Error while using ma27 solver #681

wissamkafa · 2023-07-19T12:15:25Z

wissamkafa
Jul 19, 2023

Hi,
I have a python program that uses CasADi and IPOPT.
The code runs as required sith mumps solvers, but when i tried to use ma27, i have some problems.
I get the HSL packages from their offecial website.
extracted and compiled them follwoing the steps in CasADi
https://github.com/casadi/casadi/wiki/Obtaining-HSL
"
extraction folder: /home/user/ThirdParty-HSL/coinhsl

./configure --prefix=/usr/local LIBS="-llapack" --with-blas="-L/usr/lib -lblas" CXXFLAGS="-g -O2 -fopenmp" FCFLAGS="-g -O2 -fopenmp" CFLAGS="-g -O2 -fopenmp"
make
make install
"

then I created the symbolic link to obtain libhsl.so as required
ln -s /usr/local/lib/libcoinhsl.so /usr/local/lib/libhsl.so

then to make IOPOT see the solvers i added the environment variable using
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"
when using: echo $LD_LIBRARY_PATH i get:
:/usr/local/lib:/home/user/ThirdParty-HSL/coinhsl/lib

now when using ('nlpsol_opts': {'ipopt.linear_solver': 'MA27'}) within the code to use ma27 i got them message:
Exception of type: DYNAMIC_LIBRARY_FAILURE in file "/work/build/external_projects/src/ipopt-external/src/Common/IpLibraryLoader.cpp" at line 72:
Exception message: libhsl.so: cannot open shared object file: No such file or directory
EXIT: Library loading failure.

It seems that the paths could be found but it could not load the library. I don't know whya and I almost tried everything, but did not work.

I am using ubuntu 22.04
python 3.10 - Anaconda
pycharm environment
CasADi 3.6.3
and I think all required packages and compilers are installed as required.

I have been working to overcome this problem but with no success, and it is now driving me craizy

could you please advise me what to do? and if some one has similar problem how he solved it?

svigerske · 2023-07-19T12:38:20Z

svigerske
Jul 19, 2023
Maintainer

Maybe set LD_DEBUG=all (https://man7.org/linux/man-pages/man8/ld.so.8.html) and hope for some useful output, e.g., which paths the dynamic library loader has tried.

My guess is that LD_LIBRARY_PATH is not set in the environment from which CasADi is run.

4 replies

wissamkafa Jul 19, 2023
Author

Thanks for your reply.

Actually, I tried explicitly putting LD_LIBRARY_PATH in ~/.bashrc using nano editor from terminal
and also I tried (sudo gedit /etc/ld.so.conf.d/randomLibs.conf) and put it inside
Also, I now used LD_DEBUG=all
but with the same result

do you have any ideas on how could I check that "LD_LIBRARY_PATH is not set in the environment from which CasADi is run"? or how to make them in the same environment?

svigerske Jul 19, 2023
Maintainer

LD_DEBUG=all alone won't fix anything, it should print additional information that may help you to diagnose the problem. If you don't get any additional output, you have not set it in the environment where the application is run.

Similarly, if you just changed ~/.bashrc, then this change will be active to any new shell. Did you start your program from a new shell?

Further, if you edited /etc/ld.so.conf.d, I think you need to run ldconfig for the changes to take effect.

wissamkafa Jul 19, 2023
Author

Actually, when I execute this "LD_DEBUG=all" from the terminal I just get nothing. I am afraid that I am not following what you mean by (you have not set it in the environment where the application is run) - I am not so experienced with Linux and dealing with such environment commands.
can you please explain this(you have not set it in the environment where the application is run), do you mean by application is the Python code I am executing?

for the ~/.bashrc and /etc/ld.so.conf.d I already did both steps you mentioned.

svigerske Jul 19, 2023
Maintainer

There is a reason I also included a link to the documentation, so you won't have to google it.

So if your python program is mypythonprogram.py and you run this via python mypythonprogram.py, then now run this via LD_DEBUG=all python mypythonprogram.py.
Somewhere before the DYNAMIC_LIBRARY_FAILURE exception, there should be some output about what the system tried to find libhsl.so.

Probably LD_DEBUG=libs is sufficient.

wissamkafa · 2023-07-19T14:01:05Z

wissamkafa
Jul 19, 2023
Author

Thanks for instructions:

I have a project with multiple files and other imported libraries and usually I run it within pycharm.
Now, from terminal I used "LD_DEBUG=libs python3 main.py", as mian.py is the main file in the project (it includes import casadi).
i get long output and but I could not find in that outbut any mention to libhsl.so (also when i used LD_DEBUG=all)
also it seems that it could not find casadi (ModuleNotFoundError: No module named 'casadi'). "but it works from pycharm"
if necessary I could post the output here

here is a sample from the output, it looks like searching in /home/user/ThirdParty-HSL/coinhsl/lib

64256: search path=glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:tls/haswell/x86_64:tls/haswell:tls/x86_64:tls:haswell/x86_64:haswell:x86_64::/home/user/ThirdParty-HSL/coinhsl/lib (LD_LIBRARY_PATH)
64256: trying file=glibc-hwcaps/x86-64-v3/libpthread.so.0
64256: trying file=glibc-hwcaps/x86-64-v2/libpthread.so.0
64256: trying file=tls/haswell/x86_64/libpthread.so.0
64256: trying file=tls/haswell/libpthread.so.0
64256: trying file=tls/x86_64/libpthread.so.0
64256: trying file=tls/libpthread.so.0
64256: trying file=haswell/x86_64/libpthread.so.0
64256: trying file=haswell/libpthread.so.0
64256: trying file=x86_64/libpthread.so.0
64256: trying file=libpthread.so.0
64256: trying file=/home/user/ThirdParty-HSL/coinhsl/lib/libpthread.so.0
64256: search cache=/etc/ld.so.cache
64256: trying file=/lib/x86_64-linux-gnu/libpthread.so.0
64256:
64256: find library=libgfortran-040039e1.so.5.0.0 [0]; searching
64256: search path=/usr/local/lib/python3.10/dist-packages/numpy/core/../../numpy.libs(RPATH from file /usr/local/lib/python3.10/dist-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so)
64256: trying file=/usr/local/lib/python3.10/dist-packages/numpy/core/../../numpy.libs/libgfortran-040039e1.so.5.0.0

5 replies

svigerske Jul 19, 2023
Maintainer

If it cannot find casadi, then I doubt it ever ran ipopt, so it wouldn't get to the point where it tries to load the HSL lib.

If you cannot reproduce the original problem when running from the command line, you won't be able to analyze it. So maybe fix up your Python package search path. Or find out how to debug with pycharm.

wissamkafa Jul 20, 2023
Author

I reproduced the problem using the code from CasADi GitHub on the link:
https://github.com/casadi/casadi/blob/2c8cb5510802135a8c4b094e7ee8c1a8f54c3d94/docs/examples/python/dae_multiple_shooting.py#L103
I also removed the CasADi package and uninstalled it using "pip install casadi"

then debugging from the terminal using "LD_DEBUG=libs": the code runs but with Error:
EXIT: Library loading failure.
solver : t_proc (avg) t_wall (avg) n_eval
total | 13.17ms ( 13.17ms) 855.89us (855.89us) 1

And from the output in the terminal, I think it is searching for the 'libhsl.so' in the path "/home/user/ThirdParty-HSL/coinhsl/lib"
which already exists (it was created after compiling the HSL packages by creating a symbolic link with the file 'libcoinhsl.so')

here is the output before losing library failure:

 29903:	find library=libhsl.so [0]; searching
 29903:	 search path=/usr/local/lib/python3.10/dist-packages/casadi:/usr/local/lib/python3.10/dist-packages/casadi/./glibc-hwcaps/x86-64-v3:/usr/local/lib/python3.10/dist-packages/casadi/./glibc-hwcaps/x86-64-v2:/usr/local/lib/python3.10/dist-packages/casadi/./tls/haswell/x86_64:/usr/local/lib/python3.10/dist-packages/casadi/./tls/haswell:/usr/local/lib/python3.10/dist-packages/casadi/./tls/x86_64:/usr/local/lib/python3.10/dist-packages/casadi/./tls:/usr/local/lib/python3.10/dist-packages/casadi/./haswell/x86_64:/usr/local/lib/python3.10/dist-packages/casadi/./haswell:/usr/local/lib/python3.10/dist-packages/casadi/./x86_64:/usr/local/lib/python3.10/dist-packages/casadi/.		(RPATH from file /usr/local/lib/python3.10/dist-packages/casadi/_casadi.so)
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./glibc-hwcaps/x86-64-v3/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./glibc-hwcaps/x86-64-v2/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./tls/haswell/x86_64/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./tls/haswell/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./tls/x86_64/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./tls/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./haswell/x86_64/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./haswell/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./x86_64/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./libhsl.so
 29903:	 search path=/usr/local/lib/python3.10/dist-packages/casadi:/usr/local/lib/python3.10/dist-packages/casadi/.		(RPATH from file /usr/local/lib/python3.10/dist-packages/casadi/_casadi.so)
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/libhsl.so
 29903:	  trying file=/usr/local/lib/python3.10/dist-packages/casadi/./libhsl.so
 29903:	 search path=glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:tls/haswell/x86_64:tls/haswell:tls/x86_64:tls:haswell/x86_64:haswell:x86_64::/home/user/ThirdParty-HSL/coinhsl/lib		(LD_LIBRARY_PATH)
 29903:	  trying file=glibc-hwcaps/x86-64-v3/libhsl.so
 29903:	  trying file=glibc-hwcaps/x86-64-v2/libhsl.so
 29903:	  trying file=tls/haswell/x86_64/libhsl.so
 29903:	  trying file=tls/haswell/libhsl.so
 29903:	  trying file=tls/x86_64/libhsl.so
 29903:	  trying file=tls/libhsl.so
 29903:	  trying file=haswell/x86_64/libhsl.so
 29903:	  trying file=haswell/libhsl.so
 29903:	  trying file=x86_64/libhsl.so
 29903:	  trying file=libhsl.so
 29903:	  trying file=/home/user/ThirdParty-HSL/coinhsl/lib/libhsl.so
 29903:	
 29903:	find library=libgfortran.so.5 [0]; searching
 29903:	 search path=glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:tls/haswell/x86_64:tls/haswell:tls/x86_64:tls:haswell/x86_64:haswell:x86_64::/home/user/ThirdParty-HSL/coinhsl/lib		(LD_LIBRARY_PATH)
 29903:	  trying file=glibc-hwcaps/x86-64-v3/libgfortran.so.5
 29903:	  trying file=glibc-hwcaps/x86-64-v2/libgfortran.so.5
 29903:	  trying file=tls/haswell/x86_64/libgfortran.so.5
 29903:	  trying file=tls/haswell/libgfortran.so.5
 29903:	  trying file=tls/x86_64/libgfortran.so.5
 29903:	  trying file=tls/libgfortran.so.5
 29903:	  trying file=haswell/x86_64/libgfortran.so.5
 29903:	  trying file=haswell/libgfortran.so.5
 29903:	  trying file=x86_64/libgfortran.so.5
 29903:	  trying file=libgfortran.so.5
 29903:	  trying file=/home/user/ThirdParty-HSL/coinhsl/lib/libgfortran.so.5
 29903:	 search cache=/etc/ld.so.cache
 29903:	  trying file=/lib/x86_64-linux-gnu/libgfortran.so.5
 29903:	
 29903:	find library=libquadmath.so.0 [0]; searching
 29903:	 search path=glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:tls/haswell/x86_64:tls/haswell:tls/x86_64:tls:haswell/x86_64:haswell:x86_64::/home/user/ThirdParty-HSL/coinhsl/lib		(LD_LIBRARY_PATH)
 29903:	  trying file=glibc-hwcaps/x86-64-v3/libquadmath.so.0
 29903:	  trying file=glibc-hwcaps/x86-64-v2/libquadmath.so.0
 29903:	  trying file=tls/haswell/x86_64/libquadmath.so.0
 29903:	  trying file=tls/haswell/libquadmath.so.0
 29903:	  trying file=tls/x86_64/libquadmath.so.0
 29903:	  trying file=tls/libquadmath.so.0
 29903:	  trying file=haswell/x86_64/libquadmath.so.0
 29903:	  trying file=haswell/libquadmath.so.0
 29903:	  trying file=x86_64/libquadmath.so.0
 29903:	  trying file=libquadmath.so.0
 29903:	  trying file=/home/user/ThirdParty-HSL/coinhsl/lib/libquadmath.so.0
 29903:	 search cache=/etc/ld.so.cache
 29903:	  trying file=/lib/x86_64-linux-gnu/libquadmath.so.0
 29903:	
 29903:	/home/user/ThirdParty-HSL/coinhsl/lib/libhsl.so: error: symbol lookup error: undefined symbol: isamax_ (fatal)

Exception of type: DYNAMIC_LIBRARY_FAILURE in file "/work/build/external_projects/src/ipopt-external/src/Common/IpLibraryLoader.cpp" at line 72:
Exception message: /home/user/ThirdParty-HSL/coinhsl/lib/libhsl.so: undefined symbol: isamax_

EXIT: Library loading failure.
solver : t_proc (avg) t_wall (avg) n_eval
total | 13.17ms ( 13.17ms) 855.89us (855.89us)

there is one line indicating a problem
/home/user/ThirdParty-HSL/coinhsl/lib/libhsl.so: error: symbol lookup error: undefined symbol: isamax_ (fatal)
but I could not know what does it mean and found nothing about such error when googling it
any ideas?

svigerske Jul 20, 2023
Maintainer

isamax is a Lapack function. Probably your HSL lib misses to link against Lapack, look at the output of ldd /home/user/ThirdParty-HSL/coinhsl/lib/libhsl.so whether that lists lapack.

The instructions you use are from the casadi wiki. I cannot really say much about it. My suggestion would have been to use the instructions from https://github.com/coin-or-tools/ThirdParty-HSL to build the HSL lib.

wissamkafa Jul 20, 2023
Author

Actually, I tried both compiling methods, but with the same results.
when using 'ldd' I get:

ldd /home/user/ThirdParty-HSL/coinhsl/lib/libcoinhsl.so

linux-vdso.so.1 (0x00007ffd45975000)
libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f74f8400000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f74f87df000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f74f8000000)
libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f74f8797000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f74f8777000)
/lib64/ld-linux-x86-64.so.2 (0x00007f74f8926000)

it seems that it does not list Lapack.
but when trying to install Lapack, it appears I have the latest version.

sudo apt-get install libblas-dev liblapack-dev

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libblas-dev is already the newest version (3.10.0-2ubuntu1).
liblapack-dev is already the newest version (3.10.0-2ubuntu1).

I have been stacking with this issue for about two weeks and honestly, I am now thinking to give up.
should using a library be so complicated?

svigerske Nov 1, 2023
Maintainer

Difficult to say what is happening without seeing any logs.

In the very beginning, you had configure flags LIBS="-llapack" --with-blas="-L/usr/lib -lblas". Try removing these. Maybe the LIBS=-llapack creates confusion. If automatic detection of lapack/blas flags doesn't work, then use the --with-lapack-lflags argument to specify linker flags for lapack.

A last resort could also be to set LD_PRELOAD to the lapack library.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error while using ma27 solver #681

{{title}}

Replies: 2 comments 9 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Error while using ma27 solver #681

wissamkafa Jul 19, 2023

Replies: 2 comments · 9 replies

svigerske Jul 19, 2023 Maintainer

wissamkafa Jul 19, 2023 Author

svigerske Jul 19, 2023 Maintainer

wissamkafa Jul 19, 2023 Author

svigerske Jul 19, 2023 Maintainer

wissamkafa Jul 19, 2023 Author

svigerske Jul 19, 2023 Maintainer

wissamkafa Jul 20, 2023 Author

here is the output before losing library failure:

EXIT: Library loading failure. solver : t_proc (avg) t_wall (avg) n_eval total | 13.17ms ( 13.17ms) 855.89us (855.89us)

svigerske Jul 20, 2023 Maintainer

wissamkafa Jul 20, 2023 Author

svigerske Nov 1, 2023 Maintainer

wissamkafa
Jul 19, 2023

Replies: 2 comments 9 replies

svigerske
Jul 19, 2023
Maintainer

wissamkafa Jul 19, 2023
Author

svigerske Jul 19, 2023
Maintainer

wissamkafa Jul 19, 2023
Author

svigerske Jul 19, 2023
Maintainer

wissamkafa
Jul 19, 2023
Author

svigerske Jul 19, 2023
Maintainer

wissamkafa Jul 20, 2023
Author

EXIT: Library loading failure.
solver : t_proc (avg) t_wall (avg) n_eval
total | 13.17ms ( 13.17ms) 855.89us (855.89us)

svigerske Jul 20, 2023
Maintainer

wissamkafa Jul 20, 2023
Author

svigerske Nov 1, 2023
Maintainer