undefined symbol: sgesdd_ and dgebal_

[eserte]

#5 does not seem to be fixed in later PDL-LinearAlgebra versions (seen the error in 0.32):

Can't load '/home/cpansand/.cpan/build/2022050621/PDL-LinearAlgebra-0.32-kd16Pj/blib/arch/auto/PDL/LinearAlgebra/Real/Real.so' for module PDL::LinearAlgebra::Real: /home/cpansand/.cpan/build/2022050621/PDL-LinearAlgebra-0.32-kd16Pj/blib/arch/auto/PDL/LinearAlgebra/Real/Real.so: undefined symbol: sgesdd_ at /usr/lib/x86_64-linux-gnu/perl/5.24/DynaLoader.pm line 187.

or

Can't load '/home/cpansand/.cpan/build/2022050623/PDL-LinearAlgebra-0.32-1/blib/arch/auto/PDL/LinearAlgebra/Real/Real.so' for module PDL::LinearAlgebra::Real: /home/cpansand/.cpan/build/2022050623/PDL-LinearAlgebra-0.32-1/blib/arch/auto/PDL/LinearAlgebra/Real/Real.so: undefined symbol: dgebal_ at /opt/perl-5.26.2D/lib/5.26.2/x86_64-linux-ld/DynaLoader.pm line 193.

Still Fedora is the only system amongst my smokers which has pass reports.

[mohawk2]

Thanks for the report! I'd really like to crack this properly.

This really doesn't look like it got added recently to LAPACK (no mention in release notes going back to 3.7.0 in Dec 2016), but - would you know which version of LAPACK you have on your various systems? Would you also be able to use nm on the LAPACK libraries in question to see if they have these, with and/or without a leading or trailing _?

• sgesdd (apparently missing)
• sgetrf (another one that's used)

Also, what ExtUtils::F77 returns for trail_?

I'm seeing the library find all relevant symbols and pass tests on Ubuntu, MacOS, and Strawberry Perl.

[eserte]

It seems that the problem happens if liblapack-dev is not installed. In the end the *.so files were built without lapack linked in. I installed now liblapack-dev on my debian:11 smoker (but only there), and at least one perl there has a passing test suite. Builds for other perls are running right now.

[mohawk2]

Aha! Yes, one interesting artifact of the current setup is that P:LA does all its own declarations of (the C-ish analogue of the Fortran functions), so the lack of a -dev wouldn't hold it back. What's a bit mysterious is that with the Devel::CheckLib stuff, that's actually a test of finding the relevant libraries (it tries to run ilaenv). I don't understand how that passed. Could this be that my attempt at defeating the optimiser failed?

pdl-linearalgebra/Makefile.PL

Line 39 in 33a30c6

if (argc > 2) printf("%ld", i); /* try to stop optimiser eliminating */

In any case, do the builds work now? I'd like to avoid what seems to me a false positive before (of the presence of LAPACK), so any insights you can offer will be incredibly helpful for the near-future Alien::LAPACK :-)

[KJ7LNW]

Not sure if this helps or not, but here's what I did to troubleshoot on CentOS 7 for this case:

# find files in packages that contain the missing function:
rpm -ql `rpm -qa |grep openblas` `rpm -qa |grep atlas`  | xargs grep sgesdd_ | grep matches | cut -f3 -d' ' | sort -u > /tmp/libs

# grep the missing functions from the matching libs:
for a in `cat /tmp/libs`; do readelf -Ws --dyn-syms $a | while read b ; do echo "$a: $b"; done;done | grep -P '[^_](dgebal|sgesdd)_$'

You could do something similar on Ubuntu/Debian with dpkg -L <pkgname> instead of rpm -ql <pkgname> for listing files.

The rpm out looks like this:

/usr/lib64/atlas/libsatlas.so: 5948: 00000000001db4c0  2944 FUNC    GLOBAL DEFAULT   11 dgebal_
/usr/lib64/atlas/libsatlas.so: 6136: 000000000032b8a0 21223 FUNC    GLOBAL DEFAULT   11 sgesdd_
/usr/lib64/atlas/libsatlas.so.3: 5948: 00000000001db4c0  2944 FUNC    GLOBAL DEFAULT   11 dgebal_
/usr/lib64/atlas/libsatlas.so.3: 6136: 000000000032b8a0 21223 FUNC    GLOBAL DEFAULT   11 sgesdd_
/usr/lib64/atlas/libsatlas.so.3.10: 5948: 00000000001db4c0  2944 FUNC    GLOBAL DEFAULT   11 dgebal_
/usr/lib64/atlas/libsatlas.so.3.10: 6136: 000000000032b8a0 21223 FUNC    GLOBAL DEFAULT   11 sgesdd_
/usr/lib64/atlas/libtatlas.so: 5975: 00000000001f4480  2944 FUNC    GLOBAL DEFAULT   11 dgebal_
/usr/lib64/atlas/libtatlas.so: 6172: 0000000000344860 21223 FUNC    GLOBAL DEFAULT   11 sgesdd_
/usr/lib64/atlas/libtatlas.so.3: 5975: 00000000001f4480  2944 FUNC    GLOBAL DEFAULT   11 dgebal_
/usr/lib64/atlas/libtatlas.so.3: 6172: 0000000000344860 21223 FUNC    GLOBAL DEFAULT   11 sgesdd_
/usr/lib64/atlas/libtatlas.so.3.10: 5975: 00000000001f4480  2944 FUNC    GLOBAL DEFAULT   11 dgebal_
/usr/lib64/atlas/libtatlas.so.3.10: 6172: 0000000000344860 21223 FUNC    GLOBAL DEFAULT   11 sgesdd_
/usr/lib64/libopenblas64-r0.3.3.so: 7422: 0000000001a82b10  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblas64-r0.3.3.so: 8860: 0000000001944960 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblas64.so: 7422: 0000000001a82b10  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblas64.so: 8860: 0000000001944960 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblas64.so.0: 7422: 0000000001a82b10  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblas64.so.0: 8860: 0000000001944960 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblaso64-r0.3.3.so: 7619: 0000000001b035b0  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblaso64-r0.3.3.so: 9095: 00000000019c5360 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblaso64.so: 7619: 0000000001b035b0  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblaso64.so: 9095: 00000000019c5360 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblaso64.so.0: 7619: 0000000001b035b0  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblaso64.so.0: 9095: 00000000019c5360 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblaso-r0.3.3.so: 7617: 0000000001b0bb20  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblaso-r0.3.3.so: 9093: 00000000019ccda0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblaso.so: 7617: 0000000001b0bb20  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblaso.so: 9093: 00000000019ccda0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblaso.so.0: 7617: 0000000001b0bb20  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblaso.so.0: 9093: 00000000019ccda0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblasp64-r0.3.3.so: 7626: 0000000001b03110  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblasp64-r0.3.3.so: 9102: 00000000019c4f60 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblasp64.so: 7626: 0000000001b03110  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblasp64.so: 9102: 00000000019c4f60 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblasp64.so.0: 7626: 0000000001b03110  3110 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblasp64.so.0: 9102: 00000000019c4f60 19913 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblasp-r0.3.3.so: 7624: 0000000001b0b680  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblasp-r0.3.3.so: 9100: 00000000019cc9a0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblasp.so: 7624: 0000000001b0b680  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblasp.so: 9100: 00000000019cc9a0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblasp.so.0: 7624: 0000000001b0b680  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblasp.so.0: 9100: 00000000019cc9a0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblas-r0.3.3.so: 7420: 0000000001a83080  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblas-r0.3.3.so: 8858: 00000000019443a0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblas.so: 7420: 0000000001a83080  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblas.so: 8858: 00000000019443a0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/libopenblas.so.0: 7420: 0000000001a83080  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/libopenblas.so.0: 8858: 00000000019443a0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_
/usr/lib64/R/lib/libRblas.so: 7420: 0000000001a83080  3160 FUNC    GLOBAL DEFAULT   12 dgebal_
/usr/lib64/R/lib/libRblas.so: 8858: 00000000019443a0 19594 FUNC    GLOBAL DEFAULT   12 sgesdd_

so depending on your build then (maybe) you need to add -l<something> or just pass one of the .so's above the link line at build time. For testing just hack it into the right place in your makefile or set LD_FLAGS='/lib/libfoo.so' before building to see if it works. If it does, then maybe something more graceful can be implemented that works across platforms.

I'm guessing you want one of these:

• serial atlas: /usr/lib64/atlas/libsatlas.so.3
• threaded atlas: /usr/lib64/atlas/libtatlas.so.3
• serial openmp: /usr/lib64/libopenblas.so.0
• openmp openblas: /usr/lib64/libopenblaso.so
• pthreads openmp: /usr/lib64/libopenblasp.so

I've collected a bunch of LAPACKE/CLAPACK library names by distribution with comments and names in the mathlibs structure here if that helps find the right library:

• https://github.com/KJ7LNW/xnec2c/blob/master/src/mathlib.c#L29

[mohawk2]

Thank you for this research! Are you able to repro @eserte's results of failing to find the function, then change that to success? I'd like to get the most idiomatic way to fix this in our Makefile.PL (and in a future Alien::LAPACK), since obviously setting LD_FLAGS and hacking makefiles is not a viable option.

[KJ7LNW]

ok so this fixes it for me at build time:

LD_PRELOAD='/usr/lib64/atlas/libtatlas.so.3' make test

[KJ7LNW]

@eserte, do you get these errors at perl Makefile.PL?

Package 'lapack', required by 'virtual:world', not found
Package 'blas', required by 'virtual:world', not found

In Makefile.PL:10 it tries pkg-config on the libs in @pkgs = qw(lapack blas) but if those packages do not exist then it falls back to a default of '-L/usr/lib/atlas -llapack -lblas -latlas'

This fallback is probably the problem since it is unlikely to be valid when pkg-config fails. I sort of think that Makefile.PL should just fail if pkg-config can't find the libraries. Also it might be best if @pkgs was evaluated in order. Collect any errors and fail to finish Makefile.PL with the error report.

I'm still working on a final solution that builds here, so more to come...

[KJ7LNW]

Ok, so here are several solutions that build for me on Oracle Linux 8 (RHEL 8, etc). Here is what I found:

1. So far as I can tell, the @pkgs list in Makefile.PL don't do anything because all the .so's that need linked are in Real/ Trans/ and Complex/. Maybe it does something in win32? or maybe something needs to be done so the root Makefile.PL passes its library choices to the subdir Makefile.PL's?
2. The {Real,Trans,Complex}/Makefile.PL files do not define libraries in %ldloadlibs, but they need to. I tried each of these and they work for me but unfortunately they are distro specific:
   • %ldloadlibs = (LIBS => `pkg-config atlas --libs`); # distro default, probly serial
   • %ldloadlibs = (LIBS => -lsatlas); # serial atlas
   • %ldloadlibs = (LIBS => -ltatlas); # threaded atlas
   • %ldloadlibs = (LIBS => '-lopenblaso'); # OpenMP
   • %ldloadlibs = (LIBS => '-lopenblasp'); # pthreads
   • %ldloadlibs = (LIBS => '-lopenblas'); # serial
   • %ldloadlibs = (LIBS => '-llapack'); # old slow blas

So this is where my MakeMaker knowledge breaks down. Really what you want is to search for libraries in performance order and only choose one.

I'm mocking up a PR for comment.

[KJ7LNW]

make test

t/1.t ....... ok     
t/cgtsv.t ... ok    
t/gtsv.t .... ok     
t/legacy.t .. ok     
All tests successful.
Files=4, Tests=410,  3 wallclock secs ( 0.07 usr  0.01 sys +  2.59 cusr  0.13 csys =  2.80 CPU)
Result: PASS

[mohawk2]

We're discussing this on IRC, but I wanted to correct a misapprehension here for future generations.

The {Real,Trans,Complex}/Makefile.PL files do not define libraries in %ldloadlibs, but they need to.

They do not need to, as that is not how subdirs' Makefile.PLs work. They are each called in turn by EUMM as a result of the top-level M.PL's WriteMakefile(), in the same process, so the various global variables (and defined functions) are available there.

[KJ7LNW]

Thanks for the correction. Commit message and branch have been updated: KJ7LNW@6f0742d