Openblas: Still can't compile 0.3.7 with PGI (19.10)

[christopherwharrop-noaa]

As reported in issues #2236 and #2237 version 0.3.7 has problems with PGI.

Now that I have access to PGI 19.10, which fixes the MMX instructions problems, I tried again to build 0.3.7 after applying changes in #2223 (#2240 is no longer needed due to fixes in 19.10). I have tried both the llvm and nollvm compilers. They both produce the same result.

Now, the compile fails with (for example):

pgcc -c -O2 -DMAX_STACK_ALLOC=2048 -mp -tp p7-64 -DF_INTERFACE_PGI -fPIC -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.7\" -DASMNAME=dtrsm_iun\
ucopy -DASMFNAME=dtrsm_iunucopy_ -DNAME=dtrsm_iunucopy_ -DCNAME=dtrsm_iunucopy -DCHAR_NAME=\"dtrsm_iunucopy_\" -DCHAR_CNAME=\"dtrsm_iunucopy\" -DNO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX  -DDOUBLE -UCOMPLEX -UOU\
TER -ULOWER -DUNIT generic/trsm_uncopy_4.c -o dtrsm_iunucopy.o
<inline asm>:2:13: error: invalid operand for instruction
        addq    $128, %dl
                      ^~~
<inline asm>:5:15: error: invalid base+index expression
        vmulpd  -128(%dl), %xmm0, %xmm4

The invalid base+index expression error was mentioned in the discussion in #2237 here.

Does anyone know if a solution to the invalid base+index expression error was ever found? Or if there were other modifications found that were needed to get a clean build with PGI 19.10?

[martin-frbg]

Weird I'm fairly certain I test-build on SkylakeX when 19.10 became available. Can you try with a snapshot of the current develop branch ?

[christopherwharrop-noaa]

I just tried the current develop (8d2a796) with several compilers and the only one that succeeded was gcc-4.8.5. All the rest died with some version of undefined reference error, I think during builds for tests. It does appear, though, that the invalid base+index expression is gone.

PGI 19.10 failed with:

make -j 80 -C test all
make[1]: Entering directory `/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/test'
pgfortran -O2 -tp p7-64 -mp -fPIC -c sblat1.f  -o sblat1.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c dblat1.f  -o dblat1.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c cblat1.f  -o cblat1.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c zblat1.f  -o zblat1.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c sblat2.f  -o sblat2.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c dblat2.f  -o dblat2.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c cblat2.f  -o cblat2.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c zblat2.f  -o zblat2.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c sblat3.f  -o sblat3.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c dblat3.f  -o dblat3.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c cblat3.f  -o cblat3.o
pgfortran -O2 -tp p7-64 -mp -fPIC -c zblat3.f  -o zblat3.o
pgfortran -O2 -tp p7-64 -mp   -o zblat1 zblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
pgfortran -O2 -tp p7-64 -mp   -o sblat1 sblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
pgfortran -O2 -tp p7-64 -mp   -o cblat1 cblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
pgfortran -O2 -tp p7-64 -mp   -o dblat1 dblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
../libopenblas_haswellp-r0.3.8.dev.a(blas_server.o): In function `exec_blas':
/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/driver/others/./blas_server_omp.c:339: undefined reference to `_mp_penter'
/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/driver/others/./blas_server_omp.c:339: undefined reference to `_mp_lcpu'
/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/driver/others/./blas_server_omp.c:339: undefined reference to `_mp_ncpus'
/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/pgi/19.10/driver/others/./blas_server_omp.c:351: undefined reference to `_mp_pexit'
make[1]: *** [zblat1] Error 2
make[1]: *** Waiting for unfinished jobs....

gcc 8.3.0 and 9.2.0 failed with:

gcc -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8.dev\" -march=skylake-avx512 -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I..  -w -o linktest linktest.c ../libopenblas_skylakexp-r0.3.8.dev.so -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib64/../lib64 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib/../lib64 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib64 -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib -L/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../..  -lgfortran -lm -lgomp -lquadmath -lm -lpthread -lc   && echo OK.
/tmp/ccFAaQY2.o: In function `main':
linktest.c:(.text.startup+0x2f4e): undefined reference to `clagge_'
linktest.c:(.text.startup+0x2f55): undefined reference to `claghe_'
linktest.c:(.text.startup+0x2f5c): undefined reference to `clagsy_'
linktest.c:(.text.startup+0x2f63): undefined reference to `clahilb_'
linktest.c:(.text.startup+0x2f6a): undefined reference to `clakf2_'
linktest.c:(.text.startup+0x2f71): undefined reference to `clarge_'
linktest.c:(.text.startup+0x2f78): undefined reference to `clarnd_'
linktest.c:(.text.startup+0x2f7f): undefined reference to `claror_'
linktest.c:(.text.startup+0x2f86): undefined reference to `clarot_'
linktest.c:(.text.startup+0x2f8d): undefined reference to `clatm1_'
linktest.c:(.text.startup+0x2f94): undefined reference to `clatm2_'
linktest.c:(.text.startup+0x2f9b): undefined reference to `clatm3_'
linktest.c:(.text.startup+0x2fa2): undefined reference to `clatm5_'
linktest.c:(.text.startup+0x2fa9): undefined reference to `clatm6_'
linktest.c:(.text.startup+0x2fb0): undefined reference to `clatme_'
linktest.c:(.text.startup+0x2fb7): undefined reference to `clatmr_'
linktest.c:(.text.startup+0x2fbe): undefined reference to `clatms_'
linktest.c:(.text.startup+0x2fc5): undefined reference to `clatmt_'
linktest.c:(.text.startup+0x302e): undefined reference to `dlagge_'
linktest.c:(.text.startup+0x3035): undefined reference to `dlagsy_'
linktest.c:(.text.startup+0x303c): undefined reference to `dlahilb_'
linktest.c:(.text.startup+0x3043): undefined reference to `dlakf2_'
linktest.c:(.text.startup+0x304a): undefined reference to `dlaran_'
linktest.c:(.text.startup+0x3051): undefined reference to `dlarge_'
linktest.c:(.text.startup+0x3058): undefined reference to `dlarnd_'
linktest.c:(.text.startup+0x305f): undefined reference to `dlaror_'
linktest.c:(.text.startup+0x3066): undefined reference to `dlarot_'
linktest.c:(.text.startup+0x306d): undefined reference to `dlatm1_'
linktest.c:(.text.startup+0x3074): undefined reference to `dlatm2_'
linktest.c:(.text.startup+0x307b): undefined reference to `dlatm3_'
linktest.c:(.text.startup+0x3082): undefined reference to `dlatm5_'
linktest.c:(.text.startup+0x3089): undefined reference to `dlatm6_'
linktest.c:(.text.startup+0x3090): undefined reference to `dlatm7_'
linktest.c:(.text.startup+0x3097): undefined reference to `dlatme_'
linktest.c:(.text.startup+0x309e): undefined reference to `dlatmr_'
linktest.c:(.text.startup+0x30a5): undefined reference to `dlatms_'
linktest.c:(.text.startup+0x30ac): undefined reference to `dlatmt_'
linktest.c:(.text.startup+0x310e): undefined reference to `slagge_'
linktest.c:(.text.startup+0x3115): undefined reference to `slagsy_'
linktest.c:(.text.startup+0x311c): undefined reference to `slahilb_'
linktest.c:(.text.startup+0x3123): undefined reference to `slakf2_'
linktest.c:(.text.startup+0x312a): undefined reference to `slaran_'
linktest.c:(.text.startup+0x3131): undefined reference to `slarge_'
linktest.c:(.text.startup+0x3138): undefined reference to `slarnd_'
linktest.c:(.text.startup+0x313f): undefined reference to `slaror_'
linktest.c:(.text.startup+0x3146): undefined reference to `slarot_'
linktest.c:(.text.startup+0x314d): undefined reference to `slatm1_'
linktest.c:(.text.startup+0x3154): undefined reference to `slatm2_'
linktest.c:(.text.startup+0x315b): undefined reference to `slatm3_'
linktest.c:(.text.startup+0x3162): undefined reference to `slatm5_'
linktest.c:(.text.startup+0x3169): undefined reference to `slatm6_'
linktest.c:(.text.startup+0x3170): undefined reference to `slatm7_'
linktest.c:(.text.startup+0x3177): undefined reference to `slatme_'
linktest.c:(.text.startup+0x317e): undefined reference to `slatmr_'
linktest.c:(.text.startup+0x3185): undefined reference to `slatms_'
linktest.c:(.text.startup+0x318c): undefined reference to `slatmt_'
linktest.c:(.text.startup+0x31ee): undefined reference to `zlagge_'
linktest.c:(.text.startup+0x31f5): undefined reference to `zlaghe_'
linktest.c:(.text.startup+0x31fc): undefined reference to `zlagsy_'
linktest.c:(.text.startup+0x3203): undefined reference to `zlahilb_'
linktest.c:(.text.startup+0x320a): undefined reference to `zlakf2_'
linktest.c:(.text.startup+0x3211): undefined reference to `zlarge_'
linktest.c:(.text.startup+0x3218): undefined reference to `zlarnd_'
linktest.c:(.text.startup+0x321f): undefined reference to `zlaror_'
linktest.c:(.text.startup+0x3226): undefined reference to `zlarot_'
linktest.c:(.text.startup+0x322d): undefined reference to `zlatm1_'
linktest.c:(.text.startup+0x3234): undefined reference to `zlatm2_'
linktest.c:(.text.startup+0x323b): undefined reference to `zlatm3_'
linktest.c:(.text.startup+0x3242): undefined reference to `zlatm5_'
linktest.c:(.text.startup+0x3249): undefined reference to `zlatm6_'
linktest.c:(.text.startup+0x3250): undefined reference to `zlatme_'
linktest.c:(.text.startup+0x3257): undefined reference to `zlatmr_'
linktest.c:(.text.startup+0x325e): undefined reference to `zlatms_'
linktest.c:(.text.startup+0x3265): undefined reference to `zlatmt_'

Intel 17.0.5.239, 18.0.5.274, and 19.0.5.281 all failed during testing with

icc -O2 -DMAX_STACK_ALLOC=2048 -wd981 -fopenmp -DF_INTERFACE_INTEL -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8.dev\" -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I..  -w -o linktest linktest.c ../libopenblas_haswellp-r0.3.8.dev.so -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/daal/../tbb/lib/intel64_lin/gcc4.4 -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/daal/lib/intel64_lin -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/tbb/lib/intel64/gcc4.7 -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/mkl/lib/intel64_lin -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/compiler/lib/intel64_lin -L/apps/intel/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/daal/../tbb/lib/intel64_lin/gcc4.4/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/daal/lib/intel64_lin/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/tbb/lib/intel64/gcc4.7/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/mkl/lib/intel64_lin/ -L/apps/intel/parallel_studio_xe_2018.4.057/compilers_and_libraries_2018/linux/compiler/lib/intel64_lin/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/ -L/usr/lib64 -L/usr/lib  -lifport -lifcoremt -limf -lsvml -lm -lipgo -liomp5 -lirc -lpthread -lsvml -lc -lirc_s -ldl -lc  -lgfortran && echo OK.
../libopenblas_haswellp-r0.3.8.dev.so: undefined reference to `xerbl_'
make[1]: *** [../libopenblas_haswellp-r0.3.8.dev.so] Error 1
make[1]: Leaving directory `/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/build/OpenBLAS/8d2a796/intel/18.0.5.274/exports'
make: *** [shared] Error 2

[martin-frbg]

Your PGI seems to be looking for things from libpgmp.so but I wonder why and how it manages to
do this selectively for only some of the tests. Even less sure what to make of the other two - the gcc8/9 linktest seems to be missing all functions from the (netlib) LAPACK "matgen" helpers for the included regression tests that should be exactly the same as seen by gcc-4.8.5, Intel seems to have hit a mutilated reference to xerbla somewhere that nobody else noticed)

[christopherwharrop-noaa]

I had set USE_OPENMP=1 in my make command. I am going to remove that and try them all again to see if that makes any difference.

[martin-frbg]

BTW the trsm_uncopy_4.c that seemed to cause the assembler error should be unchanged from 0.3.7 (note the "invalid operand for instruction" would have been generated by pgcc itself from the C source !) so either the compiler options are different now (in a way we neglected to document in the earlier tickets) or your build is (sometimes?) using an older assembler (or even an older pgcc).
Anything special about that system, like network file systems mounted on demand or from a distributed storage that could be out of sync ?

[martin-frbg]

Hmm. I can force the "invalid base+index" on Haswell by removing the -D__MMX__ -Mnollvm from the CFLAGS in Makefile.system (the bits from #2240, so apparently it is not fixed completely despite https://github.com/xianyi/OpenBLAS/issues/2237#issuecomment-539250130 @cparrott73 )

[cparrott73]

I don't believe the fix was propagated to the NoLLVM compilers, but I would have to double check with our developer here. In general, the NoLLVM compilers are deprecated and will not see any new releases after 20.4 or so.

[martin-frbg]

@cparrott73 it works _with_ both options set but fails without, so it seems to be the llvm variant is the one that is (still?) having problems here.

[cparrott73]

Okay, sorry about that - I guess I misread the above. I assume you are referring to the inline assembly issue above, then? I'll probably need to file a new bug report on that one. I'll work to reproduce it here first.

[martin-frbg]

Build of current (unmodified) "develop" with the NoLLVM compiler works, but the openblas_utest binary gets built as a dummy binary with none of the tests. (I think this is a regression, but cannot be sure and would need to reinstall PGI 19.4 first to confirm). Building with the LLVM version _appeared_ to work for the most part but here the openblas_utest binary segfaults on startup in _mp_preinit2 (in what was not supposed to be an OpenMP build)

And I just confirmed that a gcc 9.2.0 build is not missing any bits from MATGEN/TESTING in linktest here. No icc on that system at the moment to reproduce any "xerbl_"

[cparrott73]

I have opened TRS#28084 on this issue against PGI 19.10 here. I will let you know when I have more information.

[martin-frbg]

Thank you very much. BTW I can also reproduce the link error (undefined references to _mp_penter etc as noted above) when trying to compile with USE_OPENMP=1. Symbols mp_penter, mp_lcpu, mp_ncpus and mp_pexit do not appear to be defined in any of the libraries installed with 19.10 ?

[christopherwharrop-noaa]

I am getting some strange inconsistent behavior. The gcc failures puzzled me so I reran them and am now seeing them succeed/fail randomly. I suspect it might be an issue with parallel make, but I'm not 100% sure. I'm running my make command with -j1 to try to eliminate build issues, but it appears that the tests use -j 80 despite my efforts:

[Christopher.W.Harrop@Hera:hfe04 xml]$ grep "make " ../log/openblas/openblas_8d2a796_gcc*.log
../log/openblas/openblas_8d2a796_gcc_4.8.5.log:+ make -j1 CC=gcc CXX=g++ FC=gfortran F90=gfortran F77=gfortran
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:+ make -j1 CC=gcc CXX=g++ FC=gfortran F90=gfortran F77=gfortran
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../slaswp_plus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../slaswp_minus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../dlaswp_plus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../dlaswp_minus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../claswp_plus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../claswp_minus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../zlaswp_plus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:cd x86_64 && make ../zlaswp_minus.o
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C SRC
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C TESTING/MATGEN
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C LAPACKE
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C src
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -C utils
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -j 80 -C test all
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -j 80 -C utest all
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -j 80 -C ctest all
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:To install the library, you can run "make PREFIX=/path/to/your/installation install".
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:+ make -j1 CC=gcc CXX=g++ FC=gfortran F90=gfortran F77=gfortran PREFIX=/scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/OpenBLAS/8d2a796/gcc/8.3.0 install
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:make -j 80 -f Makefile.install install
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:Generating OpenBLASConfig.cmake in /scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/OpenBLAS/8d2a796/gcc/8.3.0/lib/cmake/openblas
../log/openblas/openblas_8d2a796_gcc_8.3.0.log:Generating OpenBLASConfigVersion.cmake in /scratch2/BMC/gsd-hpcs/Christopher.W.Harrop/bass-develop/opt/OpenBLAS/8d2a796/gcc/8.3.0/lib/cmake/openblas

This might be affecting my other builds too. I'm going to go digging to see if I can turn off the parallel make.

[martin-frbg]

You should be able to override this in Makefile.rule if commandline or CFLAGS has no effect.

[cparrott73]

I'll take a look at the OpenMP issue here, but just FYI:

pgfortran -O2 -tp p7-64 -mp -o zblat1 zblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc pgfortran -O2 -tp p7-64 -mp -o sblat1 sblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc pgfortran -O2 -tp p7-64 -mp -o cblat1 cblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc pgfortran -O2 -tp p7-64 -mp -o dblat1 dblat1.o ../libopenblas_haswellp-r0.3.8.dev.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc

Explicitly appending the OpenMP runtime libraries on the compile line here is redundant - they are implied by -mp being present when linking. In fact, there is actually an alternate OpenMP implentation, which is available in 19.10 as libnvomp. libnvomp will be the default in the 20.x releases, so explicitly appending these libraries will break then.

If you want to test it out with 19.10, you can specify -mp=nvomp instead of just -mp on the command line.

[martin-frbg]

This is probably autodetected by c_check (which builds a trivial c file and examines the verbose output of the compiler) - perhaps we need to strip it from the list of libraries seen in the link step ?
BTW the invalid assembly messages could in part be self-inflicted as we are building with -tp p7-64 rather than tp haswell (or other cpu as appropriate) ?

[cparrott73]

-tp controls the codegen setting for the compiler. PGI is sort of the opposite of GCC, in that GCC tries to be as generic as possible unless you tell it otherwise via a -march flag. PGI tries to detect your CPU and automatically target its codegen for it, unless you tell it not to. I often have to pass -tp p7-64 on our build server (a Haswell system) to generate code for generic x86 CPUs when building applications and libraries, or I end up seeing illegal instruction errors on older CPUs like Sandy Bridge.

I believe that -tp does not apply when the compiler processes inline assembly - but I would have to ask around here to be sure.

[cparrott73]

I took a look at the unresolved symbols. I believe this is not a bug, but a symptom of mixing pgcc -Mnollvm and pgfortran (which appears to be LLVM). Note that the OpenMP runtimes for LLVM vs. NoLLVM are different, and you cannot mix the two. To fix, I would suggest either removing -Mnollvm from pgcc (which I realize is in place as a workaround for the inline assembly problem), or add -Mnollvm to pgfortran when compiling the Fortran source files.

NoLLVM uses the old PGI OpenMP library, which is libpgmp.so. Again, as NoLLVM is deprecated, we opted not to change it to use the newer OpenMP implementations.

Let me know if this is unclear.

[cparrott73]

And note if you add -Mnollvm to pgfortran, you will need to do away with the explicit OpenMP library references as mentioned above, as they will be incorrect for NoLLVM.

[martin-frbg]

I had just removed the library references (which made libpgc unhappy as it was missing kmpc_fork_call and several others), will try adding the -Mnollvm to the fortran flags. (Still getting linker errors from libpgc.so missing references to __kmpc_push_num_threads and other symbols apparently in libgomp with that. Possibly some of the auto-misdetected libraries like -lpgatm, -lnspgc need to be replaced with non-omp counterparts even ? Need to catch some sleep now.)

[martin-frbg]

Quick update:

1. I cannot get rid of the assembly errors with the LLVM flavor of the compiler no matter which -tp option I try (With LLVM proper, there used to be an option to disable its integrated assembler - I still need to check if this is available in PGI). NoLLVM does not have this problem.
2. The link errors were indeed caused by mixups between LLVM and NoLLVM library paths (probably caused by the initial compiler invocation in c_check always running with the default (llvm) option and adding its (linux-x86-64-llvm) -L path to Makefile.conf. In view of (1) the best way around this seems to be to preset CC and FC with the full paths to the NoLLVM compilers rather than relying on -Mnollvm alone
3. The "empty" openblas_utest can be worked around in the utest Makefile by setting OBJS to just utest_main2.o when the compiler is PGI (this avoids having ctest compile the list of test entry points itself and so far was necessary only with MSVC). Happily all tests then pass.

[martin-frbg]

1. Also the AVX512 code in sgemm_kernel_direct_skylakex.c fails to compile ("Illegal use of symbol __mm512 ... Type not specified, int assumed"), -tp skylake and/or -mavx512f -mavx512vl does not help. From https://www.pgroup.com/userforum/viewtopic.php?t=6686 this could be related to TPR27520 which was supposed to be fixed post 19.9 ?

[cparrott73]

Regarding #4 - we are aware of the issue. I have raised it with our developers, and unfortunately, the work to be done here to support the required AVX512 intrinsic functions has not yet found its way to the top of their list of priorities. Is there any way we could work around this issue with inline assembly in OpenBLAS for PGI for these kernels?

[martin-frbg]

Unfortunately I am away from my SkylakeX system right now, so I am not sure about the extent of the problem - if it just the functions used in sgemm_kernel_direct_skylakex (where compilation aborted), it should be possible to undefine the related parameter in param.h. Otherwise OpenBLAS needs to be built with NO_AVX512=1 which would make it use the Haswell kernels.

[peterukk]

I just tried to compile v0.3.10 on a Zen1 system with PGI compilers and got the following error when running make CC=pgcc FC=pgfortran :

pgcc -O2 -DMAX_STACK_ALLOC=2048 -tp p7-64 -D__MMX__ -Mnollvm -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.10.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=sgemm_kernel -DASMFNAME=sgemm_kernel_ -DNAME=sgemm_kernel_ -DCNAME=sgemm_kernel -DCHAR_NAME=\"sgemm_kernel_\" -DCHAR_CNAME=\"sgemm_kernel\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/x86_64/sgemm_kernel_8x4_haswell_2.c -o sgemm_kernel.o
pgcc -O2 -DMAX_STACK_ALLOC=2048 -tp p7-64 -D__MMX__ -Mnollvm -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.10.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=sgemm_incopy -DASMFNAME=sgemm_incopy_ -DNAME=sgemm_incopy_ -DCNAME=sgemm_incopy -DCHAR_NAME=\"sgemm_incopy_\" -DCHAR_CNAME=\"sgemm_incopy\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/x86_64/../generic/gemm_ncopy_8.c -o sgemm_incopy.o
PGC-S-0025-Illegal character: '\37777777757' (../kernel/x86_64/sgemm_kernel_8x4_haswell_2.c: 1)
PGC-S-0025-Illegal character: '\37777777673' (../kernel/x86_64/sgemm_kernel_8x4_haswell_2.c: 1)
PGC-S-0025-Illegal character: '\37777777677' (../kernel/x86_64/sgemm_kernel_8x4_haswell_2.c: 1)
pgcc -O2 -DMAX_STACK_ALLOC=2048 -tp p7-64 -D__MMX__ -Mnollvm -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.10.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=sgemm_itcopy -DASMFNAME=sgemm_itcopy_ -DNAME=sgemm_itcopy_ -DCNAME=sgemm_itcopy -DCHAR_NAME=\"sgemm_itcopy_\" -DCHAR_CNAME=\"sgemm_itcopy\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/x86_64/../generic/gemm_tcopy_8.c -o sgemm_itcopy.o 
pgcc -O2 -DMAX_STACK_ALLOC=2048 -tp p7-64 -D__MMX__ -Mnollvm -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.10.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=sgemm_oncopy -DASMFNAME=sgemm_oncopy_ -DNAME=sgemm_oncopy_ -DCNAME=sgemm_oncopy -DCHAR_NAME=\"sgemm_oncopy_\" -DCHAR_CNAME=\"sgemm_oncopy\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/x86_64/../generic/gemm_ncopy_4.c -o sgemm_oncopy.o
pgcc -O2 -DMAX_STACK_ALLOC=2048 -tp p7-64 -D__MMX__ -Mnollvm -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.10.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=sgemm_otcopy -DASMFNAME=sgemm_otcopy_ -DNAME=sgemm_otcopy_ -DCNAME=sgemm_otcopy -DCHAR_NAME=\"sgemm_otcopy_\" -DCHAR_CNAME=\"sgemm_otcopy\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/x86_64/../generic/gemm_tcopy_4.c -o sgemm_otcopy.o
pgcc -O2 -DMAX_STACK_ALLOC=2048 -tp p7-64 -D__MMX__ -Mnollvm -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.10.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=sgemm_beta -DASMFNAME=sgemm_beta_ -DNAME=sgemm_beta_ -DCNAME=sgemm_beta -DCHAR_NAME=\"sgemm_beta_\" -DCHAR_CNAME=\"sgemm_beta\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/x86_64/gemm_beta.S -o sgemm_beta.o
pgcc -O2 -DMAX_STACK_ALLOC=2048 -tp p7-64 -D__MMX__ -Mnollvm -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.10.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=strmm_kernel_LN -DASMFNAME=strmm_kernel_LN_ -DNAME=strmm_kernel_LN_ -DCNAME=strmm_kernel_LN -DCHAR_NAME=\"strmm_kernel_LN_\" -DCHAR_CNAME=\"strmm_kernel_LN\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -DTRMMKERNEL -UDOUBLE -UCOMPLEX -DLEFT -UTRANSA ../kernel/x86_64/sgemm_kernel_8x4_haswell.c -o strmm_kernel_LN.o
pgcc -O2 -DMAX_STACK_ALLOC=2048 -tp p7-64 -D__MMX__ -Mnollvm -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.10.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=strmm_kernel_LT -DASMFNAME=strmm_kernel_LT_ -DNAME=strmm_kernel_LT_ -DCNAME=strmm_kernel_LT -DCHAR_NAME=\"strmm_kernel_LT_\" -DCHAR_CNAME=\"strmm_kernel_LT\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -DTRMMKERNEL -UDOUBLE -UCOMPLEX -DLEFT -DTRANSA ../kernel/x86_64/sgemm_kernel_8x4_haswell.c -o strmm_kernel_LT.o
PGC/x86-64 Linux 19.10-0: compilation completed with severe errors

[martin-frbg]

Does the start of that file in your copy look the same as here ? https://github.com/xianyi/OpenBLAS/blob/develop/kernel/x86_64/sgemm_kernel_8x4_haswell_2.c
(Not sure what to make of the numeric representation, probably something got converted to UTF8 or UTF16 when it should not have been ?)

[wyphan]

I had the same issue with 0.3.9. It's an invisible Unicode character 0xFEFF ("zero width no-break space" / Unicode byte order mark). I deleted it with a hex editor (ghex) and OpenBLAS 0.3.9 compiles using the nollvm version!

Edit: I'm on a Zen2 system.
Edit 2: clarified version. OpenBLAS 0.3.10 still fails to build, see comment below

[martin-frbg]

Ha, good catch, thanks. file tells me that some of the Haswell/Zen strsm kernel files are also affected, will clean them up later when I am a bit more awake

[wyphan]

By the way, any reason why the default flags for PGI compiler requires the NoLLVM version -Mnollvm and the -tp p7-64 flag? I should probably put this in another issue, but currently I'm trying to build 0.3.10 with PGI 20.1 on Summit (POWER9) to no avail, since both flags are not applicable...

Edit: typo
Edit 2: it's issue #2718

[martin-frbg]

Simply because nobody has tried building with PGI on POWER yet, and the flags you mentioned are required to get a usable build on x86_64.
There is probably something else going wrong earlier however, as POWER architecture should get its compiler arguments from Makefile.power

[wyphan]

Ha, good catch, thanks. file tells me that some of the Haswell/Zen strsm kernel files are also affected, will clean them up later when I am a bit more awake

I'm building 0.3.10 on Zen2. It still has the unstripped version. Makes sense, since #2687 was merged right after the release 😅

Anyway, the build fails at the testing routines. Looks like the error has to deal with name mangling?

/home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:252: undefined reference to `dznrm2_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:256: undefined reference to `dzasum_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:260: undefined reference to `zscal_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:265: undefined reference to `zdscal_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:270: undefined reference to `izamax_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:293: undefined reference to `zdscal_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:306: undefined reference to `zdscal_'
/usr/bin/ld: zblat1.o: in function `check1_':
zblat1.f:(.text+0x7f4): undefined reference to `zdscal_'
/usr/bin/ld: zblat1.o: in function `check1_':
/home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:288: undefined reference to `zscal_'
/usr/bin/ld: zblat1.o: in function `check2_':
/home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:535: undefined reference to `zdotc_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:539: undefined reference to `zdotu_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:543: undefined reference to `zaxpy_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:547: undefined reference to `zcopy_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./zblat1.f:551: undefined reference to `zswap_'
make[1]: *** [Makefile:167: zblat1] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: cblat1.o: in function `check1_':
/home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:252: undefined reference to `scnrm2_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:256: undefined reference to `scasum_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:260: undefined reference to `cscal_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:265: undefined reference to `csscal_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:270: undefined reference to `icamax_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:293: undefined reference to `csscal_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:306: undefined reference to `csscal_'
/usr/bin/ld: cblat1.o: in function `check1_':
cblat1.f:(.text+0x7c6): undefined reference to `csscal_'
/usr/bin/ld: cblat1.o: in function `check1_':
/home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:288: undefined reference to `cscal_'
/usr/bin/ld: cblat1.o: in function `check2_':
/home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:535: undefined reference to `cdotc_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:539: undefined reference to `cdotu_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:543: undefined reference to `caxpy_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:547: undefined reference to `ccopy_'
/usr/bin/ld: /home/wyp/compile/OpenBLAS-0.3.10/test/./cblat1.f:551: undefined reference to `cswap_'
make[1]: *** [Makefile:164: cblat1] Error 2

Make was invoked with make CC=pgcc FC=pgfortran USE_OPENMP=1
Full build log here

Edit: add Zen2 build log

[cparrott73]

Try the attached patch. It should work if you aren't using OpenMP.

If you are using OpenMP, you will need to recompile a handful of C files containing OpenMP directives with pgcc -mp, e.g. driver/others/blas_server_omp.c.

I actually don't have this in my patch anymore, as our current development compilers will actually support the GOMP OpenMP interface when they are released, making such recompilations unnecessary. (i.e. you can compile the C OpenMP files with gcc -fopenmp and they will work just fine with our compilers in the near future.)

Note the changes to f_check are probably not needed for the current release compilers, but our current development compilers produce objects that are more GNU-like, and cause f_check to incorrectly conclude they need the G77 interface. This patch includes a fix for that.

oblas.diff.txt

[cparrott73]

Oh, that f_check fix might be problematic for existing releases. We added support for -fopenmp in the compilers under development, which is aliased to -mp. You might need to change it to the latter if the release you have chokes on the former.

[wyphan]

In addition to the above patch (which has been merged as #2727), looks like one of the recent pull requests linked to #2718 fixed the compilation problem. It might have been #2731. The current development version compiles successfully on Zen2! As before, I used make CC=pgcc FC=pgfortran USE_OPENMP=1.
@peterukk Can you try compiling again?