Openblas: 0.3.7 fails to compile with PGI 19.4

[christopherwharrop-noaa]

After adding HOSTCC=cc I was able to build 0.3.7 with PGI version 18.10. However, for PGI version 19.4, the build fails with compiler error:

pgcc -c -O2 -DMAX_STACK_ALLOC=2048 -mp -tp p7-64 -DF_INTERFACE_PGI -fPIC -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.7\" -DASMNAME=drot_k -DASMFNAME=drot_k_ -DNAME=drot_k_ -DCNAME=drot_k -DCHAR_NAME=\"drot_k_\" -DCHAR_CNAME=\"drot_k\" -DNO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX -UCOMPLEX -UCOMPLEX -DDOUBLE  ../kernel/x86_64/rot_sse2.S -o drot_k.o
PGC-F-0249-#error --  "MMX instruction set not enabled" (/apps/pgi/linux86-64-llvm/19.4/include/pgi/mmintrin.h: 27)
PGC/x86-64 Linux 19.4-0: compilation aborted
make[1]: *** [sgemm_oncopy.o] Error 2
make[1]: *** Waiting for unfinished jobs....
PGC-F-0249-#error --  "MMX instruction set not enabled" (/apps/pgi/linux86-64-llvm/19.4/include/pgi/mmintrin.h: 27)
PGC/x86-64 Linux 19.4-0: compilation aborted
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/amax_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/amax_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/iamax_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/amax_sse2.S: -1)
make[1]: *** [sgemm_beta.o] Error 2
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/amax_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/copy_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/iamax_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/iamax_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/iamax_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/rot_sse2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/nrm2.S: -1)
PREPRO-I-0222-Redundant definition for symbol __extension__ (../kernel/x86_64/asum_sse2.S: -1)
make[1]: Leaving directory `/path/to/OpenBLAS/0.3.7/pgi/19.4/kernel'
make: *** [libs] Error 1

My build command was:

make USE_OPENMP=1 CC=pgcc CXX=pgc++ FC=pgfortran F90=pgfortran F77=pgfortran HOSTCC=cc

Any hints would be appreciated.

[brada4]

Apparently it needs different instruction sets enabled in the way different from gcc and clang
Here is the list in whole:
https://gcc.gnu.org/onlinedocs/gcc-9.2.0/gcc/x86-Options.html#index-mmmx

[christopherwharrop-noaa]

I've tried forcing builds with -Mvect=simd and -tp skylake -fastsse. No luck. I did find this somewhat related post in a PGI forum:

https://www.pgroup.com/userforum/viewtopic.php?t=6686&p=25800

[brada4]

Thats MMX missing (Actually from filename - SSE2 that every AMD64/x86_64 supports)
Since documentation is not public - ask their support what to do with that error. Sort of source file with full command is in your hands.

[martin-frbg]

You need to build OpenBLAS with the "nollvm" version of PGI at least for now - seems the llvm-based one does not handle the full power8 instruction set

[martin-frbg]

See also #2217

[cparrott73]

You need to build OpenBLAS with the "nollvm" version of PGI at least for now - seems the llvm-based one does not handle the full power8 instruction set

Unfortunately, the PGI compilers for POWER are LLVM-only. There is no "nollvm" variant for POWER, only x86_64.

[martin-frbg]

sorry for any confusion - the "power8" probably crept in as I had just read a comment related to build testing on that architecture, this should obviously have been x86_64 above

[cparrott73]

I have filed a bug on the PGI compilers for this particular issue, as I can reproduce it here.

Note that this issue even exists with 18.10, but you did not notice it there, as 18.10 was the last version of PGI in which the "nollvm" compilers were the default on x86_64. If you explicitly use the 18.10 "llvm" compilers instead, you will likely see this same issue there, as well.

[cparrott73]

Just spoke a developer, who fixed the issue internally last week. Fix should appear in an upcoming PGI release very soon.

[christopherwharrop-noaa]

Thank you all for your assistance. I will try out the "nollvm" version and let you know how it goes. Glad to hear a fix from PGI is on the way.

[christopherwharrop-noaa]

I tried using the nollvm compiler along with the two fixes from #2219 and was unsuccessful. I used the nollvm compiler binaries directly along with the -Mnollvm option. It's still giving me problems. Here is an example:

$ /apps/pgi/linux86-64-nollvm/19.4/bin/pgcc -Mnollvm -O2 -DMAX_STACK_ALLOC=2048 -mp -tp haswell -DF_INTERFACE_PGI -fPIC -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.7\" -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I. -O2 -DMAX_STACK_ALLOC=2048 -mp -tp haswell -DF_INTERFACE_PGI -fPIC -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=80 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.7\" -DASMNAME=dgemm_beta -DASMFNAME=dgemm_beta_ -DNAME=dgemm_beta_ -DCNAME=dgemm_beta -DCHAR_NAME=\"dgemm_beta_\" -DCHAR_CNAME=\"dgemm_beta\" -DNO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX -c -DDOUBLE -UCOMPLEX ../kernel/x86_64/dgemm_beta_skylakex.c -o dgemm_beta.o
PGC-W-0221-Redefinition of symbol ASMNAME (../kernel/x86_64/dgemm_beta_skylakex.c: -1)
PGC-W-0221-Redefinition of symbol ASMFNAME (../kernel/x86_64/dgemm_beta_skylakex.c: -1)
PGC-W-0221-Redefinition of symbol NAME (../kernel/x86_64/dgemm_beta_skylakex.c: -1)
PGC-W-0221-Redefinition of symbol CNAME (../kernel/x86_64/dgemm_beta_skylakex.c: -1)
PGC-W-0221-Redefinition of symbol CHAR_NAME (../kernel/x86_64/dgemm_beta_skylakex.c: -1)
PGC-W-0221-Redefinition of symbol CHAR_CNAME (../kernel/x86_64/dgemm_beta_skylakex.c: -1)
PGC-F-0249-#error --  "MMX instruction set not enabled" (/apps/pgi/linux86-64-nollvm/19.4/include/mmintrin.h: 27)
PGC/x86-64 Linux 19.4-0: compilation aborted
$

As you can see I changed it to use -tp haswell. It fails with the default -tp p7-64 as well. The _skylakex.c file extensions in the warnings above happen regardless of the -tp setting. I am indeed trying to build on a skylake system, so maybe that is adding some additional complexity. Using -tp skylake doesn't help either.

[brada4]

Error 0249 should go to PGI. Absent MMX in cone without any MMX is sort of dead end.

[martin-frbg]

Sorry, I cannot reproduce my "assumed" earlier success with the changes from #2219 either, seems I had some ugly hack in place. :-(
While I assume that the "MMX not enabled" thing is what cparrott73 already reported to PGI, a sufficient workaround appears to be to add -D__MMX__ to the PGI defines in Makefile.system.
(And I also found out that my recent zdot.c "improvement" from #2222 does not work with PGI - nor the Sun compiler - due to their implementations of the CREAL and CIMAG macros. Luckily this only affects the current develop branch and not 0.3.7...)

[martin-frbg]

NB the "llvm" build still fails at compiling files like dscal_microk_haswell-2.c as it mishandles some inline assembly: it substitutes "%dl" in instructions like " "vmovups %%xmm5 ,-112(%1) " and subsequently complains that "-112(%dl)" is an "invalid base+offset expression".

[martin-frbg]

Also, the SkylakeX checks currently in OpenBLAS are gcc-centric (which is why your log snippet shows "-DNO_AVX512" which will make this a "Haswell" build). I have patches for this, but AVX512 support in 19.4 appears to be very sketchy anyway - I get "illegal use of symbol, __m512" when I try to compile the SkylakeX SGEMM kernel with it. (see also https://www.pgroup.com/userforum/viewtopic.php?t=6686, the TPR mentioned there is not on the list of issues fixed in 19.7)

[christopherwharrop-noaa]

Thank you! After applying fixes in #2223 and #2240 and using the PGI 19.4 nollvm compilers directly, I have achieved a successful build. Hopefully PGI will address its inline assembly issues soon so that we don't have to continue with these gymnastics.

Thank you everyone for your help.

[christopherwharrop-noaa]

FYI, if I add

FCOMMON_OPT += -tp p7-64 -Mnollvm

to line 851 of Makefile.system and use the "llvm" compilers, I get similar behavior that you reported in #2217 . It builds fine, but dies during testing. In my case, the sblat1 executable seg faults.

make -j 80 -C test all
make[1]: Entering directory `/scratch2/BMC/gsd-hpcs/bass/build/OpenBLAS/0.3.7/pgi/19.4-nollvm/test'
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c sblat1.f  -o sblat1.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c dblat1.f  -o dblat1.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c cblat1.f  -o cblat1.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c zblat1.f  -o zblat1.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c sblat2.f  -o sblat2.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c dblat2.f  -o dblat2.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c cblat2.f  -o cblat2.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c zblat2.f  -o zblat2.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c sblat3.f  -o sblat3.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c dblat3.f  -o dblat3.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c cblat3.f  -o cblat3.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp -fPIC -c zblat3.f  -o zblat3.o
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp   -o cblat1 cblat1.o ../libopenblas_haswellp-r0.3.7.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.4/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp   -o zblat1 zblat1.o ../libopenblas_haswellp-r0.3.7.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.4/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp   -o dblat1 dblat1.o ../libopenblas_haswellp-r0.3.7.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.4/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
/apps/pgi/linux86-64-llvm/19.4/bin/pgfortran -O2 -tp p7-64 -Mnollvm -mp   -o sblat1 sblat1.o ../libopenblas_haswellp-r0.3.7.a -lm -lpthread -lm -lpthread -L/apps/pgi/linux86-64-llvm/19.4/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64  -lpgatm -lpgkomp -lomptarget -lomp -lpthread -lpgmath -lnspgc -lpgc -lm -lc  
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./sblat1
make[1]: *** [level1] Segmentation fault (core dumped)
make[1]: *** Waiting for unfinished jobs....

But, at least I have a working recipe now using the "nollvm" compilers.

[cparrott73]

FYI - the "instruction set not enabled" error should now be fixed as of the PGI 19.9 release. If you are using the PGI CE, the fix should be available when the PGI 19.10 CE is released.

[martin-frbg]

I see that PGI Community Edition 19.10 was released yesterday, so this issue should now be resolved for all users.