Openblas: Patches for OpenBLAS 0.3.12 to support NVIDIA HPC SDK Compilers release 20.11 and later
Good news! I have been working with our developers to clean up a number of the residual issues involved with compiling OpenBLAS with the NVIDIA HPC SDK Compilers, and as of the 20.11 release (due out shortly) - there are only a small number of changes needed against the OpenBLAS 0.3.12 source tree to allow it to compile it out of the box with our compilers.
Summary of these changes:
-Mnollvmis no longer needed to compile OpenBLAS with our compilers. In fact, the old NoLLVM compilers do not even exist anymore. This flag no longer has any effect and should be removed from the OpenBLAS build with modern releases of our compilers.The
-D__MMX__flag used to work around a previous issue is also no longer needed. That issue has been fixed.f_checkneeds a small tweak to correctly detectnvfortranas a PGI variant, and thus switch in the appropriate calling conventions for complex numbers.In order to compile kernels that use
_mm256and_mm512intrinsics with our compilers, you have to pass the-mavx512fflag. Otherwise, these intrinsics are not enabled in our compilers by default. I have modified the SKYLAKEX and COOPERLAKE build rules inkernel/Makefileto pass this flag. I also tested this flag with GCC 10.2.0, and confirmed that it does not adversely affect compiling OpenBLAS with GCC.
Note I built OpenBLAS 0.3.12 with these patches with our 20.11 compilers as follows:
make CC=nvc FC=nvfortran HOSTCC=gcc DYNAMIC_ARCH=1 USE_OPENMP=1 NUM_THREADS=512 all
Please give these patches a test with our 20.11 compilers once they are released, and if you agree they work well for you, consider integrating them into the OpenBLAS source tree ahead of the next OpenBLAS release. I think they will greatly help our customers who also use OpenBLAS. Cheers!
cparrott73
All 72 comments
Update - there is some internal discussion that -march=skylake-avx512 should imply -mavx512f automatically, as it apparently does in GCC. We may try to fix this ahead of the next release. In that case, the patches that add this flag will no longer be necessary.
cparrott73
on 7 Dec 2020
Version detection would be handy, mm256 iz like AVX2 thing?
brada4
on 8 Dec 2020
@brada4 - I am not sure I really understand what you mean here. OpenBLAS already has the ability to dynamically detect which kind of CPU is present on a given system, and then switch in the appropriate tuned BLAS kernels for that CPU at runtime. This feature is enabled with the DYNAMIC_MODE=1 build flag. Only kernels tuned for Haswell and later processors would use the AVX2 intrinsics here, such as _mm256. DYNAMIC_MODE=1 causes the kernels for every supported CPU to be built, so some kernels built with this will require _mm256 and _mm512 intrinsics, while others will not.
cparrott73
on 8 Dec 2020
I have an additional set of patches to compile OpenBLAS 0.3.12 for OpenPOWER architecture systems with the NVIDIA HPC compilers release 20.11. Here is a summary of the changes:
In
kernel/arm/zdot.c, at lines 51-56, there is a section that is guarded by#if !defined(__PPC__). Further down at lines 76-81, there is another section that is guarded instead by#if !defined(__POWER__)instead. I am not sure if this difference is intentional. However, the result is the the NVIDIA HPC compilers for OpenPOWER define__PPC__but not__POWER__, so the first section correctly conditionally compiles to theOPENBLAS_MAKE_COMPLEX_FLOAT()call, but the second section does not. I have fixed this discrepancy by changing the second section to also be guarded by#if !defined(__PPC__)instead of#if !defined(__POWER__).NVIDIA HPC compilers do not (and probably never will) support POWER6. The POWER6 kernels will compile, but they will be compiled with
-tp pwr8, meaning that they will potentially have POWER8 instructions in them. I'm not sure if that will work on a POWER6 system, and we do not have any POWER6 systems available internally to test on. Just to be on the safe side, I have removed POWER6 from the list of kernels that are included with DYNAMIC_ARCH when using the NVIDIA HPC compilers, and I also make the Makefile spit out a warning if you try to compile them explicitly.
POWER10 is also not yet supported. I do not have any information as to when or if that might happen, currently. I have conditionally removed POWER10 kernels from the dynamic arch list for NVIDIA HPC compilers as well, although perhaps a better solution here would be to just make them fall back to POWER9 instead. (Some performance is better than none, right?) In the future, should we support POWER10, we can add this back in in such a way that makes sense for our compilers.
- In
driver/others/dynamic_power.c, the NVIDIA HPC compilers do not yet support the GCC builtin function__builtin_cpu_is(). I have an open ticket internally to add support for this function, but it has not yet been implemented. For now, I have created a workaround version that is only dynamically switched in ifC_PGIis defined. This one should work the same way as the GCC version, although it only will match on POWER8 and POWER9 CPUs, as those are the only ones we support right now. This allowsget_coretype()to compile as written with our compilers.
I will suggest backing this out once we fully implement __builtin_cpu_is(), but this will work for now.
With these patches, I can build a fully working version of OpenBLAS 0.3.12 on OpenPOWER systems using the NVIDIA HPC compilers, release 20.11, as follows:
make CC=nvc FC=nvfortran HOSTCC=gcc DYNAMIC_ARCH=1 USE_OPENMP=1 NUM_THREADS=512 all
cparrott73
on 8 Dec 2020
Topic (1) is/was a typo/thinko I made a while ago and should already be fixed in current develop
(2) could probably also be "fixed" by conditional code in KERNEL.POWER6, but I guess the same compiler feature will also mean that the common code will compile to pwr8 instructions
Happy to merge these once 20.11 is released, thanks for all your work on this.
martin-frbg
on 8 Dec 2020
Topic (1) is/was a typo/thinko I made a while ago and should already be fixed in current
develop
(2) could probably also be "fixed" by conditional code in KERNEL.POWER6, but I guess the same compiler feature will also mean that the common code will compile to pwr8 instructions
Happy to merge these once 20.11 is released, thanks for all your work on this.
Wonderful about (1) - and glad I could help! Certainly if there is a better way to address (2), I'm all for it.
BTW, I am also testing 0.3.12 with our aarch64 (ARM64) compilers, but it looks like we have not yet implemented the SIMD intrinsic functions found in arm_neon.h on aarch64 for the newer vectorized kernels in 0.3.12. I am filing a ticket on this with our developers, but it probably will not be resolved soon.
cparrott73
on 8 Dec 2020
I have a patch that disables the NEON intrinsic calls when C_PGI is defined on ARM. This will at least allow the library to compile on ARM with our compilers, although we understand that the performance will be lower than GCC without the NEON vectorization in the kernels. I have opened a ticket to have these implemented in our compiler, but it will likely be a while before that happens.
I am testing the rest of the build on ARM now, and will send my patch for that up here once I confirm success.
cparrott73
on 9 Dec 2020
Also found another issue where our inline assembler cannot handle the vector load directives in zdot_thunderx2t99.c right now. I have opened a ticket on that one as well, and will modify those KERNEL.* files to fall back to zdot.S when compiling with our compilers.
cparrott73
on 9 Dec 2020
I see that 20.11 has been released now...
martin-frbg
on 13 Dec 2020
Does the same patch apply to 20.7 or 20.9?
Edit: I've downloaded and installed 20.11 on my x86_64 workstation, but haven't configured the Lmod modules yet that I wanna use with it -- things like CUDA 11.1, UCX, OpenMPI 4, and OpenBLAS 0.3.12 or newer. Unless 20.11 actually already bundles OpenMPI 4 in addition to the default OpenMPI 3 @cparrott73 ?
Edit 2: Some details: with both 20.7 and 20.9, it complains about -march=native:
$ make clean
nvc-Error-Unknown switch: -march=native
make: *** [Makefile.prebuild:66: getarch] Error 1
Makefile.system:1279: Makefile.: No such file or directory
make: *** No rule to make target 'Makefile.'. Stop.
$ nvc --version
nvc 20.7-0 LLVM 64-bit target on x86-64 Linux -tp zen
NVIDIA Compilers and Tools
Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
md5-df4cbd0d4649573d80d18d10a76332b7
$ nvc --version
nvc 20.9-0 LLVM 64-bit target on x86-64 Linux -tp zen
NVIDIA Compilers and Tools
Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
wyphan
on 15 Dec 2020
@wyphan that error is probably why he has "HOSTCC=gcc" in his commandline (missing a check for "nvc" in Makefile.system where it currently only looks for the old name "pgcc").
@cparrott73 unfortunately I see several hundred failed tests in make lapack-test on Haswell target when I compile without OpenMP, and what looks like a thread deadlock in ssymm with it enabled. (This was with the default "-tp p7-64", but results for "-tp haswell" are only marginally better,
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1180155 302 (0.026%) 0 (0.000%)
DOUBLE PRECISION 1214923 284 (0.023%) 4 (0.000%)
COMPLEX 730902 34 (0.005%) 0 (0.000%)
COMPLEX16 675998 239 (0.035%) 0 (0.000%)
--> ALL PRECISIONS 3801978 859 (0.023%) 4 (0.000%)
Does this match your results, or could there be a crucial part of the patch missing ?
martin-frbg
on 15 Dec 2020
BTW 20.11 also triggers a noisy warning "cc clobber ignored" for the (cpuid calls in cpuid_x86.c and common_x86_64.h,
but that one appears to be already known as TPR 29277 according to a thread on stackoverflow
martin-frbg
on 15 Dec 2020
"cc clobber ignored"
@martin-frbg Are you referring to this link? Yeah, I got that warning too with 20.11. For me, with 0.3.13 (after patching with @cparrott73 's patch above in the OP) and NVIDIA HPC SDK 20.11, the build failed at creating the .a file using ranlib (but that might be from my imperfect self-maintained Lmod module files)
ar: `u' modifier ignored since `D' is the default (see `U')
/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/ranlib ../../libopenblas_zenp-r0.3.13.a
make[2]: /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/ranlib: Command not found
make[2]: *** [Makefile:601: ../../libopenblas_zenp-r0.3.13.a] Error 127
make[2]: Leaving directory '/home/wyp/compile/OpenBLAS-0.3.13/lapack-netlib/SRC'
make[1]: *** [Makefile:27: lapacklib] Error 2
make[1]: Leaving directory '/home/wyp/compile/OpenBLAS-0.3.13/lapack-netlib'
make: *** [Makefile:244: netlib] Error 2
Here's the full build log, in case you wanna take a closer look.
build-nv20.11.log.gz
I wonder if this ranlib issue is related to issue #3037?
Also, I just finished building Reference LAPACK 3.9.0 from the tarball on Netlib on this x86_64 machine (henceforth I will refer to as BaseCamp). The -Kieee flag is required to avoid hundreds of numerical errors, even for reference LAPACK. Full details here: https://github.com/Reference-LAPACK/lapack/issues/430
wyphan
on 15 Dec 2020
Not sure about the ranlib thing, the bug uncovered in #3037 would wrongly prefix the ranlib command but not prepend a path I think(?) (I do have the fix in my branch though, as misdetecting CROSS has other drawbacks like not running the tests.)
Thanks for the reminder about -Kieee ,I thought I had already added that here but apparently did not - as OpenBLAS' LAPACK is Reference-LAPACK with just a few substitutions it is not really surprising that the same applies here )
martin-frbg
on 15 Dec 2020
BTW, I am also testing 0.3.12 with our aarch64 (ARM64) compilers, but it looks like we have not yet implemented the SIMD intrinsic functions found in
arm_neon.hon aarch64 for the newer vectorized kernels in 0.3.12. I am filing a ticket on this with our developers, but it probably will not be resolved soon.
@cparrott73 Interesting, maybe I should also try building OpenBLAS 0.3.13 with nvhpc 20.11 on my Jetson Nano Developer Kit, just for fun.
wyphan
on 15 Dec 2020
Hm, with -Kieee added I still get several errors, though much fewer than before
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1298235 13 (0.001%) 0 (0.000%)
DOUBLE PRECISION 1309007 0 (0.000%) 0 (0.000%)
COMPLEX 752310 55 (0.007%) 0 (0.000%)
COMPLEX16 760898 74 (0.010%) 0 (0.000%)
--> ALL PRECISIONS 4120450 142 (0.003%) 0 (0.000%)
Does the same patch apply to 20.7 or 20.9?
Edit: I've downloaded and installed 20.11 on my
x86_64workstation, but haven't configured the Lmod modules yet that I wanna use with it -- things like CUDA 11.1, UCX, OpenMPI 4, and OpenBLAS 0.3.12 or newer. Unless 20.11 actually already bundles OpenMPI 4 in addition to the default OpenMPI 3 @cparrott73 ?Edit 2: Some details: with both 20.7 and 20.9, it complains about
-march=native:$ make clean nvc-Error-Unknown switch: -march=native make: *** [Makefile.prebuild:66: getarch] Error 1 Makefile.system:1279: Makefile.: No such file or directory make: *** No rule to make target 'Makefile.'. Stop.$ nvc --version nvc 20.7-0 LLVM 64-bit target on x86-64 Linux -tp zen NVIDIA Compilers and Tools Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.$ nvc --version nvc 20.9-0 LLVM 64-bit target on x86-64 Linux -tp zen NVIDIA Compilers and Tools Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
Unfortunately, I don't think this patch will work for 20.7 or 20.9. Several bugs that were causing some compilation errors with some of the kernels that use inline assembly got fixed for 20.11. You can certainly try, but I would highly recommend updating to 20.11 if possible.
cparrott73
on 15 Dec 2020
@wyphan that error is probably why he has "HOSTCC=gcc" in his commandline (missing a check for "nvc" in Makefile.system where it currently only looks for the old name "pgcc").
@cparrott73 unfortunately I see several hundred failed tests inmake lapack-teston Haswell target when I compile without OpenMP, and what looks like a thread deadlock in ssymm with it enabled. (This was with the default "-tp p7-64", but results for "-tp haswell" are only marginally better,--> LAPACK TESTING SUMMARY <-- SUMMARY nb test run numerical error other error ================ =========== ================= ================ REAL 1180155 302 (0.026%) 0 (0.000%) DOUBLE PRECISION 1214923 284 (0.023%) 4 (0.000%) COMPLEX 730902 34 (0.005%) 0 (0.000%) COMPLEX16 675998 239 (0.035%) 0 (0.000%) --> ALL PRECISIONS 3801978 859 (0.023%) 4 (0.000%)Does this match your results, or could there be a crucial part of the patch missing ?
@martin-frbg - I think we enable the equivalent of gcc's -ffast-math at -O2. -Kieee will certainly force the use of more accurate math routines, at the cost of performance. It's the classic tradeoff of performance vs. accuracy...
cparrott73
on 15 Dec 2020
BTW 20.11 also triggers a noisy warning "cc clobber ignored" for the (cpuid calls in cpuid_x86.c and common_x86_64.h,
but that one appears to be already known as TPR 29277 according to a thread on stackoverflow
Yep, known issue. I think I'm the one that filed a ticket on this. Doesn't seem to be very high on their list of things to fix at the moment, but I'll poke them again.
cparrott73
on 15 Dec 2020
Does the same patch apply to 20.7 or 20.9?
Edit: I've downloaded and installed 20.11 on my
x86_64workstation, but haven't configured the Lmod modules yet that I wanna use with it -- things like CUDA 11.1, UCX, OpenMPI 4, and OpenBLAS 0.3.12 or newer. Unless 20.11 actually already bundles OpenMPI 4 in addition to the default OpenMPI 3 @cparrott73 ?Edit 2: Some details: with both 20.7 and 20.9, it complains about
-march=native:$ make clean nvc-Error-Unknown switch: -march=native make: *** [Makefile.prebuild:66: getarch] Error 1 Makefile.system:1279: Makefile.: No such file or directory make: *** No rule to make target 'Makefile.'. Stop.$ nvc --version nvc 20.7-0 LLVM 64-bit target on x86-64 Linux -tp zen NVIDIA Compilers and Tools Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.$ nvc --version nvc 20.9-0 LLVM 64-bit target on x86-64 Linux -tp zen NVIDIA Compilers and Tools Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
Missed a comment above...we actually do bundle Open MPI 4 with 20.11, but it is not default. It is considered a "beta" right now, built against UCX 1.9.0. You are certainly welcome to give it a go and see how it fares, although it is entirely possible that there could be issues. It seems to work well in my testing so far, but we are still putting it through its paces.
cparrott73
on 15 Dec 2020
@cparrott73 Interesting, maybe I should also try building OpenBLAS 0.3.13 with nvhpc 20.11 on my Jetson Nano Developer Kit, just for fun.
I have found an optimizer bug with the 20.11 ARM compilers when compiling OpenBLAS. I am still investigating it so I can open a ticket on it. It will work if you drop the opt level down to -O0, but I realize that is far from ideal. ARM is our newest arch to support, so it may still be a little rougher around the edges than x86_64 right now.
cparrott73
on 15 Dec 2020
@martin-frbg - I think we enable the equivalent of gcc's -ffast-math at -O2. -Kieee will certainly force the use of more accurate math routines, at the cost of performance. It's the classic tradeoff of performance vs. accuracy...
Probably more than just that, as dropping optimization to -O0 did not help..
BTW lapack-test results are markedly better for -tp p7-64 compared to -tp haswell above but the failures in single-precision real may point to a miscompilation of the SGEMM microkernel or similar.
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1298235 23 (0.002%) 0 (0.000%)
DOUBLE PRECISION 1309007 0 (0.000%) 0 (0.000%)
COMPLEX 768366 0 (0.000%) 0 (0.000%)
COMPLEX16 769178 0 (0.000%) 0 (0.000%)
--> ALL PRECISIONS 4144786 23 (0.001%) 0 (0.000%)
I wonder if it's possible to automatically detect the -tp option with nvhpc, just like with other compilers? For instance, on BaseCamp the optimal -tp option would be -tp zen instead of -tp haswell or -tp p7-64
wyphan
on 15 Dec 2020
I wonder if it's possible to automatically detect the
-tpoption withnvhpc, just like with other compilers? For instance, on BaseCamp the optimal-tpoption would be-tp zeninstead of-tp haswellor-tp p7-64
Worth noting that the NVIDIA HPC compilers default to the equivalent of -march=native. Unless you explicitly pass a -tp flag, they will generate code for the CPU detected on the host automatically. We normally throw -tp p7-64 when building OpenBLAS to mimic what gcc does, and produce object code that will work on any CPU.
So, if you want to target the BaseCamp CPUs by default, the easiest thing to do here would be to just simply remove -tp p7-64 from the OpenBLAS makefile.
cparrott73
on 15 Dec 2020
@martin-frbg - I think we enable the equivalent of gcc's -ffast-math at -O2. -Kieee will certainly force the use of more accurate math routines, at the cost of performance. It's the classic tradeoff of performance vs. accuracy...
Probably more than just that, as dropping optimization to -O0 did not help..
BTW lapack-test results are markedly better for-tp p7-64compared to-tp haswellabove but the failures in single-precision real may point to a miscompilation of the SGEMM microkernel or similar.--> LAPACK TESTING SUMMARY <-- SUMMARY nb test run numerical error other error ================ =========== ================= ================ REAL 1298235 23 (0.002%) 0 (0.000%) DOUBLE PRECISION 1309007 0 (0.000%) 0 (0.000%) COMPLEX 768366 0 (0.000%) 0 (0.000%) COMPLEX16 769178 0 (0.000%) 0 (0.000%) --> ALL PRECISIONS 4144786 23 (0.001%) 0 (0.000%)
Hmmmm, interesting. If we can pinpoint this to a specific file, I'll open a ticket on it.
cparrott73
on 15 Dec 2020
Certainly possible to make the -tp match the detected (or specified) target eventually, p7-64 happened to be the one that "mostly" worked with earlier releases of PGI (and from the above test results I am not yet convinced that this has already changed)
(Currently retrying with -O1 again to see if that "fixes" the problem)
martin-frbg
on 15 Dec 2020
Certainly possible to make the
-tpmatch the detected (or specified) target eventually, p7-64 happened to be the one that "mostly" worked with earlier releases of PGI (and from the above test results I am not yet convinced that this has already changed)
(Currently retrying with -O1 again to see if that "fixes" the problem)
Yes, -tp p7-64 is the "generate code for a baseline x86_64 CPU" switch - essentially the equivalent of gcc's default codegen for x86_64. Normally you want to pass this one when compiling code that is expected to run on any x86_64 CPU, or you will get illegal instruction errors when building without this flag on a newer CPU (e.g. Haswell with AVX2 instructions) and then try to run the code on an older CPU (e.g. Sandy Bridge with only AVX instructions).
cparrott73
on 15 Dec 2020
Right, but unfortunately I am not even talking about running the code on some other cpu, just building and testing on Haswell (well, Kaby Lake actually) - 142 numerical errors across all four data types in lapack-test with -tp haswell vs. 23 in single-precision real with p7-64 (and no change at -O1). I assume nvc honors the gcc-style "-msse, -mavx" etc. flags that OpenBLAS started to use for intrinsics support recently, as it does not complain ?
martin-frbg
on 15 Dec 2020
I can help test on my laptop (Skylake i5-6300U) too. Which Makefile should I modify?
wyphan
on 15 Dec 2020
The -tp stuff lives in Makefile.system, optimization level can be set as COMMON_OPT in Makefile.rule. Failing lapack tests appear to be SGV and its driver (2x 11 errors with deviations of 8E6 so certainly significant, plus the "usual" and probably harmless single fault in STFSM that also comes up with gcc). Building with -O0 brought no change. Will try to find time to debug this later
martin-frbg
on 16 Dec 2020
Right, but unfortunately I am not even talking about running the code on some other cpu, just building and testing on Haswell (well, Kaby Lake actually) - 142 numerical errors across all four data types in lapack-test with
-tp haswellvs. 23 in single-precision real with p7-64 (and no change at-O1). I assume nvc honors the gcc-style "-msse, -mavx" etc. flags that OpenBLAS started to use for intrinsics support recently, as it does not complain ?
Sorry, I know I quoted this here, but I think in my head I was thinking about @wyphan's question about regarding the BaseCamp system. :-)
Yes, if there is an issue with our handling of the inline assembly kernels that even affects -tp p7-64 at -O1, let me know and I'll file the ticket.
We do support the -msse, -mavx, etc. style flags, although I am aware of a bug involving -mskylake-avx512 not implying -mavx512f like it should. I believe that one will be fixed in the next release, which will be 21.1, due sometime in January or thereabouts.
cparrott73
on 16 Dec 2020
As promised, here are the build logs and test logs for OpenBLAS 0.3.12 on Skylake (i5-6300U). I applied the patch in the OP, then manually changed -tp p7-64 to -tp haswell in Makefile.system. I didn't change the optimization level.
$ nvc --version
nvc 20.11-0 LLVM 64-bit target on x86-64 Linux -tp haswell
NVIDIA Compilers and Tools
Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
$ nvc -tp
nvc-Fatal-Switch -tp must have a value
-tp=px|bulldozer|piledriver|zen|zen2|sandybridge|haswell|knl|skylake|host|native
Select target processor
px Generic x86 Processor
bulldozer AMD Bulldozer processor
piledriver AMD Piledriver processor
zen AMD Zen architecture (Epyc, Ryzen)
zen2 AMD Zen 2 architecture (Ryzen 2)
sandybridge Intel SandyBridge processor
haswell Intel Haswell processor
knl Intel Knights Landing processor
skylake Intel Skylake Xeon processor
host Link native version of HPC SDK cpu math library
native Alias for -tp host
md5-ce8630a394610cc75976a391dd3e50b9
$ patch -p1 < 3020-openblas-nvhpc.patch
$ make USE_OPENMP=1 > build-nv20.11-haswell.log 2>&1
$ make install PREFIX=/opt/nvlibs/openblas-0.3.12/nv20.11 >> build-nv20.11-haswell.log 2>&1
$ make blas-test lapack-test > test-nv20.11-haswell.log 2>&1
md5-df4cbd0d4649573d80d18d10a76332b7
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 970425 367 (0.038%) 22 (0.002%)
DOUBLE PRECISION 1115633 360 (0.032%) 26 (0.002%)
COMPLEX 542628 3572 (0.658%) 401 (0.074%)
COMPLEX16 593564 3652 (0.615%) 420 (0.071%)
--> ALL PRECISIONS 3222250 7951 (0.247%) 869 (0.027%)
Full build log and testing results:
build-nv20.11-haswell.log.gz
test-nv20.11-haswell.log.gz
Will try again tomorrow with -Kieee active, and with -tp skylake.
wyphan
on 16 Dec 2020
next release, which will be 21.1, due sometime in January or thereabouts.
Is the release cycle really that quick @cparrott73? One version every two months? Are there plans to make designated "LTS" versions or something, for supercomputing centers? (I feel bad for the sysadmins having to update the stack on say, Summit or Cori, every two months!)
wyphan
on 16 Dec 2020
Hmm, this is interesting, using -Kieee with -tp haswell actually increases the number of errors:
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1192185 2095 (0.176%) 444 (0.037%)
DOUBLE PRECISION 1189549 82 (0.007%) 31 (0.003%)
COMPLEX 664596 3491 (0.525%) 381 (0.057%)
COMPLEX16 673184 3502 (0.520%) 406 (0.060%)
--> ALL PRECISIONS 3719514 9170 (0.247%) 1262 (0.034%)
Full build and testing logs:
build-nv20.11-haswell-Kieee.log.gz
test-nv20.11-haswell-Kieee.log.gz
Updated patch:
3020-oblas-nvhpc-Kieee.patch.txt
wyphan
on 16 Dec 2020
Hm, weird, but I'm not sure if this should be applied to the C options at all ?
martin-frbg
on 16 Dec 2020
Using -tp skylake both crashes the build and testing phase with Illegal instruction, which is kinda counterintuitive? Unless not all Skylake processors are created equal... It's just a laptop Core i5 processor, after all, not a Xeon processor.
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./sblat2 < ./sblat2.dat
Illegal instruction (core dumped)
make[1]: *** [Makefile:28: level1] Error 132
make[1]: *** Waiting for unfinished jobs....
Illegal instruction (core dumped)
make[1]: *** [Makefile:88: level2] Error 132
Full build and testing logs:
build-nv20.11-skylake.log.gz
test-nv20.11-skylake.log.gz
wyphan
on 16 Dec 2020
@martin-frbg Let me try removing -Kieee from CCOMMON_OPT and rebuild. Also, should I test the different optimization levels too?
wyphan
on 16 Dec 2020
Changing optimization levels did not change anything in my tests so far... and maybe their "tp skylake" is skylake-x with avx512 ?
(I'm currently going off on a tangent as it appears that some of the test failures in complex occur with gcc+gfortran as well - git bisect just led to an unlikely candidate however. And what prompted this chase was that gcc+nvfortran fared no better than nvc+nvfortran in the lapack tests - which would disprove the "miscompiled C kernels" theory)
martin-frbg
on 16 Dec 2020
With -tp haswell and -Kieee only applied to FCOMMON_OPT but not CCOMMON_OPT. the number of errors decreased only slightly:
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1192185 2090 (0.175%) 432 (0.036%)
DOUBLE PRECISION 1189549 79 (0.007%) 34 (0.003%)
COMPLEX 664596 3506 (0.528%) 385 (0.058%)
COMPLEX16 673184 3441 (0.511%) 398 (0.059%)
--> ALL PRECISIONS 3719514 9116 (0.245%) 1249 (0.034%)
Full build and testing logs:
build-nv20.11-haswell-Kieee-fonly.log.gz
test-nv20.11-haswell-Kieee-fonly.log.gz
wyphan
on 16 Dec 2020
next release, which will be 21.1, due sometime in January or thereabouts.
Is the release cycle really that quick @cparrott73? One version every two months? Are there plans to make designated "LTS" versions or something, for supercomputing centers? (I feel bad for the sysadmins having to update the stack on say, Summit or Cori, every two months!)
@wyphan - we have always done releases on a 1-2 month cadence, but only two of the releases each year are intended to be feature releases that introduce new features. The ones in between are intended to be mostly bug fix/maintenance releases.
With the old PGI compilers, the feature releases were typically the x.1 and x.7 releases, where x corresponded to the last two digits of the year. e.g. 19.1, 19.7, etc.
With the rebranding to the NVIDIA HPC SDK, and to better align with NVIDIA's marketing strategies, things are changing around a bit now. The next feature release is planned for 21.3. 21.1 will actually be mainly a maintenance release that builds on 20.11.
LTS releases for compilers are pretty uncommon - even GCC doesn't do them. Language features are constantly updating and evolving. If developers want compilers with C++20 standards implemented, they aren't going to want to stick to a two year old LTS release that still only supports C++17.
cparrott73
on 16 Dec 2020
Using
-tp skylakeboth crashes the build and testing phase withIllegal instruction, which is kinda counterintuitive? Unless not all Skylake processors are created equal... It's just a laptop Core i5 processor, after all, not a Xeon processor.OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./sblat2 < ./sblat2.dat Illegal instruction (core dumped) make[1]: *** [Makefile:28: level1] Error 132 make[1]: *** Waiting for unfinished jobs.... Illegal instruction (core dumped) make[1]: *** [Makefile:88: level2] Error 132Full build and testing logs:
build-nv20.11-skylake.log.gz
test-nv20.11-skylake.log.gz
@wyphan - our compilers will report what CPU they detect on your system if you pass the -V option, e.g.
$ nvc -V
nvc 20.11-0 LLVM 64-bit target on x86-64 Linux -tp skylake
NVIDIA Compilers and Tools
Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
Note the -tp skylake here.
It sounds like you built OpenBLAS for a CPU architecture that your i5 doesn't support. Worth noting that not all "Skylakes" are created equal. -tp skylake for our compilers means a Xeon-class CPU that supports AVX-512. A lot of consumer-grade Intel CPUs that are called "Skylake" or later do not actually support AVX-512, and so our compilers will actually report those as -tp haswell instead. I would suggest changing -tp skylake to -tp haswell in your makefile, and rebuilding. That will probably work for you.
cparrott73
on 16 Dec 2020
Changing optimization levels did not change anything in my tests so far... and maybe their "tp skylake" is skylake-x with avx512 ?
(I'm currently going off on a tangent as it appears that some of the test failures in complex occur with gcc+gfortran as well - git bisect just led to an unlikely candidate however. And what prompted this chase was that gcc+nvfortran fared no better than nvc+nvfortran in the lapack tests - which would disprove the "miscompiled C kernels" theory)
This is correct. -tp skylake is essentially the same as -mavx512f in the GNU world, plus (I think) some other things. (I would have to consult our developers to understand the exact differences, as things get really tricky with AVX-512.) AVX-512 is not monolithic, like AVX/AVX2, and different processors support different strains of AVX-512. Wikipedia actually gives a pretty good breakdown of this:
https://en.wikipedia.org/wiki/AVX-512
But I'm probably preaching to the choir here. :-)
At any rate, -tp skylake essentially implies a Xeon-class CPU that supports most strains of AVX-512. Most consumer-grade "Skylake" CPUs report as -tp haswell by our compilers.
cparrott73
on 16 Dec 2020
(I'm currently going off on a tangent as it appears that some of the test failures in complex occur with gcc+gfortran as well - git bisect just led to an unlikely candidate however. And what prompted this chase was that gcc+nvfortran fared no better than nvc+nvfortran in the lapack tests - which would disprove the "miscompiled C kernels" theory)
I've also observed the same thing with some of my testing here. gcc+gfortran also reports some test failures for me, too. I seem to recall some of my colleagues did some analysis of some of the LAPACK tests a long time ago, and found a number of them weren't numerically stable. Even numbers being off by as little as one bit in the representation would blow up into large errors in some of those tests. But that was a long time ago, and I don't know if they have cleaned any of those up since then.
cparrott73
on 16 Dec 2020
I believe some cleanup has been taking place in recent years, and the failure I am looking into occured after 0.3.12 - it seems to be "related to" adding the -mfma option (but nothing as obvious as a direct causation by the fma intrinsics code that got added to the x86_64 srot.c and drot.c at that time - I can revert those and still see the errors arise with -mfma). In any case this is most likely unrelated to the test errors with nvc/nvfortran and -tp p7-64 that I had intended to debug.
martin-frbg
on 16 Dec 2020
With -tp p7-64 and default optimization level, the build crashes here for me:
/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvc -c -O2 -DMAX_STACK_ALLOC=2048 -mp -tp p7-64 -DF_INTERFACE_PGI -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=4 -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION=\"0.3.12\" -msse3 -mssse3 -msse4.1 -DASMNAME=sdot_k -DASMFNAME=sdot_k_ -DNAME=sdot_k_ -DCNAME=sdot_k -DCHAR_NAME=\"sdot_k_\" -DCHAR_CNAME=\"sdot_k\" -DNO_AFFINITY -I.. -UDOUBLE -UCOMPLEX -UCOMPLEX -UDOUBLE ../kernel/x86_64/sdot.c -o sdot_k.o
"../common_x86_64.h", line 163: warning: "cc" clobber ignored
("cpuid": "=a" (*eax), "=b" (*ebx), "=c" (*ecx), "=d" (*edx) : "0" (op), "2" (count) : "cc");
^
"../common_x86_64.h", line 163: warning: "cc" clobber ignored
("cpuid": "=a" (*eax), "=b" (*ebx), "=c" (*ecx), "=d" (*edx) : "0" (op), "2" (count) : "cc");
^
"../kernel/x86_64/saxpy_microk_haswell-2.c", line 69: warning: "cc" clobber
ignored
: "cc",
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 6: error: identifier
"__m256" is undefined
typedef __m256 v_f32;
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 7: error: identifier
"__m256d" is undefined
typedef __m256d v_f64;
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 31: error: identifier
"__m256" is undefined
BLAS_FINLINE float v_sum_f32(__m256 a)
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 33: error: identifier
"__m256" is undefined
__m256 sum_halves = _mm256_hadd_ps(a, a);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 33: warning: function
"_mm256_hadd_ps" declared implicitly
__m256 sum_halves = _mm256_hadd_ps(a, a);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 35: warning: function
"_mm256_castps256_ps128" declared implicitly
__m128 lo = _mm256_castps256_ps128(sum_halves);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 35: error: a value of type
"int" cannot be used to initialize an entity of type "__m128"
__m128 lo = _mm256_castps256_ps128(sum_halves);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 36: warning: function
"_mm256_extractf128_ps" declared implicitly
__m128 hi = _mm256_extractf128_ps(sum_halves, 1);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 36: error: a value of type
"int" cannot be used to initialize an entity of type "__m128"
__m128 hi = _mm256_extractf128_ps(sum_halves, 1);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 41: error: identifier
"__m256d" is undefined
BLAS_FINLINE double v_sum_f64(__m256d a)
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 43: error: identifier
"__m256d" is undefined
__m256d sum_halves = _mm256_hadd_pd(a, a);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 43: warning: function
"_mm256_hadd_pd" declared implicitly
__m256d sum_halves = _mm256_hadd_pd(a, a);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 44: warning: function
"_mm256_castpd256_pd128" declared implicitly
__m128d lo = _mm256_castpd256_pd128(sum_halves);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 44: error: a value of type
"int" cannot be used to initialize an entity of type "__m128d"
__m128d lo = _mm256_castpd256_pd128(sum_halves);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 45: warning: function
"_mm256_extractf128_pd" declared implicitly
__m128d hi = _mm256_extractf128_pd(sum_halves, 1);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 45: error: a value of type
"int" cannot be used to initialize an entity of type "__m128d"
__m128d hi = _mm256_extractf128_pd(sum_halves, 1);
^
"../kernel/x86_64/../arm/../simd/intrin_avx.h", line 60: warning: last line of
file ends without a newline
#define v_zero_f64 _mm256_setzero_pd
^
"../kernel/x86_64/../arm/sum.c", line 69: warning: function "_mm256_setzero_ps"
declared implicitly
v_f32 vsum0 = v_zero_f32();
^
"../kernel/x86_64/../arm/sum.c", line 75: warning: function "_mm256_add_ps"
declared implicitly
vsum0 = v_add_f32(vsum0, v_loadu_f32(x));
^
"../kernel/x86_64/../arm/sum.c", line 75: warning: function "_mm256_loadu_ps"
declared implicitly
vsum0 = v_add_f32(vsum0, v_loadu_f32(x));
^
"../kernel/x86_64/../arm/sum.c", line 81: warning: function "_mm256_add_ps"
declared implicitly
vsum0 = v_add_f32(
^
"../kernel/x86_64/../arm/sum.c", line 85: warning: function "_mm256_loadu_ps"
declared implicitly
vsum0 = v_add_f32(vsum0, v_loadu_f32(x + i));
^
10 errors detected in the compilation of "../kernel/x86_64/../arm/sum.c".
make[1]: *** [Makefile.L1:691: ssum_k.o] Error 2
Full build log:
build-nv20.11-p7-64.log.gz
wyphan
on 16 Dec 2020
Wonder why it has only the -msse options but no -mavx here, which would be required to enable the avx intrinsics ?
martin-frbg
on 17 Dec 2020
I don't think we enable the _mm256 intrinsics unless -mavx2 is thrown, but I would have to double check that.
cparrott73
on 17 Dec 2020
our compilers will report what CPU they detect on your system if you pass the -V option
I find it interesting to note that BaseCamp has a Ryzen 5 3600X, and is detected as tp zen instead of -tp zen2, even though AMD markets it as "Zen 2/Matisse", and the new Ryzen 5000-series processors as "Zen 3/Vermeer". The server equivalent is "Milan", which I believe will power both Frontier (OLCF) and Perlmutter (NERSC). (Don't quote me on Frontier, though. It might be Zen 4 or Zen 5, since they only say "next-generation AMD EPYC")
Anyway, what new instruction is introduced here, that corresponds to -tp zen2?
wyphan
on 17 Dec 2020
I don't think there is any major instruction set difference between zen vs. zen2 vs. zen3, or zen vs. haswell for that matter. I think it mostly comes down to instruction scheduling to benefit zen vs. haswell.
cparrott73
on 17 Dec 2020
So whatever it is in detail, the post-0.3.12 problem with the LAPACK test results is definitely related to the -mfma option with both gcc and nvc.
For gcc (versions ranging from 7.5 to 10.2), we get accuracy errors in single and double precision complex when -mfma is applied globally rather than just to the kernel files that actually need it (srot.c and drot.c). With nvc the accuracy errors in single precision real - or more precisely SGGEV/SGGEV3 - persist unless I actually disable the FMA codepath in kernel/x86_64/srot.c when C_PGI is defined.
(I have not yet confirmed that the almost identical code in drot.c is unused with nvc, thus avoiding similar test failures in DGGEV, but it seems plausible as the preprocessor commands there are more specific than just requiring HAVE_FMA3 to be set)
martin-frbg
on 17 Dec 2020
Power8 builds with 20.11 (and the patch, as merged in #3046 ) appear to hang early in single-precision real part of the lapack-test unfortunately - throws lots of both numerical and XERBLA errors in SST/SE2 (xeigtsts <se2.in) and eventually locks up there.
Test summary with the test of the LAPACK routines set to F in TESTING/se2.in - all failures are in the EIG section
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1187745 2486 (0.209%) 276 (0.023%)
DOUBLE PRECISION 1196333 2492 (0.208%) 266 (0.022%)
COMPLEX 652800 3639 (0.557%) 211 (0.032%)
COMPLEX16 661388 3494 (0.528%) 273 (0.041%)
--> ALL PRECISIONS 3698266 12111 (0.327%) 1026 (0.028%)
@cparrott73 Interesting, maybe I should also try building OpenBLAS 0.3.13 with nvhpc 20.11 on my Jetson Nano Developer Kit, just for fun.
I have found an optimizer bug with the 20.11 ARM compilers when compiling OpenBLAS. I am still investigating it so I can open a ticket on it. It will work if you drop the opt level down to
-O0, but I realize that is far from ideal. ARM is our newest arch to support, so it may still be a little rougher around the edges than x86_64 right now.
@cparrott73 Any updates on this? I'm trying to compile 0.3.10 on my Jetson Nano Developer Kit (4GB version). So far it fails at the testing phase for dblat2 (level1) with a segfault.
@martin-frbg Do you know if 0.3.10 is also affected by the gcc/nvc accuracy issues? Can I safely conclude that 0.3.10 is the current "stable" version for production?
Also, happy holidays! Wishing y'all a brighter 2021.
wyphan
on 30 Dec 2020
@wyphan the lapack-test issues appear only with nvc/nvfortran, not with gcc, and I have not tried to compile 0.3.10 with the current NVIDIA compiler. Possibly the current develop branch of OpenBLAS is still missing some fix that is contained in PGI/NVIDIAs internal use copy, but i do not think this is something that crept in after 0.3.10.
martin-frbg
on 30 Dec 2020
@wyphan - sorry - if you were referring to the FMA-related errors on x86_64, these indeed came in after 0.3.12 (and are solved on the develop branch). Not sure if cparrott73's patch for 20.11 would apply cleanly to 0.3.10 however.
martin-frbg
on 30 Dec 2020
@martin-frbg Great, I'll use 0.3.10, for now, until this issue is fixed. The Jetson Nano is an ARM64 platform, btw. I had to modify both Makefile.system and Makefile.arm64 to remove -tp p7-64, as well as the -march and -mtune flags (which is GCC-specific, I think?).
On this platform, the available options for -tp are:
$ nvc -tp
nvc-Fatal-Switch -tp must have a value
-tp=host|native Select target processor
host Link native version of HPC SDK cpu math library
native Alias for -tp host
Uh, did I miss a patch from cparrott73 for ARM64, or did you manage to get around the NEON support limitations in nvc that he mentioned ? (Doesn't look like they leave one with much choice for -tp on that platform :smile:)
martin-frbg
on 30 Dec 2020
I don't think the patch is here yet for ARM64. At least not in this issue...
wyphan
on 30 Dec 2020
OK, so was your experiment with removing the spurious options successful all the same ? The Jetson appears to be using a Cortex A57 so should be supported when compiling with gcc at least.
martin-frbg
on 30 Dec 2020
Yes, the gcc build completed without any issues. The nvc build, on the other hand, still segfaults at the testing phase. Maybe I need to drop to -O0 like Chris suggested. Here's my rudimentary patch for 0.3.10. Feel free to adapt it for the latest dev version.
wyphan
on 30 Dec 2020
POWER8 built single-threaded
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1278219 1 (0.000%) 0 (0.000%)
DOUBLE PRECISION 1286807 0 (0.000%) 0 (0.000%)
COMPLEX 731034 57 (0.008%) 0 (0.000%)
COMPLEX16 739622 43 (0.006%) 0 (0.000%)
--> ALL PRECISIONS 4035682 101 (0.003%) 0 (0.000%)
so part of the problem _appears_ to be with/from threading (in a VM instance with just two cores even)
martin-frbg
on 30 Dec 2020
Unfortunately I am seeing similar problems on ARMV8 - which like PPC is an architecture with weak memory ordering, but there are/were no ill effects seen with gcc/gfortran lately. After commenting all the gcc-specific options in Makefile.arm64 and redirecting the ARMV8 sdot back to sdot.S to avoid the unimplemented intrinsics from sdot.c, USE_OPENMP=1 is absolutely required to get sane behaviour in the BLAS tests. And even with it, the SE2 LAPACK tests lock up here too, and the overall result
is currently not too pretty:
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1187745 2225 (0.187%) 278 (0.023%)
DOUBLE PRECISION 1196333 2082 (0.174%) 368 (0.031%)
COMPLEX 22690 355 (1.565%) 0 (0.000%)
COMPLEX16 28484 173 (0.607%) 17 (0.060%)
--> ALL PRECISIONS 2435252 4835 (0.199%) 663 (0.027%)
(where the very low number of COMPLEX tests run is due to memory constraints in my Raspberry PI 4B)
Single-threaded run results:
--> LAPACK TESTING SUMMARY <--
SUMMARY nb test run numerical error other error
================ =========== ================= ================
REAL 1278219 1 (0.000%) 0 (0.000%)
DOUBLE PRECISION 1286807 0 (0.000%) 0 (0.000%)
COMPLEX 22690 355 (1.565%) 0 (0.000%)
COMPLEX16 28484 173 (0.607%) 17 (0.060%)
--> ALL PRECISIONS 2616200 529 (0.020%) 17 (0.001%)
Hi gang - sorry for the delayed response, I was away for a couple of weeks due to the winter holidays. I hope you are all having a happy and healthy new year so far. :-)
I do not yet have any new progress to report on the ARM crash with our compilers with OpenBLAS. I hope to get back on that horse again this week, and I'll see if I can make some progress with debugging it. I'm not as familiar with the ARM architecture as I am with x86_64, so it has been a bit of a learning curve for me.
@martin-frbg - I can't promise anything, but I will talk to my superiors and see if there is any way we can get you some better hardware. I have a RPI 4B 4 GB myself, and while it's a nice little board for the price, the memory can be a limiting factor.
cparrott73
on 4 Jan 2021
BTW, I have opened a ticket on NEON intrinsics support in our ARM64 compilers, but it's going to be a fair amount of work to implement those, and I do not expect anything to be available and ready for prime time anytime soon.
cparrott73
on 4 Jan 2021
@cparrott73 thank you - the Pi4 was what I happened to have available locally for a "quick" test during the holiday week, I will see if I can redo the build on a system in the gcc compile farm later this week.
martin-frbg
on 4 Jan 2021
@cparrott73 Isn't there also a 8GB version for the RPi 4B? One of my friends bought that model for Christmas to set up a Steam Link streaming machine (and I bought a RPi 4B 4GB off him, since he ordered both the 4GB and 8GB model that weekend).
wyphan
on 12 Jan 2021
Yeah, they came out with an 8 GB RPi 4B, although that was after the initial batch were released. When I got mine, the 4 GB one was the largest size available at the time.
Have thought about upgrading to an 8 GB one, but it hasn't been super high on my priority list at the moment.
We do have some ARM boards (e.g. NVIDIA Jetson Xaviers) and some third-party rackmount ARM servers that we use for testing and development within our organization.
cparrott73
on 12 Jan 2021
NVIDIA Jetson Xaviers
That's a lot more beefy than the usual NVIDIA Jetson Developer Kits, esp. because it packs a Volta GPU instead of a Maxwell one, and 8 GB RAM (Jetson Xavier NX) / 32 GB RAM (Jetson AGX Xavier) instead of just 4 GB. They also aren't as cheap, unfortunately, otherwise I would've bought one for myself...
wyphan
on 12 Jan 2021
I've run a few builds on an X-Gene Mustang (target identified as EMAG8180) in the gcc compile farm but have not made any
significant progress. (Also not entirely sure if finding workarounds for building with (semi-) commercial compilers is what gccfarm is supposed to be used for...)
martin-frbg
on 12 Jan 2021
Meh, I'm an idiot - all that was wrong on ARMV8 (apart from a few unsupported instructions in the DOT kernels) was the declaration of the interface for passing complex values to/from nvfortran.
martin-frbg
on 12 Jan 2021
@martin-frbg - great news, that jibes with some of my observations that the ARM NV HPC SDK crashes were localized to the Complex routines. I have not had a lot of cycles to dig into it further, though.
I do have an open ticket on supporting the missing ARM NEON intrinsics, but the work on that is estimated to require several months. I do not yet have a target release for those, but I will let you know when we do.
How soon are you thinking of releasing 0.3.14? I have some folks here that are eager to see some of this stuff in a release, though I suppose there is nothing stopping me from cloning a snapshot of the repo as it is in order to try it. Just curious, more than anything else.
All - in case it wasn't clear, our ARM compilers do not yet differentiate among target ARM64 architectures all that much. I think they only support the base ARMv8 stuff right now. So in essence, I don't think the -tp flag has any effect on ARM at the moment. That might change in the future.
cparrott73
on 12 Jan 2021
No set date yet, primarily there is some ongoing grief about strangely increased memory requirements seen by the numpy crowd.
Seeing that we are just one month past the latest release, I think end of February would be a sensible timeframe unless something dramatic comes up earlier.
martin-frbg
on 12 Jan 2021
Sounds good. I'll probably just pull 0.3.13 and sync it with these NV HPC compilers patches in the interim.
cparrott73
on 12 Jan 2021
Related issues
aytekinar
路
10Comments
winpassuser
路
8Comments
marker68
路
8Comments
santo4ul
路
9Comments
hholst80
路
17Comments