Openblas: Compiler Failure on POWER8 with /kernel/power/i{c,d,s,z}a{min,max}.c

[grisuthedragon]

I updated my installations on my POWER8 machine and since version 0.3.6 up to the current development branch, the build process fails with:

$ make  MAKE_NB_JOBS=1
...
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8.dev\" -DASMNAME=isamax_k -DASMFNAME=isamax_k_ -DNAME=isamax_k_ -DCNAME=isamax_k -DCHAR_NAME=\"isamax_k_\" -DCHAR_CNAME=\"isamax_k\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UCOMPLEX -UDOUBLE -DUSE_ABS  -UUSE_MIN ../kernel/power/isamax.c -o isamax_k.o
../kernel/power/isamax.c: In function ‘siamax_kernel_64’:
../kernel/power/isamax.c:288:1: internal compiler error: in build_int_cst_wide, at tree.c:1210
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
Preprocessed source stored into /tmp/ccriHyPV.out file, please attach this to your bugreport.
make[1]: *** [isamax_k.o] Error 1
make[1]: Leaving directory `/root/OpenBLAS/kernel'
make: *** [libs] Error 1

OS Details:
CentOS 7.6 ppc64el, gcc 4.8.5, gfortran 4.8.5, IBM POWER8 LC822

[martin-frbg]

Interesting - this file is indeed new in 0.3.6, but (from debugging #2233) the ICE in question appears to be fixed in more recent compilers. (Unfortunately the source line 288 from the error message tells us nothing as it is simply the closing brace on the last line of the file).

[grisuthedragon]

If I find some time I will take a closer look inside the routines and why they are crashing.

[martin-frbg]

I see now that this has already been reported (against gcc 7.1.0) by susilehtlola as https://bugzilla.redhat.com/show_bug.cgi?id=1740539
(From the tests for #2233, the code compiles with gcc 5.4 and again from 7.3 onwards (7.3.0/8.2.1/9.1), so it could be that something was fixed, broken, and fixed again)

[martin-frbg]

Reproduced in the unicamp.br minicloud. Preprocessed source of isamax.c is here:
isamax_preprocessed.txt if somebody wants to add it to the redhat bugzilla entry (@susilehtola)
Also the problem goes away at -O0 (instead of our default -Ofast) so can be worked around with
a #pragma GCC optimize "O0" in the affected files.

[martin-frbg]

Also of note is that in my tests (gcc-4.8.5 20150623 Red Hat 4.8.5-36 on CentOS 7) it is only the
single precision versions (isamin/isamax.c and icamin/icamax.c) that cause an ICE.

[susilehtola]

@martin-frbg thanks, uploaded. However, in my experience there's very little movement on RHEL bugzilla tickets.

If someone has a RHEL subscription, they should make an issue about the bug with Red Hat; things get moving when a paying customer complains.

[grisuthedragon]

I did some experiments with isamax.c and found the around line 111 the snippets:

temp0 += temp1;                                                                                                                                                                                                    
temp0 += temp1; //temp0+32

and around line 170

    quadruple_indices = vec_sel( quadruple_indices,ind2,r3);
    quadruple_values= vec_sel(quadruple_values,vv0,r3);      

    temp0+=temp1;
    temp0+=temp1; //temp0+32

seem to disturb the compiler.

Update: narrow the range a bit.

[quickwritereader]

Could you comment/remove some lines to find which parts cause this error?

[grisuthedragon]

@quickwritereader That was exactly what I did. Commenting out the above mentioned lines, the code compiles.

[quickwritereader]

addition? could you remove that c++ style comment or replace it with c style /* */

[grisuthedragon]

The c++ style comment is already in OpenBLAS and not from me. I tried to change "+=" to "temp0 = temp+temp1" because I had this problem in ancient times with gcc and the vetorization of a piece of code but that did not change anything.

[quickwritereader]

Maybe to rename/refactor temp0 to other names. feel free to change refactor. comment blocks to c style.
temp0 = vec_add( temp1 , vec_add(temp0,temp1));

[grisuthedragon]

Renaming and replacing the addition does not help.

[quickwritereader]

from what versions of gcc it disappears?
remove all register hints too.
Also rewrite + operations with this style result=vec_add(operand1,operand2)

[grisuthedragon]

I tried it with gcc 5.4 and there it works. The problem is as long as RHEL7 with gcc 4.8.5 is the recommend OS from IBM for the POWER8, it should work with the old gcc as well.

Rewritting the + to vec_add does also not help.

[martin-frbg]

Strangely I saw no effect of removing these lines (on the gcc 4.8.5 build that is), and have not managed to narrow it down to less than the entire siamax_kernel_64 routine. My current impression is that this is some more general issue with the __register vector declarations or perhaps an interaction with OpenMP (at least I found some resolved gcc/gfortran issues about failure of build_int_cs_wide in conjunction with OpenMP from the gcc 4.x timeframe).

[grisuthedragon]

Since the gcc 4.x series will pass away, a possible fix would be do deactivate the kernels for GCC < 5.x and bypass this problem this way.

[quickwritereader]

ok lets replace double addition with one introducing new variable before for loop after temp1=temp1 <<1 ; line
__vector unsigned int add_32= temp1<<1;
or
__vector unsigned int add_32= {32,32,32,32};
then change two additions with one
temp0=vec_add(temp0,add_32);
Afaik there should be enough registers to hold additional number.

[martin-frbg]

As long as we do not suspect anything fundamentally wrong with these files, perhaps the easiest
solution is to wrap the #pragma I suggested above in a GCC version check - probably
#if (( defined(__GNUC__) && __GNUC__ < 6 to be on the safe side.
There is no need to deactivate these kernels as such, just to deactivate the default -Ofast compiler optimization for them.

[quickwritereader]

with recent issues, I see inline assembly was the best choice but it doubles work for the new architecture.

[martin-frbg]

I now see that a gcc 8.3.0 build on power8 (Ubuntu 19.04) returns 200+ errors in the single-precision complex LAPACK tests. Disabling optimization of icamin/icamax.c works around this as well (and incidentally also removes the single failure in CTFSM that we have been seeing across all architectures)

[quickwritereader]

I will try to check those this week.

[martin-frbg]

More importantly (and thanks to isuruf's help with debugging our Travis script), DYNAMIC_ARCH=1 with gcc 5.4 fails in the power9 part with another ICE:

./kernel/power/caxpy.c: In function ‘caxpy_kernel_16’:
../kernel/power/caxpy.c:62:33: internal compiler error: in rs6000_emit_le_vsx_move, at config/rs6000/rs6000.c:9157
         register __vector float vy_0 = vec_vsx_ld( offset_0 ,vptr_y ) ;
                                 ^

and this time the #pragma GCC optimize "O0" does not help. (This is probably why the CI for #2243 had failed - I had merged your PR as I could not reproduce the error in the unicamp minicloud, but now I realize I had only built for the power8 target with this old gcc)
Update: this error does not occur with (at least) gcc 8.3.0.

[edelsohn]

Disabling features to work-around issues in GCC 4.8.5 is okay, although IBM should work with RH to backport the fixes. But ICEs in GCC 7/8 should be reported and fixed in GCC. None of this was reported to the GCC community.

[quickwritereader]

@martin-frbg how about to use gcc generated optimized assembly files for both power8 and power9 directly?
I intended to convert them inline assemblies but had to postpone.

[martin-frbg]

Yes I guess that would work (assuming the generated assembly is not totally unreadable from a human perspective). Once we are convinced that we have found a working version that is - 8.3.0
miscompiling icamin/icamax certainly complicated matters, and I suspect creating self-contained test cases for the GCC folks may be a non-trivial task as well.

[edelsohn]

It's not great to create too much hand-written assembly if the compiler produces efficient code. The compiler will adapt to future processors. Hand-written assembly code creates another location to update.

If the testcase cannot be reduced, please open a GCC Bugzilla with the pre-processed source file.

[quickwritereader]

@martin-frbg could you check if it passes after reverting caxpy fix.
I had to use intrinsics because of power8 error if you remember.

[edelsohn]

Also, you can try to use creduce program to create a minimal testcase. Or upload the entire pre-processed source file and the IBM Toolchain team will reduce it.

[quickwritereader]

@martin-frbg thanks for the generating and merging assembly kernels using conditions.
One could later further optimize generated assemblies.

[martin-frbg]

Closed by the commit, reopening as the underlying compiler issue has not yet been reported to the gcc team (while the related RH bugzilla entry has seen no activity). Might also make sense to review my choice of a minimum gcc version for PPC9 support - perhaps turning this into a choice between the original and intrinsics versions of the affected code(s) would work as well.

[edelsohn]

I believe that the original bug already was fixed. It works in GCC 5.4, 7.3, and 8.x because of backports from GCC 8. For some reason GCC 6.x was overlooked.

IBM will work with RH to backport the patch to GCC 4.8.5 RH in RHEL.

I am concerned about the second bug:

I now see that a gcc 8.3.0 build on power8 (Ubuntu 19.04) returns 200+ errors in the single-precision complex LAPACK tests. Disabling optimization of icamin/icamax.c works around this as well (and incidentally also removes the single failure in CTFSM that we have been seeing across all architectures)

The reduced test corresponds to the first bug. The second bug separately needs to be opened upstream in GCC Bugzilla. It appears to be a separate, undiagnosed and unfixed issue.

[martin-frbg]

There is also the ICE with GCC 5.4 compiling caxpy.c with -mcpu=power9 (no problem with the exact same source and -mcpu=power8). mentioned above
preprocessed source: caxpy.txt

[edelsohn]

I don't have access to GCC 5.4.

IBM Advance Toolchain 9.0 includes a patched version of GCC 5.5. I cannot reproduce the failure, although you don't provide the exact invocation. IBM AT9.0 is far out of date and no longer supported. GCC 5.x is out of date and no longer supported.

The comment above about GCC 5.4 (https://github.com/xianyi/OpenBLAS/issues/2254#issuecomment-532110686) reports that it does not fail with GCC 8.3. IBM Advanced Toolchain 12.0 include GCC 8.3, and I similarly do not elicit and error.

Why does the failure on GCC 5.4 matter? Other than GCC 4.8.5 RH, I don't understand why any of the older, out of date releases of GCC matter, nor why OpenBLAS should make any changes to accommodate them.

Is Unicamp Power Minicloud using GCC 5.4?

[martin-frbg]

GCC 5.4 is or was what Ubuntu 16 came with, which just happens to be what our current Travis CI job uses. One of the twenty or so Unicamp VM choices comes with Ubuntu 16.04, which allowed me to reproduce and debug the initial CI build failure. Other than that, the GCC 5.4 ICE probably does not matter, as most affected users should be able to update their compiler. And an obvious compile failure is certainly better in this regard than a silent miscompilation.

[edelsohn]

Can you please produce pre-processed source code for the icamax.c error with a specific compiler invocation for GCC 8.3? You can open a GCC Bugzilla issue or I can open one.

[martin-frbg]

Preprocessed source : icamax.txt
Compiler options are -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -fno-fast-math -fopenmp -m64
"Bad" intermediate assembly from this: icamax_badS.txt
"Good" intermediate assembly from overriding the -Ofast with a pragma GCC optimize "O0": icamax_goodS.txt

GCC info: gcc version 8.3.0 (Ubuntu 8.3.0-6ubuntu1)

[edelsohn]

Do you have a driver program that demonstrates the wrong result?
Have you tried anything other than O0? The differences are vast between highly optimized and no optimization, so it is difficult to ascertain what is miscompiled.

[martin-frbg]

No simple driver program so far, this problem was originally exposed only by running the LAPACK testsuite included in the netlib reference implementation. (Which is why I wrote "silent miscompilation" of the library earlier). I do not think I did experiment with other optimization levels for this particular case - the O0 workaround was already in place for the older gcc versions at the time (and there O1 was already sufficient to cause the original ICE that started all this), I really should not be spending time on this right now, but I'll see if I can do a quick test at -O1

[edelsohn]

I can try to run through the variants if you can provide the incantation to run the LAPACK testsuite in OpenBLAS.

It really would be helpful to try -O3 and -O2 -- something closer to -Ofast.

[martin-frbg]

It is simply make; make lapack-test - "good" shows no numeric errors for single-precision complex, "bad" has 275 of them. And it already goes wrong at -O1, I'll see if I can narrow it down to one of the 40-odd options enabled by that.

[edelsohn]

It fails with -O1? Okay. The individual flags do not control all of the -O1 options, so you may not be able to find the exact optimization.

[martin-frbg]

Indeed none of the options documented in https://gcc.gnu.org/onlinedocs/gcc-8.3.0/gcc/Optimize-Options.html as being enabled by -O1 triggers the test failures. Trying now with the options flagged in a diff of the gcc -Q --help=optimize output at -O0 and -O1. (Update: No hit there either)

[edelsohn]

First, Abdel always recommended that I build OpenBLAS with NO_LAPACK=1.

I built OpenBLAS on CentOS 7 running on a Power8 system in the GNU Compile Farm (gcc112) using IBM AT12.0, which is GCC 8.3.1 with additional patches backported.

The initial results are

            -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error     other error  
================    =========== =================   ================  
REAL                1284869     1   (0.000%)    5   (0.000%)
DOUBLE PRECISION    1039157     1   (0.000%)    3   (0.000%)
COMPLEX             14868       67620   (454.802%)  111 (0.747%)
COMPLEX16           424579      0   (0.000%)    2   (0.000%)

--> ALL PRECISIONS  2763473     67622   (2.447%)    121 (0.004%)

I see no excessive failures for REAL. The failures for COMPLEX seem to be related to memory and I noticed the comments in the documentation about ulimit -s unlimited. I am re-running the tests with that environment variable. I will report the results when the test completes.

At the moment I cannot reproduce the failures with GCC 8.3.1 to which I have access. My initial assessment is this failure already was fixed in a more recent release of GCC (not necessarily GCC 8.3.1).

[edelsohn]

I misread that the problem was with REAL and not single precision COMPLEX. There does seem to be a problem with COMPLEX. Re-running with ulimit -s unlimited shows numerous errors.

            -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error     other error  
================    =========== =================   ================  
REAL                1284869     1   (0.000%)    5   (0.000%)
DOUBLE PRECISION    1293457     0   (0.000%)    5   (0.000%)
COMPLEX             20868       99053   (474.665%)  1129    (5.410%)
COMPLEX16           753628      0   (0.000%)    10  (0.001%)

--> ALL PRECISIONS  3352822     99054   (2.954%)    1149    (0.034%)

[quickwritereader]

if it compiles fine I will re-write them all explicitly using vec_vsx_ld, vec_vsx_st.

[quickwritereader]

2233 I feel it is another gcc-7 vs gcc-8 problem

[edelsohn]

GCC for Power originally implemented builtins in a very old and crude manner that treated them like a black box. The PPC Toolchain team has been converting them to expand to the internal representation, when there is a correspondence like loads and stores, so that the compiler can reason about them and optimize them. This transition is occurring across the compilers that you are testing (GCC 4.8 to GCC 9).

There also were some early bugs in the swapping of vector elements in registers for little endian. It's possible that hand written assembly written with an earlier compiler assumed an incorrect element ordering and now is producing incorrect results when presented with elements in the correct (Little Endian) ordering.

Reading #2233 this seems like the same issue and it never fully was diagnosed.

With more recent releases of GCC, ideally one should be using normal assignment to load and store instead of explicit vec_vsx_ld / vec_vsx_st to allow the compiler to choose the best load and store flavors.

[quickwritereader]

I will reconsider that ,this time. I will try to use macroses so that we could change it to subscript [index] or vec_vsx_ld in future and also I will try to use vec_step for converting my loops.

[martin-frbg]

Your test results for COMPLEX are much worse than what the Ubuntu gcc 8.3.0 produced (for a pre-#2260 snapshot of the develop branch), which revision did you build ? If we are up against an immature compiler, providing workarounds for known problems seems all the more appropriate to me.

[edelsohn]

The compiler is not immature. I would appreciate that you not make accusations like that. All software has bugs.

GCC 8 has extensive tests for all of the vectorization builtins. It is building entire PPC64LE Linux distributions, including other SIMD-intensive software (OpenCV, x264, libvp9, dav1d).

There is a bug, either in GCC 8 or OpenBLAS. I would like us to focus on finding and fixing the bug. The IBM GCC Team and the Red Hat GCC Team are standing by to debug and fix any compiler bug, if we can get a clear, reproducible, self-contained testcase. Comparing -O0 and -O1 assembly language output of entire files has very limited benefit to debug the problem when we don't know exactly what is miscompiled.

I have tried building OpenBLAS with additional options of -mno-optimize-swaps, -mcpu=power7 -mtune=power7. The lapack tests seemed to hang for both.

[quickwritereader]

I re-wrote icamax icamin isamin isamax to ensure that for Power8 it chooses vec_vsx_ld .
I did not check Power9 caxpy cdot compilation errors using older compilers.

I also run lapack-test but I am not aware of lapack lib . I dont know If its errors are related to Openblas kernels

ulimit -s unlimited
make lapack-test 

Testing REAL              Linear-Equation-routines-LIN/xlintsts < stest.in > stest.out  *** XERBLA was called from SSYSV_AA_2STAGE with INFO =      7 instead of 11 ***
 passed: 397224
Info Error: 1

Testing REAL              RFP-linear-equation-routines-LIN/xlintstrfs < stest_rfp.in > stest_rfp.out   STFSM auxiliary routine:     1 out of  7776 tests failed to pass the threshold
 passed: 5352
failing to pass the threshold: 1

Testing DOUBLE PRECISION Linear-Equation-routines-LIN/xlintstd < dtest.in > dtest.out  *** XERBLA was called from DSYSV_AA_2STAGE with INFO =      7 instead of 11 ***
 passed: 397224
Info Error: 1

Testing COMPLEX           Linear-Equation-routines-LIN/xlintstc < ctest.in > ctest.out  *** XERBLA was called from CHESV_AA_2STAGE with INFO =      7 instead of 11 ***
  *** XERBLA was called from CSYSV_AA_2STAGE with INFO =      7 instead of 11 ***
 passed: 410639
Info Error: 2

Testing COMPLEX           RFP-linear-equation-routines-LIN/xlintstrfc < ctest_rfp.in > ctest_rfp.out   CTFSM auxiliary routine:     1 out of  7776 tests failed to pass the threshold
 passed: 5352
failing to pass the threshold: 1

Testing COMPLEX16          Linear-Equation-routines-LIN/xlintstz < ztest.in > ztest.out  *** XERBLA was called from ZHESV_AA_2STAGE with INFO =      7 instead of 11 ***
  *** XERBLA was called from ZHESV_AA_2STAGE with INFO =      7 instead of 11 ***
 passed: 410639
Info Error: 2



SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1284869         1       (0.000%)        1       (0.000%)
DOUBLE PRECISION        1293457         0       (0.000%)        1       (0.000%)
COMPLEX                 745040          1       (0.000%)        2       (0.000%)
COMPLEX16               753628          0       (0.000%)        2       (0.000%)

--> ALL PRECISIONS      4076994         2       (0.000%)        6       (0.000%)



md5-17bf708dc59e58ff14f7c5013e0f9406



Testing REAL              Nonsymmetric-Eigenvalue-Problem-EIG/xeigtsts < nep.in > snep.out  SHS:    2 out of  1764 tests failed to pass the threshold
 passed: 7056
failing to pass the threshold: 2
Testing REAL              Linear-Equation-routines-LIN/xlintsts < stest.in > stest.out  *** XERBLA was called from SSYSV_AA_2STAGE with INFO =      7 instead of 11 ***
 passed: 397224
Info Error: 1

Testing REAL              RFP-linear-equation-routines-LIN/xlintstrfs < stest_rfp.in > stest_rfp.out   STFSM auxiliary routine:     1 out of  7776 tests failed to pass the threshold
 passed: 5352
failing to pass the threshold: 1

Testing COMPLEX           Nonsymmetric-Eigenvalue-Problem-EIG/xeigtstc < nep.in > cnep.out  CHS:    1 out of  1764 tests failed to pass the threshold
  CHS:    1 out of  1764 tests failed to pass the threshold
  CHS:    2 out of  1764 tests failed to pass the threshold
 passed: 3528
failing to pass the threshold: 4
Testing COMPLEX16          Linear-Equation-routines-LIN/xlintstz < ztest.in > ztest.out  *** XERBLA was called from ZHESV_AA_2STAGE with INFO =      7 instead of 11 ***
  *** XERBLA was called from ZHESV_AA_2STAGE with INFO =      7 instead of 11 ***
 passed: 410639
Info Error: 2


SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1283105         3       (0.000%)        1       (0.000%)
DOUBLE PRECISION        1293457         0       (0.000%)        1       (0.000%)
COMPLEX                 747524          4       (0.001%)        2       (0.000%)
COMPLEX16               753628          0       (0.000%)        2       (0.000%)

--> ALL PRECISIONS      4077714         7       (0.000%)        6       (0.000%)

quickwritereader/OpenBLAS

gcc -versions
Power8 gcc (Ubuntu 8.3.0-16ubuntu3~16.04) 8.3.0
Power9 gcc (Ubuntu 8.3.0-6ubuntu1~18.04.1) 8.3.0

[martin-frbg]

@quickwritereader most of the LAPACK routines call BLAS functions, so lapack-test failures are almost certainly caused by problems in the respective OpenBLAS kernels (at least under the assumption that the fortran compiler is working correctly - but compiling a pure netlib 3.8.0 would
be sufficient to prove that if necessary. I think at least some of the XERBLA warnings are due to errors in the testsuite itself, but the numerical errors are genuine).
@edelsohn can you explain the effect of -mno-optimize-swaps ? It appears to mangle the SGEMM and DGEMM kernels, after which all bets are off for the lapack test. (This is old code from wernsaar that has been there for years, with my amateurish patches to fix vsx register overwriting) . I'd like to think OpenBLAS on PPC is not _that_ broken.

[quickwritereader]

using BLAS-TESTER on power9 my strmm failed with precision errors. I will check it next week.
@edelsohn Could you check quickwritereader/OpenBLAS on your Pc with gcc-8.3. as I got only 1 numerical and 3 errors on power8 and power9, not 99053
thanks

[martin-frbg]

There seems to be something wrong with my half-handcrafted caxpy_power8.S - I get lots of caxpy-related errors with gcc 9.1.1 now (opensuse tumbleweed image in the unicamp minicloud) and they go away when I switch back to the caxpy.c from #2243. ~Possibly~ a mis-edit from my merging of the conjugating and non-conjugating versions slipped through.
Fixed by #2269

[martin-frbg]

... and I realize now that it is the SGEMV/DGEMV and related functions that are failing with -mno-optimize-swaps, the dgemm canary in the utest is affected through its use of dgesvd. So more alignment (or actually element ordering, if I understand edelsohn' comment) issues like #2233 ?

Reverting the sgemv kernel updates from #1994 and dgemv_t from #1499 leaves just one error
with SGER that can be removed by reverting the saxpy optimization from #1963.
At this stage, BLAS-Tester finds no errors. The lapack test results appear to be good for REAL,DOUBLE and COMPLEX, but COMPLEX16 reports about 20000 tests failing to pass the threshold before it hangs (gdb puts the hang in ZLARTG which has no obvious BLAS dependency).
Removing the zgemv kernels from #1499 makes the lapack tests run to completion again, without numerical errors in COMPLEX16 (only a single error in REAL, which is the "familiar" one from STFSM).
So this appears to be a relatively recent systematic oversight that does not necessarily show up with default compiler settings.

[quickwritereader]

I only re-wrote cgemv pairs using vec_vsx_ld. Did it pass with -mno-optimize-swaps?

[martin-frbg]

Certainly. This latest report is entirely about getting the develop branch working with -mno-optimize-swaps after edelsohns comment. (I understood this to be an indication of which files need to be checked for possibly incorrect vector load/store operations)

[quickwritereader]

fixed strmm on power9 that I detected using BLAS-TESTER. But it did not affect lapack-test errors I posted above. Plus on power9 there are no alignment issues, afaik.
For now, I think it is just because of my fork is old and behind many commits. I will check it, later
I'm glad that I re-checked strmm and fixed its bug which was introduced after the latest sgemm performance improvement. They share the same kernel file with ifdefs.

[martin-frbg]

Thanks for the strmm fix, merged. FYI, lapack test results with gcc 9.2.1 (fedora rawhide, power9) are

                       -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error  
================        ===========     =================       ================  
REAL                    1283105         3       (0.000%)        1       (0.000%)
DOUBLE PRECISION        1293457         0       (0.000%)        1       (0.000%)
COMPLEX                 749288          4       (0.001%)        2       (0.000%)
COMPLEX16               424579          0       (0.000%)        2       (0.000%)

--> ALL PRECISIONS      3750429         7       (0.000%)        6       (0.000%)

(same result with and without -mno-optimize-swaps)

[quickwritereader]

I am getting the same on power9. I am interested in what could cause those numerical errors. I tested Blas-Tester and fixed the only problem(strmm) I found and still, it gives the same errors

[martin-frbg]

It was not obvious (at least to me) from the testing_results.txt. (Also I have just received what looks like an automated message from unicamp.br telling me my minicloud account is "about to expire" - no idea if there are any limits to (re)activation)

[edelsohn]

I'm uncertain why you chose Unicamp over the Oregon State University Powerdev environment. You should be able to ask Unicamp to extend the account. IBM can support the request if there is a question. OSU has more resources.

[quickwritereader]

@martin-frbg if there be problems getting a new instance from Oregon Uni. let me know. I got those computers for OpenBLAS and I can grant access to power8 and power9

[quickwritereader]

I run Blas-Tester on power9 blas3. It passed all cases that I tried for single real and complex.
Unfortunately, I am not able to find the source of lapack test failures for power9 to investigate it.
Noting that power8 kernels do not make the same errors.
Errors are related to Nonsymmetric-Eigenvalue-Problem. Help from anyone who is aware of how that problem works are appreciated.

[quickwritereader]

There is an option to add a print for matrix input and output to blas3 for power8 and power9 and later diff them after running lapack-tests.

[martin-frbg]

I have run a few tests with varied optimization levels and with using the power8 kernels on power9, but did not observe a difference in the reported CHS failures. Reducing the optimization level for the fortran part even caused a single failure in CST to appear. The single CTFSM (and STFSM) failures are possibly spurious, as they show up on x86_64 as well and vanish as soon as one makes any changes in the number of tests defined in the input file in an attempt to track them down.

[martin-frbg]

Revisiting this, I believe the original Power8 issue is solved as far as OpenBLAS itself is concerned (though the subtopic of deriving meaningful bug reports for gcc is still open), and also big-endian Power8 now works (with Linux at least, there is a separate ticket open for AIX assembler woes).
According to my tests the Power9 LAPACK-TESTING failures are directly attributable to the SGEMM and CGEMM kernels (causing test failures in EIG/shst01.f and chst01.f respectively, which is reduction of a general matrix to upper Hessenberg form via GEMM). Using the corresponding power8 kernels makes these failures go away, but my knowledge of PPC assembly is too poor to spot the functional difference causing this. I will see if i can at least narrow it down to
either of the transpose/non-transpose invocation paths exercised by shst01.f

[quickwritereader]

thanks @martin-frbg . I will try to look at this problem again next week myself too
I could say that all power9 gemm is using butterfly rank1 update and Cgemm and Sgemm ones have swap permute masks. But I need to see value differences to see what could go wrong.
But BLAS_TESTer test cases passed so I postponed it

[antonblanchard]

Committing the direct output of an old version of gcc is pretty ugly, can we at least check the compiler version and only use the workaround files when necessary?

I came here because these fixes causes a SEGV on my POWER9 box. The local entry point doesn't get created correctly and we end up modifying r2 when we shouldn't.

I submitted a fix for it in #2324, but any function with ".localentry" is also suspect.

[martin-frbg]

I certainly agree that this is a very messy workaround, the only argument in favor of this was that it appeared to work while various older gcc versions produced wrong results. Not sure why I did not see the SEGV in the OpenPOWER minicloud.
The dubious .localentry will be in every kernel thus "improved", i.e. caxpy, iXamin, iXamax.

[martin-frbg]

With #2328, the workaround on POWER9 should only be used when the gcc is older than 9.x. I have also fixed the precompiled caxpy which suffered from the same problem as cdot (which I now understand to be just another case of forgetting an ifdef CONJ to support generating caxpyc from the same file, so the iamin/iamax are not affected after all).

[antonblanchard]

@martin-frbg thanks!

[martin-frbg]

Unfortunately I still have no idea what is wrong with the POWER9 sgemm and cgemm, except that the lapack test failures were introduced with the respective PRs for the POWER9-specific kernels. The misbehaviour varies with overall optimization level, suggesting that a register is not cleared or restored somewhere (?)

[quickwritereader]

I dont have any idea. Maybe there is hidden typo or bug. Last time I fixed sgemm typo. Probably it was because somehow I tested outdated not refactored one. After fixes it passed Blas-Tester.
This issue belongs to me. I will try to fix it.

[quickwritereader]

I need to reproduce lapack failure or at least I have to see what numbers differ.
@martin-frbg do we have print options for input and output as compile preprocessor ,there ? I will check source code myself too

[martin-frbg]

I agree that it must be just one special condition that is not handled, something that does not get exercised by BLAS-Tester or most of the lapack tests. Not sure if I understand your question regarding print options but I guess you would need to add your own #ifdef DEBUG ... printf... in the level3 code (or even the kernel itself, if you know how to write to stdout/stderr from powerpc assembly). And/or write out the matrix elements in EIG/shst01.f (or chst01.f) to get a general idea of the good vs. bad result.

[quickwritereader]

ok, so far I think that its something related to corruption if I correctly understood
I wanted to reproduce single failure it did not occur for single case
I met strange behaviour only 4th case fails

Here are details:
folder: OpenBLAS/lapack-netlib/TESTING
test failures occurred using this test case at 4th step:
./EIG/xeigtsts < nep.in
nep.in

NEP:  Data file for testing Nonsymmetric Eigenvalue Problem routines
7                                 Number of values of N
0 1 2 3 5 10 16                   Values of N (dimension)
5                                 Number of values of NB, NBMIN, NX, INMIN, IN WIN, INIBL, ISHFTS, and IACC22
  1   3   3    3   20             Values of NB     (blocksize)
  2   2   2    2    2             Values of NBMIN  (minimum blocksize)
  1   0   5    9    1             Values of NX     (crossover point)
 11  12  11   15   11             Values of INMIN  (LAHQR vs TTQRE crossover point, >= 11)
  2   3   5    3    2             Values of INWIN  (recommended deflation window size)
  0   5   7    3  200             Values of INIBL  (nibble crossover point)
  1   2   4    2    1             Values of ISHFTS (number of simultaneous shifts)
  0   1   2    0    1             Values of IACC22 (select structured matrix multiply: 0, 1 or 2)
20.0                              Threshold value
T                                 Put T to test the error exits
1                                 Code to interpret the seed
NEP  21

I got 4th one and replicated it 4 times. (for 3 times it does not fail)
named f4nep.in
command : ./EIG/xeigtsts

Strangely, replicated the same case many times but still 4th fails.

[quickwritereader]

ok so seeding the same as test case helped.
from docs:

2                                 Code to interpret the seed`
number1 number2 number3 number4 

(Seed numbers are from nep.in test failure)
here is new fnep.in to reproduce the failure for the seed numbers

NEP:  Data file for testing Nonsymmetric Eigenvalue Problem routines
1                                Number of values of N
16                   Values of N (dimension)
1                             Number of values of NB, NBMIN, NX, INMIN, IN WIN, INIBL, ISHFTS, and IACC22
  3     Values of NB     (blocksize)
  2     Values of NBMIN  (minimum blocksize)
  9     Values of NX     (crossover point)
 15     Values of INMIN  (LAHQR vs TTQRE crossover point, >= 11)
  3     Values of INWIN  (recommended deflation window size)
  3     Values of INIBL  (nibble crossover point)
  2     Values of ISHFTS (number of simultaneous shifts)
  0     Values of IACC22 (select structured matrix multiply: 0, 1 or 2)
20.0                              Threshold value
F                                 Put T to test the error exits
2                              Code to interpret the seed 
1507 2152 3082 3573
NEP  1
20

So now it runs 14 test cases 11th and 13th fails
SHS: 2 out of 14 tests failed to pass the threshold
schkhs.f

[martin-frbg]

Yes, I just got as far as that as well. (And missing the importance of the seed numbers may well be why I failed to reproduce the third one - the single STFSM test failure - with a reduced test case earlier)

[martin-frbg]

Seems to be the second invocation of SGEMM in shst01.f https://github.com/xianyi/OpenBLAS/blob/c6ecb195e6beafecd68948a0882f86e9120f3c8d/lapack-netlib/TESTING/EIG/shst01.f#L200 that returns a different result on power9 (the attached is a dump of the WORK array from running xeigtsts with the modified snep.in and the power8 and power9 versions of the sgemm kernel, .
sgemmdiff.txt

[quickwritereader]

I will try to test all sgemm calling shapes that occurs during that test. (m,n,k,lda,ldb,ldc, transA,transB)

[quickwritereader]

individually testing calling shapes passing. it seems I am obliged to print all numbers that is calling sgemm inside test itself

[martin-frbg]

Not sure I understand why inputs would differ, when the same tests pass with the equivalent POWER8 kernel inserted into KERNEL.POWER9. (And at least my impression was that the first of the two GEMM calls in shst01.f returned the same output for both, which would then be used as input for the second. Or did the first call damage the input array Q as a side effect ?)

[quickwritereader]

I mostly was printing from inside interface/gemm
but for shst one I did it from fortran ,too
frankly I dont know fortran it got fixed format and etc things.
Used go to's to make test for only one case inside schkhs.f but it did not even call sgemm , probably I messed tests

to print matrixes on fortran

      SUBROUTINE PROMPT( NAME, NOTA,  M, K, A, LDA)
      CHARACTER  (len = *)         NAME
      INTEGER K,LDA,M,NOTA
*     ..
*     .. Array Arguments ..
      REAL A(LDA,*) 

      WRITE(*,*)  ':::++:: ', NAME,' ',M,' ',K,' --',LDA
          IF (NOTA.EQ.0) THEN
                 print *, 'Not'
                  DO 80 L = 1,K 
                          DO 70 I = 1,M
                            WRITE(*,500, advance="no")  A(I,L)
   70                     CONTINUE 
                          print * , ""  
   80             CONTINUE 


          ELSE
                 print *, 'Tr'
                  DO 110 I = 1,M
                      TEMP = ZERO
                      DO 100 L = 1,K
                          WRITE(*,500, advance="no") A(L,I) 
  100                 CONTINUE 
                      print * , ""  
  110             CONTINUE

          END IF
      print * , "============================"    
  500    FORMAT(' ',G16.6,' ')
      RETURN
      END

[quickwritereader]

I could not find the problem, yet. I fix-restricted ABI registers usage said maybe FORTRAN and C mix linkage. I replicated precision error with those values but I could not replicate the failure case. Other errors happen when input comes kinda swapped way.
I will look at it with the fresh brain, again.
Thx @martin-frbg for the help

[martin-frbg]

I have now dumped the matrix Q before & after, and I see that it is already different before it even gets to the sgemm, making it harder to compare results. (Not just small differences beyond the 4th decimal place, but some adjacent elements interchanged.) Reading the comment section at the top of shst01.f it seems likely that I missed an earlier invocation of SGEMM via SGEHRD or SORGHR (where Q is generated). It does appear to be SGEMM and not its STRMM persona, as changing just the SGEMMKERNEL line is sufficient to remove the two failures.

[quickwritereader]

I also printed inputs and feed it to my tester. (i did not dump binary but with %.7g )
for same inputs, both power8 and power9 sgemm gave an exact result of outputs inside lapack.

Though they both did not pass my own tester. as I never considered the option where C and AB too big
so when comparing numbers with opposite side plus (alpha was -1 betta was 1) C - (AB) can give huge differences

[quickwritereader]

implementation side: power8 sgemm using broadcasting but power 9 using permutations. but both rely on fma instructions. thats why, imho, there could be a precision loss but not noticeable

[quickwritereader]

I tested with those numbers 1x16*16x16 case. This time I got different numbers
So still it is precision compared to Reference. Power8 numbers slightly better
```

np.std(power8)
15321.777719677071
np.std(power9)
15661.965138513111
np.std(reference)
14413.334581213328

( on power9 I even  removed registers that got asterisk mark on it thinking it is ABI problem )
So the main question is why power9 side fails but not power8

1x16 * 16x16 case
betta =1 alpha=-1 so C-AB

./calc2_power8
 ```
1 16 16 with leading vals : 16 132 16  scalars: alpha {-1.000000 -1.000000} betta {1.000000 1.000000}


---C_OUT ---
 0
 22876.5684
 30262.5098
 23754.8242
 14170.3457
 -20926.9766
 -9972.62988
 -15702.0068
 -3534.34985
 803.840881
 24843.4102
 -7408.46826
 13761.8174
 -2845.65381
 -4006.54956
 -8668.51172

---REFERENCE:---
 0
 18112
 33280
 23168
 20224
 -19968
 -14456
 -3968
 -1984
 -2048
 11776
 -9728
 13312
 9216
 896
 -10080

./calc2_power9

1 16 16 with leading vals : 16 132 16  scalars: alpha {-1.000000 -1.000000} betta {1.000000 1.000000}

---C_OUT ---
 0
 20480
 32768
 24576
 16384
 -24576
 -8192
 -8192
 -16384
 8192
 24576
 0
 13312
 0
 8192
 -8192

[quickwritereader]

gemm_calc
to compile

export DIR_P8=_path_to_power8_blas__
export DIR_P9=_path_to_power9_blas__
export CC=gcc
${CC}  gemm_calc.c -I${DIR_P8} ${DIR_P8}/libopenblas.a -lm -fopenmp -o calc_power8
${CC} gemm_calc.c -I${DIR_P9} ${DIR_P9}/libopenblas.a -lm -fopenmp -o calc_power9

export PARAM_M=1
export PARAM_N=16
export PARAM_K=16
export BLAS_LDB=132
export BLAS_BETTA=1
export BLAS_ALPHA=-1

./calc_power8
./calc_power9

[RajalakshmiSR]

If BLAS_ALPHA=-1.1 , I don't see a big difference in output for the above testcase. Will check what is special about 1.0 .

[RajalakshmiSR]

Ignore the above comment.. I still see some big differences with -2.0

[RajalakshmiSR]

I noticed this happens only when M is not a multiple of 4. (M = 4/8/12/16 gives same result for POWER8 and POWER9)
So I guess it could be related to float-double conversions flow in macros
4x2/8x2 and 4x1/8x1 in POWER9.[sgemm_macros_power9.S].

[quickwritereader]

thx @RajalakshmiSR
Good catch that conversion thing. I will try to check those cases.

[quickwritereader]

I forgot about them at all as I was thinking 16x16 cases. I think it should be fixed to make precision consistent against Reference

[quickwritereader]

I looked at Macroses other than small strmm typo which is not used ,actually , everything seemed ok. Still need to be sure
small unused typo

so I mainly use single->double->single conversion for saving but double as accumulators only for smaller blocks without loops. I could convert them to single to make consistent with the reference case.
but power8 is using double registers as accumulators in many areas, especially for all small cases which I am reluctant to do.

[quickwritereader]

So for now, I cant see errors other than precision ones and its hard to compare exactly.
worth to read:
how_do_i_interpret_lapack_testing_failures

[RajalakshmiSR]

@quickwritereader The only thing that is different in the second SGEMM call in shst01.f is,
same WORK array is used for both input and output. Do you think it can affect the result?

[quickwritereader]

I think then it should fail for all cases. Currently we reproduce failure with provided seed. Thats why Im more inclined toward precision roundness problem. Actually for small blocks it was easier to write with double. Considering mostly bigger block used while calls that is not performance critical. lxssp pairs load float converting it to double. Power8 code is using that and just use ordinary double multiplication avoiding any simd in reality.
We could switch to it as there should not be any loss performance wise. But we could also avoid using double at all even for save cases to see what happens.
But I could be wrong too considering sometimes small typos could lead to such. So there is still benefit to investigate

[RajalakshmiSR]

@quickwritereader JFYI I tried changing only STRMMKERNEL to use strmm_kernel_16x8_power8.S in KERNEL.POWER9 and keep SGEMM as power9 version and the tests are passing.

[quickwritereader]

you mean lapack test that mentioned above is passing?
If so I think we be closer enough to find the real reason instead of thinking how presicion errors could lead to that.

[RajalakshmiSR]

you mean lapack test that mentioned above is passing?
Yes

[martin-frbg]

Strangely changing only SGEMMKERNEL and leaving STRMMKERNEL at its POWER9 version worked as well (as I noted earlier). SGEHRD (to create the matrix Q) calls SGEMM first, followed by STRMM - I wonder if the SGEMM result from the POWER9 kernel is subtly different in some way that the POWER9 STRMM cannot cope with ?

[martin-frbg]

I have now dumped the A matrix in SGEHRD after the GEMM and TRMM calls during the SHST01 run for all three scenarios (both SGEMM and TRMM using power9 code, and either of the two replaced with the corresponding power8 version). Matrix elements here are in the e+10 range, so differences in accuracy in the fifth or sixth decimal place are quite significant in absolute numbers. What I do not understand right now is that switching only the STRMMKERNEL to its power8 version has an effect on some elements of the _GEMM_ result in the second cycle - could there be an uninitialized storage area at play ? (Attachments contain dumps of a(i,j) in the form of a double loop for i=1 to lda { for j=1 to n; print a(i.j)})
sgehrd_p9both.txt
sgehrd_p8trmm.txt
sgehrd_p8gemm.txt

[quickwritereader]

@martin-frbg thanks. I am upset all these codes turned to headache.
could you dump all information of input for sgemm case so we could test it using
against reference cases ,too

[quickwritereader]

gemm_calc
to compile

export DIR_P8=_path_to_power8_blas__
export DIR_P9=_path_to_power9_blas__
export CC=gcc
${CC}  gemm_calc.c -I${DIR_P8} ${DIR_P8}/libopenblas.a -lm -fopenmp -o calc_power8
${CC} gemm_calc.c -I${DIR_P9} ${DIR_P9}/libopenblas.a -lm -fopenmp -o calc_power9

export PARAM_M=1
export PARAM_N=16
export PARAM_K=16
export BLAS_LDB=132
export BLAS_BETTA=1
export BLAS_ALPHA=-1

./calc_power8
./calc_power9

here

[quickwritereader]

Dumped values could be copied using

    f_order_copy (  AA,16,16,16, a, nrowa ,  ncola, lda );
    f_order_copy (  BB,16,16,16, b, nrowb ,  ncolb,  ldb );
    f_order_copy (  CC,16,16,16,  c,m ,  n, ldc ); 
    f_order_copy (  CC,16,16,16,  cref,m ,  n, ldc ); 

for above case all dumps was 16X16 with lda-16
so we can copy it to differnt ldas diffrence m,n,
and use different alphas bettas to see
I did not add equality check with reference. but from previous code
if betta was small or zero it passed equality check. but for C - A*B case it did not as
it was hard to compare them with relative error because of C impact.

[quickwritereader]

for example when I pass M=1
then I get for power 9
20480
but for power 8
22876.5684
but for when M=16
both power 8 power9 are 20480
The reason was that power8 using double precision for 1x16.

so I concluded that there is not anything missing on calculation side.

[quickwritereader]

so I hope that I can find something typo or missing condition or calculation. But if it turns out that its just precision thing
then we don't have any option other than either to ignore or switch to power8 style double operations

[quickwritereader]

@martin-frbg
I think we could play this trick for your second gemm call. before that call set input matrixes to exact copy of some numbers.
and check resulted values. how far they are different.

I think it would be better than checking separately I noted above

[quickwritereader]

@martin-frbg could we ask lapack team about severity. for threshold 31 it passes as failure got 30. and 22 others are passing even for 3.0

[quickwritereader]

plus if I change seed numbers it passes

[martin-frbg]

Certainly I would not have any doubts about this if it did not look like a regression compared to the previous code. (But compiling for TARGET=POWER6 in the same environment shows some failures with a threshold excession in the range of 1e6 so probably our time and energy is better spent elsewhere)

[RajalakshmiSR]

@quickwritereader @martin-frbg Can this be closed? Do you prefer opening a separate issue for lapack threshold change?

[sh1ng]

Still reproducible on 0.3.8

../kernel/power/isamax.c: In function 'siamax_kernel_64':
../kernel/power/isamax.c:288:1: internal compiler error: in build_int_cst_wide, at tree.c:1210
 }
 ^
../kernel/power/icamax.c: In function 'ciamax_kernel_32':
../kernel/power/icamax.c:326:1: internal compiler error: in build_int_cst_wide, at tree.c:1210
 }
 ^
../kernel/power/icamin.c: In function 'ciamin_kernel_32':
../kernel/power/icamin.c:264:1: internal compiler error: in build_int_cst_wide, at tree.c:1210
 }
 ^

gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)

NAME="CentOS Linux"
VERSION="7 (AltArch)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (AltArch)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7:server"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[martin-frbg]

Is that big-endian POWER8 ? On little-endian it is supposed to pick up isamax_power8.S (isamax.c as processed by a newer gcc that does not iCE), unless I messed up the "if not big-endian" in KERNEL.POWER8.

[sh1ng]

ppc64le
cpu : POWER8E (raw), altivec supported

[edelsohn]

Use IBM AT12.0 or AT13.0 or RH Developer Toolset or something newer than GCC 4.8.5 RH.

[martin-frbg]

It should not even try to compile isamax.c in 0.3.8 - are you building with cmake by any chance ? (The simple KERNEL file parser in utils.cmake simply skips over conditional expressions.) Unfortunately I have lost OpenPower cloud access again so will need to reapply.

[edelsohn]

@martin-frbg When you have the account, please contact me so that we can set it not to expire.

[sh1ng]

@martin-frbg no I'm using vanilla make. Will try with cmake

[sh1ng]

@edelsohn I'm aware of the options, but prefer to use default gcc

[edelsohn]

If you want to create complications for yourself, that's your choice.

[martin-frbg]

@sh1ng cmake definitely won't be better, it would only have explained why my dirty trick with the conditionals in KERNEL.POWER8 does not work. Unfortunately too many related problems came up during the lifetime of this issue, ppc64be, power9, the relevance of lapack-test deviations... in particular I seem to have bungled my original gcc48 workaround with a later hack for ppc64be.

[RajalakshmiSR]

I verified that 0.3.8 builds fine on POWER8BE/ POWER8LE and POWER9 using AT12.

@martin-frbg I think the check for not big endian in KERNEL.POWER8 needs a correction
-ifneq ($(__BYTE_ORDER__),$(__ORDER_BIG_ENDIAN__))
+ifneq ($(__BYTE_ORDER__),"__ORDER_BIG_ENDIAN__")

[martin-frbg]

@RajalakshmiSR thanks - silly me. And I guess i should combine that fix with a "GCC<9" version check .

[amelvill-umich]

Can still reproduce with gcc 8.2.0, OpenBlas v0.3.8 RHEL 7.5, ppc64le.

For what it's worth, I can compile this without any problems on AMD64 GCC 9.3.0, Ubuntu 20.20

../kernel/power/isamax.c: In function ‘siamax_kernel_64’:
../kernel/power/isamin.c:288:1: internal compiler error: in build_int_cst_wide, at tree.c:1210
 }
 ^
../kernel/power/isamax.c:288:1: internal compiler error: in build_int_cst_wide, at tree.c:1210
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=cgemv_o -DASMFNAME=cgemv_o_ -DNAME=cgemv_o_ -DCNAME=cgemv_o -DCHAR_NAME=\"cgemv_o_\" -DCHAR_CNAME=\"cgemv_o\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -UDOUBLE -DCOMPLEX -UTRANS -UCONJ -DXCONJ ../kernel/power/cgemv_n.c -o cgemv_o.o
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=cgemv_u -DASMFNAME=cgemv_u_ -DNAME=cgemv_u_ -DCNAME=cgemv_u -DCHAR_NAME=\"cgemv_u_\" -DCHAR_CNAME=\"cgemv_u\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -UDOUBLE -DCOMPLEX -DTRANS -UCONJ -DXCONJ ../kernel/power/cgemv_t.c -o cgemv_u.o
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=cgemv_s -DASMFNAME=cgemv_s_ -DNAME=cgemv_s_ -DCNAME=cgemv_s -DCHAR_NAME=\"cgemv_s_\" -DCHAR_CNAME=\"cgemv_s\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -UDOUBLE -DCOMPLEX -UTRANS -DCONJ -DXCONJ ../kernel/power/cgemv_n.c -o cgemv_s.o
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=cgemv_d -DASMFNAME=cgemv_d_ -DNAME=cgemv_d_ -DCNAME=cgemv_d -DCHAR_NAME=\"cgemv_d_\" -DCHAR_CNAME=\"cgemv_d\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -UDOUBLE -DCOMPLEX -DTRANS -DCONJ -DXCONJ ../kernel/power/cgemv_t.c -o cgemv_d.o
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=csymv_U -DASMFNAME=csymv_U_ -DNAME=csymv_U_ -DCNAME=csymv_U -DCHAR_NAME=\"csymv_U_\" -DCHAR_CNAME=\"csymv_U\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -DCOMPLEX -UDOUBLE -ULOWER ../kernel/power/../generic/zsymv_k.c -o csymv_U.o
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=csymv_L -DASMFNAME=csymv_L_ -DNAME=csymv_L_ -DCNAME=csymv_L -DCHAR_NAME=\"csymv_L_\" -DCHAR_CNAME=\"csymv_L\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -DCOMPLEX -UDOUBLE -DLOWER ../kernel/power/../generic/zsymv_k.c -o csymv_L.o
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=chemv_U -DASMFNAME=chemv_U_ -DNAME=chemv_U_ -DCNAME=chemv_U -DCHAR_NAME=\"chemv_U_\" -DCHAR_CNAME=\"chemv_U\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -DCOMPLEX -UDOUBLE -ULOWER -DHEMV ../kernel/power/../generic/zhemv_k.c -o chemv_U.o
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=chemv_L -DASMFNAME=chemv_L_ -DNAME=chemv_L_ -DCNAME=chemv_L -DCHAR_NAME=\"chemv_L_\" -DCHAR_CNAME=\"chemv_L\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -DCOMPLEX -UDOUBLE -DLOWER -DHEMV ../kernel/power/../generic/zhemv_k.c -o chemv_L.o
cc -c -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=160 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=chemv_V -DASMFNAME=chemv_V_ -DNAME=chemv_V_ -DCNAME=chemv_V -DCHAR_NAME=\"chemv_V_\" -DCHAR_CNAME=\"chemv_V\" -DNO_AFFINITY -I.. -UDOUBLE  -DCOMPLEX -DCOMPLEX -UDOUBLE -ULOWER -DHEMV -DHEMVREV ../kernel/power/../generic/zhemv_k.c -o chemv_V.o
../kernel/power/icamax.c: In function ‘ciamax_kernel_32’:
../kernel/power/icamax.c:326:1: internal compiler error: in build_int_cst_wide, at tree.c:1210
 }

With 0.3.9 it gives me... a segfault?

gfortran -Ofast -mcpu=power8 -mtune=power8 -mvsx -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -Wall -frecursive -fno-optimize-sibling-calls -m64 -fopenmp  -O2 -frecursive -mcpu=power8 -mtune=power8 -malign-power -DUSE_OPENMP -fno-fast-math -fopenmp -fno-optimize-sibling-calls  -o zblat3 zblat3.o ../libopenblas_power8p-r0.3.9.a -lm -lpthread -lgfortran -lm -lpthread -lgfortran -L/usr/lib/gcc/ppc64le-redhat-linux/4.8.5 -L/usr/lib/gcc/ppc64le-redhat-linux/4.8.5/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/ppc64le-redhat-linux/4.8.5/../../..  -lc
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./sblat1

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x3FFFA2167193
#1  0x3FFFA2167D13
#2  0x3FFFA23C0477
#3  0x3FFFA2309C2C
#4  0x3FFFA23261B7
#5  0x100036DF in MAIN__ at sblat1.f:?
make[1]: *** [level1] Segmentation fault

I notice it seems to be still trying to link against the GCC 4.8.5 libraries, so there might be something I still have to look at on my end.

[martin-frbg]

The fix from #2411 mentioned above went into what became 0.3.9, not 0.3.8
The 0.3.9 release definitely builds and passes tests with both gcc 9.3.1 and gcc 8.3.1 (fedora 30 & 28). A mixup between compiler versions (mismatched gcc and gfortran, or picking up an older runtime) is likely to cause a crash

[quickwritereader]

@martin-frbg did really good job fixing those issues with generating assemblies.
In my forked project, I have updated sources for icamax icamin C source files. But never Pull requested them as better and tested fixes were already supplied.
People having these types of issues could also check it.

[martin-frbg]

(Re)checked 0.3.9 on Centos7 with a gcc8.2.1 from developer toolset (yum install devtoolset-8-gcc-8.2.1 devtoolset-8-gcc-gfortran-8.2.1) now as well, no crash, no test failures, nothing suspicious in lapack-test.

[amelvill-umich]

It appears there was something wrong with the cluster's configuration, it seems to be working now. Sorry for the mixup!