Ompi: PGI compiler failures

[awlauria]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

All versions are affected. Most failures are seen in MTT.

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

All - compiled by PGI (version 19.10)

• Operating system/version: RHEL *

---

Details of the problem

There are some intermittent PGI compiler problems. Some of the failures are legitimate and are easily fixed by PR's #7371 and #7372 . But others are seemingly the result of a bug in the compiler itself after inspecting the errors and finding nothing wrong (plus xlc and gcc never see these failures). I did a valgrind config/build run of OMPI 3.1.x branch to try and diagnose the bogus compiler errors, and got numerous invalid accesses reported and use of uninitialized variables. For example:

pgcc_valgrind.9955-==9955== Invalid read of size 2
pgcc_valgrind.9955-==9955==    at 0x100C6024: semant2 (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
pgcc_valgrind.9955-==9955==    by 0x1009CA77: deferred (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
pgcc_valgrind.9955-==9955==    by 0x1009F663: parser (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
pgcc_valgrind.9955-==9955==    by 0x10010887: main (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)

and

==106769== Invalid read of size 8
==106769==    at 0x103867D0: cmp_items.constprop.8 (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x1038CFF3: assemble_end (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x10013133: main (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==  Address 0x57e5540 is 102,800 bytes inside a block of size 103,168 free'd
==106769==    at 0x4086930: realloc (vg_replace_malloc.c:785)
==106769==    by 0x1008254B: sccrelal (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x100833C3: stg_need (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x10083883: stg_next (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x10023C73: get_type (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x100BD8DB: semant (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x1009D29B: deferred (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x1009F663: parser (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x10010887: main (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==  Block was alloc'd at
==106769==    at 0x4086930: realloc (vg_replace_malloc.c:785)
==106769==    by 0x1008254B: sccrelal (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x100833C3: stg_need (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x10083883: stg_next (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x10023C73: get_type (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x100BD8DB: semant (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x1009D29B: deferred (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x1009F663: parser (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)
==106769==    by 0x10010887: main (in /smpi_dev/mpiczar/local/linuxpower/19.10/bin/pgc-llvm)

I submitted a bug report with these details to PGI, case number SR1000807.

[awlauria]

PGI recommended that we switch to pgcc18 - and that pgcc has some lingering issues. I'm testing that out now to see if it is any better.

PGI18 support needed to be fixed as configure was thinking PGI18 was a gcc compiler, and would fail on the configure step. This was fixed in #7385

Their full reply in reference to SR1000807:

"Compile again with CC=pgcc18. Pgcc is the legacy front-end that has old issues. We're planning on phasing out pgcc and making pgcc18 the new pgcc. 19.10 still has both with pgcc set as the default. Let me know if that does the trick."

[jjhursey]

I'm using:

bash-4.2$ pgcc --version

pgcc 19.10-0 linuxpower target on Linuxpower 
PGI Compilers and Tools
Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
bash-4.2$ pgcc18 --version

pgcc18 19.10-0 linuxpower target on Linuxpower 
PGI Compilers and Tools
Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.

If I configure/make with CC=pgcc CXX=pgc++ FC=pgfortran and the current master (wiht your fix from PR #7385 it build successfully. Yay!

However, if I configure with CC=pgcc18 CXX=pgc++ FC=pgfortran then configure fails with the following in the OpenPMIx check:

checking alignment of bool... configure: WARNING: *** Problem running configure test!
configure: WARNING: *** See config.log for details.
configure: error: *** Cannot continue.
configure: /bin/sh './configure' *failed* for opal/mca/pmix/pmix4x/openpmix
checking if v4.x component is to be used... yes - using the internal v4.x library
configure: WARNING: INTERNAL PMIX FAILED TO CONFIGURE
configure: error: CANNOT CONTINUE

OpenPMIx has just incorporated the change in https://github.com/openpmix/openpmix/pull/1628

With https://github.com/openpmix/openpmix/pull/1628 then CC=pgcc18 CXX=pgc++ FC=pgfortran works. Yay! Note that the PGI compiler is throwing a ton of warnings now that probably should be cleaned up.

So the suggestion from PGI is to move to use pgcc18. So we will want to backport these fixes to the release branches, correct?

In the meantime I'm going to turn back on the IBM PGI CI test for master. We can add the release branches once the fix is backported.

[rhc54]

FWIW: I "upped" the openpmix pointer to include the referenced change in #7386. Just waiting to get that one thru CI

[awlauria]

@jjhursey I'm testing PGI18 right now on 3.0.x.

I think there's no harm in pushing these to the release branches and update our MTT to use it at this point though. I can do that cherry-picking.

I am seeing a lot more warnings as well that should get looked at.

[jjhursey]

I'll do the cherry-picks for OpenPMIx and PRRTE, if you can do the OMPI branches.

We will try to get this into the OpenPMIx 3.1.5 for the OMPI v4.0.x stream.

[awlauria]

ompi pr's:

7390

7391

7392

[jjhursey]

PRRTE (done):

• https://github.com/openpmix/prrte/commit/8dcf76b874c072654e4e8ccae605ab45448da4ce

OpenPMIx

• master (done) https://github.com/openpmix/openpmix/pull/1628
• v3.1 (done) https://github.com/openpmix/openpmix/pull/1630
  
  
  
  • Fix will be in v3.1.5
  
  

[awlauria]

pgcc18 seems _better,_ but it still has issues. Once the branches are ready we'll merge the change on our end to use pgcc18, and will monitor for the same bogus errors. I tested on 3.0.xrc6 for reference.

I let PGI know that I was still able to reproduce some invalid memory accesses that might cause intermittent failures with CC=pgcc18.

[awlauria]

With the change made to CC=pgcc18, looks like there is more configury work to do on the 2.x and 4.x branches.

2.x failed with the same error that 3 and 4.x branches saw, with configure thinking pgcc = gcc.
Since I assume 2.x is closed, we'll need to figure out some way to use CC=pgcc for it I think.

4.x failed with what looks like a similar configure issue in PMIx.

[rhc54]

4.x failed with what looks like a similar configure issue in PMIx.

Fixed in the upcoming PMIx v3.1.5 release, scheduled for next OMPI v4.0.x

[awlauria]

I haven't seen any mtt PGI failures from ibm in a while, so I am closing this.