Ompi: Error in using openMPI with my fortran code

Created on 2 Feb 2019  路  25Comments  路  Source: open-mpi/ompi

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Please describe the system on which you are running

  • Operating system/version:
  • Computer hardware:
  • Network type:

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

Note: If you include verbatim output (or a code block), please use a GitHub Markdown code block like below:

shell$ mpirun -np 2 ./hello_world
Awaiting user information question
Source
picture MostafaMAshry

All 25 comments

I try to use openMPI in Linuxmint OS, i write a FORTRAN90 code using Modules,
when I try to run my code it gives this error:

[ashry-Precision-T1700:14520] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:14520] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:14522] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:14522] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:14522] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:14521] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:14521] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:14521] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:14520] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:14522] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:14519] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:14519] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:14519] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:14519] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:14521] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:14520] OPAL ERROR: Error in file pmix3x_client.c at line 113
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:14521] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:14519] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:14520] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:14522] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[56733,1],2]
  Exit code:    1
--------------------------------------------------------------------------

And my mpirun --version is
mpirun (Open MPI) 2.1.1

Report bugs to http://www.open-mpi.org/community/help/

I don't know the source of this error, I will run this program in workstation core i 32, but first i want to check it first in my PC

MostafaMAshry picture MostafaMAshry on 2 Feb 2019

Can you submit answers to the question from the template bug report? (what version of Open MPI are you using, how did you install it, ...etc.)

jsquyres picture jsquyres on 2 Feb 2019

My Open MPI version is 4.0.0,and
I install it :

$ ./configure --prefix=$HOME/openmpi
$ make all
$ make install

My operating system is LINUXmint

EDIT: added proper github markdown to make the copy-n-paste sections readable.

MostafaMAshry picture MostafaMAshry on 3 Feb 2019

When i try to use my host file 

mpirun -np 4 --map-by node -machinefile myhosts GGeoid
it gives this error:
ssh: Could not resolve hostname node3: Name or service not known
ssh: Could not resolve hostname node4: Name or service not known
ssh: Could not resolve hostname node2: Name or service not known
ssh: Could not resolve hostname node1: Name or service not known
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:

* not finding the required libraries and/or binaries on
  one or more nodes. Please check your PATH and LD_LIBRARY_PATH
  settings, or configure OMPI with --enable-orterun-prefix-by-default

* lack of authority to execute on one or more specified nodes.
  Please verify your allocation and authorities.

* the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
  Please check with your sys admin to determine the correct location to use.

*  compilation of the orted with dynamic libraries when static are required
  (e.g., on Cray). Please check your configure cmd line and consider using
  one of the contrib/platform definitions for your system type.

* an inability to create a connection back to mpirun due to a
  lack of common network interfaces and/or no route found between
  them. Please check network connectivity (including firewalls
  and network routing requirements).
--------------------------------------------------------------------------
[ashry-Precision-T1700:16150] 2 more processes have sent help message help-errmgr-base.txt / failed-daemon-launch
[ashry-Precision-T1700:16150] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
----------------------------------

this is my-host file:

#data
node1
node2
node3
node4

when i try tompirun GGeoid
mpirun GGeoid

EDIT: added proper github markdown to make the copy-n-paste sections readable.

MostafaMAshry picture MostafaMAshry on 3 Feb 2019

Note -- you can use Github's "Markdown" syntax to make copy-n-paste error output render significantly nicer -- put a line with 3 single tick marks, then all your verbatim output, and then another 3-single-tick-mark line to close off the verbatim section. See Fenced Code blocks in the Github Markdown spec.

Your situation is typically symptomatic of accidentally mixing multiple versions of Open MPI across multiple nodes. E.g., did you set your PATH and LD_LIBRARY_PATH in your shell startup files? See this FAQ entry. You might also consider adding --enable-mpirun-prefix-by-default to your configure line, if setting PATH / LD_LIBRARY_PATH in your shell startup files is not an option for some reason.

Additionally, you might want to try to latest Open MPI 4.0.1 nightly snapshot (it's not released yet, but it fixes some bugs from 4.0.0). See https://www.open-mpi.org/nightly/v4.0.x/

jsquyres picture jsquyres on 3 Feb 2019

Thank you very much for your reply, can you tell me the steps of adding --enable-mpirun-prefix-by-default to your configure line.
Should I remove Openmpi-4.0.0 before setup OpenMPI 4.0.1, and how I do that.
Thanks again

MostafaMAshry picture MostafaMAshry on 4 Feb 2019 
mpirun --prefix -by-default -np 4 ./GGeoid

and another time i try to use

mpirun --prefix /opt/openmpi-4.0.0 -np 4 ./GGeoid

and using this 

mpirun ./GGeoid

all of them gives this error

[ashry-Precision-T1700:20821] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:20821] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:20821] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:20824] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:20824] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:20824] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:20821] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:20824] OPAL ERROR: Error in file pmix3x_client.c at line 113
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:20824] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:20821] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[46207,1],3]
  Exit code:    1
MostafaMAshry picture MostafaMAshry on 4 Feb 2019

I install Open MPI 4.0.x and I face this warning, is this make any seance:

libtool: warning: relinking 'mca_regx_fwd.la'
---
----------------------------------------------------------------------
make[3]: Leaving directory '/home/ashry/openmpi-v4.0.x-201902010241-4dfb938/orte/mca/ras/slurm'
make[2]: Leaving directory '/home/ashry/openmpi-v4.0.x-201902010241-4dfb938/orte/mca/ras/slurm'
Making install in mca/regx/fwd
make[2]: Entering directory '/home/ashry/openmpi-v4.0.x-201902010241-4dfb938/orte/mca/regx/fwd'
make[3]: Entering directory '/home/ashry/openmpi-v4.0.x-201902010241-4dfb938/orte/mca/regx/fwd'
make[3]: Nothing to be done for 'install-exec-am'.
 /bin/mkdir -p '/home/ashry/openmpi/lib/openmpi'
 /bin/bash ../../../../libtool   --mode=install /usr/bin/install -c   mca_regx_fwd.la '/home/ashry/openmpi/lib/openmpi'
libtool: warning: relinking 'mca_regx_fwd.la'
libtool: install: (cd /home/ashry/openmpi-v4.0.x-201902010241-4dfb938/orte/mca/regx/fwd; /bin/bash "/home/ashry/openmpi-v4.0.x-201902010241-4dfb938/libtool"  --silent --tag CC --mode=relink gcc -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -mcx16 -pthread -module -avoid-version -o mca_regx_fwd.la -rpath /home/ashry/openmpi/lib/openmpi regx_fwd_component.lo regx_fwd.lo ../../../../orte/libopen-rte.la -lrt -lm -lutil -lz )
libtool: install: /usr/bin/install -c .libs/mca_regx_fwd.soT /home/ashry/openmpi/lib/openmpi/mca_regx_fwd.so
libtool: install: /usr/bin/install -c .libs/mca_regx_fwd.lai /home/ashry/openmpi/lib/openmpi/mca_regx_fwd.la
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/ashry/.openmpi/bin:/home/ashry/.openmpi/bin:/sbin" ldconfig -n /home/ashry/openmpi/lib/openmpi
----------------------------------------------------------------------
MostafaMAshry picture MostafaMAshry on 4 Feb 2019 
mpirun --prefix -by-default -np 4 ./GGeoid
 min  -260.75557500000002     
 min  -260.75557500000002     
 min  -260.75557500000002     
 min         569        1074
 min         569        1074
 min  -260.75557500000002     
 min         569        1074
 min         569        1074
 kmax           0
 kmax           0
 kmax           0
 kmax           0
 kmax           4
 kmax           4
 kmax           4
 kmax           4
           4
 flag_region           0
           4
 flag_region           0
           4
           4
 flag_region           0
 flag_region           0
[ashry-Precision-T1700:07869] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:07869] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:07869] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:07869] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:07870] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:07870] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:07870] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:07870] OPAL ERROR: Error in file pmix3x_client.c at line 113
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:07869] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:07870] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
[ashry-Precision-T1700:07872] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:07871] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:07871] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:07871] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:07871] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:07871] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[ashry-Precision-T1700:07872] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:07872] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:07872] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:07872] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
MostafaMAshry picture MostafaMAshry on 4 Feb 2019

you previously claimed mpirun --version is 2.1.1 and you are now installing Open MPI 4,0.0.

did you double check you are not using mpirun version 2.1.1 with your app linked with Open MPI 4,0.0 ?

what if you try a simple example such as hello_c from the examples directory ?

ggouaillardet picture ggouaillardet on 4 Feb 2019

Thanks for your time, I try to run hello_c.c

ashry@ashry-Precision-T1700:~/openmpi-4.0.0/examples$ mpicc hello_c.c
ashry@ashry-Precision-T1700:~/openmpi-4.0.0/examples$ ./a.out
Hello, world, I am 0 of 1, (Open MPI v4.0.0, package: Open MPI ashry@ashry-Precision-T1700 Distribution, ident: 4.0.0, repo rev: v4.0.0, Nov 12, 2018, 122)
ashry@ashry-Precision-T1700:~/openmpi-4.0.0/examples$ mpirun --version
mpirun (Open MPI) 2.1.1

Report bugs to http://www.open-mpi.org/community/help/
ashry@ashry-Precision-T1700:~/openmpi-4.0.0/examples$
MostafaMAshry picture MostafaMAshry on 4 Feb 2019 
ashry@ashry-Precision-T1700:~/openmpi-4.0.0/examples$ mpif90 hello_mpifh.f
ashry@ashry-Precision-T1700:~/openmpi-4.0.0/examples$ ./a,out
bash: ./a,out: No such file or directory
ashry@ashry-Precision-T1700:~/openmpi-4.0.0/examples$ ./a.out
Hello, world, I am  0 of  1: Open MPI v4.0.0, package: Open MPI ashry@ashry-Precision-T1700 Distribution, ident: 4.0.0, repo rev: v4.0.0, Nov 12, 2018                                                                                                                                      
ashry@ashry-Precision-T1700:~/openmpi-4.0.0/examples$ mpirun ./a.out
[ashry-Precision-T1700:01234] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:01233] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:01233] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:01233] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:01233] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:01234] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:01236] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:01236] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:01236] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:01234] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:01236] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:01234] OPAL ERROR: Error in file pmix3x_client.c at line 113
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:01233] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[ashry-Precision-T1700:01235] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:01235] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:01235] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:01235] OPAL ERROR: Error in file pmix3x_client.c at line 113
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:01236] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:01234] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[ashry-Precision-T1700:01235] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[57827,1],0]
  Exit code:    1
--------------------------------------------------------------------------
MostafaMAshry picture MostafaMAshry on 4 Feb 2019

For the third time, the root cause is you use mpirun 2.1.2 with an app linked with Open MPI 4.0.0.
Use the right Open MPI (e.g. fix your $PATH) and you will be just fine.

ggouaillardet picture ggouaillardet on 4 Feb 2019
👍1

sorry, but I don't know how to fix the path, Linuxmint

MostafaMAshry picture MostafaMAshry on 4 Feb 2019

/home/ashry/openmpi/bin/mpirun ./a.out

ggouaillardet picture ggouaillardet on 4 Feb 2019
👍1

Also see the Open MPI FAQ for some detailed information:

Finally, please use proper "fenced code blocks" when pasting giant amounts of verbatim text into the github issue -- it makes your text significantly easier for us to read and help you.

jsquyres picture jsquyres on 4 Feb 2019

Thanks a million, I will read this and try to fix the path,
thanks again

MostafaMAshry picture MostafaMAshry on 4 Feb 2019 
---<
ashry@ashry-Precision-T1700:~$ echo 'export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/ashry/.openmpi/lib/"' | tee -a /home/$USER/.bashrc
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/ashry/.openmpi/lib/">---
---<
ashry@ashry-Precision-T1700:~$ echo 'export PATH="/usr/bin/mpirun:/home/ashry/.openmpi/lib/mpirun"' | tee -a /home/$USER/.bashrc
export PATH="/usr/bin/mpirun:/home/ashry/.openmpi/lib/mpirun"
ashry@ashry-Precision-T1700:~$ which mpirun
/usr/bin/mpirun
ashry@ashry-Precision-T1700:~$ which mpif90
/home/ashry/.openmpi/bin/mpif90
>---

I try to change the path of mpirun but I don't know, kindly, could you correct this error for me!!

EDIT: added proper github markdown to make the copy-n-paste sections readable.

MostafaMAshry picture MostafaMAshry on 4 Feb 2019

I think you just overwrote your entire .bashrc, which is almost certainly not what you wanted to do.

You really just wanted to edit your existing .bashrc and add the statements to append to your PATH and LD_LIBRARY_PATH in to that existing .bashrc.

Alternatively, as I mentioned above, you could just add --enable-mpirun-prefix-by-default to the ./configure command line when you run it to build Open MPI. I.e.:

# Remove your old install first, just to be sure
$ rm -rf $HOME/openmpi

# Configure it with the proper option
$ ./configure --prefix=$HOME/openmpi --enable-mpirun-prefix-by-default

# Clean your previous build
$ make clean

# Make and install Open MPI again
$ make all
$ make install
jsquyres picture jsquyres on 4 Feb 2019

I still have the same problem, I try more than three times but unfortunately,
First, I reinstall OpenMPI using the:
```< $ rm -rf $HOME/openmpi
$ ./configure --prefix=$HOME/openmpi --enable-mpirun-prefix-by-default
$ make clean
$ make all
$ make install >


Then I try to check the version of mpirun, it gives:

< ashry@ashry-Precision-T1700:~/NGDPgeoid/src$ which mpirun
/usr/bin/mpirun
>

and when i try to know which mpif90, I found that there isn't mpif90,

<ashry@ashry-Precision-T1700:~/NGDPgeoid/src$ mpif90 --version

Command 'mpif90' not found, but can be installed with:

sudo apt install libmpich-dev
sudo apt install libopenmpi-dev
>


The error was:

<ashry@ashry-Precision-T1700:~/NGDPgeoid/src$ mpirun ./GGeoid
min -260.75557500000002
min -260.75557500000002
min -260.75557500000002
min 569 1074
min 569 1074
min 569 1074
min -260.75557500000002
min 569 1074
kmax 0
kmax 0
kmax 0
kmax 0
kmax 4
kmax 4
kmax 4
kmax 4
4
4
4
4
flag_region 0
flag_region 0
flag_region 0
flag_region 0
[ashry-Precision-T1700:08155] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:08155] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:08155] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:08155] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:08156] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:08156] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:08156] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:08156] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:08157] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:08157] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:08157] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:08157] OPAL ERROR: Error in file pmix3x_client.c at line 113
[ashry-Precision-T1700:08158] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 206
[ashry-Precision-T1700:08158] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 147
[ashry-Precision-T1700:08158] PMIX ERROR: UNPACK-PAST-END in file client/pmix_client.c at line 225
[ashry-Precision-T1700:08158] OPAL ERROR: Error in file pmix3x_client.c at line 113
* An error occurred in MPI_Init
 An error occurred in MPI_Init
 on a NULL communicator
 MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
 An error occurred in MPI_Init
 on a NULL communicator
 MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
 and potentially your MPI job)
 An error occurred in MPI_Init
 on a NULL communicator
 MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
 and potentially your MPI job)
 and potentially your MPI job)
[ashry-Precision-T1700:08155] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[ashry-Precision-T1700:08158] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[ashry-Precision-T1700:08156] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
 on a NULL communicator
 MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
* and potentially your MPI job)

[ashry-Precision-T1700:08157] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

Primary job terminated normally, but 1 process returned

a non-zero exit code.. Per user-direction, the job has been aborted.

mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[64249,1],2]

Exit code: 1

```

MostafaMAshry picture MostafaMAshry on 11 Feb 2019

I reinstall Ubuntu 18.10,
and install OpenMPi 4.0.0
and this is tthe config.log
config.log

now, I don't find either mpif90 nor mpirun installed in my system??

MostafaMAshry picture MostafaMAshry on 11 Feb 2019

This is the error now !!!!!

ashry@ashry-Precision-T1700:~/NGDPgeoid/src$ mpirun ./GGeoid
./GGeoid: error while loading shared libraries: libgfortran.so.4: cannot open shared object file: No such file or directory
./GGeoid: error while loading shared libraries: libgfortran.so.4: cannot open shared object file: No such file or directory
./GGeoid: error while loading shared libraries: libgfortran.so.4: cannot open shared object file: No such file or directory
./GGeoid: error while loading shared libraries: libgfortran.so.4: cannot open shared object file: No such file or directory
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------

and my Make file 

# ------------------ Macro-Defs ---------------------
#MF= Makefile
#FC= gfortran
FC= mpif90
#FFLAGS= -qsuffix=f=f90 -q64 -O3 -qarch=pwr4 -qtune=pwr4 -qsmp=omp,noauto
FFLAGS= -O3 -pipe -ftracer -fivopts -ftree-loop-linear -ftree-vectorize -fforce-addr -fomit-frame-pointer -fno-bounds-check -mtune=core2 -march=core2 -mfpmath=sse -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -funroll-loops
#FFLAGS= -O3 -xHost #-ipo
#SHFLAGS=-free -I$(SHTOOLSMODPATH) -I/opt/intel/mkl/include/fftw -I/opt/intel/mkl/include -L$(SHTOOLSLIBPATH) -lSHTOOLS2.8 -lm -m64 -mkl
#SHFLAGS=-free -I$SHTOOLSMODPATH -L$SHTOOLSLIBPATH -lSHTOOLS2.8 -lfftw3 -lm -llapack -lblas -O2 -m64 -Tf
#FFLAGS= -O3 -ffast-math -march=native \
    #-L /home/Frogge/Software/intel/lib/ia32/ -lsvml -limf -lintlc #svml
        #-mveclibabi=acml -m32 -L /home/Frogge/Software/library/acml/gfortran32_mp/lib/ -lacml_mp #acml
 #-lsvml -limf -lintlc

#Fopenmp= -fopenmp
#Fopenmp= -openmp 
#-heap-arrays
LFLAGS= $(FFLAGS)
# 0x40000000 is 1G, hexadecimal
-Wl,-rpath=.,-stack_size,0x40000000

#Target=regional
platform=$(shell uname)

MPIrun=mpirun
NP= -np 4


EXE= GGeoid
#EXE2= RGeoid
MostafaMAshry picture MostafaMAshry on 12 Feb 2019

I add the following two lines to the end of .bashrc:

export PATH=$PATH:$HOME/openmpi/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/openmpi/lib

then I got this error::

ashry@ashry-Precision-T1700:~/NGDPgeoid/src$ mpirun -np 4 ./GGeoid
mpirun: Symbol `orte_rml' has different size in shared object, consider re-linking
[ashry-Precision-T1700:00504] mca_base_component_repository_open: unable to open mca_ess_hnp: /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_ess_hnp.so: undefined symbol: orte_regx_base_framework (ignored)
MostafaMAshry picture MostafaMAshry on 12 Feb 2019

You're closer. A few points in no particular order...

The system was complaining about no mpif90 because it found no mpif90 in your PATH, but it knew that mpif90 was provided by the two packages that it suggested. I.e., if you installed openmpi-dev, then you would have mpif90. Specifically: your system is apparently dividing Open MPI up into multiple packages: a) stuff that you need to run Open MPI applications, and b) stuff that you need to build Open MPI applications. You probably installed a) by default, but not b). And b) is what contains mpif90.

That being said, just to completely avoid all confusion, you might want to fully uninstall the system-installed Open MPI (i.e., both the a) and b) packages). That way, you'll only be left with the Open MPI installed under your $HOME, and there's zero chance of confusion.

Once you do that, then your existing export statements in your .bashrc should be sufficient (that being said, if you leave the system-installed Open MPI, then you need to re-arrange the export statements to put your $HOME/openmpi entries at the beginning if your PATH and LD_LIBRARY_PATH -- so that the system finds your installation of Open MPI before it finds the system installation of Open MPI).

Once you have this all straightened out -- e.g., which mpirun returns /home/whatever/openmpi/bin/mpirun (and not /usr/bin/mpirun), you should recompile your application from scratch and try mpiruning again.

jsquyres picture jsquyres on 12 Feb 2019
👍1

Dear Jeff Squyres,
Thank you very much, finally the problem has been solved,

ashry@ashry-Precision-T1700:~/NGDPgeoid/src$ which mpif90
/home/ashry/openmpi/bin/mpif90
ashry@ashry-Precision-T1700:~/NGDPgeoid/src$ which mpirun
/home/ashry/openmpi/bin/mpirun

Thanks

MostafaMAshry picture MostafaMAshry on 12 Feb 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

yoshihingis picture yoshihingis  路  3Comments
PeterGottesman picture PeterGottesman  路  6Comments
amckinlay picture amckinlay  路  8Comments
abeltre1 picture abeltre1  路  4Comments
pharthiphan picture pharthiphan  路  7Comments