Ompi: package version for man pages not set properly in Makefiles

[DmitryLyakh]

Thank you for taking the time to submit an issue!

Background information

I am consistently getting Open MPI build failure on multiple Ubuntu 18.04 machines:

Making all in mca/crs
make[2]: Entering directory '/home/dima/src/openmpi-3.1.3/build_gcc820/opal/mca/crs'
  CC       base/crs_base_open.lo
  CC       base/crs_base_close.lo
  CC       base/crs_base_select.lo
  CC       base/crs_base_fns.lo
  CCLD     libmca_crs.la
ar: `u' modifier ignored since `D' is the default (see `U')
  GENERATE opal_crs.7
Option package-version requires an argument
Usage: ../../../../ompi/mpi/man/make_manpage.pl --package-name=<package name> --package-version=<package version> --ompi-date=<ompi date> --opal-date=<opal date> --orte-date=<orte date> --input=<input file> --output=<output file> [--nocxx] [ --nofortran] [--nof08]
Makefile:2213: recipe for target 'opal_crs.7' failed
make[2]: *** [opal_crs.7] Error 1
make[2]: Leaving directory '/home/dima/src/openmpi-3.1.3/build_gcc820/opal/mca/crs'
Makefile:2377: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/dima/src/openmpi-3.1.3/build_gcc820/opal'
Makefile:1899: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

v.3.1.3

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From source:

cd source
mkdir build
cd build
../configure --prefix=path
make (or make all)
FAILURE (above)

Please describe the system on which you are running

Ubuntu 18.04
x86_64
Single node (laptop and desktop)

• Operating system/version:
• Computer hardware:
• Network type:

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

I am consistently getting Open MPI build failure on multiple Ubuntu 18.04 machines:

Making all in mca/crs
make[2]: Entering directory '/home/dima/src/openmpi-3.1.3/build_gcc820/opal/mca/crs'
  CC       base/crs_base_open.lo
  CC       base/crs_base_close.lo
  CC       base/crs_base_select.lo
  CC       base/crs_base_fns.lo
  CCLD     libmca_crs.la
ar: `u' modifier ignored since `D' is the default (see `U')
  GENERATE opal_crs.7
Option package-version requires an argument
Usage: ../../../../ompi/mpi/man/make_manpage.pl --package-name=<package name> --package-version=<package version> --ompi-date=<ompi date> --opal-date=<opal date> --orte-date=<orte date> --input=<input file> --output=<output file> [--nocxx] [ --nofortran] [--nof08]
Makefile:2213: recipe for target 'opal_crs.7' failed
make[2]: *** [opal_crs.7] Error 1
make[2]: Leaving directory '/home/dima/src/openmpi-3.1.3/build_gcc820/opal/mca/crs'
Makefile:2377: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/dima/src/openmpi-3.1.3/build_gcc820/opal'
Makefile:1899: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

[DmitryLyakh]

It looks like the configuration step is not setting --package-version in many Makefiles properly, needed for generating man pages.

[ggouaillardet]

fwiw, I am unable to reproduce this issue on an ubuntu 18.04 (I am using stock gcc 7.3, but I doubt this is even relevant here)

can you

from source
cd build
cd opal/mca/crs
rm -f opal_crs.7
make V=1 opal_crs.7

also, can you double check ../configure contains the following lines

# Identity of this package.
PACKAGE_NAME='Open MPI'
PACKAGE_TARNAME='openmpi'
PACKAGE_VERSION='3.1.3'
PACKAGE_STRING='Open MPI 3.1.3'
PACKAGE_BUGREPORT='http://www.open-mpi.org/community/help/'
PACKAGE_URL=''

[jsquyres]

In addition to what @ggouaillardet asked for, can you attach the information requested for build problems listed here: https://www.open-mpi.org/community/help/

Thanks!

[DmitryLyakh]

Here is what I see in my configure script, which explains why the package version is missing in Makefiles:

Identity of this package.

PACKAGE_NAME='Open MPI'
PACKAGE_TARNAME='openmpi'
PACKAGE_VERSION=''
PACKAGE_STRING='Open MPI '
PACKAGE_BUGREPORT='http://www.open-mpi.org/community/help/'
PACKAGE_URL=''

[DmitryLyakh]

So, more precisely, I do see PACKAGE_VERSION in the original configure, but at some point it gets lost, not sure why would that be happening ...

[DmitryLyakh]

I also see this message when running "make all", I guess could be related:
sh: 1: config/opal_get_version.sh: not found

[DmitryLyakh]

Essentially, the workaround is to run "configure", then "make all", which will fail because it erases PACKAGE_VERSION from "configure". Then, just restore PACKAGE_VERSION in "configure" manually and rerun "make all". It builds successfully. I am OK closing this issue on my side. Thanks!

[jsquyres]

This is the problem:

config/opal_get_version.sh: not found

Are you running autogen.pl? Or are you just downloading an Open MPI tarball and building from there?

[DmitryLyakh]

This was a tarball from the Open MPI website (version 3.1.3) and I just followed the build instructions therein. Specifically:
configure
make all
make install
I did not do anything else.

[DmitryLyakh]

By the way, on my systems the "config/opal_get_version.sh: not found" issue shows up with Open MPI 3.1.0, 3.1.3, and 4.0.0rc5, that is, all versions I have tried so far. "make all" erases PACKAGE_VERSION in "configure" for some reason.

[ggouaillardet]

configure is overwritten explicitly when you run autogen.sh (but you do not need that since you are building from an official tarball) or implicitly by make if some dependencies are more recent.

That should never happen when building from an official tarball, and it does not happen in my environment. An option is the timestamps were messed up (for example you are not building from a fresh and official tarball, or you are building from a network filesystem with loose mount options related to timestamps or your workstation and the file server clocks are not synchronized).

I would suggest you download the official tarball again, untar it on a local filesystem, and then run configure && make.
If it fails, then please compress and upload the output so we can figure out what is going wrong.

[DmitryLyakh]

That's what I was doing, already multiple times:
Download official tarball to my local workstation;
Extract the source;
cd source;
mkdir build;
cd build;
../configure --prefix=path;
make all;
make install;

Fails every time. Which specific output do you need? Configure + "make all" log?

[jsquyres]

I am unable to replicate your error; config/opal_get_version.sh is present in every one of the tarballs that I have tested.

Please upload all the data requested here: https://www.open-mpi.org/community/help/

[gongotar]

Same issue here, Have you found the solution?
I get repeated messages indicating opal get version script is not found after running configure.

config/opal_get_version.sh: not found

Then in the make call I get the error message:

recipe for target 'opal_crs.7' failed

The configure file contains also following lines:

# Identity of this package.
PACKAGE_NAME='Open MPI'
PACKAGE_TARNAME='openmpi'
PACKAGE_VERSION='3.1.3'
PACKAGE_STRING='Open MPI 3.1.3'
PACKAGE_BUGREPORT='http://www.open-mpi.org/community/help/'
PACKAGE_URL=''

I run all the commands in an Ubuntu 16.04.5 LTS system and I'm installing MPI according to the instructions in the documentation (./configure --prefix=path; make; make install).

[ggouaillardet]

both @jsquyres and I are unable to reproduce the issue, and we did not receive any logs so far.

can you double check your opal/mca/crs/Makefile, it should contain

.7in.7:
        $(OMPI_V_GEN) $(top_srcdir)/ompi/mpi/man/make_manpage.pl \
          --package-name='Open MPI' \
          --package-version='3.1.3' \
          --ompi-date='Oct 29, 2018' \
          --opal-date='Oct 29, 2018' \
          --orte-date='Oct 29, 2018' \
          --input=$< \
          --output=$@

if some fields are missing, can you restart from a fresh tarball, run configure and then compress and upload the output ?

also, can you post the output of

pwd
cat /proc/mounts

so I can double check the filesystem (and mount options) you are installing from.

[ggouaillardet]

@jsquyres I cannot find config/opal_get_version.sh in any tarballs (I tested 3.0.3, 3.1.2 and 3.1.3).

We have config/opal_get_version.m4sh and config/opal_get_version.m4 (config/opal_get_version.sh is basically generated at autogen.pl time).

But we should not need config/opal_get_version.sh if all the timestamps are fine ...

[gongotar]

The file opal/mca/crs/Makefile contains following:

.7in.7:
        $(OMPI_V_GEN) $(top_srcdir)/ompi/mpi/man/make_manpage.pl \
          --package-name='Open MPI' \
          --package-version='' \
          --ompi-date='May 07, 2018' \
          --opal-date='May 07, 2018' \
          --orte-date='May 07, 2018' \
          --input=$< \
          --output=$@

The output of pwd is as follows:
/scratch/mpi_path
Should I post the rest of required information in the developers mailing list?

[ggouaillardet]

before posting all the logs, can you post the output of

cat /proc/mounts
cd opal/mca/crs
make V=1 opal_crs.7

[jsquyres]

The only way you should be looking for config/opal_get_version.sh is if you are somehow running autogen.pl, which you should not be running from a bootstrapped tarball (i.e., it's already been run / that's what a bootstrapped tarball is).

One possibility is that autogen is being invoked because you have a file server time mismatch that is causing autogen to get invoked.

We need to see your full logs.

[gongotar]

Output of make V=1 opal_crs.7:

../../../ompi/mpi/man/make_manpage.pl \
  --package-name='Open MPI' \
  --package-version='' \
  --ompi-date='Oct 29, 2018' \
  --opal-date='Oct 29, 2018' \
  --orte-date='Oct 29, 2018' \
  --input=opal_crs.7in \
  --output=opal_crs.7
Option package-version requires an argument
Usage: ../../../ompi/mpi/man/make_manpage.pl --package-name=<package name> --package-version=<package version> --ompi-date=<ompi date> --opal-date=<opal date> --orte-date=<orte date> --input=<input file> --output=<output file> [--nocxx] [ --nofortran] [--nof08]
Makefile:2213: recipe for target 'opal_crs.7' failed
make: *** [opal_crs.7] Error 1

Output of cat /proc/mounts:

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=65737904k,nr_inodes=16484476,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=13191572k,mode=755 0 0
/dev/sda1 / xfs rw,noatime,attr2,inode64,noquota 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=24,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
systemd-1 /nfs/csr autofs rw,relatime,fd=38,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
systemd-1 /nfs/tmpe autofs rw,relatime,fd=39,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
systemd-1 /nfs/software autofs rw,relatime,fd=45,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
systemd-1 /nfs/scratch autofs rw,relatime,fd=46,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
systemd-1 /nfs/dasw autofs rw,relatime,fd=47,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
tracefs /sys/kernel/debug/tracing tracefs rw,relatime 0 0
none /tmp tmpfs rw,nosuid,nodev,relatime,size=10555168k 0 0
none /var/tmp tmpfs rw,nosuid,nodev,relatime,size=10555168k 0 0
/dev/sdb1 /scratch xfs rw,nosuid,nodev,noatime,attr2,inode64,noquota 0 0
/dev/sda2 /local xfs rw,nosuid,nodev,noatime,attr2,inode64,noquota 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
data-domain:/global/home/csr /nfs/csr nfs rw,nosuid,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nolock,noacl,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=<ip>,mountvers=3,mountport=45797,mountproto=tcp,local_lock=all,addr=<ip> 0 0
tmpfs /run/user/5989 tmpfs rw,nosuid,nodev,relatime,size=13191572k,mode=700,uid=5989,gid=2068 0 0
data-domain:/global/dasw /nfs/dasw nfs rw,nosuid,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nolock,noacl,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=<ip>,mountvers=3,mountport=45797,mountproto=tcp,local_lock=all,addr=<ip> 0 0
tmpfs /run/user/6208 tmpfs rw,nosuid,nodev,relatime,size=12191572k,mode=700,uid=6208,gid=2068 0 0
tmpfs /run/user/6220 tmpfs rw,nosuid,nodev,relatime,size=12191572k,mode=700,uid=6220,gid=2068 0 0

I have also posted the logs (make and config outputs) in the devel mailing list.

[jsquyres]

config.out.txt
make.out.txt

Per https://www.open-mpi.org/community/help/, the devel mailing list has a max attachment size of 150KB; you did not compress the files, so the mail was over 150KB and it did not post to the list. I attached your files here.

The problem is as we suspected: you are building Open MPI on a network filesystem, and the machine you are building on is not time-synchronized to the file server. This causes the timestamps on the filesystem to be all out of whack, and rules fire that should not fire. Specifically: when you run make, it ends up invoking the GNU Autotools, which, in turn, try to run things like opal_get_version.sh.

There are two ways to fix this:

1. Build on a non-networked filesystem (i.e., on a local filesystem, where the file timestamps will be correct).
2. Time-synchronize your build machine with your fileserver. This will ensure that the timestamps on the network filesystem will be correct, and Open MPI (and any Autotools-based package) will build properly.

Make sense?

[DmitryLyakh]

Yet another quite simple way to build it, as I mentioned before, is to do this:
The workaround is to run "configure", then "make all", which will fail because it erases PACKAGE_VERSION from "configure". Then, just restore PACKAGE_VERSION in "configure" manually and rerun "make all". It builds successfully. Worked for me.

[gongotar]

Thanks for the answers, I have changed the code and install directory to /local/openmpi-3.1.3 and /local/mpi_path which is local to the build machine. But still I get same outputs from configure and make. Is there any other possibilities explaining why this happens?

[jsquyres]

I'm sorry if I was not clear: it’s not your prefix that is the problem, it’s your build directory.

Build on a local directory (or fix the time sync with the file server), and I promise that the problem will go away.

The problem with the network file system timestamps being wrong is twofold:

1. You’re getting effectively non-deterministic behavior because you’re using the system in a way for which it was not designed. The Open MPI build system expects file system timestamps to be accurate. If they are not, we cannot guarantee that you are getting a correct build. It may seem to be correct, but be aware that open MPI is a large, complicated code base. There may be something incorrect deep in a small subsystem somewhere.
2. When you invoke make, it should not be invoking autogen and configure again. It is doing so because it senses errors and is trying to compensate. You are seeing one side effect (no package version), but there could well be others.

[gongotar]

I have moved the code to the local directory /local/openmpi-3.1.3 (this is the downloaded code from the website). Then I cd to the code directory (which is local /local/openmpi-3.1.3) and call ./configure --prefix=/local/mpi_path on that directory. If I am correct this means that the build directory is local to the machine because the code is built on the same directory /local/openmpi-3.1.3 (as well as the prefix directory /local/mpi_path which is local).

Am I missing something?

[jsquyres]

That is correct: the build directory on /local/blah should be ok (assuming /local is, as its name implies, is a local filesystem).

But I wouldn't move the source tree there -- you already have a mucked up source/build tree, and just moving it to a local filesystem isn't going to fix it. I'd re-extract the Open MPI tarball from scratch. Something like this:

$ cd /local
$ tar xf openmpi-4.0.0.tar.bz2
$ cd openmpi-4.0.0
$ ./configure --prefix=/whatever ... |& tee config.out
$ make -j 32 |& tee make.out
# If your prefix is not writable by your user, use "sudo"
$ sudo make install |& tee install.out

[gongotar]

OK, so I have just moved the tarball to the destination and extracted it. After that I could run make successfully.
What confuses me is that by applying this trick (extract the tarball _after_ moving to the destination instead of extracting it _before_ the movement) now I can compile it also on non-local destinations (for example in /scratch).
I have repeated the tests to confirm this and every time I get same results (extract the tarball locally on my laptop then scp -r the directory to the cluster vs scp the tarball to the cluster and extract it there). I would say maybe it isn't about timestamps at all.

[ggouaillardet]

glad to hear your issue is fixed ! aclocal-1.15 is invoked by make, and make use timestamps (and file existence) in order to compute dependencies. That being said, you are free to believe this was not a timestamp/clock issue.

[DmitryLyakh]

Same worked for me, extracting .tar in the same directory solves the problem.

[DmitryLyakh]

So, is it moving the extracted folder that messes up the timestamps? If so, I think it is a good idea to document this in the README, to help avoid future help requests :)

[jsquyres]

@gongotar You're seeing the non-deterministic effects of having a filesystem have incorrect timestamps. It throws off all the hierarchy of rules and dependencies in the mesh of Makefiles that are used to build Open MPI.

@DmitryLyakh Fair point; this question has come up several times over the years. I thought we documented this somewhere, but if we did, it probably isn't strong enough (and/or we need to make the answer be google-able). We're all at the supercomputing trade show this week -- so we probably won't have time to do it this week, but it's now on the to-do list.

So just to be clear: the underlying issue is when the machine on which you are building is not time-synchronized to the file server that is providing the filesystem on which you are using (i.e., /scratch, in this example). If you get both the build machine and the file server time synchronized properly (e.g., if they both synchronize to an NTP server in the same or similar stratum), then the timestamps will be correct, the mesh of dependencies in Open MPI's Makefiles will be executed correctly, ...etc.

[jsquyres]

I strengthened the FAQ text about network filesystem timestamps: https://www.open-mpi.org/faq/?category=building#unsynchronized-timestamps

I also filed #6092 to add text to README about this issue.

Thanks for reporting the issue!

[swprf9]

Well, I had the same issue. There is a lot of manual work with manpages compilation in ompi directory (bug?), and finally I wrote script, which do this automatically. Execute the following script from the ompi directory:

!/bin/bash

for i in mpi/man/man3/*.3in;
do
j=${i/.3in/.3}
echo $i $j
../ompi/mpi/man/make_manpage.pl \
--package-name='Open MPI' \
--package-version='3.1.4' \
--ompi-date='Apr 15, 2019' \
--opal-date='Apr 15, 2019' \
--orte-date='Apr 15, 2019' \
--input=$i \
--output=$j
done

[AlessioLuciani]

I had the same issue, I had not installed the gfortran compiler. Once I did, I runned again the ./configure and the make again. It worked just fine.