Ompi: Installation Failed! (UCX compile error)

[usamazf]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

4.0.1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

distribution tarball.

Please describe the system on which you are running

• Operating system/version: Ubuntu 18.04
• Computer hardware: Tesla K40c (2 slots)
• Network type:

---

Details of the problem

I am having problem with installation of openmpi v4.0.1 with cuda support. Previously I installed mpi v3.1.4 and it installs perfectly but v4.0.1 fails to install giving out following error:

btl_uct_module.c: In function ‘mca_btl_uct_reg_mem’:
btl_uct_module.c:214:22: error: ‘UCT_MD_MEM_ACCESS_REMOTE_GET’ undeclared (first use in this function); did you mean ‘MCA_RCACHE_ACCESS_REMOTE_READ’?
         uct_flags |= UCT_MD_MEM_ACCESS_REMOTE_GET;
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
                      MCA_RCACHE_ACCESS_REMOTE_READ
btl_uct_module.c:214:22: note: each undeclared identifier is reported only once for each function it appears in
btl_uct_module.c:217:22: error: ‘UCT_MD_MEM_ACCESS_REMOTE_PUT’ undeclared (first use in this function); did you mean ‘UCT_MD_MEM_ACCESS_REMOTE_GET’?
         uct_flags |= UCT_MD_MEM_ACCESS_REMOTE_PUT;
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
                      UCT_MD_MEM_ACCESS_REMOTE_GET
btl_uct_module.c:220:22: error: ‘UCT_MD_MEM_ACCESS_REMOTE_ATOMIC’ undeclared (first use in this function); did you mean ‘UCT_MD_MEM_ACCESS_REMOTE_PUT’?
         uct_flags |= UCT_MD_MEM_ACCESS_REMOTE_ATOMIC;
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                      UCT_MD_MEM_ACCESS_REMOTE_PUT
btl_uct_module.c:225:21: error: ‘UCT_MD_MEM_ACCESS_ALL’ undeclared (first use in this function); did you mean ‘UCT_MD_MEM_ACCESS_REMOTE_PUT’?
         uct_flags = UCT_MD_MEM_ACCESS_ALL;

Any help please?

[usamazf]

I have noticed that the error occurs with version 4.0.0 also. But when I go back to version 3.1.4 installation proceeds perfectly.

[jsquyres]

You're having a problem with the UCX BTL plugin module (which was not included in the v3.1.x series). Do you have the latest version of UCX, perchance?

If it continues to be a problem, you can disable building the UCX BTL with:

$ ./configure --enable-mca-no-build=btl-ucx ...

[jladd-mlnx]

Which version of UCX are you using, @Usama-Zafar? UCX BTL is no longer maintained and is not compatible with the latest version of (UCX v1.6.)

FYI @hppritcha - now might be a good time to remove the UCT BTL.

[usamazf]

@jsquyres I have tried to do disable UCX BTL as you advised but the problem still seems to occur.

@jladd-mlnx I am not sure how to check the version of UCX, I am using an already configured server for my task.

[rhc54]

$ ./configure --enable-mca-no-build=btl-ucx ...

This is a typo - the correct configure line is:

$ ./configure --enable-mca-no-build=btl-uct ...

[jsquyres]

Oops! @rhc54 is correct -- sorry for the typo...

[hppritcha]

@Usama-Zafar are you using UCX master? There is no 1.6 release of UCX at this time.

[hppritcha]

@Usama-Zafar what version of UCX are you using? @jladd-mlnx there isn't a 1.6 release of ucx, at least not on github.

[yosefe]

Seems like a pretty old release of UCX is used (1.3?), since the issue is that UCT_MD_MEM_ACCESS_REMOTE_PUT flag is not found in UCT API.

[jladd-mlnx]

Bottom line, UCT BTL only works for UCX v1.4 and v1.5. Without a maintainer, it will be DOA when UCX v1.6 is released this month.

[rhc54]

I think we need to be careful to separate out the problems here so we don't confuse this user. The issue encountered here is that @Usama-Zafar is using too old a version of UCX - the problem can be resolved, therefore, by either updating UCX or by configuring to no-build the uct btl.

A separate issue that has been raised during the discussion is what to do with that BTL going forward. That isn't really relevant to this user's question and is something perhaps best discussed on the developer's telecon. As @jsquyres has pointed out, our release rules preclude a simple removal, and as @jladd-mlnx has noted, we don't currently have a designated maintainer. Those are not user-facing issues, but rather are internal matters.

[usamazf]

@rhc54 I was able to successfully install openmpi v4.0.1 with the configuration flags specified. Thanks! I will go ahead and close the issue!

[hppritcha]

we need to fix this so users can just do a simple configure and go

[jsquyres]

Marking this as a blocker for v4.0.x.

[hjelmn]

You need to TAG ME when you open a btl/uct bug if you want it looked at.

[hjelmn]

Will be fixed later today.

[hjelmn]

So, the user's version appears to be older than v1.3.0. Will disable btl/uct when the version is that old. Have no interest in supporting that old a version. v1.6.x support is being tested now. Will have master support ready tomorrow.

[rhc54]

Thanks @hjelmn !

[gpaulsen]

Fixed by #6670 (master)

[gpaulsen]

PRed to v4.0.x via pr #6748

[gpaulsen]

pr #6748 failing in cray Ci. Possible cluster issue.