Ompi: Vader in a Docker Container

[ax3l]

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

OpenMPI 3.0.0 (and 2.1.2 for comparisons)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From source (via spack) inside a docker container.

Please describe the system on which you are running

• Operating system/version: Ubuntu "xenial" 16.04.4 LTS with kernel 4.4.0-116 (host), Ubuntu "xenial" 16.04.4 LTS with kernel 4.4.0-116 (container)
• Computer hardware: two NUMA nodes with each an Intel Xeon E5-2698v4 (Nvidia DGX-1)
• Network type: only in-node relevant for now.

---

Details of the problem

Starting MPI with more than one rank will result in errors of the form

Read -1, expected <someNumber>, errno =1
Read -1, expected <someNumber>, errno =1
Read -1, expected <someNumber>, errno =1
Read -1, expected <someNumber>, errno =1
Read -1, expected <someNumber>, errno =1
...

as soon as communication is performed (send, receive, reduce, etc.). Simple programs that only contain MPI startup (Init, Get_Rank, Finalize) and shutdown run without issues.

The only way to work-around this issue was for me to down-grade to OpenMPI 2.X which still supports "sm" as a BTL and deactivating vader, e.g. with export OMPI_MCA_btl="^vader".

Is it possible the detection/test of a working CMA is not fully functional? This issue is likely caused by either a non-existent or not-fully forwarded CMA kernel support inside the docker container. Do you have any recommendations on how to use vader in such an environment as the in-node BTL?

[jsquyres]

What are those error messages from -- are they from your program? I.e., what exactly are those error messages indicating?

FWIW: I do not believe we have any tests -- in configure or otherwise -- to check for non-functional CMA. If we find CMA support, we assume it's working. Vader should work just fine if CMA support is not present -- it will fall back to regular copy-in-copy-out shared memory.

[hppritcha]

Are the processes within different docker containers? If so, then its likely CMA is failing because the containers may be in different name spaces. The workaround is to disable CMA on the mpirun command line:

--mca btl_vader_single_copy_mechanism=none

[ax3l]

Thank you for the details!

What are those error messages from -- are they from your program?

No they originate from OpenMPI. I guess from here:

https://github.com/open-mpi/ompi/blob/v3.0.0/opal/mca/btl/vader/btl_vader_get.c#L74-L78

Are the processes within different docker containers?

No it's the same container on a Nvidia DGX-1 (low detail datasheet, detailed guide) which has two Xeon packages in case that is relevant. Might be, I am not sure if they have regular QPI on it. QPI is there (last link, page 6)

We will try to debug it hands-on again with Nvidia engineers next week. I was just wondering if the error (see code lines above) already tells you something that could give me pointers on how to debug CMA (or if you had runtime CMA tests in place).

I see that you define OPAL_BTL_VADER_HAVE_CMA at compile time, which is of course a bit tricky for an image that is shipped around (might exist on the build machine but not on the run machine).

[ggouaillardet]

@ax3l the issue here is that process_vm_readv() fails with EPERM which is suspicious.

The root cause could be docker prevents this, and some sysadmin config might be required.

From the man page

In order to read from or write to another process, either the caller must have the capability CAP_SYS_PTRACE, or the real user ID, effective user ID, and saved set-user-ID of the remote process must match the real user ID of the caller and the real group ID, effective group ID, and saved set-group-ID of the remote process must match the real group ID of the caller. (The permission required is exactly the same as that required to perform a ptrace(2) PTRACE_ATTACH on the remote process.)

you might want to manually check those conditions are met as well.
there is a theoretical risk your kernel incorrectly supports this in the context of docker (e.g. namespaces).

I would suggest you first try to run your app on the host first, and then in the container.
As pointed earlier

mpirun --mca btl_vader_single_copy_mechanism none ...

might help you

Note the btl/sm is still available in recent Open MPI versions, so

mpirun --mca btl ^vader ...

might also help you here.

[hjelmn]

@ggouaillardet btl/sm is not needed. The way we are recommending to deal with docker if the ptrace permissions can't be fixed is to set OMPI_MCA_btl_vader_single_copy_mechanism=none. That will disable CMA.

[felker]

I recently upgraded the OpenMPI library in our project's Travis CI setup from 2.1.1 to 3.0.2 (and tried 3.1.0), and we were observing

Read -1, expected 5120, errno = 1
Read -1, expected 5120, errno = 1
Read -1, expected 5120, errno = 1
....

as soon as mpirun was launched with 2 or more ranks on the Ubuntu Trusty Docker image build environments.

Setting

export OMPI_MCA_btl_vader_single_copy_mechanism=none

before launching the jobs, as @hjelmn suggested, seems to have fixed the problem in 3.0.2 for us.

[hjelmn]

Note if you want better performance you want CMA to work. It will only work if all the local MPI processes are in the same namespace.

An alternative (I haven't tested this) would be to use xpmem: http://gitlab.com/hjelmn/xpmem (there is a version on github but it will no longer be maintained).

[blechta]

Note that one can allow ptrace permissions by docker run --cap-add=SYS_PTRACE ... which seems to get CMA working.

[JiaweiZhuang]

Setting

export OMPI_MCA_btl_vader_single_copy_mechanism=none

Got exactly the same issue with OpenMPI 3.1.3 in Docker. This fixes the problem!

[milthorpe]

Same problem on OpenMPI 4.0.0 in Docker; disabling vader single copy mechanism as suggested above fixes it.

[AdamSimpson]

FWIW the root cause of this is likely that process_vm_readv()/process_vm_writev() are disabled in the default Docker seccomp profile. A slightly less heavy handed option than --cap-add=SYS_PTRACE would be to modify the seccomp profile so that process_vm_readv and process_vm_writev are whitelisted, by adding them to the syscalls.names list. Having done that I was able to use vader and UCX with CMA without issue.

Edit:
It is also worth mentioning that if you're still running into CMA issues: https://github.com/openucx/ucx/issues/3545

[ssharma94]

export OMPI_MCA_btl_vader_single_copy_mechanism=none

indeed it worked!!!

[adrianreber]

This should be fixed with #6844 (Assuming docker uses user namespaces).