This is on current master, running on a 2-node cluster where each node has two subnets on it. When attempting to execute a 1-ppn MPI ring application, the job hangs for a minute or so and then emits the following error:
$ mpirun -npernode 1 ./ring
Rank 0 has cleared MPI_Init
Rank 1 has cleared MPI_Init
--------------------------------------------------------------------------
WARNING: Open MPI failed to TCP connect to a peer MPI process. This
should not happen.
Your Open MPI job may now hang or fail.
Local host: rhc001
PID: 13325
Message: connect() to 10.10.10.2:1024 failed
Error: Operation now in progress (115)
--------------------------------------------------------------------------
My ifconfig output shows the following subnet definitions (trimmed of unnecessary stuff):
For node1:
$ ifconfig
ens2f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.161 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::21e:67ff:fe3d:9c99 prefixlen 64 scopeid 0x20<link>
ens2f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.1 netmask 255.255.255.0 broadcast 10.10.10.255
inet6 fe80::21e:67ff:fe3d:9c9a prefixlen 64 scopeid 0x20<link>
For node2:
$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.2 netmask 255.255.255.0 broadcast 10.10.10.255
inet6 fe80::21e:67ff:fe4d:bd21 prefixlen 64 scopeid 0x20<link>
Note that the subnet devices on the two nodes have different names. I was unable to find any value of btl_tcp_if_include that would work - I keep hitting errors like the following:
$ mpirun -npernode 1 -mca btl_tcp_if_include eth0,ens2f1 ./ring
--------------------------------------------------------------------------
WARNING: An invalid value was given for btl_tcp_if_include. This
value will be ignored.
Local host: rhc001
Value: eth0
Message: Unknown interface name
--------------------------------------------------------------------------
Rank 0 has cleared MPI_Init
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[34022,1],0]) is on host: rhc001
Process 2 ([[34022,1],1]) is on host: unknown!
BTLs attempted: self
Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
I used to be able to run on my cluster without any BTL parameters, so something definitely has broken, leaving me unable to execute any multi-node MPI job.
Advice appreciated.
rhc54
All 11 comments
mpirun --mca btl_tcp_if_include 10.10.10.0/24
should do the trick for the time being.
can you please post the output of
netstat -nr
on both your nodes ?
ggouaillardet
on 26 Mar 2018
Thanks - that worked! Here is what you requested:
$ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 ens2f0
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 ens2f1
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens2f0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
$ ssh rhc002 netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
I checked the git log and could not spot any recent changes in btl/tcp.
Though that should not cause the hang, I noted there is no default route on rhc002, was this network configuration changed recently ?
ggouaillardet
on 26 Mar 2018
Not that I'm aware of. Note that I don't regularly test MPI jobs any more, so I don't know when it was broken.
rhc54
on 26 Mar 2018
My understanding is that rhc001 binds on 192.168.0.161 and (try to) connect to rhc002 on 10.10.10.2, but the reply never returns because there is no route to 192.168.0.161 from rhc002.
So unless this is a bug in the reachable component (e.g. there is an interface in the same subnet, so no other interface that requires routing should be tried), I am afraid this cannot work out of the box.
So you either need to add a default route on rhc002, restrict to the 10.10.10.0/24 network as you already did, and i guess --mca btl_tcp_if_exclude ens2f0 would work too.
ggouaillardet
on 26 Mar 2018
I don't see how to fix the underlying issue, but we are not supposed to abort as long as other connections either exists or at least are possible.
bosilca
on 26 Mar 2018
I'll go ahead and close this as setting a default param solves the problem. However, please note that OMPI worked fine for me out-of-the-box until now, and that prior releases still do. Also, master worked fine, then was broken, then worked again, and now is broken again. So it isn't that my cluster configuration simply cannot be supported - it is something that is being impacted by changes made to the connection code.
rhc54
on 26 Mar 2018
I'm going to reopen this issue and assign it to me. Chances are that it's a change my team made, and we should drive that to ground.
bwbarrett
on 26 Mar 2018
I run into this very same issue when running ring example on a 4 node cluster:
mpirun -np 4 -H node01,node02,node03,node04 /var/datasets/ompi_examples/examples/ring_c
and I got the following error
WARNING: Open MPI failed to TCP connect to a peer MPI process. This
should not happen.
Your Open MPI job may now hang or fail.
...
I also noticed if my nodes{01,02,04} had a configuration like below:
$ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:3a:1c:b0:de txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.23.167.122 netmask 255.255.255.0 broadcast 10.23.167.255
ether 00:1e:67:c9:bb:fd txqueuelen 1000 (Ethernet)
RX packets 8572169 bytes 6247097415 (5.8 GiB)
RX errors 0 dropped 9709 overruns 0 frame 0
TX packets 53451293 bytes 77941046440 (72.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 00:1e:67:c9:bb:fe txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 100300 bytes 136571135 (130.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 100300 bytes 136571135 (130.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
while on node03 network config looked different:
$ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:9b:83:69:2a txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.23.167.124 netmask 255.255.255.0 broadcast 10.23.167.255
ether 00:1e:67:cb:a5:d8 txqueuelen 1000 (Ethernet)
RX packets 45316938 bytes 57581973047 (53.6 GiB)
RX errors 0 dropped 86337 overruns 0 frame 0
TX packets 3919488 bytes 485910079 (463.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 00:1e:67:cb:a5:d9 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 21531 bytes 39461416 (37.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21531 bytes 39461416 (37.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
For me the easiest way to to resolve the issue was just to rename the interface enp3s0f0 (although I renamed enp3s0f1 as well).
So from node03 I run the commands:
ifconfig enp3s0f0 down && ip link set enp3s0f0 name eth0 && ifconfig eth0 up
ifconfig enp3s0f1 down && ip link set enp3s0f1 name eth1 && ifconfig eth1 up
and then I checked ifconfig on node03 again:
$ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:9b:83:69:2a txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.23.167.124 netmask 255.255.255.0 broadcast 10.23.167.255
ether 00:1e:67:cb:a5:d8 txqueuelen 1000 (Ethernet)
RX packets 45322870 bytes 57588792751 (53.6 GiB)
RX errors 0 dropped 86347 overruns 0 frame 0
TX packets 3920299 bytes 485985862 (463.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 00:1e:67:cb:a5:d9 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 21533 bytes 39461744 (37.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21533 bytes 39461744 (37.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Finally I run the test again and it passed successfully:
$ mpirun -npernode 72 -mca btl_tcp_if_include eth0 -H node01:72,node02:72,node03:72,node04:72 /var/datasets/ompi_examples/examples/ring_c
Process 0 sending 10 to 1, tag 201 (288 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 1 exiting
Process 2 exiting
Process 3 exiting
Process 4 exiting
...
Process 283 exiting
Process 284 exiting
Process 285 exiting
Process 286 exiting
Process 287 exiting
ashahba
on 13 Aug 2018
There's two changes you made between the working and not working cases. First, you changed the name of the ethernet device (which probably did not fix the issue). Second, you changed the list of valid interfaces to use for communication to include only eth0. My guess is that you could change btl_tcp_if_include to btl_tcp_if_exclude with the arguments docker0,lo0 and get the same result. Your network topology has two hosts with local IP ranges, that likely aren't reachable between each other (particularly since they have the same IP address). So Open MPI tries to use the docker0 ethernet device, and packets don't get to the right place, and the abort happens.
bwbarrett
on 13 Aug 2018
Any updates on this? I see errors similar to this with v3.1.3 where 1.10.7 doesn't exhibit this behavior:
[r7i2n4][[59828,0],541][connect/btl_openib_connect_udcm.c:1531:udcm_find_endpoint] could not find endpoint with port: 1, lid: 1416, msg_type: 100
[r7i2n4][[59828,0],541][connect/btl_openib_connect_udcm.c:2036:udcm_process_messages] could not find associated endpoint.
[r7i6n16][[59828,0],2597][connect/btl_openib_connect_udcm.c:1531:udcm_find_endpoint] could not find endpoint with port: 1, lid: 2251, msg_type: 100
[r7i6n16][[59828,0],2597][connect/btl_openib_connect_udcm.c:2036:udcm_process_messages] could not find associated endpoint.
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[59828,0],2597]) is on host: r7i6n16
Process 2 ([[59828,0],2629]) is on host: unknown!
BTLs attempted: vader openib self smcuda tcp
Your MPI job is now going to abort; sorry.
Since we're using Slurm, I guess https://github.com/open-mpi/ompi/issues/5478 seems more directly related to this error I am seeing.
jrood-nrel
on 2 Jan 2019
Related issues
AboorvaDevarajan
路
5Comments
ashwinvis
路
5Comments
ax3l
路
7Comments
jsquyres
路
4Comments
egorgam
路
5Comments
Most helpful comment
There's two changes you made between the working and not working cases. First, you changed the name of the ethernet device (which probably did not fix the issue). Second, you changed the list of valid interfaces to use for communication to include only
eth0. My guess is that you could changebtl_tcp_if_includetobtl_tcp_if_excludewith the argumentsdocker0,lo0and get the same result. Your network topology has two hosts with local IP ranges, that likely aren't reachable between each other (particularly since they have the same IP address). So Open MPI tries to use thedocker0ethernet device, and packets don't get to the right place, and the abort happens.