mosh: Nothing received from server on UDP port 60001

[whoizit]

Fresh Arch-based linux distro install on VPS, without firewall.
On the client side there is also no firewall.
Mosh installed on server and client, same version:

$ mosh -V
mosh 1.3.2 [build mosh-1.3.2]

SSH works.
UDP port opens and works.

On server:

$ nc -lu 60001

On client:

$ nc -zvu 46.46.46.46 60001
Connection to 46.46.46.46 60001 port [udp/*] succeeded

On server:

$ ip a s ens18
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether c6:51:3a:b6:96:a3 brd ff:ff:ff:ff:ff:ff
    inet 46.46.46.46/24 scope global ens18
       valid_lft forever preferred_lft forever
    inet6 2a02:2a02:2a02:2a02:2a02:2a02:2a02:0/48 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::c451:3aff:feb6:96a3/64 scope link 
       valid_lft forever preferred_lft forever

$ mosh --ssh='ssh -p 2022' [email protected]



md5-1ec7360a6709ed3d4e323c30eef9777d



mosh: Nothing received from server on UDP port 60001
mosh did not make a successful connection to 46.46.46.46:60001.
Please verify that UDP port 60001 is not firewalled and can reach the server.

Also I tried to use it over yggdrasil overlayed network (with ipv6 ip) to exclude possible firewall on provider side. But I getting same behaviour. IDK what to do.

[andersk]

Because UDP is connectionless, nc -z needs to send data in order to test the connection, and has no way to automatically verify that the data was received. Did you see the data received on the server (this manifests as nc -l outputting XXXXX)? It would be better to test that data is flowing in both directions; see the instructions in the wiki.

[whoizit]

server:

$ nc -4ulv 60001
Listening on [0.0.0.0] (family 2, port 60001)
Connection from <host> 34775 received!
XXXXXhello world

client:

$ echo "hello world" | nc -4vu 46.46.46.46 60001
Connection to 46.46.46.46 60001 port [udp/*] succeeded!

client:

$ nc -4ulv 60001
Listening on [0.0.0.0] (family 2, port 60001)
Connection from <host> 48970 received!
XXXXXhello world2

server:

$ echo "hello world2" | nc -4vu 92.92.92.92 60001
Connection to 92.92.92.92 60001 port [udp/*] succeeded

[eminence]

Are you familiar with how to use tcpdump? While trying to run mosh, tcpdump both the client and the server to see if that gives any hints. Are the client-to-server packets being sent to the right destination? Are they being received by the server? Are there any reply packets being sent by the server?

Note that if the mosh-server process doesn't receive any packets from the client within 60 seconds of startup, it will shutdown. This is something important to keep in mind

[netchild]

I have a similar problem on FreeBSD (13-current = in-development version). Mosh is at 1.3.2. At the end of this report is an assert failure message.

Debugging:

"Mosh by hand" from client to Server:

% ssh -S none -o ProxyCommand='mosh --fake-proxy -- %h %p' -n -tt [email protected] -- 'mosh-server new'
MOSH IP 89.238.82.207
Enter passphrase for key '/home/netchild/.ssh/id_rsa':


MOSH CONNECT 60001 hJb/8YYWEzgeNf9SBq5RWw

mosh-server (mosh 1.3.2) [build mosh 1.3.2]
Copyright 2012 Keith Winstein <[email protected]>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 19346]

Warning: termios IUTF8 flag not defined.
Character-erase of multibyte character sequence
probably does not work properly on this platform.
Connection to bastille.leidinger.net closed.

NC on client:

% echo "hello world" | nc -4 -v -u bastille.leidinger.net 60001
Connection to bastille.leidinger.net 60001 port [udp/*] succeeded!

NC on destination:

% nc -4 -u -l -v 60001
Connection from p5B165082.dip0.t-ipconnect.de 44697 received!
XXXXhello world

Mosh manually (server side):

% date; mosh-server new -vvvv -i 89.238.82.207 -p 60001; date
So. 19 Mai 2019 14:11:23 CEST


MOSH CONNECT 60001 mSyotVGryhhd6cjVhk527Q

mosh-server (mosh 1.3.2) [build mosh 1.3.2]
Copyright 2012 Keith Winstein <[email protected]>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 25877]

Warning: termios IUTF8 flag not defined.
Character-erase of multibyte character sequence
probably does not work properly on this platform.
So. 19 Mai 2019 14:11:23 CEST

Mosh manually (client side):

% MOSH_KEY=mSyotVGryhhd6cjVhk527Q mosh-client -vvv 89.238.82.207 60001; date
mosh did not make a successful connection to 89.238.82.207:60001.
Please verify that UDP port 60001 is not firewalled and can reach the server.

(By default, mosh uses a UDP port between 60000 and 61000. The -p option
selects a specific UDP port number.)
[mosh is exiting.]
select: got poll (timeout 0)
                            Assertion failed: (false), function init_diff, file ./../statesync/user.h, line 92.
                                                                                                               [1]    17816 abort      MOSH_KEY=mSyotVGryhhd6cjVhk527Q mosh-client -vvv 89.238.82.207 60001
                                                                                      So. 19 Mai 2019 14:11:30 CEST

tcpump on the server ("mosh bastille.leidinger.net"):
mosh.zip

Anything else I could provide?

[gitgitgat-zz]

Same problem with Ubuntu 16.04.6 LTS, the VPS is firewalled and the rules look like:

it cannot connect and disconnects due to timeout, it was working yesterday

[gitgitgat-zz]

removed firewall and started working again for a while. It is not working with ufw firewall.

[eminence]

@netchild The tcpdumps show incoming UDP packets, but no attempts at all to send a reply. The most obvious cause of this is that the packets weren't reaching mosh-server. Was it still running? (Remember that mosh-server will exit if it doesn't receive any packets within 60 seconds).

Also, that assertion failure looks concerning. If you can reproduce it, can you please open a new github issue?

@gitgitgat Can you take the same debugging steps I describe above? It will be helpful to know if the client-to-server messages aren't reaching the server, or if the server-to-client messages aren't reaching the client

[netchild]

The client side was executed within the 60 seconds window. I remember that the server was telling after the client already died with the assert, that it will exit due to timeout. Is there a way to get some more debug output of the server and keep it in the foreground (I didn't see anything in the man-page)? Would it help to run a FreeBSD ktrace (or dtrace) to trace the syscalls mosh-server is doing?

I will open a new issue for the client assert.

[chengguangnan]

In my case, it seems that ISP filtered my packages. I can mosh to the target server from another server that is within the same data center but not from my laptop. The problem is resolved after re-dial PPPoE.

[oblitum]

removed firewall and started working again for a while. It is not working with ufw firewall.

Same here, I've opened 22/tcp and 60001/udp on ufw, but I couldn't make it work. It just timeouts. Disabling ufw make it work.

[oblitum]

Fixed my problem by rebooting machines.

[netchild]

Any ideas how to debug this further?

[shelbyKiraM]

On iOS & macOS connecting to Ubuntu server, I get the same thing. Built from GitHub source or from package managers (apt for my Ubuntu server & brew on my Mac). Using wireshark on my Mac shows packets being sent to 60001, but no reply... I added an iptables rule to allow the ports.

```$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:openvpn
ACCEPT udp -- anywhere anywhere multiport dports 60000:61000

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- 10.8.0.0/24 anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- 10.8.0.0/24 anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (4 references)
target prot opt source destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (4 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
```

IDK what to do from here. Restarting server didn't help. Afaik it was working until I made an image & changed to a new AWS availability zone (I hadn't used mosh for a while)… I can run any commands y'all think might help to troubleshoot, server- or client-side!

[whoizit]

I believe the problem was on the host side, they blocked all traffic by default. I found a place to turn off the firewall.

[netchild]

Hi,

just a note. The closing of this triggered me testing this again. I have absolutely no change in the firewal rules on the target machine which exhibited this issue (timestamp of pf.conf is not changed, system rebooted prior to testing). The only difference is that I'm running a more recent FreeBSD-current and I may have recompiled mosh with a more recent version of llvm.

This works for me now.

Bye,
Alexander.