Yarp: Problems with (new) yarprun on master

[EliseiFrederic]

Dear All,

We just upgraded to the last YARP master:
YARP version 3.2.0+174-20190911.2+gitcc6dfda
Now, trying to execute remote commands with yarpmanager/yarprun get stucks with no answer, and temporary ports like these:
yarp: Sending output from /tmp/port/1 to /desktop-302 using tcp
(no return)

Previously, we were using a version with icub-cluster.py icub-cluster-run.sh and icub-cluster-server.sh
I read the annoucement about the "yarpmanager++" which includes this cluster management facility :
https://github.com/robotology/yarp/issues/730

After moving cluster-config.xml ymanager.ini in /.local/share/yarp/contexts/yarpmanager
I can use the cluster facility from yarpmanager, to start/stop yarprun on remote computers. _I can use the "exec" subwindow to execute various commands (like ls, seeing the result in the console). EDIT: this is using SSH, not yarprun, so no point mentionning it here..._
But anything from Apps get stucked.

I have the same problem when using yarprun manually. Even from a fresh yarp installation (with no iCub environment)

./yarpserver --write
__ __ ___ ____ ____
\ \/ // || _ \ | _ \
\ // /| || |/ / | |/ /
/ // ___ || _ \ | _/
/_//_/ |_||_| _|_|
========================

Call with --help for information on available options
Using port database: :memory:
Using subscription database: :memory:
IP address: default
Port number: 10000
yarp: Port /root active at tcp://192.168.70.41:10000/

Registering name server with itself:

• register "/root" tcp "192.168.70.41" 10000
  
  
  
  • set "/root" offers http name_ser local tcp fast_tcp mcast udp text text_ack
  
  
  • set "/root" accepts http name_ser local tcp fast_tcp mcast udp text text_ack
  
  
  • set "/root" ips "127.0.0.1" "192.168.70.41" "::1" "fe80::911f:5956:5ba4:a42f%2"
  
  
  • set "/root" process 23496
  
  
• register fallback mcast "224.2.1.1" 10000
  
  
  
  • set fallback offers http name_ser local tcp fast_tcp mcast udp text text_ack
  
  
  • set fallback accepts http name_ser local tcp fast_tcp mcast udp text text_ack
  
  
  • set fallback ips "127.0.0.1" "192.168.70.41" "::1" "fe80::911f:5956:5ba4:a42f%2"
  
  
  • set fallback process 23496
  
  
• set "/root" nameserver "true"
  Name server can be browsed at http://192.168.70.41:10000/

Ok. Ready!

(now running in background)

./yarprun --server /desktop-302

• register "/desktop-302"
  
  
  
  • set "/desktop-302" offers http name_ser local tcp fast_tcp mcast udp text text_ack
  
  
  • set "/desktop-302" accepts http name_ser local tcp fast_tcp mcast udp text text_ack
  
  
  • set "/desktop-302" ips "127.0.0.1" "192.168.70.41" "::1" "fe80::911f:5956:5ba4:a42f%2"
  
  
  • set "/desktop-302" process 23502
    
    -> announce "/desktop-302"
  
  
  • set "/desktop-302" yarprun true
    
    [INFO]Yarprun successfully started on port: /desktop-302
  
  

(now running in background)

./yarprun --on /desktop-302 --as test --cmd ls

• register "..."
• set "/tmp/port/1" offers http name_ser local tcp fast_tcp mcast udp text text_ack
• set "/tmp/port/1" accepts http name_ser local tcp fast_tcp mcast udp text text_ack
• set "/tmp/port/1" ips "127.0.0.1" "192.168.70.41" "::1" "fe80::911f:5956:5ba4:a42f%2"
• set "/tmp/port/1" process 23514
  -> announce "/tmp/port/1"
  yarp: Port /tmp/port/1 active at tcp://192.168.70.41:10003/
• query "/tmp/port/1"
• query "/desktop-302"
• query "/desktop-302"
  yarp: Sending output from /tmp/port/1 to /desktop-302 using tcp
  ^C

I have the same problem from the yarpmanager interface, wether I activate 'log' or not. With a fresh install on a different computer too, whether Debian or Ubuntu.

Actually, all my problems look exactly like :
https://github.com/robotology/yarp/issues/1173
but I have an up-to-date version ! And I'm not mixing old/new yarprun/yarpmanager

I suspect some syntax or usage may have changed, rather than a bug, but I cannot find what. Or maybe it is related to stdio or a pipe not working. Or a defaut console related to the server that need to run now ?
yarprun not using the same name resolving depending of the task/options ?
Any idea / reminiscence from the time you updated to yarprun with --log of the yarpmanager++ ?

Thanks in advance for your help,
Frédéric

[pattacini]

@EliseiFrederic thanks for reporting.
Just transferred the issue here as it is prominently concerned with yarp.

[EliseiFrederic]

In case it helps, here are new details from the yarpmanager. The cluster widget (or the old python cluster manager) can run/stop the remote yarprun, check succesfully if they are alive or not, but errors appear when:

1/ I try to launch app or modules:

[ERR] (pattern_tracking) cannot run vrmonitor on icubsrv : cannot ask /icubsrv to run vrmonitor (Remote host does not respond)

2/ opening an app or refreshing the status, with the yarprun service is running and the resource appears green (available) in both widgets of yarpmanager reports error on every line after some short timeout:

[ERR] (pattern_tracking) cannot initialize broker. : icubsrv does not exist. check yarprun is running as server.
[ERR] (pattern_tracking) icubsrv does not exist. check yarprun is running as server.

When using localhost, the module is launched or stopped as expected

[EliseiFrederic]

In case it helps, some output from the faulty yarprun:

YARP version 3.2.0+174-20190911.2+gitcc6dfda

$ YARP_VERBOSE=1 yarprun --on /icubsrv --as test --cmd ls
yarp: YARP_VERBOSE environment variable is set
yarp(1ac3): register contact ...
yarp(1ac3): Scanning. I'm scanning. I hope you like scanning too.
yarp(1ac3): Loading configuration files related to plugins from /usr/local/src/robot/yarp/build/share/yarp/plugins.
yarp(1ac3): Loading configuration files related to plugins from /usr/local/src/robot/icub-main/build/share/iCub/plugins.
yarp(1ac3): Plugin [name: yarp_bayer] [dll: yarp_bayer] [fn: bayer]
yarp(1ac3): Trying plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_bayer.so] [fn: bayer]
yarp(1ac3): Found plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_bayer.so] [fn: bayer]
yarp(1ac3): Plugin [name: yarp_portmonitor] [dll: yarp_portmonitor] [fn: portmonitor]
yarp(1ac3): Trying plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_portmonitor.so] [fn: portmonitor]
yarp(1ac3): Found plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_portmonitor.so] [fn: portmonitor]
yarp(1ac3): Plugin [name: yarp_priority] [dll: yarp_priority] [fn: priority]
yarp(1ac3): Trying plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_priority.so] [fn: priority]
yarp(1ac3): Found plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_priority.so] [fn: priority]
yarp(1ac3): Plugin [name: yarp_tcpros] [dll: yarp_tcpros] [fn: rossrv]
yarp(1ac3): Trying plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_tcpros.so] [fn: rossrv]
yarp(1ac3): Found plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_tcpros.so] [fn: rossrv]
yarp(1ac3): Plugin [name: yarp_shmem] [dll: yarp_shmem] [fn: shmem]
yarp(1ac3): Trying plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_shmem.so] [fn: shmem]
yarp(1ac3): Found plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_shmem.so] [fn: shmem]
yarp(1ac3): Plugin [name: yarp_tcpros] [dll: yarp_tcpros] [fn: tcpros]
yarp(1ac3): Trying plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_tcpros.so] [fn: tcpros]
yarp(1ac3): Found plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_tcpros.so] [fn: tcpros]
yarp(1ac3): Plugin [name: yarp_xmlrpc] [dll: yarp_xmlrpc] [fn: xmlrpc]
yarp(1ac3): Trying plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_xmlrpc.so] [fn: xmlrpc]
yarp(1ac3): Found plugin [dll: /usr/local/src/robot/yarp/build/lib/yarp/yarp_xmlrpc.so] [fn: xmlrpc]
yarp(1ac3): <<< received from nameserver: [ok]

yarp: Port /tmp/port/15 active at tcp://192.168.70.52:10041/
yarp(1ac3): working on connection /tmp/port/15 to /icubsrv (connect)
yarp(1ac3): query name /tmp/port/15
yarp(1ac3): <<< received from nameserver: registration name /tmp/port/15 ip 192.168.70.52 port 10041 type tcp
* end of message

yarp(1ac3): query name /icubsrv
yarp(1ac3): <<< received from nameserver: registration name /icubsrv ip 192.168.70.52 port 10003 type tcp
* end of message

yarp(1ac3): asking tcp://tmp/port/15: [list] [out] "/icubsrv"
yarp(1acf): TextCarrier::expectSenderSpecifier
yarp(1acf): /tmp/port/15: Port /tmp/port/15 received command [list] [out] "/icubsrv"
yarp(1ac3): * asking tcp://tmp/port/15: [add] "tcp://icubsrv"
yarp(1ad0): TextCarrier::expectSenderSpecifier
yarp(1ad0): /tmp/port/15: Port /tmp/port/15 received command [add] "tcp://icubsrv"
yarp(1ad0): query name /icubsrv
yarp(1ad0): <<< received from nameserver: registration name /icubsrv ip 192.168.70.52 port 10003 type tcp
** end of message

yarp: Sending output from /tmp/port/15 to /icubsrv using tcp
^C

[EliseiFrederic]

As a reference, what we from a computer with the older version of yarp/yarprun:
In bold at the end, what we do not receive on the newer version
(P.S. --verbose is no more accepted by the newer yarprun)

YARP version 2.3.72

$YARP_VERBOSE=1 yarprun --on /icub-001 --as test --cmd ls
yarp: YARP_VERBOSE environment variable is set
yarp(9e551740): register contact ...
yarp(9e551740): Scanning. I'm scanning. I hope you like scanning too.
yarp(9e551740): Loading configuration files related to plugins from /usr/local/src/robot/yarp/build-icub_cluster/share/yarp/plugins.
yarp(9e551740): Loading configuration files related to plugins from /usr/local/src/robot/icub-main/build-icub_cluster/share/iCub/plugins.
yarp(9e551740): Loading configuration files related to plugins from /usr/local/src/robot/iCubContrib/share/ICUBcontrib/plugins.
yarp(9e551740): Plugin [name: yarp_bayer] [dll: yarp_bayer] [fn: bayer]
yarp(9e551740): Trying plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_bayer.so] [fn: bayer]
yarp(9e551740): Found plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_bayer.so] [fn: bayer]
yarp(9e551740): Plugin [name: yarp_portmonitor] [dll: yarp_portmonitor] [fn: portmonitor]
yarp(9e551740): Trying plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_portmonitor.so] [fn: portmonitor]
yarp(9e551740): Found plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_portmonitor.so] [fn: portmonitor]
yarp(9e551740): Plugin [name: yarp_priority] [dll: yarp_priority] [fn: priority]
yarp(9e551740): Trying plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_priority.so] [fn: priority]
yarp(9e551740): Found plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_priority.so] [fn: priority]
yarp(9e551740): Plugin [name: yarp_tcpros] [dll: yarp_tcpros] [fn: rossrv]
yarp(9e551740): Trying plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_tcpros.so] [fn: rossrv]
yarp(9e551740): Found plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_tcpros.so] [fn: rossrv]
yarp(9e551740): Plugin [name: yarp_tcpros] [dll: yarp_tcpros] [fn: tcpros]
yarp(9e551740): Trying plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_tcpros.so] [fn: tcpros]
yarp(9e551740): Found plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_tcpros.so] [fn: tcpros]
yarp(9e551740): Plugin [name: yarp_xmlrpc] [dll: yarp_xmlrpc] [fn: xmlrpc]
yarp(9e551740): Trying plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_xmlrpc.so] [fn: xmlrpc]
yarp(9e551740): Found plugin [dll: /usr/local/src/robot/yarp/build-icub_cluster/lib/yarp/yarp_xmlrpc.so] [fn: xmlrpc]
yarp(9e551740): <<< received from nameserver: [ok]

yarp: Port /tmp/port/1 active at tcp://192.168.70.36:10003/
yarp(9e551740): working on connection /tmp/port/1 to /icub-001 (connect)
yarp(9e551740): query name /tmp/port/1
yarp(9e551740): <<< received from nameserver: registration name /tmp/port/1 ip 192.168.70.36 port 10003 type tcp
* end of message

yarp(9e551740): query name /icub-001
yarp(9e551740): <<< received from nameserver: registration name /icub-001 ip 192.168.70.36 port 10002 type tcp
* end of message

yarp(9e551740): asking tcp://tmp/port/1: [list] [out] "/icub-001"
yarp(99708700): TextCarrier::expectSenderSpecifier
yarp(99708700): /tmp/port/1: Port /tmp/port/1 received command [list] [out] "/icub-001"
yarp(9e551740): * asking tcp://tmp/port/1: [add] "tcp://icub-001"
yarp(98f07700): TextCarrier::expectSenderSpecifier
yarp(98f07700): /tmp/port/1: Port /tmp/port/1 received command [add] "tcp://icub-001"
yarp(98f07700): query name /icub-001
yarp(98f07700): <<< received from nameserver: registration name /icub-001 ip 192.168.70.36 port 10002 type tcp
** end of message

yarp: Sending output from /tmp/port/1 to /icub-001 using tcp

yarp(9e551740): working on connection /tmp/port/1 to /icub-001 (disconnect)
yarp(9e551740): query name /tmp/port/1
yarp(9e551740): <<< received from nameserver: registration name /tmp/port/1 ip 192.168.70.36 port 10003 type tcp
* end of message

yarp(9e551740): query name /icub-001
yarp(9e551740): <<< received from nameserver: registration name /icub-001 ip 192.168.70.36 port 10002 type tcp
* end of message

yarp(9e551740): asking tcp://tmp/port/1: [list] [out] "/icub-001"
yarp(99708700): TextCarrier::expectSenderSpecifier
yarp(99708700): /tmp/port/1: Port /tmp/port/1 received command [list] [out] "/icub-001"
yarp(9e551740): * asking tcp://tmp/port/1: [del] "/icub-001"
yarp(98f07700): TextCarrier::expectSenderSpecifier
yarp(98f07700): /tmp/port/1: Port /tmp/port/1 received command [del] "/icub-001"
yarp: Removing output from /tmp/port/1 to /icub-001
yarp(9e551740): <<< received from nameserver: ** end of message

RESPONSE:

4508
STARTED: server=/icub-001 alias=test cmd=ls pid=4508

[EliseiFrederic]

Looks like 'yarp exists' behave correctly with all yarp ports,
but freeze with the ones created by yarprun

[pattacini]

Hi @EliseiFrederic

In the effort to narrow down the problem, I've just made a quick test with yarprun solely - e.g. no yarpmanager - with good results.

On my Windows system with the latest yarp/devel (e.g. 3.2.100+20190923.10+gitfe483e2eb), I did the following:

On console 1

$ yarpserver &
$ yarprun --server /server &

On console 2

$ yarprun --on /server --as test --cmd velocityObserver
$ yarprun --on /server --sigterm test

As a result, velocityObserver started off normally and got shut down gracefully.
Could you try out the same and report back what you'll get?

[EliseiFrederic]

Hi @pattacini ,

Thanks for trying to help. On our Debian (Jessie), I got the same issue as reported previously: the "yarprun --on" commands stay freezed, without executing the intented action. While the yarprun server stays alive.

Console 2:

yarprun --on /server --as test --cmd velocityObserver
yarp: Port /tmp/port/1 active at tcp://192.168.70.52:10003/
yarp: Sending output from /tmp/port/1 to /server using tcp

Console 3:

$ yarprun --on /server --sigterm test
yarp: Port /tmp/port/2 active at tcp://192.168.70.52:10004/
yarp: Sending output from /tmp/port/2 to /server using tcp

Commands are still stucked, while the server(s) run, as visible here:
```$ ps -edf | grep yarprun
icub 3127 3121 0 16:59 pts/0 00:00:00 yarprun --server /server
icub 3129 3127 0 16:59 pts/0 00:00:00 yarprun --server /server
icub 3142 2346 0 17:00 pts/2 00:00:00 yarprun --on /server --as test --cmd velocityObserver
icub 3160 2329 0 17:00 pts/1 00:00:00 yarprun --on /server --sigterm test

If that helps, I might add than performing `yarp rpc /server`, with the intent to send `(cmd velocityObserver) (as test)` locks in the same way, with no command sent, after a connexion negociation (receiving the port adresses from the name server)

**Console 1:**

yarp server &
[1] 3124
__ __ ___ ____ ____
\ \/ // || _ \ | _ \
\ // /| || |/ / | |/ /
/ // ___ || _ \ | _/
/_//_/ |_||_| _|_|
========================

Call with --help for information on available options
Using port database: :memory:
Using subscription database: :memory:
IP address: default
Port number: 10000
yarp: Port /root active at tcp://192.168.70.52:10000/

Registering name server with itself:

• register "/root" tcp "192.168.70.52" 10000
  
  
  
  • set "/root" offers http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
  
  
  • set "/root" accepts http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
  
  
  • set "/root" ips "127.0.0.1" "192.168.70.52" "::1" "fe80::eef4:bbff:fe17:bf9c%2"
  
  
  • set "/root" process 3124
  
  
• register fallback mcast "224.2.1.1" 10000
  
  
  
  • set fallback offers http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
  
  
  • set fallback accepts http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
  
  
  • set fallback ips "127.0.0.1" "192.168.70.52" "::1" "fe80::eef4:bbff:fe17:bf9c%2"
  
  
  • set fallback process 3124
  
  
• set "/root" nameserver "true"
  Name server can be browsed at http://192.168.70.52:10000/

Ok. Ready!
yarprun --server /server

• register "/server"
  
  
  
  • set "/server" offers http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
  
  
  • set "/server" accepts http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
  
  
  • set "/server" ips "127.0.0.1" "192.168.70.52" "::1" "fe80::eef4:bbff:fe17:bf9c%2"
  
  
  • set "/server" process 3127
    
    -> announce "/server"
  
  
  • set "/server" yarprun true
    
    [INFO]Yarprun successfully started on port: /server
  
  
• list
• register "..."
• set "/tmp/port/1" offers http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
• set "/tmp/port/1" accepts http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
• set "/tmp/port/1" ips "127.0.0.1" "192.168.70.52" "::1" "fe80::eef4:bbff:fe17:bf9c%2"
• set "/tmp/port/1" process 3142
  -> announce "/tmp/port/1"
• query "/tmp/port/1"
• query "/server"
• query "/server"
• register "..."
• set "/tmp/port/2" offers http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
• set "/tmp/port/2" accepts http name_ser local tcp fast_tcp mcast udp text text_ack bayer portmonitor priority rossrv shmem tcpros xmlrpc
• set "/tmp/port/2" ips "127.0.0.1" "192.168.70.52" "::1" "fe80::eef4:bbff:fe17:bf9c%2"
• set "/tmp/port/2" process 3160
  -> announce "/tmp/port/2"
• query "/tmp/port/2"
• query "/server"
• query "/server"

```
I tried sending/reading commands on homemade software using rpc ports, and it works ok.

I also have a fresh Bionic Ubuntu, with just yarp, that works as you describe (replacing velocityObserver by another graphical tool, as it is an iCub tool). But this computer is not the laptop that connects to the robot !

I also tried recompiling on our Debian with the 'skip ACE' option, but I did not notice any improvement.

Thanks for your examples,
Frédéric

[EliseiFrederic]

If that helps, when I stop the yarprun --server while the yarprun --on clients are freezed, both report:
yarp: BottleImpl reader failed, unrecognized object code 89

[pattacini]

Debian/jessie is quite old actually, from 2015. Don't you want to update to Debian/stretch?
Also, the fact that it's working on Ubuntu/bionic can lead me to think that it might play a role.

@drdanz what do you think?

[drdanz]

I can confirm this behaviour, after bisecting this seems to be the responsible commit 3229150fc689ade3de74e3c27b1554e1ca7db05e (yes, that was me :sweat_smile: )

[pattacini]

Oki then. Out of sheer curiosity, why is it working on the latest distributions?

[drdanz]

Because it was broken in YARP 3.2 but I think that latest distribution is still running 3.1

[pattacini]

Because it was broken in YARP 3.2 but I think that latest distribution is still running 3.1

But on my yarp/devel it is working... 🤔

[drdanz]

@EliseiFrederic it should be fixed in latest master, and yarp-3.2 branch, and will be fixed in devel tomorrow, after the nightly merge. Thanks for reporting the very detailed reporting.

[drdanz]

But on my yarp/devel it is working...

I think that the write function on windows might have a different behaviour than on linux

[EliseiFrederic]

Thanks for this successfull hunt:
Your code modification effectively removes the buggy behaviour on both icubsrv (Debian/Jessie) and our cluster (Debian/Stretch)
Well done :+1: