Singularity: SSH into singularity container

[sulantha2006]

Version of Singularity:

3.0.3

Expected behavior

User should be able to ssh into a running container with portmaped 22 port.

Actual behavior

Was able to ssh from localhost, but not from another host.

Steps to reproduce behavior

The container was created with --sandbox and run with --writable. openssh-server was running in the container.
localhost was able to connect to the container when using the bridge network, but any other host was able to.

sudo singularity shell --writable --net --network-args "portmap=2226:22/tcp" --network-args "portmap=2226:22/udp" <sandbox_path>

Any help/steps is appreciated.

[cclerget]

Hi @sulantha2006, you can reach ssh from other hosts through the host IP (ssh -p2226 hostIP), container are accessible only from host where container is running.

[sulantha2006]

Hi, @cclerget, thanks but this doesn't work. I tried with ssh -v -p2226 hostIP, it hangs at connecting to hostIP port 2226.

[cclerget]

@sulantha2006 I think you have firewall rules on your host, I just tested :

host1 > sudo singularity exec --net --network-args="portmap=2222:22/tcp" docker://alpine nc -lp 22
host2 > ssh -p 2222 root@host1

and it works fine, I see client connection with banner SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 from nc process

[sulantha2006]

Hi, Seems it may be, but I cannot figure out, what causes it. Here are the ufw logs and status.

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To Action From
-- ------ ----
22/tcp ALLOW IN Anywhere
Anywhere ALLOW IN 192.168.3.39
22/tcp (v6) ALLOW IN Anywhere (v6)

Jan 27 13:08:53 newton kernel: [167645.258189] [UFW BLOCK] IN=eno1 OUT=sbr0 MAC=ac:1f:6b:26:50:a2:a8:1e:84:f2:9c:bb:08:00 SRC=192.168.3.39 DST=10.22.0.23 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=36832 DF PROTO=TCP SPT=35822 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

[cclerget]

Sorry I can't really help you on that, I'm not familiar with UFW, but at first look, seems like iptables rules set by CNI network plugin are blocked by prior rules

[QwertyJack]

Tested under 3.2.0

1. First build a container with openssh-server or something equivalent.
2. Run the container: sudo singularity shell --writable --net --network-args "portmap=2226:22/tcp" <sandbox_path>
   SSH uses tcp so that there is nothing to do with udp.
3. Make sure sshd is running inside the container.
   You may try whether port 22 is open using lsof -i:22 or telnet localhost 22 in the singularity shell.
4. Config the hosts' firewall. At least:
5. Chain FORWARD is enabled for the sbr device; you can enable it by sudo iptables -I FOWARD -j ACCEPT, which is DROP by default
6. Chain INPUT should allow traffic through port 2226
   For test purpose I would recommend to disable ufw for a while.
7. Container ssh is available via .

Besides, keeps the container shell open, or you also lose the ssh server.

[Flamefire]

As I had a similar use case: You might want to start with --bind /etc/passwd,/etc/shadow to have the hosts authentication data.

[biocyberman]

@sulantha2006 I think you have firewall rules on your host, I just tested :

host1 > sudo singularity exec --net --network-args="portmap=2222:22/tcp" docker://alpine nc -lp 22
host2 > ssh -p 2222 root@host1

and it works fine, I see client connection with banner SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.1 from nc process

Haven't tried it, but from the look, I think it would requires that host1 enables root login for this to work.

[mem]

I believe this is working as expected.

[agoose77]

Just adding a comment here for others who have trouble:

Here's a definition file for an Ubuntu image which supports SSH into the container (container.def):

Bootstrap: docker
From: ubuntu:18.04
Stage: final

%post -c /bin/bash
set -e

export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y --no-install-recommends openssh-server openssl gdb rsync

echo 'root:root' | chpasswd

mkdir /run/sshd
chmod 755 /run/sshd

sed -ri 's/^#?PermitRootLogin\s+.*/PermitRootLogin yes/' /etc/ssh/sshd_config
sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config

%startscript
/usr/sbin/sshd

I haven't run and tested this image (extracted this from my working definition file), but these are all the steps required.

I then launch the container with

alias sudop='sudo -E env PATH=$PATH' # For me to be able to find singularity

sudop singularity instance start  --writable-tmpfs --net --network-args="portmap=3000:22/tcp" container.sif ssh-runtime

And connect with

ssh root@localhost -p 3000