Singularity: Instance start/stop behaviour

[gdolle]

I've a question/suggestion concerning singularity instances. I found the current behaviour a bit misleading (at least for me).
I thought instances was working like docker or systemd services which could be start/stop anytime. But it seems (to my understanding) that it's more like a container process isolation for the current session (NB. after a reboot, instances are still listed but inner processes are stopped).

To manage containers and instances, I would have expect a behaviour closer to the following

# store container path
singularity instance new container.sif webservice

# store command, pid for the instance (stopped by default).
singularity instance add webservice s1
singularity instance add webservice s2 nginx
# Duplicate instances
singularity instance cp s2 s3

# Start the instance (stopped after shutdown)
singularity instance start s1
singularity instance start s2 s3
singularity instance stop s1
singularity instance stop s2 s3

# enable/disable boot flag (start instance at boot)
singularity instance enable s1 
singularity instance disable s1 

# Remove instances
singularity instance remove s1
singularity instance remove s2

# Delete services.
singularity service delete webservice

# List services and instances (check pids?)
singularity instance list
singularity instance list --stopped
singularity instance list --active

# Display the command (and/or startscript)
singularity instance show s2

Eventually an init script (for systemd, openrc, ..) could be provided to run instances during boot ?

[cclerget]

Hi @gdolle , you're right, instances are not persistent across reboot but you can already achieve what you describe, singularity instance start can take a --pid-file argument allowing you to manage instances as part of service scripts. I think all the above command could be implemented as part of a third party wrapper managing singularity instances like you described above. This is a really nice idea and it would be cool to have it as a third party project because I really don't think it would be good to integrate it in the core of singularity as we need to stay general (and minimal) to rightly have those kind of project on top of singularity.

This is just my opinion maybe other team members have different point of views if that would be beneficial.

[gdolle]

Ok! Thanks for the answer.

I really don't think it would be good to integrate it in the core of singularity as we need to stay general (and minimal) to rightly have those kind of project on top of singularity.

I expected this was why it was not implemented yet :)

it would be cool to have it as a third party project

:+1: Maybe a service plugin would be the way then, I'll think about it.

[al3x609]

@cclerget the flag --pid-file is not reflected in the help menu.

[user@machine ~]$ singularity instance start --help
Start a named instance of the given container image

Usage:
  singularity instance start [start options...] <container path> <instance name> [startscript args...]

Description:
  The instance start command allows you to create a new named instance from an
  existing container image that will begin running in the background. If a
  startscript is defined in the container metadata the commands in that script
  will be executed with the instance start command as well. You can optionally
  pass arguments to startscript

  singularity instance start accepts the following container formats

  *.sif               Singularity Image Format (SIF). Native to Singularity 3.0+

  *.sqsh              SquashFS format.  Native to Singularity 2.4+

  *.img               ext3 format. Native to Singularity versions < 2.4.

  directory/          sandbox format. Directory containing a valid root file
                      system and optionally Singularity meta-data.

  instance://*        A local running instance of a container. (See the instance
                      command group.)

  library://*         A container hosted on a Library (default
                      https://cloud.sylabs.io/library)

  docker://*          A container hosted on Docker Hub

  shub://*            A container hosted on Singularity Hub

Options:
      --add-caps string        a comma separated capability list to add
      --allow-setuid           allow setuid binaries in container (root only)
      --apply-cgroups string   apply cgroups from file for container
                               processes (root only)
  -B, --bind strings           a user-bind path specification.  spec has
                               the format src[:dest[:opts]], where src and
                               dest are outside and inside paths.  If dest
                               is not given, it is set equal to src.
                               Mount options ('opts') may be specified as
                               'ro' (read-only) or 'rw' (read/write, which
                               is the default). Multiple bind paths can be
                               given by a comma separated list.
      --boot                   execute /sbin/init to boot container (root only)
  -e, --cleanenv               clean environment before running container
  -c, --contain                use minimal /dev and empty other
                               directories (e.g. /tmp and $HOME) instead
                               of sharing filesystems from your host
  -C, --containall             contain not only file systems, but also
                               PID, IPC, and environment
      --dns string             list of DNS server separated by commas to
                               add in resolv.conf
      --docker-login           login to a Docker Repository interactively
      --drop-caps string       a comma separated capability list to drop
  -h, --help                   help for start
  -H, --home string            a home directory specification.  spec can
                               either be a src path or src:dest pair.  src
                               is the source path of the home directory
                               outside the container and dest overrides
                               the home directory within the container.
                               (default "/home/cbernal")
      --hostname string        set container hostname
      --keep-privs             let root user keep privileges in container
                               (root only)
  -n, --net                    run container in a new network namespace
                               (sets up a bridge network interface by default)
      --network string         specify desired network type separated by
                               commas, each network will bring up a
                               dedicated interface inside container
                               (default "bridge")
      --network-args strings   specify network arguments to pass to CNI plugins
      --no-home                do NOT mount users home directory if home
                               is not the current working directory
      --no-privs               drop all privileges from root user in container
      --nv                     enable experimental Nvidia support
  -o, --overlay strings        use an overlayFS image for persistent data
                               storage or as read-only layer of container
  -S, --scratch strings        include a scratch directory within the
                               container that is linked to a temporary dir
                               (use -W to force location)
      --security strings       enable security features (SELinux,
                               Apparmor, Seccomp)
  -u, --userns                 run container in a new user namespace,
                               allowing Singularity to run completely
                               unprivileged on recent kernels. This
                               disables some features of Singularity, for
                               example it only works with sandbox images.
      --uts                    run container in a new UTS namespace
  -W, --workdir string         working directory to be used for /tmp,
                               /var/tmp and $HOME (if -c/--contain was
                               also used)
  -w, --writable               by default all Singularity containers are
                               available as read only. This option makes
                               the file system accessible as read/write.
      --writable-tmpfs         makes the file system accessible as
                               read-write with non persistent data (with
                               overlay support only)


Examples:
  $ singularity instance start /tmp/my-sql.sif mysql

  $ singularity shell instance://mysql
  Singularity my-sql.sif> pwd
  /home/mibauer/mysql
  Singularity my-sql.sif> ps
  PID TTY          TIME CMD
    1 pts/0    00:00:00 sinit
    2 pts/0    00:00:00 bash
    3 pts/0    00:00:00 ps
  Singularity my-sql.sif>

  $ singularity instance stop /tmp/my-sql.sif mysql
  Stopping /tmp/my-sql.sif mysql


For additional help or support, please visit https://www.sylabs.io/docs/

singularity version:

singularity version 3.2.1-7.1.ohpc.1.3.8

[dtrudg]

@al3x609 - apologies, this was added after 3.2.1 - looks like it's only in the upcoming 3.5.0 release (tomorrow).

  -p, --pid                    run container in a new PID namespace
      --pid-file string        write instance PID to the file with the
                               given name

[dtrudg]

Am going to close this now, as it hasn't attracted further discussion. I concur with @cclerget that persistent instances would be better managed outside of singularity through systemd unit files, supervisord etc. as this avoids us implementing a solution which won't work for people who have different requirements for process supervision.