Singularity: Singularity 3.1.0 rc Checklist

[ArangoGutierrez]

Release candidate CheckList

Heya all, we have a new RC in our hands!

This is a QA checklist for 3.1-rc1

Each check MUST be done with a comment, showcasing how the new feature was tested.
Comment must be signed by author or will not be considered valid to check mark the bullet point.

New features

• [x] Can now build CLI on darwin for limited functionality on Mac
• [x] Added the scratch bootstrap agent to build from anything
• [x] Reintroduced support for zypper bootstrap agent
• [x] Added the ability to overwrite a new singularity.conf when building from RPM if desired
• [x] Fixed several regressions and omissions in SCIF support
• [x] Added caching for containers pulled/built from the Container Library
• [x] Changed keys command group to key (retained hidden keys command for backward compatibility)
• [x] Created an RPMPREFIX variable to allow RPMs to be installed in custom locations

• [x] Greatly expanded CI unit and end-to-end testing

  New Commands

• [x] Added cache command group to inspect and manage cached files

• [ ] Introduced the OCI command group to support a new OCI compliant variant of the Singularity runtime:
  
  
  
  • [ ] attach Attach console to a running container process
  
  
  • [x] create Create a container from a bundle directory
  
  
  • [ ] delete Delete container
  
  
  • [ ] exec Execute a command within container
  
  
  • [ ] kill Kill a container
  
  
  • [x] mount Mount create an OCI bundle from SIF image
  
  
  • [ ] pause Suspends all processes inside the container
  
  
  • [ ] resume Resumes all processes previously paused inside the container
  
  
  • [ ] run Create/start/attach/delete a container from a bundle directory
  
  
  • [ ] start Start container process
  
  
  • [ ] state Query state of a container
  
  
  • [ ] umount Umount delete bundle
    
    

    - [ ] update Update container cgroups resources

  
  

Also all bugs related to 3.1-rc1 MUST link this issue, for tracking purposes .

Once all the checks had been marked, 3.1-rc will be considered for GA release.

Update [Feb 12/2019]

A rc2 has been released :

• https://github.com/sylabs/singularity/releases/tag/v3.1.0-rc2

Fixes since RC1:

• Set OCI default args to execute runscript #2685
• Revert GC fix as it breaks OCI engine #2679
• Fix terminal issue with oci commands #2684

[ArangoGutierrez]

Check Mark

• Greatly expanded CI unit and end-to-end testin
  Signed-off-by: Eduardo Arango [e-mail]

this is not a singularity feature, more a expanded CI, test infrastructure enhancement over past releases.

• #2670
• #2627
• #2574
• #2569
• #2565
• #2564
  
  
  
  • #2563
  
  
  • #2533
  
  
  • #2459
  
  

[ArangoGutierrez]

• Can now build CLI on darwin for limited functionality on Mac
  Signed-off-by: Eduardo Arango [email protected]

Build on MacOS Mojave, Version 10.14.2 (18C54)
``[eduardo:singularity]$ ./mconfig Configuring for projectsingularity' with languages: C, Golang
=> running pre-basechecks project specific checks ...
=> running base system checks ...
checking: host C compiler... cc
checking: host C++ compiler... c++
checking: host ar' path... /Library/Developer/CommandLineTools/usr/bin/ar checking: hostld' path... /Library/Developer/CommandLineTools/usr/bin/ld
checking: host ranlib' path... /Library/Developer/CommandLineTools/usr/bin/ranlib checking: hostobjcopy' path... objcopy
checking: target C compiler... cc
checking: target C++ compiler... c++
checking: target ar' path... /Library/Developer/CommandLineTools/usr/bin/ar checking: targetld' path... /Library/Developer/CommandLineTools/usr/bin/ld
checking: target ranlib' path... /Library/Developer/CommandLineTools/usr/bin/ranlib checking: targetobjcopy' path... objcopy
checking: host compiles static binaries... no
checking: target compiles static binaries... no
checking: host os type... darwin
checking: host architecture... x86_64
checking: target architecture... x86_64
checking: host architecture word size... 64
checking: target architecture word size... 64
checking: project version... 3.1.0-rc1
checking: project short version... 3.1.0
checking: golang compiler version... 1.11
=> running post-basechecks project specific checks ...
checking: namespace: CLONE_NEWPID... no
checking: namespace: CLONE_FS... no
checking: namespace: CLONE_NEWNS... no
checking: namespace: CLONE_NEWUSER... no
checking: namespace: CLONE_NEWIPC... no
checking: namespace: CLONE_NEWNET... no
checking: namespace: CLONE_NEWUTS... no
checking: namespace: CLONE_NEWCGROUP... no
checking: feature: NO_NEW_PRIVS... no
checking: feature: MS_SLAVE... no
checking: feature: MS_REC... no
checking: feature: MS_PRIVATE... no
checking: user capabilities... no
checking: header linux/securebits.h... no
checking: header linux/capability.h... no
checking: function setns... no
WARNING: instance feature is disabled due to lack of kernel support

checking: libssl+headers... no
checking: libuuid+headers... no
checking: libseccomp+headers... no
=> generating fragments ...
=> building Makefile ...
=> project singularity setup with :
- host arch: x86_64
- host wordsize: 64-bit
- host C compiler: cc
- host system: darwin
---
- target arch: x86_64
- target wordsize: 64-bit
- target C compiler: cc
---
- config profile: release
---
- SUID install: no
- Network plugins: no
---
- verbose: no
---
- version: 3.1.0-rc1
=> /Users/eduardo/go/src/github.com/sylabs/singularity/builddir/Makefile ready, try:
$ cd /Users/eduardo/go/src/github.com/sylabs/singularity/builddir
$ make
[eduardo:singularity]$ make -C builddir/ -j8
GEN etc/bash_completion.d/singularity
GO singularity
[+] GO_TAGS "containers_image_openpgp"
GEN singularity.conf from /usr/local/etc/singularity/singularity.conf
[eduardo:singularity]$ sudo make -C builddir/ install
Password:
INSTALL /usr/local/bin/singularity
INSTALL /usr/local/etc/bash_completion.d/singularity
INSTALL /usr/local/etc/singularity/singularity.conf
DONE

Version and pull from library

[eduardo:singularity]$ singularity version
3.1.0-rc1
[eduardo:singularity]$ singularity pull alpine_macOS.sif library://alpine
WARNING: Authentication token file not found : Only pulls of public images will succeed
[eduardo:singularity]$ ls -ll alpine_macOS.sif
-rwxr-xr-x 1 eduardo staff 2180027 Feb 11 11:27 alpine_macOS.sif

Remote Build

[eduardo:singularity]$ singularity build --remote -d library://sylabsed/testing/remote:3.1-rc examples/busybox/Singularity
Build submitted! Once it is complete, the image can be retrieved by running:
singularity pull --library https://library.sylabs.io library://sylabsed/testing/remote:3.1-rc

Alternatively, you can access it from a browser at:
https://cloud.sylabs.io/library/sylabsed/testing/remote:3.1-rc

[eduardo:singularity]$ singularity push alpine_macOS.sif library://sylabsed/testing/push:3.1-rc
INFO: Now uploading alpine_macOS.sif to the library
2.08 MiB / 2.08 MiB [=================================================================================================================================================================================================] 100.00% 1.44 MiB/s 1s
INFO: Setting tag 3.1-rc
```

[ArangoGutierrez]

• Reintroduced support for zypper bootstrap agent
  Signed-off-by: Eduardo Arango [email protected]

Zypper at bootstrap on definition file

BootStrap: zypper
OSVersion: 42.3
MirrorURL: http://download.opensuse.org/distribution/leap/%{OSVERSION}/repo/oss/
Include: zypper

%runscript
    echo "This is what happens when you run the container..."


%post
    echo "Hello from inside the container"
    zypper -n install bc

Build and check

$> sudo singularity build zypper_test.sif examples/opensuse/Singularity 
[...lots of output...]
$> singularity exec zypper_test.sif cat /etc/os-release 
NAME="openSUSE Leap"
VERSION="42.3"
ID=opensuse
ID_LIKE="suse"
VERSION_ID="42.3"
PRETTY_NAME="openSUSE Leap 42.3"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:42.3"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"

[ArangoGutierrez]

• Changed keys command group to key (retained hidden keys command for backward compatibility)
  Signed-off-by: Eduardo Arango [email protected]

Run

for i in list newpair pull push search ; do
 singularity {key/keys} $i
done
[..same output...]

[ArangoGutierrez]

• Added caching for containers pulled/built from the Container Library
• Added cache command group to inspect and manage cached files
  Signed-off-by: Eduardo Arango [email protected]

Tested

  clean       Clean your local Singularity cache
  list        List your local Singularity cache

With library/docker/oci/blob
Steps followed:

• listed existing cache
• removed existing cache
• pulled from library/docker 2 times (without and with existing cache)
• listed new cache

[ikaneshiro]

• Added the scratch bootstrap agent to build from anything
• Signed-off-by: Ian Kaneshiro [email protected]

Empty definition to build empty base container

Bootstrap: scratch

Properly builds an empty rootfs with metadata. Doesn't run because no programs are installed in the container as expected. Also able to modify file system throug %setup section, though that is not shown here.

$ sudo singularity build -s scratch scratch.def
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
INFO:    Creating sandbox directory...
INFO:    Build complete: scratch
$ tree -a scratch
scratch
├── dev
├── environment -> .singularity.d/env/90-environment.sh
├── etc
│   ├── hosts
│   └── resolv.conf
├── .exec -> .singularity.d/actions/exec
├── home
├── proc
├── root
├── .run -> .singularity.d/actions/run
├── .shell -> .singularity.d/actions/shell
├── singularity -> .singularity.d/runscript
├── .singularity.d
│   ├── actions
│   │   ├── exec
│   │   ├── run
│   │   ├── shell
│   │   ├── start
│   │   └── test
│   ├── env
│   │   ├── 01-base.sh
│   │   ├── 90-environment.sh
│   │   ├── 94-appsbase.sh
│   │   ├── 95-apps.sh
│   │   ├── 99-base.sh
│   │   └── 99-runtimevars.sh
│   ├── labels.json
│   ├── libs
│   ├── runscript
│   ├── Singularity
│   └── startscript
├── sys
├── .test -> .singularity.d/actions/test
├── tmp
└── var
    └── tmp

13 directories, 23 files

[GodloveD]

I deleted all of the check marks and we can start again now on rc4 (hopefully final rc)

[GodloveD]

rc4 builds on mac

09:27 PM
david@davids-MacBook-Pro-3:~/go/src/github.com/sylabs/singularity
$ sudo make -C builddir/ install
Password:
 INSTALL /usr/local/bin/singularity
 INSTALL /usr/local/etc/bash_completion.d/singularity
 INSTALL /usr/local/etc/singularity/singularity.conf
 DONE

09:33 PM
david@davids-MacBook-Pro-3:~/go/src/github.com/sylabs/singularity
$ singularity
Usage:
  singularity [global options...] <command>

Available Commands:
  build       Build a new Singularity container
  exec        Execute a command within container
  key         Manage OpenPGP key stores
  pull        Pull a container from a URI
  push        Push a container to a Library URI
  run         Launch a runscript within container
  search      Search the library
  shell       Run a Bourne shell within container
  sign        Attach cryptographic signatures to container
  test        Run defined tests for this particular container
  verify      Verify cryptographic signatures on container
  version     Show application version


09:33 PM
david@davids-MacBook-Pro-3:~/go/src/github.com/sylabs/singularity
$ singularity version
3.1.0-rc4

09:33 PM
david@davids-MacBook-Pro-3:~/go/src/github.com/sylabs/singularity
$ singularity pull library://alpine
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Downloading library image
 2.08 MiB / 2.08 MiB [===============================] 100.00% 3.20 MiB/s 0s

[GodloveD]

rc4 scratch bootstrap agent works

vagrant@ubuntu-bionic:~$ sudo singularity build alpine go/src/github.com/sylabs/singularity/examples/scratch/Singularity.alpine
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
INFO:    Running setup scriptlet
+ alpine_minirootfs_url=http://dl-cdn.alpinelinux.org/alpine/v3.8/releases/x86_64/alpine-minirootfs-3.8.1-x86_64.tar.gz
+ curl http://dl-cdn.alpinelinux.org/alpine/v3.8/releases/x86_64/alpine-minirootfs-3.8.1-x86_64.tar.gz
+ tar xz -C /tmp/sbuild-780693376/fs --exclude=./dev --exclude=./etc/hosts
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2093k  100 2093k    0     0  2953k      0 --:--:-- --:--:-- --:--:-- 2949k
INFO:    Creating SIF file...
INFO:    Build complete: alpine

[GodloveD]

rc4 zypper bootstrap agent works

david@dcg-sles12:~> sudo `which singularity` build opensuse go/src/github.com/sylabs/singularity/examples/opensuse/Singularity
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Adding repository 'repo-oss' .............................................................................................................................................................................[done]
Repository 'repo-oss' successfully added

URI         : http://download.opensuse.org/distribution/leap/42.3/repo/oss/
Enabled     : Yes
GPG Check   : Yes
Autorefresh : No
Priority    : 99 (default priority)

Repository priorities are without effect. All enabled repositories share the same priority.
Retrieving repository 'repo-oss' metadata -------------------------------------------------------------------------------------------------------------------------------------------------------------------[\]

Automatically importing the following key:

  Repository:       repo-oss
  Key Name:         openSUSE Project Signing Key <[email protected]>
  Key Fingerprint:  22C07BA5 34178CD0 2EFE22AA B88B2FD4 3DBDC284
  Key Created:      Mon 05 May 2014 08:37:40 AM UTC
  Key Expires:      Thu 02 May 2024 08:37:40 AM UTC
  Rpm Name:         gpg-pubkey-3dbdc284-53674dd4
[...snip...]
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: opensuse

[GodloveD]

rc4 key command works

[vagrant@localhost ~]$ singularity help
[...snip...]
  key         Manage OpenPGP key stores

and

[vagrant@localhost ~]$ singularity key search [email protected]
Search results for '[email protected]'

Type bits/keyID     Date       User ID
--------------------------------------------------------------------------------
pub  4096R/8EE0DC4A 2018-10-08 Dave Godlove (demo) <[email protected]>
--------------------------------------------------------------------------------
pub  4096R/F54EBE59 2018-10-23 Dave Godlove (this is the best key) <[email protected]>
--------------------------------------------------------------------------------
pub  4096R/D8A1A670 2018-10-24 Dave Godlove (demo) <[email protected]>
--------------------------------------------------------------------------------
pub  4096R/71D4E11D 2018-10-24 Dave Godlove <[email protected]>
--------------------------------------------------------------------------------
pub  4096R/F643F561 2018-10-25 Dave Godlove <[email protected]>
--------------------------------------------------------------------------------
pub  4096R/A8D0DABB 2018-11-02 Dave Godlove <[email protected]>
--------------------------------------------------------------------------------
pub  4096R/7EB52F9C 2018-12-05 Dave Godlove <[email protected]>
--------------------------------------------------------------------------------
pub  4096R/096357EB 2019-02-04 Dave Godlove <[email protected]>
--------------------------------------------------------------------------------

and

[vagrant@localhost ~]$ singularity key newpair
Enter your name (e.g., John Doe) : some dude
Enter your email address (e.g., [email protected]) : [email protected]
Enter optional comment (e.g., development keys) :
Generating Entity and OpenPGP Key Pair... Done
Enter encryption passphrase :

[syl-ilumb]

rc4 setup for oci runtime testing:

vagrant@ubuntu-bionic:~$ singularity --version
singularity version 3.1.0-rc4
vagrant@ubuntu-bionic:~$ singularity pull docker://alpine
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0
 2.63 MiB / 2.63 MiB [======================================================] 2s
Copying config sha256:57c41aa0b96eb7fcf351321550126306e1da1bc1b1c1c847f4976fd201d95ba1
 585 B / 585 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: alpine_latest.sif

[syl-ilumb]

rc4 testing of oci mount works:

vagrant@ubuntu-bionic:~$ singularity oci mount alpine_latest.sif /var/tmp/alpine_latest
FATAL:   this command is required to be run as root

vagrant@ubuntu-bionic:~$ sudo singularity oci mount alpine_latest.sif /var/tmp/alpine_latest

vagrant@ubuntu-bionic:~$ sudo df -k /var/tmp/alpine_latest
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda1       10098432 2137788   7944260  22% /

vagrant@ubuntu-bionic:~$ sudo ls /var/tmp/alpine_latest
config.json  overlay  rootfs

[syl-ilumb]

rc4 testing of oci create works:

vagrant@ubuntu-bionic:~$ singularity oci create -b /var/tmp/alpine_latest oci_alpine
FATAL:   this command is required to be run as root

vagrant@ubuntu-bionic:~$ sudo singularity oci create -b /var/tmp/alpine_latest oci_alpine

[syl-ilumb]

rc4 testing of oci umount works:

vagrant@ubuntu-bionic:~$ singularity oci umount /var/tmp/alpine_latest
FATAL:   this command is required to be run as root

vagrant@ubuntu-bionic:~$ sudo singularity oci umount /var/tmp/alpine_latest

vagrant@ubuntu-bionic:~$ sudo df -k /var/tmp/alpine_latest
df: /var/tmp/alpine_latest: No such file or directory

[jscook2345]

Results of running OCI validation suite in a docker container (centos7):

3519 passing (3m)
433 pending
28 failing

[jmstover]

mconfig -v --prefix=/usr/local/singularity/3.1.0-rc4
make -C builddir/ old_config=$(pwd)/old_singularity.conf

builddir/singularity.conf then uses old_singularity.conf as a base. You must use a full path though.

[ArangoGutierrez]

https://github.com/sylabs/singularity/releases/tag/v3.1.0