Singularity: Remote build failed to parse DefFile header

[audreystott]

Version of Singularity:

What version of Singularity are you using? Run:

$ singularity version

3.3.0-1

singularity build -r scrnaseq.sif scrnaseq.def

Expected behavior

What did you expect to see when you do...?

Build to happen as per normal

Actual behavior

What actually happend? Why was it incorrect?

FATAL: Unable to build from scrnaseq.def: failed to parse DefFile header: invalid header keyword found: stage

Steps to reproduce this behavior

How can others reproduce this issue/problem?

Build multistage image on remote endpoint cloud.staging.sylabs.io

What OS/distro are you running

$ cat /etc/os-release

How did you install Singularity

Write here how you installed Singularity. Eg. RPM, source.

HPC installation by system administrators

[jscook2345]

@audreystott

Can you please include the contents of scrnaseq.def in a comment? Thanks!

[audreystott]

@jscook2345
Sure! Thanks for your reply.

Bootstrap: docker
From: continuumio/miniconda:4.7.12
Stage: devel

%post
    echo "name: scrnaseq
channels:
  - defaults
  - bioconda
  - conda-forge
dependencies:
  # Default bismark
  - fastqc=0.11.8
  - r-seurat=3.0.2
  - libopenblas=0.3.6
  - umap-learn=0.3.10
  - xorg-libxrender=0.9.10
  - xorg-libx11=1.6.9
  - xorg-libxext=1.3.4
  - xorg-libxau=1.0.9" >> /opt/conda.yml
    echo "source /opt/conda/etc/profile.d/conda.sh" >> $SINGULARITY_ENVIRONMENT
    echo "source activate $(head -1 /opt/conda.yml | cut -d' ' -f2)" >> $SINGULARITY_ENVIRONMENT
    /opt/conda/bin/conda env create -f /opt/conda.yml

BootStrap: docker
From: ubuntu:bionic
Stage: final

%post
    mkdir -p /tmp/cellranger-build \
    && cd /tmp/cellranger-build \
    && apt-get update \
    && apt-get install -y wget \
    && wget -O cellranger-3.1.0.tar.gz "http://cf.10xgenomics.com/releases/cell-exp/cellranger-3.1.0.tar.gz?Expires=1575380182&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cDovL2NmLjEweGdlbm9taWNzLmNvbS9yZWxlYXNlcy9jZWxsLWV4cC9jZWxscmFuZ2VyLTMuMS4wLnRhci5neiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTU3NTM4MDE4Mn19fV19&Signature=iwKIyXJVzlS7zyYQ0BzyYkN4hejK22alEPtd36ymMgHMuTCXAT~~crM3M3LKcMyiZ7TRSk6pBcTyscNENX13wtP5gOxL4xTgbzmVOT4J8UcuHK-L0AAP1kh7YSDEUrU~4DtvXSDz1YrHB88YhBTybTDgjj2hHJvn94M-PkxEWM9xbGMADIEyk3xKEApQNF8g7AGLyQUlXeAFFBhoGozv3R8CFeOHfRyfinvXBD-9DS2o2r3MRf3C5CY~gWDTABZUN-PO4Pk08rJNDiWfgKHd2ixShe78orr61EAkgCH9jAt-i4X-u9UiXMnJQrPh5UXBn1DrCfZBtHUdsHFGPUZ~7Q__&Key-Pair-Id=APKAI7S6A5RYOXBWRPDA" \
    && mkdir /apps \
    && mv cellranger-3.1.0.tar.gz /apps/ \
    && cd /apps \
    && tar -xzvf cellranger-3.1.0.tar.gz \
    && rm -f cellranger-3.1.0.tar.gz \
    && rm -rf /tmp/cellranger-build

%files from devel
    /opt/conda
    conda.yml

%environment
    export PATH=/apps/cellranger-3.1.0:$PATH
    export PATH=/opt/conda/envs/$(head -1 /opt/conda.yml | cut -d' ' -f2)/bin:$PATH

[dtrudg]

Hello @audreystott - support for remote multi stage build was added in Singularity 3.4.0. I'm afraid you will need to upgrade to use this feature.

(Ref - relevant commit is: https://github.com/sylabs/singularity/pull/4204/commits/b27a01736ab8b2d2e084865a1319ccea52e7ec7c)

[audreystott]

Hi @dctrud, thanks. However, I tried it on the remote builder on the Sylabs staging webpage and wasn’t able to either.

[dtrudg]

@audreystott - please don't use the staging site - this is not the production instance of the cloud services, and it may be inaccessible and data may be removed without warning.

@EmmEff - can you comment on the remote build / multi-stage support? Thanks.

[audreystott]

@dctrud noted. I think I found it through one of the issues here, but will not use it now that you’ve cautioned.

[EmmEff]

@audreystott did you perform this test recently? Using your def file above, I was able to start the multistage build successfully in the staging (and production, for that matter) environment.

[audreystott]

@EmmEff yes earlier this week I did. I will try it again. But we are upgrading to 3.5.1 soon so that'll work too. :)

[EmmEff]

Ahh, there was pre-parsing done on the client-side to validate the def file before it's sent to the remote build server. That change happened in the master branch of the OSS version approximately 4 months ago. v3.4.0 and later has support for multistage through the remote builder.

[audreystott]

Thanks for clarifying. Glad to know it wasn't user error :D I'll close this now then. Thanks again everyone.

[twsaari]

I'm seeing this same issue with Singularity 3.5.2

bioc.def
--------------
Boostrap: docker
From: bioconductor/bioconductor_docker:RELEASE_3_10

%environment
    export PASSWORD=bioc

singularity build --remote bioc.simg bioc.def
FATAL: Unable to build from scRNAseq.def: failed to parse deffile header: invalid header keyword found: boostrap

[EmmEff]

You have a typo... it should be "bootstrap" (you have "boostrap") :)

[twsaari]

Ah my mistake. Thanks for catching that!