Singularity: Long command causes FATAL: Failed to initialize runtime engine: engine "" is not found error

[chrarnold]

Version of Singularity:

singularity version 3.5.2-1.1.el7

Expected behavior

Command doesn't throw error

Actual behavior

Immediate error is thrown upon executing a 125.000 characters long command:

FATAL: Failed to initialize runtime engine: engine "" is not found error

Steps to reproduce this behavior

Execute something like this:
singularity exec bash -c \' long command (>120k chars)\' . If I remove the absolute path from the command, that makes the command so long in the first place, it works, so it must be related to the total command length.

What OS/distro are you running

NAME="CentOS Linux"
VERSION="7 (Core)"

How did you install Singularity

No idea, was done by IT.

[cclerget]

Hi @chrarnold , is it a requirement to pass 120kb of arguments ?

This error could be handled more gracefully than actually, for information Singularity has a limit of 131072 bytes to store the configuration in JSON format (environment, command arguments, container setup), the error Failed to initialize runtime engine: engine "" is not found error means that the JSON configuration was truncated at 131072.

But 120k really ? Why not using a script directly or pipe ?

[chrarnold]

Hi, unfortunately, this can happen more easily than one thinks: I am using Snakemake, a automated workflow manager that constructs the command line calls, and this error happens when integrating Snakemake and Singularity. Essentially, with many arguments (here, over 700 files as input), this limit can easily be reached as the full path is used for all files. Is it possible to increase the limit to a higher value?

[cclerget]

unfortunately, this can happen more easily than one thinks

Sure but hitting this limit is not a good sign imo

Is it possible to increase the limit to a higher value?

No because the value is hardcoded and is actually big enough for most usage.

There is also few things to keep in mind :

• all arguments (+environment variables) are subject to the OS limit that you can get with getconf ARG_MAX (often value 2097152)
• additionally one argument can't be longer than the hardcoded OS value 131072, so in the case of running bash -c ' long command (>120k chars)'the command argument may reach this limit and returns Argument list too long error

As I mentioned you could use pipe and get rid of Singularity limit simply by piping your arguments over stdin like this :

echo "long command (>120k chars)" | singularity exec image bash -c "read -u 0 line; set -- \$line; exec \"\$@\""

[ironslay]

I'm not contesting the comment that hitting this limit is not a good sign. But I would agree that this can happen more easily than one thinks.

HPC clusters that provide large curated application environments via Modules/Lmod, build all their own dependencies, and link dynamically can hit this limit. We provide a R environment that includes ~5k curated packages and when fully loaded the user environment is around 133 KB. I'm sure we're not the only folks in this boat.

[chrarnold]

Totally agree. For us, a total different use case exceeds the limit, and we are very limited because of that and cannot use singularity at all. In bioinformatics pipelines, particularly those are automated via workflow managers, hitting this limit happens easily as I tried to elaborate. Just concatanate a few hundred files, each with a long path, and you are already over it. I dont even want to mention the use cases where thousands of files are integrated, 100k is simply not enough at all. I strongly ask again for increasing this hard coded limit to something above 500k at least, or 1 million. Since even such long commands will be correctly embedded via quotation marks etc, I don't see an upper limit that would cause any other issues. Thank you.

[dmachi]

I am seeing this same issue during the build processes. I have a singularity definition, mostly made up of apprun (something like 300 apprun sections) that is 84k. I get this same error during the build process. This same image was working when we only had about 30 apprun sections. I am not generating any particularly long commands, so I'm guessing that there is some environment variable being created that exceeds this limit. Is there any way to work around this?

[chrarnold]

Hi @dctrud , I see development for this issue. Could you please summarize the change, will the newest SIngularity versions have a bigger limit, or, as I understood it, they can be compiled with a bigger limit, but this must be set specifically when installing it?

[dtrudg]

The current releases of Singularity are limited due to the manner they pass configuration around internally.

The master branch now has a PR merged from @cclerget that uses a different method to pass the engine configuration, allowing a configuration of up to approx. 1MB which will permit long command lines, apprun sections etc, up to that limit.

This isn't going to appear very quickly in a patch release as it's a significant change to the way the singularity engine handles configuration, and should go through more testing. I'm not a single person making a decision on that, but I'd guess it will be in an official release of 3.7 in November.

We'd be grateful for any testing of the new code with the scenarios that were mentioned in this thread.

[nsheff]

Thanks @dtrudg -- can you provide any additional update here as to the status of the longer line allowance in singularity versions? Did this make it into 3.7? Based on reading the changelog I think it may have, but I am having some trouble interpreting it.

[dtrudg]

@nsheff - yes, this is in 3.7