Moby: Dockerfile: RUN over multiple lines without && \

[d2xdt2]

It is very common to see Dockerfile's which uses RUN over several lines like so

RUN cd / && \
    git clone https://github.com/ && \
    cd /fileadmin/ && \
    bundle install && \
    rake db:migrate && \
    bundle exec rails runner "eval(File.read 'createTestUser.rb')" && \
    mkdir /pending && \
    mkdir /live && \
    chmod 777 /pending && \
    chmod 777 /live

where && \ is used many times.

The problem is that it is tedious to read and write.

Proposal

RUN should by itself add && \ at end of line, and terminate at the first blank line. The above example would become

RUN cd /
    git clone https://github.com/
    cd /fileadmin/
    bundle install
    rake db:migrate
    bundle exec rails runner "eval(File.read 'createTestUser.rb')"
    mkdir /pending
    mkdir /live
    chmod 777 /pending
    chmod 777 /live

Much easier to read and write.

[duglin]

I'm not a fan of implicit things like this. If people want to continue a line they should let us know it with a . Leading spaces (or blank lines) are too indeterminate to mandate that it means "continue previous line".

[cpuguy83]

Agree with @duglin

[tianon]

@duglin what about heredoc support? :smile:

RUN <<EOR
    cd /
    git clone https://github.com/
    cd /fileadmin/
    bundle install
    rake db:migrate
    bundle exec rails runner "eval(File.read 'createTestUser.rb')"
    mkdir /pending
    mkdir /live
    chmod 777 /pending
    chmod 777 /live
EOR

[duglin]

@tianon if we were allowed to make changes to the builder, then heck yea!!

[d2xdt2]

heredoc would be a great solution, as many now have a RUN for each line, just to avoid having to type and look at && \. Many official images on Docker Hub does it right now.

[thaJeztah]

heredoc was proposed/discussed a few times before (e.g. https://github.com/docker/docker/issues/1554, https://github.com/docker/docker/issues/1799)

However, there's currently a freeze on the Dockerfile syntax, pending moving the builder to the client; https://github.com/docker/docker/issues/14298. It looks like @jlhawn's POC builder does support heredoc; https://github.com/jlhawn/dockramp

[thaJeztah]

I'm going to close this issue, following the discussion above; supporting heredoc is probably a nicer solution to this problem, however, cannot currently be implemented due to the Dockerfile syntax currently being frozen; https://github.com/docker/docker/blob/master/ROADMAP.md#2-frozen-features

[thaJeztah]

Thanks for suggesting this though @d2xdt2! It's much appreciated to get feedback and feature suggestions.

[pierreozoux]

@thaJeztah is it still frozen? Is there a chance that will see this landing one day? (It is indeed ugly :) )

[AlJohri]

my multiline commands gets rendered like this (with large amounts of spaces between commands) when docker runs it. is there any interest in fixing this part?

[thaJeztah]

@AlJohri that's the way the shell sees it, that's no different from using line-continuation in your shell. Try running this in your shell;

echo "hello-world" && \
    echo "foo-bar"

then press the "up" arrow to get the command from your history, and it'll show;

echo "hello-world" &&     echo "foo-bar"

The "gaps" are probably because you aligned the lines in your Dockerfile, taking RUN into account (four spaces) when aligning.

[AlJohri]

@thaJeztah yup!

I was mostly just curious if it were possible to render without the spaces by regexing them out or something

[thaJeztah]

@AlJohri No, that won't be possible without fully replicating the shell and "interpreting" the steps. Sure, it's easy to strip out multiple consecutive white-spaces in the output, but that would not be the right thing to do, because white-space can have (and in most cases _has_) meaning. For example, in the following you _want_ the whitespace to be preserved in the output of docker build;

echo "hello \
            world"

[AlJohri]

Figured as much, thanks for explaining!

[gukoff]

I'm not a fan of implicit things like this. If people want to continue a line they should let us know it with a . Leading spaces (or blank lines) are too indeterminate to mandate that it means "continue previous line".

It is perfectly explicit. There are a lot of successful examples of the whitespace indentation, e.g. Python, YAML, Haskell. Te proposal isn't about 'continuing a line', it is about combining the lines.

[inf3rno]

I support heredoc too. Just wondering why it is so hard to add a simple feature requested by many ppl...

[thaJeztah]

@inf3rno see https://github.com/moby/moby/issues/34423

[piotrminkina]

My own Dockerfile coding standard:

# Example with apt-get update, install and clean it's cache
RUN apt-get update -yq \
 && DEBIAN_FRONTEND=noninteractive \
    apt-get install -yq pkg1 pkg2 pkg3 \
 && find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete

# Example with many packages to install at one time
RUN apt-get update -yq \
 && DEBIAN_FRONTEND=noninteractive \
    apt-get install -yq \
    pkg1 \
    pkg2 \
    pkg3 \
    ... \
    pkg10 \
 && find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete

# Example untar with pipes
ARG WEBSOCKIFY_VERSION="v0.8.0"
RUN mkdir -p /opt/websockify \
 && curl -sSL https://github.com/novnc/websockify/archive/${WEBSOCKIFY_VERSION}.tar.gz \
  | tar -zx -C /opt/websockify --strip 1

# Example unzip without pipes and fixing files permissions
ARG BROWSERMOB_PROXY_VERSION=2.1.0-beta-3
RUN apt-install --no-install-recommends openjdk-8-jre \
 && curl -sSLo /tmp/browsermob-proxy-bin.zip \
    https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-${BROWSERMOB_PROXY_VERSION}/browsermob-proxy-${BROWSERMOB_PROXY_VERSION}-bin.zip \
 && unzip /tmp/browsermob-proxy-bin.zip -d /opt/ \
 && chown -R 0:0 "/opt/browsermob-proxy-${BROWSERMOB_PROXY_VERSION}/" \
 && find "/opt/browsermob-proxy-${BROWSERMOB_PROXY_VERSION}/" -type d -print0 | xargs -r -0 chmod 0755 \
 && find "/opt/browsermob-proxy-${BROWSERMOB_PROXY_VERSION}/" -type f -print0 | xargs -r -0 chmod 0644 \
 && chmod a+x "/opt/browsermob-proxy-${BROWSERMOB_PROXY_VERSION}/bin/browsermob-proxy" \
 && ln -sfn "/opt/browsermob-proxy-${BROWSERMOB_PROXY_VERSION}/" /opt/browsermob-proxy \
 && rm -f /tmp/browsermob-proxy-bin.zip

# Example call command with many arguments at one time (my propose is use long parameters names)
RUN some-command1 \
    --some-option1=some-value1 \
    --some-option2=some-value2 \
    --some-option2=some-value2 \
    --some-option2=some-value2 \
 && some-command2 \
    --some-option1=some-value1 \
    --some-option2=some-value2 \
    --some-option2=some-value2 \
    --some-option2=some-value2

[jen-soft]

Using pydocker ( https://github.com/jen-soft/pydocker )

[ Dockerfile.py ]

from pydocker import DockerFile  # sudo pip install -U pydocker

d = DockerFile(base_img='debian:8.2', name='jen-soft/custom-debian:8.2')

d.RUN_bash_script('/opt/set_repo.sh', r'''
cat >/etc/apt/sources.list <<EOL
deb     http://security.debian.org/ jessie/updates main
deb-src http://security.debian.org/ jessie/updates main
EOL
apt-get clean && apt-get update
''')

d.EXPOSE = 80
d.WORKDIR = '/opt'
d.CMD = ["python", "--version"]

# d.generate_files()
d.build_img()

# sudo wget -qO- https://get.docker.com/ | sh

python Dockerfile.py
docker images

[whirlwin]

What should be the expected behaviour be when omitting &&?

Should the commands in the chain only be executed one after another on success (like with &&). Or should they be independent, so that one command does not halt the next commands.

This is not perfectly clear clear. Why && and not ||?

[thaJeztah]

The && is mainly used if you want all commands to complete successfully (e.g. do something && do something else if the previous command completed sucessfully). You can use different approaches (e.g., set -eux; some command; some other command;).

By default (in a Linux Dockerfile), RUN <shell commands> is equivalent to sh -c '<shell commands>, and docker build will execute that step, and commit the results once the sh -c terminates successfully. If the sh -c exits with a non-zero exit code, the build is marked "failed"

[chapmanjacobd]

a trick I learned recently is that you can use subshells to run multiple commands and you don't need to use \ to put multiple lines. I think this should work with docker run but I didn't test it yet:

docker run {CONTAINER_TAG} /bin/bash -c '( (
    cd /
    git clone https://github.com/
    cd /fileadmin/
    bundle install
    rake db:migrate
    bundle exec rails runner "eval(File.read 'createTestUser.rb')"
    mkdir /pending
    mkdir /live
    chmod 777 /pending
    chmod 777 /live
    ) &) &'

you might need to wrap it twice ( ( ) & ) & as shown above. otherwise below should work... if not working try double quote wrapping instead of single quote or vice versa

or

RUN (cd / 
# or if ^^that^^ doesn't work try
RUN /bin/sh -c "(cd /
    git clone https://github.com/
    cd /fileadmin/
    bundle install
    rake db:migrate
    bundle exec rails runner "eval(File.read 'createTestUser.rb')"
    mkdir /pending
    mkdir /live
    chmod 777 /pending
    chmod 777 /live
    )"