Moby: Multiline Dockerfile syntax

[mzdaniel]

Dockerfile is a powerful building tool.
It will be more visually appealing adding multiline capability to it.
As example:

run /bin/echo -e '#!/bin/bash\necho This is a long \
shell line; echo Next line

To detect the multiline we probably want to use a regex like '[ \t]*\n'

[mzdaniel]

/cc @creack

[vieux]

Fixed by #1838

[from-nibly]

+1 on that. running commands against a postgres server requires a bunch of &, && and ; and makes a really cluttered looking RUN command

[d11wtq]

Anyone know how to use heredoc with this syntax? Docker wants to bunch it onto a single line and causes a bash syntax error:

RUN cat > /etc/example.conf <<'EOF' \
This is one multiple lines \
and should stay that way \
EOF

[crosbymichael]

@d11wtq Why not just add the file instead of catting in the dockerfile?

[d11wtq]

@crosbymichael because (and correct me if I'm wrong), if I edit the file (and I will), Docker won't use my edits, due to the way it caches. I could completely disable caching, but this image includes the entire llvm toolchain and is very big as it is, so I do want caching of the commands, I just need ADD to be aware of changes in the files it is adding. Maybe it already works that way, but IIRC it does not.

[crosbymichael]

Yes, it will use your edits. It does a checksum on the content to pick up chagnes.

[d11wtq]

@crosbymichael Ah, that is excellent. Is that new? I could have sworn I'd been bitten by this in the past. That does indeed solve my issue then :)

[crosbymichael]

Yes as of 0.8 maybe

[develCuy]

Try this one:

RUN echo 'All of your\n\
multiline that you ever wanted\n\
into a dockerfile\n'\
>> /etc/example.conf

[thiagokronig]

This works for me:

RUN echo $'All of your\n\
multiline that you ever wanted\n\
into a dockerfile\n'\
>> /etc/example.conf

[RichardJECooke]

And if your string contains quotes you can do this:

RUN echo $'All of your\n\
multiline'"'"'s that you ever wanted\n\
> /etc/example.conf

[ChipmunkV]

Fixed by #1838

No, it's not.

[searene]

@develCuy @thiagokronig What if my command contains quotes, like the following one

RUN echo 'a\n\
'b'\n\
c\n'\
>> /test

It doesn't work.

[lucaszanella]

Would be awesome if possible to do:

RUN echo $'All of your\n\
multiline that you ever wanted\n\
into a dockerfile\n'\

/etc/example.conf

That is, identation in the dockerfile without getting tabs in the example.conf

[thaJeztah]

See https://github.com/moby/moby/issues/34423

[lonix1]

I want to write a small script, so I tried:

````dockerfile
RUN echo '\n\
#!/bin/bash\n\
foo\n\
bar\n'\

/usr/local/bin/myscript.sh
RUN chmod +x /usr/local/bin/myscript.sh
````

But the first line is considered to be a comment, and so is ignored. Can it be escaped somehow?

[aarzamasov]

I want to write a small script, so I tried:

RUN echo '\n\
  #!/bin/bash\n\
  foo\n\
  bar\n'\
  > /usr/local/bin/myscript.sh
RUN chmod +x /usr/local/bin/myscript.sh

But the first line is considered to be a comment, and so is ignored. Can it be escaped somehow?

RUN echo  $'\n\
  #!/bin/bash\n\
  foo\n\
  bar\n'\
  > /usr/local/bin/myscript.sh
RUN chmod +x /usr/local/bin/myscript.sh

Hi, try ti add $ after the echo.

[jen-soft]

It's help for me ( https://github.com/jen-soft/pydocker )

[ Dockerfile.py ]

from pydocker import DockerFile  # sudo pip install -U pydocker

d = DockerFile(base_img='debian:8.2', name='jen-soft/custom-debian:8.2')

d.RUN_bash_script('/opt/set_repo.sh', r'''
cat >/etc/apt/sources.list <<EOL
deb     http://security.debian.org/ jessie/updates main
deb-src http://security.debian.org/ jessie/updates main
EOL
apt-get clean && apt-get update
''')

d.EXPOSE = 80
d.WORKDIR = '/opt'
d.CMD = ["python", "--version"]

# d.generate_files()
d.build_img()

# sudo wget -qO- https://get.docker.com/ | sh

python Dockerfile.py
docker images

[JamshadAhmad]

I have tried several different varients of that echo command, none of them worked for me.
So I end up creating a file and copying it into container like
COPY ./uploads.ini /etc/example.ini

[comrumino]

Fanatics of printf may prefer a little something like this

RUN printf ''\
'#!/usr/bin/env bash\n'\
"printf 'qrcode command: qrencode -o user.png otpauth://totp/%s@%s?secret=%s'\n"\
'vsftpd /etc/vsftpd.conf'\
 "${ftpuser?}" "${pasv_address?}" "${oathuser_b32seed?}" > "/usr/bin/vsftpd-docker-init"

when it makes sense of course.
Caveat: be mindful of quotes and whitespace

[thondeboer]

I tried a couple of things with the echo command and could not get it to work, until I did this:

RUN echo  '#!/bin/bash\n\
export GRANT_SUDO=no\n\
export HOME=''"''/data''"''\n\
start-notebook.sh \n\
--NotebookApp.base_url=/${BGP_NAMESPACE}-n-${BGP_NOTEBOOK}-i${BGP_NODEINSTANCE} \n\
--NotebookApp.token=${BB_TOKEN} \n\
  --NotebookApp.allow_password_change=False \n\
  --NotebookApp.tornado_settings=''"''{'"'"'headers'"'"': { '"'"'Content-Security-Policy'"'"': '"'"'frame-ancestors *;'"'"' } }''"'' \n\
  --ip=0.0.0.0 \n\
  --config=/config/config.json \n\
  --Session.username=bluebee \n\
  --LabApp.user_settings_dir=/data/.jupyter/lab/user-settings \n\
  --LabApp.workspaces_dir=/data/.jupyter/lab/workspaces \n\
'> /usr/local/bin/bb_script.sh
RUN chmod +x /usr/local/bin/bb_script.sh

It wil then show up as this: (Note the quotes)

#!/bin/bash
export GRANT_SUDO=no
export HOME="/data"
start-notebook.sh
--NotebookApp.base_url=/${BGP_NAMESPACE}-n-${BGP_NOTEBOOK}-i${BGP_NODEINSTANCE}
--NotebookApp.token=${BB_TOKEN}
  --NotebookApp.allow_password_change=False
  --NotebookApp.tornado_settings="{'headers': { 'Content-Security-Policy': 'frame-ancestors *;' } }"
  --ip=0.0.0.0
  --config=/config/config.json
  --Session.username=bluebee
  --LabApp.user_settings_dir=/data/.jupyter/lab/user-settings
  --LabApp.workspaces_dir=/data/.jupyter/lab/workspaces

Note I did NOT use the echo $' trick, since that just inserted a $ in my code.
I could only get it to work to use the #!/bin/bash right in the same line as the echo, otherwise lines with # will still be considered comments, but this is a functioning script.

[RichardBronosky]

How is it that thousands of engineers waste millions of hours on this horrible UX decision in one of the most important Free/Libre Open Source Software projects? If Docker™ are not going to fix this, then we need to just create a Dockerfile preprocessor that converts:

RUN <<ANY_DELIMETER
...
ANY_DELIMETER

or even

...
ENDRUN

into the unintelligible format that Docker™ finds superior. @jen-soft has proven this is possible.

[thaJeztah]

This issue is not the right place for that discussion. This is a 7 year old enhancement request, asking for RUN lines to be wrapped over multiple lines, which was not possible before. That enhancement was addressed by https://github.com/moby/moby/pull/1838, which is why this issue was closed.

How is it that thousands of engineers waste millions of hours on this horrible UX decision in one of the most important Free/Libre Open Source Software projects? If Docker™ are not going to fix this

It's ironic that you mention "free/libre" and "open source", but then expect a company to do the work. This project _is_ open source, especially so that people can participate and contribute fixes, enhancements, and features. While this project was _started_ by "Docker™", it is driven by its maintainers, who maintain the project, and are responsible for making decisions on features and enhancements. While Docker™ pays some of their engineers to work on maintaining this project and developing features, the majority of the maintainers is not even employed by Docker.

into the unintelligible format that Docker™ finds superior.

No-one mentions it's superior; support for newlines (using \ as a line-continuation symbol to allow you to wrap lines) fixed the issue at hand. While adding support for wrapping lines was sufficient at the time, things evolved since, and people started creating more complex Dockerfiles, which asks for more flexible/advanced options.

Commenting on a 7 year old closed issue is not the right channel for proposing changes; to propose changes, open a new issue with a proposal. Here's a proposal that I wrote (based on some other proposals) to add support for a here-doc-style syntax: https://github.com/moby/moby/issues/34423. While (I think) maintainers agree that would be a good addition to the syntax, no work was done on it yet (there's a finite amount of engineering time, and other changes may have priority). That said; this project is open source, and I welcome people to participate on that proposal, or perhaps work on the implementation.

then we need to just create a Dockerfile preprocessor that converts:

Since this ticket was opened, the "builder" has been rewritten from scratch, and the next-generation builder (BuildKit https://github.com/moby/buildkit) was developed. BuildKit can both be used in Docker (currently opt-in by setting DOCKER_BUILDKIT=1, but it will become the default in a future release: https://github.com/moby/moby/issues/40379), or run standalone.

BuildKit allows adding alternative syntaxes ("front-ends") to extend the Dockerfile syntax. For example, the experimental Dockerfile syntax that is used to test / "incubate" new features in the Dockerfile syntax is distributed as a Docker image, but you could even implement your own syntax that's wholly different as Dockerfiles. For example, if you want to build your images using a YAML file: here's a blogpost about someone creating a format called "Mockerfile"), or even build using Buildpacks.

Distributing those alternative syntaxes/front-ends as images provides a mechanism to implement them without making code changes in Docker itself (they can be used on Docker 18.09 and higher when enabling BuildKit, or even older versions when using buildx) - see the "experimental" Dockerfile syntax mentioned earlier.

The proposed here-doc syntax could be implemented as a custom front-end, and very likely would be accepted for inclusion in the experimental Dockerfile syntax (before promoting it to the "stable" syntax). Contributions for that are welcome if someone wants to work on that.

[Bessonov]

@thaJeztah thank you very much for your great answer!