Buildkit: dockerfile2llb: support `ADD git://` with git source op

[AkihiroSuda]

@tonistiigi Is this intentionally left unimplemented?

[tonistiigi]

What git:// URL support do you mean? git context should work an use git source op. (although it doesn't seem to be implemented 100% correct yet https://github.com/moby/moby/issues/38254)

[AkihiroSuda]

I meant ADD git:// URL

[tonistiigi]

I don't think old builder supports it, right? This has not been added as a new feature. If you find it useful we can discuss it (and maybe use the experimental branch for it). Not sure I have an opinion myself atm. I think FROM git:// might be more useful.

[FernandoMiguel]

I don't personally have a use case for this, but
ADD sounds far more logical to me than FROM

OTOH, COPY vs ADD is already confusing enough.
If FROM becomes similar, we should really move down to just one 😅

[thaJeztah]

At some point the ADD instruction was marked for deprecation because it did too much magic (remote sources, automatically extracting archives (but _only_ if local) and such), which can be confusing, and doesn't provide a great amount of control by the user. I'm not sure if ADD git:// would be the best approach.

Some thoughts I've been playing with;

COPY --from=git://... /some/file /somewhere
COPY --from=https://... /some/file /somewhere

Last one may be tricky; what if the URL specified in --from specifies a single file? Guess we just produce an error.

Slightly off-topic, but another thing I was thinking about is to allow mounting remote sources:

RUN --mount type=remote,src=git://...,dest=/somewhere
RUN --mount type=remote,src=https://...,dest=/somewhere

The above would fetch the remote content, and mount it at /somewhere (dest). This would allow, for example:

RUN --mount type=remote,src=https://download.docker.com/linux/static/stable/x86_64/docker-18.09.0.tgz,dest=/temp/ \
  tar -xzf /temp/docker-18.09.0.tgz --strip 1  -C /usr/local/bin docker/docker

[FernandoMiguel]

COPY --from=git://... /some/file /somewhere
COPY --from=https://... /some/file /somewhere

I like this!

but how do you deal with auth?

[thaJeztah]

Good one; didn't really give that much thought yet ☺️ - suggestions welcome 👍

[tonistiigi]

Slightly off-topic, but another thing I was thinking about is to allow mounting remote sources:

If we have FROM git:// you can already mount them automatically.

FROM git:// as mygitstage
FROM alpine
RUN --mount=target=/dest,from=mygitstage cmd

The advantage is that you can now use ARG to switch between the different source location.

[tonistiigi]

but how do you deal with auth?

For git we can do automatic ssh auth (like the ssh forwarding support for RUN). For https I don't think it makes sense to complicate this with one specific form of auth. You can always do the same things with an extra stage running curl and mount secrets into that stage for any type of authentication.

[FernandoMiguel]

Slightly off-topic, but another thing I was thinking about is to allow mounting remote sources:

If we have FROM git:// you can already mount them automatically.

FROM git:// as mygitstage
FROM alpine
RUN --mount=target=/dest,from=mygitstage cmd

The advantage is that you can now use ARG to switch between the different source location.

ohhhh wow

[thaJeztah]

If we have FROM git:// you can already mount them automatically.

True, but I _do_ find that construct a bit confusing, as in; FROM git:// is roughly the equivalent of;

FROM scratch
ADD git://... /

So that stage won't be a functional stage (unless git://... provides a rootfs. I agree that the advantage is that ARG can be used (flags currently don't handle variable substitution), but idk, it feels a bit odd to use FROM just to fetch the source, only to use it for --mount later on.

[tonistiigi]

So that stage won't be a functional stage (unless git://... provides a rootfs

Not sure what you mean by that. As there are millions of ways to have a nonfunctional stage if you use wrong commands. And nothing is stopping the user from making the files in git to be functional.

Perhaps I was not clear that by having FROM git:// is automatically means that COPY --from=git:// works as well if you are not using stage names/args as FROM and COPY --from take the same arguments. As do the mounts as this also works already:

#syntax=docker/dockerfile:experimental
from alpine
run --mount=from=busybox,target=/busybox ls -l /busybox