Pipelines: Add Support for Docker-in-Docker in DSL (See https://github.com/argoproj/argo/tree/master/examples#docker-in-docker-using-sidecars)

[cliveseldon]

We would like to have steps in a pipeline that allow image builders. This might require docker-in-docker.
How can this be easily facilitated?

[jlewi]

I think one option would be to have your pipeline step run a separate K8s job. Then you could set whatever options (e.g. DockerInDocker, Security context are needed).

[paveldournov]

@cliveseldon - Please take a look at the KFP SDK, and the methods that allow building containers from Python functions. The SDK is creating a K8S job for building containers images, you can either use the SDK within a pipeline step or use the same technique for creating an image. Please let me know if that works.

[vicaire]

@cliveseldon, Argo supports building containers inside containers.

See https://github.com/argoproj/argo/tree/master/examples#docker-in-docker-using-sidecars

Would making this functionality available from the DSL work for you?

[cliveseldon]

Sorry, missed this. Yes that would be very useful if there was easy access to docker-in-docker.

[vicaire]

/cc @hongye-sun (Adding Hongye who may have some input on this).

@cliveseldon, would you be interested in contributing so that it is easy to specify the following in the DSL?

https://github.com/argoproj/argo/tree/master/examples#docker-in-docker-using-sidecars

[cliveseldon]

@vicaire Happy to help. Just a bit new to the code base so not sure where to start.

[hongye-sun]

@cliveseldon, the docker in docker support depends on sidecar feature, which is currently working in the PR (https://github.com/kubeflow/pipelines/pull/879). Once the PR is in, you should be able to build similar pipeline as argo example does.

[gaoning777]

This #879 is merged. Feel free to add the docker-in-docker feature.

[hongye-sun]

I don't think there is more work need to be done to support dind other than the sidecar support. Closing this bug.