Skaffold: Two-way sync

[balopat]

Syncing in Skaffold to the container works well. However, applications sometimes change files on the filesystem themselves. If this happens inside the container, it is hard to get that change quickly during development.

There might be two types of file changes:
1) inside the code repository, e.g. under the project files in an IDE
2) outside of the code repo, e.g. in /var/log/, or somewhere else

Mounting directories is tricky and slow, also can collide with Skaffold's one directional filesync.
Maybe Skaffold should be able to help with syncing back from the container in dev mode.

I would like to collect feedback from users who face this and see what use cases are there.

[wstrange]

Our use case: We have a complex application with 100+ json configuration files. These files are updated by the application UI running in the container. To save them in git we need to get them back out to the developers environment.

We have tried using minikube volume mounts for this use case. It works- but has some drawbacks - such as complexity (we want to keep things simple for the developer), it doesn't work on non minikube environments, and you can get into a sync loop race with skaffold as files are updated.

Today we use kubectl cp to pull the files out of the container. It is low tech - but is working well enough. A little bit of chrome / ease of use around this workflow would be nice.

[nthypes]

Our use case: We have a complex application with 100+ json configuration files. These files are updated by the application UI running in the container. To save them in git we need to get them back out to the developers environment.

We have tried using minikube volume mounts for this use case. It works- but has some drawbacks - such as complexity (we want to keep things simple for the developer), it doesn't work on non minikube environments, and you can get into a sync loop race with skaffold as files are updated.

Today we use kubectl cp to pull the files out of the container. It is low tech - but is working well enough. A little bit of chrome / ease of use around this workflow would be nice.

Great to hear from you. We are facing the same challenge. The problem with the kubectl cp is that we need to give the pod name, which is dynamic.

So kubectl cp works only manually.

[wstrange]

The problem with the kubectl cp is that we need to give the pod name, which is dynamic.

We use selector labels on kubectl cp (-l app=myapp)- so the shell script is reasonably generic.

[strikeout]

Can we take some inspiration by how DevSpace solved bi-directional host<>pod sync?
They inject a small binary into the container which in turn scans and passes a list of files not present on the host.

https://github.com/devspace-cloud/devspace/tree/master/sync/inject

DevSpace establishes a bi-directional code synchronization between the specified local folders and the remote container folders. It automatically recognizes any changes within the specified folders during the session and will update the corresponding files locally and remotely in the background. It uses a small helper binary that is injected into the target container to accomplish this.
The algorithm roughly works like this:
Inject a small helper binary via kubectl cp into the target container
Download all files that are not found locally
Upload all files that are not found remotely
Watches locally and remotely for changes and uploads or downloads them

[dsebastien]

I don't know if my use case is valid as I'm just starting to use K8S, but I currently have a multi-stage docker build for two micro-services which need to be deployed to K8S.

I'm using Minikube and plan to deploy the "dev" stage of my dockerfiles to K8S. That "dev" stage's goal is to provide me with an environment in which I have all the necessary dependencies and tools to build (node, npm, etc). As such my goal is to develop, build and run from within K8S.

This setup was working fine with docker + docker-compose, where I could bind-mount my local source code into the containers and have it sync both ways.

At the moment I'm looking at solutions for this with K8S, but have found quite a lot of different things so far.. :)

Skaffold looks really appealing for multiple reasons, but at the moment I'm thinking about combining skaffold and devspace to reach my goals. If Skaffold could do it all, then all the better :)

[paolomainardi]

Maybe I am missing something, but should’t be easy to just using an hostpath volume as docker does for local development ?

[maluio]

@paolomainardi
Mounting the volume using a hostpath works.
However, I run into a lot of problems due to user id mapping between the user in the container and the user on the host system. There is a way to handle this mapping properly, but it is IMHO non-trivial to set up https://docs.docker.com/engine/security/userns-remap/

In case I am missing something a nudge into the right direction is highly appreciated.

[EnziinSystem]

Two-way file synchronization is really necessary.
In development mode, we often have to create models through scaffold programs such as migrate files, controllers, etc.

[tstromberg]

Two-way sync can be especially tricky with remote clusters, where reverse mounts are not easily done.

I haven't heard about anyone actively planning on working on this, so I'm going to downgrade the priority. If someone wants to propose a design document, I'd love to take a look.

[EnziinSystem]

@tstromberg

Two-way sync can be enabled locally for development mode, not a remote cluster.

For example:

I am a developer Ruby on Rails, I need to generate a model.

rails generate scaffold Post name:string title:string content:text

Many files generated in the pod/container, but I cannot copy them by the manual.

[pierreyves-lebrun]

Though tricky, two-way sync is essential as people developing in containers simply can’t use Skaffold at the moment.

Solutions such as Okteto or Devspace do have that feature so developers don’t have to pollute their local machine with dev dependencies.

[simonoff]

Guys! Such feature is a must have for full local development inside Kubernetes. I have reviewed the ways how to implement it - and only one way looks is better for me - inject a sidecar container which will do such sync. In similar way how the devspace doing it. What do you think?

[tstromberg]

@simonoff - Personally speaking, sidecars seem like a very reasonable approach.

[MattShirley]

This feature would also be helpful to me.

My scenario is similar to the other posts. I have a Python/Django application and want to generate database migrations, which requires an active connection to the database.

Similar to other requests, this is only important for local development on minikube and is not needed for remote clusters.

[ncri]

The problem with the kubectl cp is that we need to give the pod name, which is dynamic.

We use selector labels on kubectl cp (-l app=myapp)- so the shell script is reasonably generic.

@wstrange How would you do this? Do you have a full example? I don't see the option to use a label when in the cp examples (kubectl cp --help)

[wstrange]

The problem with the kubectl cp is that we need to give the pod name, which is dynamic.

We use selector labels on kubectl cp (-l app=myapp)- so the shell script is reasonably generic.

@wstrange How would you do this? Do you have a full example? I don't see the option to use a label when in the cp examples (kubectl cp --help)

Apologies - our shell script gets the pod name first using the label selector (in our case there is only pod in dev) - then we invoke cp

[ncri]

@wstrange Ah, got it, thanks.