Tilt: Copy files back from container to local fs (or: volumes via Tilt) (or: two-way file sync)

[maiamcc]

Some people want files changed on the container to be copied back to their local filesystem; e.g. running yarn update on container and copying back the updated yarn.lock file so it can be git commit'd).

E.g. Docker Compose supports this behavior via volumes: changes made to the directory on the container get propagated back to the directory locally, and vice versa.

[maiamcc]

Requested once upon a time by Picnic.

[nicks]

it's worth pointing out that this issue is describing the same problem as https://github.com/windmilleng/tilt/issues/1688

i.e., a workflow for lock files (yarn.lock, Gemfile.lock, or whatever)

[hyu]

Addressing this issue can solve an issue raised by Tamer Hassan in #tilt (2019-08-13). See https://github.com/windmilleng/tilt/issues/1688

[ostrolucky]

Addressing this issue would also solve an issue at https://github.com/windmilleng/tilt/issues/2756. Bear in mind though that in that case project source code decides when to generate files, so it's not possible to workaround this by creating wrapper command around composer, yarn etc which would copy files to host after main task is over.

[jazzdan]

To expand on what @ostrolucky said: symfony does runtime codegen to create a routing table (basically a highly optimized regex) that you can interact with over the symfony CLI tool to like list all routes, list routes that match a url, etc

https://symfony.com/doc/current/routing.html#debugging-routes

That functionality requires the codegen’d file to exist (usually it’s in a place place /var/cache/symfony/routes.gen.php) and is used by IDEs to do smart things with routing

[ilourt]

Is this issue in your roadmap? If yes, do you have an idea of when it will be available?

[nicks]

@ilourt currently it's not on our roadmap, but that may change as we get more people asking about it, and more examples of problems that people want to solve. :smile:

Currently, Tilt has a local_resource feature that lets you run scripts manually from the Tilt UI. We're collecting feedback on how people use it. You could conceivably use it right now to, for example, run a script that copies yarn.lock to your local file system.

Based on feedback we've gotten so far, I think "two-way sync" encapsulates a few different features and use-cases:

1) Artifacts of a "clean" build (e.g., your docker_build generates a yarn.lock, and you want to use that as an input to subsequent builds)

2) Artifacts of a one-off build step inside a running container (e.g., you want to upgrade a dependency by running "yarn update" in the container, and then copy out the lockfile)

3) Artifacts generated at runtime (e.g., you want to interact with a UI that generates some artifacts, and then copy out those artifacts, e.g., Symfony)

I don't think there's a concrete proposal on the table yet, or a clear idea of how that proposal would weigh these different use-cases.

[landism]

Aside from manually syncing files out, a solution we've used is to have a local_resource run the build in a container using toast. Toast gives you a simple way to run some commands in a container and copy some files back out.

e.g.:

in the Tiltfile:

proto = str(local('find shared/proto -name "*.proto"')).rstrip("\n").split("\n")
local_resource('proto', 'toast', deps=proto + ['toast.yml'])

in toast.yml:

image: myimage
tasks:
  proto:
    input_paths:
      - shared/proto/foo.proto
      - shared/proto/bar.proto
    output_paths:
      - shared/proto/foo.pb.go
      - shared/proto/bar.pb.go
    command:
      - cd shared && ./protogen.sh

[mouzfun]

To me this feature seems essential to local development with docker, ability to sync changes/dependencies from containers with right permissions and paths is required for local IDE to function properly, and IDE without autocomplete is not much more than vim.

local_resource hack above seems sufficient for now, but that it what it is - a hack. Seems like sync_back like action would be much appreciated.

Dont get me wrong, this is not a complain or a damand. merely a suggestion. Thank you very much for making tilt open source and for it in general, you are doing a very good job :)

[ajkamel]

Similarly we are experiencing the need for this so that we can use the deployed version of our package manager to install the correct dependencies. Since we hope to develop entirely remotely this is problematic in that we would have to run our container locally or install the correct version of our package manager to generate the lock files we need for our application dependencies.

For example. If I run npm upgrade on my pod. Can I have those package.json and package-lock.json files sync back to my local filesystem in order to be committed to code?