Tilt: short-term fix for custom_build caching semantics

[nicks]

All Tilt image builds (docker_build and custom_build) use content-based immutable tags. In other words,

• Tilt builds the image
• Tilt checks the contents of the image to compute a digest
• Tilt injects the digest into the kubernetes yaml. Kubernetes is smart enough not to redeploy the image if the digest hasn't changed.

For details on this, see:
https://docs.tilt.dev/custom_build.html#why-tilt-uses-immutable-tags

There are two ways this can fall down.

The first is that if the custom_build builds remotely, Tilt doesn't have any way to compute the digest of the output image. This only affects builds that use custom_build(skips_local_docker=True)

The second way is if the custom_build has its own caching mechanism, not based on content hashing. For example if it wants to tag the deployment in some way, and always reconnect to the deployment if the tag matches what it expects.

I can think of a few different options to fix this, based on what stage we want to do the caching:

1) Compute a digest of inputs, as suggested here: https://github.com/tilt-dev/tilt/issues/3690. This puts all the implementation on the Tilt side, but weird things might happen if the user has specified deps wrong.

2) Provide a way for custom_build to specify the digest of the output image. Then Tilt could compare the digest with what's currently deployed, and skip deployment if the digest hasn't changed.

3) Provide a way for custom_build to do its own caching check. i.e., tilt passes in the existing digest/deployment, and the custom_build says "yes, i want to reuse that one!" This might run either pre-image-build or post-image-build

Note that these options aren't redundant - it would be reasonable to implement all of them. (3) is probably the biggest footgun -- a buggy user script could really corrupt your dev environment in weird and difficult-to-diagnose ways. The exact semantics and API might be hard to get right - i'm not totally sure if pre-image-build or post-image-build is right. (2) is probably the safest, but I think it would be hard for users to implement well -- it would be more as a building block for other remote builders.

[nicks]

I feel like this is super painful for some teams when it happens. We'd prefer a short-term fix that we can iterate on rather than trying to solve for everyone the first time

[victorwuky]

@joseph-zhong @calvinytong: Do you mind reviewing this and seeing if it makes sense?

[calvinytong]

This looks reasonable to me. I think 2 would be enough to unblock us.

Also we might be a little weird, but what we're really just looking at a single file (our conda env yaml). If that changes we want to rebuild otherwise we'd rather just reconnect and live-sync if possible. Not sure if that further simplifies things.

For more context we're using custom_build(skips_local_docker) with a custom built remote docker build service

[victorwuky]

Thanks @calvinytong

[nicks]

@calvinytong thanks! let me try to do something quick that does (2) and see how that works.

quick question: if you're only looking at a single conda env yaml to compute the digest, is there a risk that there will be a mismatch between the local files and what's in the existing deployment? will you also need a mechanism to make sure all the local source files are live-synced?

[joseph-zhong]

I don't think there isn't a risk for a mismatch between local files and the deployment since we rely on the live-sync for all our files -- what @calvinytong is referring to is that conda-env yaml tracks third-party versions and other dependency packages which would require a rebuild (conda install ...) to include them

I'm not exactly sure if I fully understand your second question: There are instances where a file is accidentally deleted in the deployment, and the local file isn't changed, so the only way to re-sync is to touch the local file to trigger a live-sync.
But yes a mechanism that ensures the local sources are live-sync'ed such as a manual "live-sync files listed in the Tiltfile" could be useful in this scenario, our Docker context is ~50-75MB so it's not a super large cost to upload

[nicks]

i'm going to add
custom_build(outputs_image_ref_to='./path/to/file.txt')

And if that is set, Tilt will read that file after the cmd runs to get the image ref, and inject that image ref into the YAML.

The underlying infra will assume that if the ref doesn't change, then the pod doesn't need to be restarted.

[joseph-zhong]

Awesome I think that would be great to let us get started on prototyping, let us know when it's in either a release or even if you want us to prototype with a branch we could even start with that, whichever is easiest, thanks for the quick turnaround Nick!

[nicks]

@joseph-zhong Here's a branch with a basic implementation:
https://github.com/tilt-dev/tilt/tree/nicks/ch9358
if you don't mind building from a branch, so you can see how it works with your build script

[calvinytong]

We're prototyping a solution right now. We want to implement graceful reconnects per branch, so we're thinking a simple implementation like this works best for us:

1. Generates a tag of form {user}-{branch-name}-{short hash of branch point}
2. The build scripts checks the outputs_image_ref
   
   
   
   1. If the ref exists:
      
      
      
      
      
      1. Skip the build step
      
      
      
      2. Update the live sync
      
      
      
   
   
   2. If the ref does not exist:
      
      
      
      
      
      1. Run the build step
      
      
      
   
   
   3. Write the ref to the specified outputs_image_ref
   
   

What might be helpful is if Tilt had a way to "update the live sync". We're basically thinking of rsyncing the changes up right now, but not sure if that plays well with how tilt is configured right now.

[nicks]

@calvinytong

Ya, I think that's a good solution in the short term -- connect to the existing deployment, and rsync the files you need. Happy to give pointers if you need help.

Ya, "updating the live sync" in a general way is tricky...custom solutions that can exploit specifics of your setup are easier. The problem we run into is that if the pod crashes or gets moved to a different node for some reason, Kubernetes loses the file-system state...and we have to deal with race conditions if the pod starts before we can get to it...

• there's currently a Tilt extension https://github.com/tilt-dev/tilt-extensions/tree/master/file_sync_only that uses local_resource to sync files that works in some narrow cases

• I've been playing around with the idea of a live-sync system that syncs to Nodes (rather than Pods), then mounts the folder into the container https://github.com/tilt-dev/tilt/issues/3282. But mounts have their own gotchas (since the mount is the whole directory).

Do either of those sound like they would work for the way your pods are set up?

[nicks]

outputs_image_ref_to should be out in v0.17.5! just sent out a pr adding it to docs.tilt.dev

going to leave this open, because i suspect we might need additional iterations (though it also might make sense to make those separate issues as we talk more)

[calvinytong]

Awesome! Thanks @nicks. We're working on a graceful reconnect implementation right now. Will keep you updated

[calvinytong]

Just did a first implementation and it's working great, but still haven't done the "update the live sync" step. I'm thinking the cleanest way to implement it would be with some kind of "post-deploy-hook" where we could run a script after Tilt marks the pod as deployed and ready to go. The flow we would have to reconnect would then be:

1. Generates a tag of form {user}-{branch-name}-{short hash of branch point}
2. Check the registry if the tag exists
   
   
   
   1. If the ref exists:
      
      
      
      
      
      1. Skip the build step
      
      
      
   
   
   2. If the ref does not exist:
      
      
      
      
      
      1. Run the build step
      
      
      
   
   
   3. Write the ref to the specified outputs_image_ref
   
   
3. After the pod is up and running rsync any fs differences to the container

[calvinytong]

I think we could probably use local resource for this.

[nicks]

ya, you can probably use local_resource to hack this together immediately, but it might have some gaps, because i think it will be hard to make the local_resource run reliably every time a new image build happens.

let me spitball some ideas

1) maybe a custom_build(img, on_container_running_cmd=...), where the cmd gets invoked everytime Tilt sees a new container with this image?

2) alternatively, a custom_build(img, live_init=[sync(), run()]) that has similar semantics to live_update, but runs on each new container? could lead to some weird pathological cases if a pod was in a crash loop, but might be ok.

3) have a way to trigger a local_resource in response to events (new pod comes up) and feed it some event data (like the name of the pod)? This starts to get into https://brigade.sh/ territory, but maybe a Tiltfile needs some better event-driven programming.

:thinking:

[calvinytong]

Hmm yeah, I think (3) would be a bit overkill for what we need. (1) or (2) would both work for this simple use-case. (1) would probably be easiest, but not sure what the best general solution would be

[nicks]

are there any race conditions we need to worry about?

Like, in the general case, there might be problems if the container starts before the files are initialized....

But maybe in your specific case, it's ok if the init files show up a few seconds after container start (e.g., if it's a jupyter server that you're syncing notebooks to)

[calvinytong]

Yeah, I think it would be cleanest to avoid these kinds of race conditions, but in our case it shouldn't matter too much.

[calvinytong]

I've implemented it with a local resource and it seems to be working fine.

One other painpoint is that we use a local registry for our CI tests. It used to be that Tilt would infer when we are using local registries and inject the correct repo, but now that we're using output_refs we've lost that functionality. It's not too hard to reimplement, but it is something that we might want to document.

[nicks]

nice!

OK, I'm going to close this, and if we need to do the live_init thing, we'll follow that up on a new issue.

@calvinytong i need to add this to the docs, but if you're using outputs_image_ref_to, and a local registry Tilt will inject the environment variable REGISTRY_HOST into your build script. does that work?

[calvinytong]

Awesome! Thanks, this is amazing