Tilt: Snapshot does not work / unexpected character / 500 internal server error

[FFdhorkin]

Clicking the "get link" button doesn't do anything. I do see this in the logs when I click it, though:

Error creating snapshot: posting snapshot failed. status: 500 Internal Server Error, response: error creating snapshot

Also noticed this in my console:

SyntaxError: "JSON.parse: unexpected character at line 1 column 1 of the JSON data" HUD.tsx:195:32
sendSnapshot HUD.tsx:195

[victorwuky]

Thanks @FFdhorkin for logging this!

[victorwuky]

That UI is also not very friendly since there isn't immediate feedback. We should add some type of loading UI as the snapshot is being generated. The bug here may be related to problems with load / too many services. And we need better designs to help the user and gracefully degrade the experience.

[FFdhorkin]

That UI is also not very friendly since there isn't immediate feedback. We should add some type of loading UI as the snapshot is being generated. The bug here may be related to problems with load / too many services. And we need better designs to help the user and gracefully degrade the experience.

I'm skeptical about it being load or too many services, but it's possible. I'm trying to transition our local environment from docker-compose to k8s (w/ Tilt) to better match our production environments. Things aren't working yet, so the logs are mostly errors about some services not being able to talk to one another. (We've got roughly a dozen.)

Please let me know if I can help with investigation!

[jazzdan]

I think this is probably the problem where if there are too many logs it exceeds the GRPC message size limit in the backend,

[jazzdan]

Oh, I think @landism mentioned a while back that this is actually a limit to the amount of data that can be in one field in Firestore.

[andymartin-sch]

I think this is probably the problem where if there are too many logs it exceeds the GRPC message size limit in the backend

yeah this is not reproducible for us at the beginning of our tilt up runs but is consistently reproducible later on after the logs have grown in size past ~2M

it would be nice if the "take snapshot" functionality would grab the last N lines from each service such that it always stays under the limit

[FFdhorkin]

I think this is probably the problem where if there are too many logs it exceeds the GRPC message size limit in the backend

yeah this is not reproducible for us at the beginning of our tilt up runs but is consistently reproducible later on after the logs have grown in size past ~2M

it would be nice if the "take snapshot" functionality would grab the last N lines from each service such that it always stays under the limit

This repros for us almost immediately upon tilt up, but that's probably because our logs are very spammy. I've never managed to take a snapshot.

[victorwuky]

We are aiming to work on https://github.com/tilt-dev/tilt/issues/3128 in the coming weeks, and plan to work on investigating this bug and fixing it at the same time.

[FFdhorkin]

I haven't had a chance to work on getting Tilt working with our project since April, but I'm anticipating trying again next week sometime.

If you ping this issue when you believe you have a fix, I'd be happy to confirm whether the snapshots are working. (Private snapshots also are a must for us, so these two happening at the same time is good!)

[victorwuky]

All: Thanks for participating in this issue and reporting the problems. We are now working on a fix. In the meantime, the latest release now will surface an error if snapshot creation fails. Previously it was failing silently as noted in this issue. See https://github.com/tilt-dev/tilt/releases/tag/v0.14.2 and https://github.com/tilt-dev/tilt/pull/3400.

We will keep this issue open until there is a full fix for the problem. Thanks!

cc @jazzdan

[jazzdan]

This should be fixed now!