Themekit: Theme watch stalls when more than one file is updated quickly

[stewartknapman]

Describe the bug
When several files are updated too quickly theme watch will stall and stop sending files.
This usually happens when running a build task that adds or updates multiple files in the themes assets directory. This happens intermittently, but enough to be frustrating. It will not usually happen until you've updated these files a couple of times.

To work around this we need to stop theme watch and then run it again.
I'm not sure which version this started happening in bit it has been happening for maybe the last 9-12 months.

To Reproduce
Steps to reproduce the behavior:

1. run theme watch
2. run build task i.e. npm run task that updates multiple files. This is usually watching and/or compiling multiple JS files with browserify or similar.
3. update any files that cause theme watch to push files to Shopify.

Expected behavior
Theme watch should not stall.

Environment (please complete the following information):

• ThemeKit 0.8.1 darwin/amd64

Additional context
Here is a video of the issue: https://monosnap.com/file/DMIQCidYU3vuLPGIqIljvktYELAqek
One thing I missed in the video is that once it's stalled it's not just the JS files that won't get pushed, any liquid file (snippet, template, etc.) will also no longer be pushed up once it's stalled.

[tanema]

I think this might be because of this fixed issue here: https://github.com/Shopify/themekit/issues/577 It is likely that themekit is getting stuck while you are trying to upload a specific file because of that bug. Please try updating to the prerelease and try again by running theme update --version=v1.0.0-pre

[stewartknapman]

@tanema Cool, I'll give it a go shortly and report back.

[stewartknapman]

@tanema After an hour or so I have not seen it stall once, it all seems pretty smooth. However, it does look like it uploads some files twice but only those added by the build tools, and not always.

[tanema]

are the files that are being uploaded twice, plain js, or css files instead of liquid files? Once they were uploaded twice did it happen again for that file? I have a suspicion that themekit is just retrying the file after it found that it needed to clean up a compiled file.

[stewartknapman]

This was one save: https://monosnap.com/file/7voEgFhZ6Enuw3NVFCrbZsmdgX1Few
They are plain JS files. There is a minified and non-minified version of each. Some of them (in this instance the non-minified versions) get uploaded twice, but not all of them.

[tanema]

@stewartknapman and this did not happen on older versions of themekit? This would make sense since the file watching back-end has changed in the newest version. I will have to see what I can find out about it. I am not sure how one save leads to this though, I would expect that only a single minified file would be outputted from your pipeline when a single file is saved, and themekit doesn't make up new events, how are all these other files involved? (The answer maybe be obvious but I am not exactly a front-end specialist so my apologies)

[stewartknapman]

Correct, in onther versions of themekit I have not seen the same file added from the pipeline uploaded more than once.

I mean it’s possible that the npm script is triggering multiple events when updating the file? I assume an update is a removal and then a creation. A couple of times yesterday I’d reload the page and nothing (from the file I was working on) happened. It’s as if it uploaded an empty file... which when I think about it feels familiar. I have seen that before but a while ago, maybe it was with the old theme gem?
I don’t know, maybe that’s not helpful.

Also to clarify the minified and non-minified versions are expected. We do this because I’ve inherited new clients and themes with minified code and no src which is painful to work with so we add both to the theme so anyone picking it up can use the non-minified version if they really need to.

Oh, also also, my npm scripts will run all the tasks when a single file is saved, outputting a bunch of unrelated files. It’s not ideal and something I need to look into.

[tanema]

Okay thanks for the info, I am looking into it. This will help the rollout of v1.0.0 go a bit smoother as well so it is much appreciated

[stewartknapman]

@tanema I've cleaned up my build scripts so that they now only output the relevant file on a save. It looks like it is theme kit that is sending multiple copies of the same file: https://monosnap.com/file/HBWodei1q2sEOy9IEd933OaCDuG2nI

But again this is intermittent: https://monosnap.com/file/a7QAvzUXncEuOCw1e4y3MUWgQ1DT6c

[tanema]

Yeah, it seems that I need to debounce events coming from the new watching backend. I was under the impression (from their docs) that they already did this, but here we are. Can you give me an idea of your setup? (webpack build or whatever lib you are using) If I can fully reproduce this myself, I make sure this will not happen to you or others in the future.

[stewartknapman]

Yea sure.
I'm using npm scripts, I've put the tasks and dependancies that I use in a gist here: https://gist.github.com/stewartknapman/9cd36b2783b8d67f4cde245af18d20ec

Let me know if you need anymore information.

[tanema]

Okay excellent! I was able to reproduce it.

[tanema]

Okay this was fixed by #601 You can run theme update --version=v1.0.0-pre again and it will install the most recent version of the pre-release.