Themekit: Detecting remote changes to theme

[monting]

One of the biggest pain-points when I work with a Shopify client is:

I have my own deployment process using the CLI, but clients want the flexibility to make changes in the Shopify admin as well. When they make changes and forget to tell me, their changes get overwritten when I deploy. The other way is to download the whole theme and merge the changes into my version control system.

Could we get a feature where we get notified of remote changes to the theme? Maybe check the timestamp of the last remote update compared to a timestamp of last upload kept locally? Ideally we'd be able to know which files were updated remotely.

Thanks for all your work!

[tanema]

Right now what shopify people do is download and find the changes with git. How would you expect all of this to work? If I warned every time the updated files have different timestamps it would be terrible for themes being worked on by teams.

[monting]

@tanema Yea that's exactly what I have to do now - download the whole theme and use git. But downloading the whole theme each time takes a long time and wastes API bandwidth.

What about a command "status" that shows the updated_at attribute of the theme, and the API key/user that made the change? That would be enough for me to script what I need around it.

If we can run that "status" command on individual files, to get that file's list of its older versions + timestamp, that would be amazing.

Again, thanks for your work, and humouring me on discussing this 😃

[tanema]

@NathanPJF @macdonaldr93 Do you have any feelings about this functionality? What I am thinking is maybe a --strict flag on upload/replace that will log an error when the dates are different between client and server. The main problem with this is that editing on the server isn't the only reason file dates would be different. They could be different just because people don't always run watch when they change all files.

[illarionvk]

Tracking the theme file changes with Git is the answer.

I always download the theme before starting my work and before deploying. This way I can see if the store owner (or an app!) made any changes.

[tanema]

Thanks @illarionvk I appreciate extra input!

[macdonaldr93]

@tanema it feels a bit weird to extend themekit in this way. It's kinda like once we start down this path, we'd have to keep going for it to feel complete which, imo would mean way better git integration for themes and that's another beast to tackle in of itself.

@monting do you usually have the admin open too? I have an internal chrome extension I use that adds the theme id and the last updated information to the theme cards in the admin. Maybe I could look at pulling that functionality out into a little chrome extension and make it public. No promises though haha just an idea.

[monting]

@macdonaldr93 I agree with pretty much everything you said.

I do think though, that being able to check versions/timestamps/previous uploads of a particular file seems to fit into themekit.

E.g. theme versions templates/somefile.html.liquid would list previous uploads with their timestamps. With this feature, I could keep track of timestamps and check for timestamp mismatch in my deploy script.

About the extension - it sounds interesting, but more granular, file specific information would be much more helpful. If I wanted information about the whole theme, I could get it through the shopify api, and then download the whole theme if there's a mismatch. This is kinda my backup plan 😆

[calebl]

Something like this would be really helpful for our deployment process. It seems like it is pretty standard for our clients to be modifying things in the Live theme while we are working on code changes locally. Doing a full theme download every time is feasible but doesn't seem like the best solution. Some of our clients have themes that are over 1 GB in size at this point. Being able to quickly diff the changes between my local copy and the copy in Shopify would be hugely helpful in giving me more confidence in deployments not overwriting something.

Some standardized way to integrate with the Shopify versioning system would be awesome.

[tanema]

@calebl What about download only downloading files that do not exist or files that have been updated?

[calebl]

@tanema That would be a good start. There is still to possibility that I've changed the same files locally which would necessitate a merge of some kind. It would be really easy for my local files to get overwritten without me realizing it unless I followed a pretty strict process:

1. make sure I have committed everything locally
2. create a new branch
3. download new file changes from theme
4. merge branches

You can't cover everything, but I would recommend having a confirmation of some kind at least with the download showing the files that will be overwritten to make sure the user knows.

In the ideal world, what I want is a git branch that automatically updates itself from the Shopify theme that I can merge from at well. Ha. This would be pretty simple to setup if theme download only downloaded the difference between the local copy and the copy in Shopify.

[illarionvk]

I still think this is beyond ThemeKit's scope and makes things too complex. In the end, it's down to a strict development process with version control.

My process:

• Duplicate the live theme
• Make changes to the duplicated theme using default development environment
• Track my changes in Git
• Download the live theme using production environment
• Merge client's changes, if any
• Deploy and publish

P.S. I was able to reduce the overall theme size greatly for my clients by using the new "image_picker" setting type.

[monting]

@illarionvk That's exacly what I did (but I essentially stopped doing it, because it wastes too much time, and is error prone).

I'll give you an example where that process you mentioned doesn't scale. For my client, they have multiple Shopify Plus stores, each serving a different country. All the stores are "compiled" from a single codebase, by a deploy script, allowing for customization of each country's store.

In this case, merging is a pain and error-prone, and I'd have to download from multiple production environments.

Also, it sounds like the frequency with which you deploy at isn't that high? For my client, we sometimes deploy multiple times daily, to multiple stores. Downloading the whole theme each time is excessive.

[calebl]

@illarionvk While I agree that some of what I want is beyond the scope of Themekit, being able to only download the differences between my local folder and the files in a Shopify theme does not seem beyond the scope to me. It seems like it would be beneficial to everyone to cut down on downloading files unnecessarily.

[illarionvk]

@monting That's exactly what I meant when I said it's beyond ThemeKit's scope. 90% of ThemeKit's users will never reach your level of complexity.

@calebl I agree, it might be beneficial to download only the difference between local and remote files. But in this case I'd like to have a good difference check too to avoid uncertainty, and that would be costly to implement.

[monting]

"90% of ThemeKit's users will never reach your level of complexity." - No argument from me there.

But I do believe that Shopify is heavily investing in its Plus/enterprise product, so I think it makes sense to address the tooling needs of enterprise-level customers, whether that be by Shopify or the OSS community

[tanema]

I would love to find a solution that at least alleviates this issue because it is apparent that this is a situation that most theme developers encounter. I am definitely going to keep this VCS independent and hopefully alleviate this situation for most. I will have to experiment to see what I can do because I am not really sure if VCS's like git have effects on file timestamps when you change branches so I may not be able to depend on file timestamps.

[tanema]

Okay so there are a few things about this not working very well.

• I know git changes file timestamps. This means that any time you checkout another branch it will change the timestamps.
• There is a problem with just checking the difference of time stamps because it would require you to be strict about checking them before you edit anything which is basically more of a hassle than just creating a branch, downloading and merging. Which is what you would have to do anyways if you forgot to download before you started editing the theme.
• I could list the assets changed time from the server but not the user who made the change because it is not available through the API (and potentially sensitive data that would not be available to all users).

With all this in consideration I would say you are better off just adding file filters (*.mp4, *.png, *.gif, ect...) so that your largest files that are unlikely to be edited are not downloaded in the process.

I could also add an extra list command that could list the info like @monting originally suggested but I could make it so that it could list themes as well as assets and their updated time. This may be possibly useful but not sure how useful.

[sdn90]

I don't have the same problem as OP but I'm interested in having some kind of user extension feature. I haven't really thought it through completely, but what do you guys think about something like Git hooks?

Example:

1. $ theme upload templates/page.liquid
2. execute some shell script
3. script returns 0 to continue uploading, else exit

This is my use case:

1. $ theme watch --env=feature-branch-1
2. file changes
3. only upload if the current Git branch is feature-branch-1

[calebl]

@tanema It seems like this is not an easy fix. I continue to run into issues though, even if I follow the approach outlined by @illarionvk. When I do a full theme download, git puts the file updated at time as the download time instead of when the file was actually modified. This means it's possible that my local changes (which happened later) do not get merged in properly, meaning I have to be very careful at this point that I'm not losing my changes. Maybe there is no good solution that you can implement to help with this. So far it has forced me to a lot of copying and pasting instead of merging, which means I have a lot less confidence that things are correct.

Is there a way to get notified by the Shopify API when a file has changed? If so, we might be able to use that to keep a local branch up to date with the theme in Shopify.

[illarionvk]

As far as I know, Shopify API has notifications only for theme-level events:

Webhook API reference

[calebl]

Has anyone worked with the theme/update webhook before? It looks like the only info you get back is the about the theme itself, nothing showing what was updated in the theme.

[tanema]

I would say from developing with these webhooks that you can not make anything time dependant on them and they don't always show up. Also themes/updateis only meant to convey that the theme was updated and prompt you to pull down the theme.

[monting]

@tanema how about instead of timestamps, keeping locally, a SHA hash for each file that's uploaded, and when downloading a file, you can check the downloaded file's SAH hash against the local hash?

[tanema]

@monting That is a good idea I will have to talk to the API people though, I am not sure they will love the idea but we will see.

[tanema]

APIs people say no but I have reported the problem to the Theme's theme and they are thinking on it. The main problem with adding the checksums would be that they would need to be added on file creation/update. So for it to be fully supported it would need to be backfilled on all the assets on shopify which would be a monumental task.

[monting]

@tanema yeah having the checksum on the backend would be ideal, but clearly monumental. While less than ideal, it can all be done on the client-side.

E.g.
theme upload --strict templates/x.liquid templates/y.liquid would download the 2 files from remote, compare their checksums with locally saved checksum of previously uploaded version of the files, and then error out if they don't match. If they do match, upload the file, and update the locally saved checksum of last uploaded version.

[frehner]

Great discussion. I've also ran into this issue and have had to do some not-so-great workarounds to make sure I'm not overwriting admin changes when I deploy new code:

• We keep ALL (even product page text) in the locales/en.default.json file*.
• I develop on a different store so that I can make sweeping changes without affecting the live store, and then deploy the whole theme package to the live store.
• What this means is that I can't allow my team to update the en.default.json file on the live site, for fear of overwriting their changes when I deploy new code. They must make text updates on the development store and thus it's part of my deployment to the live store. It makes it so that they're dependent on me deploying code to update text.
• This also applies to using the new Shopify Sections feature, since that is also stored in JSON files in the theme config.

I know this is a really non-standard setup, so I haven't complained too much. But if this could somehow be figured out, it would really help me and my team out a bit.

*The reason for this is because each product page has a completely different layout, since we have a small amount of products that we want to be able highlight unique features. Doing something like this just isn't possible, as far as I can tell, by just using the product description box in the admin.

[illarionvk]

@frehner If you are tracking your changes in Git, you can setup a second environment in your ThemeKit config for your live store and use it to download your team's changes.

Download your team's changes before deploying, then merge them with your code using a diff tool.

A good diff tool would help a lot, please take a look at Kaleidoscope, Changes, or DeltaWalker.

[frehner]

@illarionvk thanks. I am tracking the changes locally in git, so that part is good. The only issue I have is the following situation:

1. I pull down their changes
2. I validate it, diff it, commit it, and and merge it into master
3. While that happens, they make more changes
4. I deploy to the website

In this case, I've just overwritten their latest changes and no one may notice for awhile.

I didn't want to have to deal with that sort of situation, so that's why I'm doing what I do now.

But thank you very much for the suggestion! It's got me thinking again on how to best solve this problem.

[tanema]

I am talking to the API developers about exposing some sort of checksum still. Hopefully we can make themekits operations safer.

[chryton]

@frehner What we do with our clients is give them a changes blackout window where content is "frozen" while we stage and deploy. If something does need to go out in that time it is handed to us to integrate.

@tanema would a stop gap for checksums be the size attribute for assets in assets.json? I know it wouldn't be nearly as full-proof but it could be put behind an --experimental flag for the time being vet _how_ this would work.

[frehner]

@chryton Yeah, it was either do it that way or do it the way I currently do it. I decided that doing it the way I'm currently doing it is better for how my team works, but both cases are not ideal.

And for the moment, I can't think of how this could be better handled -- that is, how to allow a team to update the locales' and section's .json files and deploy new code/changes without overwriting what they did.

[hdytsgt]

Hey guys,

I believe checksum attached to an asset would be a pretty great addition to the Asset API. We built an app (Motifmate) that already support detecting remote changes and display the difference to avoid overwrite (Diff Viewer).

Although we think this is good enough for now, we feel this workaround is still expensive as we need to download the remote asset first.

+1 for exposing some sort of checksum!

[tanema]

Hey all I thought I would give you an update on this issue. Since the theme API devs already have a custom checksum computed for shopify assets, what we are looking at doing is sending this checksum out with the file data as a version attribute. So when updating a file, the checksum can be passed back to the server and it will respond with a 409 if the file is different than the file you are expecting to update.

This means that with theme kit I will be able to

• save the checksums (somewhere I have not decided yet, open to input)
• on upload and watch I would pass the checksum along with the update information.
• if the file is newer than the one we are expecting (we receive a 409 response) then I have two options
  
  
  
  • prompt you to overwrite.
  
  
  • display an error prompting you to re-download the file
  
  
  • re-download the file overwriting your changes
  
  
  • [extra hard] try and merge the two before uploading again
  
  
  • save a .diff file for inspection of differences and let user resolve issues.
  
  

Any feedback on this process would be appreciated and very helpful! Cheers!

Extra Notes It was decided to do it this way because the sums already existed but using a custom hashing function so users cannot compute the sum themselves. Also this allows for future stability if the computation of the checksum changes.

@hdytsgt I hope this will also enable motifmate to have better file change protection as well, please let me know how this process sounds to you as well.

[EDIT] Added an option

[chryton]

I think option #2 of displaying an error and prompting to redownload might fit the most use cases since we will need to consider build tools and processes like Slate and how that might affect their output.

[tanema]

Yes I agree however if you are using watch you definitely do not want to stop the command, run download then start watch again. The best option may be that I implement a few of these behaviours and have a flag to decide which should be used.

[chryton]

True, flags may be best. That way at least a user get the flexibility to do what they need.

I know our development process at Rocket Code does not use themekit's watch so while it'd be inconvenient, it would be a nice safeguard to have available if it isn't the default.

[illarionvk]

Great news! I would prefer a no-prompt fail action that logs a warning and outputs .diff file, because I might want to keep my Git working copy changes.

[tanema]

I thought I would give you an update on this process. I have a WIP PR open that is starting to work pretty well and should have a beta open soon. Please give me your feedback on the current design, you don't need to review the code just the design notes laid out in the PR. I think I have struck a pretty good balance on how to handle the errors for this as well as added some optimizations to themekit as a whole as a result of this work.

[allthetime]

I read through some of this thread but just want to poke in to say, at least it would be nice if the download only downloaded files that have been updated instead of downloading the entire theme every time. Simply checking against a timestamp would be enough wouldn't it?

Even during the dev process by myself, sometimes it makes more sense to make changes in the shopify console than in code and it's pretty annoying to have to redownload the entire thing for almost nothing.

[illarionvk]

@allthetime If you know which particular files were changed, you can download them individually. For example, run this command to download two files only:

theme download templates/product.liquid snippets/header.liquid

[tanema]

@allthetime This work is in progress on the open WIP PR. "checking against a timestamp" is non trivial when VCS systems come into play because when a branch is checked out, all of the files times are changed. So the only way to do that is to keep a record of all the times which I was not willing to do to just speed up a single command. However now that I have implemented version checking which creates a file to keep track of versions, this enables me to speed all commands that interact with the shopify servers.

[chryton]

On a first pass the design looks solid. A couple of questions:

• Is the "Serialized themekit data format" an example of what is going in the theme.lock? If not, what are you planning on having it look like?
• Would it be helpful for the preflight output to show the date/time for updated files?

[tanema]

@chryton Yes that is what the contents of the theme.lock file look like. And yes I can add the timestamps to the diff that is a pretty good idea.

[chryton]

And the timestamps will be in server/Shopify time, correct?

[tanema]

yes

[tanema]

Hey all 👋 . We have reached a reasonably stable state, If anyone would like to help beta test this feature you can update to the release candidate by running theme update --version=v0.0.0-rc0.7.0

There is still one main pain point to be aware of but I am not sure how often it is actually encountered. The problem is that flags and environment variables kind of change with this version. Since the command will now load every environment to check asset versions, the flags and environment variables will get applied to all the environments making some values not make sense. For instance it would not make sense for a password to be applied across all environments. Flags and env vars may have to be applied to active environments only. There is also a question if I should throw an error or just log a warning about not being able to load an environment for checking. I would love feedback on how often you use flags like --password and --themeid on commands like upload/replace/watch

Thanks for all of your help so far on this issue!

[tanema]

Hey All, the changes have been merged and will be deployed sometime this week. I just want to make sure that everything is functional. So please make sure that you try out the sometime soon and report any issues.

[BrunoBoehm]

Hey thanks for this improvement. Has anyone written a post about how to use several environments and the new theme.lock function? The theme.lock page on github is full of information but an example of workflow would be welcome ;)

[tanema]

@BrunoBoehm I will contact the Shopify documentation team about helping me with that soon.

[BrunoBoehm]

Ok @tanema ! So from what I understand what you recommend based on themekit configuration page, is to have a dev environment (as an inactive theme - ignoring gifs, jpegs and settings_data file), a production environment (live theme), and maybe a test one if needed.

How do you recommend to push changes since the production env is "readonly" ?

• Manually switching the dev theme to live theme would also switch settings_data.json...? not great
• Removing manually the readonly and then doing a "theme upload" command? Would it only update the files based on theme.lock ?

Thanks in advance for your advice!

[tanema]

@BrunoBoehm that is not a recommended configuration, that is an example config with many options to show you examples of how to use them.

The idea of the read_only attribute is to make it actually a task to remove it from your config so you have to be very deliberate about deploying to that environment.

[BrunoBoehm]

@tanema it's indeed a good safety. So I should remove "readonly" manually, and do a "theme upload", then put "readonly" back on ?
Would the "theme upload" cmd only load the files that have changed according to theme.lock ?

I used to switch the dev inactive theme to live theme, it's not optimal + erases settings_data.json. Have another idea ?

[tanema]

@BrunoBoehm please contact me at tim.[email protected] for further support on this.

[psmod2]

Is this implemented? I don't see any notes on the documentation on how to use it? Or should it work by default?
I updated a page on Shopify Admin, then in local development and didn't see any warning?

Thanks.

[tanema]

@psmod2 This was removed because it was causing inconsistencies and difficulties. https://github.com/Shopify/themekit/issues/540