Themekit: Move API passwords to a global file

[maximevaillancourt]

I'm opening this issue specifically to spark a discussion. Feel free to pitch in!

Current situation

A theme's config.yml file contains both deploy environment details (store_url & theme_id) and authentication secrets (password). Because of this, the config.yml file is never committed to version control in order to prevent API password leaks. However, this also prevents multiple developers from sharing a common config.yml to specify deploy environment details that can be used from different machines.

Suggested solution

In order to both (a) allow multiple developers to use the same environments config file and (b) offer improved separation of concerns between authentication and environments, API passwords could be moved to a global file, such as ~/.themekit/keys, which could contain a list of key-value pairs binding a store URL to API passwords.

Example file content

my-testing-store.myshopify.com:6e58f821ebf850760bbca8ad1c28a514
a-friends-store.myshopify.com:1fd6205d985d02e03971940a7d08cf66
big-client-store.myshopify.com:9f1c877fe5c2c36dc149463e4152ae56

Discussion

Feel free to improve upon this solution or suggest something different.

Pings: @t-kelly @tanema

[chryton]

To clarify, themekit would parse the URL for the store in config.yml then match that to a key:value in the keys file, correct? I'm trying to think of a situation if having multiple stores in one config.yml file might cause a problem.

A potential issue I see is that for an agency that file could get very big, very fast.

[maximevaillancourt]

@chryton

Yes, exactly — ThemeKit would parse the store property of the requested environment from config.yml and find the matching store in the user's keys file to get the relevant API password.

About having different stores in config.yml, I don't see it being a problem. It wouldn't be any different than the way it is now with multiple different environments.

As for having multiple key:value pairs in the keys file, I agree that it could get very big, very fast if a developer works on multiple stores at once. Since it's just a hashmap however, access to the store's API key should be rather fast, and nothing stops the developer from removing ununsed pairs from the file.

[VladimirCatrici]

Wouldn't be better to allow store password option in an another file, e.g. .password or .env?

[denisinla]

You can also just setup ENV variables to your .bashrc with the required content and use them in your config.yml file. I've been doing this for a while

[chryton]

@denisinla that could get unwieldy if you are a freelancer or working at an agency managing multiple sites at once since then your .bashrc file would just be a bloated mess of keys. It also wouldn't be great if you're using CI to build, test, and deploy your theme because you don't really want environment data hidden in an rc file that has nothing to do directly with the build.

[tanema]

So this gave me an idea, and I opened a PR as an example (https://github.com/Shopify/themekit/pull/589) It will expand any environment variable in the config so that you can make your config a lot more flexible. This would allow to have a .env file that looks like

export SHOP1_PASSWD=0bwef09hn23048sdkl2345n2k3
export SHOP2_PASSWD=0bwef09hn23048sdkl2345n2k3

Config looks like this:

development:
  password: ${SHOP1_PASSWD}
  theme_id: "105994886"
  store: can-i-buy-a-feeling.myshopify.com
prod:
  password: ${SHOP2_PASSWD}
  store: can-i-buy-a-feeling.myshopify.com

The commands can be run like this

$> source .env && theme download

And then do not commit the .env file to your repo.

Edit:
I would just like to say that I am resistant to adding file needed for configuration to themekit in a users config path or something like that because I want themekit flexible enough for anyones workflow but I also want it easy for beginner to figure out. Instead of asking "Why don't I need to put my password in my config? Where is it coming from?" It would be a progression to, "How can I store my config without my password in it?" Plus if we can lean more heavily on standard *nix practices like environment variables, themekit will work better as a tool that people can integrate into their own workflows and tools.

[tanema]

@chryton @denisinla ?

[denisinla]

Perhaps off topic but docker-compose does this by default. The advanced Shopify devs would be able to make usage of this as an improvement to their workflows, especially usage with github repos(Who in their right mind would just edit their theme files on Shopify admin willy-nilly?). Those who are not comfortable won't need to use it.

[tanema]

I could easily add a .env functionality to themekit. Load it up before running any commands. I am worried about how this could conflict with other tools though.

Edit: The difference between docker and themekit is that themekit has to work within an environment and docker is the environment

[cameroncowden]

I support it being optional. I have run into a few use-cases where it would have been nice to have the password separate from the config file, but most times for me this would be an unneeded complexity. That being said - I have no problem committing config.yml to private repos. Keep API permissions to read/write theme files only and make sure you only give repo access to trusted parties. I actually have different config.yml files for each branch so that each feature development has its own staging theme. FWIW, my usual workflow is simply git checkout branch then theme open (to verify), then theme watch and start working.

[chryton]

Could there be a minimal amount of settings to avoid the conflicts added to the YML file for filename, path, and if the keys should be pulled in from an external source?

ex:

keys:
  external: true
  filename: .env
  path: ~/keys

development:
  password: ${SHOP1_PASSWD}
  store: dev.myshopify.com
  theme_id: 1234567890

[tanema]

@chryton I won't do that. I am not interested in any implementation that breaks that current schema of the standard config.yml

Currently in the POC PR I have experimented with just loading a .env file in the background silently, so that it won't worry or bother less advanced developers however I think I will add in a flag to specify a path to where the env file might be located.

[denisinla]

^ Yes this works for me. 100% optional and available via a flag.

[chryton]

@tanema Your solution works for me.

I still think there should be something in the config file rather than just the command line because changing the config file ostensibly makes it easier to update a build process: you just add a new line to the environment's config instead of changing the command in your toolchain. Something like this would still keep the current schema:

development:
  key_path: ~/keys
  password: ${SHOP1_PASSWD}
  store: dev.myshopify.com
  theme_id: 1234567890

[tanema]

@chryton that is actually a pretty good point, most of the time the path will not change and it would get annoying to input a flag for every command. I will have to think this over though because loading one key file would effect how every environment is loaded so it would be weird to only specify it on one.

[chryton]

Yeah, I thought about that it could get annoying but I could also see an argument that different environments might be different files (US store vs. EU store vs. Dev store). I guess it comes down to how configurable you think it should be.

[tanema]

I can only load one environment for running a single themekit instance unless I introduce minimal containerization which seems overblown. So if I load multiple environment files, all of the variables would be available from all of the environment files. Which, to add complexity, the environment files need to be loaded before the actual config so that I can appropriately process the variables within the config file.

There is another option to add a file like this to a generally accepted config path like ~/.config/themekit/env (%APPDATA/themekit/env for windows) and have everyone use that one config path. This would satisfy the original intention of this issue. However, the thing I like about this, over the original proposed solution is that you can use any variables meaning that you could commit your config file without exposing which clients you are working for as well.

[chryton]

Hmmm. So putting it in the ~/.config/themekit/env placement would effectively allow me to set up variables for all the environments, commit the config.yml file, and each machine could in theory have it mapped differently based on the file in the above path.

I think that would be a really solid solution: good for single stores and agencies. Keeps repos clean and also allows teams to set up a canonical config file where everyone can have individual keys/IDs/etc.

[tanema]

Okay, I have updated my PR with the new functionality that I have added, please give me any feedback you have on this functionality. And thanks again for you help in working towards a better tool!
https://github.com/Shopify/themekit/pull/589

[denisinla]

Make it so number 1.

[tanema]

Please try it out by updating to the prerelease: theme update --version=v1.0.0-pre

If you want to provide a variables file you can add it to the following paths
Windows: C:\\Users\\<User>\\AppData\\Roaming
Linux/BSDs: ~/.config/Shopify/Themekit/variables
MacOSX: ~/Library/Application Support/Shopify/Themekit/variables

It will also find a file named variables in your project directory and there is a --vars flag to provide a path to a file as well.

The variables files is like any .env file

DEV_PASSWD=password12345
DEV_THEMEID=12356
DEV_SHOP=can-i-buy-a-feeling.myshopify.com

Then you can use the variables in your config:

development:
  password: ${DEV_PASSWD}
  theme_id: ${DEV_THEMEID}
  store: ${DEV_SHOP}

If you have any issues immeadiate issues with it please let me know. Otherwise I will close this issue and we can open new issues for changes.

[tanema]

Okay it seems like this issue now has a resolution. I am going to close it going forward and you can open any new issues for the prerelease until the official release happens.