Vercel: Storing complex secrets

[jberglinds]

Solution: https://github.com/zeit/now/issues/749#issuecomment-533873759

Hello,
I can't figure out how to store my Google API private key for use in a now deployment.
The key looks something like:

-----BEGIN PRIVATE KEY-----
lines
lines
lines
-----END PRIVATE KEY-----

What I've tried:

• now secret add google-private-key "<full key>"
  
  
  
  • Not possible because of dashes, https://github.com/zeit/now-cli/issues/80
  
  
• now secret add google-private-key -b $(echo "<full key>" | base64)
  
  
  
  • Only gives me "[object Object]" in js, see https://github.com/zeit/now-cli/issues/80
  
  
• Adding the key to a .env-file and then run with now --dotenv=.env
  
  
  
  • This seems to replace all newlines with whitespace making the auth with the key fail.
  
  
  • When running locally using the dotenv node package to set env vars from .env it works, it preserves newlines
  
  

How should it be done?

[sholladay]

I think your second example is almost correct, just omit the -b. I also used cat instead of echo below to ensure the newline problem isn't related to your terminal.

now secret add google-private-key "$(cat my-keyfile | base64)"

Unfortunately, whatever the intention behind the -b flag was, it seems to be quite broken.

https://github.com/zeit/now-cli/blob/f8c0288bb3e3f05b38ab6a75ba78d5dcc0e57c7a/src/providers/sh/commands/secrets.js#L236-L240

[malixsys]

Use @sholladay's method to put the secret in PROD:

now secret add google-private-key "$(cat my-keyfile | base64)"

then in now.json put:

{
  "env": {
    "google_private_key": "@google-private-key"
  }
}

then in your code:

const atob = require('atob');
const privateKey = atob(process.env.google_private_key);
// use privateKey

Locally, you can use the following to load your secrets:
In .env:

google-private-key=<btoa() of your key>

then use this script in package.json:

  "scripts": {
    "dev": "nodemon --exec \"node -r dotenv/config\" server.js",
  }

IMPORTANT!

Make sure to add .env to .gitignore and .npmignore!!

[Qix-]

Definitely - action item for this issue is to make sure we're elegantly handling different types of secrets. I'm not sure environment variables are a silver bullet here.

This definitely deserves some investigation.

[hakusaro]

Anything on this issue lately? Fixing this is kind of vital for apps like GitHub’s own Probot to be deployed successfully with Now.

[malixsys]

@hakusaro Did you try the workaround above? (e.g. for google-private-key)

[hakusaro]

I feel like I've been endlessly repeating this. I don't have this issue _personally_ anymore, but it violates the principle of least surprise that the following works:

heroku config:set SOME_PRIVATE_KEY=[some multiline blob here]

but the following, which is stupidly close in syntax, does not:

now -e SOME_PRIVATE_KEY=[some multiline blob here]

Similarly, now doesn't echo back environment variables it sets, whereas heroku does. This means that a user has no idea that setting a private key failed. It just silently blows up.

[malixsys]

Sorry did not see it before... I agree with you on "_principle of least surprise_", but it is NOT a blocker, is it?

Fixing this is kind of vital for apps like GitHub’s own Probot to be deployed successfully with Now.

[hakusaro]

It's not a blocker _by the strictest definition_, but the probot maintainers (of which I am regrettably not) currently provide three solutions for deploying probot to the internet. They go to admirably extreme lengths to make the deployment process as simple as possible on the platforms they support.

Having this issue introduces enough friction that they would rather exclude the entire platform. I think the fear is that this response:

I'm not sure environment variables are a silver bullet here.
This definitely deserves some investigation.

and no movement on patching this gives an indication that now isn't frictionless enough to warrant further inclusion.

---

Speaking from my own experience, I knew already that probot was a GitHub product, so I took the suggestion to use now as an implicit recommendation that it was stable enough for GitHub to suggest using it. GitHub's reputation was carried over to it a little in my mind. I think the reason for the removal suggestion comes partially from this. It isn't a good user experience for someone to try to deploy a GitHub product on a platform using a workaround for something that _should work_.

The problem is that I have a soft spot for now in that I found it better than the other two options in a lot of areas. I'm sticking my head out because I don't want to see a stupid issue quash the potential joy that people get from using now. It should be a trivial enough fix to get this deployment to work, and I think that any ideological loss that happens from going with what heroku does is negligible compared to potential user loss due to this kind of issue.

Notice that I'm not even caring about Google API private keys. For every person who reports an issue or comments on an issue about a bug, there are ten others who sigh, drop the platform, and move on to the next. There's no way of telling how many people don't use now as a result of this than if it just worked.

[malixsys]

Indeed! Hope the path gets smoother... If only I had time to create a PR...

CC @rauchg ?

[TooTallNate]

As of https://github.com/zeit/now-cli/pull/1033, the --base64 argument has been removed. As mentioned in the original post, it has been broken for some time (returns the string "[object Object]"). Aside from that, it's never been clear how exactly to get the binary data back out from the env var.

The recommended way to store complex and/or binary data with secrets is with the base64 CLI, but without --base64:

now secret add google-private-key "$(cat key.txt | base64)"

Then in your Node.js server code you can use Buffer to turn it back into a binary blob:

const key = new Buffer(process.env.GOOGLE_PRIVATE_KEY, 'base64');

Or if it's a Docker deployment then do something like:

echo "$GOOGLE_PRIVATE_KEY" | base64 -d > key.txt

I hope that helps!

[leo]

Closing this after @TooTallNate's answer – please do it like he described 😊

[Maxhodges]

great so everyone has to waste hours trying things that don't work as expected and consulting expensive consultants before hopefully discovering this issue and proposed workaround.

[VincentTam]

Thanks for raising this issue. I'm having the same problem.

Then in your Node.js server code you can use Buffer to turn it back into a binary blob:

const key = new Buffer(process.env.GOOGLE_PRIVATE_KEY, 'base64');

@TooTallNate new Buffer(...) has deprecated.

[mckernanin]

@VincentTam a node deprecation isn't an issue with now, and in the page you linked there's a guide on how to do this in a non-deprecated way, which looks like this:

const key = Buffer.from(process.env.GOOGLE_PRIVATE_KEY, 'base64');

[hleumas]

There is actually much easier way of storing complex secrets in now.

Just do this:

now secret add google-private-key -- "`< my-key-file`"

The -- is a bash convention to indicate the end of command options so this resolves the problem with key starting with ----.

I've tested this and it works for me. This also resolves https://github.com/zeit/now-cli/issues/80.

[fcarreno]

There is actually much easier way of storing complex secrets in now.

Just do this:

now secret add google-private-key -- "`< my-key-file`"

The -- is a bash convention to indicate the end of command options so this resolves the problem with key starting with ----.

I've tested this and it works for me. This also resolves #80.

Thanks @hleumas for this tip!.
I'm a Windows user and didn't take the time to check how to accomplish the same from regular Windows cmd/PowerShell, but I was able to use the suggested command, by switching to WSL terminal, and use the same command with a slight variation.

now.exe secret add firebase_private_key -- "`< g.key`"

( .exe seems to be required when executing Windows tools from WSL, and being g.key a file present in your current path, including your -----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY----- key string)

[paulmelnikow]

@hleumas @fcarreno Storing it in a file means it needs to be shipped as part of the deploy where it becomes part of an immutable asset. I think the goal is to store it securely in an environment variable and pull it from there at runtime.

[Qix-]

What would just be absurdly obvious, trivial to implement, and wildly helpful would be to add now secret add -f some-file.txt secret-name and cat something | now secret add secret-name -, and in cases where you want a literal - as the value, then now secret add secret-name -- -. now secret add secret-name (with no value) would prompt. 🙃

[fcarreno]

Can't speak about complexity to implement, but I'm sure pull requests are welcome if you anticipate such a simple task.

And, @paulmelnikow - you can always have the file created, generate the secret from it, and get its value from an env variable...That file does not need to be committed. In that sense, the secret will be no different than env variables, in my opinion.

[hleumas]

@paulmelnikow the example assumed you already have key stored in file as that's something most of keygen tools do, they create files. However, nothing prevents you from pasting the content into commandline instead.

[paulmelnikow]

Getting the multi-line env var into a file via the environment _is_ the challenge. See e.g. probot/friction#17.

[hleumas]

@paulmelnikow I don't understand your point. The solution I pointed out solves exactly that problem.

[paulmelnikow]

I'm confused. https://github.com/zeit/now-cli/issues/749#issuecomment-487322395 appears to pull the data from a file into Google Cloud config. It doesn't mention environment variables, explain how to store a multi-line variable in the Now environment, or how to take such a variable and transfer it into a file.

[hleumas]

This:

now secret add -- google-private-key "`< my-key-file`"

creates a now secret named google-private-key and stores a content of your local file my-key-file into it.

Then, when you deploy your now app via:

now -e MY_GOOGLE_KEY=@google-private-key

it will deploy your app and run your app with environment variable MY_GOOGLE_KEY set to the value of the secret google-private-key.

This has nothing to do with Google Cloud. I used name google-private-key just for illustration, it can be my-super-secret-key instead and nothing would change.

Also, I am not storing a content of a variable into the file. I am creating a now secret from the content of your local file.

Is that a bit more clear now?

[paulmelnikow]

Ahh yes, that's a lot clearer now, thanks!

I'd stopped using Now for GCP because of challenges with this, and it's good to know there's a solution in the future.

[danieltodonnell]

Hi there,
I'm trying to make an auth portal that will be easy to use for our clients, but I honestly can't expect our users to use the -- workaround to express a file as a config. Is there any solution for this in the upcoming pipeline?

[alexander-daniel]

@hleumas your suggestion worked great for our use case. One thing I'll add here just in case someone else runs into the same issue:

it did not work when my file was key.txt but once i changed it to key.pem to have the .pem extension it did work.

🎉 thanks for the solution!

[bastienrobert]

@hleumas, I still don't understand how did you use your solution with google.auth.getClient :/

[hleumas]

@bastienrobert it would really be super-duper helpful next time if you'd be more specific when you don't understand something. But nevertheless, let my try again:

Following creates secret named my-dog-name with value Cooper

now secret add my-dog-name Cooper

The problem begins when the name of your dog begins with --, such as -----Cooper. This is problem because it gets interpreted as flag for the now secret add command. We need to indicate there are no more flags to be expected:

now secret add -- my-dog-name ----Cooper

The -- is Posix standard way to indicate there won't be any more flags for the command.

To store your private key, all you need to do is:

now secret add -- my-private-key "-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAKBgQCqGKukO1De7zhZj6+H0qtjTkVxwTCpvKe4eCZ0FPqri0cb2JZfXJ/DgYSF6vUp
wmJG8wVQZKjeGcjDOL5UlsuusFncCzWBQ7RKNUSesmQRMSGkVb1/3j+skZ6UtW+5u09lHNsj6tQ5
1s1SPrCBkedbNf0Tp0GbMJDyR4e9T04ZZwIDAQABAoGAFijko56+qGyN8M0RVyaRAXz++xTqHBLh
3tx4VgMtrQ+WEgCjhoTwo23KMBAuJGSYnRmoBZM3lMfTKevIkAidPExvYCdm5dYq3XToLkkLv5L2
pIIVOFMDG+KESnAFV7l2c+cnzRMW0+b6f8mR1CJzZuxVLL6Q02fvLi55/mbSYxECQQDeAw6fiIQX
GukBI4eMZZt4nscy2o12KyYner3VpoeE+Np2q+Z3pvAMd/aNzQ/W9WaI+NRfcxUJrmfPwIGm63il
AkEAxCL5HQb2bQr4ByorcMWm/hEP2MZzROV73yF41hPsRC9m66KrheO9HPTJuo3/9s5p+sqGxOlF
L0NDt4SkosjgGwJAFklyR1uZ/wPJjj611cdBcztlPdqoxssQGnh85BzCj/u3WqBpE2vjvyyvyI5k
X6zk7S0ljKtt2jny2+00VsBerQJBAJGC1Mg5Oydo5NwD6BiROrPxGo2bpTbu/fhrT8ebHkTz2epl
U9VQQSQzY1oZMVX8i1m5WUTLPz2yLJIBQVdXqhMCQBGoiuSoSjafUhV7i1cEGpb88h5NBYZzWXGZ
37sJ5QsW+sJyoNde3xH8vdXhzU7eT82D6X/scw9RZz+/6rCJ4p0=
-----END RSA PRIVATE KEY-----"

Notice my use of "" because the key contains newlines.

Some might argue copy-pasting your key like that is really ugly. So in a case you have it saved in a file, let's say my-private-key.txt, you can easily use another shell trick:

now secret add -- my-private-key "`< my-private-key.txt`"

The `< my-private-key.txt` takes the content of the my-private-key.txt file and pastes it into the command line for you. Alternatively, you can also do `cat my-private-key.txt` which would do the same.

Is that more clear now?

[bastienrobert]

Much more. Sorry, my bad, I thought I was going to throw my computer through a window. I got a little upset 😅.

I just add something to your answer:

For people who want to use this solution with the google.auth.getClient method (or anything equivalent), the best way I could find was to use the given JSON file locally (don't forget to add it to the gitignore / npmignore) and, for the deployment using now, add a check on the NODE_ENV to pass the credentials with the solution specified above by @hleumas by parsing the key:
JSON.parse(process.env.GOOGLE_APPLICATION_CREDENTIALS)

[Vadorequest]

@jberglinds Could you please update your main post by linking to the solution at https://github.com/zeit/now/issues/749#issuecomment-533873759 ? That'd be super useful for folks to avoid reading everything in between and just go to the solution. Thanks!

@hleumas Could this be added to the official Zeit documentation? Something like "Storing file's content as a secret", under https://zeit.co/docs/runtimes#advanced-usage/advanced-node-js-usage, I think that's a good fit. :) _(actually, probably not the best fit, but I'll let you guys decide what is!)_

[hleumas]

@Vadorequest I am sure the zeit team will appreciate pull requests into their documentation.

[Vadorequest]

I've looked around and I'm not sure how to do that. There isn't enough documentation (for maintainers) for getting started on my own, don't know where to start.

[hleumas]

@Vadorequest here is the repo. It also includes guidelines on how to contribute.

[leerob]

I had tried pretty much everything on here and _still_ it wasn't working. I think the root cause was because when I copy/pasted my private key from Firebase, it still had the new line characters (\n) instead of actually having new lines. After switching to remove those, it worked.

now secret add -- firebase-private-key "-----BEGIN PRIVATE KEY-----
FIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCnRDtsgXGilmwB
...
...
...
7n2UQ9E66roET+PSC/4wrw==
-----END PRIVATE KEY-----"

Correct me if I'm wrong, but I believe since the secrets get injected at build time, you need to re-deploy every time you make a secret change. This could be helpful to know when debugging.

[Vadorequest]

@leerob Secrets are dynamics, but if your app uses them at build time then yes, you'll need to redeploy.

[OlliV]

@leerob I'd probably just upload the private key as password protected and then create a secret with the password. That applies to any other "complex" or huge secret as well. Though, I must admit that I find myself a bit uneasy uploading a private key as a file, even when it's encrypted.

[goran-paunovic]

If you are using windows then use GitBash.
I was trying to add my firebase key from VSCode terminal using
now secret add --
and, of course, it didn't work. I realised I wasn't reading that Posix part 🙄 .
I changed my shell to Git Bash end successfully set secret and everything is working now 😄 .

[leerob]

@OlliV After adding my base64 secret above, I got this build error:

Environment variables must not be greater than 4kb when JSON-stringified (got 4.21KB)

That led me to encrypt the service account and deploy it alongside my repo. Overall, I feel this is an easier solution than dealing with messy base64 variables.

Here's how if you're interested.

[Ponjimon]

@OlliV After adding my base64 secret above, I got this build error:

Environment variables must not be greater than 4kb when JSON-stringified (got 4.21KB)

That led me to encrypt the service account and deploy it alongside my repo. Overall, I feel this is an easier solution than dealing with messy base64 variables.

Here's how if you're interested.

The link doesn't work for me anymore :(

[leerob]

I must have messed up my redirect. This should work -> https://leerob.io/blog/vercel-env-variables-size-limit

[Ponjimon]

Thank you!

[ScreamZ]

See also https://github.com/kuzzleio/kuzzle-vault

[advl]

@malixsys @leerob
Small improvement on your answers, add your secret without line wrapping

now secret add google-private-key "$(cat my-keyfile | base64 -w 0)"

The -w 0 option removes line wrapping. See this stackoverflow answer for more details https://superuser.com/questions/1225134/why-does-the-base64-of-a-string-contain-n/1225139

(I found myself dealing with different default line wrapping options in base64 depending on the OS)

[tettoffensive]

@779g On osx base64 doesn't have the -w 0 option so I had to do

vc secret add firebase-private-key "$(cat my-keyfile | base64 | tr -d \\n)"

[transitive-bullshit]

On macOS, I do:

now secret add -- google-application-credentials "`base64 keyfile.json`"

[leerob]

Has anyone figured out how to do this using the environment variables UI? I just tried adding a service account there and failed.

[HermanNygaard]

Has anyone figured out how to do this using the environment variables UI? I just tried adding a service account there and failed.

@leerob Did you figure it out? Either on the webapp or with vercel CLI? Having the same issue

[leerob]

@HermanNygaard I did. I needed to convert new line characters \n to actual new lines. That worked!

[HermanNygaard]

@leerob thanks, I'm still getting an error. Did you include string literals? And was it on the web ui?

[manuganji]

@HermanNygaard are you sure you removed you replaced all the \n with new lines? I could get it to work.

[manuganji]

By "string literals" if you meant surrounding the key with quotes, no I didn't do that. I pasted only the key value into the field. But I'm seeing something weird when I do vercel secrets ls in the console. It's showing the variable name with a few extra characters appended in the end. I believe this would cause issues when I try to reference the variable as @var_name in vercel.json

[HermanNygaard]

@manuganji thanks! It worked now. I'm not sure what I changed, maybe I had the string quotes included with the input :D

[dsernst]

This is still pretty hard. Been fighting with it for a while now.

I have no trouble importing a .pem as an environment variable for a Next api/ route on dev w/ .env.local.
But it is failing on Vercel builds.

Manually replacing \n with new lines didn't work for me.

• One thing that would helpful is if there was a button to redeploy right from the Vercel Environment Variables web control panel. 👍👍

I don't see a way to redeploy from the web ui, and need to switch to the CLI to run vercel --prod or push to git (not ideal just for testing new environment variables).

• And it could be helpful if Vercel CLI had an option to echo back new env varables, as @hakusaro suggested in https://github.com/vercel/vercel/issues/749#issuecomment-351196434

[dsernst]

Manually replacing \n with new lines didn't work for me.

Ah, finally got this working. 🎉

It turns out manually replacing all the \n with carriage returns did work, but I had only seen & fixed the two obvious ones at the start and end, missing the other 25 buried within the long base64 key.

[hipdev]

This worked for me without replacing something :
I created a vercel.json in the root path with:

{
  "scope": "<userOrTeamName>",
  "env": {
    "FIREBASE_PRIVATE_KEY": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADA     ...        O\n-----END PRIVATE KEY-----\n"
  }
}

Then just --> vc or vc --prod (For prod deployment)

Using the Vercel UI still not working for me 😢

[goleary]

thanks for the suggestion @hipdev that worked for me.

Although I had to use NPM_RC for a private package too and it didn't work when set that way. So I had to set my github token in the UI and set my firebase key from vercel.json + cli.

I should add that even after succesfully deploying from the CLI, any new commits on the repo fail to deploy, so I have to use the CLI every time...definitely not ideal.

[grikomsn]

Here's my take on storing Firebase private key on Vercel secrets:

---

Take 1: store long secrets with unescaped newlines (does not work) ❌

As like everyone else, tried storing the private key as-is and resulted with this error log which I assume is because the unescaped newlines.

View command line log

vc secrets add firebase-private-key "-----BEGIN PRIVATE KEY-----\nXXXXXXXXXXXXXXXXXXXXXXXX\n-----END PRIVATE KEY-----"
internal/validators.js:120
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Array
    at validateString (internal/validators.js:120:11)
    at Object.resolve (path.js:980:7)
    at Object.getGlobalPathConfig [as default] (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:221212:43)
    at Object.79919 (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:221070:41)
    at __webpack_require__ (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:231483:43)
    at Object.13289 (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:229150:34)
    at __webpack_require__ (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:231483:43)
    at Object.86137 (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:229423:19)
    at __webpack_require__ (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:231483:43)
    at Module.69739 (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:208259:13) {
  code: 'ERR_INVALID_ARG_TYPE'
}

---

Take 2: store the whole json as the secret (does work, but exceeds size limit) ❌

Tried storing the actual JSON content of the Firebase credentials to Vercel secrets by running something like:

vc secrets add firebase-creds-json "$(< ./creds.json)"

Then do a JSON.parse(process.env.FIREBASE_CREDS_JSON) when retrieving the env var. Technically it works, but when deploying you'll get the error of exceeding maximum env var size limit (similar to previous comment).

---

Take 3: store long secret as json-compliant secret (does work) 🎉🎉🎉

Since take 2 works but it exceeds the secret size limit, now I tried only storing the private key but also as a JSON-compliant value (either a basic quoted string or an object):

# as a single string value, note the extra quote
vc secrets add firebase-private-key '"-----BEGIN PRIVATE KEY-----\nXXXXXXXXXXXXXXXXXXXXXXXX\n-----END PRIVATE KEY-----"'

# as a json object
vc secrets add firebase-private-key '{"privateKey":"-----BEGIN PRIVATE KEY-----\nXXXXXXXXXXXXXXXXXXXXXXXX\n-----END PRIVATE KEY-----"}'

Then same as take 2, retrieve the value using JSON.parse(process.env.FIREBASE_PRIVATE_KEY) if it's a single string or const { privateKey } = JSON.parse(process.env.FIREBASE_CREDS_JSON) if it's a JSON object. Make sure you use double quotes in your .env file on local development:

# firebase env vars
FIREBASE_PRIVATE_KEY_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
FIREBASE_PRIVATE_KEY='"-----BEGIN PRIVATE KEY-----\nXXX\n-----END PRIVATE KEY-----"' # make sure to double wrap with quotes

---

Hope this helps! 🎉

[saitodisse]

Thanks @grikomsn, I finally could configure the GOOGLE_APPLICATION_CREDENTIAL without using the json file way.

@grikomsn 's Take 3 and this blog post

GOOGLE_APPLICATION_CREDENTIAL without JSON FILE

---

getAuthToken.ts

import { google } from 'googleapis'

export const SCOPES = ['https://www.googleapis.com/auth/spreadsheets']

// pega credentials das variáveis de ambiente
export async function getAuthToken() {
  if (typeof window !== 'undefined') {
    throw new Error('NO SECRETS ON CLIENT!')
  }

  const { privateKey } = JSON.parse(process.env.GOOGLE_PRIVATE_KEY || '{ privateKey: null }')
  const auth = new google.auth.GoogleAuth({
    scopes: SCOPES,
    projectId: process.env.GOOGLE_PROJECTID,
    credentials: {
      private_key: privateKey,
      client_email: process.env.GOOGLE_CLIENT_EMAIL,
    },
  })
  const authToken = await auth.getClient()
  return authToken
}

---

env.local

GOOGLE_PROJECTID=sheets-xxxxxxxxxxxxxx
GOOGLE_PRIVATE_KEY='{"privateKey":"-----BEGIN PRIVATE KEY.........-----END PRIVATE KEY-----\n"}'
GOOGLE_CLIENT_EMAIL=[e-mail]