Brigade: Can a custom npm registry be used with brigade.json?

[LittleColin]

I've tried using the .npmrc to configure a private verdaccio npm repo but it doesn't seem to be used by yarn in Brigade which still goes to public yarn registry. Using yarn 1.7.0 manually outside of brigade (i.e. copying brigade.json to package.json and running yarn) does work, so I suspect it could be an issue with the version of yarn used by brigade or the way it's used?

.npmrc:
@devops:registry=http://verdaccio:4873

brigade.json:

{
    "dependencies": {
        "@devops/my-devops-brigade": "0.1.0"
    }
}

Error in logs:

Started build 01cjpc09gvdj8vtwbv61jvfzhr as "brigade-worker-01cjpc09gvdj8vtwbv61jvfzhr"
installing @devops/[email protected]
prestart: src/brigade.js written
{ ChildProcessError: Command failed: yarn add @devops/[email protected]
error An unexpected error occurred: "https://registry.yarnpkg.com/@devops%2fmy-devops-brigade: Not found".
 `yarn add @devops/[email protected]` (exited with error code 1)

[radu-matei]

Hi, @LittleColin, thanks for raising this!

Would the .npmrc be the same one used by the rest of the project, or a custom one, only for the Brigade pipeline?

I'm trying to figure out if we would need separation between what the Brigade pipeline needs and other dependencies of a Node project that would already use a .npmrc file.

[LittleColin]

Thanks for the response. I'm thinking it would be custom one only for the Brigade pipeline since the brigade.json is separate from a package.json used by the project.

The brigade.json may call "npm install" on the project which could potentially use a different set of registries. On our project, there is a distinction between the developers who would maintain the package.json and the devops guys who would be interested in the brigade.json+brigade.js file.

Does this help?

[radu-matei]

Yes, this is exactly what I was looking for - it means that since this registry is for Brigade only and not the entire project, the registry setting could sit in brigade.json - but since this file sits in source control, is there anything that would prevent us from having the registry setting as part of it? (config, SSH keys..?)

Otherwise, we might have to use the project's Kubernetes secret.

Thanks!

[technosophos]

Or create a custom worker Dockerfile that does something like this:

FROM deis/brigade-worker:latest

COPY path/to/.npm .npm

And you could put this in your own container registry if you're concerned about keeping sensitive info there. Then in your project, you would need to set the worker image to point to this new worker:

values.yaml

worker:
  registry: myreg
  name: my-brigade-worker
  tag: latest
  pullPolicy: IfNotPresent

[LittleColin]

Thanks @technosophos, that would work nicely I think - I guess you still end up with a secret in a docker image albeit in your own private registry?

In case there would be benefit in pursuing a .npmrc-like solution, .npmrc allows tokenisation based on environment variables.

However I see that in brigade docs, using environment variables can end up in logs.

If this is going into brigade.json anyway as @radu-matei proposes and therefore processed by brigade, could it get values directly from the project secrets - e.g. @devops:registry=http://verdaccio:4873:_authToken=${project.secrets.npmToken}

Related recent article on securing secrets in .npmrc in case it's of interest/relevant:
https://www.alexandraulsh.com/2018/06/25/docker-npmrc-security/

[technosophos]

See also #477, where this came up in the comments

[LittleColin]

Thanks, so assuming a custom worker that copies in a .npmrc file - there is just the issue that brigade goes to the external https://registry.yarnpkg.com/ rather than a custom registry. Do you want a separate issue to track this?

I still have the fun of actually trying to generate a .npmrc file with creds that don't expire for our npm registry! But all that will be moot if we can't point at the registry in the first place :-)

[technosophos]

Let's keep tracking on this issue for now. I'm mainly hoping that you find a good solid pattern that will direct how we modify things in the future. This is definitely one of those parts of Brigade that we're experimenting with at the moment.

[LittleColin]

For what its worth we decided to follow your suggestion above of creating a custom brigade worker. This has the advantage of faster build times as we aren't pulling down another package each time we build and we don't have the complication of authentication with a custom npm registry so thanks for that 👍

We can also locally validate our brigade.js file by mocking brigadier and pulling in our custom library locally - but that's tangential to this issue.

[radu-matei]

Closing, but feel free to reopen for any additional questions.

Thanks!

[phsteve]

A note for anyone trying to do this now: use the brigadecore/brigade-worker image as your base image instead of deis/brigade-worker as mentioned in @technosophos 's comment above (https://github.com/brigadecore/brigade/issues/558#issuecomment-405980462).