Skaffold: Production vs Development environment variables.

[clanstyles]

What is the right way to store values in a deployment but switch them out for production? If I include secrets, do I mock the secrets with blank data and set them by hand? If the YAML for the secret changes, Skaffold would automatically update them. Some of these values aren't passwords, they might just be flags for "development" to switch out hostnames.

What's the recommended workflow for this?

[jstrachan]

if you use Helm for packaging up your application, you can put environment specific configuration values in your values.yaml file in your chart - then any installation of your chart can override any of those configuration values via helm install mychart --set foo.bar=123

On the Jenkins X project we use helm for packaging up applications and installing them in each environment and we're big fans of GitOps - so prefer to store these environment specific configuration values in a git repository - each environment gets its own git repository to manage those values in a values.yaml file - along with storing all the charts to be installed in the environment along with the versions of the charts to install.

Then to inject the values from values.yaml into your actual Deployment you can either expose the values as environment variables - or you can put the YAML inside a ConfigMap and mount it to the file system as a configuration file. e.g. if you are using Spring Boot you could mount the YAML from values.yaml as your application.yml file on your classpath

[dlorenc]

Let's use this issue to track the overall environment variable design/support. There have been a number of issues about this that I'll dupe against this bug.

[stevenbarragan]

There is helm secrets, I'm trying to figure out how to put it together with skaffold, has anybody tried it?

[cpoole]

This would be super helpful for us.

We have a static container that serves a react app. This app needs to be compiled/minified/etc when we go to dev through prod. But on our local machines we would like to skip the full compile and just run a continuous webpack server. Waiting a full minute for a total react rebuild isnt reasonable.

If we could pass a build arg to the dockerfile we could skip the react build in the dockerfile and this would be much more reasonable.

[bhack]

What is the problem with profiles?

[Place1]

It'd be nice if skaffold would allow you to defined variables that are rendered into your manifests. These ideally could be global or profile specific and could be sourced from environment variables e.g.

apiVersion: skaffold/v1alpha2
kind: Config
profiles:
- name: prod
  # ...  
  deploy:
    kubectl:
      manifests: ./k8s/*
    values:
      namespace: "prod"
      domain: "myapp.com"
- name: feature-branch
  # ...  
  deploy:
    kubectl:
      manifests: ./k8s/*
    values:
      namespace: "${CI_ENVIRONMENT_SLUG}"
      domain: "${CI_ENVIRONMENT_SLUG}.myapp.com"

then your k8s manifests could have the following:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myservice-ingress
  namespace: "${namespace:-default}"
spec:
  rules:
  - host: "myservice.${domain:-local}"
    http:
      paths:
      - path: /
        backend:
          serviceName: myservice
          servicePort: 80

I think this also allows skaffold to fill a nice middle ground between basic deployments with static manifests and helm deployments which adds a cluster component and multiple extra configuration files that feel like boilerplate when all you need is some basic template rendering.

[Place1]

What is the problem with profiles?

One limitation of profiles is that it requires you to hardcode values into skaffold.yaml which isn't desirable when the values are secrets.

This closed issue touched on a solution to this problem.

Also, I've found profiles to be very verbose. Consider the case where you want to change a docker build argument (as mentioned in #562) between development and production. If you used profiles for this then you'd be copy/pasting your development profile into a new 'production' profile and changing 1 value.

i'm not sure what the solution is but these are some things i've bumped into using skaffold

edit:
I should add that using environment variables for everything can be great in a CI environment where it's likely already standardized and setup for you, but in most other use-cases it can be really hard to use because you end up having to run commands like MY_ARG=value MY_OTHER_THING=value skaffold dev.

[stevenjm]

Another use case: The ability to interpolate environment variables within a skaffold.yaml would also be useful for builds. We have a build that makes use of a .npmrc with an authentication token that gets passed to Docker from Skaffold as a buildArg. (It is a multi-stage Docker build, so the image which uses this token never leaves the build machine.)

Rather than hard coding this in skaffold.yaml, it would be useful to be able to provide it from the environment. I've been using the following patch for testing, which works but is a hack:

diff --git a/pkg/skaffold/schema/v1alpha2/config.go b/pkg/skaffold/schema/v1alpha2/config.go
index e497490d..4a9735a2 100644
--- a/pkg/skaffold/schema/v1alpha2/config.go
+++ b/pkg/skaffold/schema/v1alpha2/config.go
@@ -17,6 +17,7 @@ limitations under the License.
 package v1alpha2

 import (
+       "github.com/GoogleContainerTools/skaffold/pkg/skaffold/util"
        "github.com/pkg/errors"
        yaml "gopkg.in/yaml.v2"
 )
@@ -223,6 +224,18 @@ func (c *SkaffoldConfig) Parse(contents []byte, useDefaults bool) error {
                return err
        }

+       for i, a := range c.Build.Artifacts {
+               for k, v := range a.DockerArtifact.BuildArgs {
+                       t, err := util.ParseEnvTemplate(*v)
+                       if err == nil {
+                               r, err := util.ExecuteEnvTemplate(t, map[string]string{})
+                               if err == nil {
+                                       c.Build.Artifacts[i].DockerArtifact.BuildArgs[k] = &r
+                               }
+                       }
+               }
+       }
+
        if useDefaults {
                if err := c.setDefaultValues(); err != nil {
                        return errors.Wrap(err, "applying default values")

I'm not sure what the best way is to solve this in a general way.

[bryanlarsen]

What is the problem with profiles

My build section is 50 lines of YAML and only about half complete. One of the buildArgs is unique per developer, so I'd need to add 17 * 100 lines to my skaffold.yaml.

[nathkn]

Personally I'd be in favor of allowing all fields in the skaffold.yaml to be able to be evaluated as envTemplates, just for flexibility of workflows.

I think a separate but related issue being discussed here is simply that skaffold doesn't allow profiles to inherit much of the defined yaml, as is spelled out in #789, so it might make sense to move that discussion over there.

[arnaud-deprez]

I'm definetely in favour of allowing all fields to be templatized.
I can see different use cases at build or deploy time that can be parameterized depending the environment.

And IMOH, we should also consider default value for these parameters so if it is nothing is specify on dev workstation for example, default is applied. May bash syntax ${FOO:-fooDefault} is good candidate ?

[Place1]

an alternative to the bash syntax which might be nice considering skaffold.yaml already uses go templating might be '{{ .MyEnvVar | default "a-default-value" }}' using a custom template function.

[r2d4]

Some tools have built templating over skaffold. You might want to check them out

https://github.com/jkozera/j2skaffold is an example.

[RoryShively]

I just opened an issue to continue the work on getting helm-secrets to work with skaffold
https://github.com/GoogleContainerTools/skaffold/issues/1178

[brpaz]

A common way to define environment variables during development is using a ".env" file. Docker-compose supports loading env files by default. It would be great if we can have the same with skaffold.

It could be added to the "docker" section of the skaffold.yaml file like:

docker:
    env_file: "./.env"

[Place1]

I've been maintaining a small fork that addresses some of the things in this issue. It allows you to:

https://github.com/Place1/skaffold#things-ive-added

1. use environment variables for buildArgs
2. use environment variables and templates for the kubectl deployer
3. rollout status for deployments, statefulsets and daemon sets (#459)

It looks a bit like this: https://github.com/Place1/skaffold/blob/master/examples/manifest-templating/skaffold.yaml

I think it'd be easy to add support for a .env file as @brpaz suggested. I'm also keen to add a multi CLI flag such as --set <variable>=<value> so that users can define these variables in the environment, a flag or file (in order of precedence).

[nagonzalez]

@Place1 I'd love to give your fork a shot but I'm unable to get it to work. Below are steps I took to build. Am I missing something?

build steps:

git clone https://github.com/Place1/skaffold.git
cd skaffold
go get -u -d github.com/Place1/skaffold
make install

skaffold.yml:

apiVersion: skaffold/v1alpha5
kind: Config
build:
  artifacts:
  - image: aws_ecr/target
    docker:
      buildArgs:
          PYPI_INDEX: '{{ .PYPI_INDEX }}'
      dockerfile: ./build/Dockerfile

[Place1]

@nagonzalez here's how I run skaffold.

1. clone the repo into ~/go/src/github.com/Place1/skaffold
2. cd ~/go/src/github.com/Place1/skaffold/cmd/skaffold
3. go build ./skaffold.go
4. use the built binary ./skaffold

[Place1]

perhaps https://github.com/kontena/mortar cloud be used as a deployer. it seems to satisfy most of the features being raised in this issue.

[fikriauliya]

My issue is similar to @cpoole :

• The generated Docker image for development and production should differ
• The development docker run the app in development mode (faster compile)
• The production docker needs more complex build (longer compile). Currently we offload this process to CloudBuild (configured in cloudbuild.yaml)

I wish Skaffold can:

• Have different build mode based on environment
• Read cloudbuild.yaml for production build

[cpoole]

@fikriauliya I fixed my issue with docker args
set the local_build arg to true by default and then in your run block do the following:

RUN if [ "$local_build" == "false" ]; then \
             npm build prod stuff; \
         fi;

and in our jenkins build we pass local_build false

[jakolehm]

@Place1 I heard rumours that @jnummelin is working on mortar support.

[bholemt]

if docker build arg command can pick values up from environment, then why shouldn't skaffold. skaffold should support all arguments to docker build in the same standard manner. why try to reinvent the whole thing and plug in one more tool? .env file is also another new way of doing the same thing. it doesn't work for private keys.

[rcoh]

It would be nice if you could define the environment variables in the profile rather than having to pass them in. Ideally I could just do skaffold run --profile staging but I think instead I need to maintain to different sets of environment variable files and use them appropriately to get access to the env var templating?

[tibbon]

Definitely need this. Is there anything something needs to help make this happen?

[stevenpall]

@Place1 I just ran into the variable substitution in buildArgs issue and came across your fork which attempts to add this functionality. Unfortunately, I'm unable to actually run your version of skaffold after compiling it. I get the following errors running skaffold dev or skaffold run:

Error: reading configuration: Config version out of date: run `skaffold fix`
FATA[0000] reading configuration: Config version out of date: run `skaffold fix`

Running skaffold fix, I get

ERRO[0000] Unable to parse config

I thought this might be due to my skaffold.yaml apiVersion. I was using skaffold/v1beta8 but have also tried other versions all the way back to skaffold/v1alpha4 to no avail. If this is simply an API version issue, I was wondering if you'd be able to rebase your fork against upstream so that the latest API version should work.

On a related note, do you think submitting these changes as a PR to upstream would be appropriate? It seems like a lot of people are looking for this functionality to be added to the core product.

Thanks!

[Place1]

I was really keen to push this upstream but I got rejected. Here’s a link
if you want to read why. I still think the features would be great but I
understand their (skaffold’s) goals.
https://github.com/GoogleContainerTools/skaffold/pull/892

On Tue, 23 Apr 2019 at 4:33 am, Steven Pall [e-mail]
wrote:

@Place1 https://github.com/Place1 I just ran into the variable
substitution in buildArgs issue and came across your fork which attempts
to add this functionality. Unfortunately, I'm unable to actually run your
version of skaffold after compiling it. I get the following errors running skaffold
dev or skaffold run:

Error: reading configuration: Config version out of date: run skaffold fix
FATA[0000] reading configuration: Config version out of date: run skaffold fix

Running skaffold fix, I get

ERRO[0000] Unable to parse config

I thought this might be due to my skaffold.yaml apiVersion. I was using
skaffold/v1beta8 but have also tried other versions all the way back to
skaffold/v1alpha4 to no avail. If this is simply an API version issue, I
was wondering if you'd be able to rebase your fork against upstream so that
the latest API version should work.

On a related note, do you think submitting these changes as a PR to
upstream would be appropriate? It seems like a lot of people are looking
for this functionality to be added to the core product.

Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleContainerTools/skaffold/issues/543#issuecomment-485505918,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACZQDFVAKR3ORZDTJ6IKUT3PRYAHRANCNFSM4E7RHGFA
.

[demisx]

This is a highly desirable feature for our team as well. Using the good old envsubst as a workaround in the meantime:

# In skaffold.yaml
...
docker:
    dockerfile: app/api/Dockerfile
    target: dev-env
    buildArgs:
        AWS_ACCOUNT_ID: "$AWS_ACCOUNT_ID"
        NAMESPACE: "$NAMESPACE"

$ envsubst < skaffold.yaml > skaffold.gen && skaffold dev -f=skaffold.gen --port-forward=false

[reegnz]

@demisx you dont need envsubst for that anymore, skaffold can already template buildArgs. It's been in 0.29.0: https://github.com/GoogleContainerTools/skaffold/releases/tag/v0.29.0:

Allow environment variables to be used in docker build argument #1912

Only restriction is that you will have to build through the docker cli, not the docker API:

build:
  local:
    useDockerCLI: true

I guess this issue is more about the kubernetes manifests, and not the docker build.

[demisx]

@reegnz Thank you! Worked like a charm!

# In skaffold.yaml
    ...
    build:
      local:
        useDockerCLI: true
      artifacts:
        - image: my-docker-image
          docker:
            dockerfile: path/to/Dockerfile
            buildArgs:
              DOCKER_ARG_1: "{{.ENV_VAR_1}}"
              DOCKER_ARG_2: "{{.ENV_VAR_2}}"
              ...

[dkirrane]

Hi, Looking to use Env variables inside Helm deployment

This currently doesn't work for var1 & var2 below

apiVersion: skaffold/v1beta11
kind: Config

deploy:
 helm:
    releases:
      - name: my-chart
        chartPath: ./helm/my-chart
        setValueTemplates:
          image.repository: "{{.IMAGE_NAME}}"
          image.tag: "{{.VERSION}}"
        overrides:
            var1: {{.ENV_VAR_1}}
            var2: {{.ENV_VAR_2}}

[demisx]

Are the environment variables currently supported deploying plain YAML resources and not Helm releases?

[tejal29]

Related to #2849

[tejal29]

I am going to close this and we should track this in #2849