Skaffold: Skaffold generates strange warnings when running with many profiles

[petr-buchyn]

Expected behavior

I expect Skaffold to run in dev-mode without errors and warnings, when configured correctly

Actual behavior

We use Skaffold to develop locally a big platform, which consists from many services.
We use Helm as our deployer.
We have separate profiles for different services in skaffold.yaml, therefore our skaffold command looks like following:

skaffold dev -p=profile-one,profile-two,profile-three,profile-n --cache-artifacts=true --auto-build=false --auto-deploy=false --rpc-http-port=50070

However, after Helm completes deploy, skaffold starts to generate errors like following:

WARN[0096] error adding label to runtime object: patching resource default/some-secret: Secret "some-secret" is invalid: metadata.labels: Invalid value: "profile-one__profile-two__profile-three__profile-n": must be no more than 63 characters 

This warnings are generated for every single resource we have: configmaps, secrets, roles, role bindings and so on, and the label value is the names of our skaffold profiles, concatenated with __ (to underscores), in order, in which they appear in our skaffold.yaml.

Information

• Skaffold version: 1.0.1
• Operating system: MacOS Catalina
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta16
kind: Config
metadata:
  name: some-name
profiles:
- name: profile-one
  build:
    local:
      push: false
    artifacts:
    - image: some/image
      context: some-submodule
      docker:
        dockerfile: Dockerfile
    - image: some-other/image
      context: some-other-submodule
      docker:
        dockerfile: Dockerfile
  deploy:
    helm:
      releases:
      - name: release-name
        chartPath: charts/our-chart
        valuesFiles:
        - charts/our-chart/values/value-file-one.yaml
        - charts/our-chart/values/value-file-two.yaml

# Supplemental profiles
- name: profile-two
  patches:
  - op: add
    path: /deploy/helm/releases/0/valuesFiles/-
    value: charts/our-chart/values/yet-one-file.yaml
- name: profile-three
  patches:
  - op: add
    path: /deploy/helm/releases/0/valuesFiles/-
    value: charts/our-chart/values/yet-another-file.yaml

Steps to reproduce the behavior

Skaffold command: skaffold dev -p=profile-one,profile-two,profile-three,profile-n --cache-artifacts=true --auto-build=false --auto-deploy=false --rpc-http-port=50070

_Actual number of profiles, specified in command, is bigger than three, of course._

[jl-ward]

I see that there is a pr to address the warning issue I am also seeing, but I can't seem to get passing multiple profiles to work as the documentation and the op suggests. I am on version 1.0.1 of Skaffold and running macOS Mojave.
https://skaffold.dev/docs/environment/profiles/
Here is an example of a skaffold.yaml that can be used to verify:

apiVersion: skaffold/v1
kind: Config

profiles:
  - name: kafka
    deploy:
      helm:
        releases:
        - name: kafka
          namespace: kafka
          chartPath: bitnami/kafka
          remote: true
          version: 7.0.2        
  - name: zookeeper
    deploy:
      helm:
        releases:
        - name: zookeeper
          namespace: zookeeper
          chartPath: bitnami/zookeeper
          remote: true
          version: 5.1.1

Running any of the below does not work.

skaffold deploy -p kafka,zookeeper
skaffold deploy -p=kafka,zookeeper
skaffold deploy --profile=kafka,zookeeper
skaffold deploy -p kafka -p zookeeper
skaffold delete -p kafka,zookeeper
skaffold delete -p kafka -p zookeeper

There are no errors (except a very lengthy WARN message that times out on deployment), but it just seems to ignore anything other than the last item in the comma separated list or the last argument passed to -p.

It also appears that setting the SKAFFOLD_PROFILE env var to kafka,zookeeper and running skaffold deploy is only grabbing the last item in the list as well.

[petr-buchin]

@dgageot I see you created a fix for that issue. Thank you very much!
However, what do you think - maybe it's better to create some hash of profiles string, instead of trimming it to 62 characters and adding z ?

Hash from the whole profiles string will actually change with the profiles string, but trimmed string will not, if changes were made after "trim point".

[kvokka]

I experience the same behaviour with skaffold 1.0.1 on macOs 10.15.1.

Also, this bug makes

    activation:
      - kubeContext: minikube

useless, since it blocking the manual profile selection.

[kvokka]

@dgageot The bug is still in place, tried

  skaffold dev -p rails -p core -p api
  skaffold dev --profile=rails,core,api
  skaffold dev -p=rails,core,api
  skaffold dev -p rails,core,api

and always got only the latest profile executed, while the others were omitted with out any warning.
(for testing i use Skaffold 1.1.0 api version skaffold/v2alpha1)

[petr-buchin]

@kvokka your bug is actually not relative to this issue, but you can see this comment:
https://github.com/GoogleContainerTools/skaffold/issues/3310#issuecomment-559311691

Possibly you have not build and deploy sections at the top level, or have this sections empty.
Because of nonempty tag used during umnarshaling of YAML in Skaffold, this sections get removed at all, and as a result, each of your profiles may replace the whole build or deploy section, and thus you will experience this strange behavior, when only latest profile is executed.

If so, I found temporary workaround until this get fixed: you can fill top-level sections with some dummy values, and that will make your profiles work.

[kvokka]

@petr-buchin Merry Christmas! Thank you for such a quick response. But unfortunately this hack did not work, tried

skaffold build -p api,core
skaffold dev -p api,core

In both cases i got only the action for core service (since it is the latest in the list). The services in the main section and in the profiles are the same (and with out profiles or running 1 profile only it works).

Will provide fill skaffold.yaml, which may help in the problem investigation.

apiVersion: skaffold/v2alpha1
kind: Config
build:
  tagPolicy:
    gitCommit: {}
  artifacts:
  - image: us.gcr.io/replay-gaming/rails
    docker:
      dockerfile: docker/development/Dockerfile
    context: rails
    sync:
      # switch to `infer` syntax after https://github.com/GoogleContainerTools/skaffold/issues/3376
      manual:
      - src: 'app/**/*'
        dest: '.'
      - src: 'bin/**/*'
        dest: '.'
      - src: 'db/**/*'
        dest: '.'
      - src: 'lib/**/*'
        dest: '.'
      - src: 'public/**/*'
        dest: '.'
      - src: 'script/**/*'
        dest: '.'
      - src: 'spec/**/*'
        dest: '.'
  - image: us.gcr.io/replay-gaming/poker-core
    docker:
      dockerfile: docker/development/Dockerfile
    context: core
    sync:
      # switch to `infer` syntax after https://github.com/GoogleContainerTools/skaffold/issues/3376
      manual:
      - src: 'bin/**/*'
        dest: '.'
      - src: 'db/**/*'
        dest: '.'
      - src: 'lib/**/*'
        dest: '.'
      - src: 'system/**/*'
        dest: '.'
      - src: 'schemas/**/*'
        dest: '.'
  - image: us.gcr.io/replay-gaming/poker-api
    docker:
      dockerfile: docker/development/Dockerfile
    context: api
    sync:
      # switch to `infer` syntax after https://github.com/GoogleContainerTools/skaffold/issues/3376
      manual:
      - src: 'bin/**/*'
        dest: '.'
      - src: 'src/**/*'
        dest: '.'
      - src: 'tests/**/*'
        dest: '.'
  - image: us.gcr.io/replay-gaming/poker-client
    docker:
      dockerfile: docker/development/Dockerfile
    context: client
    sync:
      # switch to `infer` syntax after https://github.com/GoogleContainerTools/skaffold/issues/3376
      manual:
      - src: 'bin/**/*'
        dest: '.'
      - src: 'public/**/*'
        dest: '.'
      - src: 'src/**/*'
        dest: '.'

deploy:
  helm:
    releases:
    - name: rails
      chartPath: rails/charts/rails
      valuesFiles:
      - rails/charts/rails/values.development.yaml
      skipBuildDependencies: true
      values:
        'image.name': us.gcr.io/replay-gaming/rails
    - name: core
      chartPath: core/charts/poker-core
      valuesFiles:
      - core/charts/poker-core/values.development.yaml
      skipBuildDependencies: true
      values:
        'repository.image': us.gcr.io/replay-gaming/poker-core
    - name: api
      chartPath: api/charts/poker-api
      valuesFiles:
      - api/charts/poker-api/values.development.yaml
      skipBuildDependencies: true
      values:
        'image.repository': us.gcr.io/replay-gaming/poker-api



profiles:
  - name: dependencies
    deploy:
      helm:
        releases:
          #TODO: move it to some other path instead
          - name: redis
            chartPath: stable/redis
            remote: true
            valuesFiles:
              - support/skaffold/minikube/redis.yaml
          - name: postgresql
            chartPath: stable/postgresql
            remote: true
            valuesFiles:
              - support/skaffold/minikube/postgresql.yaml
          - name: memcached
            chartPath: stable/memcached
            remote: true
            valuesFiles:
              - support/skaffold/minikube/memcached.yaml
          - name: mysql
            chartPath: stable/mysql
            version: ~1.6.1
            remote: true
            valuesFiles:
              - support/skaffold/minikube/mysql.yaml
  # TODO: whap it in helm chart
  - name: pubsub_emulator
    build:
      artifacts:
        - image: us.gcr.io/replay-gaming/pubsub-emulator
          docker:
            dockerfile: support/skaffold/pubsub_emulator/Dockerfile
    deploy:
      kubectl:
        manifests:
          - support/skaffold/pubsub_emulator/k8s.yaml

  - name: rails
    build:
      artifacts:
        - image: us.gcr.io/replay-gaming/rails
          docker:
            dockerfile: docker/development/Dockerfile
          context: rails
          sync:
            # switch to `infer` syntax after https://github.com/GoogleContainerTools/skaffold/issues/3376
            manual:
              - src: 'app/**/*'
                dest: '.'
              - src: 'bin/**/*'
                dest: '.'
              - src: 'db/**/*'
                dest: '.'
              - src: 'lib/**/*'
                dest: '.'
              - src: 'public/**/*'
                dest: '.'
              - src: 'script/**/*'
                dest: '.'
              - src: 'spec/**/*'
                dest: '.'

    deploy:
      helm:
        releases:
          - name: rails
            chartPath: rails/charts/rails
            valuesFiles:
              - rails/charts/rails/values.development.yaml
            skipBuildDependencies: true
            values:
              'image.name': us.gcr.io/replay-gaming/rails

  - name: core
    build:
      artifacts:
        - image: us.gcr.io/replay-gaming/poker-core
          docker:
            dockerfile: docker/development/Dockerfile
          context: core
          sync:
            # switch to `infer` syntax after https://github.com/GoogleContainerTools/skaffold/issues/3376
            manual:
              - src: 'bin/**/*'
                dest: '.'
              - src: 'db/**/*'
                dest: '.'
              - src: 'lib/**/*'
                dest: '.'
              - src: 'system/**/*'
                dest: '.'
              - src: 'schemas/**/*'
                dest: '.'
    deploy:
      helm:
        releases:
          - name: core
            chartPath: core/charts/poker-core
            valuesFiles:
              - core/charts/poker-core/values.development.yaml
            skipBuildDependencies: true
            values:
              'repository.image': us.gcr.io/replay-gaming/poker-core

  - name: api
    build:
      artifacts:
        - image: us.gcr.io/replay-gaming/poker-api
          docker:
            dockerfile: docker/development/Dockerfile
          context: api
          sync:
            # switch to `infer` syntax after https://github.com/GoogleContainerTools/skaffold/issues/3376
            manual:
              - src: 'bin/**/*'
                dest: '.'
              - src: 'src/**/*'
                dest: '.'
              - src: 'tests/**/*'
                dest: '.'
    deploy:
      helm:
        releases:
          - name: api
            chartPath: api/charts/poker-api
            valuesFiles:
              - api/charts/poker-api/values.development.yaml
            skipBuildDependencies: true
            values:
              'image.repository': us.gcr.io/replay-gaming/poker-api

[petr-buchin]

@kvokka your issue is actually even simpler :)

According to the Skaffold documentation, when you provide build or deploy section in your profile, this section totally replaces your top-level configuration for the corresponding section.

But you could use json patch to get it working. Here is the docs page:

https://skaffold.dev/docs/environment/profiles/

You should look at Override via patches header here.

Your profile may look something like that:

profiles:
- name: pubsub_emulator
  patches:
  - op: add
    path: /build/artifacts/-
    value:
      image: us.gcr.io/replay-gaming/pubsub-emulator
      docker:
        dockerfile: support/skaffold/pubsub_emulator/Dockerfile
  - op: add
    path: /deploy/kubectl/manifests/-
    value: support/skaffold/pubsub_emulator/k8s.yaml

[kvokka]

Thank you for the advise, @petr-buchyn .
I have to use 2 separate profiles for dependencies kubectl && helm, so i can not put any of this parts to default build section for now (wait for #3392 ). But if I leave the build section empty, like:

apiVersion: skaffold/v2alpha1
kind: Config
build:
  artifacts: []
deploy:
  helm:
    releases: []

I can not use patches

$ skaffold build -p api
FATA[0000] creating runner: applying profiles: applying profile api: invalid path: /build/artifacts/-

Adding some value to builds fix the problem, but technically it is a spike and i do not need to build something by default.