Flux: initContainers not working

[kuburoman]

Init pods are not updated but normal containers are.

• Flux-image: abxregistry.azurecr.io/quay.io/weaveworks/flux:1.6.0*
• Fluxctl version: 1.5.0 (output from command line but installed 1.6.0)
• Kubernetes: 1.11.1
• Running on AKS

Fluxctl result

CONTROLLER                    CONTAINER  IMAGE                               RELEASE  POLICY
default:deployment/flux-test  flux-test  quay.io/stefanprodan/podinfo:0.2.1  ready    automated
                              init-pod   quay.io/stefanprodan/podinfo:0.2.0
                              init-pod2  quay.io/stefanprodan/podinfo:0.2.0

Test deployment.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    flux.weave.works/automated: "true"
    flux.weave.works/tag.init-pod: glob:develop-*
    flux.weave.works/tag.init-pod2: glob:develop-*
    flux.weave.works/tag.flux-test: glob:develop-*
  name: flux-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flux-test
  template:
    metadata:
      labels:
        app: flux-test
    spec:
      initContainers:
      - name: init-pod
        image: quay.io/stefanprodan/podinfo:0.2.0
        imagePullPolicy: Always
        command: ["echo", "test"]
      - name: init-pod2
        image: quay.io/stefanprodan/podinfo:0.2.0
        imagePullPolicy: Always
        command: ["echo", "test"]
      containers:
      - name: flux-test
        image: quay.io/stefanprodan/podinfo:0.2.1
        imagePullPolicy: Always
        command: ["caddy", "-agree", "--conf", "/etc/caddy/Caddyfile"]
        env:
        - name: ADMIN_USER
          value: admin
        - name: ADMIN_PASSWORD
          value: admin
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          limits:
            memory: 128Mi

[stefanprodan]

I've tested this and in my case the containers are not getting updated at all if there is a init container present.

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: default
  annotations:
    flux.weave.works/automated: "true"
    flux.weave.works/tag.podinfod: semver:~1.0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      initContainers:
        - name: init
          image: quay.io/stefanprodan/podinfo:1.0.0
          command:
          - ./podinfo
          - --version
      containers:
      - name: podinfod
        image: quay.io/stefanprodan/podinfo:1.0.0
        command:
        - ./podinfo
        - --port=9898
        ports:
        - name: http
          containerPort: 9898
          protocol: TCP

Flux logs:

ts=2018-09-07T10:19:53.589334403Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2018-09-07T10:19:53.659376572Z caller=images.go:79 component=sync-loop service=default:deployment/podinfo container=podinfod repo=quay.io/stefanprodan/podinfo pattern=semver:~1.0 current=quay.io/stefanprodan/podinfo:1.0.0 info="added update to automation run" new=quay.io/stefanprodan/podinfo:1.0.1 reason="latest 1.0.1 (2018-09-05 12:43:18.709127164 +0000 UTC) > current 1.0.0 (2018-08-22 10:06:03.175791271 +0000 UTC)"
ts=2018-09-07T10:19:53.659501319Z caller=images.go:79 component=sync-loop service=default:deployment/podinfo container=init repo=quay.io/stefanprodan/podinfo pattern=glob:* current=quay.io/stefanprodan/podinfo:1.0.0 info="added update to automation run" new=quay.io/stefanprodan/podinfo:1.0.1 reason="latest 1.0.1 (2018-09-05 12:43:18.709127164 +0000 UTC) > current 1.0.0 (2018-08-22 10:06:03.175791271 +0000 UTC)"
ts=2018-09-07T10:19:53.65958263Z caller=loop.go:108 component=sync-loop jobID=17ca419e-e057-2dbd-b44e-06fae410e11e state=in-progress
ts=2018-09-07T10:19:53.722048956Z caller=releaser.go:58 component=sync-loop jobID=17ca419e-e057-2dbd-b44e-06fae410e11e type=release updates=1
ts=2018-09-07T10:19:53.981945295Z caller=loop.go:118 component=sync-loop jobID=17ca419e-e057-2dbd-b44e-06fae410e11e state=done success=false err="verifying changes: failed to verify changes: the image for container \"init\" in resource \"default:deployment/podinfo\" should be \"quay.io/stefanprodan/podinfo:1.0.0\", but is \"quay.io/stefanprodan/podinfo:1.0.1\""

[squaremo]

I have reproduced this locally by adding an initContainer in a YAML in my fork of flux-example.

Also suspicious: it's not fetching image metadata for the initContainer:

$ fluxctl list-images -n hello -c deployment/helloworld
CONTROLLER                   CONTAINER   IMAGE                          CREATED
hello:deployment/helloworld  helloworld  weaveworks/hello-world         
                                         |   latest                     10 Aug 18 14:02 UTC
                                         '-> alpine                     19 Sep 17 15:25 UTC
                             init        quay.io/weaveworks/helloworld  image data not available

[squaremo]

Found it: https://github.com/weaveworks/flux/blame/master/cluster/kubernetes/resource/spec.go#L46
This lived longer than it ought to have, because there's no tests covering the release verification.