Flux: Secrets encrypted with git-secret are decrypted but not installed

[kodachimaru]

Describe the bug
I can't get working the git-secret integration with Flux.
The non encrypted resources are published to Kube just fine.
However, the secrets encrypted with git-secret are decrypted but not published

To Reproduce
Steps to reproduce the behaviour:

# Fork my repo on GitHub and replace it for the following steps
#   https://github.com/kodachimaru/gitops-demo-flux-4-git-secret.git

# Clone repo
git clone https://github.com/kodachimaru/gitops-demo-flux-4-git-secret.git
cd gitops-demo-flux-4-git-secret

# Switch to the proper branch
git checkout support-request

# Import the gpg key to verify the secret can be decrypted alright
gpg --import git-secret-demo.key

# Decrypt the secret
git secret cat the-secret.secret.yaml.secret
git secret reveal -f
        # apiVersion: v1
        # kind: Secret
        # metadata:
        #   name: the-secret
        #   namespace: staging
        # type: Opaque
        # stringData:
        #   secret-file.txt: |-
        #     THIS_IS_MY_SECRET_CONTENT

# Create minikube instance for the demo
minikube start -p flux-demo --memory 8192

# Create Secret from GPG key
kubectl create namespace flux
kubectl create secret generic git-secret-gpg-keys --from-file=git-secret-demo.key -n flux
kubectl describe secret git-secret-gpg-keys -n flux

# Init Helm
helm init

# Install Flux

helm repo add fluxcd https://fluxcd.github.io/flux

helm install \
    --name flux \
    --set helmOperator.create=true \
    --set helmOperator.createCRD=true \
    --set [email protected]:kodachimaru/gitops-demo-flux-4-git-secret.git \
    --set git.branch=support-request \
    --set git.pollInterval=5s \
    --set additionalArgs[0]="--git-secret" \
    --set gpgKeys.secretName=git-secret-gpg-keys \
    --namespace flux \
    fluxcd/flux

helm list flux
        # NAME  REVISION    UPDATED                     STATUS      CHART       APP VERSION NAMESPACE
        # flux  1           Sun Sep 22 18:54:50 2019    DEPLOYED    flux-0.14.1 1.14.2      flux     

# See GPG key imported
kubectl logs $(kubectl get pods -n flux -o name | grep flux | grep -v helm | grep -v memcached) -n flux | grep GPG
        # ts=2019-09-22T16:55:52.305875521Z caller=main.go:334 info="imported GPG key(s) from /root/gpg-import/private" files=[git-secret-demo.key]

# See git-secret being used
kubectl logs $(kubectl get pods -n flux -o name | grep flux | grep -v helm | grep -v memcached) -n flux | grep git-secret
        # ts=2019-09-22T16:55:52.441021379Z caller=main.go:623 [email protected]:kodachimaru/gitops-demo-flux-4-git-secret.git user="Weave Flux" [email protected] signing-key= verify-signatures=false sync-tag=flux-sync state=git readonly=false notes-ref=flux set-author=false git-secret=true

# Get deploy key from Flux installation
export FLUX_FORWARD_NAMESPACE=flux
fluxctl identity

# See Flux at work
kubectl logs -f $(kubectl get pods -n flux -o name | grep flux | grep -v helm | grep -v memcached) -n flux

# See the deployment is installed successfully
kubectl get all -n staging
        # NAME                                      READY   STATUS              RESTARTS   AGE
        # pod/busybox-deployment-5d65888d7f-bdf7z   0/1     ContainerCreating   0          5m42s
        # 
        # NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
        # deployment.apps/busybox-deployment   0/1     1            0           5m43s
        # 
        # NAME                                            DESIRED   CURRENT   READY   AGE
        # replicaset.apps/busybox-deployment-5d65888d7f   1         1         0       5m43s

# But not the secret!
# The pod obviously is not starting because the secret cannot be mounted in it as a volume
kubectl get secrets -n staging
        # NAME                  TYPE                                  DATA   AGE
        # default-token-jtlqn   kubernetes.io/service-account-token   3      7m11s

# Enter into the pod 
kubectl exec -it -n flux  $(kubectl get pods -n flux | grep flux | grep -v helm | grep -v memcached | cut -d " " -f 1) /bin/bash

# List the files in the "working" directory
# See that the secret is correctly revealed!
ls -la /tmp/*working*/
        # /tmp/flux-working953999195/:
        # total 44
        # drwx------    4 root     root          4096 Sep 22 17:05 .
        # drwxrwxrwt    1 root     root          4096 Sep 22 17:23 ..
        # drwxr-xr-x    8 root     root          4096 Sep 22 17:05 .git
        # -rw-r--r--    1 root     root            61 Sep 22 17:05 .gitignore
        # drwxr-xr-x    4 root     root          4096 Sep 22 17:05 .gitsecret
        # -rw-r--r--    1 root     root            87 Sep 22 17:05 README.md
        # -rw-r--r--    1 root     root          2519 Sep 22 17:05 git-secret-demo.key
        # -rw-r--r--    1 root     root           826 Sep 22 17:05 secret.deployment.yaml
        # -rw-r--r--    1 root     root            58 Sep 22 17:05 staging.namespace.yaml
        # -rw-r--r--    1 root     root           156 Sep 22 17:05 the-secret.secret.yaml       <==========
        # -rw-r--r--    1 root     root           475 Sep 22 17:05 the-secret.secret.yaml.secret

cat /tmp/flux-working*/the-secret.secret.yaml
        # apiVersion: v1
        # kind: Secret
        # metadata:
        #   name: the-secret
        #   namespace: staging
        # type: Opaque
        # stringData:
        #   secret-file.txt: |-
        #     THIS_IS_MY_SECRET_CONTENT

Expected behavior
The secret, after being successfully decrypted (verified), should have been published to Kube by Flux.

Logs

Flag --registry-poll-interval has been deprecated, changed to --automation-interval, use that instead

ts=2019-09-22T16:55:52.297688161Z caller=main.go:243 version=1.14.2

ts=2019-09-22T16:55:52.305875521Z caller=main.go:334 info="imported GPG key(s) from /root/gpg-import/private" files=[git-secret-demo.key]

ts=2019-09-22T16:55:52.305945781Z caller=main.go:372 msg="using in cluster config to connect to the cluster"

ts=2019-09-22T16:55:52.436668611Z caller=main.go:457 component=cluster identity=/var/fluxd/keygen/identity

ts=2019-09-22T16:55:52.436716894Z caller=main.go:458 component=cluster identity.pub="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD1HD6B/FNudvB6zW83kTkOGsshERnNl+CAk8rbynqHeDAQ4tR41iahjmyR8XfTEaL7j6Aar6gmGQm1q/K13jXATsMYO6+z+rwGtP4LkK0MXpcX9KZyMbPuG/QOvKuZsT5wAyuvG7S+u3hKWqOpCdbL7A1ZJIpVHJvv7G7umDB/bk88f4RNdyhoLsEYcgtQAgnaaJOe++tKDd6iwL/v1B6j4XtsKlarCrjUqO3RvgSSB2Snga2x/3I9+XiOSxBLRQuRWg2uWFUqFd2qvgtyP8EfyHUtlcKAApLwzP3BeyKtfCygV1BPljV6TWV9gzOgdFgzfaes1x6El+M9HTXclrjH"

ts=2019-09-22T16:55:52.436739074Z caller=main.go:463 host=https://10.96.0.1:443 version=kubernetes-v1.14.1

ts=2019-09-22T16:55:52.436804835Z caller=main.go:475 kubectl=/usr/local/bin/kubectl

ts=2019-09-22T16:55:52.43853157Z caller=main.go:487 ping=true

ts=2019-09-22T16:55:52.441021379Z caller=main.go:623 [email protected]:kodachimaru/gitops-demo-flux-4-git-secret.git user="Weave Flux" [email protected] signing-key= verify-signatures=false sync-tag=flux-sync state=git readonly=false notes-ref=flux set-author=false git-secret=true

ts=2019-09-22T16:55:52.45966871Z caller=main.go:724 upstream="no upstream URL given"

ts=2019-09-22T16:55:52.461150916Z caller=main.go:745 addr=:3030

ts=2019-09-22T16:55:52.462482737Z caller=loop.go:100 component=sync-loop err="git repo not ready: git repo has not been cloned yet"

ts=2019-09-22T16:55:52.462565024Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"

ts=2019-09-22T16:55:52.462582635Z caller=images.go:27 component=sync-loop msg="no automated workloads"

ts=2019-09-22T16:55:53.355463252Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.14.2

ts=2019-09-22T16:55:54.973728791Z caller=warming.go:198 component=warmer info="refreshing image" image=docker.io/fluxcd/flux tag_count=38 to_update=38 of_which_refresh=0 of_which_missing=38

ts=2019-09-22T16:55:57.462949444Z caller=loop.go:100 component=sync-loop err="git repo not ready: git clone --mirror: fatal: Could not read from remote repository., full output:\n Cloning into bare repository '/tmp/flux-gitclone765049297'...\[email protected]: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"

( ...  AFTER KEY DEPLOYED TO GITHUB REPO ... )

ts=2019-09-22T17:05:13.904423967Z caller=loop.go:126 component=sync-loop event=refreshed [email protected]:kodachimaru/gitops-demo-flux-4-git-secret.git branch=support-request HEAD=b935aee9e6a3ebb86f38860e2f7221355ffc0218

ts=2019-09-22T17:05:17.069596Z caller=sync.go:479 method=Sync cmd=apply args= count=2

ts=2019-09-22T17:05:17.449840138Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=380.097155ms err=null output="namespace/staging created\ndeployment.apps/busybox-deployment created"

ts=2019-09-22T17:05:17.465650432Z caller=daemon.go:683 component=daemon event="Sync: b935aee, no workloads changed" logupstream=false

ts=2019-09-22T17:05:21.260887092Z caller=loop.go:219 component=sync-loop state="tag flux-sync" old=0360d89784e552ff67e6ff471ad00b35c72814cc new=b935aee9e6a3ebb86f38860e2f7221355ffc0218

ts=2019-09-22T17:05:22.777187252Z caller=loop.go:126 component=sync-loop event=refreshed [email protected]:kodachimaru/gitops-demo-flux-4-git-secret.git branch=support-request HEAD=b935aee9e6a3ebb86f38860e2f7221355ffc0218

ts=2019-09-22T17:05:26.811865989Z caller=loop.go:126 component=sync-loop event=refreshed [email protected]:kodachimaru/gitops-demo-flux-4-git-secret.git branch=support-request HEAD=b935aee9e6a3ebb86f38860e2f7221355ffc0218

ts=2019-09-22T17:05:31.82711896Z caller=sync.go:479 method=Sync cmd=apply args= count=2

ts=2019-09-22T17:05:31.993944288Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=166.736ms err=null output="namespace/staging unchanged\ndeployment.apps/busybox-deployment unchanged"

Additional context
Add any other context about the problem here, e.g

• Flux version: 1.14.2
• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T18:55:03Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

• Minikube version: v1.0.1

[kodachimaru]

BTW, it's not only with Secrets. Any Kube template gets decrypted but not applied.

[hiddeco]

I think this may be due to the change described in #2429 (which contains a fix for mentioned problem but needs to be ironed into the right technical solution).

@arsiesys are you able to confirm this?

[arsiesys]

Hello ! Indeed, sadly the feature was added in the same time than a big change.. that made it not working 😅. Waiting the fix to be reviewed / improved you have two solutions :

1) take advantage of the manifest generation feature to execute a git secret reveal (I tested it, it work)

2) I can provide a link here of a working version of the docker image (that I use) waiting the fix to be published/improves if flux maintenainers are ok with that

[kodachimaru]

Hi @arsiesys

Thanks for the quick response.

As for now it's only for a POC demo at work, I'd be glad to know what's the docker image version that you use. :-)

Meanwhile I will watch for #2429 resolution.

Thanks,
K.

[arsiesys]

Hello :D,

The one I use is on our private repo but I pushed one I just built in here:
https://hub.docker.com/r/kyon/flux

I would recommend to just use it in the POC context as it contain the fix I proposed that may bring bug or unwanted behavior as it's still pending a review. It's based on the 1.14.2 version of flux. You may have to mount a known_host if you used github or a public git repository as I can't generate it from here during the docker build.. company firewall policy.. :)

You can also build this docker image from the sources I used to make the PR:
https://github.com/arsiesys/flux/tree/fix_gitsecret

If you want to support/help regarding the usage of the feature, don't hesitate to ask question.

[kodachimaru]

Awesome. Works like a charm now 👍

[hiddeco]

Closed via #2429.