Flux: Document extra setup/permissions when using OpenShift

[squaremo]

When using

fluxd --ssh-keygen-dir=/var/fluxd/keygen --git-url=whatever

This shows in the logs:

ts=2018-06-06T13:43:35.453724903Z caller=main.go:132 version=1.4.0
ts=2018-06-06T13:43:35.58808823Z caller=main.go:208 err=“exit status 255”

The error in the log is from attempting to create the SSH keyring. After a bit of fiddling,

ts=2018-06-06T14:22:11.737167092Z caller=main.go:208 err=“failed to chmod identity file: chmod /etc/fluxd/ssh/identity: read-only file system”

It's supposed to be a read-only filesystem, so that's OK; but we should only try to chmod it if it's not already the correct permissions, so something went wrong with the mount, or the original config.

Apparently, doing

oc adm policy add-scc-to-user anyuid system:serviceaccount:flux:flux

(oc is the OpenShift command-line tool) and using readOnly: false for the secret volume fixes it, but I think this is a fudge at best -- it should be entirely happy with a read-only key.

[ZeeFiix]

You wrote "After a bit of fiddling...".
I have also the problem witch exit status 255.

@squaremo : How can i increase logging so i can see the reason of the exit status 255 ?

[squaremo]

I expect I meant that I put in extra logging. You could exec into the container and check the mode bits of the file -- that would at least tell you whether it's the exact problem described above.

I don't know how to fix it through; it seems that OpenShift has its own idea of how to mount volumes.

[severi]

I got flux up and running on Openshift (v3.9) by giving the service account permissions to run as privileged

oc adm policy add-scc-to-user privileged -z flux

and adding the following the the deployment.yaml

      containers:
      - name: flux
        securityContext:
          runAsUser: 0

Didn't need to change readOnly to false as stated in the issue.

This is most likely needed because Openshift normally runs the containers with a randomly generated user. Even when generating the ssh-key manually and mounting the secret with defaultMode: 0400 flux tried to run chmod on it. I debugged this a bit and it seems that the permissions for the file were actually -r--r----- 1 root 10011200 3369 Mar 1 13:53 identity. Flux probably check whether the permissions are 0400 and realizes they arent (0440) and therefore tries to change them and fails while doing it.

Also I had to create the empty secret ( flux-secret.yaml ) before creating the deployment.

Just a side note regarding other things I changed from the default installation:

• I ran flux in a single namespace and didn't give the service account permissions to other namespaces.
• added custom known_hosts as I'm using an enterprise github installation rather than github.com
• changed the default namespace that resources are deployed to

[hkalsi]

anyone able to make this work without added privileges for flux?

[jpellizzari]

FWIW I had the wrong defaultMode argument. I was using 400 instead of 0400, the latter being unsupported by JSON.

This config appears to be working for me:

      volumes:
      - name: git-key
        secret:
          defaultMode: 256
          secretName: flux-git-deploy

From the k8s docs:

Note that the JSON spec doesn’t support octal notation, so use the value 256 for 0400 permissions

https://kubernetes.io/docs/concepts/configuration/secret/

[richardcase]

I have this working with minimal changes.

@dholbach - would you like me to document the instructions to get this working? I can then update the instructions after i have made the fix for #2651

[dholbach]

@richardcase That would be fantastic.

I wonder where the best place for this is. The FAQ works of course, but it's getting a bit crowded there and it might not be "front and center" enough for OpenShift users.

[richardcase]

@dholbach - perhaps a new tutorial...Getting started with Flux on OpenShift?

[MansM]

When I got some time I can make a crude start.

[dholbach]

Sounds great!

[squaremo]

Thank you @richardcase and @MansM!

A nudge: this makes more sense as "how-to" documentation (called Guides in our table of contents), where

• tutorial = give people a script to work through so they learn some element of the software
• how-to = give people a script to accomplish a specific task

(arguably, yes, the "Get started" docs already in place are how-to documentation -- well spotted)

The distinction makes some difference to content and tone -- in a how-to, you are giving people a recipe for getting something specific done, so you don't need to spend a lot of time on explaining the broader picture.

[alekssd]

Can someone explain how he make it work without giving root usr to flux container ? I have problems with chmod on identity file on openshift, and I didn't find any acceptable solution for it.
Christmas gift :-)

[richardcase]

@alekssd - i've been working on some changes on the openshift side of things #2652

[alekssd]

I don't see what I should add to make it work while waiting for your commit beeing accepted and merged. Is it the new SCC ?

[nfickas]

@richardcase I am still seeing issues when attempting to install flux on OpenShift. I am using this fluxctl install command: fluxctl install \ --git-url=GIT_URL \ --git-branch=GIT_BRANCH \ --git-readonly=true \ --git-email=EMAIL\ --namespace=NAMESPACE\ --add-security-context \ --output-dir=fluxcd/

I continue to get the failed to chmod identity file: chmod /etc/fluxd/ssh/identity: read-only file system error. Are there any solutions to this problem?

[dataplex]

This also happens on kubernetes if the deployment is rolled out and the flux deployment loses it's emptyDir volume mount. I think most of this information can be provided via configmap, and perhaps it would be best if these directories were not mapped the way they are?