Jx: use GitOps for the developer environment too

[jstrachan]

internally we're using GitOps for our production CI/CD environment (see https://github.com/jenkins-x/cloud-environments/tree/master/env-jx-infra) and we use GitOps for the Staging and Production environments; but we don't yet do the same for the development environment.

The extra use of GPG keys and encrypted secrets make things a bit harder though; but we should try make it easy for folks to opt into GitOps for the development environment too.

Mostly its a simpler way to manage addons & secrets really: https://github.com/futuresimple/helm-secrets everything else is in a chart or chart configuration (values.yaml) file.

[jstrachan]

one incremental move in that direction could be:

• when we install, generate a ConfigMap which contains the git repository, branch and folder used to install it (e.g. for the k8s provider)
• when doing a jx upgrade platform we can then use this ConfigMap to ensure we clone that git repo and branch, do a git pull and can then refer to the git myvalues.yaml file so that we upgrade with any additional configuration - e.g. if the user wants to add pod templates to jenkins or whatnot

[rawlingsj]

I wonder if the gitSecrets.yaml, adminSecrets.yaml and extraValues.yaml should be added to a secret that we get, decrypt and use during jx upgrade platform too? That way folks dont need to integrate with Vault or encrypted secrets in git repos?

The cluster will then have all the initial config and secrets that was used to install first time, so upgrades can reuse them. We could then build tooling that helps modify these secrets.

[jstrachan]

@rawlingsj great idea! I was wondering how we can make the GitOps for the development environment kinda painless really and something folks can incrementally adopt - so its easy to start out. That sounds good to me really!

[jstrachan]

along similar lines we should make the jx install / jx create cluster .... do a git pull in ~/.jx/cloud-environments rather than deleting it - so folks can easily work of their own forks etc

[rawlingsj]

along similar lines we should make the jx install / jx create cluster .... do a git pull in ~/.jx/cloud-environments rather than deleting it - so folks can easily work of their own forks etc

yeah on that - I've had issues if there's local uncommitted changes, so that's why I originally went with a clean ~/.jx/cloud-environments to cut down folks running into issues during the install. It might need to do more that a git pull and check for local uncommitted changes and stash / unstash maybe?

[jstrachan]

yeah; maybe we need a generic helper function gitPullAndOptionallyStash where we try to git pull and if that breaks with a conflict/error log a warning and give the user an option to stash their previous work or ignore the git pull error? (In batch mode we probably default to just ignore and log a warning?)

As we could use the same logic for both build pack and ~/.jx/cloud-environments really

[jstrachan]

relates to https://github.com/jenkins-x/jx/issues/512

[jstrachan]

ideally we'd move to Configuration as Code plugin for Jenkins so that it can be completely configured in the values.yaml file in YAML which can then be stored in git etc #474

[jstrachan]

I guess this issue can be closed once the terraform stuff is working: https://github.com/jenkins-x/jx/issues/845

[jstrachan]

a few more thoughts on how this might look.

We want to promote reusing the helm packaging and templating for the Jenkins X platform and extensions. So a git repo could look like...

extensions/
  someExtension/
    values.yaml - optional custom values to override
  anotherExtension/
    values.yaml - optional custom values to override

jenkins-x-platform/
    values.yaml - local configuration overrides
    flags.yaml - optional CLI arguments effectively passed into `jx install ....` maybe?
    templates/
       staging-env.yaml - the YAML of the staging environment
       production-env.yaml - the YAML of the production environment

Then some simple jx step dev apply or something would apply the jenkins-x-platform first (treating it a little special initially while we have the custom logic in jx install .... until we make it pure helm some day?) then iterate through all the extensions and apply them.

To be able to know what versions of each chart (platform + extensions) to use we could use requirements.yaml to list the app names, chart repo URL + version?

OR - should the entire dev environment just look like a single helm chart with composition like we do for environments? The main issue is there's still a fair bit of magic in jx install so we maybe want to treat jx install as a special case - though long term treating it like any old help chart would be awesome really!

So maybe we treat the platform specially for now, then put all apps/extensions into the extensions folder...

jenkins-x-platform/
    values.yaml - local configuration overrides
    flags.yaml - optional CLI arguments effectively passed into `jx install ....` maybe?
    templates/
       staging-env.yaml - the YAML of the staging environment
       production-env.yaml - the YAML of the production environment
extensions/
  requirements.yaml (contains someExtension + version + anotherExtension)
  values.yaml - lets us configure any value in any extension

Whenever we've a way to replace jx install with a chart we can move the platform itself into extensions?

[rawlingsj]

Yeah this is what I had in mind and started chatting to @garethjevans about it this morning.

I'm not totally sure yet on having

    templates/
       staging-env.yaml - the YAML of the staging environment
       production-env.yaml - the YAML of the production environment

in the same git repo though, I was thinking they'd be in each individual environments git repo

[jstrachan]

@rawlingsj if the Staging Environment CRD is in another git repo, how does the platform know where it is to setup the webhooks? :)

[jstrachan]

the Dev environment contains CRDs like GitService and Environment so I figured all the CRDs in the dev environment would live in the dev git repo?

[rawlingsj]

ah that's the CRD, I read staging-env.yaml as the custom environment values.yaml!

so yes agree with all a that :)

[jstrachan]

btw am trying out generating a git repo for the dev environment which looks like the staging + production git repositories from a helm perspective - lets see how this goes...

[jstrachan]

this is mostly complete now - we just need to add support for the vault operator so we can store the secrets.yaml in Vault

[msvticket]

Today I've tried jx install --no-tiller --prow --gitops --no-gitops-vault --provider=eks --namespace=core. (--no-gitops-vault since I bumped into #2527.)

But it doesn't seem to work to combine Prow with GitOps. The namespaces are not created when Prow is installed. Of some reason Prow gets installed in namespace default.

Installing Prow into namespace default

Then further along the installation fails with:

error: creating the environments: failed to create staging environment in namespace core: failed to add repo deniedboarding/environment-ore-staging to Prow config in namespace core-staging: namespaces "core" not found

[msvticket]

Trying again with installation type Static Master Jenkins and --no-tiller the application of the dev chart fails with a lot of errors like unable to decode "/../helm-template-workdir-686574220/jenkins-x/output/env/templates/dev-env.yaml": Object 'Kind' is missing in '....

It seem like the objects written to jenkins-x-dev-environment/env/templates doesn't get the correct format. So it seem like the marshalling doesn't work properly; the tags on the Environment struct doesn't seem to be heeded.

$ head jenkins-x-dev-environment/env/templates/dev-env.yaml
typemeta:
  kind: Environment
  apiversion: jenkins.io/v1
objectmeta:
  name: dev
  generatename: ""
  namespace: ""
  selflink: ""
  uid: ""
  resourceversion: ""

So, after these experiments I have to conclude that gitops for the dev environments is a bit from working...

[ccojocar]

A gitops mode is supported in the install command for dev environments, but It seems that the webhooks are not created after the dev environment installation.

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[warrenbailey]

FIxed

/close

[jenkins-x-bot]

@warrenbailey: Closing this issue.

In response to this:

FIxed

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gazal-k]

Can the gitops option be added to an existing jenkins-x? I can only see the option in jx create or jx install