Charts: [stable/jenkins] Backup/restore feature

[jpigree]

Is this a request for help?:
no

Is this a BUG REPORT or FEATURE REQUEST?:
Feature request

Adding a backup/restore mecanism could be great in the Jenkins chart (I had plenty of issues with backup jenkins plugins). Actually I implemented it quick and dirty with AWS S3 as a storage but never opensourced it. Is the community interested in this? I saw a rotten feature request #3344.

The configuration I created in the values.yaml looked like this:

backup:
  aws_access_key_id: "${BACKUP_ACCESS_KEY_ID}"
  aws_access_key_secret: "${BACKUP_ACCESS_KEY_SECRET}"
  retrieve:
    enabled: false
    name: jenkins.tar.gz
    path: "s3://${BACKUP_S3_BUCKET}/jenkins"
    image:
      repository: "devopsexperience/backup-retriever"
      tag: "1.0"
  upload:
    enabled: true
    name: jenkins.tar.gz
    path: "s3://${BACKUP_S3_BUCKET}/jenkins"
    schedule: "0 10 * * *"
    image:
      repository: "devopsexperience/backup-companion"
      tag: "1.0"

What the chart does is spawn a sidecar container which backups on schedule "/var/jenkins_home". And for the restore part, it is an initcontainer which fetches the archive, untar it then let the copy-default-config do its job.

I think, this gives the user an easy way to override too (change the container image). Will need to find a better way of passing the cloud provider credentials though.

If you are interested, I am willing to submit a PR.

[maorfr]

I think that we should go towards a solution that does not involve a sidecar.
We already have such a tool. I'll just add it :)

[jpigree]

Thanks. I looked at your PR. You used a Cronjob. Great. :)

However, I think it misses a restore mecanism. It is cool to backup our files but to be really useful, we will need to be able to create a Jenkins from those backups.

How would you do it? On my side, I did it with a initContainer capable of fetching the backups from storage and copy it to "/var/jenkins_home".

[maorfr]

Hey @jpigree,

Thanks for your feedback :)
A restore is absolutely required.
Do you think that the restore needs to be a part of the chart?

I usually like to keep restore close, so I do it manually.
Plus, this backup creates a tag for each backup, so you would have to define a tag to restore from, which makes it less suitable for a chart.

Anyway, the way you can restore is using the underlying skbn tool:
https://github.com/nuvo/skbn

Let me know what you think!

[jpigree]

Hi @maorfr . Actually, I implemented it in the chart and it always made me feel uneasy when using it.
Mostly, because the workflow required to update "retrieve" to "true", run the chart, then immediately update the "retrieve" field to "false" otherwise it will overwrite my data with the backup content when I do an upgrade.

I didn't understand the tags you are talking about. Are you talking about AWS object versions? So basically, you need to specify the object version you want to restore from. Looks okay to me.

I really like your way of doing the restore with skbn. :) I think this should be documented in the standard values.yaml though. For example, from what I understand the restore process will be something like this:

• "helm install" jenkins and wait for it to be available
• lock down jenkins in case someone tries to use it
• remove /var/jenkins_home
• copy the backup from storage to /var/jenkins_home in the jenkins pod (with skbn)
• reload jenkins configuration in the ui
• unlock jenkins

Is that it? It is a bit manual. Perhaps having a script around to do it would be useful but it shouldn't be in the chart I think. Perhaps in the jenkins repository?

[maorfr]

I've added some instruction to the chart's readme.
Now we wait for the maintainers ;)

[dbarochiya]

@jphalip @maorfr Hey,
Apologies if this doesn't make very much sense because I am very new to Kubernetes and Jenkins.

I am trying to deploy Jenkins on kubernetes with HA properties, it will be an implementation of active/standby architecture because of the way Jenkins works. I have thought of some options,

1) Create a Jenkins instance based on the base Docker image and give access to 'JENKINS HOME' stored in a shared folder to one of them.
2) Create a Jenkins instance based on the base Docker image and also the new 'JENKINS HOME' directory from the backup.

I believe the case 1 is the far better option, but I am not sure how to implement this and also which filesystem to choose such that it would be easy to share data without leading the state of inconsistency and also how to take backups.