Velero: Support pre/post backup and restore hooks

[ncdc]

These will be pre and post the entire backup. Options for what to run include

• webhooks
• exec command in container
• create new run-once pod

[ncdc]

This can be achieved with external tooling watching for new/completed backups and restores. Will reopen if needed.

[manast]

@ncdc with watching do you mean using polling or is there any other mechanism that uses push instead?

[ashish-amarnath]

@manast Watching doesn't mean active polling in your tooling. You can use the dynamic informer apis in kubernetes client-go that allows you to register handlers for create, update, delete events for objects of a resource type. In this case you you would create a dynamic informer of the backup velero CRD and register update event handler. In the handler function, you would inspect the Phase field in the status and run custom logic based on what you find there.

[manast]

all right, I got it, thanks for the reply.

[manast]

Ok, although I understand exactly what you meant, I noticed there is zero documentation for the API regarding informers, and specially in the official javascript library (the one I am using), there is one example however since no documentation is available I do not know how to extrapolate it to registering the watchers for velero backup. Sorry to bother you with this, but if you know how to do it it would be great if you could give me a hint, this is the example code:

// tslint:disable:no-console
import * as k8s from '@kubernetes/client-node';

const kc = new k8s.KubeConfig();
kc.loadFromDefault();

const k8sApi = kc.makeApiClient(k8s.CoreV1Api);

const listFn = () => k8sApi.listNamespacedPod('default');

const informer = k8s.makeInformer(kc, '/api/v1/namespaces/default/pods', listFn);

informer.on('add', (obj: k8s.V1Pod) => { console.log(`Added: ${obj.metadata!.name}`); });
informer.on('update', (obj: k8s.V1Pod) => { console.log(`Updated: ${obj.metadata!.name}`); });
informer.on('delete', (obj: k8s.V1Pod) => { console.log(`Deleted: ${obj.metadata!.name}`); });
informer.on('error', (err: k8s.V1Pod) => {
  console.error(err);
  // Restart informer after 5sec
  setTimeout(() => {
    informer.start();
  }, 5000);
});

informer.start();

This is my loose attempt to register the informer:

import k8s from "@kubernetes/client-node";

const kc = new k8s.KubeConfig();
kc.loadFromDefault();

const k8sApi = kc.makeApiClient(k8s.CoreV1Api);

const listFn = () => k8sApi.listNamespacedPod("default");

const informer = k8s.makeInformer(
  kc,
  "velero.io/v1/Backup",
  listFn
);

obviously the listFn is wrong for particular case, also as I understand I should specify a watchFn instead but couldn't find any examples for that...

[manast]

Ok, just for the record in case some other newbie wonders how to do this:

curl -X GET $HOST/apis/velero.io/v1/namespaces/velero/backups/?watch=1 --header "Authorization: Bearer $TOKEN" --insecure

That will open a stream and the events will be emitted as they occur. Most libraries support some kind of watching, but I did not have any success with the javascript library so I implemented my own watcher.

[ashish-amarnath]

@manast Sorry about not responding sooner.
I haven't used the java script client much. But I also a wrote a small program to watch for velero backups.

const k8s = require('@kubernetes/client-node');

const kc = new k8s.KubeConfig();
kc.loadFromDefault();

const watch = new k8s.Watch(kc);
watch.watch('/apis/velero.io/v1/namespaces/velero/backups',
    // optional query parameters can go here.
    {},
    // callback is called for each received object.
    (type, obj) => {
        if (type === 'ADDED') {
            // tslint:disable-next-line:no-console
            console.log('new object:');
        } else if (type === 'MODIFIED') {
            // tslint:disable-next-line:no-console
            console.log('changed object:');
        } else if (type === 'DELETED') {
            // tslint:disable-next-line:no-console
            console.log('deleted object:');
        } else {
            // tslint:disable-next-line:no-console
            console.log('unknown type: ' + type);
        }
        // tslint:disable-next-line:no-console
        //console.log(obj);
    },
    // done callback is called if the watch terminates normally
    (err) => {
        // tslint:disable-next-line:no-console
        console.log(err);
    })
.then((req) => {
    // watch returns a request object which you can use to abort the watch.
   // setTimeout(() => { req.abort(); }, 20 * 1000);
});

A couple of things you may want to look into wrt the informer code snipper you shared in your earlier comment.

1. You will want to use the velero API equivalent. Unfortunately I don't know that would be.

const k8sApi = kc.makeApiClient(k8s.CoreV1Api);

1. The second parameter to the makeInformer function call should be /apis/velero.io/v1/namespaces/velero/backups
2. You will also need to update the listFn

Also, is there a reason why you are not opting to write this in go?

[manast]

@ashish-amarnath thank you for your answer. I tried a similar code to the one you pasted but I could not get the watch functionality to work, thats why settled on using my own.
The reason I am using javascript is because my backend is in typescript and it is easier for me to watch the changes of velero backups from there.

[manast]

ah, and regarding the informer, I spend a lot of time trying to make that work too, went through the source code to understand how it works, creating my own listenFn, but I gave up at the end.

[manast]

ah, I apologize, I was an idiot and missed that I did not have this uncommented:

// setTimeout(() => { req.abort(); }, 20 * 1000);

so it works as expected :).