Client-go: Minimizing memory usage

[alexec]

I'd like to keep the memory usage of my informer low. I'm introducing it to an existing app with low memory usage (<100MB) and I know (based on another app) that, the memory usage will become significantly larger. This is what I've come up with and I'd like to request comment:

1. I don't need the whole object. I just need the namespace and name (i.e. key).
2. I can filter most of the objects out based by looking at their contents.

I implemented a version of cache.KeyListerGetter that only stores the key:

// this struct is designed to store a compact cache of meta namespace keys (i.e. "namespace/name") created by an
// informer than can be iterated safely and efficiently
type store struct {
    lock sync.RWMutex
    keys map[string]bool
}

var _ cache.KeyListerGetter = &store{}

func newStore() *store {
    return &store{sync.RWMutex{}, make(map[string]bool)}
}

func (k *store) GetByKey(key string) (interface{}, bool, error) {
    k.lock.RLock()
    defer k.lock.RUnlock()
    _, exists := k.keys[key]
    return cache.ExplicitKey(key), exists, nil
}

func (k *store) Add(obj interface{}) {
    k.lock.Lock()
    defer k.lock.Unlock()
    key, err := cache.MetaNamespaceKeyFunc(obj)
    if err == nil { // error should never happen
        k.keys[key] = true
    }
}

func (k *store) Delete(obj interface{}) {
    k.lock.Lock()
    defer k.lock.Unlock()
    key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(obj)
    if err == nil { // error should never happen
        delete(k.keys, key)
    }
}

func (k *store) ListKeys() []string {
    k.lock.RLock()
    defer k.lock.RUnlock()
    var keys []string
    for key := range k.keys {
        keys = append(keys, key)
    }
    return keys
}

And a new factory that only uses that store and combines it with a filterFunc to further reduce the memory usage:

func NewFilterUsingKeyController(restClient rest.Interface, namespace string, req labels.Selector, resource string, objectType runtime.Object, filterFunc func(d cache.Delta) bool) (cache.Controller, cache.KeyLister) {
    knownObjects := newStore()
    return cache.New(&cache.Config{
        Queue: cache.NewDeltaFIFO(cache.MetaNamespaceKeyFunc, knownObjects),
        ListerWatcher: &cache.ListWatch{
            ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
                options.LabelSelector = req.String()
                return restClient.Get().
                    Namespace(namespace).
                    Resource(resource).
                    VersionedParams(&options, metav1.ParameterCodec).
                    Do().
                    Get()
            },
            WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
                options.LabelSelector = req.String()
                options.Watch = true
                return restClient.Get().
                    Namespace(namespace).
                    Resource(resource).
                    VersionedParams(&options, metav1.ParameterCodec).
                    Watch()
            },
        },
        ObjectType: objectType,
        // note - we never re-sync
        Process: func(obj interface{}) error {
            for _, d := range obj.(cache.Deltas) {
                switch d.Type {
                case cache.Added, cache.Updated:
                    if filterFunc(d) {
                        knownObjects.Add(d.Object)
                    } else {
                        knownObjects.Delete(d.Object)
                    }
                case cache.Deleted:
                    knownObjects.Delete(d.Object)
                }
            }
            return nil
        },
    }), knownObjects
}

Is there an easier way to achieve my goals?

[kensipe]

SIG API meeting indicated that metadata informer (k8s.io/client-go/metadata/metadatainformer)

[alexec]

That is good enough I think.

[lavalamp]

@DirectXMan12 might want to look and see if this would help controller-runtime / kubebuilder users, though.

[DirectXMan12]

We've got a parallel implementation going on due to not quite using the same factories to set up our informers, but yeah, that looks about right (tracking issue is https://github.com/kubernetes-sigs/controller-runtime/issues/1159)

[alexec]

Footnote: my suggestion will use less data than the meta-data only example. If a there is a lot of meta-data (and there can be) it maybe usefule.