Origin: Duplicate caches in OpenShift master

Created on 23 Mar 2016 · 72Comments · Source: openshift/origin

Here is a list of all unique reflector caches, their type, and the location that initializes them in origin.

*api.Build from github.com/openshift/origin/pkg/build/controller/factory/factory.go:125
*api.Build from github.com/openshift/origin/pkg/build/controller/factory/factory.go:186
*api.Build from github.com/openshift/origin/pkg/build/controller/factory/factory.go:74
*api.BuildConfig from github.com/openshift/origin/pkg/build/controller/factory/factory.go:282
*api.BuildConfig from github.com/openshift/origin/pkg/build/controller/factory/factory.go:317
*api.ClusterPolicy from github.com/openshift/origin/pkg/authorization/cache/clusterpolicy.go:32
*api.ClusterPolicyBinding from github.com/openshift/origin/pkg/authorization/cache/clusterpolicybinding.go:32
*api.ConfigMap from controller/resourcequota/resource_quota_controller.go:185
*api.DeploymentConfig from github.com/openshift/origin/pkg/deploy/controller/configchange/factory.go:43
*api.DeploymentConfig from github.com/openshift/origin/pkg/deploy/controller/deploymentconfig/factory.go:43
*api.DeploymentConfig from github.com/openshift/origin/pkg/deploy/controller/imagechange/factory.go:48
*api.Group from github.com/openshift/origin/pkg/user/cache/groups.go:38
*api.ImageStream from controller/resourcequota/resource_quota_controller.go:185
*api.ImageStream from github.com/openshift/origin/pkg/build/controller/factory/factory.go:279
*api.ImageStream from github.com/openshift/origin/pkg/deploy/controller/imagechange/factory.go:37
*api.ImageStream from github.com/openshift/origin/pkg/image/controller/factory.go:40
*api.LimitRange from k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:156
*api.LimitRange from k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:156
*api.Namespace from controller/namespace/namespace_controller.go:168
*api.Namespace from controller/serviceaccount/serviceaccounts_controller.go:140
*api.Namespace from github.com/openshift/origin/pkg/project/auth/cache.go:154
*api.Namespace from github.com/openshift/origin/pkg/project/cache/cache.go:84
*api.Namespace from github.com/openshift/origin/pkg/project/controller/factory.go:37
*api.Namespace from github.com/openshift/origin/pkg/security/controller/factory.go:40
*api.Namespace from k8s.io/kubernetes/plugin/pkg/admission/namespace/lifecycle/admission.go:114
*api.Node from controller/daemon/controller.go:212
*api.Node from controller/node/nodecontroller.go:222
*api.Node from k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:220
*api.PersistentVolume from controller/persistentvolume/persistentvolume_claim_binder_controller.go:459
*api.PersistentVolume from controller/persistentvolume/persistentvolume_recycler_controller.go:325
*api.PersistentVolume from k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:224
*api.PersistentVolumeClaim from controller/persistentvolume/persistentvolume_claim_binder_controller.go:464
*api.PersistentVolumeClaim from controller/resourcequota/resource_quota_controller.go:185
*api.PersistentVolumeClaim from k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:225
*api.Pod from controller/daemon/controller.go:211
*api.Pod from controller/endpoint/endpoints_controller.go:130
*api.Pod from controller/gc/gc_controller.go:81
*api.Pod from controller/job/controller.go:141
*api.Pod from controller/node/nodecontroller.go:223
*api.Pod from controller/replication/replication_controller.go:210
*api.Pod from controller/resourcequota/resource_quota_controller.go:185
*api.Pod from github.com/openshift/origin/pkg/build/controller/factory/factory.go:189
*api.Pod from github.com/openshift/origin/pkg/build/controller/factory/factory.go:240
*api.Pod from github.com/openshift/origin/pkg/deploy/controller/deployerpod/factory.go:57
*api.Pod from k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:213
*api.Pod from k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:216
*api.Policy from github.com/openshift/origin/pkg/authorization/cache/policy.go:32
*api.PolicyBinding from github.com/openshift/origin/pkg/authorization/cache/policybinding.go:32
*api.ReplicationController from controller/replication/replication_controller.go:209
*api.ReplicationController from controller/resourcequota/resource_quota_controller.go:185
*api.ReplicationController from github.com/openshift/origin/pkg/deploy/controller/deployerpod/factory.go:39
*api.ReplicationController from github.com/openshift/origin/pkg/deploy/controller/deployment/factory.go:50
*api.ReplicationController from k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:235
*api.ResourceQuota from controller/resourcequota/resource_quota_controller.go:182
*api.ResourceQuota from controller/resourcequota/resource_quota_controller.go:182
*api.ResourceQuota from k8s.io/kubernetes/plugin/pkg/admission/resourcequota/admission.go:86
*api.ResourceQuota from k8s.io/kubernetes/plugin/pkg/admission/resourcequota/admission.go:86
*api.Secret from controller/resourcequota/resource_quota_controller.go:185
*api.Secret from controller/serviceaccount/tokens_controller.go:145
*api.Secret from github.com/openshift/origin/pkg/serviceaccounts/controllers/deleted_dockercfg_secrets.go:73
*api.Secret from github.com/openshift/origin/pkg/serviceaccounts/controllers/deleted_token_secrets.go:67
*api.Secret from k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:118
*api.SecurityContextConstraints from github.com/openshift/origin/pkg/security/admission/admission.go:51
*api.SecurityContextConstraints from github.com/openshift/origin/pkg/security/admission/admission.go:51
*api.SecurityContextConstraints from github.com/openshift/origin/pkg/security/admission/admission.go:51
*api.Service from controller/endpoint/endpoints_controller.go:129
*api.Service from controller/resourcequota/resource_quota_controller.go:185
*api.Service from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:85
*api.Service from k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:230
*api.ServiceAccount from controller/serviceaccount/serviceaccounts_controller.go:139
*api.ServiceAccount from controller/serviceaccount/tokens_controller.go:144
*api.ServiceAccount from github.com/openshift/origin/pkg/serviceaccounts/controllers/create_dockercfg_secrets.go:83
*api.ServiceAccount from k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:102
*extensions.DaemonSet from controller/daemon/controller.go:210
*extensions.DaemonSet from controller/node/nodecontroller.go:224
*extensions.HorizontalPodAutoscaler from controller/podautoscaler/horizontal.go:121
*extensions.Job from controller/job/controller.go:140
*extensions.ReplicaSet from k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:240

  12 *api.Pod
   7 *api.Namespace
   5 *api.ReplicationController
   5 *api.Secret
   4 *api.ImageStream
   4 *api.ResourceQuota
   4 *api.Service
   4 *api.ServiceAccount
   3 *api.Build
   3 *api.DeploymentConfig
   3 *api.Node
   3 *api.PersistentVolume
   3 *api.PersistentVolumeClaim
   3 *api.SecurityContextConstraints
   2 *api.BuildConfig
   2 *api.LimitRange
   2 *extensions.DaemonSet
   1 *api.ClusterPolicy
   1 *api.ClusterPolicyBinding
   1 *api.ConfigMap
   1 *api.Group
   1 *api.Policy
   1 *api.PolicyBinding
   1 *extensions.HorizontalPodAutoscaler
   1 *extensions.Job
   1 *extensions.ReplicaSet

areperformance componenrestapi prioritP0

Source

smarterclayton

All 72 comments

We shouldn't have to cache replica set yet
ProjectCache looks like we are properly sharing (Only one watcher on Policy)
Pod is low hanging fruit - large cache, frequently changed, too many copies
We should audit the non-reactive caches (caches that are pure store) and share those first
We should be able to share namespace caches for most admission controllers - verify we are doing that

This is without Node watches or any extended admission controllers enabled.

smarterclayton on 23 Mar 2016

30s sync loops:

*api.Service from controller/endpoint/endpoints_controller.go:129
*extensions.Job from controller/job/controller.go:140
*api.ReplicationController from controller/replication/replication_controller.go:209
*extensions.DaemonSet from controller/daemon/controller.go:210
*extensions.HorizontalPodAutoscaler from controller/podautoscaler/horizontal.go:121

1m sync loops:

*api.Namespace from github.com/openshift/origin/pkg/project/controller/factory.go:37

2m sync loop:

*api.Build from github.com/openshift/origin/pkg/build/controller/factory/factory.go:186
*api.Build from github.com/openshift/origin/pkg/build/controller/factory/factory.go:74
*api.BuildConfig from github.com/openshift/origin/pkg/build/controller/factory/factory.go:282
*api.BuildConfig from github.com/openshift/origin/pkg/build/controller/factory/factory.go:317
*api.ClusterPolicy from github.com/openshift/origin/pkg/authorization/cache/clusterpolicy.go:32
*api.ClusterPolicyBinding from github.com/openshift/origin/pkg/authorization/cache/clusterpolicybinding.go:32
*api.DeploymentConfig from github.com/openshift/origin/pkg/deploy/controller/deploymentconfig/factory.go:43
*api.DeploymentConfig from github.com/openshift/origin/pkg/deploy/controller/imagechange/factory.go:48
*api.DeploymentConfig from github.com/openshift/origin/pkg/deploy/controller/configchange/factory.go:43
*api.Group from github.com/openshift/origin/pkg/user/cache/groups.go:38
*api.ImageStream from github.com/openshift/origin/pkg/build/controller/factory/factory.go:279
*api.ImageStream from github.com/openshift/origin/pkg/deploy/controller/imagechange/factory.go:37
*api.Namespace from github.com/openshift/origin/pkg/project/auth/cache.go:154
*api.Pod from github.com/openshift/origin/pkg/build/controller/factory/factory.go:189
*api.Pod from github.com/openshift/origin/pkg/deploy/controller/deployerpod/factory.go:57
*api.Policy from github.com/openshift/origin/pkg/authorization/cache/policy.go:32
*api.PolicyBinding from github.com/openshift/origin/pkg/authorization/cache/policybinding.go:32
*api.ReplicationController from github.com/openshift/origin/pkg/deploy/controller/deployment/factory.go:50
*api.ReplicationController from github.com/openshift/origin/pkg/deploy/controller/deployerpod/factory.go:39

For anything with a short list interval (<5m) instead of re-listing, we should be scanning the list again. We need to investigate whether any controllers mutate the in-memory store as part of normal operations in a way that they need the store to be refreshed, and separate those out.

smarterclayton on 23 Mar 2016

@jeremyeder @timothysc duplicated caches are the likely cause of controller problems on startup and excessive memory use on the controllers. This is not all admission controllers enabled (so there are probably others), but there is plenty of dead weight here. Sync loops should also be adjusted to avoid doing a list (there is no need in most cases). Will spawn appropriate issues.

smarterclayton on 23 Mar 2016

ack, this makes sense.

/cc @rrati

timothysc on 24 Mar 2016

@smarterclayton admission plugins probably only need caches, so those shouldn't be too hard. I'll think about how to refactor the controllers to share nicely.

deads2k on 24 Mar 2016

I've added duplicate cache sharing on the Kube 1.3 list since it
affects a few of them today.

smarterclayton on 24 Mar 2016

Data also rides across the scheduler as well. Per conversations today we need ~= L2 / L4 equiv cache. There is a ton of data in the objects that are not needed for caching.

timothysc on 24 Mar 2016

We had discussed the use of flyweight caches in a few other areas in
earlier issues but I agree.

On Thu, Mar 24, 2016 at 1:39 PM, Timothy St. Clair <[email protected]

wrote:

Data also rides across the scheduler as well. Per conversations today we
need ~= L2 / L4 equiv cache. There is a ton of data in the objects that are
not needed for caching.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/openshift/origin/issues/8229#issuecomment-200943580

smarterclayton on 24 Mar 2016

Admission plugin caches are largely an upstream problem. We'd need to do surgery upstream to collapse our caches. @derekwaynecarr

limitranger (kube) - limitranges (contained by origin admission plugins)
namespaceautoprovisioner (kube) - namespaces
namespacexists (kube) - namespaces
namespacelifecycle (kube) - namespaces
resourcequota (kube) - resourcequotas (contained by origin admission plugins)
serviceaccount (kube) - serviceaccounts,secrets
securitycontextcontraints (origin) - sccs (contained by others) This is a small set, so it doesn't much matter.

deads2k on 28 Mar 2016

I am +1 in porting the wants pattern we added in origin into upstream.

I think we can ignore NamespaceExists and NamespaceAutoProvision.

derekwaynecarr on 29 Mar 2016

In 1.4.0-alpha.0:

*api.Build (2m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:191
*api.Build (2m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:77
*api.Build (5m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:130
*api.BuildConfig (2m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:288
*api.BuildConfig (2m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:324
*api.ClusterPolicy (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:103
*api.ClusterPolicyBinding (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:103
*api.ClusterResourceQuota (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:103
*api.ConfigMap (12h0m0s) from github.com/openshift/origin/pkg/quota/controller/
*api.Group (2m0s) from github.com/openshift/origin/pkg/user/cache/groups.go:51
*api.ImageStream (10m0s) from github.com/openshift/origin/pkg/image/controller/factory.go:40
*api.ImageStream (2m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:285
*api.LimitRange (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154
*api.LimitRange (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154
*api.LimitRange (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:91
*api.Namespace (0s) from github.com/openshift/origin/pkg/project/cache/cache.go:96
*api.Namespace (0s) from pkg/controller/serviceaccount/serviceaccounts_controller.go:143
*api.Namespace (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:91
*api.Namespace (10m0s) from github.com/openshift/origin/pkg/security/controller/factory.go:40
*api.Namespace (1m0s) from github.com/openshift/origin/pkg/project/controller/factory.go:37
*api.Namespace (2m0s) from github.com/openshift/origin/pkg/project/auth/cache.go:200
*api.Namespace (5m0s) from pkg/controller/namespace/namespace_controller.go:199
*api.Node (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:395
*api.Node (0s) from pkg/controller/node/nodecontroller.go:389
*api.Node (21h43m39.27814564s) from pkg/controller/daemon/daemoncontroller.go:239
*api.PersistentVolume (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:399
*api.PersistentVolume (15s) from pkg/controller/volume/persistentvolume/controller_base.go:448
*api.PersistentVolumeClaim (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:400
*api.PersistentVolumeClaim (12h0m0s) from pkg/controller/resourcequota/resource_quota_controller.go:233
*api.PersistentVolumeClaim (15s) from pkg/controller/volume/persistentvolume/controller_base.go:449
*api.Pod (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:389
*api.Pod (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:392
*api.Pod (12h25m58.182480836s) from pkg/controller/podgc/gc_controller.go:89
*api.Pod (15h1m32.428210675s) from pkg/controller/deployment/deployment_controller.go:183
*api.Pod (2m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:194
*api.Pod (5m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:246
*api.Policy (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:103
*api.PolicyBinding (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:103
*api.ReplicationController (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:410
*api.ReplicationController (10m0s) from pkg/controller/replication/replication_controller.go:221
*api.ReplicationController (12h0m0s) from pkg/controller/resourcequota/resource_quota_controller.go:233
*api.ReplicationController (30s) from pkg/controller/disruption/disruption.go:266
*api.ResourceQuota (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83
*api.ResourceQuota (5m0s) from pkg/controller/resourcequota/resource_quota_controller.go:230
*api.Secret (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/deleted_dockercfg_secrets.go:73
*api.Secret (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/deleted_token_secrets.go:67
*api.Secret (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:134
*api.Secret (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119
*api.Secret (0s) from pkg/controller/serviceaccount/tokens_controller.go:179
*api.Secret (12h0m0s) from pkg/controller/resourcequota/resource_quota_controller.go:233
*api.SecurityContextConstraints (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:91
*api.Service (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:133
*api.Service (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:405
*api.Service (12h0m0s) from pkg/controller/resourcequota/resource_quota_controller.go:233
*api.Service (30m0s) from github.com/openshift/origin/pkg/dns/serviceaccessor.go:45
*api.Service (30s) from pkg/controller/endpoint/endpoints_controller.go:158
*api.ServiceAccount (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/create_dockercfg_secrets.go:121
*api.ServiceAccount (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103
*api.ServiceAccount (0s) from pkg/controller/serviceaccount/serviceaccounts_controller.go:142
*api.ServiceAccount (0s) from pkg/controller/serviceaccount/tokens_controller.go:178
*apps.PetSet (30s) from pkg/controller/petset/pet_set.go:147
*autoscaling.HorizontalPodAutoscaler (30s) from pkg/controller/podautoscaler/horizontal.go:128
*batch.Job (10m0s) from pkg/controller/job/jobcontroller.go:148
*extensions.DaemonSet (0s) from pkg/controller/node/nodecontroller.go:391
*extensions.DaemonSet (30s) from pkg/controller/daemon/daemoncontroller.go:237
*extensions.Deployment (30s) from pkg/controller/deployment/deployment_controller.go:181
*extensions.Deployment (30s) from pkg/controller/disruption/disruption.go:268
*extensions.ReplicaSet (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:415
*extensions.ReplicaSet (14h10m6.612471037s) from pkg/controller/deployment/deployment_controller.go:182
*extensions.ReplicaSet (30s) from pkg/controller/disruption/disruption.go:267
*extensions.ReplicaSet (30s) from pkg/controller/replicaset/replica_set.go:205
*policy.PodDisruptionBudget (30s) from pkg/controller/disruption/disruption.go:264
*storage.StorageClass (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:74
*storage.StorageClass (15s) from pkg/controller/volume/persistentvolume/controller_base.go:153

Which is:

   3 *api.Build
   2 *api.BuildConfig
   1 *api.ClusterPolicy
   1 *api.ClusterPolicyBinding
   1 *api.ClusterResourceQuota
   1 *api.ConfigMap
   1 *api.Group
   2 *api.ImageStream
   3 *api.LimitRange
   7 *api.Namespace
   3 *api.Node
   2 *api.PersistentVolume
   3 *api.PersistentVolumeClaim
   6 *api.Pod
   1 *api.Policy
   1 *api.PolicyBinding
   4 *api.ReplicationController
   2 *api.ResourceQuota
   6 *api.Secret
   1 *api.SecurityContextConstraints
   5 *api.Service
   4 *api.ServiceAccount
   1 *apps.PetSet
   1 *autoscaling.HorizontalPodAutoscaler
   1 *batch.Job
   2 *extensions.DaemonSet
   2 *extensions.Deployment
   4 *extensions.ReplicaSet
   1 *policy.PodDisruptionBudget
   2 *storage.StorageClass

smarterclayton on 16 Sep 2016

Which is 74 total.

smarterclayton on 16 Sep 2016

14 upstream reflectors (started in the /vendor/ package) and 60 downstream.

smarterclayton on 16 Sep 2016

@deads2k - bump. Currently a 3.4 blocker, will this make Friday or is this too risky?

@mfojtik FYI

pweil- on 8 Nov 2016

@deads2k - bump. Currently a 3.4 blocker, will this make Friday or is this too risky?

This is a long running effort. No, it will not be resolved in 3.4 or 3.5. Maybe 3.6.

deads2k on 8 Nov 2016

Well... we're... um... better:

api.Build (2m0s)
api.Build (2m0s)
api.Build (5m0s)
api.BuildConfig (10m0s)
api.BuildConfig (2m0s)
api.ClusterPolicy (10m0s)
api.ClusterPolicyBinding (10m0s)
api.ClusterResourceQuota (10m0s)
api.ConfigMap (12h0m0s)
api.ConfigMap (12h0m0s)
api.DeploymentConfig (10m0s)
api.Event (2h0m0s)
api.Group (2m0s)
api.ImageStream (10m0s)
api.ImageStream (10m0s)
api.ImageStream (2m0s)
api.LimitRange (10m0s)
api.Namespace (0s)
api.Namespace (10m0s)
api.Namespace (10m0s)
api.Namespace (1m0s)
api.Namespace (2m0s)
api.Namespace (5m0s)
api.Node (0s)
api.Node (0s)
api.Node (10m0s)
api.PersistentVolume (0s)
api.PersistentVolume (10m0s)
api.PersistentVolume (15s)
api.PersistentVolumeClaim (0s)
api.PersistentVolumeClaim (10m0s)
api.PersistentVolumeClaim (15s)
api.Pod (0s)
api.Pod (0s)
api.Pod (10m0s)
api.Pod (2m0s)
api.Pod (5m0s)
api.Policy (10m0s)
api.PolicyBinding (10m0s)
api.ReplicationController (0s)
api.ReplicationController (10m0s)
api.ReplicationController (10m0s)
api.ReplicationController (12h0m0s)
api.ReplicationController (12h0m0s)
api.ReplicationController (30s)
api.ResourceQuota (0s)
api.ResourceQuota (5m0s)
api.Secret (0s)
api.Secret (0s)
api.Secret (0s)
api.Secret (0s)
api.Secret (0s)
api.Secret (0s)
api.Secret (12h0m0s)
api.Secret (12h0m0s)
api.Secret (20m0s)
api.SecurityContextConstraints (10m0s)
api.Service (0s)
api.Service (0s)
api.Service (10m0s)
api.Service (12h0m0s)
api.Service (12h0m0s)
api.Service (20m0s)
api.Service (2m0s)
api.Service (30m0s)
api.Service (30s)
api.ServiceAccount (0s)
api.ServiceAccount (0s)
api.ServiceAccount (0s)
api.ServiceAccount (10m0s)
apps.StatefulSet (30s)
autoscaling.HorizontalPodAutoscaler (30s)
batch.Job (10m0s)
extensions.DaemonSet (10m0s)
extensions.Deployment (30s)
extensions.Deployment (30s)
extensions.ReplicaSet (0s)
extensions.ReplicaSet (10m0s)
extensions.ReplicaSet (30s)
policy.PodDisruptionBudget (30s)
storage.StorageClass (0s)
storage.StorageClass (15s)

which is

   3 api.Build
   2 api.BuildConfig
   1 api.ClusterPolicy
   1 api.ClusterPolicyBinding
   1 api.ClusterResourceQuota
   2 api.ConfigMap
   1 api.DeploymentConfig
   1 api.Event
   1 api.Group
   3 api.ImageStream
   1 api.LimitRange
   6 api.Namespace
   3 api.Node
   3 api.PersistentVolume
   3 api.PersistentVolumeClaim
   5 api.Pod
   1 api.Policy
   1 api.PolicyBinding
   6 api.ReplicationController
   2 api.ResourceQuota
   9 api.Secret
   1 api.SecurityContextConstraints
   9 api.Service
   4 api.ServiceAccount
   1 apps.StatefulSet
   1 autoscaling.HorizontalPodAutoscaler
   1 batch.Job
   1 extensions.DaemonSet
   2 extensions.Deployment
   3 extensions.ReplicaSet
   1 policy.PodDisruptionBudget
   2 storage.StorageClass

smarterclayton on 2 Feb 2017

So we're up to 82. But we only have 5 pod loops.

Secret is the really bad one - that's 9 loops times 9 secrets per namespace. Service would be the next one.

smarterclayton on 2 Feb 2017

api.Service (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:132
api.Service (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:470
api.Service (10m0s) from github.com/openshift/origin/pkg/service/controller/ingressip/controller.go:166
api.Service (12h0m0s) from github.com/openshift/origin/pkg/quota/controller/clusterquotareconciliation/reconciliation_controller.go:120
api.Service (12h0m0s) from pkg/controller/resourcequota/resource_quota_controller.go:232
api.Service (20m0s) from github.com/openshift/origin/pkg/service/controller/servingcert/secret_updating_controller.go:109
api.Service (2m0s) from github.com/openshift/origin/pkg/service/controller/servingcert/secret_creating_controller.go:118
api.Service (30m0s) from github.com/openshift/origin/pkg/dns/serviceaccessor.go:45
api.Service (30s) from pkg/controller/endpoint/endpoints_controller.go:160

api.Secret (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/create_dockercfg_secrets.go:221
api.Secret (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/deleted_dockercfg_secrets.go:73
api.Secret (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/deleted_token_secrets.go:67
api.Secret (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:133
api.Secret (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119
api.Secret (0s) from pkg/controller/serviceaccount/tokens_controller.go:178
api.Secret (12h0m0s) from github.com/openshift/origin/pkg/quota/controller/clusterquotareconciliation/reconciliation_controller.go:120
api.Secret (12h0m0s) from pkg/controller/resourcequota/resource_quota_controller.go:232
api.Secret (20m0s) from github.com/openshift/origin/pkg/service/controller/servingcert/secret_updating_controller.go:110

api.ImageStream (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:89
api.ImageStream (10m0s) from github.com/openshift/origin/pkg/image/controller/factory.go:40
api.ImageStream (2m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:304

Image stream is also likely to be hot.

smarterclayton on 2 Feb 2017

This is a scale regression - are we sure we should be moving this to 1.6?

smarterclayton on 2 Feb 2017

I think we should cherry-pick the bad ones (service/secret) as blockers unless you think everything in the list needs addressed in the next 6 days before freeze.

pweil- on 2 Feb 2017

I mostly want to get a determination that a) the fact that we have more is not a regression, and b) that the scale team is ok with us actually holding more things in memory (so memory scale performance gets worse in 1.5) and c) that we do the minimum due diligence as a team before moving it out.

Regressing total memory used would be a reason to hold a release in many cases, especially if we had promised to improve it for some use cases.

smarterclayton on 2 Feb 2017

👍1

/cc @mffiedler We'll setup the tests soon, but we're currently backed up tracing down regression + requests from online.

timothysc on 2 Feb 2017

I'm also concerned that this is as close to low hanging fruit to dramatically reduce memory use (and increase total scale) in the system, and we haven't been able to reap the benefits of the work upstream on this in the last few releases. If we can get an extra 10-20% of headroom in online with fairly mechanical work, that's a pretty good investment in my book. Doesn't mean that we're broken, but I'd rather spend time on this than any 6 features in the backlog.

smarterclayton on 3 Feb 2017

@smarterclayton @deads2k is there a shared Secret informer in kube? (can't see it)

mfojtik on 7 Feb 2017

Might be able to pull the generated one from 1.6.

On Tue, Feb 7, 2017 at 7:20 AM, Michal Fojtik notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton @deads2k
https://github.com/deads2k is there a shared Secret informer in kube?
(can't see it)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/origin/issues/8229#issuecomment-277982712,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH2BSh55c8GnKogdcsPwGBBwYo6V11S8ks5raGF_gaJpZM4H3aGO
.

deads2k on 7 Feb 2017

@mfojtik yes, in 1.6. We should sync up on an action plan for this.

ncdc on 7 Feb 2017

@smarterclayton @deads2k i was looking at the image controller/factory and the import controller use the RetryController which is not compatible (or hard to do) with shared informers... So to fix this for images we will have to rewrite the controller to use the shared informer approach. I'm kind of reluctant to do so few days before 1.5 cut.

For the secrets and services, we will have to pick the generated shared informers from kube 1.6 as @deads2k mentioned and plumb all the controllers as well, which does not seems to be low hanging fruit for me...

I'm more than happy to take this and fix as the first thing for 1.6, will you be ok with that? (of course that depends on the performance testing and if the memory usage is not increased significantly to clarify the "regression").

mfojtik on 7 Feb 2017

I highlighted those because they were the most impactful, but if there are others any reflector we remove that is O(namespace) is a net win. The more frequently updated they are the better that improvement is.

Agree image reflector is not safe for 1.5. Informer backport is probably unreasonable as well.

smarterclayton on 7 Feb 2017

@smarterclayton we're digging in, real stupid question before we start...

to reproduce these numbers, you're using pprof somehow? Or some other tool ?
i assume maybe on a real cluster deployment ?
the api.ImageStream (10m0s) from github.com format seems to be generated by some kind of tool but not sure its the same as the https://blog.golang.org/profiling-go-programs.

jayunit100 on 10 Feb 2017

sudo openshift start master --loglevel=4 2>&1 | grep reflector.go

Will print all of these

On Feb 10, 2017, at 4:42 PM, jay vyas notifications@github.com wrote:

@smarterclayton https://github.com/smarterclayton we're digging in, real
stupid question before we start... to reproduce this, you're using pprof
somehow? Or some other tool ?

i assume maybe on a real cluster deployment ? the api.ImageStream
(10m0s) from github.com format seems to be generated by some kind of
tool but not sure its the same as the
https://blog.golang.org/profiling-go-programs.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/origin/issues/8229#issuecomment-279073972,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p5QJHIg4dYuan_x7RN5RBsJNmoZBks5rbNm1gaJpZM4H3aGO
.

smarterclayton on 11 Feb 2017

👍1

@smarterclayton Some the entries seem confusing.

Starting reflector *api.Service (30s) from pkg/controller/endpoint/endpoints_controller.go:160

Shouldn't the path be starting from github.com/openshift/origin/vendor/k8s.io/kubernetes/ and then pkg... ?

ravisantoshgudimetla on 13 Feb 2017

That's just how the site logging is configured.

On Feb 13, 2017, at 5:08 PM, ravig notifications@github.com wrote:

@smarterclayton https://github.com/smarterclayton Some the entries seem
confusing.

Starting reflector *api.Service (30s) from
pkg/controller/endpoint/endpoints_controller.go:160

Shouldn't the path be starting from
github.com/openshift/origin/vendor/k8s.io/kubernetes/ and then pkg... ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/origin/issues/8229#issuecomment-279538760,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p6LtJHfdKIMvEPeJ-3mOVpuxRoNUks5rcNRFgaJpZM4H3aGO
.

smarterclayton on 13 Feb 2017

ok, these may be a stupid questions..

1.) Where is logging configured?
2.) Some of them are pointing correctly like github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go while some like the one mentioned in above comment are pointing incorrectly, shouldn't they be consistent or am I missing something?

ravisantoshgudimetla on 13 Feb 2017

--loglevel=4 or higher

Upstream reflectors behave that way. We don't change them downstream.

smarterclayton on 14 Feb 2017

Thanks @smarterclayton. I have few more questions though, @smarterclayton and @deads2k

I see that there are some shared informers in origin at pkg/controller/shared/.go. Some of them point to originclient(authorization, buildconfig, deploy, imagestream, quota) and some of them point to kubeclient. There is one more location(which is upstream - like vendor/k8sio/kubernetes/pkg/controllers/informers where some effort to share caches is present. So, will the files pointing to kubeclient be deleted in favor of the ones coming from upstream(once 1.6 lands) or these are for different purpose altogether?
I have been trying to make changes for create_docker_config_secret.go to use shared caching for "Secret". Shouldn't they have a sharedinformer as I didn't find anything in latest upstream.(Or is it a security threat because of which it is not being shared?) or did I miss anything here?

ravisantoshgudimetla on 16 Feb 2017

@ravisantoshgudimetla some of the informers do not have support upstream yet until 1.6. @mfojtik and I have some work in progress on some but it does require temporarily carrying a legacy informer until the 1.6 rebase. You can see that work in @mfojtik's branch that this commit builds on.

@mfojtik let's make sure we coordinate with @ravisantoshgudimetla so we aren't doing the same thing twice.

pweil- on 16 Feb 2017

@pweil- Thanks for the reply. I just looked at his branch and the work you people have been doing . is similar to I have been trying to do but in different location. I was thinking I have to raise a PR in coreinformers in Kubernetes. I didn't think about purpose of legacy informers in origin. Thanks for pointing that.

Sure, please let me know how you want to co-ordinate. I am on IRC in openshift-dev channel (username-rgudimet).

ravisantoshgudimetla on 16 Feb 2017

Well, we have the 3.6 number.

126

Mostly garbage collectors, will post the breakdown. This is too much of a regression for 3.6, so marking this p0 until we are sure we can fix or tolerate.

smarterclayton on 29 Apr 2017

I can probably get gc to use shared informers.

On Sat, Apr 29, 2017 at 10:06 AM Clayton Coleman notifications@github.com
wrote:

Well, we have the 3.6 number.

126

Mostly garbage collectors, will post the breakdown. This is too much of a
regression for 3.6, so marking this p0 until we are sure we can fix or
tolerate.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/openshift/origin/issues/8229#issuecomment-298171007,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAABYlvFnXsVLGjQXIvfrY8o_eSFzjIUks5r00PHgaJpZM4H3aGO
.

ncdc on 29 Apr 2017

👍1

Here's the breakdown. 61 are GC - I suspect many of those are duplicate virtual watches.

Of the rest, looks like it's the internal vs external watch thing, which we can mitigate with adapters if we have to, or just do the straight conversion:

*api.Build (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:98
*api.Build (2m0s) github.com/openshift/origin/pkg/build/controller/factory/factory.go:92
*api.Build (5m0s) github.com/openshift/origin/pkg/build/controller/factory/factory.go:147
*api.BuildConfig (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:98
*api.BuildConfig (2m0s) github.com/openshift/origin/pkg/build/controller/factory/factory.go:281
*api.ClusterPolicy (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.ClusterPolicyBinding (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.ClusterResourceQuota (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.DeploymentConfig (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:98
*api.Endpoints (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Event (2h0m0s) github.com/openshift/origin/pkg/unidling/controller/controller.go:195
*api.Group (2m0s) github.com/openshift/origin/pkg/user/cache/groups.go:54
*api.ImageStream (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:98
*api.ImageStream (10m0s) github.com/openshift/origin/pkg/image/controller/factory.go:40
*api.ImageStream (2m0s) github.com/openshift/origin/pkg/build/controller/factory/factory.go:232
*api.LimitRange (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Namespace (0s) github.com/openshift/origin/pkg/project/cache/cache.go:109
*api.Namespace (10m0s) github.com/openshift/origin/pkg/security/controller/factory.go:41
*api.Namespace (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Namespace (1m0s) github.com/openshift/origin/pkg/project/controller/factory.go:37
*api.Namespace (2m0s) github.com/openshift/origin/pkg/project/auth/cache.go:202
*api.Pod (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Policy (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.PolicyBinding (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.ReplicationController (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.ResourceQuota (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Secret (0s) github.com/openshift/origin/pkg/serviceaccounts/controllers/create_dockercfg_secrets.go:222
*api.Secret (0s) github.com/openshift/origin/pkg/serviceaccounts/controllers/deleted_dockercfg_secrets.go:74
*api.Secret (0s) github.com/openshift/origin/pkg/serviceaccounts/controllers/deleted_token_secrets.go:68
*api.Secret (0s) github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:134
*api.Secret (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Secret (20m0s) github.com/openshift/origin/pkg/service/controller/servingcert/secret_updating_controller.go:111
*api.Secret (2m0s) github.com/openshift/origin/pkg/service/controller/servingcert/secret_creating_controller.go:143
*api.SecurityContextConstraints (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Service (0s) github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:133
*api.Service (10m0s) github.com/openshift/origin/pkg/service/controller/ingressip/controller.go:170
*api.Service (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Service (20m0s) github.com/openshift/origin/pkg/service/controller/servingcert/secret_updating_controller.go:110
*api.Service (2m0s) github.com/openshift/origin/pkg/service/controller/servingcert/secret_creating_controller.go:142
*api.ServiceAccount (0s) github.com/openshift/origin/pkg/serviceaccounts/controllers/create_dockercfg_secrets.go:221
*api.ServiceAccount (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*storage.StorageClass (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*v1.ConfigMap (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.HorizontalPodAutoscaler (30s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Job (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Namespace (5m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Node (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.PersistentVolume (15s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.PersistentVolumeClaim (15s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Pod (0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451
*v1.Pod (0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:454
*v1.Pod (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ReplicationController (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ResourceQuota (5m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Secret (0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/serviceaccount/tokens_controller.go:180
*v1.Secret (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Service (30s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ServiceAccount (0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/serviceaccount/tokens_controller.go:179
*v1.ServiceAccount (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.DaemonSet (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Deployment (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.PodDisruptionBudget (30s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.ReplicaSet (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.StatefulSet (30s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.StorageClass (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

smarterclayton on 30 Apr 2017

ImgeStreams are covered. Not sure why we have two pods watches from the scheduler

smarterclayton on 30 Apr 2017

Also would be good to verify that the reflectors aren't actually reusing (and thus something we don't have to worry about). Pretty sure though.

Secrets, Namespaces, and Services are the biggest bang after that. Those also happen to be the worst ones to watch.

smarterclayton on 30 Apr 2017

I'm taking the secret, service account, and service ones now.

smarterclayton on 5 May 2017

GC controller: https://github.com/kubernetes/kubernetes/pull/45427

ncdc on 5 May 2017

Also fixing pods in the scheduler.

smarterclayton on 6 May 2017

Will finish with 47 left.

smarterclayton on 6 May 2017

(sans GC), so 19 removed.

smarterclayton on 6 May 2017

My changes in #14086

Excluding GC:

*api.Build (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:98
*api.Build (2m0s) github.com/openshift/origin/pkg/build/controller/factory/factory.go:92
*api.Build (5m0s) github.com/openshift/origin/pkg/build/controller/factory/factory.go:147
*api.BuildConfig (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:98
*api.BuildConfig (2m0s) github.com/openshift/origin/pkg/build/controller/factory/factory.go:281
*api.ClusterPolicy (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.ClusterPolicyBinding (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.ClusterResourceQuota (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.DeploymentConfig (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:98
*api.Endpoints (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Event (2h0m0s) github.com/openshift/origin/pkg/unidling/controller/unidling_controller.go:195
*api.Group (2m0s) github.com/openshift/origin/pkg/user/cache/groups.go:54
*api.ImageStream (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:98
*api.ImageStream (10m0s) github.com/openshift/origin/pkg/image/controller/factory.go:40
*api.ImageStream (2m0s) github.com/openshift/origin/pkg/build/controller/factory/factory.go:232
*api.LimitRange (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Namespace (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Pod (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Policy (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.PolicyBinding (10m0s) github.com/openshift/origin/pkg/controller/shared/shared_informer.go:110
*api.ReplicationController (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.ResourceQuota (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Secret (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.SecurityContextConstraints (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Service (0s) github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:121
*api.Service (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.ServiceAccount (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*storage.StorageClass (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*v1.ConfigMap (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.HorizontalPodAutoscaler (30s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Job (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Namespace (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Node (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.PersistentVolume (15s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.PersistentVolumeClaim (15s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Pod (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ReplicationController (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ResourceQuota (5m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Secret (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Service (30s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ServiceAccount (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.DaemonSet (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Deployment (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.PodDisruptionBudget (30s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.ReplicaSet (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.StatefulSet (30s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.StorageClass (10m0s) github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

smarterclayton on 6 May 2017

14321 contains the upstream fixes and is merged, #14086 contains the rest of my changes, andy landed kubernetes/kubernetes#45427 which needs cherry-pick, then we should be down to 60 or so.

smarterclayton on 25 May 2017

@smarterclayton https://github.com/openshift/origin/pull/13835 contain image shared informers.

mfojtik on 25 May 2017

GC backport is #14358

ncdc on 25 May 2017

After Andy's change and my changes, 98 total, 28 GC unstructured:

   3 *api.Build
   2 *api.BuildConfig
   2 *api.ClusterPolicy
   2 *api.ClusterPolicyBinding
   1 *api.ClusterResourceQuota
   1 *api.DeploymentConfig
   1 *api.Endpoints
   1 *api.Event
   1 *api.Group
   1 *api.ImageStream
   1 *api.LimitRange
   1 *api.Namespace
   1 *api.Pod
   2 *api.Policy
   2 *api.PolicyBinding
   1 *api.ReplicationController
   1 *api.ResourceQuota
   1 *api.Secret
   1 *api.SecurityContextConstraints
   2 *api.Service
   1 *api.ServiceAccount
   1 *api.Template
   1 *rbac.ClusterRole
   1 *rbac.ClusterRoleBinding
   1 *rbac.Role
   1 *rbac.RoleBinding
   1 *storage.StorageClass
   1 *v1.ConfigMap
   1 *v1.Endpoints
   1 *v1.HorizontalPodAutoscaler
   1 *v1.Job
   1 *v1.LimitRange
   1 *v1.Namespace
   1 *v1.Node
   1 *v1.PersistentVolume
   1 *v1.PersistentVolumeClaim
   1 *v1.Pod
   1 *v1.PodTemplate
   1 *v1.ReplicationController
   1 *v1.ResourceQuota
   1 *v1.Secret
   1 *v1.SecurityContextConstraints
   1 *v1.Service
   1 *v1.ServiceAccount
   1 *v1.StorageClass
   1 *v1alpha1.PodPreset
   1 *v1beta1.CertificateSigningRequest
   1 *v1beta1.ClusterRole
   1 *v1beta1.ClusterRoleBinding
   1 *v1beta1.DaemonSet
   2 *v1beta1.Deployment
   1 *v1beta1.Ingress
   1 *v1beta1.PodDisruptionBudget
   1 *v1beta1.PodSecurityPolicy
   1 *v1beta1.ReplicaSet
   1 *v1beta1.Role
   1 *v1beta1.RoleBinding
   1 *v1beta1.StatefulSet
   1 *v1beta1.StorageClass
   1 *v1beta1.ThirdPartyResource
   1 *v2alpha1.CronJob
  28 <nil>

Seeing some identical lines showing up - are we not properly reusing an informer factory?

*v1beta1.Deployment github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Deployment github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

The remaining things to target (besides getting rid of the rest of the GC reflectors) are:

*api.Build github.com/openshift/origin/pkg/build/controller/factory/factory.go:150
*api.Build github.com/openshift/origin/pkg/build/controller/factory/factory.go:93
*api.Build github.com/openshift/origin/pkg/controller/shared/shared_informer.go:97
*api.BuildConfig github.com/openshift/origin/pkg/build/controller/factory/factory.go:235
*api.BuildConfig github.com/openshift/origin/pkg/controller/shared/shared_informer.go:97
*api.ClusterPolicy github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.ClusterPolicy github.com/openshift/origin/pkg/controller/shared/shared_informer.go:109
*api.ClusterPolicyBinding github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.ClusterPolicyBinding github.com/openshift/origin/pkg/controller/shared/shared_informer.go:109
*api.Policy github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.Policy github.com/openshift/origin/pkg/controller/shared/shared_informer.go:109
*api.PolicyBinding github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.PolicyBinding github.com/openshift/origin/pkg/controller/shared/shared_informer.go:109

@deads2k I assume the policy ones are just us needing to remove our legacy shared informer code?

smarterclayton on 28 May 2017

Lists of actual dynamic resources left. Projects shouldn't be in this list. ScheduledJobs shouldn't be in this list.

I0528 10:55:51.846979   91779 graph_builder.go:88] Listing and watching dynamic resource {authorization.openshift.io v1 clusterpolicies}
I0528 10:55:51.847052   91779 graph_builder.go:88] Listing and watching dynamic resource {oauth.openshift.io v1 oauthaccesstokens}
I0528 10:55:51.847154   91779 graph_builder.go:88] Listing and watching dynamic resource {batch v2alpha1 scheduledjobs}
I0528 10:55:51.847226   91779 graph_builder.go:88] Listing and watching dynamic resource {user.openshift.io v1 identities}
I0528 10:55:51.847254   91779 graph_builder.go:88] Listing and watching dynamic resource {user.openshift.io v1 users}
I0528 10:55:51.847311   91779 graph_builder.go:88] Listing and watching dynamic resource {template.openshift.io v1 templates}
I0528 10:55:51.847357   91779 graph_builder.go:88] Listing and watching dynamic resource {oauth.openshift.io v1 oauthclients}
I0528 10:55:51.847468   91779 graph_builder.go:88] Listing and watching dynamic resource {apps.openshift.io v1 deploymentconfigs}
I0528 10:55:51.847586   91779 graph_builder.go:88] Listing and watching dynamic resource {build.openshift.io v1 buildconfigs}
I0528 10:55:51.847612   91779 graph_builder.go:88] Listing and watching dynamic resource {build.openshift.io v1 builds}
I0528 10:55:51.847730   91779 graph_builder.go:88] Listing and watching dynamic resource {image.openshift.io v1 images}
I0528 10:55:51.847785   91779 graph_builder.go:88] Listing and watching dynamic resource {network.openshift.io v1 netnamespaces}
I0528 10:55:51.847841   91779 graph_builder.go:88] Listing and watching dynamic resource {authorization.openshift.io v1 clusterpolicybindings}
I0528 10:55:51.847975   91779 graph_builder.go:88] Listing and watching dynamic resource {authorization.openshift.io v1 policybindings}
I0528 10:55:51.848073   91779 graph_builder.go:88] Listing and watching dynamic resource {image.openshift.io v1 imagestreams}
I0528 10:55:51.848146   91779 graph_builder.go:88] Listing and watching dynamic resource {oauth.openshift.io v1 oauthclientauthorizations}
I0528 10:55:51.848191   91779 graph_builder.go:88] Listing and watching dynamic resource {network.openshift.io v1 egressnetworkpolicies}
I0528 10:55:51.848763   91779 graph_builder.go:88] Listing and watching dynamic resource {network.openshift.io v1 hostsubnets}
I0528 10:55:51.848950   91779 graph_builder.go:88] Listing and watching dynamic resource {project.openshift.io v1 projects}
I0528 10:55:51.849010   91779 graph_builder.go:88] Listing and watching dynamic resource {quota.openshift.io v1 clusterresourcequotas}
I0528 10:55:51.849124   91779 graph_builder.go:88] Listing and watching dynamic resource {network.openshift.io v1 clusternetworks}
I0528 10:55:51.849152   91779 graph_builder.go:88] Listing and watching dynamic resource {authorization.openshift.io v1 rolebindingrestrictions}
I0528 10:55:51.849353   91779 graph_builder.go:88] Listing and watching dynamic resource {extensions v1beta1 networkpolicies}
I0528 10:55:51.849385   91779 graph_builder.go:88] Listing and watching dynamic resource {oauth.openshift.io v1 oauthauthorizetokens}
I0528 10:55:51.849461   91779 graph_builder.go:88] Listing and watching dynamic resource {extensions v1beta1 horizontalpodautoscalers}
I0528 10:55:51.849510   91779 graph_builder.go:88] Listing and watching dynamic resource {authorization.openshift.io v1 policies}
I0528 10:55:51.849598   91779 graph_builder.go:88] Listing and watching dynamic resource {route.openshift.io v1 routes}
I0528 10:55:51.849626   91779 graph_builder.go:88] Listing and watching dynamic resource {user.openshift.io v1 groups}

smarterclayton on 28 May 2017

I fixed a few of these by reusing ForResource and wrapping in #14391. I think the next step is to split the openshift shared informer

Listing and watching dynamic resource {apps.openshift.io v1 deploymentconfigs}
Listing and watching dynamic resource {authorization.openshift.io v1 rolebindingrestrictions}
Listing and watching dynamic resource {batch v2alpha1 scheduledjobs}
Listing and watching dynamic resource {build.openshift.io v1 buildconfigs}
Listing and watching dynamic resource {build.openshift.io v1 builds}
Listing and watching dynamic resource {extensions v1beta1 horizontalpodautoscalers}
Listing and watching dynamic resource {extensions v1beta1 networkpolicies}
Listing and watching dynamic resource {image.openshift.io v1 imagestreams}
Listing and watching dynamic resource {image.openshift.io v1 images}
Listing and watching dynamic resource {network.openshift.io v1 clusternetworks}
Listing and watching dynamic resource {network.openshift.io v1 egressnetworkpolicies}
Listing and watching dynamic resource {network.openshift.io v1 hostsubnets}
Listing and watching dynamic resource {network.openshift.io v1 netnamespaces}
Listing and watching dynamic resource {oauth.openshift.io v1 oauthaccesstokens}
Listing and watching dynamic resource {oauth.openshift.io v1 oauthauthorizetokens}
Listing and watching dynamic resource {oauth.openshift.io v1 oauthclientauthorizations}
Listing and watching dynamic resource {oauth.openshift.io v1 oauthclients}
Listing and watching dynamic resource {project.openshift.io v1 projects}
Listing and watching dynamic resource {quota.openshift.io v1 clusterresourcequotas}
Listing and watching dynamic resource {route.openshift.io v1 routes}
Listing and watching dynamic resource {user.openshift.io v1 groups}
Listing and watching dynamic resource {user.openshift.io v1 identities}
Listing and watching dynamic resource {user.openshift.io v1 users}

smarterclayton on 28 May 2017

Looks like we need a bug for GC tracking multiple resources that are backed by the same resource, as well as resources in multiple group versions. HPA is an example.

smarterclayton on 28 May 2017

Remaining after excluding projects and authorization token types

{apps.openshift.io v1 deploymentconfigs}
{authorization.openshift.io v1 rolebindingrestrictions}
{batch v2alpha1 scheduledjobs}
{build.openshift.io v1 buildconfigs}
{build.openshift.io v1 builds}
{extensions v1beta1 horizontalpodautoscalers}
{extensions v1beta1 networkpolicies}
{image.openshift.io v1 imagestreams}
{image.openshift.io v1 images}
{network.openshift.io v1 clusternetworks}
{network.openshift.io v1 egressnetworkpolicies}
{network.openshift.io v1 hostsubnets}
{network.openshift.io v1 netnamespaces}
{oauth.openshift.io v1 oauthclientauthorizations}
{oauth.openshift.io v1 oauthclients}
{quota.openshift.io v1 clusterresourcequotas}
{route.openshift.io v1 routes}
{user.openshift.io v1 groups}
{user.openshift.io v1 identities}
{user.openshift.io v1 users}

smarterclayton on 28 May 2017

@bparees @mfojtik @pweil- for us to not regress peak memory use in 3.6 we need to

fix the bug in GC that registers duplicate listeners for resources in two versions (hpa, cronjobs)
share informers in the comment above that already exist (create and start generated informers for those groups, then use the generated informers for both our own code plus the code above).

Right now we've grown the number of caches by 20% from 1.5 -> 3.6

smarterclayton on 1 Jun 2017

@deads2k still need to know why we have two informers for RBAC

smarterclayton on 1 Jun 2017

14562, #14391, and #14289 should get us to approximately 56:

*api.Build (5m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:150
*api.BuildConfig (2m0s) from github.com/openshift/origin/pkg/build/controller/factory/factory.go:235
*api.ClusterPolicy (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:111
*api.ClusterPolicyBinding (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:111
*api.ClusterResourceQuota (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:111
*api.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Event (2h0m0s) from github.com/openshift/origin/pkg/unidling/controller/unidling_controller.go:195
*api.Group (2m0s) from github.com/openshift/origin/pkg/user/cache/groups.go:54
*api.ImageStream (10m0s) from github.com/openshift/origin/pkg/image/generated/informers/internalversion/factory.go:45
*api.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Namespace (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Pod (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Policy (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:111
*api.PolicyBinding (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.PolicyBinding (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:111
*api.ReplicationController (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.ResourceQuota (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.SecurityContextConstraints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Service (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:122
*api.Service (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Template (10m0s) from github.com/openshift/origin/pkg/template/generated/informers/internalversion/factory.go:45
*rbac.ClusterRole (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*rbac.ClusterRoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*rbac.Role (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*rbac.RoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*storage.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*v1.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Job (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Namespace (5m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Node (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.PersistentVolume (15s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Pod (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Service (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.CertificateSigningRequest (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.ClusterRoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.DaemonSet (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Deployment (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Deployment (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Ingress (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.PodSecurityPolicy (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/
*v1beta1.ReplicaSet (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.RoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.StatefulSet (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.ThirdPartyResource (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
for resource {build.openshift.io v1 buildconfigs}
for resource {build.openshift.io v1 builds}
for resource {extensions v1beta1 horizontalpodautoscalers}
for resource {extensions v1beta1 networkpolicies}
for resource {route.openshift.io v1 routes}

and then I'll do the last one to switch builds to generated informers and we'll cut two more out to 54.

smarterclayton on 10 Jun 2017

The policy part was fixed

smarterclayton on 10 Jun 2017

Well, except for policy binding, will look.

kube-proxy is still using internal in 1.6 so there's not much point to fixing service and endpoints. I think we can turn all SA and secrets to v1, and possibly the namespace informers. That could give us 3 more back.

smarterclayton on 10 Jun 2017

Caches when we only start master api:

*api.ClusterPolicy (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.ClusterPolicyBinding (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.ClusterResourceQuota (10m0s) from github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45
*api.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Group (2m0s) from github.com/openshift/origin/pkg/user/cache/groups.go:54
*api.ImageStream (10m0s) from github.com/openshift/origin/pkg/image/generated/informers/internalversion/factory.go:45
*api.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Namespace (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Policy (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.PolicyBinding (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.ResourceQuota (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.SecurityContextConstraints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Service (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Template (10m0s) from github.com/openshift/origin/pkg/template/generated/informers/internalversion/factory.go:45
*storage.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70

Templates is because of #14216, I added a comment. Secrets is because of service account admission and enforce mountable secrets, which we rarely turn on.

Service and endpoints are DNS, which ideally we move back to the nodes post 3.6.

smarterclayton on 11 Jun 2017

Looks like we're back up to 68, mostly due to aggregation:

*api.Build (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:87
*api.BuildConfig (10m0s) from github.com/openshift/origin/pkg/controller/shared/shared_informer.go:87
*api.ClusterPolicy (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.ClusterPolicyBinding (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.ClusterResourceQuota (10m0s) from github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45
*api.DeploymentConfig (10m0s) from github.com/openshift/origin/pkg/deploy/generated/informers/internalversion/factory.go:45
*api.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Event (2h0m0s) from github.com/openshift/origin/pkg/unidling/controller/unidling_controller.go:195
*api.Group (2m0s) from github.com/openshift/origin/pkg/user/cache/groups.go:54
*api.ImageStream (10m0s) from github.com/openshift/origin/pkg/image/generated/informers/internalversion/factory.go:45
*api.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Namespace (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Pod (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Policy (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.PolicyBinding (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
*api.ReplicationController (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.ResourceQuota (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.SecurityContextConstraints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Service (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:122
*api.Service (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*api.Template (10m0s) from github.com/openshift/origin/pkg/template/generated/informers/internalversion/factory.go:45
*rbac.ClusterRole (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*rbac.ClusterRoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*rbac.Role (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*rbac.RoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*storage.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
*v1.ConfigMap (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.HorizontalPodAutoscaler (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Job (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Namespace (5m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Node (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.PersistentVolume (15s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.PersistentVolumeClaim (15s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Pod (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.PodTemplate (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ReplicationController (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ResourceQuota (5m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.SecurityContextConstraints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.Service (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1alpha1.PodPreset (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.CertificateSigningRequest (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.ClusterRole (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.ClusterRoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.DaemonSet (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Deployment (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Ingress (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.PodDisruptionBudget (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.PodSecurityPolicy (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.ReplicaSet (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.Role (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.RoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.StatefulSet (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v1beta1.ThirdPartyResource (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
*v2alpha1.CronJob (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
<nil> (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:214
<nil> (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:214
<nil> (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:214
<nil> (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:214
<nil> (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:214

Worst one is endpoints and services - aggregation is using v1, upstream kube-proxy hasn't moved to to it yet, so we're on it for DNS as well.

smarterclayton on 16 Jun 2017

Looks like ClusterRole, Role, ClusterRoleBinding, and RoleBinding are also new.

smarterclayton on 16 Jun 2017

StorageClass is also duplicated.

smarterclayton on 16 Jun 2017

This may be because of ordering with the started informers (aggregation I'm going to put up a patch, because endpoints are expensive and huge).

smarterclayton on 16 Jun 2017

https://github.com/openshift/origin/pull/14694

smarterclayton on 16 Jun 2017

https://github.com/openshift/origin/pull/14679

smarterclayton on 16 Jun 2017

Looking at controllers directly, we can shave a few more off (docker cfg service accounts and secrets). And then this is basically done.

smarterclayton on 16 Jun 2017

Found duplication mostly because of GC double loading, going to open a PR to avoid that. There's a set of remaining I have to triage. We have 35 singleton caches, and then a set of duplicates (the duplicates are all large and high volume). Each of those will get its own issue and then I'll close this.

<nil> (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:214
<nil> (0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:214

Triage
  *api.ResourceQuota (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.ResourceQuota (5m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

  *api.ReplicationController (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.ReplicationController (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

  *api.Pod (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Pod (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

  *api.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

  *api.Namespace (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Namespace (5m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

  *api.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

  *api.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

  *api.Service (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:122
  *api.Service (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Service (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

  *api.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

Triaged

Small
  Has Hardcoded references
    *v1beta1.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
    *v1.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
    *storage.StorageClass (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70

Singleton
  *v1beta1.ReplicaSet (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1beta1.StatefulSet (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v2alpha1.CronJob (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1beta1.PodDisruptionBudget (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1beta1.Ingress (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1beta1.DaemonSet (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1beta1.Deployment (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1alpha1.PodPreset (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1.ConfigMap (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1.PodTemplate (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1.PersistentVolume (15s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1.PersistentVolumeClaim (15s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1.HorizontalPodAutoscaler (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1.Node (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1.Job (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1beta1.PodSecurityPolicy (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *v1beta1.CertificateSigningRequest (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *api.Build (10m0s) from github.com/openshift/origin/pkg/build/generated/informers/internalversion/factory.go:45
  *api.BuildConfig (10m0s) from github.com/openshift/origin/pkg/build/generated/informers/internalversion/factory.go:45
  *api.ClusterPolicy (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
  *api.ClusterPolicyBinding (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
  *api.ClusterResourceQuota (10m0s) from github.com/openshift/origin/pkg/quota/generated/informers/internalversion/factory.go:45
  *api.DeploymentConfig (10m0s) from github.com/openshift/origin/pkg/deploy/generated/informers/internalversion/factory.go:45
  *api.Event (2h0m0s) from github.com/openshift/origin/pkg/unidling/controller/unidling_controller.go:195
  *api.Group (2m0s) from github.com/openshift/origin/pkg/user/cache/groups.go:54
  *api.ImageStream (10m0s) from github.com/openshift/origin/pkg/image/generated/informers/internalversion/factory.go:45
  *api.Policy (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
  *api.PolicyBinding (10m0s) from github.com/openshift/origin/pkg/authorization/generated/informers/internalversion/factory.go:45
  *api.SecurityContextConstraints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *api.Template (10m0s) from github.com/openshift/origin/pkg/template/generated/informers/internalversion/factory.go:45
  *extensions.NetworkPolicy (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *rbac.ClusterRole (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *rbac.ClusterRoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *rbac.Role (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *rbac.RoleBinding (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70

smarterclayton on 16 Jun 2017

  *api.ResourceQuota (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.ResourceQuota (5m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

Should be good

  *api.ReplicationController (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.ReplicationController (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

Deployments, going to try to refactor.

  *api.Pod (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Pod (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

Builds, I'm going to try to refactor.

  *api.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.LimitRange (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

Should be ok - only admission uses the internal version

  *api.Namespace (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Namespace (5m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

Everything in API should be using internal, everything in controller should be using namespace.

  *api.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Secret (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

Several controllers in origin using internal can be converted to v1, should mean we can get to no internal secret.

  *api.Service (0s) from github.com/openshift/origin/pkg/serviceaccounts/controllers/docker_registry_service.go:122
  *api.Service (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Service (30s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70
  *api.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.Endpoints (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

kube-proxy in 1.6 uses internal, which means dns must use internal, which means API aggregation must use internal. In 1.7 kube-proxy will switch to versioned, and the others should switch.

  *api.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70
  *v1.ServiceAccount (10m0s) from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70

Upstream uses Internal for admission, we use an internal for controller manager. Can be fixed

smarterclayton on 16 Jun 2017

Done, will investigate other wins in future releass

smarterclayton on 27 Jun 2017

Was this page helpful?

0 / 5 - 0 ratings