Argo-cd: HPA objects can always be OutOfSync

Created on 1 Feb 2019  路  10Comments  路  Source: argoproj/argo-cd

HPA not being refreshed even if update is successful.
The source is a chart description.

{{- if .Values.scaling.enabled -}}
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
    chart: {{ .Values.name }}-{{ .Values.version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Values.name }}
  minReplicas: {{ .Values.scaling.minReplicas }}
  maxReplicas: {{ .Values.scaling.maxReplicas }}
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: {{ .Values.scaling.targetAverageUtilization.cpu }}
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: {{ .Values.scaling.targetAverageUtilization.memory }}
{{- end -}}

Logs from application controller.

time="2019-02-01T13:41:36Z" level=info msg="updated 'omniscient-gateway' operation (phase: Running)"
time="2019-02-01T13:41:36Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:dev,Prune:false,DryRun:false,SyncStrategy:nil,ParameterOverrides:[],Resources:[{autoscaling HorizontalPodAutoscaler omniscient-gateway}],}}" application=omniscient-gateway
time="2019-02-01T13:41:37Z" level=info msg="cache miss: apires|https://10.3.240.1:443"
time="2019-02-01T13:41:48Z" level=info msg="Comparing app omniscient-gateway state in cluster https://kubernetes.default.svc (namespace: kintohub)"
time="2019-02-01T13:41:48Z" level=info msg="Applying resource HorizontalPodAutoscaler/omniscient-gateway in cluster: https://10.3.240.1:443, namespace: kintohub"
time="2019-02-01T13:41:48Z" level=info msg="[kubectl --kubeconfig /dev/shm/473367794 -n kintohub apply -f - --dry-run]"
time="2019-02-01T13:41:49Z" level=info msg="Applying resource HorizontalPodAutoscaler/omniscient-gateway in cluster: https://10.3.240.1:443, namespace: kintohub"
time="2019-02-01T13:41:49Z" level=info msg="[kubectl --kubeconfig /dev/shm/524458921 -n kintohub apply -f -]"
time="2019-02-01T13:41:49Z" level=info msg="added resource HorizontalPodAutoscaler/omniscient-gateway status: Synced, message: horizontalpodautoscaler.autoscaling/omniscient-gateway configured" application=omniscient-gateway
time="2019-02-01T13:41:49Z" level=info msg="updated 'omniscient-gateway' operation (phase: Running)"
time="2019-02-01T13:41:49Z" level=info msg="Resuming in-progress operation. phase: Running, message: " application=omniscient-gateway
time="2019-02-01T13:41:56Z" level=info msg="Refreshing app status (force refresh)" application=omniscient-gateway
time="2019-02-01T13:42:00Z" level=info msg="Comparing app omniscient-gateway state in cluster https://kubernetes.default.svc (namespace: kintohub)"
time="2019-02-01T13:42:00Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: '' -> 'successfully synced'" application=omniscient-gateway
time="2019-02-01T13:42:00Z" level=info msg="updated 'omniscient-gateway' operation (phase: Succeeded)"
time="2019-02-01T13:42:00Z" level=info msg="Operation succeeded" application=omniscient-gateway reason=OperationCompleted type=Normal
time="2019-02-01T13:42:10Z" level=info msg="Comparing app omniscient-gateway state in cluster https://kubernetes.default.svc (namespace: kintohub)"
time="2019-02-01T13:42:10Z" level=info msg="Comparison result: prev: OutOfSync. current: OutOfSync" application=omniscient-gateway
time="2019-02-01T13:42:10Z" level=info msg="Update successful" application=omniscient-gateway
time="2019-02-01T13:42:10Z" level=info msg="Refreshing app status (force refresh)" application=omniscient-gateway
time="2019-02-01T13:42:25Z" level=info msg="Comparing app omniscient-gateway state in cluster https://kubernetes.default.svc (namespace: kintohub)"
time="2019-02-01T13:42:25Z" level=info msg="Comparison result: prev: OutOfSync. current: OutOfSync" application=omniscient-gateway
time="2019-02-01T13:42:25Z" level=info msg="Update successful" application=omniscient-gateway
time="2019-02-01T13:42:25Z" level=info msg="Refreshing app status (force refresh)" application=omniscient-gateway
time="2019-02-01T13:42:38Z" level=info msg="Comparing app omniscient-gateway state in cluster https://kubernetes.default.svc (namespace: kintohub)"
time="2019-02-01T13:42:39Z" level=info msg="Comparison result: prev: OutOfSync. current: OutOfSync" application=omniscient-gateway
time="2019-02-01T13:42:39Z" level=info msg="Update successful" application=omniscient-gateway
picture bappr

Most helpful comment

Filed upstream issue https://github.com/kubernetes/kubernetes/issues/74099.

picture jessesuen on 15 Feb 2019
❤2

All 10 comments

HPA not being refreshed even if update is successful.

Can you elaborate on this? Can you share redacted output of argocd app get APPNAME -o yaml?

jessesuen picture jessesuen on 1 Feb 2019

Sure, if you look at the log, you can see the last one being
time="2019-02-01T13:42:39Z" level=info msg="Update successful" application=omniscient-gateway
While still out of sync
time="2019-02-01T13:42:25Z" level=info msg="Comparison result: prev: OutOfSync. current: OutOfSync" application=omniscient-gateway

$ argocd app get omniscient-gateway -o yaml

metadata:
  annotations:
    applications.argoproj.io/refresh: 2019-02-01T16:00:25Z
  creationTimestamp: 2019-01-18T08:23:53Z
  generation: 1
  name: omniscient-gateway
  namespace: argocd
  resourceVersion: "4055111"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/argocd/applications/omniscient-gateway
  uid: 6404af08-1afa-11e9-8fdb-42010a8c0082
spec:
  destination:
    namespace: kintohub
    server: https://kubernetes.default.svc
  project: default
  source:
    environment: default
    path: charts/omniscient-gateway
    repoURL: https://[email protected]/kintohub/KintoInfra.git
    targetRevision: staging
    valuesFiles:
    - values-staging.yaml
status:
  comparisonResult:
    comparedAt: 2019-02-02T15:41:18Z
    comparedTo:
      environment: default
      path: charts/omniscient-gateway
      repoURL: https://[email protected]/kintohub/KintoInfra.git
      targetRevision: staging
      valuesFiles:
      - values-staging.yaml
    resources:
    - childLiveResources:
      - state: '{"apiVersion":"v1","kind":"Endpoints","metadata":{"creationTimestamp":"2019-02-01T15:26:49Z","labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway","namespace":"kintohub","resourceVersion":"3947123","selfLink":"/api/v1/namespaces/kintohub/endpoints/omniscient-gateway","uid":"cb1c7457-2635-11e9-91db-42010a8c003c"},"subsets":[{"addresses":[{"ip":"10.12.1.8","nodeName":"gke-cluster-staging-default-pool-464e3199-pwg9","targetRef":{"kind":"Pod","name":"omniscient-gateway-5b964659d7-rtdrr","namespace":"kintohub","resourceVersion":"3779313","uid":"11c1dbaf-2639-11e9-91db-42010a8c003c"}},{"ip":"10.12.2.235","nodeName":"gke-cluster-staging-default-pool-4bbeff2b-1xbr","targetRef":{"kind":"Pod","name":"omniscient-gateway-5b964659d7-zl5v7","namespace":"kintohub","resourceVersion":"3772194","uid":"15f9f0aa-2634-11e9-91db-42010a8c003c"}}],"ports":[{"name":"http","port":80,"protocol":"TCP"},{"name":"public","port":8082,"protocol":"TCP"},{"name":"websocket","port":8089,"protocol":"TCP"},{"name":"grpc","port":8090,"protocol":"TCP"}]}]}'
      health:
        status: Healthy
      liveState: '{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"omniscient-gateway\",\"applications.argoproj.io/app-name\":\"omniscient-gateway\",\"chart\":\"omniscient-gateway-0.0.1\",\"heritage\":\"Tiller\",\"release\":\"omniscient-gateway\"},\"name\":\"omniscient-gateway\",\"namespace\":\"kintohub\"},\"spec\":{\"ports\":[{\"name\":\"http\",\"port\":80,\"targetPort\":80},{\"name\":\"public\",\"port\":8082,\"targetPort\":8082},{\"name\":\"grpc\",\"port\":8090,\"targetPort\":8090},{\"name\":\"websocket\",\"port\":8089,\"targetPort\":8089}],\"selector\":{\"app\":\"omniscient-gateway\",\"release\":\"omniscient-gateway\"},\"type\":\"ClusterIP\"}}\n"},"creationTimestamp":"2019-02-01T15:26:49Z","labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway","namespace":"kintohub","resourceVersion":"3774697","selfLink":"/api/v1/namespaces/kintohub/services/omniscient-gateway","uid":"cb19f6e6-2635-11e9-ac59-42010a8c0022"},"spec":{"clusterIP":"10.79.5.242","ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":80},{"name":"public","port":8082,"protocol":"TCP","targetPort":8082},{"name":"grpc","port":8090,"protocol":"TCP","targetPort":8090},{"name":"websocket","port":8089,"protocol":"TCP","targetPort":8089}],"selector":{"app":"omniscient-gateway","release":"omniscient-gateway"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}}'
      status: Synced
      targetState: '{"apiVersion":"v1","kind":"Service","metadata":{"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway"},"spec":{"ports":[{"name":"http","port":80,"targetPort":80},{"name":"public","port":8082,"targetPort":8082},{"name":"grpc","port":8090,"targetPort":8090},{"name":"websocket","port":8089,"targetPort":8089}],"selector":{"app":"omniscient-gateway","release":"omniscient-gateway"},"type":"ClusterIP"}}'
    - childLiveResources:
      - state: '{"apiVersion":"v1","kind":"Endpoints","metadata":{"creationTimestamp":"2019-02-01T15:26:49Z","labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"api","namespace":"kintohub","resourceVersion":"3947121","selfLink":"/api/v1/namespaces/kintohub/endpoints/api","uid":"cb187b73-2635-11e9-91db-42010a8c003c"},"subsets":[{"addresses":[{"ip":"10.12.1.8","nodeName":"gke-cluster-staging-default-pool-464e3199-pwg9","targetRef":{"kind":"Pod","name":"omniscient-gateway-5b964659d7-rtdrr","namespace":"kintohub","resourceVersion":"3779313","uid":"11c1dbaf-2639-11e9-91db-42010a8c003c"}},{"ip":"10.12.2.235","nodeName":"gke-cluster-staging-default-pool-4bbeff2b-1xbr","targetRef":{"kind":"Pod","name":"omniscient-gateway-5b964659d7-zl5v7","namespace":"kintohub","resourceVersion":"3772194","uid":"15f9f0aa-2634-11e9-91db-42010a8c003c"}}],"ports":[{"name":"http","port":80,"protocol":"TCP"},{"name":"public","port":8082,"protocol":"TCP"},{"name":"websocket","port":8089,"protocol":"TCP"},{"name":"grpc","port":8090,"protocol":"TCP"}]}]}'
      health:
        status: Healthy
      liveState: '{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"omniscient-gateway\",\"applications.argoproj.io/app-name\":\"omniscient-gateway\",\"chart\":\"omniscient-gateway-0.0.1\",\"heritage\":\"Tiller\",\"release\":\"omniscient-gateway\"},\"name\":\"api\",\"namespace\":\"kintohub\"},\"spec\":{\"ports\":[{\"name\":\"http\",\"port\":80,\"targetPort\":80},{\"name\":\"public\",\"port\":8082,\"targetPort\":8082},{\"name\":\"grpc\",\"port\":8090,\"targetPort\":8090},{\"name\":\"websocket\",\"port\":8089,\"targetPort\":8089}],\"selector\":{\"app\":\"omniscient-gateway\",\"release\":\"omniscient-gateway\"},\"type\":\"ClusterIP\"}}\n"},"creationTimestamp":"2019-02-01T15:26:49Z","labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"api","namespace":"kintohub","resourceVersion":"3774693","selfLink":"/api/v1/namespaces/kintohub/services/api","uid":"cb150526-2635-11e9-ac59-42010a8c0022"},"spec":{"clusterIP":"10.79.0.76","ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":80},{"name":"public","port":8082,"protocol":"TCP","targetPort":8082},{"name":"grpc","port":8090,"protocol":"TCP","targetPort":8090},{"name":"websocket","port":8089,"protocol":"TCP","targetPort":8089}],"selector":{"app":"omniscient-gateway","release":"omniscient-gateway"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}}'
      status: Synced
      targetState: '{"apiVersion":"v1","kind":"Service","metadata":{"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"api"},"spec":{"ports":[{"name":"http","port":80,"targetPort":80},{"name":"public","port":8082,"targetPort":8082},{"name":"grpc","port":8090,"targetPort":8090},{"name":"websocket","port":8089,"targetPort":8089}],"selector":{"app":"omniscient-gateway","release":"omniscient-gateway"},"type":"ClusterIP"}}'
    - childLiveResources:
      - children:
        - state: '{"apiVersion":"v1","kind":"Pod","metadata":{"creationTimestamp":"2019-02-01T15:50:16Z","generateName":"omniscient-gateway-5b964659d7-","labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","pod-template-hash":"1652021583","release":"omniscient-gateway"},"name":"omniscient-gateway-5b964659d7-rtdrr","namespace":"kintohub","ownerReferences":[{"apiVersion":"apps/v1","blockOwnerDeletion":true,"controller":true,"kind":"ReplicaSet","name":"omniscient-gateway-5b964659d7","uid":"15f7b636-2634-11e9-91db-42010a8c003c"}],"resourceVersion":"3779313","selfLink":"/api/v1/namespaces/kintohub/pods/omniscient-gateway-5b964659d7-rtdrr","uid":"11c1dbaf-2639-11e9-91db-42010a8c003c"},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["omniscient-gateway"]}]},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"env":[{"name":"NAMESPACE_ENV","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ENV","value":"STAGING"},{"name":"GRAPHQL_HOST","value":"prisma:4466"},{"name":"GRPC_PROXY","value":"grpc-service"},{"name":"PROXY_GRPC_HOST_FORMAT","value":"http://grpc-http-proxy:3000"},{"name":"RABBIT_HOST","value":"rabbitmq-ha"},{"name":"PROXY_GRPC_X_ACCESS_TOKEN","valueFrom":{"secretKeyRef":{"key":"grpc-http-proxy-token","name":"omniscient-gateway"}}}],"image":"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4","imagePullPolicy":"IfNotPresent","name":"main","ports":[{"containerPort":80,"name":"http","protocol":"TCP"},{"containerPort":8082,"name":"public","protocol":"TCP"},{"containerPort":8090,"name":"grpc","protocol":"TCP"},{"containerPort":8089,"name":"websocket","protocol":"TCP"}],"resources":{"requests":{"cpu":"100m","memory":"256Mi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","name":"default-token-qtxkd","readOnly":true}]}],"dnsPolicy":"ClusterFirst","imagePullSecrets":[{"name":"regsecret"}],"nodeName":"gke-cluster-staging-default-pool-464e3199-pwg9","priority":0,"restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30,"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300}],"volumes":[{"name":"default-token-qtxkd","secret":{"defaultMode":420,"secretName":"default-token-qtxkd"}}]},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2019-02-01T15:50:16Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2019-02-01T15:50:19Z","status":"True","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"status":"True","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2019-02-01T15:50:16Z","status":"True","type":"PodScheduled"}],"containerStatuses":[{"containerID":"docker://5d17bb8b4d785d5e7ffea09605cff93206f349bae005675dbfae1c79ae1f4c6d","image":"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4","imageID":"docker-pullable://asia.gcr.io/linear-bounty-174410/omniscient-gateway@sha256:9c776cb1d07cdb867c6d0b608b718100f60d11a3f5fb4720c844dd2fb02952a4","lastState":{},"name":"main","ready":true,"restartCount":0,"state":{"running":{"startedAt":"2019-02-01T15:50:18Z"}}}],"hostIP":"10.140.0.50","phase":"Running","podIP":"10.12.1.8","qosClass":"Burstable","startTime":"2019-02-01T15:50:16Z"}}'
        - state: '{"apiVersion":"v1","kind":"Pod","metadata":{"creationTimestamp":"2019-02-01T15:14:36Z","generateName":"omniscient-gateway-5b964659d7-","labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","pod-template-hash":"1652021583","release":"omniscient-gateway"},"name":"omniscient-gateway-5b964659d7-zl5v7","namespace":"kintohub","ownerReferences":[{"apiVersion":"apps/v1","blockOwnerDeletion":true,"controller":true,"kind":"ReplicaSet","name":"omniscient-gateway-5b964659d7","uid":"15f7b636-2634-11e9-91db-42010a8c003c"}],"resourceVersion":"3772194","selfLink":"/api/v1/namespaces/kintohub/pods/omniscient-gateway-5b964659d7-zl5v7","uid":"15f9f0aa-2634-11e9-91db-42010a8c003c"},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["omniscient-gateway"]}]},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"env":[{"name":"NAMESPACE_ENV","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ENV","value":"STAGING"},{"name":"GRAPHQL_HOST","value":"prisma:4466"},{"name":"GRPC_PROXY","value":"grpc-service"},{"name":"PROXY_GRPC_HOST_FORMAT","value":"http://grpc-http-proxy:3000"},{"name":"RABBIT_HOST","value":"rabbitmq-ha"},{"name":"PROXY_GRPC_X_ACCESS_TOKEN","valueFrom":{"secretKeyRef":{"key":"grpc-http-proxy-token","name":"omniscient-gateway"}}}],"image":"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4","imagePullPolicy":"IfNotPresent","name":"main","ports":[{"containerPort":80,"name":"http","protocol":"TCP"},{"containerPort":8082,"name":"public","protocol":"TCP"},{"containerPort":8090,"name":"grpc","protocol":"TCP"},{"containerPort":8089,"name":"websocket","protocol":"TCP"}],"resources":{"requests":{"cpu":"100m","memory":"256Mi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","name":"default-token-qtxkd","readOnly":true}]}],"dnsPolicy":"ClusterFirst","imagePullSecrets":[{"name":"regsecret"}],"nodeName":"gke-cluster-staging-default-pool-4bbeff2b-1xbr","priority":0,"restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30,"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300}],"volumes":[{"name":"default-token-qtxkd","secret":{"defaultMode":420,"secretName":"default-token-qtxkd"}}]},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2019-02-01T15:14:36Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2019-02-01T15:14:41Z","status":"True","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":null,"status":"True","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2019-02-01T15:14:36Z","status":"True","type":"PodScheduled"}],"containerStatuses":[{"containerID":"docker://4f54e0e071fc305dac29b2673d217d565c0c6c28bf2491660e424e7883089646","image":"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4","imageID":"docker-pullable://asia.gcr.io/linear-bounty-174410/omniscient-gateway@sha256:9c776cb1d07cdb867c6d0b608b718100f60d11a3f5fb4720c844dd2fb02952a4","lastState":{},"name":"main","ready":true,"restartCount":0,"state":{"running":{"startedAt":"2019-02-01T15:14:40Z"}}}],"hostIP":"10.140.0.59","phase":"Running","podIP":"10.12.2.235","qosClass":"Burstable","startTime":"2019-02-01T15:14:36Z"}}'
        state: '{"apiVersion":"extensions/v1beta1","kind":"ReplicaSet","metadata":{"annotations":{"deployment.kubernetes.io/desired-replicas":"2","deployment.kubernetes.io/max-replicas":"3","deployment.kubernetes.io/revision":"2"},"creationTimestamp":"2019-02-01T15:14:36Z","generation":8,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","pod-template-hash":"1652021583","release":"omniscient-gateway"},"name":"omniscient-gateway-5b964659d7","namespace":"kintohub","ownerReferences":[{"apiVersion":"apps/v1","blockOwnerDeletion":true,"controller":true,"kind":"Deployment","name":"omniscient-gateway","uid":"6d6f6c6a-1afa-11e9-8fdb-42010a8c0082"}],"resourceVersion":"3947120","selfLink":"/apis/extensions/v1beta1/namespaces/kintohub/replicasets/omniscient-gateway-5b964659d7","uid":"15f7b636-2634-11e9-91db-42010a8c003c"},"spec":{"replicas":2,"selector":{"matchLabels":{"app":"omniscient-gateway","pod-template-hash":"1652021583","release":"omniscient-gateway"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","pod-template-hash":"1652021583","release":"omniscient-gateway"}},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["omniscient-gateway"]}]},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"env":[{"name":"NAMESPACE_ENV","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ENV","value":"STAGING"},{"name":"GRAPHQL_HOST","value":"prisma:4466"},{"name":"GRPC_PROXY","value":"grpc-service"},{"name":"PROXY_GRPC_HOST_FORMAT","value":"http://grpc-http-proxy:3000"},{"name":"RABBIT_HOST","value":"rabbitmq-ha"},{"name":"PROXY_GRPC_X_ACCESS_TOKEN","valueFrom":{"secretKeyRef":{"key":"grpc-http-proxy-token","name":"omniscient-gateway"}}}],"image":"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4","imagePullPolicy":"IfNotPresent","name":"main","ports":[{"containerPort":80,"name":"http","protocol":"TCP"},{"containerPort":8082,"name":"public","protocol":"TCP"},{"containerPort":8090,"name":"grpc","protocol":"TCP"},{"containerPort":8089,"name":"websocket","protocol":"TCP"}],"resources":{"requests":{"cpu":"100m","memory":"256Mi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","imagePullSecrets":[{"name":"regsecret"}],"restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}}},"status":{"availableReplicas":2,"fullyLabeledReplicas":2,"observedGeneration":8,"readyReplicas":2,"replicas":2}}'
      - state: '{"apiVersion":"extensions/v1beta1","kind":"ReplicaSet","metadata":{"annotations":{"deployment.kubernetes.io/desired-replicas":"2","deployment.kubernetes.io/max-replicas":"3","deployment.kubernetes.io/revision":"1"},"creationTimestamp":"2019-01-18T08:24:09Z","generation":5,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","pod-template-hash":"803747934","release":"omniscient-gateway"},"name":"omniscient-gateway-d47c8cf78","namespace":"kintohub","ownerReferences":[{"apiVersion":"apps/v1","blockOwnerDeletion":true,"controller":true,"kind":"Deployment","name":"omniscient-gateway","uid":"6d6f6c6a-1afa-11e9-8fdb-42010a8c0082"}],"resourceVersion":"3772206","selfLink":"/apis/extensions/v1beta1/namespaces/kintohub/replicasets/omniscient-gateway-d47c8cf78","uid":"6d73731b-1afa-11e9-8fdb-42010a8c0082"},"spec":{"replicas":0,"selector":{"matchLabels":{"app":"omniscient-gateway","pod-template-hash":"803747934","release":"omniscient-gateway"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","pod-template-hash":"803747934","release":"omniscient-gateway"}},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["omniscient-gateway"]}]},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"env":[{"name":"NAMESPACE_ENV","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ENV","value":"STAGING"},{"name":"GRAPHQL_HOST","value":"prisma:4466"},{"name":"GRPC_PROXY","value":"grpc-service"},{"name":"PROXY_GRPC_HOST_FORMAT","value":"http://grpc-http-proxy:3000"},{"name":"RABBIT_HOST","value":"rabbitmq-ha"}],"image":"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4","imagePullPolicy":"IfNotPresent","name":"main","ports":[{"containerPort":80,"name":"http","protocol":"TCP"},{"containerPort":8082,"name":"public","protocol":"TCP"},{"containerPort":8090,"name":"grpc","protocol":"TCP"},{"containerPort":8089,"name":"websocket","protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","imagePullSecrets":[{"name":"regsecret"}],"restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}}},"status":{"observedGeneration":5,"replicas":0}}'
      health:
        status: Healthy
      liveState: '{"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{"deployment.kubernetes.io/revision":"2","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"omniscient-gateway\",\"applications.argoproj.io/app-name\":\"omniscient-gateway\",\"chart\":\"omniscient-gateway-0.0.1\",\"heritage\":\"Tiller\",\"release\":\"omniscient-gateway\"},\"name\":\"omniscient-gateway\",\"namespace\":\"kintohub\"},\"spec\":{\"replicas\":2,\"selector\":{\"matchLabels\":{\"app\":\"omniscient-gateway\",\"release\":\"omniscient-gateway\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"omniscient-gateway\",\"applications.argoproj.io/app-name\":\"omniscient-gateway\",\"release\":\"omniscient-gateway\"}},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":[{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"omniscient-gateway\"]}]},\"topologyKey\":\"kubernetes.io/hostname\"}]}},\"containers\":[{\"env\":[{\"name\":\"NAMESPACE_ENV\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"ENV\",\"value\":\"STAGING\"},{\"name\":\"GRAPHQL_HOST\",\"value\":\"prisma:4466\"},{\"name\":\"GRPC_PROXY\",\"value\":\"grpc-service\"},{\"name\":\"PROXY_GRPC_HOST_FORMAT\",\"value\":\"http://grpc-http-proxy:3000\"},{\"name\":\"RABBIT_HOST\",\"value\":\"rabbitmq-ha\"},{\"name\":\"PROXY_GRPC_X_ACCESS_TOKEN\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"grpc-http-proxy-token\",\"name\":\"omniscient-gateway\"}}}],\"image\":\"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"main\",\"ports\":[{\"containerPort\":80,\"name\":\"http\"},{\"containerPort\":8082,\"name\":\"public\"},{\"containerPort\":8090,\"name\":\"grpc\"},{\"containerPort\":8089,\"name\":\"websocket\"}],\"resources\":{\"requests\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"}}}],\"imagePullSecrets\":[{\"name\":\"regsecret\"}]}}}}\n"},"creationTimestamp":"2019-01-18T08:24:09Z","generation":10,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway","namespace":"kintohub","resourceVersion":"3947122","selfLink":"/apis/extensions/v1beta1/namespaces/kintohub/deployments/omniscient-gateway","uid":"6d6f6c6a-1afa-11e9-8fdb-42010a8c0082"},"spec":{"progressDeadlineSeconds":600,"replicas":2,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app":"omniscient-gateway","release":"omniscient-gateway"}},"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","release":"omniscient-gateway"}},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["omniscient-gateway"]}]},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"env":[{"name":"NAMESPACE_ENV","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ENV","value":"STAGING"},{"name":"GRAPHQL_HOST","value":"prisma:4466"},{"name":"GRPC_PROXY","value":"grpc-service"},{"name":"PROXY_GRPC_HOST_FORMAT","value":"http://grpc-http-proxy:3000"},{"name":"RABBIT_HOST","value":"rabbitmq-ha"},{"name":"PROXY_GRPC_X_ACCESS_TOKEN","valueFrom":{"secretKeyRef":{"key":"grpc-http-proxy-token","name":"omniscient-gateway"}}}],"image":"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4","imagePullPolicy":"IfNotPresent","name":"main","ports":[{"containerPort":80,"name":"http","protocol":"TCP"},{"containerPort":8082,"name":"public","protocol":"TCP"},{"containerPort":8090,"name":"grpc","protocol":"TCP"},{"containerPort":8089,"name":"websocket","protocol":"TCP"}],"resources":{"requests":{"cpu":"100m","memory":"256Mi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","imagePullSecrets":[{"name":"regsecret"}],"restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}}},"status":{"availableReplicas":2,"conditions":[{"lastTransitionTime":"2019-01-18T08:24:09Z","lastUpdateTime":"2019-02-01T15:14:43Z","message":"ReplicaSet
        \"omniscient-gateway-5b964659d7\" has successfully progressed.","reason":"NewReplicaSetAvailable","status":"True","type":"Progressing"},{"lastTransitionTime":"2019-02-02T06:16:18Z","lastUpdateTime":"2019-02-02T06:16:18Z","message":"Deployment
        has minimum availability.","reason":"MinimumReplicasAvailable","status":"True","type":"Available"}],"observedGeneration":10,"readyReplicas":2,"replicas":2,"updatedReplicas":2}}'
      status: Synced
      targetState: '{"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway"},"spec":{"replicas":2,"selector":{"matchLabels":{"app":"omniscient-gateway","release":"omniscient-gateway"}},"template":{"metadata":{"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","release":"omniscient-gateway"}},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["omniscient-gateway"]}]},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"env":[{"name":"NAMESPACE_ENV","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"POD_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}},{"name":"ENV","value":"STAGING"},{"name":"GRAPHQL_HOST","value":"prisma:4466"},{"name":"GRPC_PROXY","value":"grpc-service"},{"name":"PROXY_GRPC_HOST_FORMAT","value":"http://grpc-http-proxy:3000"},{"name":"RABBIT_HOST","value":"rabbitmq-ha"},{"name":"PROXY_GRPC_X_ACCESS_TOKEN","valueFrom":{"secretKeyRef":{"key":"grpc-http-proxy-token","name":"omniscient-gateway"}}}],"image":"asia.gcr.io/linear-bounty-174410/omniscient-gateway:dev-820dea4","imagePullPolicy":"IfNotPresent","name":"main","ports":[{"containerPort":80,"name":"http"},{"containerPort":8082,"name":"public"},{"containerPort":8090,"name":"grpc"},{"containerPort":8089,"name":"websocket"}],"resources":{"requests":{"cpu":"100m","memory":"256Mi"}}}],"imagePullSecrets":[{"name":"regsecret"}]}}}}'
    - health:
        status: Healthy
      liveState: '{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"kubernetes.io/ingress.class\":\"nginx\"},\"labels\":{\"app\":\"omniscient-gateway\",\"applications.argoproj.io/app-name\":\"omniscient-gateway\",\"chart\":\"omniscient-gateway-0.0.1\",\"heritage\":\"Tiller\",\"release\":\"omniscient-gateway\"},\"name\":\"omniscient-gateway\",\"namespace\":\"kintohub\"},\"spec\":{\"rules\":[{\"host\":\"api.staging.kintohub.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"omniscient-gateway\",\"servicePort\":80},\"path\":\"/\"}]}}],\"tls\":[{\"hosts\":[\"api.staging.kintohub.com\"],\"secretName\":\"kintossl\"}]}}\n","kubernetes.io/ingress.class":"nginx"},"creationTimestamp":"2019-01-18T08:24:09Z","generation":1,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway","namespace":"kintohub","resourceVersion":"1111676","selfLink":"/apis/extensions/v1beta1/namespaces/kintohub/ingresses/omniscient-gateway","uid":"6d986d4c-1afa-11e9-8fdb-42010a8c0082"},"spec":{"rules":[{"host":"api.staging.kintohub.com","http":{"paths":[{"backend":{"serviceName":"omniscient-gateway","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["api.staging.kintohub.com"],"secretName":"kintossl"}]},"status":{"loadBalancer":{"ingress":[{"ip":"34.80.249.61"},{"ip":"34.80.80.219"}]}}}'
      status: Synced
      targetState: '{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway"},"spec":{"rules":[{"host":"api.staging.kintohub.com","http":{"paths":[{"backend":{"serviceName":"omniscient-gateway","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["api.staging.kintohub.com"],"secretName":"kintossl"}]}}'
    - health:
        status: Healthy
      liveState: '{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"kubernetes.io/ingress.class\":\"nginx\"},\"labels\":{\"app\":\"omniscient-gateway\",\"applications.argoproj.io/app-name\":\"omniscient-gateway\",\"chart\":\"omniscient-gateway-0.0.1\",\"heritage\":\"Tiller\",\"release\":\"omniscient-gateway\"},\"name\":\"public-omniscient-gateway\",\"namespace\":\"kintohub\"},\"spec\":{\"rules\":[{\"host\":\"public.api.staging.kintohub.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"omniscient-gateway\",\"servicePort\":8082},\"path\":\"/\"}]}}],\"tls\":[{\"hosts\":[\"public.api.staging.kintohub.com\"],\"secretName\":\"kintossl\"}]}}\n","kubernetes.io/ingress.class":"nginx"},"creationTimestamp":"2019-01-18T08:24:09Z","generation":1,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"public-omniscient-gateway","namespace":"kintohub","resourceVersion":"1111674","selfLink":"/apis/extensions/v1beta1/namespaces/kintohub/ingresses/public-omniscient-gateway","uid":"6da3e68b-1afa-11e9-8fdb-42010a8c0082"},"spec":{"rules":[{"host":"public.api.staging.kintohub.com","http":{"paths":[{"backend":{"serviceName":"omniscient-gateway","servicePort":8082},"path":"/"}]}}],"tls":[{"hosts":["public.api.staging.kintohub.com"],"secretName":"kintossl"}]},"status":{"loadBalancer":{"ingress":[{"ip":"34.80.249.61"},{"ip":"34.80.80.219"}]}}}'
      status: Synced
      targetState: '{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"public-omniscient-gateway"},"spec":{"rules":[{"host":"public.api.staging.kintohub.com","http":{"paths":[{"backend":{"serviceName":"omniscient-gateway","servicePort":8082},"path":"/"}]}}],"tls":[{"hosts":["public.api.staging.kintohub.com"],"secretName":"kintossl"}]}}'
    - health:
        status: Healthy
      liveState: '{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"kubernetes.io/ingress.class\":\"nginx\"},\"labels\":{\"app\":\"omniscient-gateway\",\"applications.argoproj.io/app-name\":\"omniscient-gateway\",\"chart\":\"omniscient-gateway-0.0.1\",\"heritage\":\"Tiller\",\"release\":\"omniscient-gateway\"},\"name\":\"ws-omniscient-gateway\",\"namespace\":\"kintohub\"},\"spec\":{\"rules\":[{\"host\":\"ws.staging.kintohub.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"omniscient-gateway\",\"servicePort\":8089},\"path\":\"/\"}]}}],\"tls\":[{\"hosts\":[\"ws.staging.kintohub.com\"],\"secretName\":\"kintossl\"}]}}\n","kubernetes.io/ingress.class":"nginx"},"creationTimestamp":"2019-01-18T08:24:09Z","generation":1,"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"ws-omniscient-gateway","namespace":"kintohub","resourceVersion":"1111677","selfLink":"/apis/extensions/v1beta1/namespaces/kintohub/ingresses/ws-omniscient-gateway","uid":"6da65543-1afa-11e9-8fdb-42010a8c0082"},"spec":{"rules":[{"host":"ws.staging.kintohub.com","http":{"paths":[{"backend":{"serviceName":"omniscient-gateway","servicePort":8089},"path":"/"}]}}],"tls":[{"hosts":["ws.staging.kintohub.com"],"secretName":"kintossl"}]},"status":{"loadBalancer":{"ingress":[{"ip":"34.80.249.61"},{"ip":"34.80.80.219"}]}}}'
      status: Synced
      targetState: '{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"ws-omniscient-gateway"},"spec":{"rules":[{"host":"ws.staging.kintohub.com","http":{"paths":[{"backend":{"serviceName":"omniscient-gateway","servicePort":8089},"path":"/"}]}}],"tls":[{"hosts":["ws.staging.kintohub.com"],"secretName":"kintossl"}]}}'
    - health:
        status: Healthy
      liveState: '{"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"autoscaling/v2beta1\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"omniscient-gateway\",\"applications.argoproj.io/app-name\":\"omniscient-gateway\",\"chart\":\"omniscient-gateway-0.0.1\",\"heritage\":\"Tiller\",\"release\":\"omniscient-gateway\"},\"name\":\"omniscient-gateway\",\"namespace\":\"kintohub\"},\"spec\":{\"maxReplicas\":10,\"metrics\":[{\"resource\":{\"name\":\"cpu\",\"targetAverageUtilization\":70},\"type\":\"Resource\"},{\"resource\":{\"name\":\"memory\",\"targetAverageUtilization\":80},\"type\":\"Resource\"}],\"minReplicas\":2,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"omniscient-gateway\"}}}\n"},"creationTimestamp":"2019-02-01T15:55:41Z","labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway","namespace":"kintohub","resourceVersion":"4053061","selfLink":"/apis/autoscaling/v2beta1/namespaces/kintohub/horizontalpodautoscalers/omniscient-gateway","uid":"d377e8a8-2639-11e9-8d7c-42010a8c003d"},"spec":{"maxReplicas":10,"metrics":[{"resource":{"name":"memory","targetAverageUtilization":80},"type":"Resource"},{"resource":{"name":"cpu","targetAverageUtilization":70},"type":"Resource"}],"minReplicas":2,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"omniscient-gateway"}},"status":{"conditions":[{"lastTransitionTime":"2019-02-02T06:28:20Z","message":"the
        last scale time was sufficiently old as to warrant a new scale","reason":"ReadyForNewScale","status":"True","type":"AbleToScale"},{"lastTransitionTime":"2019-02-02T06:01:46Z","message":"the
        HPA was able to successfully calculate a replica count from memory resource
        utilization (percentage of request)","reason":"ValidMetricFound","status":"True","type":"ScalingActive"},{"lastTransitionTime":"2019-02-02T09:15:16Z","message":"the
        desired count is within the acceptable range","reason":"DesiredWithinRange","status":"False","type":"ScalingLimited"}],"currentMetrics":[{"resource":{"currentAverageUtilization":49,"currentAverageValue":"133076992","name":"memory"},"type":"Resource"},{"resource":{"currentAverageUtilization":0,"currentAverageValue":"0","name":"cpu"},"type":"Resource"}],"currentReplicas":2,"desiredReplicas":2,"lastScaleTime":"2019-02-02T06:22:17Z"}}'
      status: OutOfSync
      targetState: '{"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"labels":{"app":"omniscient-gateway","applications.argoproj.io/app-name":"omniscient-gateway","chart":"omniscient-gateway-0.0.1","heritage":"Tiller","release":"omniscient-gateway"},"name":"omniscient-gateway"},"spec":{"maxReplicas":10,"metrics":[{"resource":{"name":"cpu","targetAverageUtilization":70},"type":"Resource"},{"resource":{"name":"memory","targetAverageUtilization":80},"type":"Resource"}],"minReplicas":2,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"omniscient-gateway"}}}'
    revision: be40193e40b7cb7d1610b4c7f47eee20b5440883
    status: OutOfSync
  health:
    status: Healthy
  history:
  - deployedAt: 2019-02-01T15:27:02Z
    id: 3
    revision: 369c64ddbebd3afdff97759802c29d65d2a4d066
  - deployedAt: 2019-02-01T15:32:07Z
    id: 4
    revision: 369c64ddbebd3afdff97759802c29d65d2a4d066
  - deployedAt: 2019-02-01T15:50:28Z
    id: 5
    revision: be40193e40b7cb7d1610b4c7f47eee20b5440883
  - deployedAt: 2019-02-01T15:55:53Z
    id: 6
    revision: be40193e40b7cb7d1610b4c7f47eee20b5440883
  - deployedAt: 2019-02-01T16:00:25Z
    id: 7
    revision: be40193e40b7cb7d1610b4c7f47eee20b5440883
  operationState:
    finishedAt: 2019-02-01T16:00:25Z
    message: successfully synced
    operation:
      sync:
        parameterOverrides: []
        syncStrategy:
          hook: {}
    phase: Succeeded
    startedAt: 2019-02-01T15:59:57Z
    syncResult:
      resources:
      - kind: Service
        message: service/omniscient-gateway unchanged
        name: omniscient-gateway
        namespace: kintohub
        status: Synced
      - kind: Service
        message: service/api unchanged
        name: api
        namespace: kintohub
        status: Synced
      - kind: Deployment
        message: deployment.extensions/omniscient-gateway unchanged
        name: omniscient-gateway
        namespace: kintohub
        status: Synced
      - kind: Ingress
        message: ingress.extensions/public-omniscient-gateway unchanged
        name: public-omniscient-gateway
        namespace: kintohub
        status: Synced
      - kind: Ingress
        message: ingress.extensions/omniscient-gateway unchanged
        name: omniscient-gateway
        namespace: kintohub
        status: Synced
      - kind: Ingress
        message: ingress.extensions/ws-omniscient-gateway unchanged
        name: ws-omniscient-gateway
        namespace: kintohub
        status: Synced
      - kind: HorizontalPodAutoscaler
        message: horizontalpodautoscaler.autoscaling/omniscient-gateway configured
        name: omniscient-gateway
        namespace: kintohub
        status: Synced
      revision: be40193e40b7cb7d1610b4c7f47eee20b5440883
  parameters:
  - name: service.publicport.internalPort
    value: "8082"
  - name: service.wsport.name
    value: websocket
  - name: deployment.env.normal.PROXY_GRPC_HOST_FORMAT
    value: http://grpc-http-proxy:3000
  - name: scaling.targetAverageUtilization.cpu
    value: "70"
  - name: deployment.antiaffinity
    value: "true"
  - name: ingress.default.hosts
    value: '[api.staging.kintohub.com]'
  - name: scaling.enabled
    value: "true"
  - name: host
    value: staging.kintohub.com
  - name: image.secret
    value: regsecret
  - name: image.tag
    value: dev-820dea4
  - name: resources.requests.memory
    value: 256Mi
  - name: service.httpport.externalPort
    value: "80"
  - name: service.grpcport.internalPort
    value: "8090"
  - name: deployment.env.normal.RABBIT_HOST
    value: rabbitmq-ha
  - name: ingress.public.tls
    value: '[map[hosts:[public.api.staging.kintohub.com] secretName:kintossl]]'
  - name: scaling.minReplicas
    value: "2"
  - name: ingress.public.enabled
    value: "true"
  - name: ingress.ws.hosts
    value: '[ws.staging.kintohub.com]'
  - name: ingress.ws.tls
    value: '[map[hosts:[ws.staging.kintohub.com] secretName:kintossl]]'
  - name: deployment.env.normal.ENV
    value: STAGING
  - name: image.repository
    value: asia.gcr.io/linear-bounty-174410/omniscient-gateway
  - name: scaling.targetAverageUtilization.memory
    value: "80"
  - name: service.httpport.internalPort
    value: "80"
  - name: service.grpcport.externalPort
    value: "8090"
  - name: ingress.default.annotations.kubernetes.io/ingress.class
    value: nginx
  - name: ingress.ws.annotations.kubernetes.io/ingress.class
    value: nginx
  - name: sslsecret
    value: devkinto
  - name: deployment.enabled
    value: "true"
  - name: deployment.env.normal.GRPC_PROXY
    value: grpc-service
  - name: deployment.name
    value: main
  - name: ingress.ws.enabled
    value: "true"
  - name: ingress.enabled
    value: "false"
  - name: service.type
    value: ClusterIP
  - name: service.wsport.externalPort
    value: "8089"
  - name: ingress.public.annotations.kubernetes.io/ingress.class
    value: nginx
  - name: ingress.public.hosts
    value: '[public.api.staging.kintohub.com]'
  - name: name
    value: omniscient-gateway
  - name: secret.enabled
    value: "false"
  - name: deployment.env.downward.POD_IP
    value: status.podIP
  - name: service.httpport.name
    value: http
  - name: ingress.annotations
    value: <nil>
  - name: service.grpcport.name
    value: grpc
  - name: version
    value: 0.0.1
  - name: deployment.env.downward.NAMESPACE_ENV
    value: metadata.namespace
  - name: ingress.default.tls
    value: '[map[hosts:[api.staging.kintohub.com] secretName:kintossl]]'
  - name: scaling.maxReplicas
    value: "10"
  - name: service.publicport.name
    value: public
  - name: service.enabled
    value: "true"
  - name: tls
    value: <nil>
  - name: infrastructure.replicas
    value: "2"
  - name: config
    value: <nil>
  - name: deployment.env.normal.GRAPHQL_HOST
    value: prisma:4466
  - name: ingress.default.enabled
    value: "true"
  - name: resources.requests.cpu
    value: 100m
  - name: image.pullPolicy
    value: IfNotPresent
  - name: ingress.hosts
    value: '[chart-example.local]'
  - name: service.publicport.externalPort
    value: "8082"
  - name: service.wsport.internalPort
    value: "8089"
bappr picture bappr on 2 Feb 2019

@bappr this payload is from Argo CD v0.10. There were a bunch of improvements to the sync logic in v0.11 (among other things). I think we'll need to see if this is still happening in v0.11 to go further. Can you upgrade to v0.11.1 and report back? Please be aware of the breaking changes in v0.11 before you upgrade, most notably the change in resource names and the CLI incompatibility. Release notes are described here:

https://github.com/argoproj/argo-cd/releases/tag/v0.11.0

jessesuen picture jessesuen on 3 Feb 2019

It s planned for our next sprint.
I ll keep you updated.
Thanks ;)

bappr picture bappr on 3 Feb 2019

I get the same problem with v0.11.1. It seems related to having multiple type: Resource entries. Example, this HPA works fine:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa
  labels:
    role: hpa
  namespace: apz
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: rails
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 75

However this one always shows Out of Sync:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa
  labels:
    role: hpa
  namespace: apz
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: rails
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 75

And the Diff from the CLI:

===== autoscaling/HorizontalPodAutoscaler apz/hpa ======
23,26d22
<       name: cpu
<       targetAverageUtilization: 50
<     type: Resource
<   - resource:
28a25,28
>     type: Resource
>   - resource:
>       name: cpu
>       targetAverageUtilization: 50
fatalsaint picture fatalsaint on 12 Feb 2019

We understand the problem. It appears that the HPA controller can re-order the metrics list. Argo CD is correctly detecting this as a difference because everywhere else, an order of a list is significant. I think the solution may be one or more of:

  1. As a best practice, the user commits in git, the HPA metrics list in a way that the order will be the same as how the HPA controller is saving it. Document this in our FAQ.

  2. If order is not deterministic, the Argo CD diff library can fuzz the difference by understanding that order is not significant for HPA spec.metrics

  3. File an upstream bug, send a PR to kubernetes HPA controller so that the metrics list preserves ordering during reconciliation.

jessesuen picture jessesuen on 15 Feb 2019

IMO, the fact that kubernetes is re-ordering the metrics list is a kubernetes bug, and should be fixed upstream.

In the mean-time, option 1 appears to be working internally, without any Argo CD code changes.

jessesuen picture jessesuen on 15 Feb 2019

Thank you - can confirm that option 1, reordering the metrics (placing memory before cpu in this case), seems to resolve the issue I was seeing.

fatalsaint picture fatalsaint on 15 Feb 2019
👍2

Filed upstream issue https://github.com/kubernetes/kubernetes/issues/74099.

jessesuen picture jessesuen on 15 Feb 2019
❤2

Thanks guys, gonna do that as well!

bappr picture bappr on 15 Feb 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

rosscdh picture rosscdh  路  3Comments
estahn picture estahn  路  3Comments
duboisf picture duboisf  路  3Comments
clintberry picture clintberry  路  3Comments
eroji picture eroji  路  3Comments