prometheus-operator 🚀 - reduce alertmanager controller's aggressive logging

The Prometheus Operator uses the go-kit logger which supports log levels. Therefore you should be able to change the log level to less than info and this log line won't be printed anymore (including other info logs).

./operator -h returns

...
  -log-level string
        Log level to use. Possible values: all, debug, info, warn, error, none (default "info")
...

You're fine with -log-level=warn.

metalmatze on 2 Oct 2018

@metalmatze I want to have visibility over other info log lines, however this particular "sync alertmanager" line floods by logging 60+ times per second. Disabling all 'info' logging is an obvious workaround but not the solution to this problem.

I don't think this is intented functionality.

What is the reasoning behind this log line and its usability? Does it make sense to be printed so many times at default settings?

nikopen on 2 Oct 2018

I would suggest to at least move it back to 'debug' logging level. It's very verbose for the default settings

nikopen on 2 Oct 2018

👍1

Alright. I guess something is wrong here then and I misinterpreted the situation at first.

Sadly we not yet have metrics for the Alertmanager (we will fix this in the next days), sorry. If you want we can start debugging this together.

Things you can help us with are:

Check the events for your alertmanager statefulset: kubectl -n monitoring describe statefulset alertmanager-main and see if there are any errors.
Could you change the log level to debug with the flag I earlier mentioned? Once you have some log lines please paste them here, as there will be more output for us to debug.

Thank you.

metalmatze on 2 Oct 2018

@metalmatze please fully read the issue before closing any pull request.

First, can you confirm that on any other operator installation this log line prints less times, and how many times? Do you have several live examples you can see?

nikopen on 2 Oct 2018

@nikopen this is irregular and is not generally the case for our clusters and test clusters running the prometheus operator. as @metalmatze mentioned, there is likely something wrong with the statefulset configuration that is causing the operator to reconcile it so aggressively.

squat on 2 Oct 2018

@squat thanks for that.

nothing irregular in the statefulset and in the logs. debug logs just show these two lines and a sync of prometheusrules/servicemonitors every ~7 seconds or so.

level=debug ts=2018-10-02T14:54:42.0462804Z caller=operator.go:366 component=alertmanageroperator msg="update handler" old=124976698 cur=124976699
level=debug ts=2018-10-02T14:54:42.046627436Z caller=operator.go:880 component=prometheusoperator msg="update handler" old=124976698 cur=124976699

each line is printed around 30 times per second.

what can cause the constant reconciliation? can you point out the reconciliation rate on your clusters?

the setup is fairly standard with some extra configuration, having changed mainly irrelevant metadata (names, namespaces etc), but it's all working. just 1 alertmanager and 1 prometheus plugged, monitoring k8s state metrics/control plane. nothing too busy on the alerting side either.

nikopen on 2 Oct 2018

@nikopen let me find a rate for you. The prometheus operator will attempt to reconcile (run the sync routine) every time that an object relevant to the operator has updated or has been created/destroyed. This means every time a prometheus object, a service monitor, a rules object, or a statefulset corresponding to a prometheus is modified, the operator will re-sync. Additionally, every time a secret or configmap is modified, all promethei in that namespace (or promethei whose service monitor namespace selector selects that namespace) will also be updated.

Is something causing dramatic resource churn in a namespace that includes your prometheus objects? To test this, it might suffice to create a new namespace just for the prometheus objects, configure the operator to only watch that namespace, and then create a new prometheus object. If the operator doesn't sync as often then it is likely that something was causing churn in your original namespace.

In any case, the best way to debug would be to turn up the log verbosity to get a hint into which resource is causing the syncing and why.

squat on 2 Oct 2018

👍1

@squat two problems with SD:

creating an operator in a new namespace still has the same issue. it polls the alertmanager of a different namespace 50+ times per sec.

alertmanager fails to join 'cluster', 'gossip mesh', but works fine:

level=warn ts=2018-10-06T16:30:30.826454666Z caller=cluster.go:219 component=cluster msg="failed to join cluster" err="1 error occurred:\n\n* Failed to resolve alertmanager-system-0.alertmanager-operated.olympus.svc:6783: lookup alertmanager-system-0.alertmanager-operated.olympus.svc on 10.96.0.10:53: no such host"
level=info ts=2018-10-06T16:30:30.826535765Z caller=cluster.go:221 component=cluster msg="will retry joining cluster every 10s"
level=warn ts=2018-10-06T16:30:30.826555648Z caller=main.go:265 msg="unable to join gossip mesh" err="1 error occurred:\n\n* Failed to resolve alertmanager-system-0.alertmanager-operated.olympus.svc:6783: lookup alertmanager-system-0.alertmanager-operated.olympus.svc on 10.96.0.10:53: no such host"
level=info ts=2018-10-06T16:30:30.826646403Z caller=main.go:320 msg="Loading configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2018-10-06T16:30:30.826620036Z caller=cluster.go:570 component=cluster msg="Waiting for gossip to settle..." interval=10m0s
level=info ts=2018-10-06T16:30:30.829805841Z caller=main.go:396 msg=Listening address=:9093
level=info ts=2018-10-06T16:31:30.826763895Z caller=cluster.go:579 component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=1m0.000061138s

is this related to the aggressive logging?

second one, the prometheus responsible for monitoring control plane / kubelets etc has the following in the 'service discovery' tab:

 nsq/kube-apiserver/0 (3/3 active targets)
nsq/kube-controller-manager/0 (6/65 active targets)
nsq/kube-scheduler/0 (6/65 active targets)
nsq/kube-state-metrics/0 (1/65 active targets)
nsq/kubelet/0 (11/65 active targets)

which means constant reconciliation of seemingly 'new' endpoints, however they are the same. what might be changing is some annotation automatically given by an inhouse operator - does prometheus SD change the target each time a pod receives a new irrelevant annotation?

nikopen on 8 Oct 2018

as a general point - my experience of servicemonitors and endpoint discovery is that it's very inconsistent and flaky, it does things it's not supposed to do, which are fixed by redeploying a resource or refreshing something. is this a known issue?

nikopen on 8 Oct 2018

As a first start we are going to add more metrics with #1984.
Hopefully we should get better insights into what's going on with your Alertmanager.

metalmatze on 8 Oct 2018

@metalmatze that's nice but I don't think it's very relevant to this case and alertmanager either..

As I'm writing above, prometheus itself does a lot of re-discovery:

nsq/kube-controller-manager/0 (6/65 active targets)
nsq/kube-scheduler/0 (6/65 active targets)
nsq/kube-state-metrics/0 (1/65 active targets)
nsq/kubelet/0 (11/65 active targets)

neither the services nor the endpoints nor the underlying pods change or restart or anything like that - it's all stable k8s wise. why does prometheus change its "active targets"? can this be explained somehow?

Main problem with this is that there's an inhouse exporter for an inhouse application metric collector, and it faces the problem of prometheus constantly rediscovering its endpoints and not being able to scrape some for minutes, leaving gaps in metric collection.

This exporter has about 117 separate endpoints on one pod - it previously was a single endpoint but prometheus couldn't handle the amount of data from a single endpoint.

@squat do you know anything in this area?

nikopen on 10 Oct 2018

Could you share your Alertmanager object? I'd like to see if we can produce it with exactly your object.

brancz on 16 Oct 2018

hi @brancz ,

here you go:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"monitoring.coreos.com/v1","kind":"Alertmanager","metadata":{"annotations":{},"name":"system","namespace":"olympus"},"spec":{"replicas":1,"resources":{"requests":{"cpu":"100m","memory":"100m"}}}}
  clusterName: ""
  creationTimestamp: 2018-10-10T12:06:29Z
  name: system
  namespace: olympus
  resourceVersion: "149504907"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers/system
  uid: eb555a55-cc84-11e8-a45f-005056b71652
spec:
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 100m

manifest is pretty bare bones:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: system
spec:
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 100m

pod object:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 10.142.130.177/32
  creationTimestamp: 2018-10-10T12:45:21Z
  generateName: alertmanager-system-
  labels:
    alertmanager: system
    app: alertmanager
    controller-revision-hash: alertmanager-system-f6bbc754f
    statefulset.kubernetes.io/pod-name: alertmanager-system-0
  name: alertmanager-system-0
  namespace: olympus
  ownerReferences:
  - apiVersion: apps/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: alertmanager-system
    uid: eb5774cb-cc84-11e8-a45f-005056b71652
  resourceVersion: "149589357"
  selfLink: /api/v1/namespaces/olympus/pods/alertmanager-system-0
  uid: 59c20cec-cc8a-11e8-a5c1-005056b79850
spec:
  containers:
  - args:
    - --config.file=/etc/alertmanager/config/alertmanager.yaml
    - --cluster.listen-address=$(POD_IP):6783
    - --storage.path=/alertmanager
    - --data.retention=120h
    - --web.listen-address=:9093
    - --web.route-prefix=/
    - --cluster.peer=alertmanager-system-0.alertmanager-operated.olympus.svc:6783
    env:
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    image: quay.io/prometheus/alertmanager:v0.15.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 10
      httpGet:
        path: /api/v1/status
        port: web
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 3
    name: alertmanager
    ports:
    - containerPort: 9093
      name: web
      protocol: TCP
    - containerPort: 6783
      name: mesh
      protocol: TCP
    readinessProbe:
      failureThreshold: 10
      httpGet:
        path: /api/v1/status
        port: web
        scheme: HTTP
      initialDelaySeconds: 3
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 3
    resources:
      requests:
        cpu: 100m
        memory: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/alertmanager/config
      name: config-volume
    - mountPath: /alertmanager
      name: alertmanager-system-db
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-rghx6
      readOnly: true
  - args:
    - -webhook-url=http://localhost:9093/-/reload
    - -volume-dir=/etc/alertmanager/config
    image: quay.io/coreos/configmap-reload:v0.0.1
    imagePullPolicy: IfNotPresent
    name: config-reloader
    resources:
      limits:
        cpu: 5m
        memory: 10Mi
      requests:
        cpu: 5m
        memory: 10Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/alertmanager/config
      name: config-volume
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-rghx6
      readOnly: true
  dnsPolicy: ClusterFirst
  hostname: alertmanager-system-0
  nodeName: node-6.kubernetes-misc.preview.ams1.xing.com
  priority: 1000
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccount: default
  serviceAccountName: default
  subdomain: alertmanager-operated
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: config-volume
    secret:
      defaultMode: 420
      secretName: alertmanager-system
  - emptyDir: {}
    name: alertmanager-system-db
  - name: default-token-rghx6
    secret:
      defaultMode: 420
      secretName: default-token-rghx6
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2018-10-10T12:45:21Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2018-10-10T12:45:28Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2018-10-10T12:45:21Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://0c57972afe19a3e19cd4ad2726313ac8c5a58c432e2b0a3be7b1b9060786f0d1
    image: quay.io/prometheus/alertmanager:v0.15.1
    imageID: docker-pullable://prom/alertmanager@sha256:f7913f36ff34f8390cba90d1c2554601e26a1dad0c65db56eed90a3d33cf34a5
    lastState: {}
    name: alertmanager
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2018-10-10T12:45:23Z
  - containerID: docker://892f04aceab2da930733afdced92bdf6b5c4fe8cc7df5d1d8f68c9e501348249
    image: quay.io/coreos/configmap-reload:v0.0.1
    imageID: docker-pullable://quay.io/coreos/configmap-reload@sha256:e2fd60ff0ae4500a75b80ebaa30e0e7deba9ad107833e8ca53f0047c42c5a057
    lastState: {}
    name: config-reloader
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2018-10-10T12:45:24Z
  hostIP: 10.12.192.43
  phase: Running
  podIP: 10.142.130.177
  qosClass: Burstable
  startTime: 2018-10-10T12:45:21Z

one thing I'm noticing is the line
- --cluster.peer=alertmanager-system-0.alertmanager-operated.olympus.svc:6783

which comes from https://sourcegraph.com/github.com/coreos/prometheus-operator/-/blob/pkg/alertmanager/statefulset.go#L273:42 ,
is it needed at all on a single replica deployment?

nikopen on 16 Oct 2018

It discovers that this is the loopback device and doesn't actually do anything, so the last thing is fine. It's also unrelated to the object being reconciled. We had similar issues with the Prometheus object in https://github.com/coreos/prometheus-operator/pull/1785 and https://github.com/coreos/prometheus-operator/pull/1737. We're looking into whether the Alertmanager object might have the same problem. Thanks for reporting!

brancz on 16 Oct 2018

👍1

@nikopen after some digging, it looks like this is due to incorrect annotation handling specific to alertmanager objects. #1737 caught some of these issues but not all. Specifically, the alertmanager controller currently overwrites annotations on the alertmanager statefulset to always be the annotations from the previous iteration [0], which causes constant reconciliation since new annotations are overwritten and do not match. I am testing a patch right now.

[0] https://github.com/coreos/prometheus-operator/blob/master/pkg/alertmanager/statefulset.go#L134-L136

squat on 16 Oct 2018

👍1

@nikopen from my testing, the annotation issue I have found seems to be independent of yours. In fact, while the code is indeed incorrect, annotating the alertmanager does not cause the controller to reconcile constantly and I am not able to recreate the issue you described. Can you increase your log verbosity to debug and post the logs so we can see more closely what exactly is triggering the reconciliation?

squat on 16 Oct 2018

@squat https://github.com/coreos/prometheus-operator/issues/1966#issuecomment-426319352

nikopen on 16 Oct 2018

https://sourcegraph.com/github.com/coreos/prometheus-operator@master/-/blob/pkg/alertmanager/operator.go#L408:36

https://sourcegraph.com/github.com/coreos/prometheus-operator@master/-/blob/pkg/prometheus/operator.go#L924:36

nikopen on 16 Oct 2018

Thanks, I thought there may have been more log statements. In any case, both Prometheus and Alertmsnager statefulsets are aggressively being updated by some process and then being reconciled by the operator. The question is if the operator is fighting itself or some other process. Can you enable audit-logging in the API server and post the audit logs for the prometheus and alertmanager statefulsets? We need to see what is triggering updates and what fields are being changed.

squat on 17 Oct 2018

Nothing on audit logs either.

It's just the Jenkins serviceaccount logging info when it applies the prometheus deployment (triggered by me):

{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:31:26Z"},"level":"Metadata","timestamp":"2018-10-16T15:31:26Z","auditID":"09504040-2c72-410a-aa42-8f39257bf4d4","stage":"RequestReceived","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-16T15:31:26.878439Z","stageTimestamp":"2018-10-16T15:31:26.878439Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:31:26Z"},"level":"Metadata","timestamp":"2018-10-16T15:31:26Z","auditID":"09504040-2c72-410a-aa42-8f39257bf4d4","stage":"ResponseComplete","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-16T15:31:26.878439Z","stageTimestamp":"2018-10-16T15:31:26.891761Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:24Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:24Z","auditID":"2ea3a654-6d2d-4d97-93ca-3a71062c6ab8","stage":"RequestReceived","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-16T15:33:24.781222Z","stageTimestamp":"2018-10-16T15:33:24.781222Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:24Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:24Z","auditID":"2ea3a654-6d2d-4d97-93ca-3a71062c6ab8","stage":"ResponseComplete","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-16T15:33:24.781222Z","stageTimestamp":"2018-10-16T15:33:24.784403Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:26Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:26Z","auditID":"7af16ec6-7dd9-48bf-ae53-07605d46a9cd","stage":"RequestReceived","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers/system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","name":"system","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-16T15:33:26.946724Z","stageTimestamp":"2018-10-16T15:33:26.946724Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:26Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:26Z","auditID":"7af16ec6-7dd9-48bf-ae53-07605d46a9cd","stage":"ResponseComplete","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers/system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","name":"system","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-16T15:33:26.946724Z","stageTimestamp":"2018-10-16T15:33:26.952455Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:27Z"},"level":"Request","timestamp":"2018-10-16T15:33:27Z","auditID":"3b41b937-4eb2-4021-9c10-7574cc3d8590","stage":"RequestReceived","requestURI":"/apis/extensions/v1beta1/namespaces/olympus/ingresses/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"ingresses","namespace":"olympus","name":"alertmanager-system","apiGroup":"extensions","apiVersion":"v1beta1"},"requestReceivedTimestamp":"2018-10-16T15:33:27.014006Z","stageTimestamp":"2018-10-16T15:33:27.014006Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:27Z"},"level":"Request","timestamp":"2018-10-16T15:33:27Z","auditID":"3b41b937-4eb2-4021-9c10-7574cc3d8590","stage":"ResponseComplete","requestURI":"/apis/extensions/v1beta1/namespaces/olympus/ingresses/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"ingresses","namespace":"olympus","name":"alertmanager-system","apiGroup":"extensions","apiVersion":"v1beta1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-16T15:33:27.014006Z","stageTimestamp":"2018-10-16T15:33:27.018766Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:27Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:27Z","auditID":"1ea97949-6c39-4039-a62f-d2e883e10b4a","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/secrets/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"secrets","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-16T15:33:27.049074Z","stageTimestamp":"2018-10-16T15:33:27.049074Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:27Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:27Z","auditID":"1ea97949-6c39-4039-a62f-d2e883e10b4a","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/secrets/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"secrets","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-16T15:33:27.049074Z","stageTimestamp":"2018-10-16T15:33:27.065914Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:27Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:27Z","auditID":"7d70b5e2-7f9f-4d63-8878-24382eebb8c2","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/secrets/alertmanager-system?timeout=30s","verb":"patch","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"secrets","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-16T15:33:27.067874Z","stageTimestamp":"2018-10-16T15:33:27.067874Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:27Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:27Z","auditID":"7d70b5e2-7f9f-4d63-8878-24382eebb8c2","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/secrets/alertmanager-system?timeout=30s","verb":"patch","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"secrets","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-16T15:33:27.067874Z","stageTimestamp":"2018-10-16T15:33:27.082388Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:27Z"},"level":"Request","timestamp":"2018-10-16T15:33:27Z","auditID":"092f36c9-a05d-4458-8b50-722d99598571","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-16T15:33:27.084967Z","stageTimestamp":"2018-10-16T15:33:27.084967Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:27Z"},"level":"Request","timestamp":"2018-10-16T15:33:27Z","auditID":"092f36c9-a05d-4458-8b50-722d99598571","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-16T15:33:27.084967Z","stageTimestamp":"2018-10-16T15:33:27.087654Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:28Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:28Z","auditID":"c1a35176-fa8b-484d-a2dc-a8f19b48a0c8","stage":"RequestReceived","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-16T15:33:28.650495Z","stageTimestamp":"2018-10-16T15:33:28.650495Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-16T15:33:28Z"},"level":"Metadata","timestamp":"2018-10-16T15:33:28Z","auditID":"c1a35176-fa8b-484d-a2dc-a8f19b48a0c8","stage":"ResponseComplete","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"0b0d99ff-d0ce-11e8-a444-005056b7cc75","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-16T15:33:28.650495Z","stageTimestamp":"2018-10-16T15:33:28.652619Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:49:58Z"},"level":"Metadata","timestamp":"2018-10-17T08:49:58Z","auditID":"c16d1b5d-e56f-4344-ae6a-3db4d19373c5","stage":"RequestReceived","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T08:49:58.522978Z","stageTimestamp":"2018-10-17T08:49:58.522978Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:49:58Z"},"level":"Metadata","timestamp":"2018-10-17T08:49:58Z","auditID":"c16d1b5d-e56f-4344-ae6a-3db4d19373c5","stage":"ResponseComplete","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T08:49:58.522978Z","stageTimestamp":"2018-10-17T08:49:58.533648Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:00Z"},"level":"Metadata","timestamp":"2018-10-17T08:50:00Z","auditID":"4754dc96-aed7-4521-9193-8957a778d8c0","stage":"RequestReceived","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers/system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","name":"system","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T08:50:00.878533Z","stageTimestamp":"2018-10-17T08:50:00.878533Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:00Z"},"level":"Metadata","timestamp":"2018-10-17T08:50:00Z","auditID":"4754dc96-aed7-4521-9193-8957a778d8c0","stage":"ResponseComplete","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers/system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","name":"system","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T08:50:00.878533Z","stageTimestamp":"2018-10-17T08:50:00.885354Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:01Z"},"level":"Request","timestamp":"2018-10-17T08:50:01Z","auditID":"e9b99b2c-bf64-447a-b630-cebc6654d1f4","stage":"RequestReceived","requestURI":"/apis/extensions/v1beta1/namespaces/olympus/ingresses/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"ingresses","namespace":"olympus","name":"alertmanager-system","apiGroup":"extensions","apiVersion":"v1beta1"},"requestReceivedTimestamp":"2018-10-17T08:50:01.062385Z","stageTimestamp":"2018-10-17T08:50:01.062385Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:01Z"},"level":"Request","timestamp":"2018-10-17T08:50:01Z","auditID":"e9b99b2c-bf64-447a-b630-cebc6654d1f4","stage":"ResponseComplete","requestURI":"/apis/extensions/v1beta1/namespaces/olympus/ingresses/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"ingresses","namespace":"olympus","name":"alertmanager-system","apiGroup":"extensions","apiVersion":"v1beta1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T08:50:01.062385Z","stageTimestamp":"2018-10-17T08:50:01.064598Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:01Z"},"level":"Metadata","timestamp":"2018-10-17T08:50:01Z","auditID":"19f2ba84-806f-4803-8c2f-c74b05bd1106","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/secrets/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"secrets","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T08:50:01.161146Z","stageTimestamp":"2018-10-17T08:50:01.161146Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:01Z"},"level":"Metadata","timestamp":"2018-10-17T08:50:01Z","auditID":"19f2ba84-806f-4803-8c2f-c74b05bd1106","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/secrets/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"secrets","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T08:50:01.161146Z","stageTimestamp":"2018-10-17T08:50:01.163768Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:01Z"},"level":"Request","timestamp":"2018-10-17T08:50:01Z","auditID":"e5952916-cc8e-432f-8000-ec0548f02db2","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T08:50:01.167295Z","stageTimestamp":"2018-10-17T08:50:01.167295Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:01Z"},"level":"Request","timestamp":"2018-10-17T08:50:01Z","auditID":"e5952916-cc8e-432f-8000-ec0548f02db2","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-system?timeout=30s","verb":"get","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-system","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T08:50:01.167295Z","stageTimestamp":"2018-10-17T08:50:01.180420Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:02Z"},"level":"Metadata","timestamp":"2018-10-17T08:50:02Z","auditID":"793acea0-fdb5-49cb-9615-be4aa821115e","stage":"RequestReceived","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T08:50:02.892892Z","stageTimestamp":"2018-10-17T08:50:02.892892Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T08:50:03Z"},"level":"Metadata","timestamp":"2018-10-17T08:50:02Z","auditID":"793acea0-fdb5-49cb-9615-be4aa821115e","stage":"ResponseComplete","requestURI":"/apis/monitoring.coreos.com/v1/namespaces/olympus/alertmanagers?limit=500\u0026timeout=30s","verb":"list","user":{"username":"system:serviceaccount:olympus:jenkins-main-robot","uid":"72fe3ef4-d197-11e8-bbf0-005056b79feb","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.227.241"],"objectRef":{"resource":"alertmanagers","namespace":"olympus","apiGroup":"monitoring.coreos.com","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T08:50:02.892892Z","stageTimestamp":"2018-10-17T08:50:03.035809Z"}

nikopen on 17 Oct 2018

@nikopen hmm interesting. Are there no logs for any requests modifying the actual StatefulSets in question? I see patches being made to the alertmanager secret but nothing else of interest.

squat on 17 Oct 2018

There actually are, missed it before somehow.

operator 'updating' statefulset alertmanager:

{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:10Z","auditID":"a584ae91-e7e2-4300-8945-27e088cfa7cf","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:10.991827Z","stageTimestamp":"2018-10-17T13:33:11.001748Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"6ea9ca7c-ecb2-4bae-b98d-a5e254be18ff","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.022654Z","stageTimestamp":"2018-10-17T13:33:11.022654Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"6ea9ca7c-ecb2-4bae-b98d-a5e254be18ff","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.022654Z","stageTimestamp":"2018-10-17T13:33:11.029544Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"654688ca-088d-4d12-830a-8b8d66661ac9","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.051530Z","stageTimestamp":"2018-10-17T13:33:11.051530Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"654688ca-088d-4d12-830a-8b8d66661ac9","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.051530Z","stageTimestamp":"2018-10-17T13:33:11.061634Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"e36f00f9-aea4-4078-8375-e0f141e57eb5","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.081581Z","stageTimestamp":"2018-10-17T13:33:11.081581Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"e36f00f9-aea4-4078-8375-e0f141e57eb5","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.081581Z","stageTimestamp":"2018-10-17T13:33:11.086967Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"9a6b927b-3aee-4b8a-9d7d-a56e2129b224","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.111769Z","stageTimestamp":"2018-10-17T13:33:11.111769Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"9a6b927b-3aee-4b8a-9d7d-a56e2129b224","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.111769Z","stageTimestamp":"2018-10-17T13:33:11.120949Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"c7749717-e725-4f27-b967-2df931fec038","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.142293Z","stageTimestamp":"2018-10-17T13:33:11.142293Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"c7749717-e725-4f27-b967-2df931fec038","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.142293Z","stageTimestamp":"2018-10-17T13:33:11.150041Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"d3996b39-fed6-49b9-907c-a52bee2fdc52","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.188462Z","stageTimestamp":"2018-10-17T13:33:11.188462Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"d3996b39-fed6-49b9-907c-a52bee2fdc52","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.188462Z","stageTimestamp":"2018-10-17T13:33:11.194162Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"0854dfb1-7479-4291-b5f5-98cc1f449b18","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.206855Z","stageTimestamp":"2018-10-17T13:33:11.206855Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"0854dfb1-7479-4291-b5f5-98cc1f449b18","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.206855Z","stageTimestamp":"2018-10-17T13:33:11.214757Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"22767dac-c649-48e2-bdeb-bae996913bb3","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.257851Z","stageTimestamp":"2018-10-17T13:33:11.257851Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"22767dac-c649-48e2-bdeb-bae996913bb3","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.257851Z","stageTimestamp":"2018-10-17T13:33:11.263315Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"f5fd1e4d-c50d-493c-b1eb-7fc853197058","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.273601Z","stageTimestamp":"2018-10-17T13:33:11.273601Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"f5fd1e4d-c50d-493c-b1eb-7fc853197058","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.273601Z","stageTimestamp":"2018-10-17T13:33:11.283253Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"905405d0-9794-4df4-a3af-a62743d36046","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.294217Z","stageTimestamp":"2018-10-17T13:33:11.294217Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"905405d0-9794-4df4-a3af-a62743d36046","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.294217Z","stageTimestamp":"2018-10-17T13:33:11.299654Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"eb822877-def5-4857-aa7e-89c6b216fa5c","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.330909Z","stageTimestamp":"2018-10-17T13:33:11.330909Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"eb822877-def5-4857-aa7e-89c6b216fa5c","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.330909Z","stageTimestamp":"2018-10-17T13:33:11.339039Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"01c673a8-8bcd-42e8-85b8-dd39070e7f38","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.359589Z","stageTimestamp":"2018-10-17T13:33:11.359589Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"01c673a8-8bcd-42e8-85b8-dd39070e7f38","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.359589Z","stageTimestamp":"2018-10-17T13:33:11.369780Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"681f38bb-06fb-47e6-8ef3-f40e29c658ab","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.383242Z","stageTimestamp":"2018-10-17T13:33:11.383242Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"681f38bb-06fb-47e6-8ef3-f40e29c658ab","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.383242Z","stageTimestamp":"2018-10-17T13:33:11.399119Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"cd6842ee-ff5e-4076-86b2-f3dfa563ffe8","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.416403Z","stageTimestamp":"2018-10-17T13:33:11.416403Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"cd6842ee-ff5e-4076-86b2-f3dfa563ffe8","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.416403Z","stageTimestamp":"2018-10-17T13:33:11.423342Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"27fb3203-c3e8-43c5-8d5a-e3062242a8bd","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.441390Z","stageTimestamp":"2018-10-17T13:33:11.441390Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"27fb3203-c3e8-43c5-8d5a-e3062242a8bd","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.441390Z","stageTimestamp":"2018-10-17T13:33:11.450156Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"35d66671-f801-42d1-957b-05f239fc503e","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.471843Z","stageTimestamp":"2018-10-17T13:33:11.471843Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"35d66671-f801-42d1-957b-05f239fc503e","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.471843Z","stageTimestamp":"2018-10-17T13:33:11.477739Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"493c1465-4637-45a7-8c4f-d3701e8ee9c3","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.501455Z","stageTimestamp":"2018-10-17T13:33:11.501455Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"493c1465-4637-45a7-8c4f-d3701e8ee9c3","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.501455Z","stageTimestamp":"2018-10-17T13:33:11.509714Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"182a524f-97ab-4d04-a28c-0ec674ff20b9","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.531538Z","stageTimestamp":"2018-10-17T13:33:11.531538Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"182a524f-97ab-4d04-a28c-0ec674ff20b9","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.531538Z","stageTimestamp":"2018-10-17T13:33:11.538285Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"618e5d61-a7fc-43f3-9d72-8b3582ef8342","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.566815Z","stageTimestamp":"2018-10-17T13:33:11.566815Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"618e5d61-a7fc-43f3-9d72-8b3582ef8342","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.566815Z","stageTimestamp":"2018-10-17T13:33:11.573608Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"12aad1a6-c07b-4dd2-a7f2-1cef8f163d4d","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.596149Z","stageTimestamp":"2018-10-17T13:33:11.596149Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"12aad1a6-c07b-4dd2-a7f2-1cef8f163d4d","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.596149Z","stageTimestamp":"2018-10-17T13:33:11.604106Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"aebf986c-6a62-468b-86a6-15dddcb8b535","stage":"RequestReceived","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"requestReceivedTimestamp":"2018-10-17T13:33:11.621843Z","stageTimestamp":"2018-10-17T13:33:11.621843Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T13:33:11Z"},"level":"Metadata","timestamp":"2018-10-17T13:33:11Z","auditID":"aebf986c-6a62-468b-86a6-15dddcb8b535","stage":"ResponseComplete","requestURI":"/apis/apps/v1beta2/namespaces/olympus/statefulsets/alertmanager-system","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"statefulsets","namespace":"olympus","name":"alertmanager-system","apiGroup":"apps","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T13:33:11.621843Z","stageTimestamp":"2018-10-17T13:33:11.638400Z"}^C

operator 'getting' and 'updating' alertmanager-operated service aggressively:

{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"d5369e18-7558-445b-b199-d4b319f7a9fd","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.080172Z","stageTimestamp":"2018-10-17T14:39:43.082822Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"852ef9b1-2222-4ab6-9b8c-ec10b16377da","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.083957Z","stageTimestamp":"2018-10-17T14:39:43.083957Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"852ef9b1-2222-4ab6-9b8c-ec10b16377da","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.083957Z","stageTimestamp":"2018-10-17T14:39:43.088389Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"3954e7cb-9d57-45bf-8ff7-faaf53d35bc3","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.107020Z","stageTimestamp":"2018-10-17T14:39:43.107020Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"3954e7cb-9d57-45bf-8ff7-faaf53d35bc3","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.107020Z","stageTimestamp":"2018-10-17T14:39:43.109093Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"865bf2ab-3c32-42a6-9cb7-76bf600fc6a5","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.114044Z","stageTimestamp":"2018-10-17T14:39:43.114044Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"865bf2ab-3c32-42a6-9cb7-76bf600fc6a5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.114044Z","stageTimestamp":"2018-10-17T14:39:43.117820Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"8bff4178-8dcd-456d-af93-847933e190c5","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.141212Z","stageTimestamp":"2018-10-17T14:39:43.141212Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"8bff4178-8dcd-456d-af93-847933e190c5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.141212Z","stageTimestamp":"2018-10-17T14:39:43.143603Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"23a6bdb3-3b9f-4128-9921-55703e23b7c8","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.144549Z","stageTimestamp":"2018-10-17T14:39:43.144549Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"23a6bdb3-3b9f-4128-9921-55703e23b7c8","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.144549Z","stageTimestamp":"2018-10-17T14:39:43.149179Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"5b605653-b0c6-4324-aea6-a178f251fd66","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.164475Z","stageTimestamp":"2018-10-17T14:39:43.164475Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"5b605653-b0c6-4324-aea6-a178f251fd66","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.164475Z","stageTimestamp":"2018-10-17T14:39:43.166942Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"2825c21d-8abd-4fec-8395-c41eea8b3472","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.176506Z","stageTimestamp":"2018-10-17T14:39:43.176506Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"2825c21d-8abd-4fec-8395-c41eea8b3472","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.176506Z","stageTimestamp":"2018-10-17T14:39:43.181383Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"3fdd05c5-b65b-47f8-9f8f-84fe4e2a2737","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.197027Z","stageTimestamp":"2018-10-17T14:39:43.197027Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"3fdd05c5-b65b-47f8-9f8f-84fe4e2a2737","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.197027Z","stageTimestamp":"2018-10-17T14:39:43.199020Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"e68c657a-c4c7-49b7-bec4-cfa30ac7d14e","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.204135Z","stageTimestamp":"2018-10-17T14:39:43.204135Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"e68c657a-c4c7-49b7-bec4-cfa30ac7d14e","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.204135Z","stageTimestamp":"2018-10-17T14:39:43.209472Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"357af059-3e56-4335-8b78-8cf8933ce805","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.224063Z","stageTimestamp":"2018-10-17T14:39:43.224063Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"357af059-3e56-4335-8b78-8cf8933ce805","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.224063Z","stageTimestamp":"2018-10-17T14:39:43.226227Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"4f978965-7f76-4c9a-b0fa-e7a7cea0fc21","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.234095Z","stageTimestamp":"2018-10-17T14:39:43.234095Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"4f978965-7f76-4c9a-b0fa-e7a7cea0fc21","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.234095Z","stageTimestamp":"2018-10-17T14:39:43.238564Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"afa883c5-7fd6-4df5-ba55-7caccb4983b9","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.255719Z","stageTimestamp":"2018-10-17T14:39:43.255719Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"afa883c5-7fd6-4df5-ba55-7caccb4983b9","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.255719Z","stageTimestamp":"2018-10-17T14:39:43.257907Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"1b1bf61d-6df0-4edd-b68c-d0de157aefce","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.264188Z","stageTimestamp":"2018-10-17T14:39:43.264188Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"1b1bf61d-6df0-4edd-b68c-d0de157aefce","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.264188Z","stageTimestamp":"2018-10-17T14:39:43.269375Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"0f0f28b5-d78b-4719-8cd4-d34b0608308a","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.295403Z","stageTimestamp":"2018-10-17T14:39:43.295403Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"0f0f28b5-d78b-4719-8cd4-d34b0608308a","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.295403Z","stageTimestamp":"2018-10-17T14:39:43.297579Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"08bf9080-953e-493e-a8e3-ec465078a677","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.298789Z","stageTimestamp":"2018-10-17T14:39:43.298789Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"08bf9080-953e-493e-a8e3-ec465078a677","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.298789Z","stageTimestamp":"2018-10-17T14:39:43.303746Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"6d236b38-3f1e-4220-914a-27d9fd1a40a6","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.315451Z","stageTimestamp":"2018-10-17T14:39:43.315451Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"6d236b38-3f1e-4220-914a-27d9fd1a40a6","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.315451Z","stageTimestamp":"2018-10-17T14:39:43.319730Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"841d16bc-ddc6-48d1-8657-3dd4394ff171","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.324952Z","stageTimestamp":"2018-10-17T14:39:43.324952Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"841d16bc-ddc6-48d1-8657-3dd4394ff171","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.324952Z","stageTimestamp":"2018-10-17T14:39:43.330913Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"495764b3-98f7-4728-9d65-4bf1e85c5dc7","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.351535Z","stageTimestamp":"2018-10-17T14:39:43.351535Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"495764b3-98f7-4728-9d65-4bf1e85c5dc7","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.351535Z","stageTimestamp":"2018-10-17T14:39:43.354026Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"efa5a01f-9707-4c79-a167-5173ac12a3a8","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.355625Z","stageTimestamp":"2018-10-17T14:39:43.355625Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"efa5a01f-9707-4c79-a167-5173ac12a3a8","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.355625Z","stageTimestamp":"2018-10-17T14:39:43.360789Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"a7267a27-5622-44bb-a296-16e2a3727b20","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.375684Z","stageTimestamp":"2018-10-17T14:39:43.375684Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"a7267a27-5622-44bb-a296-16e2a3727b20","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.375684Z","stageTimestamp":"2018-10-17T14:39:43.379616Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"ba9c0476-5af6-4731-be9f-52f50860158b","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.384569Z","stageTimestamp":"2018-10-17T14:39:43.384569Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"ba9c0476-5af6-4731-be9f-52f50860158b","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.384569Z","stageTimestamp":"2018-10-17T14:39:43.389645Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"79e0fac1-7e54-47b9-a521-de02afc64e63","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.411318Z","stageTimestamp":"2018-10-17T14:39:43.411318Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"79e0fac1-7e54-47b9-a521-de02afc64e63","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"get","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2018-10-17T14:39:43.411318Z","stageTimestamp":"2018-10-17T14:39:43.413507Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"57d8b44c-9bc9-4466-aad3-97f05cd02050","stage":"RequestReceived","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1"},"requestReceivedTimestamp":"2018-10-17T14:39:43.414267Z","stageTimestamp":"2018-10-17T14:39:43.414267Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-17T14:39:43Z"},"level":"Request","timestamp":"2018-10-17T14:39:43Z","auditID":"57d8b44c-9bc9-4466-aad3-97f05cd02050","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/olympus/services/alertmanager-operated","verb":"update","user":{"username":"system:serviceaccount:olympus:prometheus-operator","uid":"576b1af3-c7d7-11e8-90d2-005056b76baf","groups":["system:serviceaccounts","system:serviceaccounts:olympus","system:authenticated"]},"sourceIPs":["10.12.35.236"],"objectRef":{"resource":"services","namespace":"olympus","name":"alertmanager-operated","apiVersion":"v1","resourceVersion":"351100722"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"Service","apiVersion":"v1","metadata":{"name":"alertmanager-operated","resourceVersion":"351100722","creationTimestamp":null,"labels":{"operated-alertmanager":"true"}},"spec":{"ports":[{"name":"web","protocol":"TCP","port":9093,"targetPort":9093},{"name":"mesh","protocol":"TCP","port":6783,"targetPort":6783}],"selector":{"app":"alertmanager"},"clusterIP":"None","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"requestReceivedTimestamp":"2018-10-17T14:39:43.414267Z","stageTimestamp":"2018-10-17T14:39:43.422028Z"}

nikopen on 17 Oct 2018

@nikopen thanks, this is very helpful information. The controller does not watch Services for changes, but the Service is updated because the StatefulSet triggered a change. So we need to determine what is changing on the StatefulSet and why.

Could you set the audit level to Request and share the request bodies of the StatefulSet update requests from the operator (redacted as necessary) to understand what fields are being changed?

squat on 17 Oct 2018

the audit logs are already in Request level, if you see above. I'm not sure what you mean with request bodies, can't find something relevant.

There are the following aggressive entries in the audit logs:

prom-operator 'get' alertmanager-operated svc
prom-operator 'update' alertmanager-operated svc
prom-operator 'update' alertmanager sts

I think you also missed these log lines I posted 16 days ago:

level=debug ts=2018-10-02T14:54:42.0462804Z caller=operator.go:366 component=alertmanageroperator msg="update handler" old=124976698 cur=124976699
level=debug ts=2018-10-02T14:54:42.046627436Z caller=operator.go:880 component=prometheusoperator msg="update handler" old=124976698 cur=124976699

this is debug level logs of the prometheus-operator, it seems to be self inflicted. I cannot see any external entity triggering this.

nikopen on 18 Oct 2018

also tried RequestResponse audit level but don't see any related entries.

Are you not able to reproduce any of the above?
Do you have any theory that is specific to the k8s clusters I'm working with?

nikopen on 18 Oct 2018

@nikopen the following changes:

prom-operator 'get' alertmanager-operated svc
prom-operator 'update' alertmanager-operated svc
prom-operator 'update' alertmanager sts

are updated only in response to a change that is noticed in the StatefulSet. The real culprit here is the StatefulSet, which is changing and causing the operator to reconcile it and update related resources, like the Service. The Alertmanager operator only watches Alertmanager objects and StatefulSets, so a changing Service is not the cause of the issue. In fact, the log lines you posted clearly indicate that there is a change in the StatefulSet (not the Alertmanager object) that is triggering everything.

Regarding the the audit level, it looked to me like the audit log for the StatefulSet data is only writing the request metadata but not the request body itself. I see you have the request bodies for the logs related to the Services, could you post the same but for the StatefulSets? This is really the most valuable information.

squat on 18 Oct 2018

in the statefulset object, resourceVersion: "174624276" stays the same, and this can be observed via the audit logs too.

wouldn't that change if the sts was indeed altered?

another thing I tested:

https://github.com/coreos/prometheus-operator/blob/release-0.23/pkg/alertmanager/operator.go#L398

putting

spec:
  paused: true

in the alertmanager CRD, the buggy reconciliation doesn't happen anymore. If I remove it, operator starts spewing the buggy logs quite immediately.

somehow, it seems that there are always items to dequeue, else there would be a 'sync failed' message

enqueue() is called in multiple places. it will be interesting to put log lines near them to see what's happening.

re audit request body, I can't seem to get any request body, with a 'RequestResponse' level for everything even. Can you pass an audit config that works for you so I can apply it maybe?

nikopen on 18 Oct 2018

Hmm I actually don’t see a resource version in the audit logs for the StatefulSets at all. It is true that if the sts was being updated that the resource version would change, but I don’t see any logs that indicate the resource version is staying the same.

There is no sync failed message because the sync func is not returning errors, it is happily reconciling over and over again, and then returning early [0].

It is very odd that we can’t find the full audit log for the updates. If you don’t mind, I’ll build a debug Operator container for you that logs the StatefulSet objects so that we can compare what is being updated to what is in the informer cache. Is that alright with you?

[0] https://github.com/coreos/prometheus-operator/blob/ffae4dfc1abf804b36ebfc5fb70585296651b14d/pkg/alertmanager/operator.go#L311-L313

squat on 19 Oct 2018

A debug operator would be great, thanks for that.

nikopen on 22 Oct 2018

@squat is the test image easy to make?

nikopen on 19 Nov 2018

@nikopen sorry for the delay, this fell off of my radar.

I created a new docker image for you based on v0.23.2 that should log a full diff of the changes. It's tagged as: squat/prometheus-operator:nikopen. You can also inspect the changes or build it from source yourself [0].

Please give it a shot and post the logs once you have them. It should be quite verbose, but will help us see what is changing that is causing the reconciliation. Hopefully by understanding what is changing we will be able to determine why.

[0] https://github.com/coreos/prometheus-operator/compare/v0.23.2...squat:nikotest?expand=1

squat on 19 Nov 2018

👍1

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

stale[bot] on 14 Aug 2019

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

stale[bot] on 20 Oct 2019

Closing due to inactivity.

brancz on 21 Oct 2019

Prometheus-operator: reduce alertmanager controller's aggressive logging

All 35 comments

Related issues