Postgres-operator: podAnnotations section seems not to work

[thepotatocannon]

I tried to add custom annotations to postgres pods using _podAnnotations_ option in the cluster manifest:

  podAnnotations:
    prometheus.io/scrape: "true"

But it seems that this option does nothing, here are annotations from pods created after deployment:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    status: '{"conn_url":"postgres://10.244.2.178:5432/postgres","api_url":"http://10.244.2.178:8008/patroni","state":"running","role":"replica","version":"1.6.0","xlog_location":1845494080,"timeline":1}'

[FxKu]

are you using the latest version of the operator or the 1.2.0 release image?

[thepotatocannon]

It's 1.2.0, do I have to upgrade it?

[FxKu]

yes, this feature got just introduced a few days ago :smiley:

[thepotatocannon]

Quick update: I cloned the repo with newest changes, but it still does nothing. This is my configuration for now:
operator values:

 # annotations attached to each database pod
  # custom_pod_annotations: prometheus.io/scrape:"true"
  custom_pod_annotations: 
    keya:test

cluster manifest:

  podAnnotations:
    prometheus.io/scrape: "true"
    annotation.key: test

Am I doing something wrong?

[FxKu]

Have you updated the deployment yaml file to use the latest operator image? Use latest instead of v1.2.0

[thepotatocannon]

That's what I was missing, thanks a lot for the help. :)

[thepotatocannon]

I'm sorry, but I have to reopen this issue. For my understanding, everything should work fine, this is my configuration:
values.yaml:

  custom_pod_annotations: 
    prometheus.io/scrape: true

postgres manifest:

  podAnnotations:
    prometheus.io/scrape: true

However pods don't get that annotation:

kamil@kamil:~$ kubectl get pod test-postgres-0 -o json
{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": {
        "annotations": {
            "status": "{\"conn_url\":\"postgres://10.244.1.127:5432/postgres\",\"api_url\":\"http://10.244.1.127:8008/patroni\",\"state\":\"running\",\"role\":\"master\",\"version\":\"1.6.3\",\"xlog_location\":50356680,\"timeline\":1}"
        },
        "creationTimestamp": "2019-12-19T08:36:46Z",

edit:
Forgot to mention, I already use operator in 1.3.0 version.

[FxKu]

In the values.yml you have to use this format:

  custom_pod_annotations: "prometheus.io/scrape:true"

as it's used by the ConfigMap. No nesting allowed. Wonder why the CRD validation doesn't catch it - ah yes because its for ConfigMap and not CRD confg. But it should work with podAnnotations nevertheless. Have you started from scratch and deleted all operator related resources before? What's your K8s env?

[thepotatocannon]

custom_pod_annotations: "prometheus.io/scrape:true"

this format resulted in error while installing operator via helm:

validation failure list:
configuration.kubernetes.custom_pod_annotations in body must be of type object: "string"

I was trying to make it work by using helm upgrade, because while deleting helm release I cannot install it again until I delete crd for operator and postgres, which causes new operator not to see running clusters. Am I doing something wrong generally, that this happens?

[FxKu]

the configuration property only exists in the CRD-based configuration. So you are using the values-crd.yaml file when installing the operator. If the operator is already running and you want to upgrade, simply edit the deployment and change the docker image. This will create a new operator pod which should deal with custom pod annotations for newly created clusters.

[thepotatocannon]

Yes, I am using crd one, so how should this look like?

  custom_pod_annotations: 
    prometheus.io/scrape: "true"

like this or without quotes?

[thepotatocannon]

simply edit the deployment and change the docker image. This will create a new operator pod which should deal with custom pod annotations for newly created clusters.

I don't really understand how is this supposed to work?

[FxKu]

Quotes are not necessary I think, but don't hurt either for string.
When you deploy the operator there should be a deployment, replicaset and pod. Simply patch the image field in the deployment resource (kubectl apply/patch/edit etc.). This will create a new replicaset + pod.

[thepotatocannon]

Okay, what I can see at the moment is:

1. This is my current operatorconfiguration:

$ kubectl get operatorconfigurations.acid.zalan.do czk-postgres-operator -o yaml
apiVersion: acid.zalan.do/v1
configuration:
  aws_or_gcp:
    aws_region: eu-central-1
  debug:
    debug_logging: true
    enable_database_access: true
  docker_image: registry.opensource.zalan.do/acid/spilo-cdp-12:1.6-p16
  enable_crd_validation: true
  enable_shm_volume: true
  etcd_host: ""
  kubernetes:
    cluster_domain: cluster.local
    cluster_labels:
      application: spilo
    cluster_name_label: cluster-name
    custom_pod_annotations:
      prometheus.io/scrape: "true"
    enable_init_containers: true
    enable_pod_antiaffinity: false
    enable_pod_disruption_budget: true
    enable_sidecars: true
    oauth_token_secret_name: czk-postgres-operator
    pdb_name_format: postgres-pdb
    pod_antiaffinity_topology_key: kubernetes.io/hostname
    pod_management_policy: ordered_ready
    pod_role_label: spilo-role
    pod_service_account_name: czk-postgres-operator
    pod_terminate_grace_period: 5m
    secret_name_template: '{username}.{cluster}.credentials'
    spilo_privileged: false
    watched_namespace: '*'
  load_balancer:
    db_hosted_zone: db.example.com
    enable_master_load_balancer: false
    enable_replica_load_balancer: false
    master_dns_name_format: '{cluster}.{team}.{hostedzone}'
    replica_dns_name_format: '{cluster}-repl.{team}.{hostedzone}'
  logging_rest_api:
    api_port: 8080
    cluster_history_entries: 1000
    ring_log_lines: 100
  logical_backup:
    logical_backup_docker_image: registry.opensource.zalan.do/acid/logical-backup
    logical_backup_s3_access_key_id: ""
    logical_backup_s3_bucket: my-bucket-url
    logical_backup_s3_endpoint: ""
    logical_backup_s3_secret_access_key: ""
    logical_backup_s3_sse: AES256
    logical_backup_schedule: 30 00 * * *
  max_instances: -1
  min_instances: -1
  postgres_pod_resources:
    default_cpu_limit: "3"
    default_cpu_request: 100m
    default_memory_limit: 1Gi
    default_memory_request: 100Mi
  repair_period: 5m
  resync_period: 10m
  scalyr:
    scalyr_cpu_limit: "1"
    scalyr_cpu_request: 100m
    scalyr_memory_limit: 1Gi
    scalyr_memory_request: 50Mi
  teams_api:
    enable_team_superuser: false
    enable_teams_api: false
    pam_role_name: zalandos
    protected_role_names:
    - admin
    team_admin_role: admin
    team_api_role_configuration:
      log_statement: all
  timeouts:
    pod_deletion_wait_timeout: 10m
    pod_label_wait_timeout: 10m
    ready_wait_interval: 3s
    ready_wait_timeout: 30s
    resource_check_interval: 3s
    resource_check_timeout: 10m
  users:
    replication_username: standby
    super_username: postgres
  workers: 4
kind: OperatorConfiguration
metadata:
  creationTimestamp: "2019-12-19T15:25:00Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: czk
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: postgres-operator
    helm.sh/chart: postgres-operator-1.3.0
  name: czk-postgres-operator
  namespace: postgres
  resourceVersion: "5546940"
  selfLink: /apis/acid.zalan.do/v1/namespaces/postgres/operatorconfigurations/czk-postgres-operator
  uid: b8ff11c5-2273-11ea-8814-fa163e76e357

and these are annotations on newly created postgres pods:

$ kubectl get pod test-postgres-0 -o 'jsonpath={.metadata.annotations}'
map[status:{"conn_url":"postgres://10.244.1.144:5432/postgres","api_url":"http://10.244.1.144:8008/patroni","state":"running","role":"master","version":"1.6.3","xlog_location":52505576,"timeline":3}]

Why are there no custom annotations?

[FxKu]

Check the postgres-operator deployment. What's the version of the image field. Is it really running on v1.3.0. If you have used the latest helm chart from the repo it should have installed the latest version. I can even see it in the annotation of your configuration.

You can also delete the postgres-operator deployment (not the CRDs!) and postgres clusters will keep running. When you recreate the operator it will discover running clusters. As a last resort you could also check the logs from the operator logs to spot some errors.

[thepotatocannon]

Yes, it is version 1.3.0, operator has no errors. Does that mean this feature doesn't work properly then?

[ekeih]

I think I can confirm that this does not work. So far I tried it with global settings and with cluster specific settings without any success. I will try to investigate this further after the holidays. (version 1.3.0 installed via the current helm chart)

[FxKu]

I think, I found the problem. For some reason the v1.3.0 image I pushed did not reflect the latest state. I've created a separate dirty image with our internal build pipeline and now the annotations work with a fresh setup. Before, it seems I've only tested with a cached v1.3.0 image. Therefore, I did not spot the problems you were facing.

[ekeih]

@FxKu Thanks for finding this! Looking forward for the next release with the correct image :)

[FxKu]

It's already out. Simply use tag v1.3.0-dirty.

[ekeih]

@FxKu Ah nice, thanks! :)
(I still think a normal image release would be a good idea, so other people get the correct software release when they install the operator.)

[FxKu]

Done. I created another minor release v1.3.1. Will close this issue now.

[JSchlarb]

Hey @FxKu, thanks a lot for the fix!

the build for your latest commit ( 077f9af4e399d7d50177c0c5074672911d88fc1c / #780) failed unfortunately for. Can you trigger it again?