Postgres-operator: Service creation fails at first sync after enabling/disabling Load Balancer type
I have a bare-metal k8s cluster, which doesn't have a LoadBalancer. I only use postgresql for cluster database connection so load balancer is not needed.
postgres-operator tries to deploy a load balancer and it stuck in "Load balancer is being provisioned". So far there's no problem but it makes me worry something might happen. I tried to change the service to ClusterIP but the operator changes it back after a while.
ntcong
All 18 comments
The operator does not create load balancer, maybe the service is wrongfully set to provision a load balancer. Please check that in your config for the operator and in the postgresql manifest both load balancer options are disabled.
https://github.com/zalando/postgres-operator/blob/master/manifests/configmap.yaml#L25
https://github.com/zalando/postgres-operator/blob/master/manifests/complete-postgres-manifest.yaml#L22
Jan-M
on 8 Nov 2019
but if I don't set enable_master_load_balancer, there's no k8s service created. Do we need a service to connect to the master pod
ntcong
on 8 Nov 2019
The service for master access and replica access is always created. You must not access pods directly.
Jan-M
on 8 Nov 2019
Here is kubectl get svc output when enableMasterLoadBalancer is set to true
kubectl -n postgresql get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
acid-jandb LoadBalancer 10.43.188.196 <pending> 5432:32482/TCP 116s
acid-jandb-config ClusterIP None <none> <none> 2d22h
acid-jandb-repl ClusterIP 10.43.27.0 <none> 5432/TCP 22s
postgres-operator ClusterIP None <none> 8080/TCP 2d22h
postgres-operator-ui ClusterIP 10.43.174.21 <none> 80/TCP 2d23h
I set enableMasterLoadBalancer to false and svc is still the same
kubectl -n postgresql get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
acid-jandb LoadBalancer 10.43.188.196 <pending> 5432:32482/TCP 3m44s
acid-jandb-config ClusterIP None <none> <none> 2d22h
acid-jandb-repl ClusterIP 10.43.27.0 <none> 5432/TCP 2m10s
postgres-operator ClusterIP None <none> 8080/TCP 2d22h
postgres-operator-ui ClusterIP 10.43.174.21 <none> 80/TCP 2d23h
Setting enableReplicaLoadBalancer from true to false correctly create a ClusterIP service
kubectl -n postgresql get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
acid-jandb LoadBalancer 10.43.188.196 <pending> 5432:32482/TCP 5m7s
acid-jandb-config ClusterIP None <none> <none> 2d22h
acid-jandb-repl ClusterIP 10.43.140.121 <none> 5432/TCP 13s
postgres-operator ClusterIP None <none> 8080/TCP 2d22h
postgres-operator-ui ClusterIP 10.43.174.21 <none> 80/TCP 2d23h
Here is operator log when I set enableMasterLoadBalancer from true to false
time="2019-11-08T16:04:00Z" level=info msg="\"UPDATE\" event has been queued" cluster-name=postgresql/acid-jandb pkg=controller worker=0
time="2019-11-08T16:04:00Z" level=info msg="update of the cluster started" cluster-name=postgresql/acid-jandb pkg=controller worker=0
time="2019-11-08T16:04:00Z" level=debug msg="final load balancer source ranges as seen in a service spec (not necessarily applied): [\"127.0.0.1/32\"]" cluster-name=postgresql/acid-ja
ndb pkg=cluster
time="2019-11-08T16:04:00Z" level=debug msg="final load balancer source ranges as seen in a service spec (not necessarily applied): [\"127.0.0.1/32\"]" cluster-name=postgresql/acid-jandb pkg=cluster time="2019-11-08T16:04:00Z" level=debug msg="No load balancer created for the replica service" cluster-name=postgresql/acid-jandb pkg=cluster
time="2019-11-08T16:04:00Z" level=debug msg="No load balancer created for the replica service" cluster-name=postgresql/acid-jandb pkg=cluster
time="2019-11-08T16:04:00Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE acid-jandb nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-jandb.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PGUSER_STANDBY sta
ndby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-jandb.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PAM_OAUTH2 nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/11/bin\"},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=postgresql/acid-jandb pkg=cluster
time="2019-11-08T16:04:00Z" level=debug msg="Generating Spilo container, environment variables: [{SCOPE acid-jandb nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {KUBERNETES_LABELS application=spilo nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.acid-jandb.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.acid-jandb.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PAM_OAUTH2 nil} {HUMAN_ROLE zalandos nil} {SPILO_CONFIGURATION {\"postgresql\":{\"bin_dir\":\"/usr/lib/postgresql/11/bin\"},\"bootstrap\":{\"initdb\":[{\"auth-host\":\"md5\"},{\"auth-local\":\"trust\"}],\"users\":{\"zalandos\":{\"password\":\"\",\"options\":[\"CREATEDB\",\"NOLOGIN\"]}},\"dcs\":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=postgresql/acid-jandb pkg=cluster
time="2019-11-08T16:04:00Z" level=debug msg="syncing roles" cluster-name=postgresql/acid-jandb pkg=cluster
time="2019-11-08T16:04:00Z" level=debug msg="closing database connection" cluster-name=postgresql/acid-jandb pkg=cluster
time="2019-11-08T16:04:00Z" level=info msg="cluster has been updated" cluster-name=postgresql/acid-jandb pkg=controller worker=0
I found a relevant log
time="2019-11-08T16:16:16Z" level=info msg="reason: new service's type \"ClusterIP\" doesn't match the current one \"LoadBalancer\"" cluster-name=postgresql/acid-jandb pkg=cluster time="2019-11-08T16:16:16Z" level=error msg="could not sync services: could not sync master service: could not update master service to match desired state: could not create service \"postgresql/acid-jandb\": object is being deleted: services \"acid-jandb\" already exists" cluster-name=postgresql/acid-jandb pkg=cluster
So maybe when I chaged the cluster loadbalancer settings, the operator tries to remove the old service and create a new one, but fail because the deletion process take a few seconds.
So I think we should have some kind of sleep timer here: https://github.com/zalando/postgres-operator/blob/6e682fd6b5707efa90e8c40684914436e457de78/pkg/cluster/resources.go#L378
ntcong
on 8 Nov 2019
I can independently verify that this operator indeed creates a LB.
grafana-pg LoadBalancer 10.97.126.188 localhost 5432:31029/TCP 5d20h
apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
name: grafana-pg
spec:
teamId: grafana
volume:
size: 1Gi
numberOfInstances: 2
users:
grafana: # database owner
- superuser
- createdb
foo_user: [] # role for application foo
databases:
grafana: grafana # dbname: owner
postgresql:
version: "11"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-operator
data:
# additional_secret_mount: "some-secret-name"
# additional_secret_mount_path: "/some/dir"
api_port: "8080"
aws_region: europe-north1
cluster_domain: cluster.local
cluster_history_entries: "1000"
cluster_labels: application:spilo
cluster_name_label: version
# custom_service_annotations:
# "keyx:valuez,keya:valuea"
db_hosted_zone: db.example.com
debug_logging: "true"
# default_cpu_limit: "3"
# default_cpu_request: 100m
# default_memory_limit: 1Gi
# default_memory_request: 100Mi
docker_image: registry.opensource.zalan.do/acid/spilo-11:1.6-p1
# enable_admin_role_for_users: "true"
# enable_database_access: "true"
enable_master_load_balancer: "true"
# enable_pod_antiaffinity: "false"
# enable_pod_disruption_budget: "true"
enable_replica_load_balancer: "false"
# enable_shm_volume: "true"
# enable_team_superuser: "false"
enable_teams_api: "false"
# etcd_host: ""
# infrastructure_roles_secret_name: postgresql-infrastructure-roles
# inherited_labels: application,environment
# kube_iam_role: ""
# log_s3_bucket: ""
# logical_backup_docker_image: "registry.opensource.zalan.do/acid/logical-backup"
# logical_backup_s3_bucket: "my-bucket-url"
# logical_backup_schedule: "30 00 * * *"
master_dns_name_format: '{cluster}.{team}.staging.{hostedzone}'
# master_pod_move_timeout: 10m
# max_instances: "-1"
# min_instances: "-1"
# node_readiness_label: ""
# oauth_token_secret_name: postgresql-operator
# pam_configuration: |
# https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees
# pam_role_name: zalandos
pdb_name_format: "postgres-{cluster}-pdb"
# pod_antiaffinity_topology_key: "kubernetes.io/hostname"
pod_deletion_wait_timeout: 1m
# pod_environment_configmap: ""
pod_label_wait_timeout: 1m
pod_management_policy: "ordered_ready"
pod_role_label: spilo-role
pod_service_account_name: "zalando-postgres-operator"
pod_terminate_grace_period: 20s
# postgres_superuser_teams: "postgres_superusers"
# protected_role_names: "admin"
ready_wait_interval: 3s
ready_wait_timeout: 30s
repair_period: 1m
replica_dns_name_format: '{cluster}-repl.{team}.staging.{hostedzone}'
replication_username: standby
resource_check_interval: 3s
resource_check_timeout: 1m
resync_period: 1m
ring_log_lines: "100"
secret_name_template: '{username}.{cluster}.credentials'
# sidecar_docker_images: ""
# set_memory_request_to_limit: "false"
spilo_privileged: "false"
super_username: postgres
# team_admin_role: "admin"
# team_api_role_configuration: "log_statement:all"
# teams_api_url: http://fake-teams-api.default.svc.cluster.local
# toleration: ""
# wal_s3_bucket: ""
watched_namespace: "*" # listen to all namespaces
workers: "4"
which is the quickstart config.
haf-afa
on 11 Nov 2019
Yes, enable_master_load_balancer: "true".
Jan-M
on 11 Nov 2019
This should be false if you do not want a load balancer to be created. The K8s service is always created for you to connect to.
Jan-M
on 11 Nov 2019
To me that's unclear, because a ClusterIP is an internal L4 load balancer; specifically one that transparently fails over to the replica if need-be. Also the above config is from the quickstart, so perhaps the quickstart should be updated not to expose the service? (alike: kubectl expose ...)
haf-afa
on 12 Nov 2019
We are updating the config map in the example/default.
Yeah, i guess this is where habbit or current environment comes into play. For me LoadBalancer means service of type load balancer, and actually some other component acting as LoadBalancer (ELB/NLB/F5). The service, especially in the Postgres case with just one endpoint for the master was just not a load balancer for me.
Jan-M
on 12 Nov 2019
@FxKu
Jan-M
on 12 Nov 2019
Yes, the word is a bit over loaded... What tripped me up was the "enable_master..." and "enable_replica_..." which I took to mean the ability to address the master via a service (which I want) and the ability to address the replica via a service (e.g. for read-only use-cases, data-analytics and such).
One way that the Jaeger operator didn't trip me up, was that they reused much of the CRD definitions from K8S core, allowing me to specify explicitly Service.spec.type and then merged that into the created Service CRD instance; (IRL it allowed me to specify volume claim templates, but you get the idea); that way there's no confusion about where the property lands in the end. This pattern is also used with Helm to a lesser effect.
haf-afa
on 12 Nov 2019
Both services are always created as K8s services don't cost money. So you can have it all false and use the master service for writing and the replica service for offloading reads.
Jan-M
on 12 Nov 2019
Makes sense!
haf-afa
on 12 Nov 2019
Hi @Jan-M . Did you take a look at my earlier comment about deleting/recreating the service?
https://github.com/zalando/postgres-operator/issues/713#issuecomment-551895559
ntcong
on 12 Nov 2019
@ntcong will check it, but last time I enabled/disabled load balancer after provisioning the cluster the services we're synced correctly - although it took a while. I did not check the logs though.
FxKu
on 12 Nov 2019
Every time I change enableMasterLoadBalancer from true to false, postgres-operator tries to delete the old service and failed to create a new one.
I think that's really bad because if it fails it leaves the cluster in an unusable state
ntcong
on 12 Nov 2019
I could reproduce this. Deploying a ClusterIP, setting enablingMasterLoadBalance = true, getting:
level=error msg="could not sync services: could not sync master service: could not update master service to match desired state: could not create service \"default/acid-minimal-cluster\": object is being deleted: services \"acid-minimal-cluster\" already exists"
...
level=error msg="could not connect to PostgreSQL database: dial tcp ... i/o timeout"
...
level=error msg="could not connect to PostgreSQL database: dial tcp: lookup acid-minimal-cluster.default.svc.cluster.local ...: no such host"
But then at the next sync it should create the correct service:
level=info msg="created missing master service \"default/acid-minimal-cluster\""
But, you're right, this shouldn't happen. I will add the same wait logic we use for StatefulSets.
FxKu
on 12 Nov 2019
Related issues
k0da
路
4Comments
hjacobs
路
5Comments
aixeshunter
路
6Comments
RichieSams
路
6Comments
GeertJohan
路
7Comments