Timescaledb: Kubernetes helm chart?

[k1ng440]

Hello,
please create kubernetes helm chart that supports cluster and automatic

see
https://github.com/CrunchyData/postgres-operator
https://github.com/zalando-incubator/postgres-operator

[pyros2097]

Since timescaledb is just a module for postgres you can use the default postgres helm chart.
You just need to specify timescaledb as the image and specify the version in the imageTag parameters for the chart.
https://github.com/kubernetes/charts/tree/master/stable/postgresql

[k1ng440]

this helm chart to works perfectly fine.
https://github.com/kubernetes/charts/tree/master/stable/postgresql

helm install --name "timescaledb" -f timescaledb/values.yaml stable/postgresql

` image: timescale/timescaledb imageTag: latest-pg10 postgresPassword: xxxxxx metrics.enabled: true

[samkrew]

this helm chart to works perfectly fine.
https://github.com/kubernetes/charts/tree/master/stable/postgresql

helm install --name "timescaledb" -f timescaledb/values.yaml stable/postgresql

image: timescale/timescaledb
imageTag: latest-pg10
postgresPassword: xxxxxx
metrics.enabled: true

Chart stable/postgresql uses bitnami docker image which handles all staff like replication.
timescale/timescaledb will not work same as bitnami/postgresql.

[husseinm]

@samkrew Yes, you are right that the timescaledb container is not the bitnami container. That's exactly what was proposed above your post. He suggested you override it with timescaledb's image.

Personally, when I deployed to helm, I had an issue with the securityContext. Specifically, I kept getting this error preventing the pod from fully starting:

initdb: could not look up effective user ID 1001: user does not exist

What worked was to disable the securityContext in the values.yml file above, however, I believe this is just a hacky workaround. Does anyone have a suggestion for how to properly correct the issue?

Context: Am using GKE with Kubernetes 1.11.1 & Helm 2.11.0

[sebastianvoss]

I'm running into the same issue as @husseinm: initdb: could not look up effective user ID 1001: user does not exist. If someone has an advice (other as switching off securityContext) it would be appreciated.

[husseinm]

@sebastianvoss I currently am focused on my project (which will take a couple of months) and when I am finished if this is still not fixed, I might swing back and try to resolve this. However, if you would like to take a swing at it, I have a feeling it most likely has to do with the user ID given in the values.yml, AFAIK that's supposed to map to a user on the Linux system. The solution is probably to change the ID in the .yml to whatever the platform you are running K8/Helm on supports/gives permission for. Another solution could be giving permissions to UID 1001 to enable the DB to initialize and start.

As a starting point - are you on GKE? Try putting your platform, K8 version & Helm version as I did in my Context note.

[sebastianvoss]

@husseinm I'm on EKS (k8s 1.10.3) and Helm v2.11.0.

I think I solved the issue. Timescale is using the Alpine Postgres image as base. Alpine contains a postgres user out of the box but with different UID/GID.

When the correct IDs are configured in values.yaml everything works fine:

securityContext:
  enabled: true
  fsGroup: 70
  runAsUser: 70

[husseinm]

@sebastianvoss I recreated my DB and it worked, Alpine does have a different User System/defaults then Ubuntu (which is the default OS the original helm chart we are basing this off of uses).

Regardless, I reproduced the fix and it worked. +1

[joaocc]

Has anyone checked (or knows) of the settings/vars expected by bitnami/postgresql image are compatible with the ones from _/postgres?
In https://github.com/helm/charts/commit/7f1e47c5a8b10f24547aa385d782294622f8f439 the base image was changed from postgres:9.6.2 to bitnami/postgresql:10.5.0.
A quick look at the charts and images points to many changes, which may cause some issues.

[husseinm]

@joaocc Yes, I tested it and it worked. That commit you pointed to is 21 days old, I tested with the latest Helm chart 6 days ago. Let me clarify a few things for you and anyone else that sees this.

Sorry for the wall of text, but I guess it had to be done:

Explanation for people to be aware

Helm has many stages/components used to install the DB. However, at its core, it uses Docker Containers. That container has the OS and the environment to execute in a self-contained file.

The Helm Chart in question uses Bitnami's Docker image. Their Docker image appears to use Ubuntu. (not 100% sure, am assuming, but if I had to bet on it that's probably what it would be knowing Bitnami)

The fix that has been proposed in this ticket is to replace the container image from the Bitnami one to the TimescaleDB one available on the public Docker Registry. The issue with that is the Registry container uses a different OS, Alpine. See, when PG starts up it runs in a user context, this makes sure PG doesn't do anything "evil" (the rationale behind this is the same one why we don't run every CLI command with sudo it's the concept of giving programs the least access necessary).

My initial hack was to simply disable the running of PG in a special Postgres user environment and instead of running it in the default user context, giving it a different set of powers/privileges.

Then Sebastian came and found the proper User for Alpine, noting that Alpine already had a Postgres user made for us by default, we simply had to update the values.yaml to point to it before deploying our Helm charts.

Mini-Rant

Now, on to the feature differences between Timescale and the normal Helm Chart. Again, remember how I said they are both the same Postgres, just running on different operating systems and this one has the TimescaleDB extension downloaded and ready to be initialized (again you still need to initialize Timescale like in the getting started guide). All standard PG features that the helm chart will work, the following I have tested to work:

• Creating a DB on chart creation (without PSQL)
• Overriding default Username & Password on chart creation (without PSQL)
• Overriding the Postgres Config on chart creation (without PSQL)
• The Prometheus Metics Exporter

The features I am concerned about and do not think/know will work from the Chart:

• Kubeapps
• Custom Config Scripts (Literally added as a feature yesterday)
• Production Horizontal Slave Scaling
• Persistence (The chart does successfully create a Stateful Set, but the Chart README says it stores all PG Data/Configs in /bitnami/postgresql - I am not sure if that directory gets installed on the Alpine Install)

The reason I am not sure about the above is they are either non-standard (Kubeapps), Dependent on the image configuration (Custom Config Scripts & Persistence) or difficult/complex and easy to screw up (Production Horizontal Slave Scaling). It would be fantastic if someone could test these.

Regardless, the important thing to note from that is that we should have a more standard way of doing this besides editing the current image, ideally, someone from TimescaleDB (or from the community) would do the following:

• Monitor the Helm chart and give instructions to edit the Chart for Postgres.
• Document what works/doesn't works and keep that documentation up to date.

This I feel is a fairly common demand now that K8 has a more mature Stateful Set implementation and TimescaleDB should really support it since deployment is what is hampering many from using Timescale (can't use RDS/Cloud SQL, must deploy to EC2 and monitor/manage it, hard to scale instances, must implement own backup system with Wal-e, etc.) - K8/Helm Charts are probably the sanest apporach atm which is unfortunate but we must make do.

Regardless, for the newcomers who have made it past my wall of text, here are the instructions I tested to work with the lastest Helm chart on 11/08/18:

TL;DR - How do I fix it!

1. Download the latest values-production.yaml, you can also use the non-production values.
2. Edit the file to use the TimescaleDB image, specifically edit the following:

image:
  registry: docker.io
  repository: bitnami/postgresql
  tag: 10.6.0

to instead be:

image:
  registry: docker.io
  repository: timescale/timescaledb
  tag: latest-pg10

NOTE: Ideally you change that tag to a specific PG version, especially if you plan to do Horizontal Scaling

1. Edit the file to use the correct Alpine security context, specifically edit the following:

securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001

to

securityContext:
  enabled: true
  fsGroup: 70
  runAsUser: 70

1. Review and edit the values.yaml file accordingly to your desired spec/deployment.
2. Deploy the helm chart with helm install -f ./values-production.yaml stable/postgres

[joaocc]

@husseinm Thanks for the clarification.
I am in the process of testing TimescaleDB and was caught by the transition of the helm chart format. While the docker image seems to be compatible, chart values settings for the library-based series are not compatible with the ones in bitnami-based images, neither are postgres-related environment variables nor the folders where data and configs are stored.

• https://github.com/docker-library/postgres/tree/master/10
• https://github.com/bitnami/bitnami-docker-postgresql/blob/10.6.0-debian-9-r2/10/debian-9

For instance, TimescaleDB expects POSTGRES_PASSWORD but the bitnami-based chart is setting POSTGRESQL_PASSWORD.
As a user-to-be, I would be happy with any helm chart for TimescaleDB that is maintained or recommended by the community.

[husseinm]

At a quick glance the first issue I can see is that the Bitnami image is Debian based and not Ubuntu, that alone will cause a couple of issues with directories and such, Alpine is closer to Ubuntu than Debian (obviously it is not 100% compatible with either)

You are right and that's what we have been collecting in this issue, what values/chart settings need to be changed, what works and what doesn't.

The persistence issue I mentioned sound like what you are having issues with and that simply boils down to the directory used by the chart, I think it might work I just haven't had a moment to test it (I would assume it should 'just work' since it really shouldn't matter if a linux directory has the name 'bitnami' in it, it's still a linux directory that files can be written to)

---

Quite frankly, I am extremely hesitant to recommend this method at the moment since all the benefits of Helm don't really apply with this method, we are probably better off just deploying the OG Docker image to an EC2 or follow the Timescale guides on installation for now until someone picks up the slack. Let's also not forget that K8 has not always been the best with Stateful Sets/DBs.

Also, kinda unrelated but, TimescaleDB can be slower on many things until you hit that ~100 million row mark simply because of the hypertable breaking things down to too small pieces than Postgres is used to. If that comes as news to you, I would read some of their benchmarking a little closer on their blog. So if you are testing TimescaleDB for a brand new project, it might not make sense to start with it off the bat, you might be better off waiting until that magic mark and then migrating over (That's the beauty of the extension method, it isn't hard to switch)

But yeah, I agree the Helm chart isn't something I plan to deploy to production in it's current state.

[joaocc]

@husseinm Thanks for the advice.
In the meantime, I created a docker image for TimescaleDB based on bitnami (https://github.com/timescale/timescaledb-docker/pull/38) so i can use the helm chart.
It is my understanding that the charts based on bitnami will be better supports (since the official repo gone in that direction).
For convenience I have created a repo with both chart versions here (https://github.com/hsz-devops/timescaledb--timescaledb-docker).

[joaocc]

In case it is of interest, I have started (but not finished) work of converting the current chart to a statefull set (2.x) version, but still not finished (https://github.com/hsz-devops/helm--timescaledb). Prob wont continue the work since I already have a way to use the 2.x bitnami chart with the bitnami-based image for timescaledb.

[LeeHampton]

Thanks for doing this @joaocc ! This should be getting merged shortly. It adds a very useful option for deploying TimescaleDB on Kubernetes.

Just to recap the state of things, with @joaocc 's PR and @husseinm 's comments, we currently seem to have 4 different approaches for deploying TimescaleDB on Kubernetes:

(1) Use the official helm chart with the official timescaledb image using the workaround outlined in @husseinm 's comment: https://github.com/timescale/timescaledb/issues/522#issuecomment-438716313 …. This gets your foot in the door but as @husseinm outlines in his post you may run into other snags depending on your needs

(2) Use the new bitnami-compatible image @joaocc has been kind enough to open in a PR against our official docker image. This addresses the main concerns around the workaround in option (1).

(3) Use the patroni helm chart for a more fully featured HA Postgres/Timescale deployment. Patroni seems to be pretty faithfully tracking TimescaleDB releases (they’re already tracking 1.0.1 which we released last week). However, if you don’t need automatic failover, this might be overkill. It introduces dependencies on etcd and patroni daemons running in each postgres pod.

(4) If you have a decent amount of Kubernetes knowledge, it may actually be better to just build your own stateful set based off of the TimescaleDB image. PostgreSQL has enough extensions and different architectural approaches that helm charts can be a bit constraining. Helm charts are great for getting bootstrapped quickly, but especially in the world of postgres, its extension ecosystem, and all of the different possible architectures for HA, one size doesn’t necessarily fit all.

If anyone else wants to weigh in on the pros and cons of these options or has other options to add to the list, this would be a good place to start that discussion.

[husseinm]

Thanks, I think the option left to us by @joaocc is definitely better than the workaround I left, again thank you for that!

4 is definitely an option - but again not exactly conducive for a great dev experience.

So my vote is between option 2 & 3.

My only concern with the bitnami route is backing up the WALs or at the least the pgdumps. It doesn't seem that there is an automated solution to provide backups with @joaocc's PR & the default postgres helm chart (1000% not his fault, rather the fault of that helm chart that is currently in stable). If we could integrate the TimescaleDB WAL-E image into the same helm chart that would be perfect and mitigate that large concern (because obviously you really shouldn't have a production DB without backups 😅)

Another option is to simply say someone has to use the patroni chart in production because let's be honest if you have 100 million rows+ where TimescaleDB is actually going to be of a large benefit, then the HA is way more important than the increase of resources.

So this is definitely a question of practicality, development experience, production resource overhead and what direction the TimescaleDB creators want to steer us in. My thought process is:

1. Should we recommend @joaocc's PR for development purposes only?
   
   
   
   • If not, then we should try and add backups so that it can be easily used with confidence in production, IMO.
   
   
   • If yes, we could just say no and steer production towards patroni.
   
   
2. How early should one start using Timescale?
   
   
   
   • Day One? Then we need @joaocc cheap easy button solution.
   
   
   • Only when it will start to improve performance (high row count)? Then patroni makes more sense.
   
   

One last note: With patroni, the documentation on deploying via helm & TimescaleDB is rather limited and if that is the recommended route there should be more docs on that. I see you provided an analysis of patroni v. stalone on the Timescale blog, but no deployment info/instructions/how to integrate Timescale with it. Also, is the recommendation to use the Helm chart (still in incubator status) or to use it directly?

That's how I feel, what's the recommendation? @LeeHampton

[jwmullally]

For anyone looking for an example of installing Timescale with CrunchyData postgres-operator, see this: https://github.com/CrunchyData/crunchy-containers/issues/747#issuecomment-459546319
YMMV.

[seguidor777]

This can be useful too
helm install --name timescaledb --set image.repository=timescale/timescaledb,image.tag=latest-pg11,postgresqlPassword=xxxxxxxx,postgresqlDatabase=postgres,securityContext.fsGroup=70,securityContext.runAsUser=70,metrics.enabled=true stable/postgresql

[mfreed]

Hi @k1ng440 @seguidor777 @jwmullally @husseinm @joaocc @pyros2097 thanks for this discussion and requests.

You can now found native Kubernetes helm charts for TimescaleDB at the following repo. It includes deployments with single-node in a replicated environment (with automated HA and failover via Patroni) and incremental backup (via pgBackrest). It also includes charts for multi-node scale-out TimescaleDB.

https://github.com/timescale/timescaledb-kubernetes

You can find more information in our recent blog post:

https://blog.timescale.com/blog/new-helm-charts-for-deploying-timescaledb-on-kubernetes/