Vector: New `timescale` / `postgres` sink

Created on 26 Sep 2019  路  4Comments  路  Source: timberio/vector

I'd like to support Timescale as a sink, and after talking to them it appears we just need to support Postgres since Timescale is built on Postgres. I'm hesitant to call it a Postgres sink given that we'll probably want to make certain assumptions about how we're writing data. Before we begin work we should determine if this is the appropriate course of action.

Ref https://github.com/timberio/vector/issues/935

metrics sinks approval requirements feature

Most helpful comment

A general unified SQL sink might be challenging given the language variants between the different databases.

But @jamessewell is correct, there should be no difference in writing data between Timescale and Postgres (but there may be other reasons for creating a Timescale specific sink).

All 4 comments

I鈥檓 not familiar with Timescale, but wonder would it make sense support unified SQL sinks, which would include Postgres, ClickHouse, and at least for the sake of easy tests, SQLite. The idea is to have same configuration structure for them and, ideally, support some kind of automatic table creation/migrations.

As far as writing data there is no difference between Timescale and Postgres.

It would be up to the user to configure the destination tables (and hypertables in the Timscaledb case) before the sink was used.

A general unified SQL sink might be challenging given the language variants between the different databases.

But @jamessewell is correct, there should be no difference in writing data between Timescale and Postgres (but there may be other reasons for creating a Timescale specific sink).

I forgot to note our progress here: this can be achieved using PostgREST (a REST gateway for PostgreSQL) . Performance is very good.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

binarylogic picture binarylogic  路  3Comments

leebenson picture leebenson  路  3Comments

a-rodin picture a-rodin  路  3Comments

lewisthompson picture lewisthompson  路  3Comments

LucioFranco picture LucioFranco  路  3Comments