Loki: Using Cassandra to store both indexes and chunks

[Konpon96]

I'm trying to configure Loki to use Apache Cassandra both for index and chunk storage. By following the example from the documentation and tweaking it slightly (newer schema version, different names, dropping fields with default values) I've succeeded to do the former - Loki creates keyspace and the table for the Loki indexes. Here is screenshot from DataGrip:

Due to fact the usage of Cassandra for storing chunks doesn't seem to be documented anywhere, I've tried to configure it analogically. I expected Loki to create necessary tables automatically, just like with the index table. However, when I launched Promtail and started providing Loki with data, I keep getting the following errors in the Loki logs:

level=error ts=2020-01-12T00:04:42.532601078Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:04:42.533099558Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:05:12.529906601Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:05:12.530103736Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:05:12.530123701Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:05:12.530141404Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:05:12.530180373Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:05:12.530190617Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:05:12.530208735Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"
level=error ts=2020-01-12T00:05:12.531913961Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="unconfigured table chunk"

As it can be seen on the screenshot above, there is no "chunk" table indeed. However, there is also no mention how the table schema should look like if I was to create it manually - which I doubt to be a right choice.

Here is excerpt from my Loki configuration:

schema_config:
  configs:
  - from: 2020-01-01
    store: cassandra
    object_store: cassandra
    schema: v10
    index:
      prefix: loki_index
    chunks:
      prefix: chunk
storage_config:
  cassandra:
    username: cassandra
    password: password
    addresses: cassandra.logging.svc.cluster.local
    auth: true
    keyspace: loki

What am I missing? My versions are as follows:

• Loki: 1.2.0 (official Helm Chart, version 0.22.0)
• Cassandra: 3.11.5 (Bitnami Helm Chart, version 4.1.13)

[slim-bean]

I don't know if Loki is able to use cassandra as a chunk store, but given that most of this code is borrowed from cortex I would think this should work. @gouthamve do you have any insights here?

[Konpon96]

I don't know if Loki is able to use cassandra as a chunk store, but given that most of this code is borrowed from cortex I would think this should work. @gouthamve do you have any insights here?

According to the documentation (https://github.com/grafana/loki/blob/master/docs/operations/storage/README.md#supported-stores) Cassandra can be used both as a chunk and index store. What's more, the configuration schema implies that it is possible to use one store for both - if the object_store field in the schema_config is skipped, Loki uses the same store for chunk storage as it uses for the index, specified by the store field (again, as stated in the documentation files: https://github.com/grafana/loki/blob/master/docs/configuration/README.md#schema_config).

[shokada]

Probably, "chunk" table will not be created if you do not use the index table period.
https://github.com/cortexproject/cortex/blob/master/pkg/chunk/table_manager.go#L246-L284

So, If you set up like the following, "chunk" table should be created.

index:
  prefix: loki_index
  period: 360h
chunks:
  prefix: chunk
  period: 360h

[Konpon96]

Probably, "chunk" table will not be created if you do not use the index table period.
https://github.com/cortexproject/cortex/blob/master/pkg/chunk/table_manager.go#L246-L284

So, If you set up like the following, "chunk" table should be created.

index:
  prefix: loki_index
  period: 360h
chunks:
  prefix: chunk
  period: 360h

I've analyzed the same code part too, adding the period option for both index and chunks made it work!

The documentation doesn't make this behavior clear though:

index:
  # Table prefix for all period tables.
  prefix: <string>
  # Table period.
  [period: <duration> | default = 168h]
  # A map to be added to all managed tables.
  tags:
    [<string>: <string> ...]

I'm not sure but from what I remember from analyzing the Cortex's source code, the Period field is an integer, which in Go is 0 by default. In such case the default = 168h part from the docs disagrees with the config.IndexTables.Period == 0 in the Cortex's Go code.

As much as I'm happy that we finally got Loki working fully on-premises, diving into the dependency's source code to figure out the behavior is a bit counter-productive, to say the least. The field clearly isn't required and there is some specific behavior if it's not provided, but it definitely isn't equal to setting it to 168h. However, just saying that the field is required to force people into providing a value is not a solution, cause the field is not required and it's presence is not validated during Loki startup.

Any idea how could I rewrite that docs part in my PR to Loki's repo so that other people won't stumble into the same time-consuming analysis?

[cyriltovena]

I would definitively add a paragraph there about this configuration.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.