Loki: Unrecognized storage client after schema upgrade

[SockenSalat]

I'm runing loki in a kubernetes v1.19.2 kluster. Loki has been installed manually - without helm or tanka. After following the upgrade instructions from the v2.0.0 release I tried to update the schema.

When I add the follwoing section to my schema config:

      - from: "2020-10-28"
        index:
          period: 24h
          prefix: index_
        object_store: filesystem
        schema: v11
        store: boltdb-shipper

Loki reports Unrecognized storage client , choose one of: aws, s3, gcs, azure, filesystem.

When I remove this section loki comes up and works as expected.

my complete config (with the section causing the annoyance) looks like this;

    auth_enabled: false
    chunk_store_config:
      max_look_back_period: 0s
    ingester:
      chunk_block_size: 262144
      chunk_idle_period: 3m
      chunk_retain_period: 1m
      lifecycler:
        ring:
          kvstore:
            store: inmemory
          replication_factor: 1
      max_transfer_retries: 0
    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
    schema_config:
      configs:
      - from: "2018-04-15"
        index:
          period: 168h
          prefix: index_
        object_store: filesystem
        schema: v9
        store: boltdb
      - from: "2020-10-28"
        index:
          period: 24h
          prefix: index_
        object_store: filesystem
        schema: v11
        store: boltdb-shipper
    server:
      http_listen_port: 3100
    storage_config:
      filesystem:
        directory: /data/loki/chunks
      boltdb:
        directory: /data/loki/index
      boltdb_shipper:
        active_index_directory: /data/loki/boltdb-shipper-active
        cache_location: /data/loki/boltdb-shipper-cache
        cache_ttl: 24h
        shared_store: filesystem
    table_manager:
      retention_deletes_enabled: true
      retention_period: 1512h

[owen-d]

~Weird, this looks good to me. Apologies for asking, but can you double check that you're running 2.0.0?~

Edit: just tracked this error message to Cortex, it looks like the storage client is registering as "" somehow. Investigating further.

[SockenSalat]

Just to be sure, kubectl says: Image: grafana/loki:2.0.0

[slim-bean]

Yeah I know what causes this, and we should handle this more gracefully.

Running Loki 2.0.0 in the single binary mode with boltdb-shipper index will start the boltdb-shipper compactor.

If no compactor config is defined, Loki will fail with this error message you are seeing.

You will need to add a compactor config similar to this:

compactor:
  working_directory: /data/loki/boltdb-shipper-compactor
  shared_store: filesystem

Then things should work!

[SockenSalat]

@slim-bean Thnx a lot :+1: everthing is fine now.

But a nicer error message in the logs would have been nice.

[hassanfarid]

@slim-bean - This issue is still present. Is loki following longer release cycle now?

[hassanfarid]

You will need to add a compactor config similar to this:

compactor:
  working_directory: /data/loki/boltdb-shipper-compactor
  shared_store: filesystem

Then things should work!

what will happen here? boltdb-shipper-compactor will use the defined location for one time only? do we need to preserve this volume outside docker container?

Also, is this a root level configuration?

[hassanfarid]

I did incorporate the suggested change above, but I see the same error again.

I am upgrading from 1.6.1 to 2.0.0, and configurations look like this.

# The module to run Loki with. Supported values
# all, querier, table-manager, ingester, distributor
target: "all"

# Enables authentication through the X-Scope-OrgID header, which must be present
# if true. If false, the OrgID will always be set to "fake".
auth_enabled: false

# Configures the server of the launched module(s).
server:
    http_listen_port: 3100
    log_level: "info"

# Configures the ingester and how the ingester will register itself to a
# key value store.
ingester:
    lifecycler:
        ring:
            kvstore:
                store: "inmemory"
            replication_factor: 1
        # Duration to sleep before exiting to ensure metrics are scraped.
        # final_sleep: 5s
    chunk_retain_period: 30s
    max_chunk_age: 12h

# temporary configuration for loki:2.0.0
# for using boltdb-shipper-compactor
# ref: https://github.com/grafana/loki/issues/2815
compactor:
    working_directory: /loki/boltdb-shipper-compactor
    shared_store: filesystem

# Configures the chunk index schema and where it is stored.
schema_config:
    configs:
      - from: 2018-04-15
        store: "boltdb"
        object_store: "filesystem"
        schema: "v11"
        index:
            prefix: "index_"
            period: 24h
        chunks:
            prefix: "chunk_"
            period: 24h
      - from: 2020-10-24
        store: "boltdb-shipper"
        object_store: "filesystem"
        schema: "v11"
        index:
            prefix: "index_"
            period: 24h
        chunks:
            prefix: "chunk_"
            period: 24h

# Configures where Loki will store data.
storage_config:
    # Configures storing index in BoltDB. Required fields only
    # required when boltdb is present in config.
    boltdb:
        # Location of BoltDB index files.
        directory: /loki/index
    # Configures storing the chunks on the local filesystem. Required
    # fields only required when filesystem is present in config.
    filesystem:
        # Directory to store chunks in.
        directory: /loki/chunks

# Configures limits per-tenant or globally
limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 1h
    # Maximum number of log entries that will be returned for a query. 0 to disable.
    # [max_entries_limit: <int> | default = 5000 ]

# Configures how Loki will store data in the specific store.
chunk_store_config:
    max_look_back_period: 0s

# Configures the table manager for retention
table_manager:
    chunk_tables_provisioning:
        inactive_read_throughput: 0
        inactive_write_throughput: 0
        provisioned_read_throughput: 0
        provisioned_write_throughput: 0
    index_tables_provisioning:
        inactive_read_throughput: 0
        inactive_write_throughput: 0
        provisioned_read_throughput: 0
        provisioned_write_throughput: 0
    retention_deletes_enabled: true
    retention_period: 720h

[slim-bean]

@hassanfarid I think in this case it might be complaining because you are lacking the boltdb_shipper section in the storage_config:

  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h         
    shared_store: filesystem

This example taken from the example config file

[slim-bean]

This issue is still present. Is loki following longer release cycle now?

Not necessarily, I've fallen behind on the next release, hoping to get to it in the next few days.

[slim-bean]

what will happen here? boltdb-shipper-compactor will use the defined location for one time only? do we need to preserve this volume outside docker container?

The working directory does not need to be persisted, the compactor needs a place to download and store the boltdb files as boltdb requires this.