Loki: Grafana errors when using GCS with Loki

[kerneljack]

Hi, I'm trying to use Google Cloud Storage with Loki and I wasn't sure what the config file format should be so I looked at the code and roughly worked out the following:

apiVersion: v1
data:
  loki.yaml: |
    auth_enabled: false

    server:
      http_listen_port: 3100

    ingester:
      lifecycler:
        ring:
          store: inmemory
          replication_factor: 1
    schema_config:
      configs:
      - from: 0
        store: gcp
        object_store: gcp
        schema: v9
        index:
          prefix: index_
          period: 168h
    storage_config:
      gcs:
        bucket_name: my_testbucket_name
kind: ConfigMap
metadata:
  labels:
    app: loki
  name: loki
  namespace: loki

I'm using the following Deployment for Loki with a GOOGLE_APPLICATION_CREDENTIALS environment variable which points to a Secret. This Secret contains Service Account credentials for a Service Account I created specifically for Loki. The permissions I have given to my bucket are "Storage Object Admin" and "Storage Object Creator".

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: loki
  name: loki
  namespace: loki
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: loki
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: loki
    spec:
      containers:
      - env:
          - name: GOOGLE_APPLICATION_CREDENTIALS
            value: /var/secrets/google/key.json
        args:
        - -config.file=/etc/loki/loki.yaml
        image: grafana/loki:master
        imagePullPolicy: Always
        name: loki
        ports:
        - containerPort: 3100
          name: loki
          protocol: TCP
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/loki
          name: config
        - name: google-cloud-key
          mountPath: /var/secrets/google
        resources:
          requests:
            cpu: 300m
            memory: 1Gi
          limits:
            cpu: 300m
            memory: 1Gi
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: loki
      serviceAccountName: loki
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: loki
        name: config
      - name: google-cloud-key
        secret:
          secretName: loki-gcs-secret

I can confirm that the correct env var gets created and that the correct service account credentials DO exist in /var/secrets/google/key.json. I can see no errors from Loki in the pod logs when it starts up so I assume my GCS config is correct, but am not sure.

When I now attempt to use Grafana, it's happy to add the datasource, and it can see all the labels that Promtail is sending to Loki, but it gives me the following error when I attempt to query a particular pod's logs:

rpc error: code = InvalidArgument desc = When parsing 'projects//instances//tables/index_2560' : Table name expected in the form 'projects/<project_id>/instances/<instance_id>/tables/<table_id>'.

I've attached a screenshot of the same error in case it's helpful.

[davkal]

Hi! Looks like you're missing the bigtable entry in the storage config. Here is how it should look like in jsonnet:
https://github.com/grafana/loki/blob/ab4c5bebab7f3e8a371e3caa9e1b8b4e55369d21/production/ksonnet/loki/config.libsonnet#L55

Should be straightforward to translate this yaml. Hope that helps.

[kerneljack]

@davkal thanks, I don't have a bigtable instance though, all I'm trying to do is to store the data in GCS, instead of the default /tmp/loki/chunks and /tmp/loki/index - is this not possible?

Do I need a bigtable instance in addition to GCS?

[nouney]

@davkal so, do we need a bigtable instance?

[hypnoglow]

GCS is desirable and cheap storage for logs for many cases, but Bigtable can be quite expensive for small projects. Can we find out a solution for GCS chunk storage that is not bound to Bigtable? Any help wanted?

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[nouney]

That sucks, a lot.

[garbetjie]

Just a note on this. It seems like you need to supply configuration values for the bigtable storage, but they don't actually need to exist.

For example, the snippet below is an extract from my configuration that seems to work so far:

schema_config:
  configs:
    - from: 2000-01-01
      store: boltdb
      object_store: gcs
      schema: v10
      index:
        prefix: idx_
        period: 168h

storage_config:
  bigtable:
    instance: nonexistent-instance
    project: nonexistent-project
  gcs:
    bucket_name: my-loki-logs
  boltdb:
    directory: /data/loki/indexes