Loki: Loki not retaining logs more than 5 mins

[AnandPalani92]

I have deployed the loki tool on my kebernetes cluster using helm, i used following commands to create deployments

helm install loki/loki-stack -n loki-stack \                   
  --set fluent-bit.enabled=true,promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=25Gi,loki.config.table_manager.retention_deletes_enabled=true,loki.config.table_manager.retention_period=120h \
  --namespace=logging

I am able to create a deployment without any issues and able to create datasources and pointed to loki service.

Loki explorer displayed the logs for each deployments, after some time i couldn't see the older logs on the Grafana UI.
But I given the data retention for 5 days , I can see only for 5 mins of logs for container.

You can see my configuration:

auth_enabled: false
chunk_store_config:
  max_look_back_period: 0
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
schema_config:
  configs:
  - from: "2018-04-15"
    index:
      period: 168h
      prefix: index_
    object_store: filesystem
    schema: v9
    store: boltdb
server:
  http_listen_port: 3100
storage_config:
  boltdb:
    directory: /data/loki/index
  filesystem:
    directory: /data/loki/chunks
table_manager:
  retention_deletes_enabled: true
  retention_period: 120h

[rfratto]

Hi, I think the problem may be that your retention period is not a multiple of your index period. The configuration docs mentions:

# How far back tables will be kept before they are deleted. 0s disables
# deletion. The retention period must be a multiple of the index / chunks
# table "period" (see period_config).
[retention_period: <duration> | default = 0s]

Your period is currently set to 168h, can you try lowering that to 24h? You might have to wait for up to another 15 minutes to see if the problem goes away.

[rfratto]

This _may_ be related to #1187, which we're releasing as part of 1.1.0 today.

[rfratto]

@anandecemptc Sorry for all the random guesses. The reason you're not getting any logs past 15 minutes is because Grafana limits the amount of logs returned in an individual query to 1000. In your screenshot, you can see that you had reached the max amount of lines for that query.

There's a few things you can do here:

1. Make the query period smaller (i.e., 5 minutes, 10 minutes) and then move the time range back if you want an earlier period
2. Make the query smaller, either through filtering or using different labels.

[AnandPalani92]

Thanks Robert for all your comments.

When i clicked 1 hour period of logs for the POD and I can see the 2
minutes logs only.

I can't get 1 hour log details. how do we achieve this?

Regards,
Anand Palani

On Wed, Dec 4, 2019 at 9:01 PM Robert Fratto [e-mail]
wrote:

Closed #1362 https://github.com/grafana/loki/issues/1362.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grafana/loki/issues/1362?email_source=notifications&email_token=ADZKWSD7Y7L4VLWZI4BB6KLQW7ELZA5CNFSM4JVI3ENKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOVIRUTGQ#event-2854439322,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADZKWSHPPGISZ5WIV4IYGJTQW7ELZANCNFSM4JVI3ENA
.

[AnandPalani92]

@anandecemptc Sorry for all the random guesses. The reason you're not getting any logs past 15 minutes is because Grafana limits the amount of logs returned in an individual query to 1000. In your screenshot, you can see that you had reached the max amount of lines for that query.

There's a few things you can do here:

1. Make the query period smaller (i.e., 5 minutes, 10 minutes) and then move the time range back if you want an earlier period
2. Make the query smaller, either through filtering or using different labels.

When i clicked 1 hour period of logs for the POD and I can see the 2
minutes logs only.

I can't get 1 hour log details. how do we achieve this?

[pstibrany]

Log less! :) (sorry, bad joke)

I don't think there is any solution to 1000 entries limit in Explore tab currently, other than using logcli directly (logcli is a command line client for Loki, which can be used instead of Grafana Explore tab, it can be downloaded from here: https://github.com/grafana/loki/releases).

[pstibrany]

Related: https://github.com/grafana/grafana/issues/19354

[pstibrany]

Note that you can configure Loki data source (in data source settings) to use more max lines.

[AnandPalani92]

Thanks pstibrany.

I am just changed the log lines limit on grafana loki datasources and am able to see all the log lines

[suneeldevabhaktuni]

Hi Anand,
How did you change the log lines limit

[tomweston]

This no longer seems to work

helm upgrade --install prod-loki-stack loki/loki-stack --namespace monitoring --set grafana.enabled=false,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=true,prometheus.server.persistentVolume.enabled=true,loki.config.table_manager.retention_deletes_enabled=true,loki.persistence.enabled=true,loki.config.table_manager.retention_period=120h,loki.persistence.storageClassName=generic,loki.persistence.size=25Gi

loki pod goes into CrashLoopBackOff

prod-loki-stack-0 0/1 CrashLoopBackOff 2 46s

Containers:
  loki:
    Container ID:  docker://712292505e23f51f864e75d118775546a431d241cc0702a60e0822cb5ee883e9
    Image:         grafana/loki:1.6.0
    Image ID:      docker-pullable://grafana/loki@sha256:72fdf006e78141aa1f449acdbbaa195d4b7ad6be559a6710e4bcfe5ea2d7cc80
    Port:          3100/TCP
    Host Port:     0/TCP
    Args:
      -config.file=/etc/loki/loki.yaml
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 20 Oct 2020 21:09:19 +0100
      Finished:     Tue, 20 Oct 2020 21:09:19 +0100
    Ready:          False
    Restart Count:  1
    Liveness:       http-get http://:http-metrics/ready delay=45s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http-metrics/ready delay=45s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /data from storage (rw)
      /etc/loki from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from prod-loki-stack-token-5mfcd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  storage-prod-loki-stack-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prod-loki-stack
    Optional:    false
  prod-loki-stack-token-5mfcd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prod-loki-stack-token-5mfcd
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

It does not seem to like the following parameters:

loki.config.table_manager.retention_deletes_enabled=true,loki.persistence.enabled=true,loki.config.table_manager.retention_period=120h

The pod boots correctly without the above config values however /etc/loki/loki.yaml returns to default values :(

☁  loki-stack [master] ⚡  k exec -it prod-loki-stack-0 -n monitoring -- /bin/sh
/ $ cat /etc/loki/loki.yaml
auth_enabled: false
chunk_store_config:
  max_look_back_period: 0s
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  max_transfer_retries: 0
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
schema_config:
  configs:
  - from: "2018-04-15"
    index:
      period: 168h
      prefix: index_
    object_store: filesystem
    schema: v9
    store: boltdb
server:
  http_listen_port: 3100
storage_config:
  boltdb:
    directory: /data/loki/index
  filesystem:
    directory: /data/loki/chunks
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Any ideas @pstibrany / @tomwilkie ?