Loki: Retention period should now be a multiple of periodic table duration

[uhthomas]

Describe the bug
Cannot create a fresh instance of Loki when specifying retention in the helm chart.

To Reproduce
Steps to reproduce the behavior:

1. Create a new Helm release and PVC
2. Apply the manifests
3. Check the logs

Expected behavior
Loki should startup normally and adhere to the retention period.

Environment:

• Infrastructure: Kubernetes v1.17.5
• Deployment tool: Flux and Helm operator v1.0.2

Screenshots, Promtail config, or terminal output

Helm release and PVC:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: loki
  namespace: grafana
spec:
  releaseName: loki
  chart:
    repository: https://grafana.github.io/loki/charts
    name: loki
    version: 0.29.0
  values:
    persistence:
      enabled: true
    config:
      chunk_store_config:
        max_look_back_period: 720h
      table_manager:
        retention_deletes_enabled: true
        retention_period: 720h
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: loki
  namespace: grafana
  finalizers:
  - kubernetes.io/pvc-protection
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  volumeMode: Filesystem

Logs:

➜  ~ kubectl -n grafana logs loki-0 loki
level=info ts=2020-05-29T21:14:39.625097812Z caller=server.go:179 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=error ts=2020-05-29T21:14:39.625400562Z caller=log.go:140 msg="error initialising loki" err="retention period should now be a multiple of periodic table duration\nerror initialising module: table-manager\ngithub.com/grafana/loki/pkg/loki.(*Loki).initModuleServices\n\t/src/loki/pkg/loki/loki.go:194\ngithub.com/grafana/loki/pkg/loki.New\n\t/src/loki/pkg/loki/loki.go:134\nmain.main\n\t/src/loki/cmd/loki/main.go:76\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"

[owen-d]

Hey, Loki requires that retention (720h in your case) is divisible by the duration of each index table (168h by default). In this case, Loki creates weekly (168h) tables for its index and rotates through them before dropping them once they've elapsed your retention. You can try setting retention_period to 672, which is 168 * 4, allowing you 4 weeks of retention, or any other multiple according to your retention requirements.

Does that make sense?

[uhthomas]

That seems a little convoluted. I think the major confusion (for me) is the docs aren't very clear on it, and the examples don't seem to match reality. Will change the values and let you know how I get on. Thank you!

[shapeofarchitect]

@uhthomas did that work for you ?

[uhthomas]

@shapeofarchitect Yep! Take a look at my live manifest for an example.

Docs need to be updated though as it's very unclear.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[uhthomas]

I don't think this issue is ready to be closed. The documentation still needs to reflect reality.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[uhthomas]

not stale

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[steowens]

So basically the distribution fails out of the box because the settings set you up for failure. Pretty lame if you ask me, if you can't adjust the periodic table setting and the retention period needs to be a multiple of the periodic table then a good software engineer would at least set the default configuration to such a multiple with comments in the distributed config that explain what valid values are. A better software engineer would change the retention period to be a simple setting of retention weeks with values indicating number of weeks to retain thus not requiring every user to do maths every time they adjust the settings.

[cyriltovena]

Totally fair, I'd gladly take some help here too it's open source.