Thanos: store: Improve help of --data-dir flag.

[onitake]

Thanos, Prometheus and Golang version used:

quay.io/thanos/thanos:v0.12.0-rc.0

Object Storage Provider:
None

What happened:
When trying to run thanos store without an object storage config, I get the following error:

flag objstore.config-file or objstore.config is required for running this command and content cannot be empty.

What you expected to happen:
Thans runs and only uses the data-dir, without trying to write to external object storage.

How to reproduce it (as minimally and precisely as possible):
Based on the example k8s manifest from https://github.com/thanos-io/kube-thanos/blob/master/manifests/thanos-store-statefulSet.yaml, modify the arguments as follows:

      containers:
      - args:
        - store
        - --data-dir=/var/thanos/store
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        env:

Full logs to relevant components:

level=info ts=2020-04-14T19:14:29.794693798Z caller=main.go:152 msg="Tracing will be disabled"
level=error ts=2020-04-14T19:14:29.794941253Z caller=main.go:186 err="flag objstore.config-file or objstore.config is required for running this command and content cannot be empty.\ngithub.com/thanos-io/thanos/pkg/extflag.(*PathOrContent).Content\n\t/go/src/github.com/thanos-io/thanos/pkg/extflag/pathorcontent.go:69\nmain.runStore\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/store.go:199\nmain.registerStore.func1\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/store.go:111\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:184\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\npreparing store command failed\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:186\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"

Anything else we need to know:
The documentation suggests that writing to object storage is completely optional. For a simple proof of concept setup, I want to run thanos-store without object storage, but it seems like this is not possible.

Do I have to use a FILESYSTEM object storage target instead? Why do I also need to specify a --data-dir in that case?

[GiedriusS]

Thanos, Prometheus and Golang version used:

quay.io/thanos/thanos:v0.12.0-rc.0

Object Storage Provider:
None

What happened:
When trying to run thanos store without an object storage config, I get the following error:

flag objstore.config-file or objstore.config is required for running this command and content cannot be empty.

What you expected to happen:
Thans runs and only uses the data-dir, without trying to write to external object storage.

How to reproduce it (as minimally and precisely as possible):
Based on the example k8s manifest from https://github.com/thanos-io/kube-thanos/blob/master/manifests/thanos-store-statefulSet.yaml, modify the arguments as follows:

      containers:
      - args:
        - store
        - --data-dir=/var/thanos/store
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        env:

Full logs to relevant components:

level=info ts=2020-04-14T19:14:29.794693798Z caller=main.go:152 msg="Tracing will be disabled"
level=error ts=2020-04-14T19:14:29.794941253Z caller=main.go:186 err="flag objstore.config-file or objstore.config is required for running this command and content cannot be empty.\ngithub.com/thanos-io/thanos/pkg/extflag.(*PathOrContent).Content\n\t/go/src/github.com/thanos-io/thanos/pkg/extflag/pathorcontent.go:69\nmain.runStore\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/store.go:199\nmain.registerStore.func1\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/store.go:111\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:184\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\npreparing store command failed\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:186\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"

Anything else we need to know:
The documentation suggests that writing to object storage is completely optional. For a simple proof of concept setup, I want to run thanos-store without object storage, but it seems like this is not possible.

Do I have to use a FILESYSTEM object storage target instead? Why do I also need to specify a --data-dir in that case?

I feel like there's some confusion here. Thanos Store _is_ the component which satisfies StoreAPI calls by using the data from a specified remote object storage. Thus, this is expected. So, it is definitely not needed if you only want to use local data.

Instead, you should use Thanos Sidecar _without_ --objstore.config which will disable uploads, and Thanos Query which understands StoreAPI and can ask Thanos Sidecar for data, and show you the results in a Prometheus compatible way. Does that help? :P

[onitake]

Instead, you should use Thanos Sidecar _without_ --objstore.config which will disable uploads, and Thanos Query which understands StoreAPI and can ask Thanos Sidecar for data, and show you the results in a Prometheus compatible way. Does that help? :P

Well, that is exactly what I did and how I got the error above. :smiley:

[onitake]

Oh wait, scratch that.

You mean I should run only the sidecar and not use a store instance at all?

[GiedriusS]

Oh wait, scratch that.

You mean I should run only the sidecar and not use a store instance at all?

Yes, that's what I mean. Thanos Store only uses data in the specified remote object storage hence it needs that parameter and Thanos Sidecar is the adapter component for Prometheus.

[onitake]

I see.

I was under the impression that a data store instance is always needed, but that it would work without a backing storage service. I think I'll got for a FILESYSTEM store in that case.

In any case, what is the exact purpose of the data-dir? Does it need to be 10GiB and on persistent storage?

[yeya24]

The data dir is used to cache the block meta file and index. Maybe we should make this description more clear.
Maybe we should change it to Data directory in which to cache the index and meta file of remote blocks.

    dataDir := cmd.Flag("data-dir", "Data directory in which to cache remote blocks.").
        Default("./data").String()

[onitake]

That would certainly be helpful.

And I have another question - also apologies if this is already answered in the documentation and I didn't read it well enough:
Can thanos-store take care of storing data by itself, or does thanos-sidecar need to do this?

If solely thanos-sidecar is responsible for storing, then it would (obviously) make no sense to run thanos-store with a FILESYSTEM reference that isn't shared with thanos-sidecar, as there would never be any data in it.

[bwplotka]

Acceptance Criteria for this ticket then:

• Improve help of --data-dir flag on store

Right? (:

[bwplotka]

cc @Harshitha1234

[Harshitha1234]

That would certainly be helpful.

And I have another question - also apologies if this is already answered in the documentation and I didn't read it well enough:
Can thanos-store take care of storing data by itself, or does thanos-sidecar need to do this?

If solely thanos-sidecar is responsible for storing, then it would (obviously) make no sense to run thanos-store with a FILESYSTEM reference that isn't shared with thanos-sidecar, as there would never be any data in it.

Thanos Sidecar is responsible for implementing Store API on top of Prometheus and as mentioned in the documentation it is optionally used to upload metrics to object storage and allow Queriers to query Prometheus data with common, efficient StoreAPI.

Whereas the Thanos Store (Store Gateway) acts as an API gateway and is responsible for implementing the Store API on top of historical data in an object storage bucket. And it maintains a small amount of information about all the remote blocks on local disk and keeps it in sync with the bucket. And when it joins a Thanos cluster on startup it advertises the data that it can access. And it supports index cache for speeding up postings and series lookups from TSDB blocks indexes and to know more about the Sidecar and its supported cache types feel free to go through the below link.
https://thanos.io/components/store.md/#index-cache

So in a nutshell, the --data-dir flag on Store points to the disk space for local cache.

[bwplotka]

:+1:
Exactly! Can we improve flag help as well? :hugs:

https://github.com/thanos-io/thanos/blob/8cb72e5fc6cdf37661ba14a701571c8e7b7b79b3/cmd/thanos/store.go#L48

[Harshitha1234]

Sure @bwplotka

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[onitake]

Thank you, I think that should address my questions.