Thanos: docs/store: Document binaryindex-header usage and characteristics.

[bwplotka]

It would be nice to do before release.

AC:

• Explain how to use index-header for end-user.
• Explain characteristics (low baseline usage, but query spikes are expected)
• Note that it is still experimental.

cc @metalmatze @squat @brancz

[brancz]

We're still releasing this as an experimental feature, I think this is mostly required for when we make it default, but I agree. I think the original design is nice but maybe to detailed for the end user, so I do agree. How about we start a new section in the docs "Operational Guide"? I quite like this in etcd's op-guide section, where various aspects from alerting to expected/unexpected behaviors are documented from an operational perspective. What do you think?

[bwplotka]

Agree, I think Operational Guide sounds like good place for this content. We already have something like Troubleshooting.md that we should put there as well. Thanks 👍

[gburek-fastly]

I suggest an additional warning about open fd limits with this option.

Our store nodes are each responsible for 7989 blocks in GCS and, until now, we have run store under systemd with a LimitNOFILE=4096. Enabling this flag meant an index header for each block and an open file for each block, resulting in our process_open_fds for thanos store going from 11 to the limit very quickly.

Somehow, store came online but seemed to be unresponsive to StoreAPI queries until we raised the LimitNOFILE to 16384 and restarted. Our process_open_fds then stabilized at ~8000, or 11 + num of blocks returned for the node.

In hindsight the change documented here suggests that it will result in more open files, but I didn't immediately make the link between mmaped files and a ulimit setting for the process. Maybe a reminder to check ulimit would be helpful for others.

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[jojohappy]

Still valid.

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

AC:

• Document ulimit impact
• Explain how index-header is obtained (not kept in compactor)
• Explain the characteristics

Potentially in compactor.md

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[pracucci]

_There's the PR #2668 open._