Jaeger: Jaeger is slowly losing operations that are still available via the API

[Azim-Palmer]

Describe the bug
I have Jaeger hooked up to Elasticsearch and have it running for over a month. Over time my operations count via the jaeger ui went up to 52, however after about a week I noticed it dropped down to 32 and a day later has risen to 46.

To Reproduce
Steps to reproduce the behavior:

1. Have Jaeger hooked up to Elasticsearch for over a month
2. Create a number of different spans over the course of that month
3. Wait several days (sorry I can't be more specific)
4. Observe the # of operations has dropped

Expected behavior
Every historical operation that has spans logged against it should be available via the UI

Version (please complete the following information):

• OS: Linux
• Jaeger version: 1.17
• Deployment: Docker

What troubleshooting steps did you try?
Checked the Elasticsearch index and all the historical operations are there (and there's 100 of them)
Confirmed that the operations aren't present on the response of via /api/services/SERVICENAME/operations

I've confirmed that I can still get the spans via the api via /api/traces?end=1598868060000000&limit=200&lookback=custom&maxDuration&minDuration&operation=WebRequest%3A%2Fquotes%2F19&service=SERVICENAME&start=1588305660000000 even though the operation name isn't in the UI dropdown

Additional context
Just in case it's important, I'm using a const sampler at the moment so it's logging everything !

[jpkrohling]

Confirmed that the operations aren't present on the response of via /api/services/SERVICENAME/operations

This sounds like a problem in the ES backend, not UI.

[Azim-Palmer]

@jpkrohling How would I go about confirming that?

I've "Checked the Elasticsearch index and all the historical operations are there (and there's 100 of them)" - to elaborate I created an index patten on "jaeger-service-*" and it's returning everything that I'm expecting to see.

I don't believe I've changed anything significant on the Elastic side, the instance is literally just for Jaeger.

[jpkrohling]

Sorry, I meant the ES-specific part of the Jaeger backend. Likely around here:

https://github.com/jaegertracing/jaeger/blob/66d0d20b98126c59ce687f4d6ccfe7c28e0fc4f1/plugin/storage/es/spanstore/reader.go#L247-L270

[goku321]

Want to take this one up.

[jpkrohling]

It's yours. Let us know if you need help.

[goku321]

Why are we subtracting 24 here?
https://github.com/jaegertracing/jaeger/blob/66d0d20b98126c59ce687f4d6ccfe7c28e0fc4f1/plugin/storage/es/spanstore/reader.go#L179

[yurishkuro]

I assume because indices are expected to be daily