Victoriametrics: series count API returns old value after calling delete series API

[ledmonster]

Describe the bug
I got old return value from /api/v1/series/count after calling /api/v1/admin/tsdb/delete_series.

To Reproduce

After ingesting multiple time series,

$ curl http://localhost:8428/api/v1/series/count
{"status":"success","data":[150]}

then, delete series from VM.

$ curl -G 'http://localhost:8428/api/v1/admin/tsdb/delete_series' --data-urlencode 'match[]={__name__=~".*"}'

Now, series API returns empty result.

$ curl -G 'http://localhost:8428/api/v1/series' --data-urlencode 'match[]={__name__=~".*"}' --data 'start=0'
{"status":"success","data":[]}

But count API still returns old stats.

$ curl http://localhost:8428/api/v1/series/count
{"status":"success","data":[150]}

Expected behavior

I want to get a correct value from series count API after series deletion.

Version

• v1.41.0

Used command-line flags

• nothing

[dxtrzhang]

Try to clear cache by calling /internal/resetRollupResultCache and see if it returns old value.

[ledmonster]

@dxtrzhang thanks, now I tried that API, but count API still returns old value.

$ curl http://localhost:8428/internal/resetRollupResultCache
$ curl -G 'http://localhost:8428/api/v1/series' --data-urlencode 'match[]={__name__=~".*"}' --data 'start=0'
{"status":"success","data":[]}
$ curl http://localhost:8428/api/v1/labels/count
{"status":"success","data":{}}
$ curl http://localhost:8428/api/v1/series/count
{"status":"success","data":[780]}

[valyala]

This is expected behavior - /api/v1/series/count returns all the series stored in the lookup index used for searching time series by the given labels (aka inverted index). Information about deleted time series isn't deleted from the lookup index for up to 2x-retentionPeriod due to architecture restrictions. This is one of the reasons why it isn't recommended to use delete API for regular deletion of data - see these docs for details.

Note also that /api/v1/series/count scans index entries for all the time series stored in the database on every request. So it may execute slowly if the database contains tens or hundreds of millions of time series.

[ledmonster]

I see. Let me explain my background. Since I implements dynamic logic to ingest metrics from IoT devices to VM, I write unit tests for ingestion program with VM and confirm my logic before deploying it to production.

• start VM container locally
• run tests
• stop container

To make each test independent, I delete time series on tear down. And I noticed that count API doesn't be reset with series deletion. For tests, I can achieve my purpose without count API, so current behavior doesn't matter for me. Just for your information. Thanks.

[valyala]

@ledmonster , thanks for the explanation of your use case.
There are two options exist for such use case:

• To manually delete <-storageDataPath> directory after container is stopped.
• To send count({__name__!=''}) query to /api/v1/query after deleting time series in order to make sure that all the time series are deleted. Note that this query may consume high amounts of resources (CPU, memory) if the database contains big number of time series.

[valyala]

Closing this question as answered.