Influxdb: When delete by time range, some points not inside that range gets deleted

[inetfuture]

Bug report

__System info:__ [Include InfluxDB version, operating system name, and other relevant details]

influxdb 1.1.0

__Steps to reproduce:__

1. insert some points
2. delete by a time range

__Expected behavior:__ [What you expected to happen]

Only points inside that range gets deleted

__Actual behavior:__ [What actually happened]

Some points not inside that range gets deleted

__Additional info:__ [Include gist of relevant config, logs, etc.]

Here is the screenshot:

As you can see, after I deleted points from 1481058000000000000(2016-12-06T21:00:00.000Z) to
1496869200000000000(2017-06-07T21:00:00.000Z), points between 2016-12-04T21:00:00Z and 2016-12-05T21:00:00Z are missing.

I can not reliably reproduce this, with same set of points, sometimes points inside that range gets deleted, sometimes not.

I'm guessing it's related to the insertion order of points, which is the only different among my many tests with same set of points. I'm aggregating data from mongodb then insert them into influxdb, therefore the insertion order is not stable.

[inetfuture]

Is this somewhat related to #7582?

[inetfuture]

any progress on this?

[dehorsley]

I think I have encountered the same bug. If it is the same as mine, the data is still there but does not show up in all queries. In my case, if my query includes times in the range between the start of the deleted window and 110s after it, all values up to the end of the shard gets excluded.

Eg.
After running

> delete from met where time > '2016-10-12T17:04:03Z' and time < '2016-10-12T17:11:30Z'

running

> SELECT count(pressure) from met where time > '2016-10-12T17:04:03Z' AND time < '2016-10-17T00:00:00Z'

shows empty. But

> SELECT count(pressure) FROM met WHERE time > '2016-10-12T17:05:54Z' AND time < '2016-10-17T00:00:00Z'
name: met
time            count
----            -----
1476291954000000001 123370

Note 2016-10-17T00:00:00Z is the end of the shard. Values from other shards are included in all queries.

I am trying to construct a minimal example, but it does not seem to happen every time.

[corylanou]

@inetfuture can you show me what your retention policy duration was? Was it just the default (inf) or did it have a custom duration?

[corylanou]

For more context, I tried the following on 1.1.

> create database test
> use test
Using database test
> precision rfc3339
> insert hourlyReservations,restaurant=NonTheBund count=1 1480910400000000000
> insert hourlyReservations,restaurant=NonTheBund count=1 1480914000000000000
> insert hourlyReservations,restaurant=NonTheBund count=1 1480917600000000000
> insert hourlyReservations,restaurant=NonTheBund count=3 1480932000000000000
> insert hourlyReservations,restaurant=NonTheBund count=10 1480935600000000000
> insert hourlyReservations,restaurant=NonTheBund count=7 1480939200000000000
> SELECT * from hourlyReservations where time >= '2016-12-04T21:00:01Z' and time <= '2016-12-05T21:00:03Z' and restaurant='NonTheBund'
name: hourlyReservations
time                    count   restaurant
----                    -----   ----------
2016-12-05T04:00:00Z    1       NonTheBund
2016-12-05T05:00:00Z    1       NonTheBund
2016-12-05T06:00:00Z    1       NonTheBund
2016-12-05T10:00:00Z    3       NonTheBund
2016-12-05T11:00:00Z    10      NonTheBund
2016-12-05T12:00:00Z    7       NonTheBund

> DELETE from hourlyReservations where time >= 1481058000000000000 and time <= 1496869200000000000
> SELECT * from hourlyReservations where time >= '2016-12-04T21:00:01Z' and time <= '2016-12-05T21:00:03Z' and restaurant='NonTheBund'
name: hourlyReservations
time                    count   restaurant
----                    -----   ----------
2016-12-05T04:00:00Z    1       NonTheBund
2016-12-05T05:00:00Z    1       NonTheBund
2016-12-05T06:00:00Z    1       NonTheBund
2016-12-05T10:00:00Z    3       NonTheBund
2016-12-05T11:00:00Z    10      NonTheBund
2016-12-05T12:00:00Z    7       NonTheBund

[corylanou]

If you are able to get this to happen, can you also provide an output of SHOW SHARDS. I suspect it has to do with the edge of a shard start/end time

[jwilder]

Might be related to: #7798

[inetfuture]

@inetfuture can you show me what your retention policy duration was? Was it just the default (inf) or did it have a custom duration?

just the default

[inetfuture]

If you are able to get this to happen, can you also provide an output of SHOW SHARDS. I suspect it has to do with the edge of a shard start/end time

sorry, I don't have that environment any more...

[genofire]

We recognize the same problem on influxdb version 1.2.1 in the application.
By sending delete from %s where time < now() - %ds to these influxdb.

We make a issue on these project: https://github.com/FreifunkBremen/yanic/issues/40

[dehorsley]

My data returned after compaction completed.

[mantzas]

my delete statement was time ranged deleting 1 day (time >= '2017-03-13 00:00:00' AND time < '2017-03-14 00:00:00') but in the end it deleted 13 and some data of the 14th and 15th.

this is a very serious problem.

BTW i am on v1.2

[max3163]

Same probleme here.
I try to delete some data by time range in a measurement
Exemple :
DELETE FROM events WHERE application = 'xxxx' AND time < 1497945908227000001 AND time > 1497945908226999999
Data are delete as expected in events but all others measurements from the db are deleted too for a period of 40 minutes... except in the 'events' measurement....
It happen randomly but it's a serious issues...

[jwilder]

This might be fixed via #8576

[javitorl]

same problem here with version 1.3.5, it's not fixed yet