osquery occupies too much disk

[dabeike]

Bug report

What operating system and version are you using?

# osqueryi --line "SELECT version, build, platform FROM os_version;"
 version = CentOS Linux release 7.4.1708 (Core)
   build = 
platform = rhel

What version of osquery are you using?

# osqueryi --line "SELECT version from osquery_info;"
version = 3.3.0

What steps did you take to reproduce the issue?

I add a schedule query for osqueryd to collect disk size used by rocksdb :

select sum(size) as bytes_disk from file where path like '/var/osquery/osquery.db/%';

In order to reduce disk occupied by it, I add some limitations in osquery.flags:

But the result still turns out to be like this:

What did you expect to see?

Most of the machines take up less than 200M of disk space, while a few of it takes up to 600M of disk space.

I read the content of the .sst file, and grep it by name. Here is the command:

sudo /tmp/sst_dump --file=/var/osquery/osquery.db/ --command=scan | wc -l
sudo /tmp/sst_dump --file=/var/osquery/osquery.db/ --command=scan  | grep process_events | wc -l 
sudo /tmp/sst_dump --file=/var/osquery/osquery.db/ --command=scan  | grep socket_events | wc -l 

And here is the result:

# host_1
1380917
1281996
62917

# host_2
1149855
1126230
14024

# host_3
1693820
1649105
23780

# host_4
1173181
1117843
45165

Rocksdb saves 100w+ lines, while 99% of it are process_events data. It's obvious that the events_max flag didn't take effect.

I want to know if this is normal. And What should I do to reduce the disk usage? How osqueryd expiring data?

Need your help. Thanks!

[dabeike]

# ls -alh /var/osquery/osquery.db/
Usage 772M
drwx------ 2 root root  20K Nov 28 11:13 .
drwxrwxr-x 3 root root 4.0K Nov 28 05:36 ..
-rw-r--r-- 1 root root 2.1M Nov 22 20:30 053445.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:30 053446.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:40 053488.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:40 053489.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:40 053490.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:40 053491.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:50 053536.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:50 053537.sst
-rw-r--r-- 1 root root 203K Nov 22 20:50 053538.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:50 053539.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:50 053540.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:50 053541.sst
-rw-r--r-- 1 root root 2.1M Nov 22 20:50 053542.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:00 053582.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:00 053583.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:00 053584.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:10 053624.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:10 053625.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:10 053626.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:20 053666.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:20 053667.sst
-rw-r--r-- 1 root root 2.1M Nov 22 21:20 053668.sst
-rw-r--r-- 1 root root  13K Nov 22 21:50 053798.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058692.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058693.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058694.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058695.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058696.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058697.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058698.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058699.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058700.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058701.sst
-rw-r--r-- 1 root root 2.1M Nov 23 15:53 058702.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:03 058748.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:03 058749.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:03 058750.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:13 058788.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:13 058789.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:13 058790.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:13 058791.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:13 058792.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:23 058833.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:23 058834.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:23 058835.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:23 058836.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:33 058874.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:33 058875.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:33 058876.sst
-rw-r--r-- 1 root root 2.1M Nov 23 16:33 058880.sst
-rw-r--r-- 1 root root 1.3M Nov 23 16:33 058881.sst
-rw-r--r-- 1 root root  13K Nov 23 17:03 059005.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060234.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060235.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060236.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060237.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060239.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060240.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060241.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060242.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060243.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060244.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060245.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060246.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060247.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060248.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:23 060249.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:53 060378.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:53 060379.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:53 060380.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:53 060381.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:53 060382.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:53 060383.sst
-rw-r--r-- 1 root root 2.1M Nov 23 21:53 060384.sst
-rw-r--r-- 1 root root 8.7K Nov 23 22:13 060468.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061197.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061198.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061199.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061200.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061201.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061202.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061203.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061204.sst
-rw-r--r-- 1 root root 2.1M Nov 24 00:53 061205.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061343.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061344.sst
-rw-r--r-- 1 root root 526K Nov 24 01:23 061345.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061346.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061347.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061348.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061349.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061350.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061351.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061352.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061353.sst
-rw-r--r-- 1 root root 2.1M Nov 24 01:23 061354.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061567.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061568.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061569.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061570.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061571.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061572.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061573.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061574.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061575.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:13 061576.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061763.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061764.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061765.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061766.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061767.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061768.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061769.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061770.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061771.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061772.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061773.sst
-rw-r--r-- 1 root root 2.1M Nov 24 02:53 061774.sst
-rw-r--r-- 1 root root 2.1M Nov 24 03:43 061986.sst
-rw-r--r-- 1 root root 2.1M Nov 24 03:43 061987.sst
-rw-r--r-- 1 root root 2.1M Nov 24 03:43 061988.sst
-rw-r--r-- 1 root root 2.1M Nov 24 03:43 061989.sst
-rw-r--r-- 1 root root 2.1M Nov 24 03:43 061990.sst
-rw-r--r-- 1 root root 2.1M Nov 24 03:43 061991.sst
-rw-r--r-- 1 root root  17K Nov 24 03:53 062039.sst
-rw-r--r-- 1 root root  15K Nov 24 03:53 062040.sst
-rw-r--r-- 1 root root 101K Nov 24 05:55 062612.sst
-rw-r--r-- 1 root root 2.1M Nov 24 15:45 065373.sst
-rw-r--r-- 1 root root 2.1M Nov 24 15:45 065374.sst
-rw-r--r-- 1 root root 2.1M Nov 24 15:45 065375.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065545.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065546.sst
-rw-r--r-- 1 root root 437K Nov 24 16:25 065547.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065548.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065549.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065550.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065551.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065552.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065553.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065554.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065555.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065556.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065557.sst
-rw-r--r-- 1 root root 2.1M Nov 24 16:25 065558.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065748.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065749.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065750.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065751.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065752.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065753.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065754.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065755.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065756.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065757.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065758.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065759.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065760.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065761.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065762.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:05 065763.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065938.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065939.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065940.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065941.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065942.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065943.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065944.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065945.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065946.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065947.sst
-rw-r--r-- 1 root root 2.1M Nov 24 17:45 065948.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066133.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066134.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066135.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066136.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066137.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066138.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066139.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066140.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066141.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066142.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066143.sst
-rw-r--r-- 1 root root 2.1M Nov 24 18:25 066144.sst
-rw-r--r-- 1 root root  14K Nov 24 18:35 066192.sst
-rw-r--r-- 1 root root  14K Nov 24 18:35 066193.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066331.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066332.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066333.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066334.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066335.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066336.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066337.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066338.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066339.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066340.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:05 066341.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:45 066511.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:45 066512.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:45 066513.sst
-rw-r--r-- 1 root root 2.1M Nov 24 19:45 066514.sst
-rw-r--r-- 1 root root  17K Nov 24 19:55 066567.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:43 073004.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:43 073005.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073050.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073051.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073052.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073053.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073054.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073055.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073056.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073057.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073058.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073059.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073060.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073061.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073062.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073063.sst
-rw-r--r-- 1 root root 2.1M Nov 25 18:53 073064.sst
-rw-r--r-- 1 root root 2.1M Nov 25 20:13 073394.sst
-rw-r--r-- 1 root root 2.1M Nov 25 20:13 073395.sst
-rw-r--r-- 1 root root 2.1M Nov 25 20:13 073396.sst
-rw-r--r-- 1 root root 2.1M Nov 25 20:13 073397.sst
-rw-r--r-- 1 root root 2.1M Nov 25 20:13 073398.sst
-rw-r--r-- 1 root root 2.1M Nov 25 20:13 073399.sst
-rw-r--r-- 1 root root 2.1M Nov 25 20:13 073400.sst
-rw-r--r-- 1 root root  21K Nov 25 20:22 073450.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075184.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075190.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075191.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075192.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075193.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075194.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075195.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075196.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075197.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075198.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075199.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075200.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075201.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075202.sst
-rw-r--r-- 1 root root 2.1M Nov 26 02:53 075203.sst
-rw-r--r-- 1 root root 2.1M Nov 26 04:02 075486.sst
-rw-r--r-- 1 root root 2.1M Nov 26 04:02 075487.sst
-rw-r--r-- 1 root root 2.1M Nov 26 04:02 075488.sst
-rw-r--r-- 1 root root 2.1M Nov 26 04:02 075489.sst
-rw-r--r-- 1 root root 2.1M Nov 26 04:02 075490.sst
-rw-r--r-- 1 root root 2.1M Nov 26 04:02 075491.sst
-rw-r--r-- 1 root root 2.1M Nov 26 04:02 075492.sst
-rw-r--r-- 1 root root  23K Nov 26 04:23 075576.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076000.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076003.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076004.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076005.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076006.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076007.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076008.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076009.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076010.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076011.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076012.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076013.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076014.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076015.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076016.sst
-rw-r--r-- 1 root root 2.1M Nov 26 05:52 076017.sst
-rw-r--r-- 1 root root 2.1M Nov 26 06:23 076142.sst
-rw-r--r-- 1 root root 2.1M Nov 26 06:23 076143.sst
-rw-r--r-- 1 root root 2.1M Nov 26 06:23 076144.sst
-rw-r--r-- 1 root root 2.1M Nov 26 06:23 076145.sst
-rw-r--r-- 1 root root 2.1M Nov 26 06:23 076146.sst
-rw-r--r-- 1 root root 2.1M Nov 26 06:23 076147.sst
-rw-r--r-- 1 root root 2.1M Nov 26 06:23 076148.sst
-rw-r--r-- 1 root root 8.8K Nov 26 06:43 076242.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078452.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078454.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078455.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078456.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078457.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078458.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078459.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078460.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078461.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078462.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078463.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078464.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078465.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078466.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078467.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078468.sst
-rw-r--r-- 1 root root 2.1M Nov 26 14:43 078469.sst
-rw-r--r-- 1 root root 2.1M Nov 26 15:22 078649.sst
-rw-r--r-- 1 root root  15K Nov 26 15:32 078702.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081035.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081038.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081039.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081040.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081041.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081042.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081043.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081044.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081045.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081046.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081047.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081048.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081049.sst
-rw-r--r-- 1 root root 2.1M Nov 26 23:22 081050.sst
-rw-r--r-- 1 root root 2.1M Nov 27 00:02 081240.sst
-rw-r--r-- 1 root root 2.1M Nov 27 00:02 081241.sst
-rw-r--r-- 1 root root 2.1M Nov 27 00:02 081242.sst
-rw-r--r-- 1 root root 2.1M Nov 27 00:02 081243.sst
-rw-r--r-- 1 root root  12K Nov 27 00:12 081294.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081861.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081863.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081864.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081865.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081866.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081867.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081868.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081869.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081870.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:15 081871.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081967.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081968.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081969.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081970.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081971.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081972.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081973.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081974.sst
-rw-r--r-- 1 root root 2.1M Nov 27 02:35 081975.sst
-rw-r--r-- 1 root root 2.1M Nov 27 03:15 082177.sst
-rw-r--r-- 1 root root 2.1M Nov 27 03:15 082178.sst
-rw-r--r-- 1 root root 2.1M Nov 27 03:15 082179.sst
-rw-r--r-- 1 root root 2.1M Nov 27 03:15 082180.sst
-rw-r--r-- 1 root root 2.1M Nov 27 03:15 082181.sst
-rw-r--r-- 1 root root  12K Nov 27 03:25 082233.sst
-rw-r--r-- 1 root root 2.1M Nov 27 13:45 085358.sst
-rw-r--r-- 1 root root 2.1M Nov 27 13:45 085359.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085556.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085557.sst
-rw-r--r-- 1 root root 864K Nov 27 14:25 085558.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085559.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085560.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085561.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085562.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085563.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085564.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085565.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085566.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085567.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085568.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085569.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085570.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085571.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:25 085572.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:55 085715.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:55 085716.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:55 085717.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:55 085718.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:55 085719.sst
-rw-r--r-- 1 root root 2.1M Nov 27 14:55 085720.sst
-rw-r--r-- 1 root root 8.7K Nov 27 15:05 085770.sst
-rw-r--r-- 1 root root  12K Nov 28 03:02 089159.sst
-rw-r--r-- 1 root root 113K Nov 28 03:23 089254.sst
-rw-r--r-- 1 root root 349K Nov 28 04:43 089598.sst
-rw-r--r-- 1 root root 157K Nov 28 06:53 090176.sst
-rw-r--r-- 1 root root 2.1M Nov 28 09:02 090770.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:12 091087.sst
-rw-r--r-- 1 root root 1.1M Nov 28 10:12 091088.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091224.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091225.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091226.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091227.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091228.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091229.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091230.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091231.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091232.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091233.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091234.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091235.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091236.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091237.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:42 091238.sst
-rw-r--r-- 1 root root 555K Nov 28 10:42 091239.sst
-rw-r--r-- 1 root root 2.1M Nov 28 10:52 091282.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:02 091323.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:02 091324.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:02 091325.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:02 091326.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:02 091327.sst
-rw-r--r-- 1 root root 1.7M Nov 28 11:02 091331.sst
-rw-r--r-- 1 root root  12K Nov 28 11:10 091351.sst
-rw-r--r-- 1 root root 412K Nov 28 11:10 091354.sst
-rw-r--r-- 1 root root  422 Nov 28 11:12 091357.log
-rw-r--r-- 1 root root  237 Nov 28 11:12 091358.log
-rw-r--r-- 1 root root 3.2K Nov 28 11:12 091359.log
-rw-r--r-- 1 root root  507 Nov 28 11:12 091360.log
-rw-r--r-- 1 root root  995 Nov 28 11:12 091361.log
-rw-r--r-- 1 root root 2.1M Nov 28 11:12 091363.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:12 091368.sst
-rw-r--r-- 1 root root 369K Nov 28 11:12 091369.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:12 091370.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:12 091371.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:12 091372.sst
-rw-r--r-- 1 root root 2.1M Nov 28 11:12 091373.sst
-rw-r--r-- 1 root root 1.8M Nov 28 11:12 091374.sst

The output shows that logs of .sst file remains in the disk in the past few days which should be cleared when the log size reached 5000.

[dabeike]

I found that osquery will expire events when starting, so I made a test.
Firstly, I saw the numbers of socket_events log before restart osqueryd

sst_dump --file=/var/osquery/osquery.db/ --command=scan  | grep socket_events | wc -l
577486

And then executed:

systemctl restart osqueryd

And got this in /var/log/osquery/osquery.INFO:

Log file created at: 2018/11/28 16:07:35
Running on machine: hostname
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1128 16:07:35.702229 31583 events.cpp:862] Event publisher not enabled: syslog: Publisher disabled via configuration
I1128 16:07:45.402627 31583 events.cpp:1115] Error registering subscriber: process_file_events: Subscriber disabled via configuration
I1128 16:07:48.002794 31583 events.cpp:1115] Error registering subscriber: selinux_events: Subscriber disabled via configuration
W1128 16:07:50.805047 31583 events.cpp:311] Expiring events for subscriber: socket_events (overflowed limit 5000)
I1128 16:07:50.805352 31583 events.cpp:313] Subscriber events socket_events exceeded limit 5000 by: 15280
I1128 16:07:59.304909 31583 events.cpp:744] Subscriber expiration is too low: process_events
I1128 16:07:59.509066 31927 events.cpp:783] Starting event publisher run loop: auditeventpublisher
I1128 16:07:59.509125 31929 events.cpp:783] Starting event publisher run loop: inotify
I1128 16:07:59.509127 31930 events.cpp:783] Starting event publisher run loop: udev
I1128 16:07:59.509690 31925 auditdnetlink.cpp:302] Attempting to configure the audit service
I1128 16:07:59.509744 31925 auditdnetlink.cpp:320] Enabling audit rules for the socket_events table
I1128 16:07:59.509760 31925 auditdnetlink.cpp:329] Enabling audit rules for the process_events table

And in /var/log/osquery/osquery.WARNING:

Log file created at: 2018/11/28 16:07:50
Running on machine: hostname
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W1128 16:07:50.805047 31583 events.cpp:311] Expiring events for subscriber: socket_events (overflowed limit 5000)

And then saw socket_event count again:

./sst_dump --file=/var/osquery/osquery.db/ --command=scan  | grep socket_events | wc -l
571097

It's very strange. I have 500,000+ lines of socket_events log actually, but osqueryd only found (5000+15280=20280) of it.

And expiring while inserting data into rocksdb seems to be stopped. I waited for half an hour, and executed:

> sst_dump --file=/var/osquery/osquery.db/ --command=scan  | grep socket_events | wc -l
576331
> cat /var/log/osquery/osqueryd.INFO | grep "Subscriber events"
I1128 16:07:50.805352 31583 events.cpp:313] Subscriber events socket_events exceeded limit 5000 by: 15280

The only log is logged when starting osqueryd...

It seems that osquery lost the control of old log in rocksdb. This should be why osquery occupies so much disk in a few of my machines.

[dabeike]

I watched the source code, and found some logical error in it.

Firstly, the expireCheck is not running as expect.

And normally, events datas in rocksdb should be devided into two types:

• records.auditeventpublisher.socket_events: it records the index_key for the real socket_events logs
• data.auditeventpublisher.socket_events: it records the real log

When it's normal, it should be like this:

In the machines which didn't remove old logs, it didn't have records log:

I think it's why osqueryd can't expire these old logs, it just can't found the data log without records log.

But I don't know why osquery removed records log without remove data log. It happens in 10% of all my machines, but it affects on-line progress heavily. My solution now is remove *.sst file older than 1 day by bash shell, but I do want to find an elegant solution for this problem.

Looking forward for your reply. TKS!

[guliashvili]

This is great catch. Thank you for finding it. Will fix it as soon as I have some time.

[guliashvili]

5335 should have solved it

[uptycs-nishant]

@dabeike

./sst_dump --file=/var/osquery/osquery.db/ --command=scan | grep socket_events | wc -l

Does not mean that all the data is coming from events column family. Reading from sst files include everything

Use -
ldb --db= for exact count of events

[dabeike]

thanks, but I don't think the PR has fix all the problems. There's still a lots of old sst file in rocksdb's directory.

The content of the sst file are mainly datas. In this machine, it has 1,021,003 lines. These data's eid are not logged in any records and indexes.

I review the function expireRecord and expireIndexes in events.cpp, trying to find why osquery deleted the records and indexes line without deleted the data line. But nothing found.

[packetzero]

Do you have extensions enabled? Can you turn off extensions and try again? It looks like a code path in checkExpire() will call scanDatabaseKeys(), and if there's an external registry, the database will never be scanned... leading to event cache that never gets cleaned up. I'm going to guess that most of us that tried to reproduce this did it without extensions on.

[dabeike]

Sorry to say, I didn't use any extension. Does osquery have any default extension?
I found most of the logs are starting from Feb 15. I remember that I should have changed some config using tls (doorman), maybe there is some problem with config change?

I can send you a pack of db files. Maybe it can stimulate your inspiration.

[packetzero]

I must be wrong about that then. the get().external() must mean "if I am an extension" rather than "if I have an extension"

[dabeike]

Is there any multi_processing used when inserting when to rocksdb? If expire_records are not sorted, it's possible for osquery to delete the indexes and records without delete all the data

[uptycs-nishant]

@dabeike , is it possible for you to provide the DB for analysis?

[dabeike]

@uptycs-nishant sorry for seeing this message so late.
Yes, I can send you the db files. Please give me your email address, or contact me at slack (@fr1day). I searched your name in the slack's osquery #general group, but couldn't find you.