Smokedetector: Heavy disk write in long run

[iBug]

I set up a local instance (iBot) of Smokey in a Docker container, and recently I noticed unintelligibly high disk write amount (read from Block I/O column of docker stats). At present, the container that's been up for 7 days has accrued a total block write amount of more than 150 GiB, which divides into roughly 20 GiB per day.

Upon closer inspection, I noticed that the file bodyfetcherQueueTimings.p was written for once every few seconds, and at the time it had a size of 2.1 MiB. For every update its size grew by a few bytes. Given that it's written once for every API call to fetch posts from SE and that we're using around 10k of our daily quota, it's most likely the culprit for such a huge write amount.

Per my knowledge, that pickle file doesn't have any use for Smokey's primary functionalities (it's for data analysis by human), and this file alone is wearing out SSDs very quickly (one Samsung 970 EVO 500 GB model in less than 3 years if unattended, calculating from raw bytes). Should we take some measures to reduce unnecessary hardware workloads?

[ArtOfCode-]

That file does get written a lot, but even so, 20G/day seems excessive. Not sure what's going on there.

[iBug]

@ArtOfCode- Referencing angussidney, we're doing 10k API calls each day, and each API call writes the file once, so 2 MiB * 10k getting 20 GiB is rather accurate.

[ArtOfCode-]

Sure, but 2M per write also seems excessive. I'm sure it's not meant to be logging that much data.

[ArtOfCode-]

_Definitely_ excessive. Each API call should only be logging a few values - the number of seconds spent in the queue for each post in that site's queue, which is no more than a handful. That's something like 100 _bytes_ per API call, not 2 MiB. Something's wrong here.

[angussidney]

Every time make_api_call_for_site is run, it logs the queue timings to disk using the store_queue_timings() method:

https://github.com/Charcoal-SE/SmokeDetector/blob/master/bodyfetcher.py#L154

This then calls the _dump_pickle method:

https://github.com/Charcoal-SE/SmokeDetector/blob/master/datahandling.py#L333

which passes the "wb" option to the built-in open() function. According to the Python documentation, this truncates and overwrites the entire file.

https://github.com/Charcoal-SE/SmokeDetector/blob/master/datahandling.py#L58

Given the matching stats that iBug has given us, this seems to be the cause of the issue.

[angussidney]

I have two proposals for how we can go about fixing this issue:

1. Disable queue timing logging by default.
   This data isn't necessary for Smokey to correctly function, and isn't accessed during normal operation. We only use this data on an infrequent basis; thus, it would make sense to disable this logging by default, and only enable it on the rare occasion that we actually need to collect the data (e.g. by passing some form of option to the startup command, like we have for standby etc)

2. Change the data to a format which can be easily be appended to the end of a file, so that it doesn't require an overwrite.
   Pickle, as a serialised format, is difficult to modify without loading the binary data into Python objects, changing it, and re-writing the data to file. This downside is worth it if you're dealing with complex Python objects that are hard to store by other means; however, we're only using it to store relatively simple data (a series of lists of integers (or floats?), with each number representing the time the post waited for an API request):

   queue_timings = {
       "site": [int, int, int...],
       ...
   }
   

   (side note: wouldn't our current implementation benefit from the use of collections.defaultdict?)

   Given that all the analysis is done by a totally separate script, I don't think we're benefiting greatly from pickle's ability to store python objects directly. Instead, I propose that we take a simpler approach, and convert the data to one timing per line, in the format site timing (e.g. stackoverflow.com 231). Then, at a later time the data analysis script can aggregate the data into a dictionary, or whatever other format is most convenient for analysis. This method has the advantage that new data can easily be appended to the end of the file without having to overwrite it, thus saving disk writes.

Better ideas for a new data format are welcomed.

[angussidney]

Some initial changes have been started here: https://github.com/Charcoal-SE/SmokeDetector/tree/3243-heavy-disk-usage-fix

[stale[bot]]

This issue has been closed because it has had no recent activity. If this is still important, please add another comment and find someone with write permissions to reopen the issue. Thank you for your contributions.