Sentry: Database size has not shrinked after cleanup

[darhlyk]

My sentry.db file size was about 1go, so I did a cleanup that took several hours, but afterwards the size remain unchanged as the event counts in sentry, and if I run again the cleanup command it finishes immediately like if there's nothing to cleanup anymore.

[dcramer]

Welcome to SQL :)

[dcramer]

(sqlite is not a supported production engine, but all RDBMS engines have this issue, see also https://wiki.postgresql.org/wiki/VACUUM_FULL)

[darhlyk]

Hmmm right, I'll do a cron job to cleanup things to keep reusing that space, but still is it a feature that event counts didn't changed?

[mattrobenolt]

Fwiw, it's generally a really bad idea to run a VACUUM FULL unless you expect downtime. It will lock each table until it's done. 

With that said, the way Postgres handles is is the space from all the deleted rows is available for new rows. It's just not reclaimed by the OS. So if kept up with cleaning, the total size should never really grow much since it'll keep writing new rows over the space occupied by the old ones.

If you wait a long time, then delete data, your usage will likely plateau since new rows are solely working within already claimed space. 

You can also claim space immediately with a tool such as pg_repack. 

This is very specific to Postgres (which we fully support and recommend using) but should be pretty similar across the board. 

Hope this helps. 

We should add this to the docs or something since it's asked often. 

--Matt Robenolt

@mattrobenolt

On Tue, Nov 4, 2014 at 2:18 AM, darhlyk [email protected] wrote:

Hmmm right, I'll do a cron job to cleanup things to keep reusing that space, but still is it a feature that event counts didn't changed?

Reply to this email directly or view it on GitHub:
https://github.com/getsentry/sentry/issues/1304#issuecomment-61619080

[darhlyk]

Yes I got the fact that deleted rows space is available for new rows, that's why I said I'll do a cron job to keep cleaning and stabilize my disk usage, but thanks for the more detailed explanation =)

[mattrobenolt]

Ah, I thought you meant to cron up running VACUUM FULL or something. You'd have a really bad time with that. :)

--Matt Robenolt

@mattrobenolt

On Tue, Nov 4, 2014 at 2:51 AM, darhlyk [email protected] wrote:

Yes I got the fact that deleted rows space is available for new rows, that's why I said I'll do a cron job to keep cleaning and stabilize my disk usage, but thanks for the more detailed explanation =)

Reply to this email directly or view it on GitHub:
https://github.com/getsentry/sentry/issues/1304#issuecomment-61622670