Onpremise: Clickhouse partitions retention

[rmisyurev]

Hi!

I have a problem with Clickhouse disk space usage on my onpremise
Every partition size amouth 50Gigs and total size is over 600Gigs
Sentry retention policy is set to "7 days", but I investigated that it's not true for Clickhouse partitions
Is there any way to fix that?

Here is retention settings in sentry.conf.py and .env

And here is partition select from Clickhouse

Tnx!

[lynnagara]

Hey @rmisyurev, thanks for bringing this to our attention.

At a glance, this seems to be caused by https://github.com/getsentry/snuba/blob/master/snuba/datasets/storages/events.py#L102, though I'm not sure why it's the case that only 30 and 90 days are allowed values here - it seems to have been done this way intentionally.

If this is an urgent issue for you, the quickest fix I can suggest would be to either recreate the table with a new partition key (unfortunately it's a bit of a pain since it has to be specified on initial table creation) or to manually drop all of the partitions prior to your window periodically.

But we should also look into if there's a reason things were done this way originally and provide a longer term fix to respect the passed retention value.

[BYK]

@lynnagara is there anything we should be doing here or can the issue be closed?

[lynnagara]

@BYK I don't think there's anything we should do here since we can't easily change the partition clause on an existing table. We are actually starting the process of removing this "sentry_local" table which stores both errors and transactions and replacing it with separate tables for each. We'll make sure the new "errors_local" and "transactions_local" tables respect the provided retention values so we won't have this problem in future. (For errors, this is tackled in https://github.com/getsentry/snuba/pull/1354, transactions_local should already be doing the right thing).

[BYK]

Closing per @lynnagara's comment as there's no sensible action we can take right now.