K3s: state.db-wal grown too large

[VladoPortos]

Any idea what is in state.db-wal ?
It grown to 8+GB within a month or so... is it something that can be cleaned up, or how can I find out why it is filling up so much ?

[clook]

I had the same issue (16 GB file, leading to pod eviction).
I could vacuum the following way without problem (not sure how safe it is though):

systemctl stop k3s
sqlite3 /var/lib/rancher/k3s/server/db/state.db
sqlite> vacuum;
sqlite> .quit
# The file  /var/lib/rancher/k3s/server/db/state.db-wal is not there anymore, k3s kine sqlite driver will recreate it.
systemctl start k3s

If the vacuum has to be implemented, a specific issue should imo go there: https://github.com/rancher/kine

[Locus99]

I had a similar situation where the file got to an amazing 177GB in size. Running those commands got rid of that file. Will follow up if there are any issues from running the commands.

[brandond]

We should probably evaluate either enabling auto-vacuum, or manually vacuuming at intervals or at startup since auto-vacuum cannot be enabled on existing databases.

https://sqlite.org/lang_vacuum.html

[brandond]

For anyone encountering this in the future - could you comment here? We'd like to know more about the workload and collect some statistics from your database.

In particular, using the Precompiled Binaries for Linux from https://www.sqlite.org/download.html, run:

./sqlite3_analyzer /var/lib/rancher/k3s/server/db/state.db
./sqlite3 /var/lib/rancher/k3s/server/db/state.db << EOF
.header on
.mode column
.dbinfo
SELECT 
  COUNT(*) AS rows, 
  (SELECT prev_revision FROM kine WHERE name = "compact_rev_key" LIMIT 1) AS compact_rev,
  MAX(id) AS current_rev
FROM kine;
EOF