Duckdb: RENAME TABLE on a persistent database causes the table to lose all of its data

[Mytherin]

RENAME TABLE internally does a DROP TABLE followed by a CREATE TABLE with the same table contents - however, when logged to the WAL only the DROP and CREATE are logged. At database startup that means the contents of the table are cleared.

[jwills]

Hey y'all, I'm currently hitting this issue with my dbt integration; is there a fix for it I can put together?

[Mytherin]

I will have a look at fixing it. This issue has been bugging me for some time :)

[jwills]

I spent a pleasant day reading the code and think I have figured out how to implement this, but it feels like I'm either going to have to a) do major surgery on CatalogSet or b) copy-and-paste a lot of the logic from the CatalogSet::CreateEntry and CatalogSet::DropEntry methods into a new CategorySet::RenameEntry function that does a bit of both. The TL;DR would be that I would create a new CatalogType called RENAME_ENTRY that would become the parent type of the deleted entry and the child type of the created entry and put some special handling for it into the CommitState::WriteCatalogEntry function to ensure that a) the create entry was correctly handled as an alter table and b) the delete entry was ignored entirely.

TL;DR I feel pretty good that I now understand how all of this works, but I still don't think I have a "good" way of implementing the fix.

[Mytherin]

I have already started implementing a fix in the fixrenametable branch. Indeed there is some not so trivial rewrites required. I ended up rewriting the CatalogSet somewhat so that entries are stored separately from their names, which allows renaming in a similar manner to the other alter statements. The persistent rename table now works correctly on that branch, but I borked the dependency manager in the process, so will have to fix that. Hopefully I can get that working tomorrow.

[Mytherin]

Should be fixed now