Zotero-better-bibtex: Corrupt DB crashes Zotero on exit

[mizhozan]

• Zotero 5.0.89 (archlinux)
• BBT 5.2.54

Every time I open Zotero, BBT start from scratch, generate citation keys and I have to setup the auto-export profile again. This problem happens some point in time and after that it happens non-stop.

Here is the output from zotero -ZoteroDebug

WARNING: At least one completion condition is taking too long to complete. Conditions: [{"name":"Sqlite.jsm shutdown blocker","state":{"description":"Waiting for connections to close","state":[{"name":"better-bibtex.sqlite#0: waiting for shutdown","state":{"identifier":"better-bibtex.sqlite#0","isCloseRequested":false,"hasDbConn":true,"hasInProgressTransaction":false,"pendingStatements":0,"statementCounter":9},"filename":"resource://gre/modules/Sqlite.jsm","lineNumber":250,"stack":["resource://gre/modules/Sqlite.jsm:ConnectionData:250","resource://gre/modules/Sqlite.jsm:OpenedConnection:1143","resource://gre/modules/Sqlite.jsm:openConnection/</<:953"]}]},"filename":"resource://gre/modules/Sqlite.jsm","lineNumber":148,"stack":["resource://gre/modules/Sqlite.jsm:null:148","resource://gre/modules/XPCOMUtils.jsm:get:194","resource://gre/modules/Sqlite.jsm:ConnectionData:250","resource://gre/modules/Sqlite.jsm:OpenedConnection:1143","resource://gre/modules/Sqlite.jsm:openConnection/</<:953"]}] Barrier: profile-before-change
WARNING: At least one completion condition is taking too long to complete. Conditions: [{"name":"better-bibtex.sqlite#0: waiting for shutdown","state":{"identifier":"better-bibtex.sqlite#0","isCloseRequested":false,"hasDbConn":true,"hasInProgressTransaction":false,"pendingStatements":0,"statementCounter":9},"filename":"resource://gre/modules/Sqlite.jsm","lineNumber":250,"stack":["resource://gre/modules/Sqlite.jsm:ConnectionData:250","resource://gre/modules/Sqlite.jsm:OpenedConnection:1143","resource://gre/modules/Sqlite.jsm:openConnection/</<:953"]}] Barrier: Sqlite.jsm: wait until all connections are closed
FATAL ERROR: AsyncShutdown timeout in profile-before-change Conditions: [{"name":"Sqlite.jsm shutdown blocker","state":{"description":"Waiting for connections to close","state":[{"name":"better-bibtex.sqlite#0: waiting for shutdown","state":{"identifier":"better-bibtex.sqlite#0","isCloseRequested":false,"hasDbConn":true,"hasInProgressTransaction":false,"pendingStatements":0,"statementCounter":9},"filename":"resource://gre/modules/Sqlite.jsm","lineNumber":250,"stack":["resource://gre/modules/Sqlite.jsm:ConnectionData:250","resource://gre/modules/Sqlite.jsm:OpenedConnection:1143","resource://gre/modules/Sqlite.jsm:openConnection/</<:953"]}]},"filename":"resource://gre/modules/Sqlite.jsm","lineNumber":148,"stack":["resource://gre/modules/Sqlite.jsm:null:148","resource://gre/modules/XPCOMUtils.jsm:get:194","resource://gre/modules/Sqlite.jsm:ConnectionData:250","resource://gre/modules/Sqlite.jsm:OpenedConnection:1143","resource://gre/modules/Sqlite.jsm:openConnection/</<:953"]}] At least one completion condition failed to complete within a reasonable amount of time. Causing a crash to ensure that we do not leave the user with an unresponsive process draining resources.
WARNING: No crash reporter available
[169354, Main Thread] ###!!! ABORT: file resource://gre/modules/Sqlite.jsm, line 148
[169354, Main Thread] ###!!! ABORT: file resource://gre/modules/Sqlite.jsm, line 148
[1]    169354 segmentation fault (core dumped)  zotero -ZoteroDebug

I have also included the Zotero Debug output (html) here (which covers mostly the startup part), I saw the below part relevant, if it may help:

            <div id="errors">[JavaScript Error: "{better-bibtex error} +5099 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5177 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5037 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5173 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5195 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5154 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5164 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5091 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5179 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5133 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5115 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5214 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5374 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5228 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5075 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5608 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5203 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5135 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5229 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5072 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5280 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5118 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5162 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5206 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

[JavaScript Error: "{better-bibtex error} +5072 DB.Store.exportDatabaseSQLiteAsync: save of better-bibtex to unopened database "]

version =&gt; 5.0.89, platform =&gt; Linux x86_64, oscpu =&gt; Linux x86_64, locale =&gt; en-US, appName =&gt; Zotero, appVersion =&gt; 5.0.89, extensions =&gt; ZotFile (5.0.16, extension), Zotero LibreOffice Integration (5.0.23.SA.5.0.89, extension), Better BibTex for Zotero (5.2.54, extension)</div>
            <div id="output"><p>(3)(+0000000): {better-bibtex} +null pathSearch: texstudio found at /usr/bin/texstudio </p><p>(3)(+0000000): {better-bibtex} +0 TeXstudio: found at /usr/bin/texstudio </p><p>(3)(+0000001): {better-bibtex} +1 Startup progress: waiting for Zotero locks... </p><p>(3)(+0000003): Using data directory /run/media/mizozhan/Win10/Users/mizozhan/AppData/Roaming/zotero/zotero/4g7pxkf8.default/zotero</p><p>(3)(+0000003):
 IPC: Initializing pipe at 

Export ID: 5072, 5206, 5162, 5118, 5280, 5229, 5135, 5203, 5608, 5475, 5298, 5374, 5214, 5115, 51xx ...

Expected behavior: After restarting zotero, BBT shouldn't start from scratch and it keep the auto export profile created before.

Actual behavior: After restart I lost everything related to BBT and it start from scratch (all citation keys are generated again) and the automatic export profile is lost and I need to recreate it again.

ZoteroDebugOutput.zip

better-bibtex.sqlite

[retorquere]

Can you start zotero like that again but redirect the output to a file, reproduce the problem, and then attach that log to this issue?

Can I have a copy of your database (taken when zotero is not running! Cmd-q if you're on Mac)

[retorquere]

Oh sorry I see you've already attached the log file. Would still appreciate a copy of the DB.

[retorquere]

Never mind -- can I get a copy of the BBT db instead? Same regimen -- make sure Zotero is not running (I see you're on Linux)

[mizhozan]

@retorquere I'm not sure what you mean by BBT db? you mean zotero/better-bibtex.sqlite? The main zotero sqlite is huge (600+MB) nad I prefer to keep notes private. Moreover, I also get suspected that this might be problem of zotero itself, so let me first come up with new fresh zotero install to see if it persists or not.

[retorquere]

I'm not sure what you mean by BBT db? you mean zotero/better-bibtex.sqlite?

Yes. I only need this, and it only contains your citekeys and auto-exports, nothing else. Your log indicates that DB is corrupted and the rest is likely knock-on damage.

[retorquere]

Wait I think I can replicate the problem

[mizhozan]

@retorquere I updated the original post, and put the bibtex.sqlite at the end of the message. Hope it help you.

[retorquere]

I know this is going to sound nuts, but are you sure another program did not access the bbt database? The kind of damage I see here really should not be possible.

[mizhozan]

It seems I have zotero closing problem even with uninstalling BBT. So it may refer to zotero itself. Regarding another program, I only think of having access the same zotero profile from another Linux user (just for test) and from windows OS. That's might be the cause if you think so. The zotero folder reside in a NTFS partition (so no permission problem).

Just as a side question, Is there a way to backup things and re-create zotero profile from beginning? I no longer use Windows so I don't need to keep that profile moreover I also use zotfile to collect PDFs in special folder that's OK, but there are notes and other information as well.

[retorquere]

Wait what? If you uninstalled BBT the corrupted BBT database cannot be the cause, although the corrupted BBT DB with BBT installed will reliably trigger the problem. From another user should also not be a problem.

The thing is I just use the firefox/zotero database (which is sqlite) mechanism as a safe way to store my actual database, which is JSON-formatted data. Now your database is fine from the sqlite perspective -- I can read the data in it and SQLite has no problem, but the JSON stored in the database is corrupted. The weird thing is that I don't know how that can happen. My code that stores my database atomically writes the output of the javascript JSON serializer -- it'd be unthinkable that the JSON serializer would output broken JSON, but also unthinkable that SQLite would change a string of text before writing it to the DB. So I have absolutely no idea how you could get into this state except another progrom opening the DB, and writing malformed data back.

6983 (announced above by blip-bloop) will probably start for you with a fresh database once and then just keep working as it will write the fresh database back to better-bibtex.sqlite. With this your auto-exports will be lost but you can now set them up again and they should keep.

[retorquere]

Oh wait WRT uninstalling BBT: if you uninstall BBT it is actually still active until you've restarted Zotero, that's just how old-style firefox/zotero extensions work. So if you uninstalled BBT and then saw the same problem, that's expected. If you start again and then quit again, it shouldn't have any effects from BBT.

[mizhozan]

1. I did another test, taken you suggestion and completely restart the zotero then install BBT, restart, check, double check...no problem. I moved the previous ~/.zotero file to another folder and just start with fresh profile. There were no problem. So something has gone wrong in previous zotero profile and as you said it's not SQL problem since I done integrity test within zotero and it didn't report any problem.

2. I had a zoetero problem with previous profile which zotero do not close properly with this error:

WARNING: No crash reporter available
[169354, Main Thread] ###!!! ABORT: file resource://gre/modules/Sqlite.jsm, line 148
[169354, Main Thread] ###!!! ABORT: file resource://gre/modules/Sqlite.jsm, line 148

which also didn't allow to restart zotero flawlessly like before. With new profile this also gone.

1. I only can things of two scenarios corrupting the sql/json data:
   
   
   
   • Having two users with BBT accessing the entry at the same time (not an expert)
   
   
   • Bit rot from my harddisk (the zotero db there is 6+ years old), however I haven't seen it anywhere else.
   
   

For the zotero I will refer to the main thread at zotero forum, I just need to keep the notes, snapshots and dates to have a fresh zotero profile to fix this. Fortunately I've done most of recent notes in markdown outside zotero but still there is some old notes there...

Anyway, thanks a lot for BBT, I love it.

[retorquere]

• Having two users with BBT accessing the entry at the same time (not an expert)

The way Zotero opens databases should disallow simultaneous access (even read), but even if it were allowed, sqlite would only allow one party at a time to write to any given record. It should not be possible to get this damage by this route.

• Bit rot from my harddisk, however I haven't seen it anywhere else.

Highly unlikely.

For the zotero I will refer to the main thread at zotero forum, I just need to keep the notes, snapshots and dates to have a fresh zotero profile to fix this. Fortunately I've done most of recent notes in markdown outside zotero but still there is some old notes there...

With 6984 you should be able to keep everything as-is; BBT should start up, fail to read the database, create a new (empty) one, and write that over the existing corrupt data. You should never have to abandon your Zotero database; by design, BBT does not touch it, so worst case will always be just that the BBT database is lost.

[mizhozan]

The problem is BBT calculate 5000+ papers and that's take time, so I prefer BBT just work as before, re-creation of auto-export isn't a big deal.

Just one thing: When I remove BBT from addons, restart and then install BBT with previous profile BBT still works problematic as stated initially. Do you have any idea on this? To best my understanding there should something wrong with zotero or within its profile that when I re-install BBT it still cause it corrupting json database entry.

[retorquere]

The problem is BBT calculate 5000+ papers and that's take time, so I prefer BBT just work as before, re-creation of auto-export isn't a big deal.

No that's the thing -- it will work as before, but on the current formal release, BBT would never recover from a corrupt DB by creating a new one. 6984 does recover by creating a new one, and from that point on, it will be business as usual. You will lose the auto-exports in this recovery, but after that, it should just work again.

Just one thing: When I remove BBT from addons, restart and then install BBT with previous profile BBT still works problematic as stated initially. Do you have any idea on this? To best my understanding there should something wrong with zotero or within its profile that when I re-install BBT it still cause it corrupting json database entry.

Nope that's expected. When BBT is uninstalled, it does not remove its database, so when you re-install, the corrupt data is still there.

[retorquere]

Still not fixed.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5.2.54.6990 ("fixes #1619")

Install in Zotero by downloading test build 5.2.54.6990, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

[retorquere]

6990 should do it.

[mizhozan]

Yes, I can confirm that 6990 completely fixed the problem. It also keep auto-export profile on restart (no need to re-create) and the zotero restart problem (keeping sql connection) has also gone. Completely fixed. Does it take effect in future official releases?

[retorquere]

Yep, I'll roll out a release based on this; you can keep using this and you'll be upgraded automatically.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.