Zotero-better-bibtex: %b renaming pattern issue

[annargrs]

Testing build 5.0.28 on Zotero 5.0.24 (64-bit Linux).

Adding new items works, citation keys are generated as expected. Thanks for all the hard work you're doing!

Problem: ZotFile no longer accepts the %b renaming pattern, and there isn't an option to set it in the better bibtex preferences. I know that eventually zotero is going to have a citekey field, and both better bibtex and zotfile will hopefully be able to just use that - but in the meanwhile, would it be possible to restore the %b option?

[retorquere]

@jlegewie does Zotero.ZotFile.wildcardTable still return a dict mapping keys like %X to the relevant value of the item? I used to monkey-patch that function to add %b on the fly.

TBH I'd really prefer it if ZotFile called into BBTs citekey list rather than me monkey-patching ZotFile. I'm doing too much of it anyhow.

[jlegewie]

Zotero.ZotFile.wildcardTable is now Zotero.ZotFile.Wildcards.wildcardTable and there are probably other changes related to the transition from Zotero 4 to 5.

More generally, a wildcard for the bibtex key would be a welcome addition (%b makes sense, I just don't know whether it's already taken). But it should work with native Zotero as well and only use BBT if that is installed.

[retorquere]

It wasn't taken before, but you're the expert.

With "native Zotero" you mean "the citation key field Zotero will have in 5.1"? I plan to make use of that myself.

Detecting BBT is easy (you can test for the presence of Zotero.BetterBibTeX), and the key manager is available at Zotero.BetterBibTeX.KeyManagerbut BBT can only reliably be called once the Zotero.BetterBibTeX.ready promise resolves (IOW you can yield on it in a coroutine). It is always safe to call Zotero.BetterBibTeX.KeyManager.get(itemID) but before Zotero.BetterBibTeX.ready clears it will always return { citekey: '', pinned: false, retry: true }. After it clears, it will return the same structure, but:

• retry will be undefined
• citekey will be the item citekey
• pinned will be true if the citekey is pinned (that is it won't change even if the reference does -- you probably don't care about this)

The call to Zotero.BetterBibTeX.KeyManager.get is synchronous; the keys are cached in memory.

[jlegewie]

Thanks for the details. I added an issue but I am not going to get to it anytime soon.

[retorquere]

@ookimi I'd prefer to leave it until then. Zotero 5 is a really different beast internally and by monkey-patching ZotFile I may be introducing race conditions -- I don't know the ZotFile startup sequence, and it doesn't seem desirable that %b would return empty citekeys for a while after start.

[annargrs]

That's too bad... As far as I can see, there isn't even an ETA for 5.1 :(

My whole workflow is centered on pdf files being named with bibtex keys - that's why I haven't even tried to update to 5.0 until now. Of course there may not be many people doing that... but if there are, it's going to be a lot of manually renaming files for the foreseeable future.

Thanks anyway!

[retorquere]

I can give it a shot, but, with the caveat that it would be experimental until @jlegewie takes over... Zotero 5 is such a different beast it isn't even funny. It's taken me 5 months to re-do my own code with which I was familiar.

[retorquere]

I don't think @jlegewie has said it would have to wait until 5.1, just that it'd have to be compatible with 5.1 when that drops. But what exactly was meant is best determined by @jlegewie.

[annargrs]

thanks! I'm willing to be the guinea pig :)

[retorquere]

Err, I've looked, and @jlegewie wasn't kidding when he (I think?) said things have changed... Zotero.ZotFile.Wildcards.wildcardTable is a private function of Zotero.ZotFile.Wildcards so I can't patch it; the only public function I see there is replaceWildcard but I don't know what it's signature means.

I take it item is a Zotero Item, but rule, table, offset I don't know, nor do I know how it modifies it's parameters (if at all), or the semantics of its return value.

[annargrs]

Ok, so only @jlegewie can do something about it. Thanks for trying!

Alternatively, zotfile supports user-defined wildcards, so, if BBT keeps them in a known field and there's nothing else in it, the quickest workaround for now would be to just define a {"b": "zotero-field"} in zotfile.wildcards.user, and then use a %b renaming pattern. I tried to do that with "extra" and "Extra" fields - it should be "extra", right? But that doesn't work, while, say, "title" field works.

[retorquere]

It's possible that this mechanism can be used, but I'd still need to know how to patch into this. I really don't have the time to reverse engineer zotfile - reverse engineering zotero is enough to keep me busy and then some.

[annargrs]

Oh, I thought BBT stores the keys in the "extra" field, and so no hacking would be required - I thought I could just use that whole field. Did that change?

Where does BBT store them, then?

[jlegewie]

Any changes to zotfile from my side are potentially going to take a long time. I basically have no time to work on it and if I do, it's mostly bug fixes.

Yes custom fields might work IF there is nothing else saved in the field. There is a description on the zotfile webpage. No reverse engineering required for using custom fields. That's just a zotfile option to use any field in the zotero database and even do basic text transformation.

[retorquere]

@jlegewie I know that "basically have no time" feeling all too well 😞.

@ookimi the extra field contains a copy of the citekey, and only if it is pinned -- the keys (dynamic and pinned) actually live in BBTs own database. It's in the extra field to make sure that pinned keys sync (non-pinned keys don't, they just get (re)created on all sides of the sync; I'd rather not have it in the extra field, or even in my own DB, but this is the only way offered currently by Zotero.

On the custom keys, if you (@jlegewie) by "any field in the Zotero database" you mean "any field gettable by getField (the citekey is gettable by getField even though it doesn't live in the Zotero DB), then @ookimi: try adding a user-defined wildcard using

{
    "b": "citekey"
}

this should get you the citekey as it is displayed in the references list. You will want to apply zotfile post-processing to what's returned as there will be two things you don't want:

• non-pinned keys will be postfixed with <space>*
• before BBT init is done, the citekey for all references fetched this way will be \uFFFD (the unicode repacement character)

[annargrs]

@retorquere thanks, {"b": "citekey"} works. So no need for anybody to hack anything here! Hopefully something like this will still work in zotero 5.1, so you guys can focus on your many other things.

Thanks you both for both of your great projects.

[retorquere]

This (or something very similar) should just work in 5.1. I monkey-patch getField so it returns the citekey when that field on a reference is requested as that's the mechanism used to display things in the main references list.

[jlegewie]

Great! You can also improve the user-defined wildcard {"b": "citekey"} with transformations based on regular expressions. There are a couple of examples on the zotfile webpage http://zotfile.com/#user-defined-wildcards. For example, you can extract just the citekey using regular expressions.

[retorquere]

That's what I meant with post-processing; you will most definitely want to remove the junk I outlined above.

(never mind the feedback label. I need to look at blip-bloop again).

[DancingQuanta]

What would the regex expression be to remove BBT's junk from citekey?

[retorquere]

Something like /[\uFFFD \*]/g

[DancingQuanta]

Let me check, the regex is only necessary for unpinned citekeys? Because I did not see any additional elements such as <space>* nor \uFFFD in resulting filename after applying {"b": "citekey"}. I am a bit confused.

[retorquere]

Unpinned keys will have <space>* at the end, and all keys will be \uFFFd while the Zotero database is loading at startup (which for me sometimes take 10-20 secs, this is out of my hands, I must wait for Zotero to signal the DB is ready). I don't know exactly when Zotfile becomes active, but I take it you want none of these to show up in the filename.

[retorquere]

And, importantly, all citekeys will start as unpinned, because an unpinned key is created directly when a new item is created.

[DancingQuanta]

Okay. However I am trying to say that, after applying zotfile's rename attachment function, I did not observe any of the junk you explicitly warned about.

[retorquere]

You'll have to ask @jlegewie about what he's seeing and how he deals with that in Zotfile. I just put the citekey + markers in a field ¯\_(ツ)_/¯.

[DancingQuanta]

Haha, weird things can happen! Do you use zotfile and its file renaming capacities?
I will watch this behaviour and hoping that I will not resort to adding regex to my wildcard just yet.
Oh and I believe my current zotfile setup is set to have no space and no unicode in the filenames so maybe that is why the junk is not there.

[retorquere]

That could well explain it. I do not currently use ZotFile, no.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.