Zotero-better-bibtex: Rewrite for Zotero 5

[ericnchen]

I have installed the latest 5.0 beta for Zotero on a new computer and installed the latest version of Better Bib(La)Tex as well. The versions specifically are 5.0-beta.r39+bdec4b1 and 1.6.72, respectively.

Anyway, after installing Better Bib(La)Tex and restarting Firefox (45.3.0 ESR on CentOS 7) I received a JSON error though I didn't think to save the error. I went to Advance Settings and enabled Debug Mode for Better Bib(La)Tex and restarted Firefox. Now on restart, I get the following message:

 Better BibTeX has been disabled because it found Zotero undefined, but requires 4.0.28 or later.

I also can not edit any of the BBT options anymore because it also tells me that BBT has been disabled. The only thing I can do I presume is to just uninstall the extension from Firefox and reinstalling it and hoping it works.

Before I turned on debug mode BBT showed up in the Zotero options, at least. I was not able to export anything with BBT though. Selecting "export library" didn't provide BBT as one of my export options.

Is there a working version of BBT that will work with the 5.0 beta?

[retorquere]

There is currently not a version that works with 5.0, and I'm not certain whether there will be a compatible BBT at the release date. Zotero 5.0 is not just an incremental update to Zotero, it is really a wholly new program, and almost all non-trivial extensions will have to make extensive changes for 5.0.

Rewriting BBT to be compatible is going to be a substantial effort, partly because it moves far and wide beyond what Zotero officially allows extensions to do, but mostly because the database paradigm has changed, and BBT has some deep-rooted assumptions in that domain.

I currently lack the time to go heads-down on this and spend the time it would take to do this. I don't know exactly when I will be able to make the time, and there is also the matter that 5.0 is a transitional Zotero in any case -- there are already plans to move away from Firefox entirely to another platform (Electron), which would likely mean another substantial rewrite. Between my lack of time and the dynamic of the developments of Zotero, I'm going to wait until their plans solidify before I move (unless I suddenly get a month of nothing-to-do spare time, which is unlikely).

This is not a complaint about how Zotero moves BTW. I am unhappy with these moves because they impinge on my already cramped planning, but I understand why they are making these moves.

[ericnchen]

Thanks for the response!

[retorquere]

After a few attempts it looks like 5.0 is going to require essentially a full rewrite. With some 40k lines of code, this is going to take a while -- end of october at the very earliest, and very likely later than that.

[dbobak]

That is a very bad news. My entire workflow depends on stable BibTeX keys. Do you know any other possibility to have them in Zotero 5.0?

[retorquere]

None that I know of that don't require coding. I'm working on 5.0 compatibility, but it will require major changes to BBT. I don't have an ETA for this.

[retorquere]

(I'm juggling a full-time job, studies, a family and the work on BBT. I was doing OK with incremental updates to BBT, but the 5.0 port is not an incremental change. Basically everything is broken right now, and I'll need to fix/change everything before I see even parts of BBT work again, which makes it incredibly hard to judge how long this is going to take)

[dbobak]

Great, that you are working on it anyway. I will stick to Zotero 4.x as long as possible.

[steko]

@retorquere thank you for your continued efforts. As a devoted user, may I ask if funding would help you work on fixing BBT for Zotero 5.0?

[retorquere]

Truly appreciate the offer, but it wouldn't help. What I need is time, and my calendar has just flooded the last month. I hope to make progress during the holidays.

[adam3smith]

Hi @retorquere -- just to be able to give better feedback&support to potential users, do you have an update on this?

[retorquere]

I can't offer anything of substance at this point. I've started a full rewrite, but it's incredibly slow going.

[Frank-Zappa]

Was there any discussion to merge zotero-better-bibtex functions to Zotero 5.0 directly? I think, we should urge the Zotero dev team to consider it.

[adam3smith]

Some features of BBT make a lot of sense as an add-on. E.g. Zotero wouldn't want to expose anywhere close to as many preferences for bibtex as BBT does.
Some things will eventually happen but need more time -- stable editable citekeys most importantly.

Some things would be great but are probably just not high enough on the core devs agenda to realistically happen any time soon. Auto export is probably among those.

[retorquere]

Yeah, as long as space for cite keys isn't even present in the references, I wouldn't hold my breath for bbt integration.

BBT does have a baroque number of preferences, but all of them have sensible defaults. There's no need to have them exposed in Zotero - that could be the job of an extension. Just saying.

And there are things that really should have been separate extensions - stuff like auto-export really doesn't even belong inside bbt, but it's mostly there because it needed a caching system (zotero reference serialization is a really big bottleneck), and bbt had one, and I only need auto-export for bbt. Plus the serialization cache as it is now does some minor damage to the serialized objects by simplifying them to just data, no methods, so it's not safe for any and all translators.

I'd say that the serialization cache would be a real boon for zotero users that do frequent exports, and that a damage-free cache is possible inside zotero, but it's quite possible that only bbt users are doing frequent exports - other users just use zotero directly I'd venture to guess.

[ghost]

Are there any updates on this since Zotero 5 is out?

[retorquere]

Only that I've not given up on the idea. BBT requires a full rewrite for 5.0, BBT is a pretty complex beast, and the simple ports is tried have all failed so far.

[retorquere]

First thing I'm going to do is get my test framework back up. Without my tests I can't do anything. Next order of business will be adding the translators, but stuff like auto-export won't be there in the beginning.

[MarioJose]

@dbobak, for while, you can select all your references in Zotero, left click with mouse and export it to BibLaTeX. Zotero have a option to export to many formats. But, when you add a new reference, you have to export all your references again.

[retorquere]

I may spin off the auto export to a separate plugin if feasible. But first
things first :

1. Cite key generation
2. Basic export (but better, natch)
3. Everything else.

On Jul 14, 2017 7:04 PM, "Mario José Marques-Azevedo" <
[email protected]> wrote:

@dbobak https://github.com/dbobak, for while, you can select all your
references in Zotero, left click with mouse and export it to BibLaTeX.
Zotero have a option to export to many formats. But, when you add a new
reference, you have to export all your references again.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/retorquere/zotero-better-bibtex/issues/555#issuecomment-315412481,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIEDHRlGZ9v9IW8mzw2ceelfiTIBCxLks5sN5-ogaJpZM4JtLTL
.

[dbobak]

@MarioJose, I know that. But it does not solve the problem of stable keys. For me it's the most important feature; I could manually export up-to-date after update, but I have no guarantee that the keys will not change.

[retorquere]

Stable keys is the first order of business. I have a skeleton implemented that loads that I'm starting to populate slowly.

[dbobak]

@retorquere, marvelous news. Thank you very much.

[retorquere]

Stable keys would be tons easier to achieve btw if the zotero crew would include a means for extensions to store their own data. It's been discussed for years at this point.

[MarioJose]

Stable keys is a important issue. I didn't think that @dbobak. I thought that had a standard for that. Thank you @retorquere!

[retorquere]

Stable keys are challenging in Zotero because the part of the code that generates bibtex is isolated in a way that it can't look at the database to spot duplicates. BBT works around that in a way that zotero really shouldn't and I rather wouldn't.

[retorquere]

For the curious, activity on this happens in https://github.com/retorquere/zotero-better-bibtex/tree/z5. Nothing usable is there right now, but if you're the adventurous kind, please let me know if your willing to test once I get something running.

If you do:

1. Thank you so much, and
2. BACKUP, BACKUP, BACKUP

I have an extensive test suite that I will have running before I would even think of submitting it for test, but BBT 4 Z5 will most likely incorporate some visual changes in the library that I'd like to have checked off. This is your chance to influence those changes. Getting the test suite to pass is likely going to take at least two weeks though (wouldn't you know it, the office and the family and the study also demand some of my time. Crazy stuff).

[jhrmnn]

Signing up as an alpha tester ([email protected]). Have no experience with FF extensions so cannot offer help coding.

[retorquere]

No coding experience required, I just need feedback. I don't expect to release alpha-level stuff in any case -- whatever I release will pass at least a significant chunk of my existing tests. It's mostly that I have some plans to change where the cite keys live, technically, but these will be visible in the UI. My first testing need is that I need to know whether that visual change is acceptable.

[dbobak]

I would also help as an alpha tester ( deni+github(at)lithics.eu ). I am not a programmer in any way, but I could track and report bugs.

[hquilo]

I might be able (and would like) to help coding and refactoring ... check my info at camilorocha (dot) info and please send me an email if you think I can be of any help.

[retorquere]

@hquilo I've added rudimentary dev setup instructions in CONTRIBUTING.md, please check if they're clear enough to get you going.

[dbobak]

@retorquere It's is quite complicated to install all the required software on Windows. And I'm on Windows only. Would it be possible to publish xpi?

[retorquere]

@dbobak windows is a miserable experience for BBT dev. It should be better now bash on Ubuntu on windows is available but I haven't tried myself.

I'm nowhere near a sensible xpi yet but once I do they'll be published as usual under the "builds" release on github. But for that to happen I need to have my test & deploy scripts ready, and have at least one passing real test. None of those milestones have been hit yet, although the test & deploy scripts are making good progress.

[hquilo]

@retorquere Thanks for the instructions. I managed to advance almost to the end. I got stuck at this instruction

Run ./features/support/mkprofile. Zotero will start up, import a bunch of references, and shut down.

Where is this folder structure located?

[retorquere]

@hquilo I forgot to add instruction to checkout the z5 branch (which is where that script lives). See new updated CONTRIBUTING.md. You will have to run the bundle update and npm i steps again after changing to the z5 branch.

[retorquere]

But if you wait a few hours I'll rename the z5 branch to master; things may get confusing if you overlap. AAMOF I'd recommend getting a new checkout after I rename the branch.

[retorquere]

OK the branches have been renamed; I recommend getting fully new clone of the repo.

[retorquere]

OK, feedback time. Warning: large wall of text ahead, which contains a fair bit of technical, possibly Zotero or BBT specific mumbo-jumbo. The _TL;DR_ is at the bottom.

As I'm rebuilding BBT for Z5, I'm trying to minimise, preferably eliminate, the monkey-patches and sandbox-piercing that BBT employed to do its work, a.o because many of the monkey patches assumed Zotero would work synchronously which makes wrapping or replacing code relatively easy. Z5 is all gung-ho for async code, which is both understandable and a huge pain in the ass (and the reason that I'm talking about rewriting rather than porting BBT). Right now I'm at a crossroads with regards the generation of citation keys. My desiderata for this are:

1. When generating citation keys, I need to be able to search the whole library the reference lives in, not just the subset being exported, for potential duplicates. Given the translator sandboxing, this means citation keys must be generated outside the sandbox.
2. This citation key must be attached to the reference so it does not change unless explicitly triggered by the user
3. There are people with large libraries, so the search for duplicates needs to be efficient
4. The citation keys must be available within the translator, so the citation key generated outside the sandbox must find a way in.

What I did before was:

1. Pierce the sandbox so key generation can be initiated from within the translator.
2. This key generation searches a secondary cache that holds the citation keys, and places the new key there, because searching the "extra" field is way too slow.
3. If the key is meant to be pinned (that means doesn't change when e.g. the reference title changes), I also store it in the "extra" field so it will sync
4. Return the generated key to the translator and it's off to the races.

This however has a number of drawbacks that I'd love to get rid of while I'm rebuilding BBT.

1. As from the Phil Karlton quote "There are only two hard things in Computer Science: cache invalidation and naming things.": I'd dearly love to get away from my secondary cache. I'm seeing cases in the wild where it gets out of sync to the actual library which requires not just a full table scan but a parse of each result to find the pinned keys in the "extra" field, which gets me to
2. There is no way to efficiently search for duplicates in the "extra" field, as it requires inspecting each of the extra fields to lift the key out.
3. I don't want to pierce the translator sandbox anymore. The sandbox has some very decent defences in place that are tricky to work around. It's extra work for no benefit, and potentially risky.

The "clean" option from the Zotero pov of course is to store all citation keys in the "extra" field, pinned or not, and retain the secondary cache for searches. I don't like this because cache invalidation remains an issue. The citekeys would get to the translators without sandbox-busting, which is better than it was. I know this is always picked as the preferred option because it doesn't interfere with the regular operations of Zotero, but it sure does interfere with mine because of the efficiency and/or cache invalidation problems.

I've been experimenting with several alternate approaches, including:

Hacky alternatives that violate the spirit of the Zotero API but won't break Zotero, and which would allow for efficient finding of duplicates:

1. Store the citation keys in a tag. This will make the tag selector instantly useless because it will be flooded with citation key "tags", and I seem to recall there were performance problems when too many tags were present in a Zotero library. Perhaps the performance problem has been solved; I've poked around a bit in Zotero to see if I can just have the tags not show up in the tag picker, but haven't gotten far yet. Such hiding would involve monkey-patching.
2. Store the citation keys in a specially formatted linked URL (e.g. url="zotero://better-bibtex/citekey", title="@citekey"). Downside is that every reference will show that it has an attachment when potentially that can be a sole "attachment" just holding the citekey. I've likewise looked into hiding those in the UI but haven't gotten far on that either. The hiding again would involve monkey-patching. "zotero://" urls cannot be imported back I've just found, so I may have to settle on something like "https://better-bibtex/citekey"

Hacky alternatives that will (likely) break Zotero, so they're not options for BBT:

1. Store the citation keys in spurious relation records. When I previously tried this some years ago this screwed up my library beyond any salvation (btw, the account "emilianoheyns" that linked to it can be killed AFAIC).
2. Store the citation keys in the existing tables for custom keys. Will likely not sync, and may or may not break Zotero.

Clean option

I very much would love to see a non-hacky alternative that would just allow me to store the citation key associated to the reference, efficiently searchable, syncable, and available to the translators. Support for custom fields in Zotero has been talked about many times before; I'm hopeful now that the major work on Z5 is done, there would be time to implement custom fields, but this is a request that has in the past only gotten the conceptual "good idea, some time in the future" half-thumbs-up, so it is only a weak hope.

Finally then, my question:

If support for custom fields is not likely to emerge in the foreseeable future, I'm leaning towards the linked URL alternative at the moment. Thoughts?

TL;DR

I want to store the citation keys in a Zotero-native way. Zotero doesn't offer a formal Zotero-native way, so I'm considering stuffing them in attachments. These attachments will show up in your library. Is this OK?

[anne-urai]

I would have no problem with attachments containing citation keys, since I already have many items that contain both a pdf attachment and e.g. a Pubmed entry link, and I don't search for PDFs from within Zotero (Zotfile renames them and Mac OS Spotlight finds the files on disk).
Thanks for your work on this!

[duncdrum]

• attachments seem very reasonable, so far Zotero's handling of multiple attachments is very robust.
• extra is not very stable especially with juris-m (which might of course no longer exist if custom fields arrive).
• Tags would interfere with visualising zotero contents, and with search so that sounds like making a bad situation worse.

[retorquere]

@duncdrum I wouldn't hold my breath for Juris-M being superseded by Zotero; it does a lot more than just adding a few fields (like multi-lang support, which is not a trivial change against Zotero).

The attachments would have edge cases of their own; if you merge two references they resulting reference would have multiple citation keys. I don't yet know by which algorithm I'm going to resolve that; but then in the current situation the resulting reference would get a random pick from the merged references, so it's not really all that different. I've posted the large wall of text on zotero-dev too in hopes that the Zotero devs chime in with better ideas. But good points on the tags. So the tags option is out.

The tags would have had as side benefit though that they're searchable; citekey search in Zotero came by way of a rather unpleasant monkey-patch.

[duncdrum]

@retorquere I see your point about the searchability of tags. In my own use cases the pros of searchable citekeys, wouldn't outweigh the cons. But I guess I'm in the minority.

As for Juris-m, I still call it MLZ and refuse to give up hope 😎

[retorquere]

@duncdrum I think your tags argument is persuasive, so for the time being I'm sticking with either the extra field or with the attachment. Searchability of tags can be done in other ways, or possibly deferred until Zotero finally gets a dedicated field for citation keys.

There must be more to Juris-M (or MLZ), because I feel confident that if MLZ could be merged easily with Zotero, Frank wouldn't mind at all getting that chunck of his life back.

[hquilo]

@retorquere I figured that much last night while following the instructions. I did checkout the z5 branch and worked from there. Once I get back home (tonight) I will try to get rid of the problem.

As for the way of storing a citation key, I lean towards having them as an attachment. If Zotero finally includes a citation key field in future releases, then upgrading to this more stable solution from the temporary attachment one seems OK.

[retorquere]

@hquilo I'm actively testing that out now but it seems there's something in the Zotero notifier that's broken that I need (the alternative would be to do a moderately expensive DB call), hoping they'll get back to me soon.

[retorquere]

@hquilo I would suggest doing a clean clone because I don't think the rename will have made it with full history in the repo. I actually have very little on my hands this afternoon so at least for a few more hours I can work towards getting the test harness up and a rudimentary key generator.

[retorquere]

One thing where everyone could help is go here on Trello and add comments for features or functionality not already listed on the board. BBT is a complex beast, and as I'm not porting but rebuilding from scratch I must make sure I re-create everything. The test cases I have cover a lot, but potentially not everything, and the don't do any UI.

[troeger]

I wrote a small Python script for converting a Zotero5 BibLatex export to stable citation keys:

https://gist.github.com/troeger/87848e8485c8f009537a6e085ce16a15

This is only an intermediate emergency solution, in case you need publication work to be done. Some entry types simply don't get their notes exported with the standard mechanisms.

[andersjohansson]

If the question is just to have a working solution until citekeys are added to Zotero (hopefully in a few months time, after 4 syncing is turned off) wouldn’t it be best to choose the simplest solution that would be simplest to convert after that (and for the time being accept that it might be a little slow)?
I don’t know if using the Extra field or the linked urls would be simplest though.

Would an url be added for every item (used as cache?) or only for those with pinned citekeys? For my use-case I only have a few pinned citekeys and go with the generated (although with a customized format) most of the time. Having (visible) urls added to all vs the few items with pinned keys would be quite different experiences then, with the later not distracting at all and the former a little distracting.

[retorquere]

If indeed the citekeys are coming in a few months, and there are now indications that they might, then yes, storing them in the extra field is easier at the cost of performance. Given that the odds have just increased of this happening, I'm re-considering keeping the cite keys in the extra field.

Having the keys in an attachment would not have been a cache, it would simply be the place where the citekeys would be stored. All of them, not only the pinned ones. I want to move away from a secondary database to keep the non-pinned keys, and I do need to keep them somewhere to have stable citekeys. With the citekeys-in-attachments, all references would automatically get at least one attachment, being the citekey, and within that attachment, and indication of whether it's pinned or not. I don't really like that either, but the secondary database takes upkeep to make sure it doesn't go out of sync with the primary database, expiring entries when they go out of the primary database, etc.

[retorquere]

Ugh just thinking about this gets my blood pressure up. For a shadow database I need to react to events that Zotero emits for reference changes/additions/deletions, and any error that occurs there (I'm not perfect) throws the shadow DB out of sync. That in turn has repercussions for citekey generation. I really would prefer not having to deal with a shadow database anymore for the citekey generation.

In order to chuck this shadow database I need to have all citekeys in the primary database. I'm testing an approach where I do keep them in the extra field, but all references would have one, with a special mark for non-pinned keys. This will for sure have performance implications, but perhaps they are tolerable.

[rcoreilly]

If there is anything we can do to advocate for getting a first-class citekey field added to zotero, let me know! We have developed a separate database that maps citekeys to zotero item codes so you can add citations into zotero items from a wiki: https://grey.colorado.edu/CompCogNeuro/index.php/WikiCite -- if the citekey field existed and was directly searchable, then all the extra work of maintaining and updating this database would be alleviated. And needless to say, none of this works without BBT providing the cite keys in the first place! BBT is really the key (so to speak) that makes zotero functional for us, so without it, we can't really make use of zotero! I don't know what your relationship is with the core zotero dev team but from my perspective, BBT should just be part of the main zotero functionality, instead of having to hack around on the margins of the sandbox or whatever.

[retorquere]

@rcoreilly if I'm interpreting correctly what I've just heard, work on Zotero-native citekeys can start somewhere late this year, contingent on Zotero 4 sync being shut down. When the citekey really would appear I don't know. I don't currently know how exactly the citekey field will be implemented (but I'm guessing just an extra field on the reference) or by what way it will be populated by Zotero (if manual, then I deem it no problem for BBT to take control, if there's another mechanism in Zotero itself that wants to populate this field this could get tricky). I'll just see how to deal with it when it drops. The sooner the better, but...

I can't wait for that; it's still a ways off, for one, but to wait for it would also mean that there would be a period where BBT users would have nothing -- right now, you can still use BBT on Z4, but when Z4 sync is shut off, that would leave BBT users stranded. Not good. So I'm working on an intermediary version that will probably still store the citekeys in the extra field -- the Zotero devs seem confident that this need not be a performance bottleneck.

As for the relationship, the Zotero devs have always been polite to me and have been prompt to provide technical information when I ask for it. Other than that, I can't really speak on how they see our relationship, and I'm hesitant to speculate; sometimes they seem happy with BBT, sometimes they're a little curt, but that could be for a host of sensible reasons -- not only are the devs very likely extremely busy, but I also sometimes shoot off questions that more research on my part could have resolved, or forget that people use Zotero for non-LaTeX work too and that BBTs needs and Zotero's needs need not always be perfectly aligned.

Whether or not BBT should be built-in... I wouldn't mind personally but I think it's safe to say the Zotero devs consider BBTs feature set to be on the baroque side and properly hosted in a plugin that lives on a different release cycle. A lot could already be gained by expanding the internal API (and adding citekeys is a great start; a good dateparser would be fantastic). I'm trying to stay away from piercing the sandbox, but I've already just now hit an edge case with the date parser where I may have to do a lot of extra work to have it function in the translators, or I have to pierce the sandbox to have it run in the plugin and available in the sandbox. Decisions, decisions.

[retorquere]

I have citekey generation working (yay for progress) but scanning the extra field duplicates is intolerably slow. Guess I'll have to fall back to the shadow DB after all :cry:

[gncs]

@retorquere, thank you for keeping us posted! I am so much looking forward to having BBT in Z5!
there is something I don't understand about the keys issue: requesting a value in a database field plus grabbing whatever comes after extra seems to me like a totally reasonable thing to do. why does this take so long when everything else in Z5 feels so snappy (once everything is loaded). thanks!

[retorquere]

Several issues are at play.

BBT must do a table scan to find potential duplicates. Grabbing one reference is a non-issue; grabbing 20k references (and I know people with 20k references using BBT) and waiting 40-50 seconds because you added a period to a title is another. This is exacerbated by the fact that the results of the table scan must themselves be parsed because the citekey is lumped in with other unstructured data. This is then further exacerbated by the fact that BBT will usually spring alive when a reference is changed, meaning that Zotero is very likely itself busy with the database, so BBT and Zotero are contending for access.

Zotero has to deal with none of this. Zotero is usually itself the trigger that causes DB access, so it has a lead there. Zotero does no instant deduplication on its references (that I know of at least), so dedup only happens on explicit user interaction, when you're much more likely to tolerate lag. I can get rid of some of the contention by having BBT "sleep" an arbitrary amount of time after a change to Zotero is more likely to be done with the DB (which drops the lag about 50-fold), but I have no deterministic way to pick the delay, and I may have to deal with a higher chance of race conditions because I may have stale data in my hands by that time. Or not. I just don't know.

Z5 is a wholly different beast from Z4. I had a lot more control under Z4 because it was mostly synchronous code so I went wide and far outside the formal internal API using a process called monkey-patching and I was pretty confident I could picture what's going on, and be certain about my shadow DB and the main DB being in sync. In Z5, everything is asynchronous and a lot of the assumptions built into BBT4 don't hold. For those reasons I'm trying to do away with the shadow DB, but that has it's own drawbacks (my shadow DB was using LokiJS and was stupendously fast).

Overall, Z5 is an architectural improvement, and one that was necessary given how Javascript is put together as a language. But if I'd had started BBT on Z5 instead of Z4, a lot of things would have been done differently (or not at all); I might have waited out the arrival of citekey fields, which would at least have made some things a tad easier.

[rcoreilly]

Is it not possible to just keep all existing BBT users on zotero4 until the citekey fields are available (in 5.1?) and then switch over then? Seems like it would be a lot easier on you. Hopefully there will be some kind of conversion process that would still work at that point? I guess the issue is the central database is the key bottleneck there -- everyone needs to upgrade once the database changes, and the database can't be upgraded until everyone upgrades their client? They can't just fork the database and you could live in the old system until you're ready to switch over?

[retorquere]

No can do, because the arrival of the citation key field (by technical necessity if I understand correctly) will come after the shutdown of sync capability for Z4 clients. This would mean that there would be a period where you wouldn't be able to sync you DB while I work on the citekey field support.

To be fully honest, I had about a year to get BBT ready for Z5. I tried I think about 5 half-rewrites but the scale of the work just flat-out scared me. My life could easily be described as hectic (but then that would describe anyone who has use for a reference manager I guess) and a lot of the BBT4 stuff was just coasting on a fairly solid basis I had built over the years. I had secretly hoped that I could just tide Z5 over and go straight from Z4 to the Electron port that's soon to follow. But there's no timeline on the Electron port, Z5 is here, and Z4 sync is going the way of the dodo somewhere later this year, way before the Electron port will even be sketched out.

[lyndondrake]

In a couple of weeks I'll be in a position to test BBT with Z5, if that might help. I won't be publishing anything new for a while, and have copies of my biblatex db for the in-progress stuff. My DB has about 1200 items so not as vast as some

[floriandierickx]

I would be also happy to help where useful (testing, ...). I am not a schooled programmer, but have experience in R and Python (for PhD modeling purposes). My setup and workflow are: Mac OSX 10.12.6, Zotero 5.0 (although I should have checked compatatibility.. ), atom, markdown, pandoc, betterbibtex feature and zotero-picker (https://github.com/oztalha/zotero-picker, @oztalha). In the meantime it seems I will shift to the atom package autocomplete-bibtex to continue writing and publishing (https://github.com/apcshields/autocomplete-bibtex, @apcshields). Thanks anyway for the ongoing efforts!

[jimvine]

Just a thought, in case it's helpful. You mentioned:

Zotero does no instant deduplication on its references (that I know of at least), so dedup only happens on explicit user interaction, when you're much more likely to tolerate lag.

Is there a way, perhaps just as a tide-over until Z5.1, to switch any parts of BBT to work on an explicit-trigger basis? I'm speaking as a long-term Zotero user who has recently come across BBT but not yet got into it knowing the Z5 challenges. It looks really powerful and something I'd be keen to use in the future, but I know I'm not speaking from a position of knowing how it works either from a user or a programmer perspective, so please feel free to treat this suggestion with the derision it may deserve!

[retorquere]

The derision it deserves is none, so: done.

The explicit trigger would mean references wouldn't get citekeys by default, and you'd have to think about generating them before export. That would be too much of a burden, I think.

It would be possible to generate them only on export though... which would not interfere with the normal operation of Z5, and would defer the work to be handled in bulk just before the export is kicked off. The downside would be that you'd only see the citekey after you export. Anyhow, I'll keep this in mind should the performance become an issue.

[tpoisot]

@retorquere as far a I'm concerned, citekeys being generated only on export are fine -- I usually set folders to auto-export the references anyway, so I don't even have to think about the citekeys. Do you think most users rely on citekey autocompletion when writing?

[MarioJose]

@tpoisot, I think that the problem is that many people use exported bib file as a link to Zotero data, no copy of that will go to text/paper created. I, for instance, always use the same exported file (whole Zotero data) in my text/papers. If citekey is changed and I want recreate pdf of this texts, my bibliography will be wrong. But, if I use one bib file (whole data or selected references) to each text/paper, I will note have problem, as you said.
@retorquere, if Zotero reference ID/key is in citekey, it solve problem to check duplicated key?

[retorquere]

@tpoisot, @MarioJose the exports would still always have citation keys, but should I delay the generation of them until the moment of export, you wouldn't be able to know whet they were going to be until you actually exported; after that, they'd be stable. But I will only go there when the performance disappoints; I can't yet oversee what potential edge cases there would be for keys only generated at export (eg the interaction with sync of non-citekey'ed references).

[retorquere]

@tpoisot if you set folders to auto-export, you'd notice no difference, as for each change you make you'd still have the citekey generated right then.

[vtcarlino]

Hi everyone, I'm a phd student hoping to use zotpick + betterbibtex on Zotero 5. I'm ready for some beta testing stuff. How can I download betterbibtex for Zotero 5? I can't find the z5 branch you're talking about. Thanks!

[retorquere]

@vtcarlino you can't right now, but all my tests on BibLaTeX export currently pass; I'm working on getting the preferences screen working again and when that is done (this week or so) I'll have a version ready for test for those who are willing to test manual biblatex export.

You must currently manually pin and unpin your citation keys, which now always lives in the extra field; bibtex:... is a pinned key, bibtex*:... is a dynamic key.

Since this is a ground-up rewrite I have a pretty good picture of what everything does and I can't think of anything that would mess with your production library, but I recommend either a safe backup (taken when Zotero is not running) or starting with a clean profile by starting Zotero from the command line passing it the -P option, creating a new profile specifically for testing, and unchecking "Use the selected...". Unchecking this will always bring up the profile selector so you can after that start Zotero using the usual way.

[retorquere]

For those about to test

1. we salute you
2. if you find anything that's absent in the speclist on trello, add it as a comment here
3. For things that are listed on trello but not implemented, feel free to comment or suggest priorities (order in the "planned" column is roughly the priority)
4. for things that are claimed to be implemented (meaning it shows up in the "done" column on trello) but not working as intended, add a new Github issue

I will post a download URL here as soon as I have a version ready. During the tests, upgrades will have to be manual -- I'll post new ready-for-test versions here as they become available.

[dmbortz]

Please add me to the list of those ready to beta (or alpha) test BBT on Zotero 5. BBT is central to the way my group writes articles.

[retorquere]

It's getting there. The asyncification is a major pain in the ... but the preferences screen works. I just have some gnarly race conditions to deal with -- it currently takes some 9 seconds for Zotero to "settle" into a state where it's safe for BBT to do it's work, and I need to figure out a way to have BBT respond sensibly before that.

[retorquere]

Alright, time to gear up @azag0, @dbobak, @hquilo, @lyndondrake, @floriandierickx, @vtcarlino, @dmbortz!

The BETA release can be found here; newer BETA versions will also appear here. To reiterate,

1. The preference pane is available, but not all preferences actually work, because not all corresponding functionality has been implemented (see trello)
2. if you find anything that's absent in the speclist on trello, add it as a comment here
3. For things that are listed on trello but not implemented, feel free to comment or suggest priorities (order in the "planned" column is roughly the priority)
4. for things that are claimed to be implemented (meaning it shows up in the "done" column on trello) but not working as intended, add a new Github issue, prefixing the subject with Z5:
5. debug report submission doesn't yet work but it's on its way.

And once again, for those about to test, we salute you.

[dmbortz]

Hello, I just tried to export one item in my archive (using 3902) and got a error message of "An error occurred while trying to export the selected file." And, while the file was created, nothing is in it.

Can you point me toward a reminder of how to generate a useful bug report?

thanks!

[dbobak]

@retorquere, many thanks! Just started testing. Exporting bibliography works fine, but I have found one missing thing: there is no quick copy translator for drag & drop citation.

And just one question. I'm working with my own translators, which allows to cite in personal MediaWiki installation (with Semantic Cite extension). What would be the best way to get BetterBibTeX citation key in my translator? And also the item id and item URI (for creating links to local zotero installation as well as for web)?

[duncdrum]

@retorquere once juris-m has taken the leap I'll be happy to help test things. Seems like a good opportunity to homogenise different export formats with respect to juris-m fields.

[retorquere]

@dmbortz I expect to have the regular debug report up and running today, that will make error reporting much easier

@dbobak: thanks for the spot: added https://trello.com/c/rOHDGxOt/38-quick-copy . To get the BBT citation key right now you'd parse the extra field using /(?:^|\s)bibtex(\*?):[^\S\n]*([^\s]*)(?:\s|$)/; the captured fields will tell you whether the key is pinned or not (the first capture group will be false-ish if it's pinned) and the citation key. BBT monkey-patches the exporter, so you will get .itemID on the items in your translator; without BBT, you'd have to have your translator request legacy format ("minVersion": "3.0b3" in your header will do the trick). The item URI should just be there on items in Z5.

@duncdrum you mean to tell me JurisM is still on the 4.0 line? That's just great. not.

[retorquere]

@azag0, @dbobak, @hquilo, @lyndondrake, @floriandierickx, @vtcarlino, @dmbortz: zotero-better-bibtex-5.0.2.circle.3906.xpi is out and has debug-report under preferences-BBT-advanced-submit

If anyone is familiar with XUL CSS styling or is dying to learn about it, I could use some help. The prefs and the debug report screen look like ass and I have no idea why.

[retorquere]

3907 has right-click to submit references along with a debug report.

[dbobak]

@retorquere, thanks again! And again one question: is there a simple way to remove double curly bracing from some fields in exported Bib(La)TeX file?

[retorquere]

@dbobak https://github.com/retorquere/zotero-better-bibtex/wiki/Unnecessarily-complicated-BibTeX-output

[dmbortz]

Just tried to export again with 3908. Got the same error as above "An error occurred while trying to export the selected file". Here's the Report ID: 4YCH8F9Y

[retorquere]

@dmbortz please submit separate issues for this as this thread isn't really trackable anymore.

[dmbortz]

Ok, will do!

[retorquere]

@azag0, @dbobak, @hquilo, @lyndondrake, @floriandierickx, @vtcarlino, @dmbortz: I need to rework the database access stuff for citekeys. No testing will be required until I report back on this thread. Sorry for the disruption.

[retorquere]

Feedback time: currently, there are two triggers that cause dynamic keys to be regenerated: the reference changes (that one reference gets a new citation key) or the citation key pattern changes (all references with dynamic keys are updated).

I'm considering dropping the second trigger as it's a performance problem in the new version. This would means a pattern change would affect new/changing references from the time of change forward, but not retroactively (which could even be considered a good thing).

Acceptable?

[retorquere]

There would still be a way to regenerate dynamic keys for any selection of references, but it'd be manual.

[dbobak]

Sounds legit for me. But I do not use dynamic keys, so maybe I cannot imagine some workflows in which this would be crucial.
On the other hand, changing citation key pattern probably does not occur very often (I've changed it only once in few years), so maybe the performance problem would not be so important?

[retorquere]

I agree it's unlikely to change often, but the new setup with the keys in the reference itself there are new performance implications, and with some people having 20k+ references in Zotero, a pattern change could cause a resave of the whole library, and that would then sync out, causing all synced clients to change, etc. That would stall zotero previously, which was annoying, but now it will introduce race conditions, and I do not want to debug race conditions by proxy when it hits a bbt user.

That's it. Unless someone makes a case why the second trigger cannot be dropped, it's out. One less race condition to worry about, and much simpler code.

[jGleitz]

I think that as long as there's a button like “regenerate all dynamic keys with new pattern” in the UI (preferably next to the pattern setting), it's totally fine to drop the second trigger.

Am 09.08.2017 1:08 nachm. schrieb retorquere [e-mail]:

I agree it's unlikely to change often, but the new setup with the keys in the reference itself there are new performance implications, and with some people having 20k+ references in Zotero, a pattern change could cause a resave of the whole library, and that would then sync out, causing all synced clients to change, etc. That would stall zotero previously, which was annoying, but now it will introduce race conditions, and I do not want to debug race conditions by proxy when it hits a bbt user.

That's it. Unless someone makes a case why the second trigger cannot be dropped, it's out. One less race condition to worry about, and much simpler code.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[andersjohansson]

Or a note that if you want to regenerate citekeys you can select whatever you want (whole library?) and press "regenerate citekeys" in the right-click menu.

[retorquere]

Both of those are easy

[andersjohansson]

BTW: I mostly use dynamic keys and think it's reasonable that they don't get automatically updated if the pattern is changed, as long as the user is made aware of this. As you say, this can even perhaps be considered a feature (keeping old keys).

As an example of a use case:
I have in my years of using betterbibtex changed the pattern once. Some citekeys in my documents then became outdated, but could be updated since I write in Emacs org-mode, with org-zotxt (where the zotero-id is stored in the link format used, [[zotero://select/items/0_T6F2XYIU][@citekey]], so the citekeys could be regenerated with some elisp-hacking). But having old citekeys kept might actually have been good there.

[retorquere]

With the whole async thing Z5 has going on makes maintaining a shadow database interesting (in the apocryphal Chinese sense), so I'm trying to move the citekeys into the main Z5 database. This means in turn I have to be careful about how much changes I'm making to the references -- where in my previous shadow DB, updates were nearly instantaneous (as it was all in memory and only occasionally flushed out to disk), the Zotero DB commits every change immediately (for good reason), and the previous "DB calls are free trololo" approach I had would lead to severe trashing.

[retorquere]

@azag0, @dbobak, @hquilo, @lyndondrake, @floriandierickx, @vtcarlino, @dmbortz: ready for test again: please pick up 3931 (or later). manual reset is not yet ready.

[dbobak]

Great! Will you create xpi with changes from https://github.com/retorquere/zotero-better-bibtex/commit/38bc42cf58da99566304f0fe8723154259d44033?

[retorquere]

@dbobak that's all automatic (running here); if I commit on a branch that doesn't start with @, and the tests run green, the tested XPI gets uploaded to here. I merge back to master when the fix is confirmed, and that in turn will auto-release a new test build.

[dbobak]

@retorquere ok, I had no idea about that automation :)

[retorquere]

Doing it this way keeps me honest. If it doesn't pass the test, it doesn't get released, so people aren't bothered with testing releases with known faults.

[dmbortz]

I tried doing an export (using 3936) and it works for some references and not for others. For one that breaks it, the Report ID is: EG89WWHE

In looking closely at a reference that worked, it appears that it has an entry in the "Extra" field of "bibtex*:Wang2017IEEESpectr" The reference that doesn't have such an entry cannot be exported.

Was there something I should have done to generate keys for all bibliographic entries? I've read the discussion above, but it's not clear if there is currently a "button to push".

[retorquere]

@dmbortz can you open a new issue for EG89WWHE? This thread has become a discussion thread, it's not really doable to do tracking of specific problems anymore.

References that don't have keys will trigger an error right now, but BBT should have done a scan-and-fix at startup. Let's figure that out in a new issue.

[dbobak]

@retorquere @dmbortz I have created a new thread for the issue, its here: https://github.com/retorquere/zotero-better-bibtex/issues/698

[retorquere]

Vote time again: should the [year] pattern expand to 0768 or 768?

[retorquere]

(the reason I'm asking is that I just found out that shortyear gives you 02 for 1902, not 2. Perhaps these are different cases).

[retorquere]

For future testing reference: at least during testing, exporting will error out if a reference doesn't have a citation key. Undecided whether that will make it into the final release, but for the moment it is an expected effect if the citekey generation fails (which it shouldn't)

[Wekhrlin]

As historian, I would say that 768 would be right. But it's far ahead of my research era. ;-)

[retorquere]

Historians would know about dates I'd say. The existing behaviour stays.

[dbobak]

768 IMO. It's more elegant. And I'm not historian, but archaeologist ;)

BTW, I've just noticed some strange behaviour with really old dates. Zotero has a problem recognising dates older than 100, but BBT deals with it in some way. The line below shows zotero record and BBT export to Biblatex result:

Zotero      BBT Biblatex
99      -0099
-99     -0099
99 AD       -0099
99 BC       -0098

[adam3smith]

Yes, 768. Short years always take the last two digits (by convention). "Long" years are as long as they need to be.

[retorquere]

That "in some way" is the date parser I threw together. It handles a wide range of creatively entered dates (and perhaps some dates it really shouldn't).

[retorquere]

@adam3smith long time no see 😄 .

[retorquere]

3942 is out -- get it while it's hot

[dbobak]

Got it, checked it. Keys are created at startup. Cool 👍

[dbobak]

And getting back to the dates - these "old" dates (Before Christ, Anno Domini) are crucial for some historians. IMO 99 AD and 99 BC should be left intact.

[EmmanuelCharpentier]

Dates : By all means, 768 for reference production. But an historian might want to add the era of reference : for axample, "768 AUC" (Ab Urbe Condita) might make more sense to him/her (e. g. date of a law, or of a text quoted by another text) than an (uncertain) translation to Christian Era ; similarly, (another) historian might appreciate to be able to date a law as "Loi du 10 nivôse an II de la Liberté".

For key generation/srting purposes, "0768" gets sorted more usably (i. e. after "0079" and before "1141") that "768" that would end up before "79" and after "1179".

[retorquere]

These are stacked matters -- the date parser tries to make sense of dates, the pattern formatter chooses how to work them into the keys. Whether the keys would have 0768 or 768 is separate from what ends up in exported date fields.

@dbobak you mean the date field should say 99 AD rather than 99? @njbart, any insights?

[retorquere]

768 AUC is probably left untouched and passed through verbatim.

[dbobak]

@retorquere, I think it should be left as it was entered in Zotero. AD is default, so it should be omitted, but (maybe?) sometimes it should be left as it is. For sure BC should be left if it was entered. Also @EmmanuelCharpentier comments are very sensible.

EDIT: yes, AUC is left untouched.

[retorquere]

I smell a preference in the near future.

[retorquere]

"as entered" is not as clear a term as one might suspect BTW. Most references I get are imported, and the quality you get isn't always great -- the date parser tries to compensate for that fact. In any case, verbatim date handling is an enhancement request against what BBT4 was -- let's get the existing behavior up and running stably first, and change it later.

[dbobak]

I agree. Old functionality and stability first. But we are in discussion thread, so I believe we can left here some ideas for the future.

[EmmanuelCharpentier]

Dates : I think that what was entered in a "Date" field should be left untouched. But now, our bibliographies are monolingual. Ouch.

Sorting is probably impossible to do rigorously without adjoining to Zotero a whole date-conversion package, which would need a lot more input than currently available. Think of Roman, Christian, Buddhist, etc.. calendars. And what about the various dates of adoption of the Gregorian calendar ? What about XVII-XVIIIth century English usage of "Old style/New style" dates ?

A workaround could be the adjunction of a "quoted_date" field, holding the date of a document as published by its author(s). For example, a document dated July 27th, 1794 CE could be quoted as published on "9 thermidor an II"...

However, such an adjunction could be used only by biblatex users : I'm not aware of any other bibliographic software offering enough freedom to its users to make practical use of it (maybe bibtex, with a lot of stack juggling, mangling and stomping.. I wouldn't want to write the relevant code !).

Can someone suggest a sensible compromise ?

[retorquere]

This is indeed a discussion thread, but it's for the discussion to get BBT5 up and running. I much prefer having change/enhancement requests in separate issues. Things get buried here fast.

[EmmanuelCharpentier]

I got a curious error after a (lengthy) synchronization of Zotero on a small Windows machine against the database server : Zotero complained of a nonexistent predicate "isReplacedBy, (IIRC). This has been reported in the Zotero forum.

[EmmanuelCharpentier]

More precisely : the "error box" one can get by clicking on the "exclamation mark" button contained :

````
An error occurred during syncing:

Unsupported predicate 'dc:isReplacedBy'
````

HTH,

[lyndondrake]

I’ve just tried 3956. It installs fine, but when I try to export I get the following:

https://www.dropbox.com/s/rxtjt5wz1z4h0lu/Screenshot%202017-08-13%2008.24.01.png?dl=0 https://www.dropbox.com/s/rxtjt5wz1z4h0lu/Screenshot%202017-08-13%2008.24.01.png?dl=0

Any ideas on how to proceed?

On 11 Aug 2017, at 5:31 pm, EmmanuelCharpentier [e-mail] wrote:

More precisely : the "error box" one can get by clicking on the "exclamation mark" button contained :

An error occurred during syncing:

Unsupported predicate 'dc:isReplacedBy'
HTH,

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/retorquere/zotero-better-bibtex/issues/555#issuecomment-321829492, or mute the thread https://github.com/notifications/unsubscribe-auth/ARpuLQsnty51BQe9SvX59igSVBuuVqL5ks5sXGXIgaJpZM4JtLTL.

[retorquere]

Testing is stalled again as I need to figure out a way to safely close the &%#@ database at shutdown.

@lyndondrake you can go into the preferences, better bibtex, advanced, submit error report. But please hold off as I figure this database thing out.

[qiuzhu]

where can i download z5 bbt ,ca not wait, beta is good, where is the link

[retorquere]

I'm reworking the database access (again -- storing dynamic keys in the extra field was a problem), I really hope to have a new version out. Please do back up your Zotero.

[retorquere]

The database rework is done but testing is currently stuck on performance issues; with just 3.5k references my tests start failing because erasing the library between tests times out. I've asked the Zotero devs for insights, but I really can't do much in the interim. For the curious, this rework will show up as 3991, but I really need to sort out how I'm going to do the database layer before I can get back into functional problems.

So much fun.

[TaylanT]

Hey guys,
just installed the zotero-better-bibtex-5.0.2.3992.master.xpi and the autosync function seems not to work properly. In the library export tab i choose BibTex and keep updated.
When I click on preferences the “Better BibTeX” tab and then the “Automatic Export” tab. My current
autosyncs is not listed in the box. How can I fix that?

[Wekhrlin]

It's not implemented, yet. See: https://trello.com/b/OpJQyVTP/better-bibtex-for-zotero-5

[retorquere]

4032 now shows the citation key in the reference list and in the reference edit pane.

[dotran]

Hi, I've just downloaded 4032 from
https://github.com/retorquere/zotero-better-bibtex/releases/tag/builds
(I intentionally put the link here as I see it's so hard to find the build!)
and with my latest Zotero v5.0.17, I've observed (and need your confirmation) that I cannot export an item via Better BibTeX and I also don't have any entry on the context menu to pin the Citation Key (which I see now is appearing on top of the right pane as you've just announced).

If you think those shouldn't be like that, i.e., I should have missed something or configured my BBT wrong, kindly let me know. In case what I'm seeing is expected, could you remain the bibtex: ... and bibtex*:... in the Extra field? (or add an option to configure this, since I need to parse the citekey after exporting using Zotero's default BibTeX in case your exporter isn't working yet.)

[hivemindtheory]

Thank you very much for the link, it was indeed hard to find it somewhere.

@retorquere Thank you so much for your engagement, I totally hope you will get closer to a stable release soon, as I have to hand in my Master Thesis very soonish. ;)

[retorquere]

Hi @dotran; Better BibTeX export should work right now; since for you it doesn't, please open a new issue and submit a debug log from the references you were trying to export. Citekey pinning got broken in the latest database rework.

Dynamic keys are no longer in the extra field and the bibtex*: format is no longer recognized, but all translators will now find an .citekey property on items fetched by Zotero.nextItem() in the translator, so no need to parse them out of the extra field. For pinned keys, the .citekey should match what is still in the extra field.

[retorquere]

@hivemindtheory I also look forward to getting this port behind me :roll_eyes:

[retorquere]

(the reason for the deprecation of bibtex*: right after I introduced it was that I couldn't get the performance right)

[dotran]

Hi @retorquere, many thanks for your feedback. I check again with greater care and confirm that Extra does have the pinned citekey. Only those that are unpinned don't appear in Extra and there is no way to pin them. So this is "normal".

Regarding the Better BibTeX export failure, I check again and realize that I have the following Postscript in the Advanced tab of BBT configuration window:

if (Translator.BetterBibTeX) {
    if (this.item.dateAdded) {
        this.add({ name: 'dateadded', value: this.item.dateAdded });
    }
}

And as long as I remove this script, the Better BibTeX export works just fine. (This script used to work for me on Zotero 4.)

Anyway, I am submitting an issue with Report ID EP3TZF3B. Please let me know what's wrong the above script if you know.

[retorquere]

Oh crap -- postscripts. I'll fix those.

[retorquere]

Please submit debug reports on separate issues. This thread has gotten too unwieldy to track stuff like this on.

[dotran]

Submitted. #706

[retorquere]

Bleep bloop; this is your friendly neighborhood build bot announcing new test build 4044.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4048.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4049.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4050.

[dotran]

I just look over the Trello board and see citekey pinning on top of the Planned list. Are you somewhere close to this? I think this is quite important.

Another side question: I notice the bracing in the title field in Better BibTeX export is somewhat redundant, e.g.

title = {Robust {{Classification}} for {{Imprecise Environments}}},

while Zotero's BibTeX export gives

title = {Robust {Classification} for {Imprecise} {Environments}},

Any grounds for this or it just eases the implementation (and nothing gramatically wrong with that)?

[retorquere]

@dotran citekey pinning works, but the UI isn't done yet. If you type bibtex: xxxxx in the extra field, that's the pinned key; if you remove it, you'll get a new dynamic key. As to the braces, see this bit of insanity.

[dotran]

@retorquere bibtex: xxxxx is amazing -- just tried it out!

With this I can manually pin the inconsistent autogenerated unpinned citekeys as reported in #707. When you're done with the UI, please keep this manual pinning around. (I love it, until the autogenerated unpinned keys are truely perfect.)

[retorquere]

@dotran the bibtex: xxx way is the actual way to pin the citekeys -- I have no other way, until Zotero adds a proper field for the citekey, hopefully late this year or early next year. The shadow database I maintain for dynamic keys caches the pinned keys for efficiency, but the key in the extra field is authoritative, and the UI for (un)pinning really only adds the dynamic key to the extra field or strips the pinned key from the extra field. #707 shouldn't be hard to fix, it must be a cache invalidation problem (famous last words). A debug report will tell me whether that's the case.

[dotran]

Good to know that!

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4057.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4064.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4065.

[lyndondrake]

I've installed first 4057 and then 4065 - both seem to work ok at first glance. The time to recreate the cite keys, just fyi on my computer, can be deduced from this:
https://www.dropbox.com/s/y8nik0duitva2pk/Screenshot%202017-08-25%2011.22.00.png?dl=0

The citekey column shows up nicely:
https://www.dropbox.com/s/y1zayqtwkx8nqtb/Screenshot%202017-08-25%2017.27.21.png?dl=0

[retorquere]

I really hope to have #707 fixed today (thanks to the infinite patience of @dotran), and then a whole lot of issues will have been resolved along with that. Watch this space.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4077.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4078.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4085.

[dotran]

I have tried 4085: Pin BibTeX key works, but it wipes out what a reference already has in Extra. (For me sometimes there is a DOI for Book Chapter in Extra.)

[retorquere]

@dotran you mean it clears out the whole extra field?

[retorquere]

Damn, you're right. #711 opened to address this.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4086.

[dotran]

Yes.
Does 4086 fix that?

[retorquere]

Nope. Found the problem, still working on it.

[dotran]

OK. Drop a notification here once this is fixed. I'd love to try again.

[retorquere]

@dotran Can you watch #711 instead? That's where the fix will drop first (it's all automated)

[dotran]

OK

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4089.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4091.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4092.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4095.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4096.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4097.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4102.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4111.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4118.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4119.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4122.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4124.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4125.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4135.

[retorquere]

Imports now work.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4143.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4150.

[edVonSchleck]

Hi,

I installed your latest build (4150) and tried to change the default citation key format ( I tried it with the last "official" release from June as well).
The new citation key format appears in the UI but (even after zotero restart) the default citation key is still used.

Is this a bug or is this feature just not yet implemented?

Thanks
Ed

[retorquere]

@edVonSchleck the latest official release is for Z4 only; it will not work at all under Z5.

[retorquere]

@edVonSchleck wait, you say the new key appears in the UI but the old key is still used; still used where? In the export? That's a bug for sure.

[edVonSchleck]

Ok. Got it. Thanks. New references appear indeed with the right (new) citation key.
Sorry.

[retorquere]

New or changed references.

No apologies expected or required.

[duncdrum]

Juris-m is now available as 5.0 🎉
I ll try to set up a test system soon to chime in on BBL and juris-m in the brave new age of 5️⃣

[retorquere]

@edVonSchleck if you want all keys to update you can select them, right-click, and select "refresh".

[retorquere]

@duncdrum I'll try to set BBT up for Juris-M testing in the weekend.

[EmmanuelCharpentier]

The Z5 version of BBT seems ready for (at least) alpha test. I'd like to suggest an update fot the README.md which is displayed on the Better BibTeX GitHub home page :

• There should be a pointer to the page where test .xpi can be downloaded.
• There should be also a couple notes indicating how these .xpi can be installed, as well as a notice explaining what is expected to work and not to work, and the possible (expected ?) side effects of the installation.

Of course, these two elements could be put on a special page such as "Alpha test of BBT on Z5", pointed to on the BBT homepage, and updated when retorquere thinks that his recent work needs testing(or at least tire-kicking...).

What do you think ?

[gracile-fr]

@EmmanuelCharpentier : good idea to improve that. I'd say the readme.md should point to the wiki with a special (new) page for BBT for Z5.

[retorquere]

@gracile-fr the wiki is editable for anyone with a github account, and it's not as easy for me to keep track what still needs doing. I currently use Trello for that, I can move them to a GH project, but I need something to keep me organized and that makes it easy for me to keep the status up to date; a wiki doesn't serve that purpose efficiently while things are still moving very fast.

[retorquere]

Strike that -- GH projects are anemic to the point of being unusable. No comments, no history, no attachments, no labels -- no thank you.

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4160.

[steve-biggs]

Sorry to add to an already enormous conversation but I'm not sure how else to get in contact.

I am available for testing. How can I get the latest version that needs to be tested? How / where do you want me to report success / failure?

Thanks

[retorquere]

:pager: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4165.

[retorquere]

@steve-biggs thanks for offering, any help in testing is very much welcome. The conversation here has the nature of a chat channel now and that's OK for this specific issue.

New builds will be announced here (see the bleep bloop just above this comment); the current state of affairs can be found here; for things that are in the Done or Needs testing columns that do not work as you would expect, please open new issues, don't report them here, because it'll get drowned in the noise. For things that are In Progress, Planned or Known Problem, please don't file issues at this time, it is known that they don't work right now.

[jrennstich]

THANK you SO MUCH for bringing this essential tool back to life!

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4178.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4195.

[retorquere]

Aww yeah -- 4195 includes auto-export for collections and libraries (not search)

[jrennstich]

Build 4195 is the first one for me that did not force me to reboot Zotero 5 several times (usually 2-3) after installation. I always got an error message from the Zotero database (error - could not load, please restart). Auto-export seems to work fine so far.

[retorquere]

@jrennstich if you see that problem again, can you open a new issue? I didn't do anything specific that would address this, and I'd rather get ahead of problems like these.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4196.

[jrennstich]

Error Report ID: 37CL27LA. Installed the build 4196 with no errors. Then I upgraded to Zotero 5.0.18 and received same error message as previously (error - could not load zotero database). After restart everything is fine.

[jrennstich]

The citation keys don't seem to refresh properly. When I click on a different entry, I sometimes get a Citation key from a previously pinned or used entry. When I click on another entry and then back to the one that showed up wrongly, it's all correct, however.

[jrennstich]

Over at Zotero Forums, I saw reports similar to mine - Dan Stillman says "You're getting lots of errors from BBT":

raffam today at 6:12pm
I am unable to use Zotero anymore

Now it does start but when I click onany element in the middle of the screen it shows "An error has occurred. Please restart Zotero"

Please help

Report ID: 102237694

dstillman 3 hours ago
Upgrade to Zotero 5.0.18, now available via Help -> Check for Updates…, and if you're still having trouble, disable all your other extensions. You're getting lots of errors from BBT.

[retorquere]

Please open a separate issue. #555 is not usable for tracking problems anymore; too many people involved.

[retorquere]

@jrennstich WRT the citekey not refreshing in the details pane this is currently a known problem

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4201.

[retorquere]

@jrennstich 4201 may fix the issue you're seeing but please open separate issues if you meet problems. Problem solving requires a focused conversation between me and the reporter, and this thread has become a bit of a cocktail party; focused conversation is not really feasible here with multiple discussions interleaving.

[retorquere]

@duncdrum WRT Juris-M: https://github.com/Juris-M/zotero/issues/34 .

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4214.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing new test build 4220.

[floriandierickx]

Hey everybody, I went through all the developments, it looks promising and beautiful! I'm ready to test on my research database in Zotero (it converted automatically when updating to Z5, although I'm not enough of an expert to know what happened there..), but as I am a bit afraid of throwing my whole research database in the experiment: is there a way to safely test the extension? Should I backup zotero as it is now (https://apple.stackexchange.com/questions/101025/how-can-i-backup-an-application-in-the-applications-folder) and start testing from there, or is there a way to create a double version of zotero(+extention) in a sandbox on a mac (and keep the other zotero running as it is) ? if this is too much of a newbie-talk you can leave this comment for what it is for now, and I'll jump in at a later time, there's no urgency :) Thanks a lot for the work!

[retorquere]

I don't know if I'd call it beautiful 😄 ; befitting a long-time unix/latex user, I have no truck with UI work. In fact, if someone is deeply enthousiastic about UI work in general and CSS in particular, I have plenty of work I would pitch your way in a heartbeat 😆

Abyhow; yes on all accounts, but to be more specific:

1. BBT no longer changes your references (except citekey pinning), so it should be safe. It's in principle still possible to screw up your BBT database, but as the BBT databases live outside the Zotero database, that damage should be contained (and I haven't seen it happen yet).
2. Backing up and testing like that is perfectly safe but a bit of a nuisance
3. You can start an isolated profile if you're using Zotero 5.0.18+ by using the -P -datadir profile command line parameters when starting Zotero (how exactly you effectuate this depends on your platform)

[floriandierickx]

Oki, thanks! I'll go for option 3 and if that seems not to work out as expected, option 2 thus :)

[retorquere]

@floriandierickx what OS do you use? Linux, Windows, Mac?

[floriandierickx]

Mac with sierra 10.12.6, is this one already 'represented'?

[retorquere]

Yup. You can start it using

/Applications/Zotero.app/Contents/MacOS/zotero -P -datadir profile

A popup will appear asking you for a profile; you can create a profile for testing there.

[floriandierickx]

Ok, that's nice, thanks!

[cunio]

@retorquere how to make that Zotero automatically updates the BBT (alpha?) addon when a new build arrives?

[retorquere]

@cunio looking into that.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.16.

[retorquere]

From v5.0.16 I will start creating pre-releases that will auto-upgrade; not every build will be tagged as pre-release, but when sufficient fixes have collated I'll manually trigger a new pre-release.

For specific issue reports I will still create per-issue builds which will have to be installed manually.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.17.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.18.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.19.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.20.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.21.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.22.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing test build 4318 ("support for date lists in date parser").

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.23.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.24.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.25.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.26.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing test build 4352 ("adjust tests for biblatex parser 1.1.2").

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.27.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing pre-release v5.0.28.

[inviridi]

Just wanted to say that v5.0.28 installed flawlessly on my OpenSuse 42.3 box, imported the library (>700 entries) and even exported it without fuss. I am relieved and grateful; thanks for your hard work on BBT!

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing test build 4384 ("skip release on nightly").

[mguzmann]

This is looking great? any idea when the official release might happen?

[retorquere]

I think it's usable as it is now, but the first non-pre-release comes when the planned column is empty (sans Juris-M support, which is not a show-stopper for me).

[retorquere]

WRT "when" in terms of dates: just when it's done. I seriously have no idea on an ETA. Could be a few weeks, could be a few months. I've been going at the rewrite since early July and new surprises keep cropping up.

[mijalche]

I might be requesting too much, but is it possible the next feature that will be done to be CAYW support. Highly appreciated!

[retorquere]

CAYW is in progress, but that too has changed substantially in Zotero and is taking a lot longer than I had hoped. Which is a bit of a theme of this rewrite.

[retorquere]

[rouckas]

Hi, I tried installing the latest test build but I am told that it is not compatible with Firefox 56. Is that a known issue, or am I doing something wrong?

[adam3smith]

@rouckas Zotero 5 only exists as Standalone software, so you'll need to install the test build there, not in Firefox (under Tools --> Add-ons --> Install from file).

[cossio]

@retorquere Yeah. I just installed tried to install it manually and it is working great. Thanks a lot for the hard work! Hope the official release is finished soon. Cheers

[retorquere]

It depends on your definition of "soon". My lifetime: likely. This year: possible. This month: no.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing test build 4486 ("Merge branch 'master' into @ts").

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing test build 4497 ("reset \"Keep updated\"").

[naught101]

@retorquere: My dad is a builder. He's always lived by the saying "A finished house is a dead house" :grin:

[retorquere]

I fully agree, but it would be nice to move out of the "living in a construction site" into the "home improvement" phase :roll_eyes:. I'm also gearing up for a PhD which involves a lot more uncertainty prep time than I had anticipated. For those who've been through this before: how did you survive :fearful:.

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing test build 4502 ("Merge branch '746'").

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing test build 4510 ("es2017 seems to work").

[retorquere]

:robot: bleep bloop; this is your friendly neighborhood build bot announcing test build 4514 ("don't submit builds to #555 anymore").

[pvitt72]

Hi, I'm unable to download the XPI (test build 4514)... I get the 404 error...

[retorquere]

Test builds are only available for approximately a week; you can pick up 5.0.28 for now, I expect to put out a new pre-release somewhere this week when 4 issues I'm working on have cleared.

[pvitt72]

OK, thanks! Got the 5.0.28...

[warint]

thanks a ton for your hard work on this one! Very appreciated.

[steve-biggs]

Here here! Solid job mate. Much appreciated.

[vtcarlino]

Any progress on the cayw feature?

[retorquere]

@vtcarlino as it happens it was completed yesterday :). I'm waiting on feedback on #590, #747 and #751 so I can cut a new release. Progress tracking has now moved from Trello to a github milestone, so you can see what's stilll open (although I did forget to add CAYW to that list, now added but closed)

[retorquere]

The Trello list is now gone; work is being tracked here. If this list is still missing stuff, please file new issues and I'll add them to the milestone. Things are speeding up now, the full release could well be this month (no promises though).

[JannikBeusse]

Dear @retorquere,
Thanks for your work on better-bibtex - it's a great add-on!
I've got some troubles with Z5. The latest update works fine most of the time but there's one bigger issue:
When I'm exporting the library (doesn't matter wether with better bibtex or better biblatex) the titles of the entries are getting a weird character with capital letters for each word and braces. It looks like that in the exported file:
title = {Numerische {{Simulation}} Der {{Beanspruchung}} Eines {{Geogitters}} Bei Der {{Radüberfahrt}}}
When I'm exporting the library just with biblatex from Z5 this works fine. The problem only seems to be find in the type of entry: conference-paper.
How can I solve that?

You gonna find the corresponding Report with the ID: IU55HPSD

Thanks a lot!
Jannik

[retorquere]

@JannikBeusse this is by design, see here.

[JannikBeusse]

@retorquere thanks for the fast reply.
Does it mean that I've to put <span class="nocase">...</span> around every title to stop the capitalization of each word?

[njbart]

No. Most likely the “Language” field of your entry is either empty or contains a string that begins with “en”. Case conversion is triggered only for entries that fit this description. For German entries, use “de-DE” instead.

[JannikBeusse]

perfect! Thank you so much @njbart !

[retorquere]

To be specific, this explains why BBT does this whole spiel with the capitalization. Zotero's exporter doesn't do this to keep things simple (BBT is not simple), and the simple approach will usually work, I just don't like "usually" when it comes to citations.

[retorquere]

5.0.31 contains citekey search and CAYW.

[retorquere]

And the remaining work lives here; the last 3 issues are really only notes-to-self for cleanup; there really are only 3 issues left, and I'm not yet settled on whether SCHOMD will make it to the release -- not sure whether it's still being used when https://atom.io/packages/zotero-citations really isn't getting active maintenance, and the Brackets plugin hasn't complained about the lack of support... so maybe only 2 issues to resolve.

[EmmanuelCharpentier]

A couple questions :

• Is there any documentation for the "new features" ? For example, I suppose that "CAYW"means "Cite As You Write", but I haven't the foggiest idea of what that means in this specific context.
• What is (or means) "SCHOMD" ?

[retorquere]

These aren't new features; BBT5 is a ground-up reimplementation of what I already had for Zotero 4. It had to be done or it would have meant the end of BBT, but functionally, nothing has been accomplished the past 5 months :disappointed:.

CAYW = indeed Cite as you Write.

SCHOMD = Scholarly Markdown, but it was always experimental even in BBT4. It really needs good editor support to make it work, but I'm not getting into the editor game (again; the abandoned zotero-citations for Atom was of my hand).

[retorquere]

(If you're asking about documentation in general; what exists exists on the wiki. Which is publicly editable. Nudge nudge, wink wink, to that 100% of the population who correctly assesses that they're better than me at docs)

[EmmanuelCharpentier]

"what exists exists on the wiki".

Okay. I missed _that_. Which, by the way, should be pointed at by _big_ fat _pointers_ on the Better Bibtex "Preference" pages (it's about the only point where you can anchor it in Zotero's UI, unless you can add an item in the "help" menu, in which case a "Better Bibtex documentation" item should point to the Wiki...).

[EmmanuelCharpentier]

"... I'm not getting into the editor game (again ..."

I sympathize. But the CAYW page gives me an idea for a possible addition to RefTeX/AUCTeX...

[retorquere]

@EmmanuelCharpentier you mean big fat pointers like the Help button at the bottom-left of every preferences pane? :laughing:

I can (and do) add menu entries. As should be painfully obvious, my expertise is not UI work (the prefs pane has some alignment issues of my doing that I have no idea how to fix), and as it's usually fiddly work, I try to get away with re-using the existing UI where I can.

And ideas for addition to existing editors is exactly what CAYW is for. The Atom thing was a proof of concept. If you need a new format, just holler, they're easy to add.

[Isimoro]

@EmmanuelCharpentier I'm pretty sure you can just give the BibLaTex URL for any given Collection to Reftex/Org-Ref to use as Bibliography. No additional Script needed.

(I'm still on BBT4 don't know if this works in BBT5)

[retorquere]

This works again as of a few days. But CAYW is for picking references. The url won't automatically refresh if new references are added, it's updated when biblatex requests it, which is usually only at compile time. CAYW and pull export combine pretty nicely.

[Isimoro]

Reftex and Org-ref implement their own CAYW interface (I mean - we're talking Emacs), they just need a Bibfile to pull from. But if the url isn't always up to date this is probably not an optimal solution.

I just automatically export my Collection when I update it and pull refs from the exported file that way I don't need to always run Zotero either.

[retorquere]

That's how most people use it - pull export was just relatively easy to do so that appeared before I even thought of auto-export (and even when I did think of it it took a while because it has UI work, and I hate UI work).

I only use auto-export myself these days, and I'll typically eyeball and retype citekeys rather than using CAYW of any kind. I'm a vi user; I don't take my hands of the keyboard for anything if I can help it at all. AAMOF I don't usually have zotero running unless I'm actively managing references.

[happydemic]

What heroic efforts you have made! Thank you so much for keeping this project alive after Zotero 5.

Thanks also for keeping CAYW support going. It's great to have it in 5.0.31.

You're probably well aware, but I noticed that while the other CAYW output formats are working, scannable-cite gives:

$ curl -s http://localhost:23119/better-bibtex/cayw?format=scannable-cite
CAYW failed: TypeError: Zotero.BetterBibTeX.Pref is undefined
scannable-cite@chrome://zotero-better-bibtex/content/BetterBibTeX.js:18521:19
init@chrome://zotero-better-bibtex/content/BetterBibTeX.js:18323:36

[retorquere]

@happydemic nope, not aware at all, as I had forgotten to reactivate the cayw tests. scannable-cite should now work in .32.

[retorquere]

Yes, that means what you think it does.

[JohnLukeBentley]

@retorquere that is an awesome result. Thanks for the marathon effort. When's the BBT 5.0.35 party at your place?

[retorquere]

I think somewhere in spring of 2022. If I uncharacteristically don't screw up my pitch next Thursday, all my time and then some will be consumed by the PhD I would be starting then. But you're welcome for a beer. By sheer coincidence I happen to have a bottle of Westvleteren, reportedly the best beer in the world.

[JohnLukeBentley]

Well I'll join you with the bottle of the Chateau de la Municipality Tap Water I have to hand. Good luck with the PhD pitch.

[rcoreilly]

I'd like to add my major congratulations! Finally made the switch over and everything worked remarkably smoothly! even with 22k citations the export was not too bad -- about a minute or so. Good luck with your PhD and thanks again for all your effort on this.

[retorquere]

Thanks - today's the pitch and I'm having a major case of imposter syndrome (or so I've been told).

That minute, that's with a hot cache? The old numbers would make a minute plausible. It wasn't likely that BBT5 was going to out-perform BBT4. Still odd though. With the caching everything should be in memory, ready to just write to disk. A minute is a lot for that.

[dbobak]

I'm joining with congratulations! BBT is a really piece of good job.
And good luck with PhD - I know that pain ;)

[retorquere]

Thanks man - not a day goes by I don't get into a raging panic. The profs involved seem confident I can do it, I just have to find out why seemingly very bright people would say such things 🙄. Very, very excited to get back into the academic field though, and secretly I know if I can get this godforsaken RQ right I'm golden.

Please forgive me for what I'm about to say, as I fully realize English is not your native tongue, but it's too funny to not react - BBT aims to be more than a piece of a good job at some stage and be a fully good job 😆. That's honestly the one good thing I got out of this blasted rewrite (other than continuity, and I'll need a reference manager for that phd should I not screw up this afternoon) -- having revisited every single line of code unearthed more than a few "Oops that should not have been done this way" spots. The rewrite to typescript (much more strict than coffeescript) while I was at it helped with finding those. But my initial assessment of "this is going to be a metric fuck ton of work" was off by at least several fuck tonnes 🙄.

[dbobak]

I'm not sure if I was fully aware of the amount of work rewriting BBT was, but I was sure that was A LOT :)

And the "piece of a good job" was a loan translation from Polish, where it means "a really good work not as little as one could expect". Particularly, not a piece of cake (which in Polish translates into "bread roll with butter") :D

Let us know how's your pitch!

[retorquere]

I'm sure I was only vaguely aware of the amount of work, which is a good thing, because knowing it to any accurate degree would have scared off any sane person. Lucky for all of us I guess, "sane" is not an adjective often applied to me, and at some point I just dug in and rode it out. "Not a piece of cake" is accurate.

Very glad to have it behind me. Very, very glad.

Now I'll be very, very glad 3 hours from now, when the pitch is done. Then other mayhem awaits of course, but not pants-pooping-scary mayhem, just the usual "this isn't reasonable at all" kind of mayhem.

[retorquere]

So.... If "good job, start next week, we'll arrange a desk, here's your copromotor" is to be taken as an indication that they're actually considering me as a candidate...

[ConorIA]

Congratulations! 🎉🎉🎉

[dbobak]

Congratulations!

[cjpoor]

Thank you so much. You are a real gem! Congratulations on a brilliant effort.

[vtcarlino]

Hi,
I'd like to disable the @comment lines BBT puts at the end of .bib files, such as these : @comment{jabref-meta: databaseType:biblatex;}

I remember I have to put a "0" value in Zotero's about:config but I forgot the preference name. Could you please help?

[OlafKleinBerlin]

Congratulations. Thanks for your hard work with BBT and good look for your PhD.

[retorquere]

@vtcarlino the prefs are documented here, but I've made a few changes in 5.0.39 which exposes the pref in the UI, and sets the default to "off" (BBT is not primarily a pathway to JabRef after all)

@JohnLukeBentley I've also exposed a pref for the QR.

[JohnLukeBentley]

@retorquere that works a treat. Default to "off", for the reason you give, seems spot on.

One suggestion this kicks up is to include a table of contents at the top of wiki pages with h2s.

[retorquere]

@JohnLukeBentley I need to reconsider the wiki in any case. I'll probably go back to gh-pages.

[JohnLukeBentley]

I'm a bit unclear about the relationship between gh-pages (https://pages.github.com/) and the wiki as is. I mean I see that:

• A page like https://github.com/retorquere/zotero-better-bibtex/wiki/Bundled-translators can be directly edited; but
• A page like https://github.com/retorquere/zotero-better-bibtex/wiki/Configuration/ is autogenerated from project files.

... on the matter of the basic choice between the two I think I'd prefer direct editing. I suppose this swings on whether you, the repository/account owner, prefer:

• Less friction for your users: making it more likely they'll make contributions to the wiki;
• More friction for your users: allowing you tighter control of the edits users make.

But there may well be other issues and reasons for going with gh-pages (https://pages.github.com/), whatever that is.

[retorquere]

Those assessments are correct. gh-pages allow for more flexibility in makeup of the page. I initially started with gh-pages, then moved to the wiki, but the wiki doesn't allow arranging the pages sidebar, nor auto-generating TOCs, and the wiki didn't ever really get much user edits; of the 265 edits in the history, all but 13 are mine (although TBH, 3 of those were yours, so thanks). There's also the possibility to use the wiki as a CMS and pump that into gh-pages.

[JohnLukeBentley]

Yeah, you'd want to be able to autogenerate TOCs. And by ...

the possibility to use the wiki as a CMS and pump that into gh-pages" ...

you mean gh-pages don't, natively, expose an interface for users (who aren't the repo/account holder) but there's a possible workaround?

And, yes, given the historical evidence that user edits are infrequent (in the case of this niche tool); preserve the need for users to directly edit the wiki is not pressing. At the very least users could suggest changes by posting issues.

[retorquere]

you mean gh-pages don't, natively, expose an interface for users (who aren't the repo/account holder)

They don't expose an interface for anyone, really. There's the online edit, which for a non holder forks the repo and allows creating a pull request. But the process does pose some hurdles for casual edits.

but there's a possible workaround?

In the sense that I could build a workaround.

And, yes, given the historical evidence that user edits are infrequent (in the case of this niche tool); preserve the need for users to directly edit the wiki is not pressing. At the very least users could suggest changes by posting issues.

Or pull requests.

[JohnLukeBentley]

Got the essence of it, thanks.

I happen to have taught myself Git in recent times. So, if the need arose, I might even enjoy having to make a pull request rather than a casual edit ... as an opportunity to further practice the dark art.

[retorquere]

For the time being, the wiki will suffice

[vtcarlino]

Hi, I have an issue concerning CAYW, using the Scannable-cite format. The following URL http://localhost:23119/better-bibtex/cayw?format=scannable-cite returns { | Akrich, et al., Sociologie de la traduction: textes fondateurs, 2006 | | | zu0:TJVCIMFC }.

BBT adds the title of the document ("Sociologie de la traduction ...") while it shouldn't. The result should be { | Akrich, et al., 2006 | | | zu0:TJVCIMFC }. Any idea ?

[retorquere]

@vtcarlino please open a new issue for that.

[vtcarlino]

@retorquere Done. Sorry to spam this thread!

[retorquere]

No worries

[TheoChristiaanse]

Will there be an xpi available in the near future? I've been forced upgraded to 5.0 would like to keep using better bibtex.

[adam3smith]

BBT 5 is released, .xpis are here: https://github.com/retorquere/zotero-better-bibtex/releases and auto-update.

[retorquere]

I hate asking for money for BBT, but the time has come: my MacBook Air is not really hacking it anymore, and in order to do decent cross-platform support, a MacBook is what I need. Anything you can spare towards that goal is very much appreciated.

[JohnLukeBentley]

I've just sent a nominal amount. I wish I was in a position to give the much greater sum you deserve.

[retorquere]

Jeez dude that's not nominal. Thanks!

[jrennstich]

Done. Sent the money.

Sent from mobile device

On 21. Jan 2018, at 13:53, Emiliano Heyns [e-mail] wrote:

Jeez dude that's not nominal. Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[cjpoor]

Hi Emiliano,
Very happy to make a contribution.
Thanks again for all you excellent work!
Best regards,
Chris.

On 22 January 2018 at 03:42, Joachim K. Rennstich [e-mail]
wrote:

Done. Sent the money.

Sent from mobile device

On 21. Jan 2018, at 13:53, Emiliano Heyns [e-mail]
wrote:

Jeez dude that's not nominal. Thanks!

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/retorquere/zotero-better-bibtex/issues/555#issuecomment-359253301,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AInAXwdmaWandA_E8JgR4G8jcZnQjUE2ks5tM0y-gaJpZM4JtLTL
.

[retorquere]

Dayum thanks you guys, I'm already 20% there!

[retorquere]

26% there, thanks guys! I am deeply grateful for the very generous contributions. For people that may stray in here (or long-time followers of this thread even) I want to stress that a) this should in no way be seen as an obligation, and b) I will for sure be paying part of this myself.

Does anyone around here have experience with hooking up 2 monitors to a macbook? I assumed I could use the existing displayport MST hub I used for the windows machine from work; it never worked on my MBA but that was a known issue, but even when an MBPro would support MST, it looks like it wouldn't work with the MST hub; I have no idea what the apple support page says. Apple support said something like this should work, but at some 200 euros, I'm looking for something that will work rather than something that should work.

[pablox-cl]

EDITED: pressed enter way too soon.

PS.- I'll donate to your work: You really deserve it.

[retorquere]

@pablox-cl is something in BBT not working as you expected? Please open a separate issue to describe your problem.

[steko]

Late here but I donated a few days ago (in line with my earlier proposal). Thanks for your great work on BBT!

[retorquere]

Yeah, you were right the first time :smiley:. I didn't want to ask for money for several reasons -- the main one being that many people that I know that use reference managers are struggling undergrads, and I'm doing OK myself, another that my schedule is absolutely nuts and I can't ever promise when things get done, so I felt uncomfortable asking for people's money for little concrete in return. I had forgotten though that I got my previous laptop through a windfall that I don't hope to get again¹. We're not in any kind of dire straights, but I can't justify dropping a lot of money on a hobby project.

---

¹ funny story. I have the odious individual who totaled my car and then tried to pin the blame and cost on me to thank for this -- the low-cost juridical support I have would usually have left me out to dry, as I filled out the form wrong so on paper I was out of luck, but took it all the way including interviewing witnesses after the realization sunk in with them that I was legitimately primarily upset that he tried to put the blame on me rather than stopping to think about the cost. It took in turn a while for me to understand when they explained to me the scrapes and bruises constituted injury, that he could be made to pay for this, and that this payment would not be covered by his insurance. And that this may in some part make good for the actual injury of unjustly blaming me for what happened. That found its way to the macbook air that I now use. Still 0/10 would not recommend.

Said witnesses were also very willing to fetch his tires late at night and sell them but luckily it never got to that. The idea pleased me for a bit, though.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.