Zotero-better-bibtex: Feature Request: CAYW suggestions JSON API

[rriemann]

Hello,

I am working on a nice workflow for the Atom Editor and Pandoc conversions.

So far, the Atom plugin allows to launch the input element from Zotero.
https://atom.io/packages/zotero-citations

However, I would like to improve the integration and use instead the API of the plugin autocomplete-plus: https://github.com/atom/autocomplete-plus/wiki/Provider-API

Example:

To do so, I need to fetch suggestions that include already the formated content according to the chosen format for all search elements.

API could be for instance: http://localhost:23119/better-bibtex/cayw?format=pandoc&search=SEARCH_STR&limit=20

The result could be a json document with ≤ 20 elements in a list that contain each the pandoc formatted result and some context data to display as hint (e.g. the full citation text).

The hint is shown in the following example from https://atom.io/packages/autocomplete-latex-cite :

Best,\
Robert

[retorquere]

If you mean "search for (part of) citekey, return 20 matches in the chosen cayw format + formatted citation", that's doable. A little busy today, but this could be done tomorrow I think.

[rriemann]

Hey @retorquere, this would be a great christmas present! 🎁 I will already start working on the Atom plugin. 👍

However, I think it is important that the search does not only take into account the citekey, but kind of the same fields like the search in the search popup from cayw.

I thought one may potentially reuse the same code for the search from this popup.

We try to come up with custom translators for EU laws in PR https://github.com/zotero/translators/pull/1502 and it is super important that those very exotic fields such as legislativeBody are also considered. So I suggest to just consider all fields including the citekey. :)

[retorquere]

The popup isn't mine, and I have no influence on how it behaves and no knowledge of what it does internally; I just call a few things and it pops up, BBT is notified when it pops down, and then I can grab the results. What black magic happens in the interim I do not know.

It is possible for me to call into the existing search infrastructure, so you could get something that gives the same results as if you searched in the search box, but I'd have to choose whether that would search in "Title, Creator, Year" (pretty fast), "All fields and tags" (tolerable) or "Everything" (let's not do that).

[rriemann]

So I suggest to take "All fields and tags" or make it configurable via an API parameter (e.g. &mode=tags or &mode=basic). In our case, we will have often no creator and the title will be very long, e.g.:

• Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation)

In practice, will will have a tag with the common name "GDPR" (not part of the title) and use it to refer to the accordingly entry.

For the use in autocompletition, it would be nice if in any case also the citekey (the result of the auto-completition) is considered for the search.

[retorquere]

Looking at the gif you posted -- that shows a fully formatted citation, where the Zotero picker (which the regular CAYW uses) just shows author + title. There's a good reason for this;

• search + author/title: 0m0,269s
• search + zotero-formatted citation: 0m1,685s

I don't know how patient you are, but it looks like > 1s isn't going to be pleasant for users .

[retorquere]

any reason BTW to prefer this over doing autocompletion using an auto-exported bib file? Happy to make the change (it's done, really), but just curious.

[rriemann]

I agree, waiting for suggestions 1,7 seconds is too long!

I thought integrating this suggestions API upstream is more clean than doing it in atom. The alternative would be:

• someone types @ in atom and triggers auto-completiton
• atom fetches the entire database with many fields in json format and produces suggestions in atom.

I still think that fetching the database in atom is less clean. I may also run into the same speed problems.

So what do you suggest? One could
a) not compute the zotero-formatted citation
b) limit the number of entries to only few, e.g. 5, to keep the time small.

Anyway, the formatted zotero citation can only be a preview to help the selection process, because eventually the citations will be formatted using the configuration of the document (e.g. biblatex).

[rriemann]

On your second remark: I thought that the lookup in Zotero could consider more fields than what I can do based on an auto-exported biblatex file. However, maybe it is in fact easier to just ensure that biblatex exports some tags or note fields. This is important as the citation of laws are often only numbers and abbreviated institutions. So extra fields for context and common names are important.

If I use an auto-exported file, I have to watch the file for changes. The current https://atom.io/packages/autocomplete-latex-cite does this already.

[retorquere]

Atom doesn't need to fetch anything, perse. It could rely on a bibtex or csl auto export.

That 1.7sec was for 5 items btw. Title + author is plenty fast; the formatted citations look nice but not worth it I'd say. No idea how the stuff behind that gif does it - doesn't seem likely to me they're calling actual biblatex to do this.

[rriemann]

The author has stripped down a CSL-JSON formatter. He explains it in a comment here https://github.com/hesstobi/atom-autocomplete-latex-cite/blob/master/lib/lite-bibtex-parse.js#L4 :

'Lite' version nodified to output CSL-JSON compatible entries (e.g convert authors string to CSL authors structure) and to ignore comments and preambles Also only converts latex to utf8 for title and author fields, otherwise seems very slow.
Based on:
Original work by Henrik Muehe (c) 2010
CommonJS port by Mikola Lysenko 2013
Choice of compact (default) or pretty output from toBibtex: Nick Bailey, 2017.
Port to Browser lib by ORCID / RCPETERS

I have not understood yet if the search on many fields or the formatting for the preview is causing the significant performance drawback. If the formatting is the issue then I suggest:

• do the search in Zotero with the suggestion API and export unformatted raw CSL-JSON and use the file mentioned above to generate a preview. (I may in fact just propose a PR for atom-autocomplete-latex-cite to allow fetching suggestions via http)

Otherwise we may just do nothing and I make sure that the biblatex output file has all the information needed.

[retorquere]

I don't have to worry about bibtex parsing for this issue, but I'm curious what atom-autocomplete-latex-cite uses to generate the previews.

The problem is only the formatting. I've ran the code doing the same search with one just returning the title and one using the Zotero-internal citation processor, and that's the difference between 0.26 and 1.6 secs.

Returning CSL objects gets me down to 0.272 secs.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5170 ("json-rpc for #852").

[retorquere]

5170 has the search built in; you can call it using JSON-RPC like so using curl:

curl http://localhost:23119/better-bibtex/json-rpc -X POST -H "Content-Type: application/json" -H "Accept: application/json" --data-binary '{"jsonrpc": "2.0", "method": "item.resolve", "params": ["bugreports"] }'

(which would search for the term bugreports, although the method will change to item.search in the next build). You can learn more about JSON-RPC here. This functionality cannot go under the cayw endpoint because that would pop up the Zotero picker.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5171 ("change method to item.search").

[retorquere]

is this OK to merge?

[rriemann]

Please wait. I would like to test the integration with the Atom extension first and confirm only after some tests its usefulness. I'm optimistic, but have not tried yet.

[retorquere]

OK, but I prefer issues to not stay open too long as the code lines may diverge, leading to extra merge work later.

[retorquere]

Any progress? I don't like having longstanding open issues.

[rriemann]

I cannot download the test build 5171 from https://github.com/retorquere/zotero-better-bibtex/releases/download/builds/zotero-better-bibtex-5.0.59.5171.issue-852.xpi any longer. :(

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5272 ("Merge branch 'master' into 852").

[retorquere]

Builds are auto-expired because I don't want to deal with potentially divergent behavior between builds. 5272 is the latest version + the JSON-RPC extension.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5279 ("item.search").

[rriemann]

So I tested the last version and it seems to work well. :) 👍

Please merge!

Unfortunately, I had to learn the hard way to change my quick-copy citatation style to something different than BibLatex. I found it out by reading the source and the debug log.

For the record:

$ curl http://localhost:23119/better-bibtex/json-rpc -X POST -H "Content-Type: application/json" -H "Accept: application/json" --data-binary '{"jsonrpc": "2.0", "method": "item.search", "params": ["Helios"] }'
{"jsonrpc":"2.0","error":{"code":-32603,"message":"Internal error"},"id":null}-

And the Zotero log:

version => 5.0.33, platform => Linux x86_64, oscpu => Linux x86_64, locale => en-US, appName => Zotero, appVersion => 5.0.33, extensions => Scaffold (3.2.3, extension), Zotero LibreOffice Integration (5.0.8.SA.5.0.33, extension), Zotero Better Bib(La)Tex (5.0.66.5279.issue-852, extension)

(3)(+0018884): Starting full-text content processor

(5)(+0020645): POST /better-bibtex/json-rpc HTTP/1.1 Host: localhost:23119 User-Agent: curl/7.57.0 Content-Type: application/json Accept: application/json Content-Length: 66

(3)(+0000007): {better-bibtex} json-rpc: execute {"jsonrpc":"2.0","method":"item.search","params":["Helios"]}

(4)(+0000010): SELECT itemID FROM items WHERE itemID IN (SELECT itemID FROM items WHERE (itemID NOT IN (SELECT itemID FROM deletedItems) AND itemID NOT IN (SELECT itemID FROM itemNotes WHERE parentItemID IS NOT NULL AND parentItemID IN (SELECT itemID FROM deletedItems)) AND itemID NOT IN (SELECT itemID FROM itemAttachments WHERE parentItemID IS NOT NULL AND parentItemID IN (SELECT itemID FROM deletedItems))) AND (itemID NOT IN (SELECT itemID FROM itemNotes WHERE parentItemID IS NOT NULL) AND itemID NOT IN (SELECT itemID FROM itemAttachments WHERE parentItemID IS NOT NULL)) AND itemID NOT IN (SELECT itemID FROM items WHERE (itemTypeID IN (SELECT itemTypeID FROM itemTypesCombined WHERE typeName=?)))) AND ((itemID IN (SELECT itemID FROM itemData WHERE (fieldID IN (?,?,?,?) AND valueID IN (SELECT valueID FROM itemDataValues WHERE value LIKE ?))) OR itemID IN (SELECT itemID FROM itemData WHERE (fieldID IN (?,?,?,?,?,?,?,?,?,?) AND valueID IN (SELECT valueID FROM itemDataValues WHERE value LIKE ?))) OR itemID IN (SELECT itemID FROM itemData WHERE (fieldID=? AND valueID IN (SELECT valueID FROM itemDataValues WHERE value LIKE ?))) OR itemID IN (SELECT itemID FROM itemData WHERE (fieldID=? AND valueID IN (SELECT valueID FROM itemDataValues WHERE value LIKE ?))) OR itemID IN (SELECT itemID FROM itemData WHERE (fieldID IN (?,?,?,?) AND valueID IN (SELECT valueID FROM itemDataValues WHERE SUBSTR(value, 1, 4) LIKE ?))) OR itemID IN (SELECT itemID FROM itemCreators WHERE (creatorID IN (SELECT creatorID FROM creators WHERE TRIM(firstName || ' ' || lastName) LIKE ?))))) ['attachment', 110, 111, 112, 113, '%Helios%', 12, 115, 91, 107, 104, 119, 119, 114, 85, 86, '%Helios%', 116, '%Helios%', 44, '%Helios%', 14, 96, 52, 100, '%Helios%', '%Helios%']

(3)(+0000014): {better-bibtex} formatted-citations: export=b6e39b57-8942-4d11-8259-342c46ce395f {"mode":"export","contentType":"","id":"b6e39b57-8942-4d11-8259-342c46ce395f","locale":""}

(3)(+0000002): {better-bibtex} Error: JSON-RPC: <Exception: formatted-citations requires the Zotero default quick-copy format to be set to a citation style search@chrome://zotero-better-bibtex/content/BetterBibTeX.js:2700:19 >

(5)(+0000002): HTTP/1.0 200 OK X-Zotero-Version: 5.0.33 X-Zotero-Connector-API-Version: 2 Content-Type: application/json {"jsonrpc":"2.0","error":{"code":-32603,"message":"Internal error"},"id":null}

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5284 ("saner errors for json-rpc failurs").

[retorquere]

try 5284 for better error messages

[retorquere]

Wait, that requirement for the quick-copy setting can be dropped as the JSON-RPC no longer returns formatted citations.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5285 ("don't require quick-copy settings").

[retorquere]

merged and released

[lock[bot]]

This thread has been automatically locked because it has not had recent activity. Please open a new issue for related bugs and link to relevant comments in this thread.