Zettlr: Integration with Reference Managers (Zotero/JabRef)

[somelinguist]

It would be great to be able to open a file linked to a citation/reference (if there is one) when ctrl/cmd clicking on the citation in the editor or clicking on the reference in the attachment side bar. If it were a local file, it could open in the system viewer or a user-specified viewer. Some bibliographies also reference links to resources on the internet, which would need to be handled differently.

The links to local files should already be included in .bib files exported by Better BibTex for Zotero. The links are not included in CSL JSON format, but Better BibTex can be configured to include them in those files: https://github.com/retorquere/zotero-better-bibtex/issues/518#issuecomment-310522601

Something that would make even nicer, but more complicated, would be to open a PDF files to the page number referenced in the citation, but this would be complicated. Zotero has some logic to determine what the PDF viewer is and then pass the appropriate option for the page number. (See https://github.com/zotero/zotero/blob/3c1cdd57d2ad52cc9434b906a469579cb7b850a8/chrome/content/zotero/xpcom/openPDF.js#L26).

[nathanlesage]

To break down and understand what you want:

1. You have a file opened within Zettlr, and then click on the citation. It will start a search for all other notes that have this citation?
2. The same should be possible with the references in the right sidebar?
3. Additionally, it would be good to open the PDF at the specific page number, if you click on a citation?

Do I understand your request correctly?

[somelinguist]

Sorry for not being clear.

I was't thinking as much about searching for other notes that have the same citation, though that would also be incredibly useful.

1. Mostly I was thinking that if a citation had a link to a local PDF or file in the .bib* file, clicking on it (possibly with a modifier key) would open the PDF/file in the system viewer. Even just a link in the context menu would be really helpful. Something similar could be done if the .bib file had a link to a resource on the web.

2. The list of resources cited in the right side bar could also link to local PDFs/files or web resources in the same way.

3. It would be even better if citations in notes opened PDFs directly to the referenced page in the PDF. This is doable, but requires more logic than simply doing 1, because PDF programs differ in how the page argument is passed when executing the command or opening the url scheme. Zotero has some logic for doing this (referenced above) with several popular PDF programs on Mac, Windows, and Linux.

*.bib files often include the path to locally stored PDFs or files in the file attribute for an entry. CSL JSON files do not include this information, but Better BibTex for Zotero has an easy way to include it when exporting CSL JSON files (see the discussing referenced above).

The use case for this is mostly just easy access to referenced information, since the needed info is basically there in the citation.

[nathanlesage]

Ah, thank you for clarifying. Well, the thing is that Zettlr is meant for writing, not for citing. I do not want to implement any functionality that Zotero/JabRef/Mendeley/etc. already possess, both because one of these programs will be already installed, and because it would mean more work for effectively not adding anything. What I could offer instead would be — because, as far as I know, Zotero has some good callbacks that we could make use of — to open the respective item in Zotero. This way, the convenience would be that you don't have to search for the item, but with no added overhead.

[somelinguist]

That makes sense. It's definitely better not to duplicate functionality provided by other installed programs if it's not necessary.

I had originally explored using Zotero's url protocol to handle opening the PDFs, which would be ideal. Currently, there are three schemes provided:

zotero://open-pdf/library/items/[itemKey]?page=[page]
zotero://open-pdf/groups/[groupID]/items/[itemKey]?page=[page]
zotero://open-pdf/[libraryID]_[key]/[page]

Unfortunately, each of these schemes requires referencing an item id internal to Zotero rather than the citation key.

I just discovered that Better BibTex for Zotero does provide a scheme for opening and selecting the item Zotero given the citation key. zotero://select/items/@[citekey]. This by itself would still be very useful. I'm not sure if JabRef, Mendeley, etc. provide similar mechanisms.

[nathanlesage]

Yeah, this is what I meant! This would be good indeed. And I mean, installing BBT is already recommended in the docs, so we could just hint the users "For this functionality you need to have BBT installed" — after all, it's also Open Source and if you use Zotero either way, this should not pose any problem at all!

Concerning JabRef, Mendeley, etc., we'd need to do some research. I'll ping the guys from JabRef on twitter for that, if you'd like to, you could do some research on the other programs. Then the only additional thing we'd need is a small switch in the preferences so that users can indicate which of the programs they use to "unlock" that functionality!

[somelinguist]

Sounds good. I've opened a feature request for BBT to have a way to open the PDF with zotero://open-pdf/. The feedback so far makes it sound possible. They currently translate @citekey into the necessary Zotero id for the item, and might be able to provide another prefix that translates something like bbtpdf:citekey into the Zotero id needed for opening the PDF.

https://github.com/retorquere/zotero-better-bibtex/issues/1347

[somelinguist]

An initial look at Mendeley makes it look like they do everything with their web api, not through the the desktop program.

The desktop program registers a url protocol handler with Windows that just passes the arguments to the program, and it doesn't look like they haven any command line options to select items.

The web api requires registering an app at https://dev.mendeley.com/ It looks like one could search for documents, but I don't see any obvious way to open Mendeley Desktop to the desired document once it's found.

[nathanlesage]

This is what you get from a literature management software developed by one of the most greedy publishers globally 🙄 Then only JabRef and Zotero.

[somelinguist]

It looks like something might work with Bookends on Mac using some AppleScripts

If Bookends is configured to use BibTex, the following script will open Bookends to the item identified by CITATIONKEY:

tell application "Bookends"
    tell front library window
        set pubItem to id of first publication item whose user1 is "CITATIONKEY"
    end tell
end tell

tell application "System Events"
    do shell script "open \"bookends://sonnysoftware.com/" & pubItem & "\""
end tell

I'm not sure sure how to pass a value for CITATIONKEY, since my AppleScript skills are limited

The following script will open an attached PDF in the system viewer:

tell application "Bookends"
    tell front library window
        set attachment to attachments of first publication item whose user1 is "CITATIONKEY"
    end tell
end tell

tell application "System Events"
    do shell script "open \"$HOME/Documents/Bookends/Attachments/" & attachment & "\""
end tell

Unfortunately, PDF apps differ in the way to specify command line arguments, so it's not possible to handle opening to page numbers generically.

There are possible bugs with the script to open PDFs:

1) It assumes Bookends is using its default directory.
2) It assumes that there is one and only one attachment.

My AppleScript skills are rather limited, so I don't know how best to fix either bug. If there is more than one attachment, I understand that Bookends returns these separated by a newline character.

[nathanlesage]

This sounds like way too much work for one tool that also only works on one platform — I'd rather focus on cross-platform efforts … keeps the work small and the benefits great!

[somelinguist]

Sounds good to me. I typically only use Zotero, and sometimes JabRef, anyway. :)

[nathanlesage]

Sooo, just a short update: The issue on the JabRef tracker is created, and the team there is on it (kudos to them at this point <3), so we'll be having that API at some point.

Now to the feature here: We should now think strategically of what interactions between reference managers and Zettlr make sense, and which ones are essential (so that scalability is maintained once we implement this).

Therefore I also made the issue title resemble the broader issue we're working on here!

[somelinguist]

That sounds great. Here are some things I can think of:

From Zettlr to the reference manager

1. Choose an optional reference manager (Zotero or JabRef?) to be able to interact with its API. The easiest way to start with this would be to keep getting references from exported bib/json files as Zettlr is already doing and just use the API for getting information from the citation key.

   • If I understand JabRef correctly, the bib file IS the library, so we would know the name of the library in addition to the citation key in JabRef.
   • This is not the case in Zotero, where BBT allows you to export multiple bib/json files from a library or collection, with the option to keep those files updated. The name of the bib/json file does not (have to) correspond to the library or collection in Zotero. So we would only have the citation key in Zotero.

2. Open the reference from a selected citation in the linked reference manager to view or edit the reference details.

   • The UI for this could be implemented by a context menu item and/or, a keyboard shortcut.
   • BBT for Zotero has an API in place that would allow for this using the citation key.
   • It sounds like JabRef is willing to work on it.

3. In the Attachments sidebar on the right, offer the same or similar functionality as (2) in the References section

4. Directly open a file linked to the cited reference using an API provided by the reference manager. I would imagine in most cases, this would be a PDF. It would be nice to optionally open the file to the desired page number.

   • The UI for this could be implemented by a context menu item and/or, a keyboard shortcut.
   • Zotero supports this now if we had the internal ID (we don't), but it sounds like BBT is willing/able to add functionality to do this using a citation key.
   • Check with JabRef?
   • What to do if the citation is for a web page or Word document or something else?

5. In the autocomplete drop down for citations, include an option to open the reference manager to add a new item, which can later be inserted in the Zettlr document. This might be as simple as opening the reference manager, or if they have an API to allow it, opening it and creating a new blank item

From the reference manager to Zettlr

1. API to insert a citation at the cursor in Zettlr.

   • JabRef already has option to do this to other programs such as TeXStudio, Emacs, vim etc.. This is only really useful if Zettlr is already using the bib file that is open in JabRef. The inserted command can be specified in JabRef, but it probably works better with (La)TeX. Perhaps Zettlr could take care of adding the needed formatting (brackets and @).
   • I'm not sure if Zotero has similar functionality.

2. API to use Zettlr for notes for reference items https://github.com/JabRef/jabref/issues/5719#issuecomment-562982670

Where in the Zettlr code would these be implemented?

[nathanlesage]

Ahh, great ideas! I concur with most of the things you said, these make much sense.

First Zettlr --> Reference Manager:

1. This is basically what we've already agreed upon, so we can leave that as you've formulated it. I think the problem of database/citation key is not so much a problem we should deal with, because: If we provide JabRef with both the library AND the citeKey, we can be certain that this will address one unique item in a specified library. If the user only uses one library with JabRef, the Citekey might be enough -- but I'd say for this we can wait to see what the guys from JabRef implement; they know their code better :) And with Zotero, I don't think that it should be a problem, because you have to have BBT installed for that API to work, and BBT makes sure the citekeys are unique throughout the whole library. And even in case of collisions, we can defer the responsibility to make reasonable choices to the receiving program (which will get the non-unique citekey and then can decide upon an action)
2. I don't think a shortcut will work, because it'll be easier simply using the context menu and a small "Show Item in Zotero/JabRef"-Icon in the right sidebar to do this (activating a shortcut will leave Zettlr oblivious as to which of the citations in the currently opened file were meant)
3. Here I'd do the "Show item"-icon (similar to the "open directory" button at the top of that very sidebar, only one per reference item)
4. This point heavily depends upon the API. The easiest part for Zettlr would be to simply throw both citekey and an optional page number at the foreign API and let that one decide: If it's a PDF, and no page number is provided, simply open the PDF. If it's a PDF and a page number is provided, try to also navigate to the page directly. If it's not a PDF and a page number is provided, simply omit it. As to WHAT will be opened: In most cases this should be obvious, because for JabRef, there is one URL (or link?) property within the BibTex-library, which means it can simply open that one. And for Zotero, it's basically the "double click" action with an optional page-parameter. Double-clicking on a Zotero-item will open the PDF, if one is attached, the snapshot, if one has been made, or simply the URL-field content. But I think I'll better notify the JabRef team about that optional-page-omit-if-not-a-PDF-thingy we're talking about here!
5. This is a good idea, which would increase the similarity between Zettlr and the Word/LibreOffice-drop-ins even further. Is there an API to do so?

Now the reference manager --> Zettlr. All of the ideas that pertain to this require Zettlr itself to implement the callback API, which has been demanded since April in this issue here: https://github.com/Zettlr/Zettlr/issues/140

So we need that issue closed prior. I wanted to get to it in the next few months, so stay tuned!

[somelinguist]

That all sounds good. Thanks!

1. Re: a shortcut, would it be possible to have a shortcut that checked for a citation at the cursor/selection and try that? If not, no big deal.

2. The icon sounds good. In this section, would we want to support both opening the item in Zotero and another option to open the PDF (no page number needed here)?

3. The PDF page number is optional in Zotero, so it can handle either. If a page number is given and doesn't exist, it just opens to the first page. Reminder: we can't currently open a PDF in Zotero with the cite key (awaiting the issue linked above). Currently, if a cite key is given, it will open a random PDF (though the same one consistently).

4. I couldn't find an API to create a new item in Zotero, but zotero://select is an easy way to open/show the main Zotero window, where the user could do so.

[nathanlesage]

Mh … I don't think a shortcut makes sense at all. Simply perform the experiment to navigate to a citation with your cursor and then to press a shortcut. It's really unintuitive. So I'd suggest let's first implement that feature and if you _then_ see the need, we can discuss it again!

To the double-feature: My lazy inner self thinks "no, it's more work!", and my UX-self thinks "how could we probably sensibly implement something like that?"

It's good to hear that Zotero doesn't care also, so we can proceed on that front! Concerning the random-PDF thingy: I ask myself why you'd ever have more than one PDF for a single publication?

If there's no ability to create one, we could also ask them to implement such a thing, but we can take our time. After all, it will both take time to implement it for Zettlr, and also some time to implement it in JabRef, so no need to hurry!

[somelinguist]

That's fine about the shortcut. The context menu is definitely the first place I'd look.

[nathanlesage]

Just found a way to extract file link attachments from BibTeX-files, so I think we're closing in on the goal much sooner than I expected!

[nathanlesage]

So, here we go. What's now possible:

• If you have Zotero and the (not-yet published) right version of BetterBibTex installed, it can open your attachments.
• If you have a BibTex file with file-properties on the entries, Zettlr can also open one of these.
• As far as I understand, the file-properties are standardised, so it shoooouuuld work with all citation managers offering to export to BibTex files.

[somelinguist]

Sounds great. Thanks so much! I'm looking forward to trying it out.