Jabref: Synchronize file links issues when .bib is in the same directory as the PDFs

[felker]

• [x] I have tested the latest master version from http://builds.jabref.org/master/ and the problem persists

JabRef version 4.1 on Mac OS X 10.13.2 x86_64 with Java 1.8.0_151

I am a recent adopter of JabRef after I finally had enough with Mendeley, which had become completely incapable of opening my library. So, I exported what I could to .bib and arduously cleaned up the entries in JabRef.

The final step in this process was to ensure that the database was in 1-1 correspondence with my Published/ directory which contained all the PDFs. I was hoping to synchroize the majority of the files automatically, and fix the names of the remaining unliked PDFs with names not matching the JabRef entry BibTeX keys However, running the file link tools in JabRef has not worked for me. In addition to failing to link the eligible files in v 4.1, JabRef will open the "Undefined file type" prompt for file type "PDF" repeatedly, even after "Define PDF" is completed.

Steps to reproduce:

1. Quality > Synchronize file links
2. Select "Automatically set file links. Do not overwrite existing links."
3. Select "Check existing file links"

Log File

Opening: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib Problem executing command java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib.sav.lock at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.jabref.JabRefExecutorService.executeAndWait(JabRefExecutorService.java:60) at org.jabref.gui.externalfiles.SynchronizeFileField.run(SynchronizeFileField.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at spin.Invocation.evaluate(Invocation.java:175) at spin.off.SpinOffEvaluator$1.run(SpinOffEvaluator.java:108) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib.sav.lock at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88) at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104) at java.util.Iterator.forEachRemaining(Iterator.java:115) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.jabref.logic.util.io.CiteKeyBasedFileFinder.findFilesByExtension(CiteKeyBasedFileFinder.java:95) at org.jabref.logic.util.io.CiteKeyBasedFileFinder.findAssociatedFiles(CiteKeyBasedFileFinder.java:49) at org.jabref.gui.externalfiles.AutoSetFileLinksUtil.findAssociatedNotLinkedFiles(AutoSetFileLinksUtil.java:51) at org.jabref.gui.externalfiles.AutoSetLinks.lambda$autoSetLinks$2(AutoSetLinks.java:89) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: java.nio.file.NoSuchFileException: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib.sav.lock at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372) at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:95) ... 17 more
Note, on the development version JabRef_macos_4_2-dev--snapshot--2018-01-10--master--5cbeb2baa.dmg, the Log File is simply:

Opening: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib
java.io.IOException: Problem in finding files
java.io.IOException: Problem in finding files
java.io.IOException: Problem in finding files
java.io.IOException: Problem in finding files
...

[LinusDietz]

Yeap, that was something we worked on quite recently. (see https://github.com/JabRef/jabref/pull/3623/commits/c93ae634b9065e5322859c9a397b335f0b53b561) I guess the problem is that I can't link all files accordingly and the exception handling (maybe only the displayed message) is wrong.

[felker]

I see. So is there a way to fix the "Define PDF" issue, independent of the exception handling?

[tobiasdiez]

Short version for @felker: The problem occurs because your bib file is in the same folder as the PDFs. If you (temporarily) move your bib file to a parent directory and set the PDF file directory in File > Database properties to the sub-directory, then the file link util should work as expected.

Short version for developers: Files.find and Files.walk suck.

Long version: The reason for the problem is that some files are changed between the creation of the stream (Files.find) and the access in collect. As of now the java API does not provide a way to catch and handle exceptions that occur while walking the tree. This is pretty suboptimal since IOExceptions can happen all the time for a variety of reasons (here it that the lock file is deleted, but it could also be security problems / ... ). There is an open feature request but this is inactive since 4 years.... On the bright side of things, there even was a patch proposed. So it should not be too hard to use this patch and create our own walk and find methods in FileUtil.

[felker]

Thanks for the reply @tobiasdiez . This fixed both the Log errors and the Define PDF issue.

However, I am not sure that the "Synchronize file links" or "Automatically set file links" abilities are working correctly. For example, I have an entry with bibtexkey 'AuerPaletou1994' and no linked file. There is matching file at /Users/kfelker/Documents/Research/References/Published/Radiation/AuerPaletou1994.pdf. This should be automatically linked by JabRef-- it does recursively search subfolders by default, right? However, the following steps:

1. Quality > Synchronize file links
2. Select "Automatically set file links. Allow overwriting existing links."
3. Select "Check existing file links"

fails to add the link, and the result says "Finished synchronizing file links. Entries changed: 336" which is my entire library, but most do not end up file links. I am guessing that is due to "Check existing file links"?

My relevant settings are:

• Preferences > File > External file links
  
  
  
  • Main file directory = /Users/kfelker/Documents/Research/References/Published
  
  
  • Autolink files with names starting with the BibTeX key
  
  

In addition, I produced a new Log error (but unfortunately I did not record the steps that caused it)

Opening: /Users/kfelker/Desktop/bib_database/jabref_db.bib
Uncaught exception occurred in Thread[JavaFX Application Thread,5,main]
java.lang.IndexOutOfBoundsException: [-2147483648, 9) is not a valid range within [0, 9)
    at org.reactfx.util.Lists.checkRange(Lists.java:128)
    at org.reactfx.util.Lists.checkRange(Lists.java:123)
    at org.reactfx.collection.MemoizationListImpl.forget(MemoizationList.java:184)
    at org.fxmisc.flowless.CellListManager.cropTo(CellListManager.java:82)
    at org.fxmisc.flowless.CellPositioner.cropTo(CellPositioner.java:29)
    at org.fxmisc.flowless.Navigator.cropToNeighborhoodOf(Navigator.java:227)
    at org.fxmisc.flowless.Navigator.placeStartAtMayCrop(Navigator.java:200)
    at org.fxmisc.flowless.Navigator.visit(Navigator.java:129)
    at org.fxmisc.flowless.StartOffStart.accept(TargetPosition.java:85)
    at org.fxmisc.flowless.Navigator.layoutChildren(Navigator.java:78)
    at javafx.scene.Parent.layout(Parent.java:1087)
    at org.fxmisc.flowless.VirtualFlow.layoutChildren(VirtualFlow.java:257)
    at javafx.scene.Parent.layout(Parent.java:1087)
    at javafx.scene.Parent.layout(Parent.java:1093)
    at javafx.scene.Parent.layout(Parent.java:1093)
    at javafx.scene.Parent.layout(Parent.java:1093)
    at javafx.scene.Parent.layout(Parent.java:1093)
    at javafx.scene.Scene.doLayoutPass(Scene.java:552)
    at javafx.scene.Scene.preferredSize(Scene.java:1646)
    at javafx.scene.Scene.impl_preferredSize(Scene.java:1720)
    at javafx.stage.Window$9.invalidated(Window.java:846)
    at javafx.beans.property.BooleanPropertyBase.markInvalid(BooleanPropertyBase.java:109)
    at javafx.beans.property.BooleanPropertyBase.set(BooleanPropertyBase.java:144)
    at javafx.stage.Window.setShowing(Window.java:922)
    at javafx.stage.Window.show(Window.java:937)
    at com.sun.javafx.stage.EmbeddedWindow.show(EmbeddedWindow.java:58)
    at javafx.embed.swing.JFXPanel.lambda$addNotify$49(JFXPanel.java:799)
    at com.sun.javafx.application.PlatformImpl.lambda$null$172(PlatformImpl.java:295)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.sun.javafx.application.PlatformImpl.lambda$runLater$173(PlatformImpl.java:294)
    at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)

[tobiasdiez]

Yes, this seems to be a bug with the "Check existing file links" option. You can safely uncheck this option or try out Quality > Automatically set file links.

@JabRef/developers can we just remove the "Synchronize file links" action? To me this looks like auto-find together with a file existence checker. For the first part, there exists already a menu item and the second part of the feature is covered by the integrity check.

[Siedlerchr]

👍 for removing the synchronize file links. Autostart file link Already covers all

[koppor]

The dialog renders as follows:

With "auto-find" you mean the following?

The "integrity check" is something like that. https://github.com/koppor/jabref/issues/282 ?

Did anyone check what "Check links" is doing? --> I would move that functionality to a second dialog and not show it on Mac OS X (make it a Windows feature only).

The following functionality is not in the check integrity thingy:

I would say that we improve our check integrity so that only file-based things can be executed. Otherwise, it takes too long for a database...

[tobiasdiez]

No "auto-link" refers the "Automatically set file links (F7)" option, which does exactly the same thing as the the first part of the sync-file links.

Yes, there is no equivalent integrity check for the last dialog. But what is this dialog even asking?! I really don't understand it.

[koppor]

"PDF" vs. mimetype pdf (application/pdf)

See https://github.com/JabRef/jabref/issues/98

That means, instead of "PDF" "application/pdf" (or similar) should be used. But that was an idea of JabRef 1.x or 2.x. Maybe, this check should be dropped... Maybe we should work on the format of the file field and remove the mimetype as currently, everything works well with the file extension only. - Or do I miss something in the context of compressed files?

[Siedlerchr]

Regarding the select files dialog, it will be reworked, see my wip pr

[felker]

@tobiasdiez is correct in what I was referring to as "auto-link". As indicated in #3629 , there seems to be a problem with "Automatically set file links (F7)", so I will wait to add the links to my database.

[koppor]

DevCall decision: Remove "check links" at a separate PR. - Then, we know, which code it is and what it did. This could be a start for integrating the functionality in "check integrity".

[koppor]

@tobiasdiez We should enable to have .bib in the same directory as .pdf files. E.g., a shared drive.

DevCall decision: Reimplement Files.walk, ... -- Maybe even imperatively (AKA "traditionally") to make the code more understandable.

[felker]

Repeating my comment in #3629 here in the hopes that it is useful for others:

The "Automatically set files link" functionality seems to work well on macOS High Sierra with JabRef 3.8.2 and JDK build 1.8.0_162-b12.

My current workflow for managing my BibTex database involves switching from JabRef 3.8.2 to "Automatically set file links", to JabRef 4.1 or 4.2 to add new entries via DOI since #2879 was never patched in 3.8.2.

[charryzzz]

@tobiasdiez I have the same problem using Jabref v 4.3.1 in ubuntu 16.04.

both the bib file and PDFs are in the “main file direction”

While it works without "Check existing file links" or using the version jabref v4.0

[Siedlerchr]

Synchronize File links was removed in favor of Automatically set file links in dev-5.0.
So I am closing this issue as no longer relevant

[PierreMarchand20]

Ok, but I would like to point out that "Automatically set file links" is also not working even with dev-5.0 on macOS.

Besides, I feel like the "Automatically set file links" feature checks every file when I just want to add another file and link it (meaning that all the other files have already their links). But I could be mistaking since it crashes when I try this feature anyway.

[Siedlerchr]

@PierreMarchand20 Could you please add some details on where it crashes and if you something in the log?

[PierreMarchand20]

Well, it just hangs, so there is no log to give unfortunately. I just open my database, take a random file where I delete the link. Then I click on Quality>Automatically set file links and it freezes

[PierreMarchand20]

Ok I tried with a new database with only one entry, and this time it does not hang (so that, I really feel like it is looking at all the files in my database where I just want to update the selected entry). But it actually does not find the file (and I checked the options and it should find it).