Jabref: Performance Improvements Groups tree

[simonharrer]

Follow up of #1385 so that we only release when this problem is fixed

[AEgit]

I would like to follow up on this: From master build 3.4 (18-05-2016) to 3.4 (20-05-2016) the performance decreased substantially for large databases (>6000 entries; >900 groups). Now, editing an entry/and or saving it takes a couple of seconds (beforehands it was nearly instantaneous): see also https://github.com/JabRef/jabref/issues/1332

Is there a way for me to get the old binary 3.4 (18-05-2016) (unfortunately I've already deleted it).

[AEgit]

Nevermind - found an older build.

The problem, however - of course - persists for the new version.

[simonharrer]

Yes, we are aware of this issue. This currently blocks the 3.4 release.

[simonharrer]

@tobiasdiez any idea how to fix this? This is basically the most critical blocker for the 3.4 release targeted tomorrow.

[tobiasdiez]

There are essentially two problems:

• revalidateGroups method in GroupsSelector gets called after every change, which results in problems for batch edits. Solution: wait a few miliseconds and only redraw the group tree if no further revalidate request comes in
• revalidateGroups blocks the main thread, so that editing an entry results in waiting time. Solution: make revalidation async

But I have no idea how to implement a solution. These Java workers and Swing threads are too cryptic for me.

[simonharrer]

The best way would be to use RxJava to implement this.

The quickest way is to implement something like this by hand through a scheduler somehow.

[simonharrer]

Could you try out the current master build whether this fixes your issue? In my tests, it worked correctly.

[tobiasdiez]

Thanks Simon!

[AEgit]

Hmmm, I think this still hasn't be solved properly. With the new version 3.4 the performance is definitively better than build 3.4 (20-05-2016), but it is worse compared to build 3.4 (18-05-2016). Now, everytime an entry is edited and saved, all groups appear to be rebuilt, which takes a couple of seconds. Can you confirm this behaviour?

[AEgit]

Could you reopen this issue (see above)?

[rvb-033]

I confirm the issue with V3.4 downloaded Saturday. I have a 7000 records database and 3 groups (publication, authors and keywords).

[tobiasdiez]

We tried to further improve the performance. Could you please try out the latest development version http://builds.jabref.org/master/? Thanks!

[AEgit]

I can confirm that the performance has significantly improved with the new development version:
JabRef_windows-x64_3_5dev--snapshot--2016-06-28--master--a24b544

Adding a new entry works (nearly?) immediately (there might (?) be a minimal delay of max. half a second, but this is totally acceptable).
I would say, this bug has been fixed now.

One thing to add: If "Show number of elements contained in each group" is activated, scrolling through the groups of my large database (>6000 entries; >900 groups) seems a bit slow (compare it to a videogame where the framerate is low). If it is not activated, you can smoothly scroll through the groups.
Similarly, if this setting is activated, adding an entry to another group (using drag'n'drop) has a small delay. If it is not active, it works immediately.
This, however, is another issue and definitely not a show stopper as the problem reported above. Maybe it is possible to improve this behaviour as well, but if not, it's fine I would say.

Thank you very much for this great new build!

[rvb-033]

Sorry about it, I could not use this version. The window is frozen, and
accepts one command per minute or so. I had a message "coud't connect to
the update server" (JabRef is running on a machine that is not connected
to the web).

Regards

Le 28/06/2016 15:42, Tobias Diez a écrit :

We tried to further improve the performance. Could you please try out
the latest development version http://builds.jabref.org/master/? Thanks!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/JabRef/jabref/issues/1425#issuecomment-229052248,
or mute the thread
https://github.com/notifications/unsubscribe/ATIp1Yr9EWbAzaVa_I588zGuU44-PwvAks5qQSSxgaJpZM4IiX3b.

[simonharrer]

@bartsch-dev this is caused by #1459 can you fix this, please?

[koppor]

@AEgit Did you find an old binary? Otherwise, here is one: https://4750-17634071-gh.circle-artifacts.com/0//home/ubuntu/jabref/build/releases/JabRef-3.4dev--snapshot--2016-05-17--master--db843c9.jar. On 2016-05-18, there was more then one build and some of them touched the group tree functionality. Therefore, I point you to an older development build.

[AEgit]

@koppor

Yep, found an old binary. But it's not a problem anymore (see my recent post). The current development version has adressed the problem.

[chriba]

@simonharrer this is very strange.
I could not reproduce the frozen window or any effect on the performance of JabRef with the current master branch.
Furthermore, when the message couldn't connect to the update server is displayed the updatecheck has already finished and should have no effect anymore.

@rvb-033
Can you please describe the exact steps you did when this problem occurred?
Additional Info like your PC-Setup and database would help to identify/reproduce this problem!

[koppor]

@rvb-033 I provide a recent build without the update checker at http://builds.jabref.org/no-update-checker/. Does this behave different than a snapshot of http://builds.jabref.org/master/?

[simonharrer]

@bartsch-dev did you turn off your internet connection for testing?

[chriba]

I did, I tested it with my PC (disabled adapter) and my Laptop (disabled wifi and no lan connection).

[Braunch]

@rvb-033 Can you add further descriptions of what you did when the problem occurred? @bartsch-dev is willing to fix this but he needs more information to reproduce it.

[tobiasdiez]

Please continue the discussion about the update functionality at https://github.com/JabRef/jabref/issues/1540. Since the performance issues seem to be (finally) fixed, I close this issue now.

[rvb-033]

I am at work now (lunch pause). I will try again this evening, say in 6
hours. Is it okay for you ?
Le 4 juil. 2016 11:55, "Braunch" [email protected] a écrit :

@rvb-033 https://github.com/rvb-033 Can you add further descriptions of
what you did when the problem occurred? @bartsch-dev
https://github.com/bartsch-dev is willing to fix this but he needs more
information to reproduce it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JabRef/jabref/issues/1425#issuecomment-230252426, or mute
the thread
https://github.com/notifications/unsubscribe/ATIp1d2inpyK27P2hcoYc-MRPzxW9aIVks5qSNh-gaJpZM4IiX3b
.

[matthiasgeiger]

Well, I'm not sure whether this is really caused by the updating mechanism. This might as well be just the last update of the status bar before JabRef freezes. @rvb-033 I would be good if you can try to show the contents of the "error console" (Help -> Show error console) if this is still possible. If not, you try starting JabRef via a terminal/bash using the command java -jar jabref-3.5dev.....jar to see potential errors in the terminal. Thanks!

[rvb-033]

Good evening,

Please find appended :

• current JabRef and java version I am running,
• console capture,
• 3 screen hard copies, one as soon as the software displayed something
  (note the could'nt connect to the update server - normal this machine is
  off-line), second while it was trying to update the groups list, third,
  groups updated and records list blank (I copied the console at this
  point in time).

It takes quite a while to change something in the window. During this
time, the software accepts no other command. Note that there is
something wrong with the groups count, but I think we can live with it
for the time being.

Keep me posted if I can help,

regards,

Hervé

Le 04/07/2016 14:05, Matthias Geiger a écrit :

Well, I'm not sure whether this is really caused by the updating
mechanism. This might as well be just the last update of the status
bar before JabRef freezes. @rvb-033 https://github.com/rvb-033 I
would be good if you can try to show the contents of the "error
console" (Help -> Show error console) if this is still possible. If
not, you try starting JabRef via a terminal/bash using the command
|java -jar jabref-3.5dev.....jar| to see potential errors in the
terminal. Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JabRef/jabref/issues/1425#issuecomment-230275473,
or mute the thread
https://github.com/notifications/unsubscribe/ATIp1e7ASapVnvAcMrUUxbrLcfAuiD9Jks5qSPcRgaJpZM4IiX3b.

19:59:04.478 [AWT-EventQueue-0] INFO net.sf.jabref.importer.OpenDatabaseAction - Opening: F:\Index\Index essai correction mots clés.bib
19:59:05.951 [AWT-EventQueue-0] WARN net.sf.jabref.logic.l10n.Localization - Warning: no message translation for "Check for updates" for locale fr
19:59:08.078 [SwingWorker-pool-3-thread-1] WARN net.sf.jabref.gui.worker.VersionWorker - Couldn't connect to the updateserver.
java.net.UnknownHostException: api.github.com
at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:1.8.0_73]
at java.net.PlainSocketImpl.connect(Unknown Source) ~[?:1.8.0_73]
at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:1.8.0_73]
at java.net.Socket.connect(Unknown Source) ~[?:1.8.0_73]
at sun.security.ssl.SSLSocketImpl.connect(Unknown Source) ~[?:1.8.0_73]
at sun.security.ssl.BaseSSLSocketImpl.connect(Unknown Source) ~[?:1.8.0_73]
at sun.net.NetworkClient.doConnect(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.https.HttpsClient.(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.https.HttpsClient.New(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) ~[?:1.8.0_73]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source) ~[?:1.8.0_73]
at net.sf.jabref.logic.util.Version.getLatestVersion(Version.java:72) ~[JabRef-3.5dev--snapshot--2016-06-28--master--8db7f86.jar:?]
at net.sf.jabref.gui.worker.VersionWorker.doInBackground(VersionWorker.java:52) [JabRef-3.5dev--snapshot--2016-06-28--master--8db7f86.jar:?]
at net.sf.jabref.gui.worker.VersionWorker.doInBackground(VersionWorker.java:33) [JabRef-3.5dev--snapshot--2016-06-28--master--8db7f86.jar:?]
at javax.swing.SwingWorker$1.call(Unknown Source) [?:1.8.0_73]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:1.8.0_73]
at javax.swing.SwingWorker.run(Unknown Source) [?:1.8.0_73]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_73]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_73]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_73]
20:01:06.496 [AWT-EventQueue-0] WARN net.sf.jabref.logic.l10n.Localization - Warning: no message translation for "Couldn't connect to the update server." for locale fr
20:01:06.496 [AWT-EventQueue-0] WARN net.sf.jabref.logic.l10n.Localization - Warning: no message translation for "Please try again later and/or check your network connection." for locale fr

JabRef 3.5dev--snapshot--2016-06-28--master--8db7f86
windows 10 10.0 amd64
Java 1.8.0_73

[matthiasgeiger]

Sorry but GitHub strips the images if you reply via mail. Can you please send them directly to me or upload them via mail. Thanks in advance!

I think it would also be helpful if you could send me your bib file for debugging - of course we will not publish it but only use it internally.

[chriba]

@rvb-033 Thanks for the log. At first glance I cannot see anything indicating a problem.
Have you tried the build without the version checker? Does the performance problem occur there too?
http://builds.jabref.org/no-update-checker/

[rvb-033]

I will test this version this evening (six hours from now).

Hervé

2016-07-05 21:36 GMT+02:00 bartsch-dev [email protected]:

@rvb-033 https://github.com/rvb-033 Thanks for the log. At first glance
I cannot see anything indicating a problem.
Have you tried the build without the version checker? Does the performance
problem occur there too?
http://builds.jabref.org/no-update-checker/

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JabRef/jabref/issues/1425#issuecomment-230579915, or mute
the thread
https://github.com/notifications/unsubscribe/ATIp1WMKP4of3WYwH1_pulikoZb6nRQWks5qSrJMgaJpZM4IiX3b
.

[matthiasgeiger]

Okay...
Hervé has sent me the screenshots stripped from his last reply and I am now able to reproduce his problems.
The issues are NOT caused by a missing web connection and the auto-updater but still are related to the groups tree.

@rvb-033 You said above that you have a database consisting of 7000 entries and 3 groups - but this was not really right :wink:
Yes, there are 7000 entries - but that is no problem.

You have an huge amount of "autocreated groups" for keywords and authors - i.e., technically we are not talking of 3 but of hundreds (thousands?) of groups. But still: Working with so much groups is not that problematic.

Moreover, you are using the option "Show number of entries contained in each group".

The combination of these three aspects causes the trouble: Upon opening the file JabRef has to count for each group how much entries are contained in the group. So basically what happens is: For each single group (of your hundreds of groups), each single entry (i.e., 7000 times) it is analyzed whether it is part of the group or not. And this requires quite a lot of time - even for modern PCs :wink:

I'm not sure whether we can really solve - or at least improve this on the JabRef side soon - so two potential work arounds that might help in your case are:

• deactivate the option "Show number of entries contained in each group", or
• remove the auto-generated groups for Authors, as these groups are causing the most search effort (perhaps the improved searching options using the always visible search bar are sufficient for you now?)

@tobiasdiez I had no time to check in the code, but would it be an option to calculate the _hits per group_ using the "underline groups for the selected entry" functionality? I.e., not searching for each group which entry is contained but check the group memberships for each entry. This should decrease the computation complexity for static groups (only one iteration of the whole DB, directly incrementing the counter for each static group defined in the group field). However, I fear this will not improve the situation for dynamic groups which are based on searching the DB (Searching _n_ times a DB of _x_ groups for _n_ dynamic groups - or evaluating _n_ searches for _x_ single entries... What is better here? :))

[rvb-033]

Good evening,

Please accept my deepest apologies for the wrong manipulation of your
software and for my wrong explanations. I hope I did not cause too much
trouble.
I tested the package without updater and it is difficult to tell whether
or not there is an improvement. I did not try to measure which version
it faster.
Removing the "Show number of entries contained in each group" really
makes the difference. This causes a real improvement, giving a very
usable version. Of course, I can live without this option. As a matter
of fact I do not remember when I selected it... Apologies again. I did
not test the version with the updater without the "Show number of
entries contained in each group".
Can I offer some translation of whatever you want in french to
compensate for the trouble ?

Regards,

Hervé

Le 06/07/2016 16:07, Matthias Geiger a écrit :

Okay...
Hervé has sent me the screenshots stripped from his last reply and I
am now able to reproduce his problems.
The issues are NOT caused by a missing web connection and the
auto-updater but still are related to the groups tree.

@rvb-033 https://github.com/rvb-033 You said above that you have a
database consisting of 7000 entries and 3 groups - but this was not
really right 😉
Yes, there are 7000 entries - but that is no problem.

You have an huge amount of "autocreated groups" for keywords and
authors - i.e., technically we are not talking of 3 but of hundreds
(thousands?) of groups. But still: Working with so much groups is not
that problematic.

Moreover, you are using the option "Show number of entries contained
in each group".

The combination of these three aspects causes the trouble: Upon
opening the file JabRef has to count for each group how much entries
are contained in the group. So basically what happens is: For each
single group (of your hundreds of groups), each single entry (i.e.,
7000 times) it is analyzed whether it is part of the group or not. And
this requires quite a lot of time - even for modern PCs 😉

I'm not sure whether we can really solve - or at least improve this on
the JabRef side soon - so two potential work arounds that might help
in your case are:

• deactivate the option "Show number of entries contained in each
  group", or
• remove the auto-generated groups for Authors, as these groups are
  causing the most search effort (perhaps the improved searching
  options using the always visible search bar are sufficient for you
  now?)

@tobiasdiez https://github.com/tobiasdiez I had no time to check in
the code, but would it be an option to calculate the /hits per group/
using the "underline groups for the selected entry" functionality?
I.e., not searching for each group which entry is contained but check
the group memberships for each entry. This should decrease the
computation complexity for static groups (only one iteration of the
whole DB, directly incrementing the counter for each static group
defined in the |group| field). However, I fear this will not improve
the situation for dynamic groups which are based on searching the DB
(Searching /n/ times a DB of /x/ groups for /n/ dynamic groups - or
evaluating /n/ searches for /x/ single entries... What is better here? :))

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JabRef/jabref/issues/1425#issuecomment-230782009,
or mute the thread
https://github.com/notifications/unsubscribe/ATIp1X9xTNQ1G38QY2jqyBvCU2pMmicIks5qS7aigaJpZM4IiX3b.

[matthiasgeiger]

No need to apologize. For non-developers it is often hard to understand what might go wrong, how to describe it and to decide what information is needed to reproduce the strange behavior.

We are glad that you can now use JabRef again without any trouble. However, as other users want to use the counting feature we'll try to come up with a solution for this.

Regarding your offer: With @mlep we already have an active contributor for the translation of the texts shown directly in JabRef. However, it would be great if you could check and (if necessary) update some of the French help pages - which can be found here http://help.jabref.org/fr/

Best regards
Matthias

[tobiasdiez]

With the recent improvements, the performance should now be on an acceptable level. For really big databases the option "show number of hits" makes some problems but this is to expect from the computational complexity of the feature.
Thus I close this issue for now.
If the performance problems should resurface, then please feel free to comment here again and we will reopen the issue.