Floccus: E022: Ordner #xxx[yyy] enthält angeblich ein nicht existierendes Lesezeichen

[tkurbad]

Describe the bug
Whenever I try to sync my bookmarks to or from my Nextcloud server, the error message in the title appears after a short while and sync stops. In a browser with empty bookmark set and "always overwrite client changes" ticked, no bookmark ever appears.

To Reproduce
Steps to reproduce the behavior:

1. Take one of my machines, sync bookmarks with floccus
2. Subsequently, all sync runs on this and all other machines fail with the above error

Expected behavior
All bookmarks should be synced to and from my Nextcloud.

Desktop (please complete the following information):

• OS: Gentoo Linux
• Browser Chromium
• Browser Version 77.0.3865.75
• Floccus version: 3.5.1
• Floccus sync method: nextcloud folders

Server (please complete the following information):

• OS: Gentoo Linux
• Nextcloud version: 17.0.0
• Bookmarks app version: 2.3.1

Note: Please write down the actual version numbers instead of writing 'latest'.

Debug log
It is often useful to provide a debug log file along with the issue. You can obtain a debug log of the most recent sync runs by clicking 'Debug log' in the bottom right corner of the floccus panel in your browser.

Note: As your debug log usually contains large parts of your bookmark data, instead of attaching it publicly to the issue here, I recommend uploading it privately at the following URL https://klehranlage.net/index.php/s/7nn82NSBsfQKdsJ

• [x ] Debug log provided

Additional context
I might add that this problem already occured in floccus 3.4.x with Nextcloud 16.0.x.

[marcelklehr]

Hey @tkurbad
Thanks for the succinct bug report :) So, initially sync succeeds? Can you try running the following command (can be on your desktop) and check if the result is a bookmark (JSON-encoded)?

curl -v -u 'username:password' https://yournextcloud.com/index.php/apps/bookmarks/public/rest/v2/bookmark/20805

[tkurbad]

Hi @marcelklehr,

yes, the initial - and only that first for a fresh server side folder - sync seems to succeed.

And yes, the curl line delivers a json encoded bookmark and "success" as result.

[marcelklehr]

So, the error message indicates an inconistent server state. Floccus first fetches the list of bookmarks and then checks which bookmarks belong to which folder. In this case, a folder said it contains a bookmark that wasn't fetched beforehand, but which, thanks to the curl line, we know exists on the server. The question now is, why isn't it included in the first response.

Could you deactivate the accounts on all clients but one and sync, so I can rule out a race condition? Also, is it always the same bookmark?

[tkurbad]

Could you deactivate the accounts on all clients but one and sync, so I can rule out a race condition? Also, is it always the same bookmark?

Yes, I will do that during the weekend.

And, after the initial run, it's always the same bookmark, but it's different after each initial sync...

[tkurbad]

Ok, results are in:

First complete sync of the one single client seemed okay as I chose "Overwrite changes from server".
I didn't delete the server folder beforehand.
Second sync of that same client failed with the above error (and that exact bookmark again)

Might the problem occur when the initial sync to the server is interrupted? I mean the very first sync to a particular "root" folder on the server. Just a guess...

[marcelklehr]

Could you upload a log after doing the initial sync and the second sync again, so I can see what's happening during that first sync?

[tkurbad]

Sorry, it took me a while, but I finally uploaded a new log for you.

I tried from scratch now:

1. Removed the target for the root folder from the server.
2. Disabled all but one client. The one I've chosen is a 64 Bit Chromium Version 79.0.3945.16 on Gentoo Linux. It's essentially the best used browser on my main machine. ;-)
3. Did an initial sync with floccus. [x] "Initial sync" and [x] "overwrite changes from server" both checked. -> All fine!
4. Changed "overwrite changes from server" to "merge server and client" -> Run sync -> E022, only this time with another bookmark.

Thanks again for looking into this!

[marcelklehr]

Heh, the log window is not large enough for the amount of bookmarks you have, so I still can't see the interesting part, but I suspect it might help to reset the server by means other than setting 'overwrite changes' in floccus.

[tkurbad]

Heh, the log window is not large enough for the amount of bookmarks you have, so I still can't see the interesting part, but I suspect it might help to reset the server by means other than setting 'overwrite changes' in floccus.

Oh, I did. As I've said, I completely deleted the target folder on the server before the initial sync. I guess what I could try is to delete the folder again and then try an initial sync with the default "merge" sync theme. Will do this in a minute and report back.

[tkurbad]

Okay, the error happens upon the 2nd sync run regardless of the initial sync strategy.

I now took a completely new server folder for the bookmarks root, ticked the "merge" sync strategy and the "Initial sync". First run works perfectly fine.

The second sync (without any changes to the bookmarks or floccus settings) again produces the error.

What else can I do to help resolve this issue? Would a confidential backup of my bookmarks be of any help?

[marcelklehr]

Then it would still be beneficial to look at the log of the first sync run, so:

1. Initial sync run
2. Download logs
3. Next sync run
4. download logs again
5. Upload both logs

[tkurbad]

Done!

[marcelklehr]

Thank you! I suspect that somehow the subfolder sync feature is the problem. Could you send me (privately if you want) the output of the curl command for the currently failing bookmark (18327)?

[tom-ch1]

I have the same issue, except that it does not even work for the first sync. Here's my setup:

• Nextcloud 16.0.5
• Bookmarks 2.3.1
• Initial and successive sync from Chromium (Linux) --> works without problems
• first-time sync on Chrome (Windows) fails with "E022: Ordner #339[Android 4.3 on S3 mini (i8190)] enhält angeblich ein nicht existierendes Lesezeichen: 2370"
• apache and nextcloud log show nothing out of the ordinary
• floccus log shows the above error message

the curl-command for the corresponding bookmark (2370) outputs:
{"item":{"id":2370,"url":"http:\/\/novafusion.pl\/s3-mini\/","title":"NovaFusion - New Android experience!","user_id":"xxx","description":"","public":0,"added":1573424473,"lastmodified":1574420841,"clickcount":0,"last_preview":0,"tags":[],"folders":["387","339","381"]},"status":"success"}

Some folders have been duplicated, but I don't know when that happened and which sync-job it was. This might have happened before I tried syncing from chrome(windows)

[tkurbad]

Thank you! I suspect that somehow the subfolder sync feature is the problem. Could you send me (privately if you want) the output of the curl command for the currently failing bookmark (18327)?

Here's the output:

{"item":{"id":18327,"url":"http:\/\/firefox-browser.de\/wiki\/Portable_Firefox","title":"Portable Firefox - FirefoxWiki","user_id":"xxx","description":"","public":0,"added":1551708340,"lastmodified":1574711185,"clickcount":0,"last_preview":1573742702,"tags":[],"folders":["4872","2018","2207","1917","1728"]},"status":"success"}

[tkurbad]

@marcelklehr is there anything else I can do?

[marcelklehr]

I'll let you know when I think of something, currently, my goal is to reproduce this.

[marcelklehr]

I've published a new patch release which might fix this (personal confidence level 13%). If you could retry with the new version and report back, I'd be grateful :)

[tkurbad]

Sorry @marcelklehr, your confidence level was about right. Nothing changed (except for the particular bookmark, floccus complains about). To reproduce this, would it help if I provided you with a dump of my bookmarks (or an excerpt thereof)?

[marcelklehr]

@tkurbad I think this might indeed be the best course of action. A complete export of the folder you are syncing would be good.

[tkurbad]

@marcelklehr, I uploaded a recent backup of my bookmarks (from 11/23/2019) to your share.

I keep most of my bookmarks within folders in the bookmark bar to have them available as a kind of menu.
I then sync the bookmarks root folder to a sub-folder on my nextcloud server.

I'm really curious if you can reproduce this now...

[marcelklehr]

It turns out I cannot, yet, reproduce this. I have Nextcloud 17.0.1, Bookmarks 2.3.3, Chromium 78, wiped the bookmarks installation, imported your dump into a folder in chromium and set up sync for that folder with a subfolder on the server. Works like a charm. I noticed I'm not syncing the browser root folder, if that's what you meant :thinking:

[tkurbad]

I noticed I'm not syncing the browser root folder, if that's what you meant

Well, that's exactly what I meant. ;-)

[marcelklehr]

Even when syncing the root folder I cannot reproduce the error, though.

[tkurbad]

Hmm, that's odd! Any suggestions on how to proceed?

[marcelklehr]

Could you try to follow my steps above and see if that helps reset things in a way? i.e. wipe the server bookmarks, remove the floccus account, recreate it and sync from scratch?

[tkurbad]

Okay, I'll close this for now. The issue seems to be resolved viewed from my primary client. Will see how the second one behaves tomorrow.

I did the following steps:

• removed floccus from browser
• removed all bookmarks from NC and deinstalled bookmarks app
• exported all bookmarks from browser
• deleted all bookmarks from browser
• reimported all bookmarks
• reinstalled bookmarks app in NC
• reinstalled floccus
• recreated account
• started inital sync -> Fine!
• forced successive syncs -> All fine!

[tkurbad]

Ok, I now tried firefox with an empty bookmark set and freshly installed floccus. The error re-occured there, but only in firefox, not back in Chromium where the bookmarks came from.

[tkurbad]

So, maybe that's your test case to reproduce @marcelklehr?

[tkurbad]

Ok, I now tried firefox with an empty bookmark set and freshly installed floccus. The error re-occured there, but only in firefox, not back in Chromium where the bookmarks came from.

Same on my second client with Chromium. Again E022.

[marcelklehr]

So, since I don't have a dev instance of chromium, I have one firefox for uploading and a second one for downloading and it works flawlessly :thinking:

[tkurbad]

That is really strange! When I get the chance, I'll try again with fresh user profiles in all connected browsers and report back.

[marcelklehr]

I do see something odd in your logs, though: Your floccus instance doesn't succeed in fetching a sparse tree from the server for some reason, forcing it to download data for the whole tree, which then for some reason is inconsistent, causing the error. I will try and mimick a timeout for sparse tree fetching to force a full-tree load shortly, let's see if that will reproduce the error.

[tkurbad]

Oh, that's interesting. Could this be due to a nginx configuration problem?

[marcelklehr]

Mh, it's probably not so much a misconfiguration as a load problem: Initially floccus tries to retrieve a hash for the whole bookmarks tree, which can take an exorbitant amount of time when there's lots of bookmarks, causing it to blow past the proxy timeout i.e. HTTP 504, which floccus currently interprets as a failure and continues as if that feature is not available.

There's multiple avenues to be pursued from this (from a developer perspective):

1. Caching!!!!1 The bookmarks app is in desperate need of a caching strategy for these hashes.
2. Refined handling of 504 statuses in floccus

In general, I've gleaned a lot of insights from working with your bookmarks set and observing how floccus behaves under stress. I did try to test this with an artificial data set, but nothing trumps the real thing. So thank you for that! :)

[tkurbad]

Caching!!!!1 The bookmarks app is in desperate need of a caching strategy for these hashes.

Hmm, I'm using a redis/memcache combination for my NextCloud instance. Will check if there's something that can be done about the proxy timeout.

In general, I've gleaned a lot of insights from working with your bookmarks set and observing how floccus behaves under stress. I did try to test this with an artificial data set, but nothing trumps the real thing. So thank you for that! :)

Thanks! But I'm amongst those who will benefit the most from sharing this, so thank YOU!

[marcelklehr]

Say hello to v3.5.3 which addresses some of the issues I've found.

[tkurbad]

Thank you very much!

v3.5.3 indeed solved the issue for me! Can this be propagated to the various extension stores?

[marcelklehr]

@tkurbad Yay! It should be in both the chrome webstore and on AMO, already. Are there any other stores I'm not aware of?

EDIT: The chrome update is still pending review.

[tkurbad]

Hmm @marcelklehr, seems as if the Chrome update is still in review...

[marcelklehr]

It's published now :tada:

[tkurbad]

Thanks! On a side note, floccus is running much more stable for me now. It used to crash within hours in Chromium, but doesn't anymore. Thank you very much for the ongoing improvements!!!

[marcelklehr]

@all-contributors Let's add @tkurbad for reporting bugs :blue_heart:

[allcontributors[bot]]

@marcelklehr

I've put up a pull request to add @tkurbad! :tada:

[tkurbad]

What an honor! Thank you! :-)