Floccus: Not syncing & strange Tags

[lorddoumer]

Hi,

O'm trying to set up floccus to work, but just can't manage to...it doesn't sync anything (only creates an empty folder) but adds a lot of strange tags (floccus:>) to my bookmarks when viewed in Nextcloud. How can I get floccus to work or what is going wrong? Thanks in advance!

Software versions

• Browser: current FF and current Chrome
• Nextcloud: current version (13)
• Nextcloud Bookmarks app: current version
• Floccus: current version

Steps to reproduce

1. install floccus
2. add account
3. pres "snychronize"

Expected outcome

bookmarks are synced somehow, listed, displayed whatever...

Actual outcome

no bookmarks in browser but strange tags in bookmark-app in Nextcloud (seems to be grwoing all the time, some bookmarks have this tag multiple times)

[marcelklehr]

Hey @lorddoumer
I'm sorry you're having trouble setting up floccus. Thank you for reporting this.

The floccus tags are by design in order to synchronize the folder structure in your browser, but they shouldn't appear multiple times per bookmark. Could you

a) explain what you mean by "growing all the time"
b) lookup the exact versions of the required software
c) paste the debug logs as follows

In order to aid in debugging, please provide client-side logs.

Firefox

• Go to about:debugging
• enable debugging
• click on debug button next to floccus entry
• go to floccus options and make sure the account of your choice is connected
• trigger a sync by clicking on force Sync for the account of your choice.
• go back to the inspector window and copy the logs after waiting for the sync to finish

Chrome

• Go to chrome://extensions
• enable Developer mode
• click on dist/html/background.html button in floccus' entry next to "Inspect views: "
• go to floccus options and make sure the account of your choice is connected
• trigger a sync by clicking on force Sync for the account of your choice
• go back to the inspector window and copy the logs after waiting for the sync to finish

[lorddoumer]

With "growing" I mean the tag "floccus:>" (btw. Is it really meant to be called that? What's the purpose of this tag?) Has more occurrences than bookmarks (multiple one some) ... It started with about 50, then grew to 200, after 1h it was at 600...

Software version are all up-to-date: nextcloud 13, ff 59, floccus whatever is the latest I've installed today, chrome latest stable

Gonna check the logs tomorrow, thanks for your help :)

[lorddoumer]

So... here are the logs:
logs.txt

BTW: After clicking sync Chrome displays "error" next to the button, FF stays in this state forever (well, testet for about 2h):

And maybe also relevant: when clicking the "run test" button, I get this results:

Hope this helps :)

[lorddoumer]

sorry for spaming but in firefox something new happened: after syncing for 4h i now get the mesage "path segement is undefined" and when looking into my bookmarks (i mean the firefox bookmarks) there are multiple folders with [email protected] (one of them contianing a fraction of my bookmarks), one "[email protected]" with "foo" and "bar" subfolders. in the nextcloud bookmark app every bookmark now has one "floccus:>" tag so this seems to be OK now...

[lorddoumer]

And the last Update for now: I noticed whenever I try to use Floccus, I get logged out from my Nextcloud instance. Also, the Log is full of this error:

Error no app in context OC\ForbiddenException: This request is not allowed to access the filesystem
/PATH/nextcloud/lib/private/Files/View.php - line 1148: OC\Lockdown\Filesystem\NullStorage->mkdir('files_encryptio...')
/PATH/nextcloud/lib/private/Files/View.php - line 267: OC\Files\View->basicOperation('mkdir', '/USER/file...', Array)
/PATH/nextcloud/lib/private/Encryption/Keys/Storage.php - line 372: OC\Files\View->mkdir('/USER/file...')
/PATH/nextcloud/lib/private/Encryption/Keys/Storage.php - line 232: OC\Encryption\Keys\Storage->keySetPreparation('/USER/file...')
/PATH/nextcloud/lib/private/Encryption/Keys/Storage.php - line 117: OC\Encryption\Keys\Storage->setKey('/USER/file...', '-----BEGIN PUBL...')
/PATH/nextcloud/apps/encryption/lib/KeyManager.php - line 298: OC\Encryption\Keys\Storage->setUserKey('USER', 'publicKey', '-----BEGIN PUBL...', 'OC_DEFAULT_MODU...')
/PATH/nextcloud/apps/encryption/lib/KeyManager.php - line 257: OCA\Encryption\KeyManager->setPublicKey('USER', '-----BEGIN PUBL...')
/PATH/nextcloud/apps/encryption/lib/Users/Setup.php - line 77: OCA\Encryption\KeyManager->storeKeyPair(* sensitive parameters replaced )
/PATH/nextcloud/apps/encryption/lib/Hooks/UserHooks.php - line 180: OCA\Encryption\Users\Setup->setupUser( sensitive parameters replaced )
/PATH/nextcloud/lib/private/legacy/hook.php - line 106: OCA\Encryption\Hooks\UserHooks->login( sensitive parameters replaced )
/PATH/nextcloud/lib/private/Server.php - line 398: OC_Hook emit('OC_User', 'post_login', Array)
[internal function] OC\Server->OC{closure}( sensitive parameters replaced )
/PATH/nextcloud/lib/private/Hooks/EmitterTrait.php - line 99: call_user_func_array(Object(Closure), Array)
/PATH/nextcloud/lib/private/Hooks/PublicEmitter.php - line 36: OC\Hooks\BasicEmitter->emit('\OC\User', 'postLogin', Array)
/PATH/nextcloud/lib/private/User/Session.php - line 368: OC\Hooks\PublicEmitter->emit('\OC\User', 'postLogin', Array)
/PATH/nextcloud/lib/private/User/Session.php - line 605: OC\User\Session->completeLogin( sensitive parameters replaced )
/PATH/nextcloud/lib/private/User/Session.php - line 333: OC\User\Session->loginWithToken( sensitive parameters replaced )
/PATH/nextcloud/lib/private/User/Session.php - line 411: OC\User\Session->login( sensitive parameters replaced )
/PATH/nextcloud/lib/private/User/Session.php - line 525: OC\User\Session->logClientIn( sensitive parameters replaced *)
/PATH/nextcloud/lib/base.php - line 1052: OC\User\Session->tryBasicAuthLogin(Object(OC\AppFramework\Http\Request), Object(OC\Security\Bruteforce\Throttler))
/PATH/nextcloud/lib/base.php - line 987: OC handleLogin(Object(OC\AppFramework\Http\Request))
/PATH/nextcloud/index.php - line 37: OC handleRequest()
{main}

[marcelklehr]

a) "whatever is the latest I've installed today" -- I really need to make sure you have the correct version in order to rule out potential causes, that's why I'm asking for specfic versions. You can find the floccus version in about:addons in firefox, or chrome://extensions in chrome, and the nextcloud bookmarks version on the apps page in nextcloud.

b) The "[email protected]" folders stem from tests that failed and weren't cleaned up.

c) The firefox logs were not as helpful as I hoped, as they were not complete. How many bookmarks do you have on your server?

d) Do the same tests fail on both firefox and chrome? Can you click right on the tests panel in chrome after having run the tests, open the inspector and provide the debug logs for the tests?

(The floccus tags in nextcloud represent the folder path of the bookmark, i.e. floccus:>foo>bar>baz would mean the bookmark is in folder baz inside bar inside foo. floccus:> is the root folder, where bookmarks are placed by default, if they have no such tag initially)

[lorddoumer]

a) sorry, here are the exact versions: 2.0.1 in both browser (gonna try 2.0.2 now, wasn't available yesterday)
b) ok
c) the last two errors just repeate forever, nothing new (and no other stuff before). I have quite a lot of bookmarks (imported from FF) – should I delete the database and start the tests with just a few?
d) the test with 2.0.2 in Chrome doesn't throw an error (same errors on FF with 2.0.2 though), see results:
test-results.txt

(ah okay, thanks for clarification)

Noticed one more thing while editing the database: every time I try to sync the bruteforce-protection is triggered. gonna disable it for the next tests.

[marcelklehr]

Thanks. Good to hear that tests are passing in chrome. Could you try and provide logs for the tests as follows?

1. open about:addons, navigate to the floccus options and leave them open
2. open a separate tab with about:debugging and open the debugger for floccus
3. Go back to the floccus options and run the tests again
4. Now, switch to debugging the tests page by clicking the window icon in the top right corner of the debugger and selecting tests.html
5. Copy the logs and paste them here.

Edit: You seem to have set your nextcloud URL as the password in chrome, which is why it gives you an authentication error. Also, the many log messages in firefox don't appear to be error messages, but simply information about what is being done (tags are created for the bookmarks on your server). Apparently you have so many bookmarks that this takes a lot of time.

[lorddoumer]

sorry, I can't follow step 4 – which window icon? you don't mean the console-output produced by the debugger? btw: every time I run the tests and they fail I have to remove floccus from FF and reinstall because afterwards the popup won't open anymore (even after browser restart:

But I noticed something: when using a fresh installation with only < 10 bookmarks, everything works fine (havn't checked the logs though) but if I insert more there'll be errors when trying to sync (unfortunately it just says "5 errors" but not what exactly). As mentioned before this might be related to some security settings so I temporarly remove bruteforce-settings from NC and try again.

Yeah sorry, noticed the Chrome error too, fixed now.

[marcelklehr]

I mean the following button:

That's interesting. Regarding the errors, the debug console should have more information (if you could post that again, that would be great)

[marcelklehr]

Could it be that you entered a HTTP URL instead of a HTTPS URL in the account settings in floccus? What happens if you enter the https URL?

[lorddoumer]

hm, i can't select test.html with this button.
but I found out what the errors are, thanks to the console output:

Could not map contract ID '@mozilla.org/fxaccounts/push;1' to CID {1b7db999-2ecd-4abf-bb95-a726896798ca} because no implementation of the CID is registered. components.manifest:174
Starting sync process for account USER@https://URL.TLD/nextcloud/ Account.js:105:6
The same URL is bookmarked twice locally: "https://WEBSITE1" Account.js:157:10
The same URL is bookmarked twice locally: "https://WEBSITE2" Account.js:157:10
SERVERCREATE TocUunhcqHJV "WEBSITE3" Account.js:160:8
SERVERCREATE OsiXp8EXLVKc "WEBSITE4" Account.js:160:8
SERVERCREATE 7_Sr3XRu2v4K "WEBSITE5" Account.js:160:8
Response { type: "basic", url: "https://URL.TLD/nextcloud/index.…", redirected: false, status: 200, ok: true, statusText: "OK", headers: Headers, bodyUsed: false } Nextcloud.js:236:4
Response { type: "basic", url: "https://URL.TLD/nextcloud/index.…", redirected: false, status: 200, ok: true, statusText: "OK", headers: Headers, bodyUsed: false } Nextcloud.js:236:4
Response { type: "basic", url: "https://URL.TLD/nextcloud/index.…", redirected: false, status: 200, ok: true, statusText: "OK", headers: Headers, bodyUsed: false } Nextcloud.js:236:4
Syncing failed with Error: 2 errors
Stack-Trace:
MultiError@moz-extension://ff3f52ca-866b-4451-8926-416c95b10abe/dist/js/background-script.js:14:9
next/<@moz-extension://ff3f52ca-866b-4451-8926-416c95b10abe/dist/js/background-script.js:123:70
run@moz-extension://ff3f52ca-866b-4451-8926-416c95b10abe/dist/js/polyfill.js:4295:22
notify/<@moz-extension://ff3f52ca-866b-4451-8926-416c95b10abe/dist/js/polyfill.js:4308:30
flush@moz-extension://ff3f52ca-866b-4451-8926-416c95b10abe/dist/js/polyfill.js:1398:9

WEBSITE1 and WEBSITE2 are two bookmarks, which got duplicated somehow (haven't been before, that's for sure) - after removing them syncing is continued. I'm gonna add now folder after folder manually and see how it will work out.
It's HTTPS, just double checked – so in the end it seems to come down to two problem, both are not really realated to Floccus: Bruteforce protection is triggered when trying to sync (have to find a way around this, maybe it's only relevant on initial setup when syncing all the bookmarks) and the duplication problem, which might als be caused by FF or the NC app.

I'll report back but seems to be working now – thank you so much for your support and your patience!

[marcelklehr]

It also seems to be possible to get the logs from the tests when test.html is not selected. I'm very interested in those, as the tests still appear to be failing for you.

I'll see what I can do about the bruteforce protection kicking in :/

[lorddoumer]

hm, if test.html is not selected, it's just the "normal" console output?

I don't know what's causing the duplication-issue, but it happens between syncs: folders which have already been synced without errors suddenly throw an error because a bookmark within gets duplicated. without the dubugging-output I would be totally at a loss what causes the problem – is it really necessary to abort all sync if a duplicate is present? if so, maybe the error message (wothout debugging) could be a bit more specific to aid the user – that's just an improvement idea thoug ;)

[marcelklehr]

In my local Fx v59 the tests log to the same console as the other pages (a firefox bug, I suspect), which is why I recommended using the normal console to gather the logs produced by the tests. The tests test floccus, so the log messages may be similar to the normal floccus output. So, this would be the new guide:

1. open about:addons, navigate to the floccus options and leave them open
2. open a separate tab with about:debugging and open the debugger for floccus
3. Clear the logs by clicking the trash bin icon in the console
4. Go back to the floccus options and run the tests again
5. Copy the logs and paste them here.

[lorddoumer]

here you go:

logs-results.txt

unfortunately I have to reinstall floccus now again as described above :(
after running the tests no account-options are more available, maybe this information is also helpful:

[lorddoumer]

last message for tonight: after sync got stuck for 20min I restarted FF: sync-button is still grey and it says it's syncing but the log shows: Unchecked lastError value: Error: Could not establish connection. Receiving end does not exist.

[lorddoumer]

so, something new arise: after gradually adding my bookmarks (copy the folders in the FF bookmark manager to the sync-folder) the used memory by FF continues to grow – cpu-usage is between 20-40% and ram goes up to 5gb, then following error will be thrown:
Unhandled promise rejection out of memory polyfill.js:4326:11
uncaught exception: out of memory (unbekannt)
[...]
the same error hust gets repeated so I have to restard FF but the sync-button is greyed-out and floccus sys it's syncing but looking at the console it doesn't sync. after adding or removing some local bookmars it starts again though, so let's continue ;)

seems also be related to #65

[SomePoorBastard]

Do you have any other extensions on the browser enabled? Try disabling all but floccus and see if the same thing happens.

[marcelklehr]

It appears that the id of the sync root folder changes, or it cannot be found anymore. Do you have Firefox sync enabled?

[lorddoumer]

@SomePoorBastard only the nextcloud password app, disabld now but no changes.
@marcelklehr true, I was hoping to get the bookmarks to my Android FF installations, disabled now if this causes problems. is there any way to sync to mobile FF (native bookmarks, not an external app)?

so, after 2 days of work I do now have all bookmars (~600 bookmarks) synced with NC :)
the Chrome app works absolutely perfect! syncing all my bookmarks only takes 1-2 minutes (fresh sync) and afterwards much quicker, the FF app however is constantly syncing – as soon as it finished it starts again, always with high cpu and memory load. this renders firefox quite unusable :(
btw: I somehow can't open the debugging-window anymore (also for other extensions I tried to install), don't know why...

[billbillsen]

+1 i got the same problem....

[marcelklehr]

@lorddoumer do the tests pass in forefox when sync is disabled?

bad performance is a different issue if correctness isn't affected, see https://github.com/marcelklehr/floccus/issues/65

syncing to firefox for android is currently not possible, see #3 for that :/

[lorddoumer]

only one test fails:

ok, thanks for the info! gonna use Chrome on desktop for productive usage until performance is fixed and the bookmark-app on Android, maybe somewhere in the future it may be possible (as long as there is a workaround like the app it's fine). :)

the tagg issue is fine now, like stated earlier every bookmark has now only one tag as it's ment to be. only one tag (a floccus-path-tag) says in NC that there are more then 6000 occurrences, what is not true ;)

[marcelklehr]

I've just released v2.0.3 which should fix issues with random duplicates and dramatically increase performance on firefox. Please try it and report back. :)

[lorddoumer]

Awesome! FF 2.0.3 works now great! The initial sync is a bit slower then Chrome 2.0.2, but the incremental checks are super fast – no more constant cpu and ram load. Also thanks for the better error messages – very helpful to know, which url is duplicated ;)

Also tested Chrome 2.0.3 but can't invoke the account-popup to add my account (as instructed deleted my account and removed bookmarks prior to upgrade), maybe this consoel output from background.html helps?

Uncaught (in promise) TypeError: Cannot convert undefined or null to object
    at Function.keys (<anonymous>)
    at Function._callee17$ (Account.js:265)
    at tryCatch (polyfill.js:6900)
    at Generator.invoke [as _invoke] (polyfill.js:7138)
    at Generator.prototype.(anonymous function) [as next] (chrome-extension://bildjkcfnjfmcgnpfbhlpecpffidnbpm/dist/js/polyfill.js:6952:21)
    at step (Account.js:4)
    at Account.js:4

btw: can't get bookmarklets (like the one from the NC bookmark app) to sync – is there a limitation?

[marcelklehr]

Phew. Glad it works now. The shitty performance really had me worried...

Thanks a ton for the log snippet. Should be fixed in v2.0.4 (quite a release schedule over the last week :D)

Bookmarklets are sadly not allowed by the bookmarks app. We've had a discussion about it over here, if you'd like to chime in: https://github.com/nextcloud/bookmarks/issues/454

[lorddoumer]

quite a release schedule over the last week :D

yeah indeed, thanks a lot for your hard work, really appreciate it!

ok thanks for the information regarding bookmarklets – I hav eonly a few of them, so I'll just keep them seperate :)

[marcelklehr]

Thanks for not giving up on floccus, but being patient and helping with debugging :)

[t3mujin]

The bookmarklets is a real bummer, and not in your control, but at least you could try not to sync it to server. What's happening to me know is me is any bookmarklet I add gets removed after sync.

[marcelklehr]

@t3mujin Aah, that's a regression. I'll open a new issue for this. :/

[marcelklehr]

You can sync to XEBL/Webdav now, though, which should retain your bookmarklets.

[t3mujin]

Will do, didn't remember that one. Thanks