QOwnNotes 🚀 - Slow navigation between folders with @tag script

The @tag script has to parse every note (in memory) for tags. That might take a lot of time if you have a lot of notes. That's actually the price to pay if you have your tags in the note text and not in a performant sqlite database.

Switching note folders causes all notes to be parsed again by the script.

Writing a more performant tagging script could be one solution...

pbek on 16 Jun 2018

Maybe @Maboroshy can help. :)

pbek on 16 Jun 2018

Thank you for your reply.
So I think I will run the script once a day. Is there a simple way to have a button in the toolbar that parses the folders and syncs the tags with the database ? (not important, but it could be a workaround)

OlivierHobbes on 16 Jun 2018

You would need to write a different script for that.

On 16 Jun 2018 16:39, "iolivier87" notifications@github.com wrote:

Thank you for your reply.
So I think I will run the script once a day. Is there a simple way to have
a button in the toolbar that parses the folders and syncs the tags with the
database ? (not important, but it could be a workaround)

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/pbek/QOwnNotes/issues/943#issuecomment-397816660, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABtv1eB-lkyqX-gZSTZtQ8jhIgea_o0oks5t9RiegaJpZM4UqZkt
.

pbek on 16 Jun 2018

👍1

Will do.

Maboroshy on 17 Jun 2018

👍1

Ok, I've got the script.

Here it is

import QtQml 2.0
import QOwnNotesTypes 1.0

/**
* This script handles tagging in a note for tags in the note text like:
* @tag1 @tag2 @tag3
* @tag_one would tag the note with "tag one" tag.
*/

Script {
    property bool enableScan: true

    property string tagMarker
    property bool putToBeginning
    property string tagHighlightColor
    property bool manualScanOnly

    property variant settingsVariables: [
        {
            "identifier": "tagMarker",
            "name": "Tag word marker",
            "description": "A word that starts with this characters is recognized as tag",
            "type": "string",
            "default": "@",
        },
        {
            "identifier": "putToBeginning",
            "name": "Put tags to beginning of note rather than to end",
            "description": "If enabled tags added by UI will be put to the first line of note or right after top headline",
            "type": "boolean",
            "default": "false",
        },
        {
            "identifier": "tagHighlightColor",
            "name": "The color for tag highlighting in note preview",
            "description": "Put a <a href=\"https://www.w3.org/TR/SVG/types.html#ColorKeywords\">color name</a> or a <a href=\"http://doc.qt.io/qt-5/qcolor.html#setNamedColor\">supported</a> color code here. Leave empty to disable tag highlighting in preview.",
            "type": "string",
            "default": "purple",
        },
        {
            "identifier": "manualScanOnly",
            "name": "Disable updating tags on each note folder load",
            "description": "If enabled tags scan can only be started manually by a toolbar button.",
            "type": "boolean",
            "default": "false",
        },
    ]

    function init() {
        script.registerCustomAction("toggle_tag", "Force scan note folder for in-text tags", "Get tags", "tag");
    }

    function customActionInvoked(action) {
        script.log("ACTION START");

        if (action == "toggle_tag") {
            enableScan = true;
            mainWindow.buildNotesIndexAndLoadNoteDirectoryList(true, true);
            script.log("ACTION FINISH");
        }
    }

    // Stop scanning on opening and saving the note if auto scans are not allowed. 
    // There's no way to know scan for in-text tags is finished, so that's the best I came up with.
    function onNoteStored(note) {
        if (manualScanOnly == true) {
            enableScan = false;
        }
        script.log("SCAN IS " + enableScan)
    }

    function noteOpenedHook(note) {
        if (manualScanOnly == true) {
            enableScan = false;
        }
        enableScan = false
        script.log("SCAN IS " + enableScan)
    }


    /**
    * Handles note tagging for a note
    *
    * This function is called when tags are added to, removed from or renamed in
    * a note or the tags of a note should be listed
    *
    * @param note
    * @param action can be "add", "remove", "rename" or "list"
    * @param tagName tag name to be added, removed or renamed
    * @param newTagName tag name to be renamed to if action = "rename"
    * @return string or string-list (if action = "list")
    */
    function noteTaggingHook(note, action, tagName, newTagName) {

        var noteText = note.noteText;
        var tagRegExp = RegExp("\\B%1(?=($|\\s|\\b)) ?".arg(escapeRegExp(tagMarker + tagName).replace(/ /g, "_")));

        switch (action) {
            // adds the tag "tagName" to the note
            // the new note text has to be returned so that the note can be updated
            // returning an empty string indicates that nothing has to be changed
            case "add":
                // check if tag already exists
                if (noteText.search(tagRegExp) > 0) {
                    return "";
                }

                const tag = tagMarker + tagName.replace(/ /g, "_");

                // add the tag to the beginning or to the end of the note
                if (putToBeginning) {

                    // make an array of up to 3 first lines and other text as last item
                    var textLines = [];
                    for (var lineCount = 0, lineStart = 0, lineEnd = 0; lineCount != 3; lineCount++) {
                        lineEnd = noteText.indexOf("\n", lineStart + 1);

                        if (lineEnd == -1)
                            continue;

                        textLines.push(noteText.substring(lineStart, lineEnd));
                        lineStart = lineEnd;
                    }

                    textLines.push(noteText.substring(lineStart));

                    // if line after headline is a line for tags add tag there, 
                    // or make a new line for tags after headline
                    function appendTag(text, tag, prepend) {
                        if (text.substring(0, tagMarker.length) == tagMarker ||
                            text.substring(1, tagMarker.length + 1) == tagMarker)
                            return text + " " + tag;
                        else
                            return prepend + tag + "\n" + text;
                    }

                    // use different tag line number depending on a headline type
                    if (textLines[0].substring(0, 1) == "#")
                        textLines[1] = appendTag(textLines[1], tag, "\n");
                    else if (textLines[1].search(/=+/) != -1)
                        textLines[2] = appendTag(textLines[2], tag, "\n");
                    else
                        textLines[0] = appendTag(textLines[0], tag, "");

                    noteText = textLines.join("");
                }

                else 
                    noteText += "\n" + tag;

                return noteText;

            // removes the tag "tagName" from the note
            // the new note text has to be returned so that the note can be updated
            // returning an empty string indicates that nothing has to be changed
            case "remove":
                return noteText.replace(tagRegExp, "");

            // renames the tag "tagName" in the note to "newTagName"
            // the new note text has to be returned so that the note can be updated
            // returning an empty string indicates that nothing has to be changed
            case "rename":
                return noteText.replace(tagRegExp, tagMarker + newTagName.replace(/ /g, "_"));

            // returns a list of all tag names of the note
            case "list":

                // Don't scan notes if auto scan is not allowed and scan wasn't ran by toolbar button
                if (enableScan == false){
                    return note.tagNames();
                }

                script.log("RUNNING REGEX FOR " + note.name)

                var re = new RegExp("\\B%1([^\\s,;]+)".arg(escapeRegExp(tagMarker)), "gi"),
                    result, tagNameList = [];

                while ((result = re.exec(noteText)) !== null) {
                    tagName = result[1].replace(/_/g, " ");

                    // add the tag if it wasn't in the list
                    if (tagNameList.indexOf(tagName) == -1) {
                        tagNameList.push(tagName);
                    }
                }
                return tagNameList;
        }

        return "";
    }

    // Removes tag marker in note preview and highlights tag name with set color
    function noteToMarkdownHtmlHook(note, html) {
        if (tagHighlightColor == "")
            return;

        var re = new RegExp("\\B%1([^\\s,;]+)".arg(escapeRegExp(tagMarker)), "gi"), result;

        while ((result = re.exec(html)) !== null)
            html = html.replace(result[0], '<b><font color="%1">%2</font></b>'.arg(tagHighlightColor).arg(result[1]));

        return html;
    }

    // Escapes a string for regular expressions
    function escapeRegExp(str) {
        return str.replace(/[\-\[\]\/\{\}\(\)\*\+\?\.\\\^\$\|]/g, "\\$&");
    }

    /**
    * Hook to feed the autocompletion with tags if the current word starts with the tag marker
    */
    function autocompletionHook() {
        // get the current word plus non-word-characters before the word to also get the tag marker
        var word = script.noteTextEditCurrentWord(true);

        if (!word.startsWith(tagMarker)) {
            return [];
        }

        // cut the tag marker off of the string and do a substring search for tags
        var tags = script.searchTagsByName(word.substr(tagMarker.length));

        // convert tag names with spaces to in-text tags with "_", "tag one" to @tag_one 
        for (var i = 0; i < tags.length; i++) {
            tags[i] = tags[i].replace(/ /g, "_");
        }

        return tags;
    }
}

The idea of the script is to make an early return from list command of noteTaggingHook before it goes regexping through note text.

I made custom action that switches off early return, makes a force note folder refresh to get tag data and then switches on early return back, so casual folder refresh won't go through regexp. Since an empty return writes an empty tag data, I made early return output note tags from tag db.

But then... After force refresh makes noteTaggingHook go regexping through all notes and my custom action goes to "swich off" noteTaggingHook makes an unwanted second go with list. And, since early return is on, QON gets no actual tag data from the script. So I couldn't turn early return on right after force refresh and I didn't have any event to know when I can do it.

I made a dirty hack to resolve that - switching early return on at onNoteStored and noteOpenedHook. Script worked as specified and I got to testing.

A test on my 300 "production" notes showed no big difference between running casual refresh with early return from "list", before going through regexp, and custom action refresh that parses notes. Looks like noteTaggingHook's list brings delay by itself, no matter what list actually does.

Also I don't understand why noteTaggingHook makes two passes with list through all notes on refresh. The script has no loops, so it's somewhere in the app. Maybe there's a 50% optimisation laying around.

Summary: I wrote the script but it contains some hacks and doesn't really improve anything.

Maboroshy on 19 Jun 2018

I don't see any improvement either :-(
I have slowdowns even when "Disable updating tags on each note folder load" is checked.
Thank you for trying.

OlivierHobbes on 19 Jun 2018

Maybe @pbek can make a few tweaks to make it happen.

Maboroshy on 19 Jun 2018

What would happen if the "list" commands would be ignored. And there only would be a custom action (which can be called manually) that parses all notes for tags?
Or an external service that does the note parsing and "list" asks that service (in whatever way)...

pbek on 20 Jun 2018

The hook's list command can't be ignored. At least I haven't managed it to. If the hook is present, it will be triggered with list command for each note however script handles that command. If there's no proper output from the script, hook will tell the app that there are no tags.

There's no scripting function that could trigger hook's logic on purpose in a custom action. I can parse one note and tag it, but not all notes at once. So all I can do is to control the hook's behaviour.

External service is possible and even easier to implement. But I'm trying to stick to native scripting. The app is not suited for external changes to its tag db. And extra dependencies can scare Windows users.

Maboroshy on 20 Jun 2018

If there's no proper output from the script, hook will tell the app that there are no tags.

you could try to return the tags read from the database from the script api (if that doesn't result in an infinite loop)

There's no scripting function that could trigger hook's logic on purpose in a custom action.

it really would need to be a manual parsing of notes for tags of all notes in the note folder

pbek on 20 Jun 2018

you could try to return the tags read from the database from the script api (if that doesn't result in an infinite loop)

That's what my above script currently does on early return from the list command. For some reason that proved no noticable delay reduce over regular list with regexp.

Maboroshy on 20 Jun 2018

For some reason that proved no noticable delay reduce over regular list with regexp.

that's interesting, seems like even calling a QML method from C++ seems to be that "expensive", I'm not sure how to improve that since it's done by Qt

pbek on 20 Jun 2018

I have an illusion that hook trigger makes the app read all the note files again.

Also I clearly see that list runs two passes on each trigger. That doubles the delay.

You can try my script, I've put some debugging log messages.

Maboroshy on 20 Jun 2018

Every change in the note folder could trigger a reparsing of all notes via "list"...

pbek on 20 Jun 2018

But I do a manual refresh by app's command, why it makes two passes?

Also on each refresh app reloads all the files in the folder to RAM. So list should basically go regexping through RAM content, which is very cheap. Where's delay come from?

Maboroshy on 20 Jun 2018

Maybe the app sees the db file change event after the first run and that triggers the socond one... But by that logic it should start infinite loop.

Maboroshy on 20 Jun 2018

Hi,
I recently writed many @tags in my notes (no script activated), and I wanted to synchronize them with the database.

With the 0.0.8 script QownNotes chrashed immediately and couldn't restart. The hard disk were then used at 90% until I force the closing of the process. I had to rename my Notefolder to restart QON.

It's better with the script gived by @Maboroshy in this thread. QON can start (slowly) and I can update the database with the button in the toolbar. It freezes a few seconds, but it works.

OlivierHobbes on 2 Jul 2018

I did some extra testing by writing new @tags. When I activated the last script, QON crashed again. I couldn't restart it (and HDD was at 85%).
So, I renamed the notefolder, restarted QON, disabled the script, restarted QON with the notes...
Then I reactivated the script, and this time it didn't crash (but it was very slow).

Finally, I notice that the tags have been synced while I didn't click the button in the toolbar.

OlivierHobbes on 2 Jul 2018

I have re-written part of the script to not use regexes. It has less functionalities (~~no tag colouring in the preview,~~ tags can only be right below the title or at the end of the file). If someone wants to check whether that helps performance-wise, it's WIP here.

EDIT: added tag colouring back

bendem on 29 Jan 2019

👍2

Thanks for your work. I would like to test your script but I don't know how to install it :-/

I tried to import the local file "in-note-text-tagging.qml" with "info.json" and "tag-extraction.js" in the same folder but there are errors ("Expected a qualified name id")

OlivierHobbes on 29 Jan 2019

I have it working, but I can't say I notice much difference in performance, if any.
As I've previously posted, I don't consider regexping through strings in RAM a performance killer.

Maboroshy on 29 Jan 2019

As I've previously posted, I don't consider regexping through strings in RAM a performance killer.

The performance gain would be from parsing a maximum of 3 lines instead of the whole file.

The switch from regexes to parsing is so it's more stable and doesn't find tags in the middle of the text (like in URLs or code).

bendem on 29 Jan 2019

The performance gain would be from parsing a maximum of 3 lines instead of the whole file.

Yep, not a big deal with current hardware. Surely not the reason for a seconds of waiting I get on on folder refresh with tag parsing enabled.

The switch from regexes to parsing is so it's more stable and doesn't find tags in the middle of the text (like in URLs or code).

True. There's also "Epsilon Notes compatible tags (YAML tags)" that takes a similar approach.

Maboroshy on 29 Jan 2019

Surely not the reason for a seconds of waiting I get on on folder refresh with tag parsing enabled.

I'll have to dive into Qt to profile what's slow, that's going to take more time, but since I'm dogfeeding it, it'll keep me motivated to solve it. I'll finish polishing the script first so that it can be proposed for official inclusion. Then it'll be my next step.

bendem on 30 Jan 2019

👍1

@iolivier87 Can you please test and report if you see any differences in speed?

Waqar144 on 10 Dec 2019

I don't use tags anymore at all because I noticed that QON is faster without any tags (and because the search engine is pretty good, so I only write "@tags" in plain text and look for them now).

Anymay, I tested the tags efficiency, with QON 19.12.5 on a powerful computer (i5 9400F, 16GB RAM, and SSD) :
Without script, it's a bit slower with a note folder that contains tags but it is acceptable.
With the scripts from @Maboroshy (0.0.8 on repository and the one above), it's still very slow.

OlivierHobbes on 10 Dec 2019

Without script, it's a bit slower with a note folder that contains tags but it is acceptable.

Do you mean the search is slow?

With the scripts from @Maboroshy (0.0.8 on repository and the one above), it's still very slow.

Yes, and regex is one reason for the slowness. I have some ideas, I will try to optimize this one.

Waqar144 on 10 Dec 2019

🎉1

Without script, it's a bit slower with a note folder that contains tags but it is acceptable.

Do you mean the search is slow?

If my notes contain tags, browsing between sub-folders is a little slower than if I don't have tags at all.

OlivierHobbes on 10 Dec 2019

Yes, and regex is one reason for the slowness. I have some ideas, I will try to optimize this one.

You can take a look at the changed I made, it works fine without the regexes, I just had to fix the tag actions which I never got around to doing

bendem on 10 Dec 2019

If my notes contain tags, browsing between sub-folders is a little slower than if I don't have tags at all.

Can I ask how much slower? A second or even more than that?
There should be no lag ideally.

Can I ask how much tags you have?
I need to replicate the environment you have to reproduce it on my side and then optimize it.

You can take a look at the changed I made, it works fine without the regexes, I just had to fix the tag actions which I never got around to doing

@bendem
Thanks. I already did. I have to test it yet.
I have something else in mind, the search should happen in c++.

Waqar144 on 10 Dec 2019

Can I ask how much slower? A second or even more than that?
There should be no lag ideally.

It's about 1 second to display notes when I change of folder with tags (but without script).
Without tags, it's instant. If I search for characters in all folders at once it's instant too.

Can I ask how much tags you have?

500 notes in numerous sub-folders. All notes have several tags : from 2 to 10, the average is 4. So, 2000 tags in all.
There are about 80 different subtags organized in groups of tags.

OlivierHobbes on 10 Dec 2019

@iolivier87
Some changes have been made to improve the performance. Can you test and report if you feel any difference?

Waqar144 on 11 Dec 2019

Yes it's faster now (without script).
Still not as quick as without tags, but only 0.3 second now. So, good job !

OlivierHobbes on 11 Dec 2019

Yes it's faster now (without script).
Still not as quick as without tags, but only 0.3 second now. So, good job !

Thank you for your response. (It should be somewhat faster with script too.)

Some more changes are coming. It will be even more faster.

Waqar144 on 11 Dec 2019

👍1

I've changed my hardware after the last take on this. 6 cores, 16 GB RAM and very fast SSD and I still got 0.3-0.5 second on 19.12.5 to open a folder with "@tagging" script from the repository and almost no tags. Looks like hardware means little for the issue.

Maboroshy on 11 Dec 2019

Well, of course the script is going to be slow. It's looking for tags in full note text using regex. If you have a lot of notes, this is going to be even more slow.

Waqar144 on 11 Dec 2019

I can't agree with you.

As I've posted earlier I consider regexping through string in RAM as nothing for current hardware. I've opened https://regex101.com/, put the regexp from my script in JS mode and pasted the largest of my notes, which is 700 KB, - 0 ms. I had to paste it 5 times to get something more than 0, and that is about 3 MB. My note folder is 13 MB of plain text. I think QON could render 13 MB of markdown to html under 1 second.

Anyway, why does it go regexping through folder when I switch sub-folder? I haven't changed any note and even if I did - only one note should be rescanned, and that is 0 ms.

Maboroshy on 11 Dec 2019

I've opened https://regex101.com/, put the regexp from my script in JS mode and pasted the largest of my notes, which is 700 KB, - 0 ms.

That wouldn't apply here unless they are using Qt Regular expressions behind the scenes. The implementation of Regex mattes, the scenario for the search matters. That website has only one task when you give it text, meanwhile QON has a lot of other things to take care of besides the filtering. The recent optimizations that I did in the highlighter were mostly about removing regex and reducing calls to functions.

I think QON could render 13 MB of markdown to html under 1 second.

I haven't measured so, I can't say for sure but I highly doubt that.(on my pc). I can tell you that rendering markdown(in the editor) is definitely going to be a lot more than one second. The note that I use for checking speed is about 650KB (16K lines). It takes about 350ms, with spell checking enabled, to load it. With auto detection it takes about a second.

Anyway, why does it go regexping through folder when I switch sub-folder? I haven't changed any note and even if I did - only one note should be rescanned, and that is 0 ms.

Because whenever you change a folder, tag tree is reloaded and the tag script is called again.

Waqar144 on 12 Dec 2019

It's not rendering markdown that takes that 350 ms. It's highlighter and spell checking. I've disabled those and my 700 KB note draws up almost instantly. I even pasted my 700 KB note 20 more times in there. Not sure if it's drawn under 1 second but it's under 1.5 for sure. QON is really fast in that.

Also I found out that highlighter and spell checking keep working even if there's no editor pane opened! Free performance there.

I've posted about changing a sub-folder. Why the app scans the whole folder on a sub-folder change? For now changing a sub-folder is a stable 0.5 of time to change a folder. That is because changing a folder draws "All notes" and jumps to last selected sub-folder, basically 2 sub-folder jumps. Why is loading a folder and sub-folder takes equal time? Should it? What is the gain?

I've created a folder with 2 sub-folders with an empty note in each. Then I switched between them toggling the tagging script. There's a noticeable difference. But where does it come from if there's no text?

Maboroshy on 12 Dec 2019

It's not rendering markdown that takes that 350 ms. It's highlighter and spell checking.

When I said rendering I meant exactly that (not md to html conversion)

I've disabled those and my 700 KB note draws up almost instantly. I even pasted my 700 KB note 20 more times in there. Not sure if it's drawn under 1 second but it's under 1.5 for sure. QON is really fast in that.

If you disable those, it becomes just a simple QPlainTextEdit so, of course its faster.

Do you get any lag with 700K note?

Also I found out that highlighter and spell checking keep working even if there's no editor pane opened! Free performance there.

The highlighter and spellchecker work only if there's some text. If there's no text, they don't do anything. However if the widget is hidden, there's no simple way to stop the highlighter(spellchecker is a part of it) but it can be done I think.

I've posted about changing a sub-folder. Why the app scans the whole folder on a sub-folder change?

It doesn't. Only the selected subfolders(and their child subfolders, if subfolder option is enabled) are scanned for reloading the tag tree. @pbek can correct me if i am wrong.

For now changing a sub-folder is a stable 0.5 of time to change a folder. That is because changing a folder draws "All notes" and jumps to last selected sub-folder, basically 2 sub-folder jumps. Why is loading a folder and sub-folder takes equal time? Should it? What is the gain?

I don't really know that part of the code very well yet, so I cant say for sure. However, changing a note folder vs subfolder are two different things from the point of code. So they can't be compared like that I think

I've created a folder with 2 sub-folders with an empty note in each. Then I switched between them toggling the tagging script. There's a noticeable difference. But where does it come from if there's no text?

From the script that's for sure. :laughing:
Other than that, I will have to analyze the source code of the script to see why its slow.

Waqar144 on 12 Dec 2019

https://github.com/pbek/QOwnNotes/blob/3685ebc80cfe7e1325b9ed60d932a8d0d47acd35/src/mainwindow.cpp#L9819 does
https://github.com/pbek/QOwnNotes/blob/3685ebc80cfe7e1325b9ed60d932a8d0d47acd35/src/mainwindow.cpp#L9854 which does
https://github.com/pbek/QOwnNotes/blob/3685ebc80cfe7e1325b9ed60d932a8d0d47acd35/src/mainwindow.cpp#L7346 which takes all the notes
https://github.com/pbek/QOwnNotes/blob/3685ebc80cfe7e1325b9ed60d932a8d0d47acd35/src/mainwindow.cpp#L7950 https://github.com/pbek/QOwnNotes/blob/3685ebc80cfe7e1325b9ed60d932a8d0d47acd35/src/mainwindow.cpp#L7951 and runs a script hook on them
https://github.com/pbek/QOwnNotes/blob/3685ebc80cfe7e1325b9ed60d932a8d0d47acd35/src/mainwindow.cpp#L7953 which sends list action to all the scripts
https://github.com/pbek/QOwnNotes/blob/3685ebc80cfe7e1325b9ed60d932a8d0d47acd35/src/services/scriptingservice.cpp#L388
and the script goes regexping through each note.

Maboroshy on 12 Dec 2019

👍1

Ah yes, sorry, I was answering from the non script point of view 😬
Been working a lot on it lately so it's stuck in my head 😅

The number of notes can be reduced I think, though I am not sure what side effects that can have before testing. You know the script and the scripting service backend. Seems like a good opportunity for you to test and improve this. :wink:

Waqar144 on 13 Dec 2019

Hey, I'm just a random guy who've learned some QML to write scripts and sounds smarter than he is. I don't know C++ and it's an overkill language for a non-IT guy like me. And 11k line mainwindow.cpp doesn't look like the perfect first project.

It's not only about scripting. Looking into MainWindow::reloadTagTree I can't understand why there's no caching. I think this function is evil that eats my CPU time. There are quite enough calls for it. It goes through all sub-folders, fetches all the notes and rebuilds the tag tree from a scratch. We may not even touch a thing and it still throws out all the previous CPU work and does it again. This dates back to 2016 when the trees were tall and QON only started its path from light markdown notepad to a note management monster it's now. Back then we didn't throw 10+ MB note folders to it.

Maboroshy on 13 Dec 2019

Ah, sorry I didn't know that. C++ is definitely overkill monster language 😅
Life in Qt is much more simpler though.

I can't understand why there's no caching.

It's the way QON works..., everything is fetched from the database, every time. You will be surprised to know that it isn't always slow, only when we have to go through the whole table using * mostly.
Introducing caching here may not be very beneficial, and could potentially increase the number of bugs (due to stale data). And it would need some major amount of work just to be able to test it.

The biggest bottleneck in the function is eliminated already. (brokenTagNoteLink). The rest of the functions aren't as slow. The next slowest one is buildTagTree, I optimized it a little bit today so in the next release you will hopefully feel some difference in performance. (it's about two times faster than before according to my measurements)

And btw, note directory size doesn't matter here (without script). Only the number of notes matter. They can be empty. I am testing with empty notes (250 of them almost).

The script is slow because it has to look for tags inside the text. Full text. That means, if you have a big note, it's going to take much more time. Size matters in this case. One thing that could be done to make it faster is to have a defined criteria for tags inside notes so that the script looks for them in that place. Second thing that could be done is to just parse the text manually, which could be faster than regex. Thirdly, the tagging hook could be made more efficient, in the way that only the selected folders notes are parsed for tags. QML isn't as fast as c++, so some performance issues might still be there after everything.

I will however rethink these things, maybe there's a way to save time and get more performance out of it.

Anyways, if there are other slow things you have noticed be sure to let me know.

Waqar144 on 13 Dec 2019

I need to read something about DBs. I presumed DBs are always quite an overhead compared to storing data in object model. I'd store all the data in entity objects updating it atomically on altering so I could just read it from the object when I need.

For example, we need an untagged note counter. As of now QON goes brutal to get it: fetches all the notes, then fetches all the tags and queries a DB for each one. It doesn't use some dummy "Untagged" tag counter but calculates it like seeing the folder for the first time ever. And it does that all the time. How many performance goes to this single counter?

Why not count it once at the start and then make atomic +/- on a counter while tagging/untagging. We know what we tag. We see the external changes to notes. Yes there'd be partial rescan here and there, but on a major events, not each time we switch a sub-folder.

User created/tagged note - give him +1 on a counter and an extra item in a list. Don't recreate the whole list. He will get an instant response and praise the app.

If all that counters take much time we can give user a list without counters and update it when the counters are ready. The folder will open twice faster and the counters will take the same time.

have a defined criteria for tags inside notes

I wrote "Epsilon Notes compatible tags (YAML tags)" script with that in mind. It's little faster but it's not a game changer.

just parse the text manually

The current regex is only about 8 operations. I doubt it can be much faster, but it will be much less maintainable for sure.

QML isn't as fast as c++

Maybe we could pass the regexp definition string to C++ code and get back the results?

Maboroshy on 13 Dec 2019

DBs are always quite an overhead compared to storing data in object model.

Well, yeah. DBs are quite slower than storing data like that. But you need a persistent data storage and at some point all the data will go somewhere in file or something.
QON has two types of databases. In memory and in disk. In memory db is in the memory so its very fast. But when the db from the disk is queried, its quite slow. That's why queries like Note::fetchAll() are very fast.

For example, we need an untagged note counter. As of now QON goes brutal to get it: fetches all the notes, then fetches all the tags and queries a DB for each one.

All tag related queries go to the disk db. If it's just a single query it doesn't matter. But in a loop they can cause quite an overhead.

It doesn't use some dummy "Untagged" tag counter but calculates it like seeing the folder for the first time ever.

we can't have just one counter, every folder would need a counter. That counter would need to be updated everytime the tag gets new notes or if notes are removed from it. And it would need to be saved back to the database on application close(or maybe periodically). That is possible, theoretically. It would need sometime to implement however and a lot of care so that all the parts of code that depend on tagging stay in sync.

If all that counters take much time we can give user a list without counters and update it when the counters are ready. The folder will open twice faster and the counters will take the same time.

For that we need multithreading, and that would be very difficult,, i would say impossible. The reason is that in Qt, the db connection can't be accessed from a different thread.

I doubt it can be much faster

You wouldn't know until you try and measure.

but it will be much less maintainable for sure.

That's true, but once its done, it will be okay. I wrote the syntax highlighter without regex(it still lacks some things), but its fast. With regex it would be a lot less lines of code and probably much easier to understand for me, but then, it wouldn't be as fast.

Maybe we could pass the regexp definition string to C++ code and get back the results?

That's possible, though how much performance can be gained out of that is doubtful. But it will be faster.

The very best thing that could be done is to give the script less notes to scan. That would surely increase the performance a lot.

Waqar144 on 14 Dec 2019

All tag related queries go to the disk db.

That's where the evil is. We reduce app speed to a disk speed. We can at least have an in-RAM copy of the tag DB and dump the changes to in-disk one in another thread, without blocking the event loop.

Maboroshy on 14 Dec 2019

That's where the evil is. We reduce app speed to a disk speed. We can at least have an in-RAM copy of the tag DB and dump the changes to in-disk one in another thread, without blocking the event loop.

Sounds good. Except for the another thread part, that isn't possible I think. And we need to see what @pbek has to say about this.

Waqar144 on 14 Dec 2019

@iolivier87 Some changes were made to improve performance. Could you please test and report if you see any differences in performance?

Waqar144 on 14 Dec 2019

It seems a little faster in 19.12.9, but I don't have the appropriate tool to mesure it accurately (still between 0.2 and 0.4 second).

OlivierHobbes on 14 Dec 2019

On my potato computer (8GB RAM, SSD, but Core m3 6Y30, max speed is 2,2 GHz), It's about 0.9 to 1 second in 19.12.9.

With the task Manager, I can monitor the CPU load for future comparisons :
In idle : speed is at 0,7 GHz and utilization is 4%
Folder change without tag (instant display) : speed reaches 1,3 GHz and utilization is about 13%
Folder change with tags (1s lag) : speed jumps to 1,9 GHz and utilization is about 30%.

OlivierHobbes on 15 Dec 2019

Hi everyone!
I've just started using QOwnNotes and the discussion is quite long (I haven't read all messages) so my suggestions might be not relevant.
What if the script could be a frontend for the existing database tag system? It would work much faster. No need to parse all notes to find notes by tag. The script will be triggered when you

create a new note
modify an existing note
trigger tag sync manually for one note (when a note was modified with an external text editor)
trigger tag sync manually for all notes

Update:
It seems to work with database tags. But the tags update is not triggered on a file update for some reason

JekRock on 21 Mar 2020

Another suggestion is to add the support for multiword tags. It would require changing the tag word marker. For example, it could be something like @ multiword tag here@

JekRock on 21 Mar 2020

It seems to work with database tags. But the tags update is not triggered on a file update for some reason

Indeed, that's how it works currently. I meant to investigate more on this and PR my changes to the script, but I've had to take a very long hiatus from note taking.

I'll try and do that.

bendem on 21 Mar 2020

👍1

Multiword tags are supported. @multiword_tag in the text becomes multiword tag in the tag tree. This example is in the script's description.

Maboroshy on 22 Mar 2020

@Maboroshy I've missed that. Thanks!

JekRock on 22 Mar 2020

One more question about the tagging script. Could you please explain how it adds tags from notes to the database? I don't see any calls to the tagCurrentNote method. The script has only noteTaggingHook that works in the opposite direction (puts tags from UI to notes).
Also, it adds tags from a note to the database if you either restart the app or open "Scripting > Script settings > Press OK" only. Nothin happens if you saves a note.

JekRock on 22 Mar 2020

I will close this until there is more information.

pbek on 12 May 2020

20.8.3

there now is an option to disable the note count in the tags panel to improve the
performance on slower hardware (for #943, thank you @Waqar144)

pbek on 9 Aug 2020

There now is a new release, could you please test it and report if it works for you?

pbek on 9 Aug 2020

Yes, it's much more efficient without the tags counter.

Same computer (6Y30) and note folder as above with QON 20.8.3 :

Idle : CPU speed is at 0,63 GHz and utilization is about 4%
Folder change without tag (instant display) : speed reaches 0,9 GHz and utilization is about 10%
Folder change with tags and counter (lag is more than 1s) : speed jumps to 2,1 GHz and utilization is about 37% (even if the tags panel is not displayed !)
Folder change with tags and no counter (not instant display, but lag is much less than 1s) : speed only jumps to 1,25 GHz and utilization is only 16%.

Thank you @pbek

OlivierHobbes on 9 Aug 2020

👍1

thank @Waqar144 😁

pbek on 9 Aug 2020

20.9.11

more tagging system speed improvements were made (for #943, thank you @Waqar144)

20.9.10

the speed of the tagging system was slightly improved (for #943, thank you @Waqar144)

pbek on 24 Sep 2020

The new versions really fixed the in-text tags for me. The speed is only a bit slower than with default tagging. A huge improvement over my last experience with in-text tags. The only visible reminder of the previous issues is a sub-folder selection marker lag. Looks like it's tied up with tagged note counters update.

Maboroshy on 27 Sep 2020

👍2

Awesome. Thanks a lot to @Waqar144!

pbek on 27 Sep 2020

Thank you @Waqar144 for working on optimizations.
I just did some tests in 20.9.11:

No tag counter = same CPU utilization as 20.8.3 on my m3-6Y30. I don't know if it is quicker than before (it's noticeable but ok)
Tag counter enabled = same CPU utilization (39% at 2,17GHz), but it became extremely slow: 29s to load one folder :-(

OlivierHobbes on 27 Sep 2020

Thank you @Waqar144 for working on optimizations.
I just did some tests in 20.9.11:

* No tag counter =  same CPU utilization as 20.8.3 on my m3-6Y30. I don't know if it is quicker than before (it's noticeable but ok)

* Tag counter enabled = same CPU utilization (39% at 2,17GHz), but it became extremely slow: 29s to load one folder  :-(

That's weird, because most of the changes were made to optimize the counting, a lot of queries are much lighter now. Do you get the same lag for all folders?

Waqar144 on 28 Sep 2020

The duration varies a lot with the number of notes and tags in the selected folder.
503 notes and numerous tags (root folder) = 99s
181 notes and numerous tags = 32s
33 notes with numerous tags = 10s
43 notes with very few tags = 3s

Edit: If the tag counter is hidden, it's very fast, even for the root folder (less than 1s).

OlivierHobbes on 28 Sep 2020

Hmm. I am testing this on my machine with 980 notes and about 50 tags and the lag is very little (with count on). Without the count I don't get any lag at all.

@OlivierHobbes are you selecting a single folder or multiple?

I am not quite sure about what might be causing this :thinking:
@pbek can you review the changes and see if you notice something?

Waqar144 on 28 Sep 2020

I select the folders one by one, but there are subfolders at the same time.
But folder/subfolder doesn't seem important.
From the previous tests:
33 notes (in the same folder) with numerous tags (more than = 10s
43 notes (in 4 subfolders) with very few tags = 3s

OlivierHobbes on 28 Sep 2020

@pbek can you review the changes and see if you notice something?

@Waqar144 do you mean testing with the tag script? Is there a set of notes to benchmark with? (I don't use the script anywhere)

pbek on 28 Sep 2020

@pbek can you review the changes and see if you notice something?

@Waqar144 do you mean testing with the tag script? Is there a set of notes to benchmark with? (I don't use the script anywhere)

No, without the script. I mean if you can notice something odd in the code or if you are noticing any slowdowns.

@OlivierHobbes just to be sure, you are not using the script right?

Waqar144 on 28 Sep 2020

I've 658 notes with counting turned on in the tag panel and can't see any lag or anything...

pbek on 28 Sep 2020

@OlivierHobbes just to be sure, you are not using the script right?

Yes, it is without the tag script

OlivierHobbes on 28 Sep 2020

Maybe the note folder I'm using for tests has something broken. It is old data altered by the tag script. I don't use tags anymore since a long time because of these issues (and because the search engine is very efficient)

On my gaming computer and the same note folder, it is instant with hidden counter. But I have a very little lag when the counter is enabled.

OlivierHobbes on 28 Sep 2020

Oh yes, that was because of broken data. Sorry for your time.

I built tags from scratch with the script on another note folder. And in this one, the folders are loading very quickly (less than 1s) when the counter is enabled. (and sorry for multiple posts)

OlivierHobbes on 28 Sep 2020

Oh yes, that was because of broken data. Sorry for your time.

By any chance did you see what was broken? Maybe we can add something to the code that automatically takes care of that.

I built tags from scratch with the script on another note folder. And in this one, the folders are loading very quickly (less than 1s) when the counter is enabled. (and sorry for multiple posts)

Haha that's okay. I was really curious about the regressions anyways. I am glad we have optimized this enough to be usable for people who have large number of tags + notes.

Waqar144 on 29 Sep 2020

By any chance did you see what was broken? Maybe we can add something to the code that automatically takes care of that.

Don't care about that. There was errors with the tags. This note folder was altered by the multiple tag scripts from this thread, then encrypted with Cryptomator, and synced on multiple computers with One Drive. A lot of things could go wrong.
If i wanted to fix it I would just have to delete all the tags and run the tag script again.

OlivierHobbes on 29 Sep 2020

Qownnotes: Slow navigation between folders with @tag script

Expected behaviour

Actual behaviour

Steps to reproduce

1 : PC1 + Encrypted Note folder with Cryptomator : 20 seconds

2 : PC1 + No encryption : 11 seconds

3 : PC2 + Encrypted Note folder with Cryptomator : 7 seconds

4 : PC2 + No Encryption : 4 seconds (not instantaneous but quite usable)

5 : PC1 + Cryptomator + No script : 4 seconds

Comment

Most helpful comment

All 79 comments

20.8.3

20.9.11

20.9.10

Related issues