Cherrytree: Compile a list of features missing in the C++ version

[ForeverRainbow]

Hi,

I want to help with the porting of cherrytree to C++, however I am not sure what features are important to implement, are a priority, might be easier for someone new, etc. What I think would be really useful is to have a list of features still missing in the C++ version (possibly with the estimated level of difficulty of implementing them). I think this would definitely help any contributors who want to port things over but are unsure as to what to do.

---

List of missing features

Imports

Todo

• [ ] EssentialPIM HTML importing

Not planned

• [ ] KeyNote importing
• [ ] Knowit importing
• [ ] Basket importing
• [ ] TuxCards importing
• [ ] Treepad importing

Done

• [x] Gnote importing
• [x] KeepNote importing (#1002 but untested)
• [x] Zim importing (#853)
• [x] Tomboy importing
• [x] Mempad importing (#1011)
• [x] Leo file importing (#1019)
• [x] RedNotebook importing (Parse exported HTML) (#1026)
• [x] NoteCase importing (Parse exported HTML) (#1034)

From comments

Todo

• [ ] Progress bar when updating nodes not showing progress
• [ ] Show/Hide the main window
• [ ] Fix naming of node sanity check methods in CtActions
• [ ] Strip bad characters on URI list paste
• [ ] Fix code duplication in CtCodebox::to_xml and CtHtml2Xml::_insert_codebox
• [ ] Slowness of get_cell in CtPrintTableProxy

Done

• [x] Find a good CSV lib (Wrote own parser #914)
• [x] Custom weblink action (Open weblinks in custom application?) (#925)
• [x] Validate and check corruption of SQLite database when populating the treestore (#943)

Other

Todo

• [ ] Theme editor (issue #896)

Done

• [x] Tree summery (Tree Info) - Not too difficult but requires more in depth into the code (4b458f6)
• [x] New version checking - Trivial
• [x] Fix unable to import cherrytree files created by a version before 49e7f9b (when ts_creation and ts_lastsave were added to the schema) ( #832)
• [x] Table of contents insert (#909)

[txe]

Cool, thanks,
Also, some missing functional can be found be searching 'todo' in code.

[giuspen]

1) new version checking - trivial
2) there should be no need of a special utility to add newline in tables at all. the tables framework changed and a table cell is now like a codebox
3) tree summary - not too difficult but requires more in depth into the code
4) the proprietary external note taking applications go from the most used, I would say Zim, Gnote, Tomboy are probably the most in use...

[txe]

The main issue with importing that there is no files to test(. Also, the import can be much more efficient than in pygtk version, if you're going to work on it, I can show what to change.

[giuspen]

Agree with @txe, a big mistake I made was to not add unit tests at all. I will spend time in future just writing unit tests but you also try and do the same.

[ForeverRainbow]

I am happy to work on some of the importing features, but I am not familiar with any of the formats, looking at the python version it seems like they are mostly just XML which needs to custom parsing so not particularly difficult just will take a while (libxml++ actually has the SAX parser class to do just this).

In terms of the current importing is there anything which needs to be changed to improve it? I feel like sending the HTML through the clipboard is kinda hacky but it seems an efficient use of existing functionality.

[txe]

Right, I was talking about improving XML/HTML parsing. You can see the difference between HTMLHandler (pygtk) and CtHtml2Xml. The former has issues:

• can create really a lot of rich_text (rich_text_serialize), a few GB. CtHtml2Xml tries to concat rich_text with the same style.
• when tag is closed, instead of reverting style to previous state, it removes style. CtHtml2Xml supports the stack of styles.
  So you just need to use CtHtml2Xml as an example when you deal with XML/HTML.

[ForeverRainbow]

Noticed an issue, the python version has a fix for the addition of ts_creation and ts_lastsave fields where they are missing, since the C++ version has no such fix it is currently impossible to import such files. I will try and fix this next I think

[ForeverRainbow]

I am concerned that if all the import handler classes are placed into the same file (ct_imports) it will end up as a massive monolithic header and source file. Would it be ok to put them into seperate files? (in a subdirectory), otherwise I think the compile times (and readability issues) are going to be huge.

[giuspen]

I would go the way already walked for ct_actions_* so splitting in ct_imports_* but for now I would not make subdirs. Note that one of your unit tests is failing on msys2 but I can look at that myself later anyway :)

[ForeverRainbow]

That is weird AFAIK g_content_type_get_mime_type() should be cross-platform, possibly its an issue with connecting to the windows registory in msys2?

[giuspen]

If I remember correctly I had to do platform specific in Pygtk2 because the mime type was just looking at the file extension on windows

[ForeverRainbow]

That makes sense, from the documentation it looks like it just looks at the extension on windows and then checks the registory:

A content type is a platform specific string that defines the type of a file. [...] On Win32 it is an extension string like .doc, .txt or a perceived string like audio. Such strings can be looked up in the registry at HKEY_CLASSES_ROOT

[mahdi1234]

Hi,

gentoo added https://packages.gentoo.org/packages/app-text/cherrytree build 0.99.0_p20200523 today so I had to try it, noticed tree list bookmark icon missing, do you want this issue be commented with other things I spot or separate issues?

[txe]

Hi @mahdi1234 , separate is better

[txe]

markdown is really popular nowadays and as format it looks quite simple, maybe it will be good idea to export it as well? #579, #55

[DiagonalArg]

@txe markdown should be simple enough to import, also (you linked the issue, but didn't include it in your comment).

[ForeverRainbow]

Markdown would certainly be a good idea, we could export to markdown or maybe just send the already existing HTML through a HTML->MD parser?

I've also been thinking that cherrytree should support at least _some_ of the more common markdown formatting (like _italic_ and **bold**)

[txe]

An external package dependency is less desirable because it can become an issue during porting on Win\Mac. As just include or source, it can be ok. Also, a good parser (not necessary HTML->MD) can fix an issue of supporting different types of markdown.

I've also been thinking that cherrytree should support at least some of the more common markdown formatting (like _italic_ and bold)

I don't get that, what did you mean?

[ForeverRainbow]

The escapes don't seem to have rendered properly, I mean ** for bold text and _ for italic text. It is quite natural to me to write using markdown formatting and many rich text editors support it natively. If we had a proper markdown parser it would be easy to just send text through it so that shouldn't be too hard.

There are a zillion markdown parsers out there but they are mostly javascript unfortunatly, though I am sure there _must_ be at least one for C++, either that or a basic one can be created with (hopefully) not too much difficultly

[txe]

Can you give an example of such editor? I'd like to see how it works for users.

[ForeverRainbow]

I mean, the github comment editor is one, as is reddit, SO, etc. Discord does it in realtime (formatting is applied instantly). Most wiki-like software has at least a semblance of markdown (although since most are older they have slightly different formatting).

CherryTree already has something similar with its shorthand tags such as <> (although that is currently broken on the C++ port)

[ForeverRainbow]

One of the most useful things for me would be to just have things like italic applied immediatly and then ` tags applied when exporting (probably generating codeboxes automatically would be best)

[txe]

It looks quite reasonable and not too hard to implement. Maybe we need to move this discussion into another place, to keep this issue clean.

[txe]

FIY, I'm going to do tomboy/gnote imports

[giuspen]

Thanks @txe, another feature not ported yet is the possibility to open a text file with cherrytree. In the Pygtk2 version if the command line argument is not a cherrytree document tries to open it as text and load as single node

[txe]

OK, I'll take a look

[ForeverRainbow]

I am gonna do the table of contents inserting

[giuspen]

Cool!

[ForeverRainbow]

@txe Did #910 implement gnote and tomboy importing?

[txe]

Hi, kind of, I's still going to test it

[ForeverRainbow]

I am going to do the tree info, in regards to the CSV lib do we need to be able to handle external CSV (i.e csv not exported by cherrytree)? Since if not there is already existing the structure to tokenize text and implementing a csv parser would be trivial.

[txe]

I suppose, csv from Excel like app can be supported

[ForeverRainbow]

I am assuming that the python version uses the python csv module? Which only supports excel without custom formats so yeah, we could just implement a csv parser by extending CtTextParser

[txe]

Right

[ForeverRainbow]

Ok so tree info was done in 4b458f6 should have checked that :p, shall I try writing a csv parser for the tables then?

[txe]

Sure, btw, New version checking is also done

[giuspen]

Thanks guys, I just pushed a commit where I renamed all the stock icons which involved many source files and changed build.sh to build the Default target by default rather than Release. I will try and add a couple of unit tests over the tree info using tests/data/test.ctb and tests/data/test.ctd.

[giuspen]

The debug target is faster to build also from Travis, I will change also the windows target in Travis to build the Debug target because we don't care about optimisations there. @txe you are the Mac OS man it would be great if you could add that target to travis as well in future so that we will be more wary as to not break the build. If there is a way to try the build in a sort of VM I can do that... is there a trick I'm not aware of to test the build on Mac OS not owning a Mac?

[txe]

You won't believe, I'm mac user as much as you. So, I have macos in vmware, it can be downloaded from some places :)

[txe]

@ForeverRainbow ,
I wanted to used this markdown sample, but when I couldn't import it, the parser threw an exception.

Can you fix it or give another example that I can use for testing?

[ForeverRainbow]

@txe Was that with the pandoc import or with the markdown import? The markdown import supports limited syntax (the most basic of the common md "standard" afaik) so it wouldnt be able to parse that document

[txe]

supporting limited markdown is fine, but if the parser cannot read document and skip unsupported syntax, wouldn't it be useless for users?

[ForeverRainbow]

Yes, it shouldn't have crashed thats a problem

[ForeverRainbow]

What I think I am going to do actually is make the markdown import attempt to use pandoc and fallback to the built-in one if it cannot find it, also fix this issue with the parser xD

[ForeverRainbow]

I didn't end up doing that in the end, I improved the markdown parser built-in to cherrytree instead. Pandoc can still be used ofc but I do think its better cherrytree has support for some markdown without an external application

[ForeverRainbow]

@txe Is custom link action done? CtConfig::weblinkCustomAct doesnt seem to be used by anything, if its not done I'll have a go at implementing it now

Edit
Ok I find the todo in link_clicked, I'll do that now

[ForeverRainbow]

@txe Do you have already a way in mind for checking corruption of the SQLite database? Because afaik all that is needed is PRAGMA quick_check call

Also the backup method for SQLite databases just copies the file as far as I can see, which is potentially problematic and can cause corruption if, for example, the database file is copied without its journal, or a database transaction hasnt been fully flushed to the disk yet (unlikely but possible with consecutive saves). Really we should be using the backup API provided by SQLite

[txe]

@ForeverRainbow, the idea about checking is to prevent users to work with a broken db (also, they get frustrated not to be able to save changes), PRAGMA quick_check is sufficient.

About sqlite journal:

• According to docs, the rollback journal is usually created when a transaction is first started and is usually deleted when a transaction commits or rolls back
• we don't use explicit transaction commands (I don't know it' one transaction per connection or per query?)
• to be sure, the db connection is closed before making copy of file.

Keeping the above statements in mind, journal should not exist. And if it exists, it's already something wrong with db file (program or OS crush) Currently we do nothing about it, but we might (do some recovery, etc). At least, we can check it and show warning to user. So, I don't know how Backup API can help here

Edit: By default, SQLite operates in auto-commit mode. It means that for each command, SQLite starts, processes, and commits the transaction automatically

[ForeverRainbow]

A recovery journal is created if SQLite crashes, or losses access to the database or whatever, during a transaction. An implicit transaction is created for commits if a "manual" one is not used, I _think_ this is also the case for queries but not 100% on that one.

This means that if cherrytree crashes during save, or is force closed during a save (e.g it goes unresponsive), a recovery journal will be created and copying/renaming the database file before opening the database _will_ cause corruption (although since its opened for read on load, this isnt really an issue here, its just another potential bug).

If there is 100% certainty that all database connections are closed then just copying the file should be fine, its just another potential bug which could be nasty and very easy to trip up on (especially if multi-threading which imho, storage access should be)

[ForeverRainbow]

@txe Shall I just add a method which checks that PRAGMA quick_check doesn't return results then? (and triggers appropriate warnings to the user ofc)

[txe]

Sure, good place is in CtStorageSqlite::_open_db

[txe]

Backup are still fine. They are created before saving data but after opening connection. Either sqlite fixes journal during opening file or we check that the file is corrupted and stop process, so backup will never happen.

[ForeverRainbow]

@txe Not sure it should be in every call to open the database? The probability that it is corrupt is quite low and I am guessing that it causes some overhead (which increases with database size). Your call but maybe just check before heavy operations (like saving) or every N operations? (And check on first open ofc)

[txe]

Check before heavy operation

the idea about checking is to prevent users to work with a broken db (also, they get frustrated not to be able to save changes), PRAGMA quick_check is sufficient.

Ok, then call it not in _open, but

• void CtStorageSqlite::test_connection()
• void CtStorageSqlite::import_nodes(const fs::path& path)
• bool CtStorageSqlite::populate_treestore(const fs::path& file_path, Glib::ustring& error)
• bool CtStorageSqlite::save_treestore(const fs::path& file_path, const CtStorageSyncPending& syncPending, Glib::ustring& error)

without checking how many. In theory it will be once per file

[txe]

I'm wrong, CtStorageSqlite::save_treestore doesn't need check

[ForeverRainbow]

@txe, @giuspen I have written a keepnote importer, however I just realised I cannot test it due to not having python 2, which is going to be an issue for people moving to the C++ version as well. Is the C++ version going to support keepnote? Anyone who uses it presumably has python 2 so can run the python version to do the import if they want.

[txe]

I didn't get it, how is python2 related to import implementation?

[ForeverRainbow]

@txe Keepnote is written in python 2 and the last release I can find is in 2012. I do not have any way to test my implementation since I cannot generate keepnote files. If anybody has some keepnote files for testing that would work too I guess.

When trying to get it to run by sending it through python 2 I get import errors. I might be able to get it working by hacking it but just checking that keepnote is going to be supported

[txe]

Ah, now I see, if you have already written keepass importer then it can be left as is. If users will have any issue with it, then we can ask files to test it..

[giuspen]

I will also generate some data with a virtual machine in future and test it

[ForeverRainbow]

In terms of priority what imports should be done next?

[txe]

Treepad?

[ForeverRainbow]

Btw, the only download link for Treepad I could find was with the wayback machine here. Treepad does have export funcionality too (and HTML is supported), just not sure about reimplementing so many importers for uncommonly used software especially given that it is only an extra step (export from the software) to import from it

[giuspen]

@ForeverRainbow if a document type is very old and you hardly can prepare data to test the import skip it and I will consider whether to leave it off the future cherrytree.

[ForeverRainbow]

Researching some of these formats, it looks like most of them seem to be HTML and all that will be needed is picking the right files to import. Some (EssentialPIM and probably others) use inline CSS (in <style> tags), do we need to worry about that or just import the HTML?

Also is a seperate import option needed for formats such as EssentialPIM HTML which are just ordinary HTML files which could be done by the HTML import (as far as I can tell)

[giuspen]

The Pygtk2 version is surely not parsing the CSS so I would be conservative with that

[ForeverRainbow]

The list of things I am going to skip for now at least:

• Keynote: Apple only
• Knowit: Last release 2004, therefore provides no fedora package and compiling from source likely requires libs which are unavailable now (although I havnt tried)
• Basket: Last release is 2010, depends on Qt4 which I would prefer not to install and must be build from source
• Treepad: As mentioned above, very difficult to obtain even binaries for and can export to HTML anyway
• TuxCards: Again 2010 release, depends on Qt4 and must be built from source
• Leo files: Couldn't find anything for these, what application produce them or is it some format?

Mempad seems to be a small helper program, windows only but that is fine, RedNotebook and NoteCase seem to be maintained and I will look into their formats, EssentialPIM just uses HTML and I can wrap the import option just to call the HTML import.

[ForeverRainbow]

Ok, RedNotebook and NoteCase both have HTML export options. Imho we should use these as they will be better and more-up-to-date than trying to parse their native formats directly. Mempad has some weird custom format (and can only export to txt) for which I will set up a parser.

I suggest that instead of directly supporting native formats for everything, instead simply ensure the HTML parser can handle the exported HTML. Cherrytree could also run from the commandline in "import" mode, where it imports a file or from stdin (allowing piping from exporters). Also most of the ones I listed as skipping for now have HTML export options afaik

[giuspen]

Sure as long as the exported HTML is not losing information that we need

[ForeverRainbow]

Should I remove the import options for formats HTML will be used for or redirect them to the HTML importer? I feel like leaving them there clutters the UI but there needs to be some explanation that cherrytree does still support importing from these applications just in HTML form

[ForeverRainbow]

Ok I have found leo (https://github.com/leo-editor/leo-editor), I will do it next

[giuspen]

Thanks @ForeverRainbow , before inviting the users to make use of the HTML export of a note taking application we will have to test that parsing it recreates the tree properly in cherrytree. If you want to comment out the imports that are not really supported at the moment that is fine, maybe add a "todo" pattern so we can easily spot them.

[ForeverRainbow]

@giuspen Yes of course they need to be tested, I was just wondering whether they would be left as they were with separate options or merged into the HTML import file/folder

[giuspen]

My point is that until we see that the HTML parsing has complete information we cannot exclude in the end we will parse the proprietary data

[giuspen]

@ForeverRainbow I recently finished implementing an example of unit test that is asserting the data in the (cherry) tree after the document is read (also saved and re-read but this is not what I wanted to point you to).
This is not a requirement and I'm already infinitely thankful for the work you are doing, but if in future you would like to try and unit test an import, it should be quite simple now.
If you see https://github.com/giuspen/cherrytree/blob/master/future/tests/tests_read_write.cpp I'm using this TestCtApp that inherits from the CtApp so has access to everything. You can pass arguments and then do something specific like importing into an empty tree and then assert all you want in the tree. If you will decide to have a go please any doubt just ask.

[ForeverRainbow]

@giuspen I will have a look at that, to make life easier we might make a subfolder under tests/data which has different formats which should produce the same data when imported and then just iterate through them all

[giuspen]

@ForeverRainbow that is not possible because the external note taking apps have only a subset of the features of cherrytree so I'm afraid we cannot use the same test for all... unless many if (supported)

[ForeverRainbow]

@giuspen Surely basic things like whether the hierarchy is right can be tested, then more complex ones like formatting and possibly widgets can be done with some sort of capabilities list

[giuspen]

@ForeverRainbow the unit testing that I have done for the cherrytree document types is comprehensive of all the information that we are saving in the document but you don't necessarily have to cover all the content of the tree yourself, just few assertions can help to ensure that the import is still working if/when the code is modified in future, without need to manually re-test everything. Anyway if it builds fine I'll have a look and merge in immediately, thanks!

[ForeverRainbow]

I just looked over the NoteCase and Rednotebook HTML. In order to preserve the node hierarchy and not just put it all in one node some custom parsing will have to be done.

Rednotebook HTML is mostly simple with nodes being split by <span> tags with id set to a date format which could be matched.

NoteCase actually has 4 different formats, with the "lite" (free) version only being able to save to .ncz which is a proprietary compressed format. The python version seems to just support the .ncd format judging by the regex to find properties but I would say thats low priority since you need to buy the pro version to use that format. NoteCase can also save to .ncdb which might be nicer to parse since its an sqlite database but again its pro version only

[giuspen]

Years ago there was notecase free and the document type was .ncd which was an html enriched with also info about the hierarchy and serialised images saved in it. I would not supported importing from the notecase pro but from the free version as the Pygtk2 version.

[ForeverRainbow]

@giuspen Yeah, ncd now requires a pro license (although mine seems to have a bug, it warns about needing a pro license yet saves the file xD). The ncd format is almost identical to the exported HTML however, minus the creation date

[ForeverRainbow]

I will look at Rednote format next. Btw I think #393 would be a great idea for allowing customisation of import/exporting, but I don't have any experience with plugin systems in C++ and I doubt it will be simple

[giuspen]

@ForeverRainbow I don't have experience with plugins systems as well, we'll get back on that later

[chadfurman]

I've experience using Lua scripting in C (Janus Gateway), and generally you just have to add hooks to the code for the things you want plugins to be able to do. Specifically, if you want them to swap out the editor for example you need to define an interface with the editor module then drop the module in place and check the plugin registry for anything that hooks the module, pass the editor module to the hook and use the new editor module returned from the hook.

Obviously this is hand wavey though the thought structure is there.

[chadfurman]

GPL warning: https://github.com/notepad-plus-plus/notepad-plus-plus/tree/master/PowerEditor/src/MISC/PluginsManager

This plugin manager could be useful inspiration

[ForeverRainbow]

Interesting. So it uses a C interface and passes function pointers across the ABI gap, then exposes a struct which provides information like name, and what its shortcuts are?

Also seems to have an event system and such, sigc++ may provide an easier way to implement these than by hand but I'm just thinking out loud now :p