org-roam <> org-zettelkasten Discussion Thread

[jethrokuan]

@l3kn Since we're both working on something similar, I thought we should have some discussion about where each package should go, so efforts are not duplicated. As far as possible, maybe we can extract out certain package logic to allow for reuse.

For me, I'm just working towards emulating Roam. Unless there's some amazing feature that Roam doesn't have that I can create, i think that's as far as I'd go. For now, I'm curious as to:

1. What are you creating beyond what Roam provides, and how does it help you? Zettelkasten is meant to be a simple workflow, and I'm striving for that.
2. It seems like you will be building another external indexer. I think that would be useful in my case as well, so perhaps we can work together on that.
3. I'm just curious to understand your workflow in general, with your package. Would you mind sharing a bit more? Happy to take a video call.

[alphapapa]

This package may be relevant or give you some ideas: https://github.com/alphapapa/org-sidebar

[bbigras]

I would also be interested in knowing the difference between org-roam, org-zettelkasten and org-brain.

[l3kn]

I started working on org-zettelkasten because I ran into performance problems using other similar packages once my collection of files got larger.

In my experience, the default org-agenda takes seconds to load with > 100 agenda files.
I'd like to have a folder of > 1000 thousand and a way to get an instant (let's say <300 ms) overview of all headlines marked as TODO / NEXT tasks, or all calender entries in it.

Another example is extracting the #+TITLE keyword from files to build a file selection / searching command.

Most of the effort so far has gone into building a emacs-lisp only (no sqlite / other database) cache for this kind of data that is persisted to disk to keep the emacs startup time low, then creating versions of org-mode functions / views that work on this data.

For a similar performance problem, see https://bitbucket.org/eeeickythump/org-drill/issues/48/slow-startup where a user reports a loading time of >30s for 80 files.

With an (experimental) indexer based on my cache, I can search 1.5k files for due cards in < 1s.

Differences to org-brain

From my understanding of org-brain, the main difference between it and org-zettelkasten / org-roam is that it uses one buffer for visualizing a notes text & connection graph whereas org-zettelkasten and org-roam use the "raw" org file as a main view and add a sidebar / graph view.

Another difference is that org-brain allows linking to headlines in files, showing them as nodes in the graph. In my setup, I prefer to move headlines to separate files once I want to link to them from multiple places, so I'm not planning on adding a similar feature to org-zettelkasten.

Choosing a Package

If you want to start building a roam / zettelkasten style collection of notes,
I'd advice against using org-zettelkasten.

Once you run into similar performance issues (and once the implementation is more stable),
it should be possible to run it on top of your existing notes with little to no changes,
selectively enabling the features that seem useful to you.

[jethrokuan]

Thanks for the overview! I think a big blocker for current state of the art is a good out-of-Emacs Org-mode parser. A reduced Org-syntax with a context-free grammar should enable building of mobile solutions, web solutions and these indexers.

I remember you mentioned that when run synchronously, the backlinks cache-building takes around 16 seconds for you. To me that would be unacceptable ><

For the indexer, I'd like something that adds/removes links between nodes by watching file operations (inotify).

[jethrokuan]

After the chat, I've decided that this code can be extracted out into 3 main modules:

1. The org-roam buffer
2. The graph building and navigation
3. Other required utilities, like org-roam-insert

The backlinks cache building shall, as far as possible, be moved to org-el-cache, which will serve as the basis for other modules that @l3kn is building, like a flashcard system. Ultimately, this allows the end user to select the features they want to use, while allowing a way forward collaboratively.