Zettlr: [ENHANCEMENT] Automatic Backlinking

[DrkDigglr]

Very briefly, I've just discovered Zettlr and I have to say, it's absolutely awesome. It's really cool to see truely opensource tools being developed actively :)

Description

Ability to have Zettlr automatically prepend backlinks to beginning of destionation files in the event they are linked to from another file.

Proposed Changes

I think this would be best suited as a varient of the current linking syntax. If you don't use the new syntax a backlink won't be added to the destination file. I'd like to be able to use the current default syntax ([[]]) to do one-way links and the varient of [[[]]] to have the backlink generated.

Caveats

I can't see any real caveats. It would be nice to have the ability to configure the linking syntax alongside that which you can currently do.

Additional Information

Given the craze around Roam Research and the ability to do backlinking, I think it would be a good enhancement. I'm not a big fan of heirarchical data structures and I use Markdown as a form of knowledgebase alongside TheBrain. I'd like to fully adopt Zettlr but the inability to traverse related files is a bit of a bummer. I could manually put the backlinks in but then I'm really slowing down my productivity when making notes.

I think this feature along with the one currently going through as a pull request (#178) to create new files in the event they don't exist, would make the workflow of adding new notes and navigating through those that are related, the ideal experience.

[boring-cyborg[bot]]

Thanks so much for opening up your first issue here on the repository! 🎉 We would like to warmly welcome you to the community behind the app! ☺️ We'll check in soon and have a look at your issue. In the meantime, you can check your issue and make sure it aligns with our contribution guidelines! Here's the comprehensive list:

Enhancements

An enhancement takes a feature and improves or alters its behaviour. Please make sure to argue how your proposition will aid non-technical text workers, and why it can't be emulated easily with other features or apps!

Feature requests

Feature requests introduce whole new features into the app. This requires a lot of work, so these might be turned down if the implementation costs supersede the benefits we expect to see from implementing it. Please do not be disappointed if that happens. It likely has nothing to do with your great request but simply with us and our missing resources!
You can of course always ask someone to implement this feature, because a PR with a working new feature has much higher chances of being merged! :)

Bug reports

Please note that one of the main reasons for why bug reports cannot be addressed is that there's not enough information for us to find and fix the bug you describe, so make sure you try to pinpoint the bug as close as possible.
The ideal bug report for us has two qualities:

1. The bug is always reproducible, at least within a certain context. 2. We know exactly what specifically goes wrong, and there is consensus on
   what should happen instead.

Please note that if you encounter behaviour that does not align with your expectations of what would happen, this might as well be simply intended behaviour and we need to simply _clarify_ why the behaviour is the way it is. This is not to be considered a bug and such issues may be closed! Suggest an enhancement instead!
But now, have a great day and thank you again!

[nathanlesage]

Thanks for joining the party! :)

I can't see any real caveats.

There are quite a lot. There are a lot of things to think about — where to put the backlinks? How? How not? Edge cases, etc. etc. etc. That being said: It's something that's been on the bucket list for some time now, but it will take a considerable amount of time to implement. It may look great, but Zettlr has a big difference to Roam: It does not work with a database but with real files, and changing these takes a lot more time than a simple database. Plus, I want to make it _right_, not fast … (I can't hammer in new features like I used to because the user base has grown considerably, so the app needs to be much more stable and is less a field of experimentation :/)

Going further:

It would be nice to have the ability to configure the linking syntax

You can actually do this in the preferences!

I'm not a big fan of heirarchical data structures

Me neither, and one of the reasons why I have "neglected" the Zettelkasten functionality for quite some time now is that I believe I have a better solution to that instead of automatic backlinking (but the idea is never to automatically change files, but instead build a Graph — yet I have to read more papers on these computer linguistic things in order to make the idea a reality)

So, TLDR: There will be improvements, but currently due to the rising user base I'm more putting out small forest fires in the codebase than implementing new features, so it'll take some time :D

[DrkDigglr]

Well you did make some good points RE caveats :)

I've actually spent some time on graph databases myself. I really don't think it's required for something like Zettlr. I can see benefits for searching and performance perhaps, but on the whole I don't really think it's worth doing.

If you've got links and backlinks you've got the nodes and edges of your graph. I question how much effort it would take to store those nodes and edges in a flat config file somewhere and draw a graph dynamically from that, essentially using an in-memory database.

It's getting to the point where I'm considering rolling my own solution to generate backlinks and add them to files for me. I completey appreciate the shift in how you develop this project. I'm less bothered about having a graph and more bothered about being able to traverse related notes and build those relationships fast.

[DrkDigglr]

Sorry I'd just like to clarify the syntax of linking that I was describing earlier. I understand this doesn't mean it's going in as an official feature, but in terms of flow I think it would be pretty slick:

• [[link]] makes a link to a file as normal
• [[[link]]] makes a link to a file as normal and adds a backlink into the destination file

[nathanlesage]

If you've got links and backlinks you've got the nodes and edges of your graph. I question how much effort it would take to store those nodes and edges in a flat config file somewhere and draw a graph dynamically from that, essentially using an in-memory database.

Well, it has to do with my lazyness of linking stuff — if you have a graph database which also makes use of something like, e.g., Word2Vec, you can have links without actually putting links — and I think (albeit not verified, obviously), that this may be beneficial. But I do concur with you that it's a lot of work for, potentially, nothing!

Sorry I'd just like to clarify the syntax of linking that I was describing earlier.

Ah, so basically having the syntax describe when or when not backlinks are being generated? Do I read you right here?

[DrkDigglr]

The main problem I've had with using graph databases is finding one that is opensource. When you actually start digging into the features you realise that everything sits behind an _Enterprise license_. Word2Vec looks very interesting. Will read into that, thank you!

I think links without putting links takes a lot of away from Zettlr's ability to pick up Markdown content without having any tool lockin or specific big deviations. Product lockin is massive for me right now, TheBrain is totally locked down. I'm having to migrate to Markdown just for safety at this stage. All stuff to think about I guess, all caveats, which I thought were none 😄

Ah, so basically having the syntax describe when or when not backlinks are being generated? Do I read you right here?

Exactly. Simple, quick and effective.

[nathanlesage]

you realise that everything sits behind an Enterprise license.

Yes, because the companies are douches, as with a lot of corporate tech, heh

Exactly. Simple, quick and effective.

Unless talking about implementation :D I'll have a look!

[Arvinsanity]

Just wanted to say that I've been using a workaround in sublimeless ZK right now for this. It's why I raised this issue https://github.com/Zettlr/Zettlr/issues/806 Please check the GIF that I put in :)

[Arvinsanity]

@DrkDigglr

[jbwhit]

There's also this approach which could be a reasonably lightweight addition:

https://github.com/andymatuschak/note-link-janitor

Essentially it adds a section at the bottom of the files with the backlinks.

[indeyets]

I don't think we need something like database here, as we have ultra-fast search tools now. ripgrep can search thru any realistic collection of texts in less than a second.

So, all we need is a button, which would open an overlay pane. Then we need to fill it dynamically with ripgrep (or something similar) and cache result in RAM until user switches to another document

p.s. I believe that injecting backlinks into document is a separate optional feature and is much less useful for majority of users

[ditServices]

Notebag does this nicely. It would be a great feature to have in Zettlr

[deobald]

@nathanlesage I'm not sure where best to push my votes for things... +1's on a GitHub issue never feel terribly efficient, but I'll stick to that for now and move my votes elsewhere if this is the wrong place for voting.

Between the options of (A) forcibly writing backlinks directly into files and (B) dynamically auto-generating them, I probably can't vote hard enough for option (B). This is both as a software developer with decades of experience supporting legacy systems, and as a humble Zettlr user. :bow:

Yes, writing the backlinks into the files themselves means that they exist within the plaintext files and can be ported elsewhere, but this in effect creates a protocol on top of the Markdown files; they are no longer simple Markdown and for files to coexist between Zettlr and other apps, all other apps will need to implement this feature in the same way. Adding a special syntax ([[[linked text]]], or otherwise) unfortunately exacerbates this "invisible protocol" problem. Should other software choose to auto-generate backlinks, it will now display each note with both the forced backlinks which were written in the file by Zettlr and its own dynamic backlinks. (When I say "other software" I'm currently thinking of tools like Press and GitJournal for modifying my Zettlr notes on other platforms, so this isn't a hypothetical scenario.)

Since the backlinks themselves are inherently derivative data, any storage of that derivative data becomes a cache. This is also true of any other backing store but I would sincerely hope the first version of backlinks does not rely on anything but the Markdown files anyway. Writing backlinks into the markdown means they are user-editable (there's nothing quite like permitting your users to corrupt a cache) but writing them to disk anywhere else means the data is duplicated.

As a side-benefit of dynamically auto-generating backlinks, they can be turned off. It's quite likely that there are existing Zettlr users who have no requirement for graph-shaped data, be it implicit or explicit. For those users, Zettlr could remain as it is.

I hope this makes sense. I'm happy to clarify any of this, if not. Thank you for Zettlr!

/cc @saket @vHanda

[nathanlesage]

@deobald I completely see your point, but it tends to contradict the philosophy of Zettlr never to mess with ones files — and I count even expected "messings" among that.

This is why I have an alternative idea, which I am planning and growing since a few weeks: to enable something like a "cursor HUD": Hold Cmd or Ctrl. for a second and you can immediately insert a whole lot of text in that position: backlinks to all files linking the one, suggested tags, other stuff. This would be a compromise I'm willing to make for a program that first and foremost aims to help researchers in the social sciences get off of Microsoft Word. That's why I'm still not going with all the PKM-hype that's currently going on …

[deobald]

@nathanlesage I'm a bit confused how dynamically auto-generating backlinks qualifies as "messing with files" if it never modifies the files (which was what I was describing -- it's my strong opinion that backlinks should _never_ exist inside the file itself). Maybe you can walk me through your thinking there?

Hold Cmd or Ctrl. for a second and you can immediately insert a whole lot of text in that position: backlinks to all files linking the one, etc. -- While I understand the appeal, I'll admit I'd never use this feature. That's because any backlinks which live inside the files themselves as static text, whether automatic or manual, will inherently become out of date over time. A list of backlinks collected on one central "cluster" node in the graph (say, an idea I revisit frequently or a note discussing a white paper I frequently come back to) will either always be out of date or it will demand that I constantly delete and re-create it. That sounds like precisely the kind of manual labour that would discourage the users from creating new files... in which case we're likely to give up and go back to Emacs for managing notes. (I don't know any researchers who use Word for this, honestly, so I may be far outside the target user base.)

Regarding research vs. PKM, I completely understand the desire for explicit and manual control over backlinks if the files in Zettlr are intended to _become the documents themselves._ That's not how I use it at all, though... I only use Zettlr (and equivalents) for the research itself. My papers are still written in LaTeX and composed in Emacs. If it's the explicit intention of Zettlr to act as a replacement for that, I may have grossly misinterpreted its intended purpose. Is final document authoring an intended use case for Zettlr?

[nathanlesage]

I only use Zettlr (and equivalents) for the research itself. My papers are still written in LaTeX and composed in Emacs.

Ah, I see there's the point of friction — I write my full research papers in Zettlr, and don't use LaTeX. One of my initial motivations was that I wanted something to get rid of having to write LaTeX, but still have the ability to use some LaTeX to further finegrain that.

Effectively, Zettlr should become some form of a replacement for Word, which is why PKM-support is somewhat limited. Concerning automated backlinks however, I have to confess that my I simply want to know what is going on, so instead of forcing arbitrary solutions (I see why this might never be used) what we could think of is a plugin for automatic backlinks, but what I need would be a very verbose output (which links have been added where, etc.)!

[deobald]

Effectively, Zettlr should become some form of a replacement for Word, which is why PKM-support is somewhat limited.

Gotcha. This makes sense to me. I'm not very familiar with the PKM world (I've never used Roam, Obsidian, or their equivalents) but I am painfully familiar with the Markdown-editor-of-the-future world. Automatic backlinks interest me from a research and fieldstones perspective... I still do all my writing research (it would be a stretch to call it zettelkasten since I only looked up that term after I started using Zettlr) on paper notes at the moment but Zettlr is unbelievably close to replacing them for me. For my style of work, the appeal of automated backlinks comes from the undirected graph they create. My paper notes are inherently a directed graph... if the ideas are even properly connected. A wiki is also a directed graph so the medium doesn't provide much beyond paper. I haven't bothered committing to a wiki since offline-first matters a lot to me personally; I spend about a third of the year in an off-grid cabin (sorry, TMI) which is why I've been staring at the ocean of available markdown tools for so long.

EDIT: Sorry, when I use the word "graph" I mean that purely in the _mathematical_ sense. After submitting this comment I went and had a first look at Roam and Obsidian (I've never even considered them options, since they're not open source) and saw they both have a _visual_ graph overview of links. I'm not sure if some users of those apps find a visual graph helpful for navigation or if it's just a fun toy, but that certainly wasn't what I meant in the above paragraph.

One of my initial motivations was that I wanted something to get rid of having to write LaTeX, but still have the ability to use some LaTeX to further finegrain that.

Out of curiosity, is Zettlr likely to acquire more wordprocessing features from MS Word /Emacs/LaTeX over time? I'm honestly in love with its simplicity right now and if it's bound to become a complex word processor over its lifetime I should probably hit the road and continue my search for a desktop markdown editor that can link files. :)

what we could think of is a plugin for automatic backlinks, but what I need would be a very verbose output (which links have been added where, etc.)!

This sounds perfect! Are you hoping for external contribution on this particular feature? If so, is there a plugin endpoint you would recommend looking at?

[nathanlesage]

I'm not sure if some users of those apps find a visual graph helpful for navigation or if it's just a fun toy, but that certainly wasn't what I meant in the above paragraph.

Yeah, a lot of people REALLY like those things. I never really got the gist of it, especially given my background in statistics (see, e.g., this brilliant blogpost on the limitations of visual networks).

Out of curiosity, is Zettlr likely to acquire more wordprocessing features from MS Word /Emacs/LaTeX over time?

Hehe, interesting question — but no. It aims to become a replacement for Word, yes. But not by _becoming_ yet another Word, but by porting the useful stuff and removing all the bloat (I'm always amazed when I look at the installed size of my Word executable, which is over 2 GB)

Are you hoping for external contribution on this particular feature? If so, is there a plugin endpoint you would recommend looking at?

I think … there is something already in the pipeline? There is no plugin API, but you can plug something to the internal API, which still looks _kind of_ messy …

I'm always happy with PRs!

[deobald]

I think … there is something already in the pipeline?

Rad! I'll wait to see what comes out of this before I harass a friend(s) to contribute? I haven't done front-end things for the better part of two decades and I've never written Javascript professionally so I'm probably ill-equipped to meaningfully contribute anyway.

Hehe, interesting question — but no. ... not by becoming yet another Word

That's a relief. Thanks for taking the time to answer all my goofy questions! I really appreciate it. Looking forward to seeing what comes of the plugin approach!

[nathanlesage]

I haven't done front-end things for the better part of two decades and I've never written Javascript professionally so I'm probably ill-equipped to meaningfully contribute anyway.

Nah, same here!

That's a relief. Thanks for taking the time to answer all my goofy questions! I really appreciate it.

You're weelcome!

[mmhat]

A bit late too the party, but how about the idea of embedding search queries in notes? Neuron does that. Such an embedded query could look like this:

~~~zettel-query
query: "linksto:1234567890"
sort: ascending
...
~~~

So this is just an ordinary fenced code block with syntax "zettel-query" which in turn is a YAML document describing the query and how we plan to insert the matches.

When Zettlr renders the document it performs the lookup and renders the matches. Just like we render mermaid diagrams or KaTeX formulas.

I think such an embedded query feature might be useful on its own and I'll might open a separate issue when I have more time.

Thoughts?

[OliverEvans96]

:+1:s all around for the great project and discussion!

@DrkDigglr:

The main problem I've had with using graph databases is finding one that is opensource. When you actually start digging into the features you realise that everything sits behind an Enterprise license

As far as I can tell, Neo4j has an open-source community edition. Markdown editing probably doesn't require high-availability and some of the other enterprise features they offer ;) But I'm no expert on graph databases by any stretch of the imagination.

@deobald:

I haven't bothered committing to a wiki since offline-first matters a lot to me personally

~It's not markdown~, but are you aware of tiddlywiki? I just discovered it the other day and found it to be a pretty unique approach. Lots of atomic notes, all stored, linked, and edited within a single html file on your computer. There are several forks, including:

• tiddlyroam, which adds auto-backrefs.
• stroll, which adds auto-backrefs and two-column mode
• tiddlymap, which adds the 2D graph-vis a la obsidian

EDIT: apparently tiddlywiki has a markdown plugin

[deobald]

Hi @OliverEvans96

are you aware of tiddlywiki?

Yes, I was a big TiddlyWiki user 14 or 15 years ago when it was first released. :) I've had a few people recommend it to me in my journey to take my writing and research out of paper notebooks and into the computer but it really doesn't fit the bill for a variety of reasons that aren't relevant to this GitHub issue.

As far as I can tell, Neo4j has an open-source community edition.

I run Neo4j CE in production and ...it's fine... but I find it hard to imagine a single Zettlr user actually demands a graph database (of any kind) to persist relationships between their Markdown files. I chose Neo4j for that project (Pariyatti) because it is a huge digital library and the ontological relationships between nodes of the graph matter a lot (sometimes more than the content). This is not true of Zettlr.

I'm guessing _most_ of us use Zettlr because it's simple: plaintext md files with links. That's it. As long as Zettlr remains that simple, there is absolutely no need to duplicate those relationships in a graph database.

[OliverEvans96]

Very good points - so far it seems to me like for most use cases, generating backlinks dynamically with ripgrep as @indeyets suggested is probably the best solution

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[deobald]

@nathanlesage I just saw the stalebot notification and it's been a couple months so I thought I would check in to see what (if anything) happened with this:

I think … there is something already in the pipeline?

I'm still extremely happy using Zettlr as my primary writing tool and supporting development. I haven't found the lack of backlinks to be a huge limitation yet, given that search can provide a similar way to navigate relationships between documents. But I do keep looking for a notice about the backlinks plugin in the changelog with every upgrade. :wink:

[Whil-]

The main problem I've had with using graph databases is finding one that is opensource.

Hi @DrkDigglr - There is a fairly new one out there that seems interesting. TerminusDB. Open source. Just thought to put that out there. Cheers!

[pihentagy]

Regarding the syntax: please do not use [[[ or anything else which breaks compatibility with other markdown tools!

OTOH I'd like to have backlinks for every links. Providing some link specifiers? I'm in!