Zettlr: [Enhancement] Embed other notes in a note

[tubia]

Description

Embed/incorporate other notes/files in a specific paragraph of a note, using as a reference the UUID.

Proposed Changes

We have two notes, a.md with UUID 123 and b.md with UUID 124.
Currently, I can create a link to refer to the a.md in b.md using [[123]] in the body of the latter.
My proposal is that you can incorporate - when you export the file to .pdf or .docx and so on - the content of a.md in b.md using [[[123]]] in the latter.

Comment

This is probably a "meta" issue that could be addressed more in general through pandoc, but maybe it's possible to find a solution that involves zettlr, "before" involving pandoc. Something like:

• zettlr recognized the [[[ ]]] and searches the project folder for the file with that specific UUID
• zettlr incorporates the content of the latter file within the paragraph of the "incorporating" file, creating a new temporary .md file
• zettlr passes the resulting .md file to the pandoc pipeline

Pandoc filters

I did a bit of research and what I was able to find for pandoc have been solutions involving filters. For example in this thread of pandoc-discussion mailing-list. Then there is pandoc-include not updated since 2015 and pandoc-include-filter which seems to be more updated.

[boring-cyborg[bot]]

Thanks so much for opening up your first issue here on the repository! 🎉 We would like to warmly welcome you to the community behind the app! ☺️ We'll check in soon and have a look at your issue. In the meantime, you can check your issue and make sure it aligns with our contribution guidelines! Here's the comprehensive list:

Enhancements

An enhancement takes a feature and improves or alters its behaviour. Please make sure to argue how your proposition will aid non-technical text workers, and why it can't be emulated easily with other features or apps!

Feature requests

Feature requests introduce whole new features into the app. This requires a lot of work, so these might be turned down if the implementation costs supersede the benefits we expect to see from implementing it. Please do not be disappointed if that happens. It likely has nothing to do with your great request but simply with us and our missing resources!
You can of course always ask someone to implement this feature, because a PR with a working new feature has much higher chances of being merged! :)

Bug reports

Please note that one of the main reasons for why bug reports cannot be addressed is that there's not enough information for us to find and fix the bug you describe, so make sure you try to pinpoint the bug as close as possible.
The ideal bug report for us has two qualities:

1. The bug is always reproducible, at least within a certain context. 2. We know exactly what specifically goes wrong, and there is consensus on
   what should happen instead.

Please note that if you encounter behaviour that does not align with your expectations of what would happen, this might as well be simply intended behaviour and we need to simply _clarify_ why the behaviour is the way it is. This is not to be considered a bug and such issues may be closed! Suggest an enhancement instead!
But now, have a great day and thank you again!

[tobiasdiez]

There is also https://github.com/DCsunset/pandoc-include and for code https://github.com/owickstrom/pandoc-include-code

[canpolat]

This would be a great alternative to projects where one has to have a directory. Some people (myself included) use a flat file structure and "meta" documents would be a great way to stitch things together while keeping a good control over ordering and reordering things.

[tubia]

@canpolat yes I was thinking the same :)

@tobiasdiez thanks for pointing to these other pandoc filters! Actually, I tried to use DCsunset/pandoc-include in my pipeline in two ways:

• using the yaml frontmatter, in zettlr, to include the filter within pandoc-options: no luck
• including the filter in the pandoc command in the zettlr settings
  
  
  
  • !include Introduction.md not working, probably because the relative path is different (I think zettlr moves the files in a temp directory
  
  
  • Adding the full path in the include command resulted in a weird output, with the "include line" sanitized, hence not interpreted as a command
  
  

I still have to take a look at the code, but _maybe_ would be easier to implement this directly in zettlr - that has already in place a system to search uuid along the directories - but probably will produce less agnostic plain-text files - in which there are commands only interpretable by zettlr.

What I imagine is:

UI

• In zettlr, while editing a file, when you add [[[ a dropdown list appears with possible files to include (this could work also if using !include)
• The result is a UUID which you can Ctrl+click and has the same behaviour as normal cross-links

Building process

_When/where:_ somewhere in the zettlr pipeline, before it passes files and options to pandoc, when it looks for all custom commands

• zettlr recognises the command and searches for the file to include that has the referenced UUID
• zettlr incorporates the content of the included file in the original file
• zettlr passes the resulting modified file to the pandoc pipeline

What do you think?

[nathanlesage]

Well, we already had the issue proposing transclusion of docs (https://github.com/Zettlr/Zettlr/issues/354), so here's my stance on that:

• [[ and ]] (customizable) are already very much Zettlr-specific, and yes, Zettlr does have all the necessary code to support transclusion/embedding
• Hence, I think we should not add another identifier like [[[ and ]]] for that specific purpose, as file links should be replaced on export either way. Instead of removing the links, we could offer the settings option to actually include those links on export, which would – at least from my perspective right now – make even more sense as to simply remove the links.

Something like this:

I can see the following benefits:

• The process remains transparent to the user
• We do not need new syntax
• I cannot think of any situation where you only would want to embed some links, but not others

[canpolat]

This sounds like a good solution to me. And from what you say, the code change is also minimal 👍

[tubia]

Thanks @nathanlesage for the reply, I like the idea of having the possibility of transcluding the content of the links!

I also don't see cases in which one wants to keep certain links and include the content of other (maybe something will come to my mind in the next few days, since I am actively using Zettlr for writing a project).

Worlkaround research

Just to add something more at the conversation, I just found that there is an official lua-filter in pandoc that should support transclusion, but I still have to test it. It uses this identifier:

``` {.include}
chapters/introduction.md
chapters/methods.md
chapters/results.md
chapters/discussion.md

It's not clear if it takes also the front-matter or not. If the latter, should be easy, in my case, to move all the info related to each document in the front-matter (title, id, tags) - DCsunset/pandoc-include had the option to select or not the header but I wasn't able to use it.

So in terms of having transclusion while exporting using pandoc, although not integrated in zettlr, this seems to be a viable solution: I will let you now tomorrow as soon as I test it.

[tobiasdiez]

It would also be nice if zettlr could render the content of the included markdown in place. This is especially helpful if you want to reuse notes in a longer document, simulating a bit a similar feature of Roam Research. I was thinking about having a light gray background behind the included text, and a sentence like "Embedded from " at the end.

For this, the code-block syntax would be more suitable I guess.

[nathanlesage]

It would also be nice if zettlr could render the content of the included markdown in place. This is especially helpful if you want to reuse notes in a longer document, simulating a bit a similar feature of Roam Research. I was thinking about having a light gray background behind the included text, and a sentence like "Embedded from " at the end.

Well, as far as I know CodeMirror supports multiple documents stacked upon each other, but I am still a little bit afraid of tackling this … I want to get the editor run faster first! But then … I also know there was something with multiple editors on the horizon for side-by-side editing…

[tubia]

Hi,

a quick update: I managed to have this working with the official include-files pandoc lua-filter.

This is main .md file:

Some text

``` {.include}
C:\Users\admin\Desktop\Zettlr\zettelkasten\b.md
C:\Users\admin\Desktop\Zettlr\zettelkasten\c.md
```

Some other text

Notice that I added the full absolute path for the 2 included files.

And this is the export command for pandoc:

pandoc "$infile$" -f markdown $outflag$ $tpl$ $toc$ $tocdepth$ $standalone$ --pdf-engine=xelatex --mathjax --lua-filter="C:\Users\admin\AppData\Local\Pandoc\include-files.lua" --lua-filter="C:\Users\admin\AppData\Local\Pandoc\zotero.lua" $citeproc$ -o "$outfile$"

Notice that $citeproc$ is _after_ the two filters - the other lua-filter is the one to convert from markdown citation formatting to live citations for Word or LibreOffice made by Better BibTex for Zotero - in order to have it to process the references of the the included text.

EDIT:

So, to add to the proposal made by @nathanlesage, we could have for the UI the checkbox he proposed. Then at export time, if true, zettlr could:

• check the UUID of the file and get the file path
• substitute the [[ ]] line with the {.include} block, adding the path of the included file obtained in the previous step
  
  
  
  • (additional) check if in the same line there are more than one UUID and in that case, check for all the file paths and add them all together in the {.include} block
  
  
• set a variable $transclusion$ in the pandoc export command that will basically add --lua-filter=path_to_the_filter

[nathanlesage]

Thanks for the additional info. However I fail to see why we should add an extra filter. Zettlr already has the ability to transclude files inline, and your way seems overly complicated to achieve this…

[tubia]

@nathanlesage apologies, so is it already possible to just transclude files in Zettlr?

[nathanlesage]

Yes, at least for the codebase it's super easy to implement that (two, three additional function calls), so I guess it's also easier for the users as they don't have to set up some new filter :D

I just wondered, because I didn't know whether or not there were additional, more sophisticated benefits with regard to the filter!

[tobiasdiez]

A filter has the advantage that it would also work outside of zettlr, right?

[tubia]

@nathanlesage thanks! I thought to have overlooked a crucial feature for my workflow :D

Well, I don't know how much would be complex to implement the feature as I was suggesting with the filter, but maybe it's just a matter of packaging zettlr with that lua filter or add a procedure to install it in the documentation (it could be really just copy the file somewhere and then set up the path in Zettlr as we already do with pandoc).

Still, the file will not work outside of Zettlr due to the [[ ]] that are only used in the context of this software. But maybe being able to "export" it as vanilla markdown could do the trick.

[nathanlesage]

Still, the file will not work outside of Zettlr due to the [[ ]] that are only used in the context of this software. But maybe being able to "export" it as vanilla markdown could do the trick.

The thing is: Everyone can simply add this LUA filter themselves, if they want to use the (somewhat verbose) filter syntax. They just need to download the filter and modify the command accordingly.

However, we are only talking about transcluding stuff that is _represented within Zettlr itself with Zettlr-specific syntax anyway_, so implementing it directly in Zettlr would be the easiest way to handle this (and the most transparent, because it's hopefully clear for everybody that the internal links and the tags are purely Zettlr-internal features that just happen to work for other programs as well).

Basically this will lead to the following User Experience:

• You can still link stuff internally using the regular syntax
• If you yourself know "Hey I want to transclude that stuff on any export", Zettlr will do so if the option is activated
• If you want to transclude files _selectively_, or want to transclude stuff platform-independently, you can use this filter, and it will work out of the box regardless of other, purely internal settings. And it will also work outside of Zettlr

So, in effect we're talking about two different processes here: One, how Zettlr handles _its own syntax_, and two, how to add filters to perform stuff.

Which brings me to the thought that maybe we could add a documentation page outlining how to add and install (a) LUA-filters and (b) pre-compiled filters such as pandoc-citeproc (but only very shortly, because for the nitty gritty we must redirect users to the Pandoc manual, which will be more up-to-date either way, what's in the scope for the Zettlr docs will be how to integrate these filters into the export pipeline of Zettlr)

[cdaven]

I cannot think of any situation where you only would want to embed some links, but not others

I disagree. And for me, this has nothing to do with export, I want to embed notes in the UI.

Obsidian (also Markdown) supports both internal links, with the [[ ]] syntax, and embedding, with ![[ ]]. Why not? :-)

[GregSchofield]

Suggestion of markdown-include.

An extension of GitHub markdown --- it would be useful for me at least --- ref. markdown-include a Node extension.

#include "file.md" a more Zettlr form might be #include [filename]([[id]]).

This makes any link reference into an include.

If this is used as syntax we get a normal link if we click-it to the sub-document for editing so it can be edited, but when processed it is included. I quicklook of the document would include, but a new menu item to quicklook the snippit via the link would give a very smooth working environment.