Foam: Generating markdown link reference definitions ?

[steveklabnik]

Hey there, thanks for foam!

The docs say: https://foambubble.github.io/foam/wiki-links

The Foam for VSCode extension automatically generates markdown link reference definitions at the bottom of the file to make wiki-links compatible with Markdown tools and parsers.

If you look at link references the bottom of any Foam workspace file that uses wiki-links, you should see an automatically generated list of references that look as follows:

[wiki-links]: wiki-links "Wiki Links"
[other-page]: other-page "Other Page"
These exist to make [[wiki-links]] compatible with Markdown-consuming tools such as static site generators, VS Code plugins etc.

However, I am not seeing these links get generated. I'm not seeing any errors, either. Do I need to configure something, or press a key or something? Thanks!

[jevakallio]

Hey Steve! Try running the "Update Markdown Links" command manually -- it should happen automatically when you're in a markdown document inside a Foam workspace (a workspace that has .vscode/foam.json)

The references are generated:

• On file save,
• When file has at least one [[wiki-link]],
• To a file that exists, i.e. wiki-link.md
• That has a top level # Heading

If it still doesn't work, make sure you've reloaded/restarted VSCode since installing the foam-vscode extension.

I know that's.a lot of edge cases that could fail, I'm looking to simplify this in the coming days. Alpha software troubles.

If it still doesn't work, please give me an example workspace and I'll try to repro

Thanks for reporting!

[steveklabnik]

Ah ha! I figured it out!

I tend to use [[weekly review]], rather than [[weekly-review]]. Without the dash, they don't get generated, but with the dash, they do!

I am not sure if that makes this a bug or not :)

[jevakallio]

@steveklabnik interesting! Files with spaces have worked for me in the past. I don't recommend them but they should anyway...

Will test

[digiguru]

Hi folks. Loving this project, and I want to help.

I've spotted a bug closely related to this one with links with spaced. It turns out case sensitivity is also important.

If I add a link called [[Review]] and ctrl+Click it we see Foam automatically generates review.md as it should.

Now if I look at the auto-generated links in the original file it doesn't create a link to review.md. I'd expect a line saying

[Review]: Review "review"

I think it spawns from the id parameter of the Note data object matches the filename in the parsed markdown objects from addNoteFromMarkdown (so it will be review in my example).

https://github.com/foambubble/foam/blob/af7af05e9d8d837c14159c783af138ad66ab6ceb/packages/foam-workspace-manager/src/WorkspaceManager.ts#L107

But the list of IDs in the file comes from parseNoteLinksFromMarkdown which honours what the user has typed in the original file. (Which will be Review in my example)

This is then matched in a Map.has()
https://github.com/foambubble/foam/blob/af7af05e9d8d837c14159c783af138ad66ab6ceb/packages/foam-workspace-manager/src/WorkspaceManager.ts#L37

To fix this we could....

• Create a special Map to allow case insensitivity and space detection and use the index object
• Amend wiki links that are being processed and pretend they are actually lowercase and fix spaces as they are parsed in the document itself in findWikilinksInMarkdown.

Neither option seems very palatable to me, so I'd like guidance on what would be preferred.

[digiguru]

To be clear, I'm happy to submit a pull request but as @jevakallio explains

I'm looking to simplify this in the coming days

I am happy to take guidance and aid in this simplification effort, or fix using a preferred method. Feel free to reach out.

[jevakallio]

Thank you for the detective work @digiguru, you're absolutely correct! I don't have a perfect solution in mind yet, but in order to organise my own thoughts and get you and @jsjoeio (who reported https://github.com/foambubble/foam/issues/68) on the same page, here's the current circumstance.

Markdown Notes

On one hand, we have the Markdown Notes extension which is quite permissive, and allows both spacing and variable casing in [[links]], and normalises them to lower-kebab-case.

• [[My Big Idea]] -> my-big-idea.md
• [[my-big-idea]] -> my-big-idea.md

It also allows for slashes, but normalises them to dashes, not directories:

• [[my/big/idea]] -> my-big-idea.md

I only discovered these through other people's bug reports, because my note taking habit just happened to be kebab-cased all along. I am happy with the Markdown Notes's behaviour in this case, but if others feel strongly and can add broader support to that extension, I'd be up for supporting it too. Wonder if @kortina has opinions on this?

Only real issue with Markdown Notes are:

• It also allows for file extension, e.g. [[my-big-idea.md]] (and auto-completes for it, which is never what I want
• If you were to try to create a file with the .md extension in the link, you'd get into trouble
  
  
  
  • [[my-big-idea.md]] when file already exists, links to my-big-idea.md
  
  
  • [[my-big-idea.md]] when file doesn't exist creates my-big-idea-md.md
  
  

Foam Reference Link Generation

So, right now the Foam reference link generation is a little broken for cases other than the vanilla kebab case. I've been holding off from fixing it because it's not an inherently complex problem, it's only accidentally complex that way I implemented the reference link generation:

1. We ingest the documents and their links both up front on startup, and when a file is saved:
   https://github.com/foambubble/foam/blob/af7af05e9d8d837c14159c783af138ad66ab6ceb/packages/foam-workspace-manager/src/WorkspaceManager.ts#L107

2. We store the notes metadata and links in a normalised format where we don't really store any information about the edges, e.g. what format the [[link SyNtAx wAS-in.md]]:
   https://github.com/foambubble/foam/blob/af7af05e9d8d837c14159c783af138ad66ab6ceb/packages/foam-workspace-manager/src/WorkspaceManager.ts#L12-L33

3. When creating reference link lists, we look up the linked files from said index and not directly from the file, so we don't have access to the exact formatting and can't put it to the reference list:
   https://github.com/foambubble/foam/blob/af7af05e9d8d837c14159c783af138ad66ab6ceb/packages/foam-vscode/src/extension.ts#L137

4. To fix this, the easiest way would probably be to re-parse the links from the original markdown (which we have access to literally the line above!).
   https://github.com/foambubble/foam/blob/af7af05e9d8d837c14159c783af138ad66ab6ceb/packages/foam-vscode/src/extension.ts#L134

5. We could approach this either by:

   • Add a method to foam-workspace-manager that exposes the findMarkdownLinks behaviour to just grab us the link ids and returns them along with their Note entities based on the "normal form" of the ID
   • Changing the notes data structure map to contain information about the edges between them

@digiguru if this helps you to take the issue, I'd love a PR to fix it! Both above approaches listed in point 5 would be acceptable to me.

The reason I haven't done this work yet is that I was hoping to rewrite large parts of VS Code extension code to support Workspace Janitor, which is my next big feature, but so far I haven't found the time to do it and we shouldn't block on my schedule with this!

[jevakallio]

FYI I just landed https://github.com/foambubble/foam/pull/67 which should make adding functionality to foam-workspace-manager a lot easier, so sync your fork if you intend to work on this!

[ztratar]

Just want to say I love the active development on this, and ya'll are building something great. :)

[digiguru]

Looks like the following accent issue is related to this...

48

You can fix it for accentuated & whitespace characters fairly trivially:

// First get a "difficult" string
let str = "ÀÁÂÃÄÅ àáâãäå, ÒÓÔÕÕÖ-òóôõöøÈÉÊË_èéêëÇçÌÍÎÏìíîïÙÚÛÜùúûüÑñŠšŸÿýŽž";
// Now use normalise("NFD"), and replace characters with non-accented ones, lowercase everything, remove all punctuation
str = str.normalize("NFD").replace(/[\u0300-\u036f]/g, "").toLowerCase().replace(/[^a-z\d]/g,"-")
// outputs
// "aaaaaa-aaaaaa--oooooo-ooooo-eeee-eeeecciiiiiiiiuuuuuuuunnssyyyzz"

Which is "good" (if you don't like to see accents in filenames). Given the upset it could cause I would wrap this in a setting with a default to what we think the majority of users would expect, and the ability to opt out.

Arguably we should also deduplicate all consecutive hyphens in the string.

So...

"---this---is--a--string".replace(/[-]+/g, '-')
// outputs
// "-this-is-a-string"

Which I feel reads nicer, particularly if someone puts two punctuation marks together.

HOWEVER...

This solution cannot handle certain characters like ø which it transpires is not accented (in this case it's a danish vowel).

This means non-latin alphabets will be treated as hyphens - which won't be a great user experience - particularly for #73

[digiguru]

I've added a potential fix downstream, but there's still work to be done once it's been approved and merged.

kortina/vscode-markdown-notes#53

[digiguru]

The downstream fix has been resolved.

However - sorry folks, I'm not going to have time to explore the mirroring changes in this for another week. Feel free to pick it up if it's a burning issue.

I guess to work on this you need to really be debugging the extension, so this should help

And of course understanding how to compile the extension will also be necessary.

[digiguru]

I had a little time so made a little progress. I have added more tests to the link parser to play with a new concept that's documented in #93

[cpopell]

I'm running into these issues too. This is tremendously elegant way of solving situations that don't support [[]] format. Unfortunately, all of my stuff is sentence cased + spacing. The actual file names are (e.g.) "Summa Theologica.md". Right now I'm using Obsidian, where that's not a problem, but prior to this I used VSCode for everything so I have a hope I can keep using it. That said, capitals is only an issue for me in display name - the normalizing into lowercase-dash and then just renaming my notes would be helpful.

[shkdmhhb]

@jevakallio, for the link definitions, is there a setting for the auto generated links to include the ".md" extension?

Atom editor has this terrific extension, https://atom.io/packages/markdown-mindmap, that i used occasionally for my MD files for visual naivgation. Unfortunately, without the ".md" extension, it literally tries to open the file without the ".md".

[shkdmhhb]

nvm, show the new setting added in #133 .