The-turing-way: CI step to check for dead links in markdown files

[JimMadge]

Summary

I've spent a little time looking at dead links in the book and repository. Thinking about how to manage fixing existing dead links and helping to reducing the introduction of dead links I came across two projects with GitHub actions.

| Software | Action |
| --- | --- |
| markdown-link-check | github-action-markdown-link-check
| liche | Link Checker |

(note: I couldn't (after a brief attempt) build liche as it appears to have dependencies which no longer exist or have moved)

I think there are barriers to using one of these actions in a way where they block merges,

• There are currently quite a few dead links of various types (typos, file inside repo moved, resource moves, false negatives) which should probably be corrected in more than one commit
• Checking every link is time consuming
• The tools are not flawless. markdown-lint-check, for example, cannot handle urls with unicode characters
• Internet resources can move or disappear outside of the control of a contributor. It could make contributing harder if a contributor or reviewer has to fix links unrelated to an issue or PR

It might still be helpful to run a CI step,

• A step which passes even if there are dead links could be used to help enumerate and identify dead links
• github-action-markdown-link-check has an option to only check modified files, which could help to prevent introducing new dead links and remove existing dead links when files are updated

What needs to be done?

• [ ] Decide if a CI step for dead links would be useful
  
  
  
  • [ ] How should the step should function (blocking/non-blocking, all files/modified files)
  
  
  • [ ] Consider the available tools to find the most suitable
  
  

Who can help?

---

Updates

[KirstieJane]

I'm super into a checker for dead links! Particularly if they can check for internally broken links... I feel like every time we re-structure a chapter etc then we run the risk of breaking internal links.

@sgibson91 - what do you think?

[Remi-Gau]

Just jumping in to give you my 2 cents about the markdown-linkcheck github actiion: in case it helps make a decision.

I started using when I was refactoring a lot of the doc on one my project and that started breaking internal links "up the wazoo". And I noticed a couple of dead external links too.

So I started using markdown-linkcheck.

Overall it does a very good job: works on external, internal links and headings to a section in the same file.

It produces a report that really helps sieving through all your files and their links.

It does give some "false positive" once in a while and will report a link as dead when it is not. But the report allows you to check whether it is one of those living-dead links (:zombie: :zombie_man: :zombie_woman:) by simply clicking it.

So unless you feel bad about merging things when your continuous integration does not give you green lights all around (I do but I am working on it), this is actually quite a useful action (I promise I am being paid to promote it. :wink: )

For an example of a report see here.

[sgibson91]

@JimMadge are you suggesting an extra link-checking step to the one we already run here? The problem with checking for dead links in the markdown is that won't actually catch dead internal links in the book since Jupyter Book uses a label referencing system different to markdown (see here). This is why we run our link check on the rendered html, not the markdown.

cc: @martinagvilas

[JimMadge]

@sgibson91 I didn't notice that CI step 🤦...

I suppose the question then is, would an extra step do anything the existing check does not or can not do?

Definitely agree that the references being missed is a drawback and that testing on the html that will actually be deployed is better.

I've recently tried incorporating markdown-link-check into a project. My impressions are the same as @Remi-Gau's above.

[sgibson91]

I suppose the question then is, would an extra step do anything the existing check does not or can not do?

I'm not sure it would. Checking links in markdown would likely reliably catch external links ... And I've just realised that we have --disable-external enabled for html proofer. @martinagvilas what's the thinking behind that?

[martinagvilas]

Hey! So indeed, the CI already checks for broken internal links. It has been set up for not failing when it finds broken links, so that instead of fixing them all on one go ourselves we could open issues for new contributors. Side note: Jupyter Book also kind of checks for internal links (see the build log, some of the warnings are showing broken links).

@sgibson91 the --disable-external was carried over from the previous CI. I just kept it as it was. I honestly don't remember if at some point I read that the idea was to avoid that the CI takes a lot of time (checking for external links seemed to take a while), or if there was an issue with this particular tool and Jupyter Book. Regardless, I think introducing this check will be great!

I'm not familiar with the tools mentioned here, but I just want to point out that Jupyter Book already has a built-in option for checking for external broken links. To add it we need to add -linkcheck option to the build command (see here). I don't know how many false positive this option gives, but it might be the cleanest solution as it will never be inconsistent with the build of the book.

[sgibson91]

Thanks for that explanation! I wonder then if we could simplify and keep all link-checking (internal and external) inside the jupyterbook build command? Or do you think it's best to also keep html proofer?

[sgibson91]

It has been set up for not failing when it finds broken links, so that instead of fixing them all on one go ourselves we could open issues for new contributors.

Does such an issue exist? Should we spend some time (a Collab Cafe?) creating one? There's this issue but it's currently empty

[martinagvilas]

Thanks for that explanation! I wonder then if we could simplify and keep all link-checking (internal and external) inside the jupyterbook build command? Or do you think it's best to also keep html proofer?

I would totally go for this option! Especially since these checks are always improving in Jupyter Book, and we are not checking the output of html proofer anyways

Does such an issue exist? Should we spend some time (a Collab Cafe) creating one? There's this issue but it's currently empty

Oh yes, I've been saying that I will work on this on the collab cafe for a while but I have not been able to attend one since the Jupyter Book tutorial 😭 I also wanted to create a file that explains what each of the warnings of Jupyter Book means (they can be quite cryptic) so that people can fix them whenever a new one appears. So yes, I'm definitely in for working together on this next time!

[sgibson91]

Perhaps @JimMadge would like to join too? details

[jezcope]

Just to add to this, because I've just been running jupyter-book locally and seen that there are a lot of warnings during the build process: jupyter-book build has an option --warningiserror that will cause the build to treat any warnings as errors and abort, which seems like it would be useful for a CI check of "make sure there are no warnings". Most of the current warnings are indeed missing internal links, plus a few minor syntax errors. It also flags that there are some entries in references.bib that aren't being used (and some citations that don't have an entry there).

Along with --warningiserror there's a --keep-going flag that will run the whole process and find all the warnings before aborting. So a CI step that runs jupyter-book build --warningiserror --keep-going . might do the trick?

[sgibson91]

there's a --keep-going flag that will run the whole process and find _all_ the warnings before aborting.

More things in the world need this.

[jezcope]

More things in the world need this.

I know I need this...