Pelican: Add Markdown as an (optional) dependency

[justinmayer]

Since its inception, this project has taken the collective position that since not everyone uses Markdown, the markdown package should not be a dependency of the project and should instead be manually installed by users who want to use Markdown.

On the other hand, the docutils package that parses reStructuredText markup has always been a dependency. Particularly given Markdown's widespread popularity, I cannot come up with a compelling argument to support the continuation of this discrepancy.

This project is littered with issues caused by environments with missing markdown packages. Many folks have encountered frustration when trying to build their Markdown-content sites, only to be met with an (often inscrutable) error message. And that's just the folks who have taken the time to file issues! Imagine how many threw up their hands and just moved on to another solution…

Adding Markdown as a dependency would have significant benefits, including simplifying installation instructions, reducing frustration, and minimizing support request load. So to take another approach at this topic… What are the reasons for not adding Markdown as a dependency, and do you think they are sufficiently important to out-weigh the concerns raised above?

[justinmayer]

I'm interested in feedback on this topic from Pelican users and @getpelican/reviewers.

On a related side-note, I just added a new pyproject.toml file to the project via #2586, and in it markdown is currently included alongside the other Pelican dependencies.

[scarpent]

+1 to adding the dependency

[acidjunk]

if it solves a lot of problems for new(er) users: add it.

[martinronn]

+1 to adding the dependency from me too. Pelican should be ready use out of the box for any user.

[iKevinY]

Another +1 for including Markdown as a dependency from me.

[mcepl]

Particularly given Markdown's widespread popularity, I cannot come up with a compelling argument to support the continuation of this discrepancy.

It is very easy, I (and many others) prefer reStructuredText to Markdown, and don’t feel the need of having an useless package installed. Perhaps make docutils optional as well?

[justinmayer]

@mcepl: Thank you for your input. But is a few kilobytes of disk space important enough to outweigh the aforementioned usability disadvantages? I'm looking for _compelling_ reasons, and so far I've not heard any.

[bittner]

IIUC, there are two aspects of the problem:

only to be met with an (often inscrutable) error message.

1. Pelican supports various markups (reStructuredText, Markdown, and AsciiDoc according to the Pelican website), and
2. The error messages are not obvious, little helpful, don't tell the user what to do to fix the issue.

Shouldn't we simply fix (2) and make all non-strict dependencies optional?

Make errors user-friendly

I can envision two scenarios:

_a_) A user (blindly) installs Pelican, faces an error when compiling, and Pelican tells them what to do:

$ pelican
ERROR: Pelican could not compile your source files. You seem to have Markdown
files in the "content" folder. Please install the "markdown" Python package
to enable Markdown for Pelican, e.g. via

    pip install pelican[markdown]

See https://docs.getpelican.com/en/stable/install.html for details.

_b_) A user reads the "Installing Pelican" section in the documentation and chooses to install the required dependencies upfront. It could read like this:

Install Pelican with the support of the markup language you want to use (reStructuredText, Markdown, or AsciiDoc):

• pip install pelican[restructuredtext] or
• pip install pelican[markdown] or
• pip install pelican[asciidoc]

If you install Pelican without markup language support it will remind you later (with a compile error) to install the required flavor.

Make Pelican a slim package

In addition, when making docutils optional, I'd suggest to do a major version bump, drop Python 2 support and all related packages (e.g. six).

Instead of going the "bloatware for convenience" way we would then go the clean, minimal dependencies "quality software" way. We'll ensure the convenience by user-friendly errors instead. :nerd_face:

People wanting to continue on Python 2.7 could be suggested to stick to using Pelican<5.0. That would be a good _"Use Python 3, the good Python"_ showcase, following the Django example.

[oulenz]

I think this is a case where the benefit of removing a (few) small dependency/ies is not worth the reduced user friendliness. Am I correct in assuming that advanced users who know what they are doing could still manually overrule the installation of markdown?

[avaris]

As far as I know, you can't (easily) avoid a specific dependency. pip has --no-deps option but that's kind of the nuclear equivalent. It prevents all dependencies. You would need a hand crafted requirements.txt with particular dependency removed and use it in combination with --no-deps.

EDIT: Just to add my 2-cents. I don't mind making Markdown a regular dependency but I'd prefer it being an "extra" dependency instead (pip install pelican[markdown]).

And some clarifications:

Pelican supports various markups (reStructuredText, Markdown, and AsciiDoc according to the Pelican website), and

Actually, AsciiDoc isn't in the core anymore. It's a plugin and if we're counting those there are a couple more formats that could be added :).

The error messages are not obvious, little helpful, don't tell the user what to do to fix the issue.
Shouldn't we simply fix (2) and make all non-strict dependencies optional?

There is no error message, normally. If a reader isn't enabled for a particular extension, that file is basically invisible in terms of content (could still be a static file though). Only relevant "error" you get would be: if you write all your content for a disabled reader and run pelican, it will inform you that no content files (it can process) were found.

Trying to conjure error messages for these cases involves guesstimating user intention and that is a tricky undertaking. How would you differentiate a user that forgot to install Markdown and a user that puts a .md file in content for static file purposes only?

Make Pelican a slim package

It is more or less like that. But in the spirit of having "some batteries included", it auto includes rst. Doing pip install pelican and not being able to do anything with it right after without installing some other package is weird.

[shmcgrath]

I think Markdown should be included. The Quickstart docs have the first example written in Markdown. I know the docs say to “pip install pelican markdown” directly above the Markdown example, but it feels odd to me that the first example is something that isn’t included as a dependency. If the community chooses not to include Markdown, would it be worth updating the docs to have the Quickstart example to be a ReStructured Text example since those batteries are included?

[justinmayer]

Thanks everyone for taking the time to contribute your thoughts. Truly much appreciated.

For those who prefer Markdown to be an optional dependency... I am still not hearing _why._

I'm not saying I feel strongly about this. On the contrary — I don't. But I am simply not hearing _any_ real reasons for keeping Markdown as an optional dependency. To put this in perspective, we are talking about an 88-kilobyte wheel file. Is it just the principle of installing something you don't need? Because if that's the case, I have something to tell you about the gigabytes of files your operating system itself has already installed on your primary drive that you have probably never used. :wink:

So, please... Help me understand.

[florimondmanca]

Hi!

Building on @avaris’ two cents, why not have both docutils and Markdown optional dependencies, and let users choose which format they prefer at install time?

pip install pelican[rst]
pip install pelican[markdown]

This would require updating the docs (installation guide) to clarify that no format is supported by default.

By I would argue that this increases the number of choices users must make before getting started, thus raising the barrier to entry, which I don’t think is desirable.

So I’d rather be in favor of building everything in, so that users can just install pelican and get started right away with the format they prefer. These are my own two cents. :)

Edit: to be clear, I just can’t write anything in RST, and use Markdown in so many other places that not having Markdown builtin is definitely a pain point with adopting Pelican.

[sio]

Not having markdown as the dependency is literally the only reason I have to include the requirements.txt file in each of pelican projects directories. +1 for the idea

[davidag]

A light search on what other static site generators do shows that two of the top three, Hugo and Jekyll, support Markdown by default, while Next.js doesn't (though, it seems to be a bit different in philosophy in respect to the others).

In my opinion, all that favors user adoption and ease of use is more than welcome. If it reduces repeating issues and building problems, even better.

I can't find any good reason not to have both docutils and markdown as Pelican dependencies.

[mcepl]

A light search on what other static site generators do shows that two of the top three, Hugo and Jekyll, support Markdown by default, while Next.js doesn't (though, it seems to be a bit different in philosophy in respect to the others).

Will you ask all those other static generators to include docutils as their dependency as well?

[justinmayer]

Will you ask all those other static generators to include docutils as their dependency as well?

Do well-tested reST parsers exist for Ruby, Go, and JavaScript? If not, I'm not sure this is a fair comparison.

[justinmayer]

@avaris said:

I don't mind making Markdown a regular dependency but I'd prefer it being an "extra" dependency instead (pip install pelican[markdown]).

For the moment, I'm fine with this compromise, and I already made that call a while back when I added Markdown as an optional dependency in pyproject.toml.

Would someone please volunteer to make a corresponding setuptools-based change to the add Markdown as an extra (optional) dependency?

[justinmayer]

Thanks to @davidag, Markdown has been added as an optional dependency via the above PR. :sparkles:

[justinmayer]

This is included in the just-released Pelican v4.1.3, courtesy of AutoPub.