Eleventy: Slug filter doesn't create url safe slugs.

[edwardhorsford]

I've been using the slug filter to convert regular strings to strings suitable for use as urls. However I've realised it's not set up to remove apostrophes (and possibly other characters?).

So it's a test becomes it's-a-test, and when rendered with eleventy the url is /tags/it's-a-test/ - the ampersand breaks the url. It also looks ugly!

I see that the package supports supplying a list of characters to remove.

If the intention is that this filter is used to create 'safe' urls, could Eleventy remove these by default?

---

As an alternative I've added my own filter using the slugify() option of the string library. I might stick with this anyway as it seems to do a good job of creating 'pretty' slugs.

[hughrun]

I just noticed a similar problem with commas in the title - it leaves the commas in for the automatically slugified permalink.

[zachleat]

Oh, hmm. Well that’s not ideal.

I did try permalink: '/{{ "Hi I''m Zach" | slug }}/' (note the escaped single quote using '') which resulted in the URL http://localhost:8080/hi-i'm-zach/ which worked in both Chrome and Firefox. Where is the conversion happening in your URL? It doesn’t seem to be coming from the slugify filter.

[edwardhorsford]

I think the point is more that I'd expect apostrophes to be be stripped by the filter. And that it would generally make 'nice' urls.

In my case this was most obvious when using automatic anchor links from headings - or from tags from tag pages.

[zachleat]

Well as much as I agree with you, this would be a breaking change and would require a major version bump. So I'm trying to figure out the exact impact of this to determine priority. It doesn't seem to be breaking anything in the test case I tried.

[edwardhorsford]

Oh totally! Quite a bad breaking change too. Even with a major version bump I suspect you'll want / need to offer a compatibility mode. Although I it only impacts a certain percentage of links, and users will likely have avoided using them, it has the potential to screw up lots of links.

Some thoughts:

• You could expose more of the config of the base package so users can strip these things if they want.
• You could offer a higher level config that users opt in to (and later make default, and offer a 'legacy' option)

FWIW, I think I prefer the string library's implementation anyway - to me it handles more edge cases nicer. I'll stick with my own filter anyway.

[jevets]

@edwardhorsford

Out of curiosity:

• Are you replacing the slug filter with your own implementation? (Does Eleventy allow replacing the built-in filter?) And still using the same filter name: {{ title | slug }}?

• Or are you using your own complete implementation, something like {{ title | edsSpecialSlugifyFilter }}?

I think it makes sense to expose config/options on the built in filter (like @edwardhorsford says above), opt-in only; keep the defaults as-is and avoid breaking changes.

[edwardhorsford]

@jevets the latter - a new filter with a new name. FWIW I went with slugify

I'm not sure the long term aim should be to keep as-is - though changing it presents issues and challenges. Fundamentally (to me at least), this filter is for making slugs / urls. It takes arbitrary string input and outputs something that can be used safely as a slug / url. I think you could argue that it's a bug that it doesn't currently do that - some arbitrary strings create slugs and urls that cannot be used.

Edit - here's the filter I'm now using:

const string = require('string')

module.exports = function(input) {
    if (!input) return false;
    else return string(input).slugify().toString();
}

[kazzkiq]

@edwardhorsford Did you managed to also generate the folders with this new slug function? If so, care to explain how?

I'm also having this issue where punctuation keeps adding noise to the URL, which looks odd and is awful for usability (imagine an user having to nitpick punctuations from the URL instead of just typing alphanumeric and dashes).

This:

mywebsite.com/posts/typescript:should-i-be-using-it,or-not

Versus this:

mywebsite.com/posts/typescript-should-i-be-using-it-or-not

What is easier to identify/read/type? Yeah... Medium seems to have an answer.

[edwardhorsford]

@kazzkiq I think you should be able to use custom filters in permalink generation - have you tried?

[kazzkiq]

Yep, it worked!

So just in case anyone else ends up here, here is how to fix your slug function:

1. Implement @edwardhorsford function slugify: (Doesn't know this file? Example here)

// .eleventy.js
eleventyConfig.addFilter('slugify', input => {
  if (!input) { return false };
  return string(input).slugify().toString();
});

1. Find/Replace every | slug in your project, to: | slugify.
2. Recompile your project (the new URLs may break older shared links, etc).

[zachleat]

Thanks for sharing @kazzkiq! I wonder if you could just use slug? Does it overwrite the built in filter? I feel like that should work if it doesn't

[okitavera]

yup @zachleat , it does replace the slug, so I just do remove some chars to make it little bit prettier tho

const slugify = require("slugify");
eleventyConfig.addFilter("slug", (input) => {
  const options = {
    replacement: "-",
    remove: /[&,+()$~%.'":*?<>{}]/g,
    lower: true
  };
  return slugify(input, options);
});

[zachleat]

Excellent, thank you @okitavera.

[zachleat]

I’m going to move this into the new feature queue and it is logged for the next major version milestone.

[zachleat]

This repository is now using lodash style issue management for enhancements. This means enhancement issues will now be closed instead of leaving them open.

View the enhancement backlog here. Don’t forget to upvote the top comment with 👍!

[bridgestew]

@okitavera Thanks for posting that solution. It works great for the post titles.

@zachleat or anyone: Is there a suggestion for how to make @okitavera 's solution work with markdown-it-anchor, as well? I'm not sure where to begin because I am a Javascript noob. Thanks!

[zachleat]

@bridgestew are you using eleventy-base-blog?

If so your opts defined here (docs: https://github.com/valeriangalliat/markdown-it-anchor):
https://github.com/11ty/eleventy-base-blog/blob/master/.eleventy.js#L43

  let opts = {
    permalink: true,
    permalinkClass: "direct-link",
    permalinkSymbol: "#"
};

would be something like:

const slugify = require("slugify");
let opts = {
    permalink: true,
    permalinkClass: "direct-link",
    permalinkSymbol: "#",

    // this is the same function shared above
    slugify: function(input) {
      const options = {
        replacement: "-",
        remove: /[&,+()$~%.'":*?<>{}]/g,
        lower: true
      };
      return slugify(input, options);
    }
};

Does that make sense?

[bridgestew]

@zachleat It does and it worked. Thank you so much for the help!

[rolandtoth]

This is only partly related, but I got an error if the frontmatter title contains : and the permalink is set like this:

"title": "Tip: Eleventy rocks",
"permalink": "/{{ title | slug }}/"

The error:

Problem writing eleventy templates (more info in DEBUG output): 
{ Error: dist/tip:-eleventy-rocks contains invalid WIN32 path characters.
    at Object.mkdirs (C:\Users\tpr\AppData\Roaming\npm\node_modules\@11ty\eleventy\node_modules\fs-extra\lib\mkdirs\mkdirs.js:18:22)
    at Object.mkdirs (C:\Users\tpr\AppData\Roaming\npm\node_modules\@11ty\eleventy\node_modules\universalify\index.js:5:67)
    at pathExists (C:\Users\tpr\AppData\Roaming\npm\node_modules\@11ty\eleventy\node_modules\fs-extra\lib\output\index.js:20:11)
    at fn.apply.then.r (C:\Users\tpr\AppData\Roaming\npm\node_modules\@11ty\eleventy\node_modules\universalify\index.js:23:46)
    at <anonymous> code: 'EINVAL' }

• Win 8.1
• Eleventy 0.7.1

[Ryuno-Ki]

Hm, experience told me, that _slugging_ is an area where it is better to rely on a library instead of rolling an own solution.
So I looked into what's available on npm and came up with a list:

• slug
• @sindresorhus/slugify - note, that eleventy is currently using slugify
• slugid
• limax

Maybe it makes sense to talk about the constrains here?
Like, which languages should be supported? What about numbers? Unicode (among them, emojis) etc.

[edwardhorsford]

@Ryuno-Ki I use the string library and have found it works well.

[nhoizey]

I chose @sindresorhus/slugify for all my slugify needs, it works perfectly, and I use all sorts of special characters in titles.

[Miosame]

Any fix to this issue will break previous "wrong" slugged outputs from pagination obviously. The issue itself is easily fixed by reading the readme of slugify, so instead of doing:

https://github.com/11ty/eleventy/blob/723ac17075748ae7efce2326417b95fb11b299b4/src/Filters/Slug.js#L4-L7

You'd do:

slugify(str, {
    replacement: "-",
    remove: /[*+~.()'"!:@]/g,
    lower: true
});

That fixes the issue at hand, not sure what's going on with #26 though, that's another story.

[mbajkowski]

There is a special case of the above I ran into the other day on win10 dev environment. Because the the current slug filter does not remove dots from titles, if your title happens to have a one at the end, and you use titles for your URLs you can end up with something like: this-is-my-title./index.html

Having a dot at the end of a directory name or file turns out to be problematic in windows, and requires a rather roundabout way of deleting it: https://stackoverflow.com/questions/4075753/how-to-delete-a-folder-that-name-ended-with-a-dot

Figured some may find this useful if they have cleanup scripts to blow away the _site directory, and those scripts are suddenly failing on windows for unknown reasons, this could be one of them.

[Ryuno-Ki]

@mbajkowski Interesting finding!
As stated above, Eleventy is using a library. Would you mind to open an issue there to get it fixed upstream?

[mbajkowski]

@Ryuno-Ki, I thought eleventy uses https://www.npmjs.com/package/slugify, is this changing going forward? I'm not sure removing the dot ought to be default behavior as it could break existing URLs, but just something people need to be aware off - maybe something that needs to go into the knows pitfall in the documentation section when using windows. Easily fixed via the code shown by @okitavera

[Ryuno-Ki]

Hm, I checked again. You're right. I don't know what I looked at back then.

[Miosame]

@mbajkowski @Ryuno-Ki as stated above the slugify library that eleventy uses has in its readme how this issue is solved. (including the dot issue on windows)

the only issue is that it can't be solved without breaking people's existing generated slugs.

so is this a forever to be bug? and people will have to workaround with their own filter to have _real URL safe slugs_?

[kazzkiq]

Breaking changes should only be added on major version bumps.

11ty has 17k npm installs per week. You can only imagine the mad mob that would show up here if the slug function changed and broke their projects due to a npm install.

Perhaps it should be "fixed" on v1.0.0

[nhoizey]

I think this change is required (I would recommend @sindresorhus/slugify, stable and well maintained) to make people's life much easier in the future, with probably many more users than today.

One option to deal with it would be to release a 0.12.0 version with some deprecation/breaking change warnings (not only the slug, I think there are other changes) in the console, before a 1.0 version that really changes these.

This 0.12.0 version could already includes the new slugify function, just to make sure the warnings are shown only if the result is different from the current one.

[Miosame]

@nhoizey good idea, though it'd be much easier to check for the "to be filtered" characters instead of compiling duplicate states or even including the new filter at all yet, so if it contains any of the chars throw a warning that in the near future that will change. (incl. in quiet exec maybe? because it seems most people run it in quiet and will never get to see it otherwise)

[nhoizey]

Checking for the "to be filtered" characters would mean dive into the new slugify function and make sure it stays synced with any change in it.

Doesn't feel really safe IMHO, even for a short time period.

[Miosame]

@nhoizey assuming eleventy would stay with the same library it is now, that library isn't opinionated and you control entirely which characters to filter, the readme itself only gives a sane suggestion for most characters that don't belong into slugs - if there's need to change it, then you change what the warning triggers on, can make it an internally available variable too, so both the warning and the slugify filter (later) access the same definition of chars.

That way up until the actual implementation you can gather feedback safely on if said characters are enough.

[nhoizey]

@Miosame you're right, it might be enough.

I just wish the defaults would be more safe, but Eleventy can make it so.