Remark: Supporting only a subset of the syntax in certain tokenizers / changing defaults

[renekliment]

Hi!

Subject of the feature

We would like to do these things:

1. Support ATX headings only up to certain level. Currently hard-coded in MAX_ATX_COUNT private module variable.
2. Support strong only using **, not __.
3. Don't support <https://thisSyntaxFor.links>. ah, disabling autoLink now works, not sure why it didn't before

Problem

The current tokenizers don't allow to do this AFAIK.
We could easily take the current tokenizers as base and write our own, but that would not be very maintainable.

Would it make sense to have all these things changeable via options?
We could probably stich-up a pull request to have this upstream.

Thanks.

[renekliment]

Ah, I see the options are parser options (not tokenizer options) and one cannot add arbitrary keys to it. So this seems like a no-go. Am I correct?

[wooorm]

Why these “arbitrary” rules of what is and isn’t valid Markdown?

[renekliment]

@wooorm It's not about considering stuff invalid Markdown.

The use-case is content managed by a group of people and even though you can set rules, you want to enforce it by the app & ignore things you don't support by your app, even though it is a valid Markdown.

1. So for headings, for SEO reasons and keeping it simple / well aranged for our users / content consumers, you might want only a certain depth.
2. This actually turned out to be optional for our use-case. The original reason behind it:
   
   
   
   1. We have a function that strips our subset of Markdown on-the-fly and the less it has to support, the better.
   
   
   2. The less things / ambuguity for our content managers to learn / have, the better, so the source codes have as much consistency as possible.
   
   

[kmck]

This is already possible, but it's a little tricky. I did something similar when I wanted to repurpose __ for underlined text rather than strong text.

Basically you can add custom tokenizer options as a plugin to the unified processor. It works something like this:

import unified from 'unified';
import parse from 'remark-parse';

// this function is use()-ed by the unified processor to add custom parser behavior.
// it needs a normal function so you can access the parser via the context (this).
function customParserExtensions({/* custom parser options */}) {
  const { Parser: { prototype: parserProto } } = this;
  const {
    blockMethods,
    blockTokenizers,
    inlineMethods,
    inlineTokenizers,
  } = parserProto;

  // ADDING A CUSTOM INLINE TOKENIZER
  // 1. define a tokenizer function for the custom key
  inlineTokenizers.underline = tokenizeUnderline;
  // 2. insert the custom key into the list of tokenizers.
  // they are evaluated in order, so we splice "underline"  in before "strong", which
  // ensures that __ is processed as "underline" before the normal behavior processes it as "strong"
  inlineMethods.splice(inlineMethods.indexOf('strong'), 0, 'underline');

  // REMOVE A DEFAULT INLINE TOKENIZER
  // just filter it out and it will never run, even though the tokenizer will still exist in inlineTokenizers
  parserProto.inlineMethods = inlineMethods.filter(type => (type === 'autoLink' !== false));

  // REMOVE A DEFAULT BLOCK TOKENIZER
  // filter it out of the method list
  parserProto.blockMethods = blockMethods.filter(type => (type === 'list' !== false));
  // block methods can be nested, so there can be other references to the tokenizer (i think?)
  // replacing the tokenizer with a no-op helps to avoid unexpected behavior that filtering the
  // method list alone doesn't catch
  blockTokenizers.list = passthruTokenize;

  // REPLACING A BLOCK TOKENIZER
  blockTokenizers.atxHeading = customAtxHeadingTokenize;
}


// NO-OP TOKENIZER
// it looks like a tokenizer, but does absolutely nothing.
function passthruTokenize() {
  return undefined;
}
passthruTokenize.locator = () => -1;

// CUSTOM UNDERLINE TOKENIZER EXAMPLE
// for the sake of simplicity, this isn't the most robust implementation.
// it's not handling escape characters, for example.
function tokenizeUnderline(eat, value, silent) {
  const [match, text] = value.match(/^__[^_\s]__/) || [undefined, undefined];
  if (match) {
    if (silent) {
      return true;
    }

    const now = eat.now();
    now.column += 2;
    now.offset += 2;

    return eat(match)({
      type: 'underline',
      children: this.tokenizeInline(text, now),
    });
  }

  return undefined;
}
tokenizeUnderline.locator = (value, fromIndex) => value.indexOf('__', fromIndex);

// CUSTOM ATX HEADER
function customAtxHeadingTokenize(eat, value, silent) {
  // copy from default implementation:
  // https://github.com/remarkjs/remark/blob/master/packages/remark-parse/lib/tokenize/heading-atx.js
  // and modify as needed
}

// TELL REMARK PROCESSOR TO USE CUSTOM PARSER EXTENSIONS
const processor = unified()
  .use(remarkParse, {/* remark options */})
  .use(customParserExtensions, {/* custom parser options */});

Using the code above as a base, here's how I'd implement what you're trying to do:

1. Replace the blockTokenizers.atxHeading with a custom implementation. For example, you could customize the default atxHeading tokenizer with your own max heading number.
2. In the example above, I intercept the syntax for a different type of formatting. If you just don't want to use it for something else but just want to disable it for the strong parser, you could replace inlineTokenizers.strong with a custom implementation. Easiest thing is probably to copy the default strong tokenizer and remove the stuff about underscores.
3. Seems like you got this working, but you can also splice it out of inlineMethods and/or no-op the inlineTokenizers.autoLink.

[wooorm]

I think the ways Keith outlined are fine. You can already do what you want.
When micromark comes, we’ll have another opportunity to look at how to turn on (and off) constructs, but I think this is fine for now.