Marked: Extending with custom tags

[FireflyAndStars]

I want to create custom "tags", like text alignment blocks, inline text color, etc.
Documentation is not very clear to me how to create totally new tags or tokens.

There are other examples?

[UziTech]

you will have to modify the lexer and parser if you want to create your own tokens.

Example from the docs

This is not something that is done easily. There have been times where a fix in marked has created a bug in someones custom parser.

[FireflyAndStars]

Maintainability is not a problem if I understand how the extension system exactly works.

Should be good to have a basic example of all steps (renderer, parser, lexer, etc) to create new inline and block tokens in the documentation.

Currently, I'm analyzing the source code to understand it, because the documentation seems incomplete to me.

[UziTech]

The documentation is incomplete, but we are always accepting pull requests 😁

[FireflyAndStars]

You're right, but my knowledge is too low/insufficient for this right now :sweat_smile:

[solugebefola]

Follow up question here...I have seen examples of how to modify the inlineLexer, but how to modify the main lexer with custom rules? I don't see how to add custom tokens without completely rewriting theLexer.prototype.tokens function?

[UziTech]

@solugebefola Right now to modify the lexer you will need to rewrite Lexer.prototype.tokens. A PR would be nice to be able to modify the Lexer easier.

[solugebefola]

Was there a PR to add this feature? Or just closing b/c no activity?

[UziTech]

Sorry I was closing all of the NFE issues since they weren't going to result in a change in this repo. This issue should be listed as docs changes.

There have not been any PRs to address this.

[horacioh]

can the parser and lexer implement a similar api as the renderer to make them more extensable?

[UziTech]

@horacioh that would be great if they did. PRs are always welcome. 😁

[calculuschild]

I might work on a PR for this. There are a couple of lexer elements I want to customize how tokens are generated (tables) but I can't without rewriting the whole function.

Sounds like it would be in line with the Marked architecture if I separate each lexer token into its own named item like it is in the renderer?

[UziTech]

@calculuschild That would be great! Just make sure that creating separate functions doesn't slow it down too much. You can run npm run bench to run benchmark tests against marked and other markdown libraries.

I am also working on creating inline tokens before sending them to the parser in PR #1627

[calculuschild]

@UziTech Alright, got an initial attempt at extending the Lexer.