Githawk: Markdown issues

[rizwankce]

From the app

From the browser

Edit:

Link to the github PR https://github.com/artsy/artsy.github.io/pull/423
_{Sent with GitHawk}

[SD10]

This may be a naive question but I'm genuinely curious about the state of our markdown parser dependency. Would a better solution be to wrap GitHub's fork of CMark in a Swift API?

[rnystrom]

@SD10 any details on it? I haven’t seen it before. I remember investigating cmark before and it wasn’t an option, can’t remember the specifics.

Note that our parsing lib was written by a GitHub engineer originally.

edit: found it, looks interesting! I haven't seen this before. I'll do some digging.

edit2: here's the gfm spec

_{Sent with GitHawk}

[SD10]

@rnystrom Yeah the GFM spec is amazingly detailed. I was considering writing a parser of my own to follow that spec in Swift but decided to spend my time other places.

If I could wrap GitHub's fork for Swift that would save me from having to reinvent the wheel. (and it would be far more accurate, likely faster even)

GitHub CMark Fork:

Fast. cmark can render a Markdown version of War and Peace in the blink of an eye (127 milliseconds on a ten year old laptop, vs. 100-400 milliseconds for an eye blink).

But if you said CMark wasn't an option it may not be possible. Maybe leveraging it in another tool? I'll explore this more as well with no concrete time frame 😓

[rnystrom]

@SD10 wrapped it in swift this afternoon for fun, already wrote a custom extension for it. I think the reason I didn’t bother was it was missing lots of GitHub flavors, but I didn’t know their fork was a thing.

Seems like we could make a more declarative parser with this after having learned more about our parsing and rendering needs.

I’ll push my WIP into another repo this week.

_{Sent with GitHawk}

[SD10]

@rnystrom Awesome! Thanks for taking the initiative. I'd love to take a look at the WIP. This could eliminate a whole category of issues for GitHawk both present and future

[rnystrom]

Here's the WIP. Probably the most important part is adding a custom extension, e.g. the one I wrote to handle mentions. We'll need another one to do checkmarks.

All of the URL handling/shortening can be done outside of the lib. _However_ the auto shortlink (e.g. repo/name#123) can't be done in an extension. We'll need a post-processing step in our own custom pipeline.

What I'm thinking of doing is this:

1. cmark + extensions to build c AST
2. Use enums, inspired by this lib to convert to typed Swift AST
3. Flatten to node stream (text, quotes, images, html, tables)
4. Post-process step (e.g. shortlinks)
5. Map nodes to GitHawk's models

That breaks things apart pretty nicely making each component testable.

[SD10]

Probably the most important part is adding a custom extension, e.g. the one I wrote to handle mentions.

@rnystrom Mentions were not already handled? Surprising.

[rnystrom]

@SD10 ya it wasn’t. Finished a checkbox extension last night as well. Should be in place to start using.

_{Sent with GitHawk}

[rnystrom]

@SD10 I know this Issue is off topic, but using this as my place to give status-updates on the cmark integration:

• I built a mentions and checkbox extension on top of GitHub's cmark fork (this is worth a blog post)
• Our cmark-gfm fork + extensions are in this repo
• Pushed WIP to this branch
• Building our list models happens here. Some huge benefits:
  
  
  
  • All elements are represented w/ enums + data meaning that supported markdown data is represented in entirety when building our app models.
  
  
  • To add more feature support (e.g. images in tables), we first add the enum + parsing in our cmark fork, the compiler will err in GitHawk, then we add feature support.
  
  
  • Instead of having a single, mutable builder that we iterate over, we now have different unique builder & logic branch for each root Element type
  
  
  • Less LOC
  
  
  • Functional-ish
  
  

App is running and rendering correctly on this branch. Tables are the last TODO.