Gatsby: Convert API pages to MDX.

[tesseralis]

Summary

The following pages under www/src/pages/docs need to be converted to MDX:

• actions.js
• browser-apis.js
• node-api-helpers.js
• node-apis.js
• node-model.js
• ssr-apis.js

Basic example

See /docs/docs/graphql-api.md to see how to integrate the query

Motivation

The API files are some of the core files to translate to gatsby, and they need to be extracted to MD to make it easier to translate. In addition, the current JS implementation is clunky and relies on __dangerouslySetInnerHTML to properly render code blocks.

Related Issues

19027 #18704

[alexluong]

Hey, I'd love to help with this 👋

[tesseralis]

@alexluong sure, go ahead!

[sever1an]

Removing the JSDoc may have negative side effects for tools that use it directly such as IDEs. The JSDoc schema also enforces a rigorous format that developers are comfortable with.

Would it be possible to use https://www.npmjs.com/package/documentation to perform the conversion automatically to ensure consistency and maintain the JSDoc as an original source of truth? If the resultant markdown was tracked in git and used by the website, would this provide an equally viable thing to translate?

I also notice that the JSDoc comments are living in stub files, not the active source, which might limit their usefulness. Does Gatsby eventually want to make a more focused effort to use JSDoc or is it safe to abandon now?

[tesseralis]

Honestly... I don't know why most of the JSDoc comments are living in stub files. I'd have to ask the team that. The plan is to just convert the text above the JSDoc part to MDX and to pull in the JSDoc using the component.

[sever1an]

I definitely misread the other issue about translating source comments. It would be interesting to know what the core team thinks about JSDoc and if it will always be isolated in separate no-code js files.

[sever1an]

I have started work on this

[tesseralis]

@alexluong @sever1an would you mind splitting the work since there are multiple API files?

[sever1an]

I just noticed it was back in the help wanted pile, but would be happy to wait and see if @alexluong is still interested.

[alexluong]

If you wanna work on it @sever1an, be my guest. I haven't started yet.

[sever1an]

Having the mdx outside of the www folder seems to preclude the execution of static graphql queries within them. Moving forward it would seem that the queries would need to be extracted to components which would likely mean one component (or maybe even js file) per api doc.

[sever1an]

Having a pageQuery in mdx rendered by <MDXRenderer> seems to be impossible, so the graphql in the api docs files needs to be extracted and performed in a component. Currently I am comparing two solutions.

Queries in the MDX template

Put the jsdoc paths in frontmatter, then loads them into page context during createPage. Another graphql expression using $variables would be added to the mdx template which replaces those in the JS files.

This seems to inevitably involve a significant amount of bloat to the mdx template file, so there may be some question of relative value considering that the mdx files seem to contain only a few lines of text. Clarity in the already misty gatsby-node would also suffer.

Queries in single components per API.

Something that might perhaps be simpler considering the relatively stable number of api documents would be to simply create a component for each one containing the required graphql query that could be used from mdx. For example <GatsbyNodeApiDocs/>

Internationalization would have total control over them pending the outcome of https://github.com/gatsbyjs/gatsby/issues/19036

Conclusion

My experimental version of the frontmatter/template graphql solution is a bug farm, and feels dreadful to maintain.

I will have a PR for the individual components solution tomorrow, unless someone feels strongly that it is the wrong strategy.

[tesseralis]

Thank you so much for the detailed explanation, Bailey!

I think you're right that Single Component Queries would be the best option out of the two. It's a bit inelegant, especially if we do #19036 and these components end up still importing gatsby and relying on a graphql schema, but that's something we can deal with later.

It also seems like, for everything except actions, the queries and how they're processed are similar, so maybe we can only have one or two components instead?

[sever1an]

Something like <APIDocs name="browser-apis"/>?

[tesseralis]

Yeah, that's my thought!

[tesseralis]

There's something going on with "normalizing" in some of the pages (using two queries and combining them together) and I don’t really know what's going on with that.

[pieh]

What about this:

Instead of individual pages like they are right now, we make them templates. We would keep page query (as we kind of need page query to be able to query for different languages if we want both free-form content and jsdocs to be translated):

import { MDXRenderer } from "gatsby-plugin-mdx"

export default class MyPageLayout {
  render() {
    const funcs = sortBy(
      this.props.data.file.childrenDocumentationJs,
      func => func.name
    )

    const normalized = normalizeGatsbyApiCall(this.props.data.nodeAPIs.group)

    const mergedFuncs = funcs.map(func => {
      return {
        ...func,
        ...normalized.find(n => n.name === func.name),
      }
    })

    // according to https://www.gatsbyjs.org/packages/gatsby-plugin-mdx/#mdxrenderer
    // we can pass arbitrary props to MDXRenderer so data becomes available to use in .mdx file
    return <MDXRenderer mergedFuncs={mergedFuncs}>{this.props.data.mdx.body}</MDXRenderer>
  }
}

export const pageQuery = graphql`
  query Docs($language: String!) {
    mdx(<select_node_apis_content>, <select_proper_language>) {
      body
    }
    file(relativePath: { regex: "/src.*api-node-docs.js/" }, <select_proper_language>) {
      childrenDocumentationJs {
        name
        ...DocumentationFragment
      }
    }
    nodeAPIs: allGatsbyApiCall(filter: { group: { eq: "NodeAPI" } }) {
      group(field: name) {
        ...ApiCallFragment
      }
    }
  }
`

And then in api-node-docs.mdx it could have all the content that is currently hard-coded in react component and at the end import <APIReference> component and use mergedFuncs passed via MDXRenderer?

[pieh]

There's something going on with "normalizing" in some of the pages (using two queries and combining them together) and I don’t really know what's going on with that.

This is because the content is source from 2 different sources - we should use schema customization for it, so we use single query (and connect those at build time, rather than on runtime) - schema customization just wasn't built yet when this page/template was written.

What this does is combines jsdocs (DocumentationJs part) and plugin that looks through gatsby code base to figure out where those extension points (so in gatsby-node, gatsby-browser and gatsby-ssr) are being called in gatsby code base (GatsbyApiCall part) - result are those links in our docs ( createPages example ):

[sever1an]

@pieh Do you think it would be an acceptable solution to include the jsdoc search stuff as frontmatter with a single new template for api doc mdx? It would be possible to get the values into a page query using graphql parameters and page context.

I am not familiar enough with the inner workings of Gatsby graphql to identify whether this is a good idea, although I can definitely implement it...

---
jsdoc: "/src.*api-node-docs.js/"
apicalls: "NodeAPI"
---

[pieh]

@pieh Do you think it would be an acceptable solution to include the jsdoc search stuff as frontmatter with a single new template for api doc mdx?

This technically can be done - just the question where this should live - should be in component level or in .md level - using frontmatter will allow us to have single template for all API reference pages. But even then - we could provide this via context when calling createPage, so maybe we should just define this in gatsby-node, so .md file don't need to define/track it themselves?

[muescha]

Since there is a goal for native TypeScript (#18983) in Gatsby:

would it make sense to put the JSDoc information instead in separate api-*-docs.js better into the TypeScript .d.ts files?

Then we have this goals fixed:

• doc are decoupled from code (can be pushed to repo for translatIon)
• dts files are up to date (this is needed for right type checking)
• IDE editors can use it for assistance
• more rich descriptive types can be displayed in API Reference

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜

[sever1an]

I sincerely doubt that this issue should be closed.

[sever1an]

The auto close bot is way too aggressive