Gatsby: Serve Gatsby site as GraphQL API

[jgierer12]

Summary

Add a command to Gatsby that serves the site data as a public GraphQL API.

I believe the core functionality is already existing, in the form of GraphiQL in development mode. If we could allow the user to take that API and serve it publicly, they would be able to query their site dynamically with exactly the same schema they use for their static queries, and with minimal additional configuration.

Basic example

Let's say we're building a shop with a few hundred products. We want to implement a filter functionality, so that users can narrow down the displayed items by price as well as rating.

Since there are so many possible combinations of prices and ratings, we can't generate every possible combination at build time. So currently, we need to download all the products' data and filter it on the client. Of course, this uses a lot of bandwidth and client-side system resources.

With the new feature, we can instead connect a GraphQL client such as Apollo to our Gatsby GrahpQL API and dynamically filter the products we want to download:

// pages/shop.js
import { graphql } from "gatsby"
import { gql } from "graphql-tag"
import { query } from "some-gql-client"

export const query = graphql`
  // static page query like usual
`

// ...

// in page component, when user changes filter:
function filterItems(maxPrice, minRating) {
  const data = await query(gql`
    query {
      allMdx(filter: {
        frontmatter: {
          price: { lte: ${maxPrice} }
          rating: { gte: ${minRating} }
        }
      }) {
        edges {
          node {
            title
            price
            rating
            slug
          }
        }
      }
    }
  `)

  // display items
}

The API server would be started via a new CLI command (e.g. gatsby api).

Naturally, this would be a completely optional, additional feature. All sites can and must still be built with gatsby build and deployed on a static host such as Netlify. If the developer wants to use the API server, they have to deploy it separately from the static part to a host such as Zeit Now. The URL from that deployment can then be used in the front-end to connect a GraphQL client to the API.

Motivation

This enables many use cases. For example, complex sorting or filtering like in the example above. It also allows implementing search functionalities without relying on a third-party SaaS and managing its index. Other use cases are (extremely basic) password-protected content (filter: { password: { eq: ${password} } }), filtering based on time or location, dynamic pagination and probably many more.
It can even be used as a source for a different Gatsby site via gatsby-source-graphql!

As I mentioned above, I suspect that much of the functionality can be borrowed from or shared with the already existing GraphiQL feature.

Concerns and ideas

• It's worth noting that this will probably underlie the same limitations as GraphiQL, so no support for plugin-defined fragments (#612).

• Having two different GraphQL tags may be confusing. On the other hand, I reckon this is more for advanced users anyways.

• We should also implement a caching layer. Since the same input should always result in the same output, it's easy to cache results per query. This will increase performance for subsequent requests.

• I don't have a lot of insight yet into how the development GraphiQL works, so perhaps this isn't as trivial to implement as I assume (it probably isn't :wink:).

[KyleAMathews]

I love this! Thanks for writing this up. Something like this has been on my mind for a while and seems very promising.

gatsby api seems like the right command for this or perhaps gatsby api-server to be really explicit.

You could try this out pretty easily right now by just running gatsby develop somewhere. The main downside is that it'd also be running webpack which is kinda heavy. We do have some caching but more for builds than develop where we assume data is changing all the time. But speeds would definitely be adequate for testing and even for a website with light traffic.

Would love to hear your reports on playing more with this. The next step past that would be to write this up (mostly a copy paste) as an RFC to get more feedback! https://github.com/gatsbyjs/rfcs

[jgierer12]

Wow, I just tried running some queries against https://localhost:8000/___graphql and it's working really well already! Just got to run it separately from gatsby develop without webpack and maybe add some caching and we're pretty much good to go!

I'll put up a little demo running with gatsby develop and then write up an RFC.

[jgierer12]

Alright, here's a little demo, using gatsby develop on a heroku server: https://gatsby-api-proposal-demo.netlify.com/ (Source).

The implementation itself was a breeze. Being able to query against the Gatsby API dynamically feels great! There is a bit of boilerplate code though, since you need to write the same query twice in many use cases.

I did run into two problems during deployment:

• CORS is not enabled on the gatsby develop server, so I had to run it through an express proxy. This should definitely be easier for the dedicated gatsby api.
• Zeit Now doesn't allow deploying npm script commands anymore, instead it needs an actual file to run. I used Heroku instead, but this should be taken into consideration since Now is one of the most popular Node.js PaaS.

Otherwise, it was all good. It handles 10k items just fine!

[pieh]

We have some basic implementation of this already done - https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/commands/data-explorer.js

This is what "powers" graphiql iframes on https://www.gatsbyjs.org/docs/graphql-reference/ page.

It will probably hit same CORS problem, you had with gatsby develop.

ATM there is no command for this in gatsby-cli, but you can invoke it with something like this: https://github.com/gatsbyjs/gatsby/blob/master/scripts/www-data-explorer.js

[jgierer12]

Thanks for pointing that out @pieh! I replaced the hacky gatsby develop based server in my demo repo with data-explorer.js plus the CORS middleware. It's now a lot lighter on server resources since webpack isn't running in the background!

I think that's pretty much all we need for the feature. I'll write the RFC now to further discuss the further details.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

Thanks again for being part of the Gatsby community!

[jgierer12]

This is still something I'd really want.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[LpmRaven]

Has this happened? Any update? It would completely fix my current problem.

[CrackTheCode016]

I'd love to be able to do this!

[johnatspreadstreet]

We would also love to be able to do this. Excellent feature potential.

[gerhardsletten]

Did start on something here before summer github.com/gerhardsletten/graphql-cms

But because I have limited knowledge to the whole gatsby-js project I started with extraction stuff from the remark and file package, I also wanted it not to be to big so I easily could run in it on Zeit now, and I had the perception that the whole gatsy-js library would add a lot overhead.

But the basic idea is that you have a build-command that will read a folder for .md files and create a json that you later can serve as a graphql content api. (See the example-folder in the above repo, for usage)

The next step which I have not had time to look at is also extraction images and files linked in the markdown-files and saving them as optimized web-images in sizes from the config-file, like the gatsby-sharp packages do.

[johnatspreadstreet]

It looks like @birkir is pretty close with this repo: https://github.com/birkir/graphql-gatsby

The only issue is that it may become outdated quickly trying to reference node_module files directly (it is already throwing errors due to the schema customization changes).

[birkir]

yeah its a huge task to maintain a thing like that outside of the gatsby source...

two things

a) i can try to keep it up and running, but I'm super busy so issues would be appreciated.
b) if someone has suggestions to bypass direct node_modules access, that would be great

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[muescha]

side note for readers: the data explorer is removed from Gatsby

to see the previous solution, you can look at the related removal PRs:

• #15306 chore(gatsby-cli): Remove data explorer (was before "adding data explorer cli" - see also commmit 5c8a11a removing a proposed cli implementation)
• #25705 chore(scripts): remove www-data-explorer

[tomalex0]

Just checking, if anybody found a way to run light weight api server, without front end?

[jamo]

I'd love to hear out what particular issue this would solve and what's the use-case?

[cantide5ga]

I'd love to hear out what particular issue this would solve and what's the use-case?

I think in the case where you have something like a headless CMS around Gatsby, having access to a single source of data to support a cross-platform experience for something like a mobile app would be handy.

Currently looking for a way to architect exactly this and think this would work well.

[gburgett]

I'd love to hear out what particular issue this would solve and what's the use-case?

I have a use case for this. We use Contentful for a backend for our website. Currently in order to do previews of our content, we run a "gatsby develop" process with the host set to preview.contentful.com and load up a page in an iFrame. Refreshing content in this iFrame currently takes around 15 seconds, which is a long time for our editors to wait to see one minor text change. We would like to reduce this to 1-2 seconds.

One way to reduce the amount of time would be to load up the React component that renders the page inside the editor's browser, and then populate it using react-apollo. In fact we used to do this before upgrading Gatsby. In a previous version of Gatsby I was able to copy-paste part of the code that loaded up the Gatsby GraphQL schema, and expose it using express. This was very fast. Unfortunately the gatsby develop source has become very complex and I can no longer maintain that version.

Here is a simplified version of what I'm talking about, copied and modified from our code:

class Preview extends React.Component<{ slug: string }> {
  public render() {
    const { slug } = this.props

    return <Query>
      query={gql`
      query PageBySlug($slug: String!) {
        contentfulPage(slug: { eq: $slug }) {
          slug
          title
          text {
            childMarkdownRemark {
              html
            }
          }
        }
      }
      variables={{ slug: slug }}
      errorPolicy="all"
      fetchPolicy="no-cache"
    >
      {(resp) => {
        const { loading, error, data } = resp
        if (loading) { return <h2>Loading...</h2> }
        if (error || !data || Object.keys(data).length == 0) {
          return <div>
            <p>Error :(</p>
            {error && JSON.stringify(error)}
          </div>
        }

        return <MyPageTemplate data={data} />
      }}
    </Query>
  }
}