Gatsby: How can we access derived/computed fields using the JavaScript API?

[TMiguelT]

Summary

I need to be able to access computed fields from JavaScript, not from GraphQL.

Relevant information

I have a number of derived/computed fields, ie those with a resolve() method but where a static value is never actually assigned to the node. The trick is that I need to be able to access them from the Node API (not just from GraphQL!) in order to fulfil the resolution of another field. How can this be done?

As a simple example, I've made a simplified version of this problem below, whereby we have Traffic has many Car, and Traffic needs to access the derived field speed on its children in order to resolve averageSpeed, but it can't.

Environment (if relevant)

  System:
    OS: Linux 4.19 Ubuntu 18.04.4 LTS (Bionic Beaver)
    CPU: (8) x64 AMD FX(tm)-8320 Eight-Core Processor
    Shell: 3.1.2 - /usr/bin/fish
  Binaries:
    Node: 14.5.0 - /usr/bin/node
    npm: 6.14.5 - /usr/bin/npm
  Languages:
    Python: 2.7.17 - /usr/bin/python
  Browsers:
    Chrome: 84.0.4147.89
    Firefox: 78.0.2
  npmPackages:
    gatsby: ^2.23.12 => 2.23.12
    gatsby-image: ^2.4.9 => 2.4.9
    gatsby-plugin-manifest: ^2.4.14 => 2.4.14
    gatsby-plugin-offline: ^3.2.13 => 3.2.13
    gatsby-plugin-react-helmet: ^3.3.6 => 3.3.6
    gatsby-plugin-sharp: ^2.6.14 => 2.6.14
    gatsby-source-filesystem: ^2.3.14 => 2.3.14
    gatsby-transformer-sharp: ^2.5.7 => 2.5.7
  npmGlobalPackages:
    gatsby-cli: 2.12.13

File contents (if changed)

gatsby-config.js: N/A
package.json: N/A
gatsby-node.js:

exports.createSchemaCustomization = ({ actions: { createTypes }, schema }) => {
  createTypes([
    schema.buildObjectType({
      name: "Traffic",
      fields: {
        cars: {
          type: "[Car]",
          resolve(source, args, context) {
            return context.nodeModel.getNodesByIds({
              ids: source.cars,
              type: "Car"
            })
          }
        },
        averageSpeed: {
          type: "Float!",
          resolve(source, args, context) {
            // Firstly, I'd like to be able to access the resolved version of the "cars" field here. e.g.
            // const cars = source.cars;
            const cars = context.nodeModel.getNodesByIds({
              ids: source.cars,
              type: "Car"
            });

            // Secondly, I'd like to be able to access car.speed here, but this isn't returned
            return cars.reduce((acc, curr) => acc + curr.fields.speed, 0) / cars.length;
          }
        }
      },
      interfaces: ["Node"],
    }),
    schema.buildObjectType({
      name: "Car",
      fields: {
        distance: "Float!",
        time: "Float!",
        speed: {
          type: "Float!",
          resolve(source, args, context) {
            return source.distance / source.time
          }
        }
      },
      interfaces: ["Node"],
    })
  ])
}

exports.sourceNodes = ({ actions, createNodeId, createContentDigest }) => {
  const { createNode } = actions

  for (let car of [
    {
      id: "0",
      distance: 100,
      time: 5,
      internal: {
        contentDigest: createContentDigest("0"),
        type: "Car"
      }
    },
    {
      id: "1",
      distance: 60,
      time: 3,
      internal: {
        contentDigest: createContentDigest("1"),
        type: "Car"
      }
    },
    {
      id: "2",
      distance: 80,
      time: 4,
      internal: {
        contentDigest: createContentDigest("2"),
        type: "Car"
      }
    },
    {
      id:"3", 
      cars: ["0", "1", "2"],
      internal: {
        contentDigest: createContentDigest("3"),
        type: "Traffic"
      }
    }
  ]) {
    createNode(car)
  }
}

gatsby-browser.js: N/A
gatsby-ssr.js: N/A

[herecydev]

I'm actually not sure that's possible. It could be due to avoiding potential circular references.

From my experience you can only query "actual node data". You can always encapsulate the logic into some function which you can then reuse?

[TMiguelT]

That's unfortunate. I can see circular references being a problem, but losing a huge amount of power in our models doesn't seem worth that edge case. In any case, it seems that, with a hack/workaround, you can actually query derived fields: https://stackoverflow.com/a/63030645/2148718. However, I would like the ability to do this "officially".

[vladar]

@TMiguelT

I would recommend adding speed to the node object itself when sourcing/transforming nodes vs having a custom resolver for it.

Custom resolvers are compelling but I would recommend to use them as little as possible and instead do all calculations/transformations during source/transform step (in sourceNodes or onCreateNode).

---

That said, Gatsby has an internal mechanism to filter/sort by fields with custom resolvers. We call it materialization. And the StackOverflow link sums it up correctly. The problem is that this is not a public API. This is a sort of implementation detail that may change someday and that's why it is not documented.

We won't break it in the patch/minor version and have no plans to remove it in the next major, so I guess you can use it as a workaround for now (still not recommending this).

So for your situation you can try something like this:

averageSpeed: {
  type: "Float!",
  resolve(source, args, context) {
    const cars = context.nodeModel.runQuery({
      type: "Car",
      query: { filter: { id: { in: source.cars }, speed: { gte: 0 } } }
    });
    return cars.reduce((acc, curr) => acc + curr.__gatsby_resolved.speed, 0) / cars.length;
  }
}

But it is highly discouraged 😀 This is not a public API.

We've been thinking about alternative ways to achieve a similar effect (maybe by using custom node transforms vs custom resolvers in schema customization to keep node model always 1-1 with the schema) but that's just an idea/discussion so far.

[TMiguelT]

Thank you, this is a comprehensive answer that gives me two practical solutions. As you suggest, I think I'll try to store all the derived attributes on the nodes. My avoidance of this is the kind of habit that comes from writing SQL databases where you really aren't ever supposed to do that, so I just have to retrain myself when using Gatsby.

That said, I think it would still be beneficial to add these derived fields to the returned nodes in some way, for situations where pre-calculating the data is not possible. I look forward to updates in this area.

Happy to close this issue or keep it open, depending on whether this long term goal is tracked elsewhere or not.

[vladar]

Thank you for opening this, @TMiguelT

We're marking this issue as answered and closing it for now but please feel free to reopen this and comment if you would like to continue this discussion. We hope we managed to help and thank you for using Gatsby! 💜

[herecydev]

Custom resolvers are compelling but I would recommend to use them as little as possible and instead do all calculations/transformations during source/transform step (in sourceNodes or onCreateNode).

@vladar Can you elobarate on this? From my understanding the only negative is that custom resolves _probably_ can't be cached (because they're functions). Are there other factors that make them a poor recommendation?

An example where I use them heavily is in i18n:

createResolves({
Car: {
description: {
  type: `String`,
  resolve: (source, _, context) => { //look up localized translation based on current locale in the context  }
}
}
})

Because the "description" field is dependent on the current locale the page is targeting, the query has to be dynamic. The only other option is to produce permuations of every car with each locale i.e.

1. Car|en
2. Car|fr
3. Car|ru

As Michael pointed out, it seems counter-intuitive to duplicate data

[vladar]

@herecydev

We are thinking about ways to parallelize Gatsby builds in the future. And custom resolvers are hard to reason about in this context (unlike sourcing/transforming step).

Gatsby datastore is a bit different from classical databases. It was originally based on a concept of event sourcing (or streaming) which is a good approach for parallel data processing.

Schema customization and custom resolvers moved it a bit to a more hybrid approach for convenience. But as we consider the parallelization of builds it may become problematic.

The only other option is to produce permuations of every car with each locale i.e.

But if you build your site for all locales you will still "produce permuations of every car with each locale". The question is mainly at what step and how convenient it is from the DX perspective.

[herecydev]

Thanks, that's helpful.

But if you build your site for all locales you will still "produce permuations of every car with each locale". The question is mainly at what step and how convenient it is from the DX perspective.

I think I need to better understand how graphql queries are run (and cached?). I've got a few questions so I might open another issue after doing some research