Gatsby: Gatbsy createNode arbitrarily building nodes on `gatsby develop`

[Hannah-goodridge]

Description

I'm trying to use createNode after running a few fetch requests, but I'm finding that these nodes don't always get created. I've read a few other issues like this one and they mentioned to use promises which I've tried implementing but I still get issues.
Should I be using the onCreateNode method instead of sourceNodes?
I super new to gatsby so please excuse my ignorance.

Gatsby Node code

const credCache = new Map();
require('dotenv').config({
  path: `.env.${process.env.NODE_ENV}`,
});

require('es6-promise').polyfill();
require('isomorphic-fetch');

const crypto = require('crypto');
const qs = require('qs');

exports.onPreInit = () => {
  return new Promise(resolve => {
    fetch(process.env.TOKEN_URI, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
      },
      body: qs.stringify({
        grant_type: 'client_credentials',
        client_id: process.env.CLIENT_ID,
        client_secret: process.env.CLIENT_SECRET,
        scope: 'squidex-api',
      }),
    })
      .then(response => response.json())
      .then(data => {
        credCache.set('token', `${data.token_type} ${data.access_token}`);
        resolve();
      });
  });
};


exports.sourceNodes = ({ actions }) => {
  const { createNode } = actions;
  // Create nodes here, generally by downloading json
  // from a remote API.

  const token = credCache.get('token');
  // get all the squidex schemas
  const getSchemas = () => {
    return fetch(`${process.env.GENERAL_URI}`, {
      method: 'GET',
      headers: {
        Authorization: token,
      },
    })
      .then(result => result.json())
      .then(json => json.items.map(item => item.name));
  };

  const getContents = endpoints => {
    return Promise.all(
      endpoints.map(endpoint => {
        return fetch(`${process.env.CONTENT_URI}${endpoint}`, {
          method: 'GET',
          headers: {
            Authorization: token,
          },
        }).then(result => result.json());
      }),
    ).then(result => {
      result.forEach(schemaData => {
        if (schemaData.items.length) {
          const name = `Squidex-${schemaData.items[0].schemaName}`;
          const type = name.replace(/-/g, '');

          schemaData.items.forEach(datum => {
            const { id, createdBy, lastModifiedBy, data, isPending, created, lastModified, status, version, children, parent } = datum;

            const internal = {
              type,
              contentDigest: crypto.createHash('md5').update(JSON.stringify(datum)).digest('hex'),
            };

            const node = {
              id,
              createdBy,
              lastModifiedBy,
              isPending,
              created,
              lastModified,
              status,
              version,
              children,
              parent,
              internal,
            };

            const keys = Object.keys(data);
            keys.forEach(key => {
              node[key] = data[key].iv;
            });

            createNode(node);
          });
        }
      });
    });
  };

  Promise.resolve(getSchemas()).then(schemas => getContents(schemas));
};

Expected result

I'm always expecting the node to be present when I run gatsby dev but its unreliable.

Actual result

What happened.

Environment

System: OS: macOS Mojave 10.14.6 CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz Shell: 5.3 - /bin/zsh Binaries: Node: 12.14.1 - ~/.nvm/versions/node/v12.14.1/bin/node Yarn: 1.22.0 - /usr/local/bin/yarn npm: 6.13.4 - ~/.nvm/versions/node/v12.14.1/bin/npm Languages: Python: 2.7.14 - /usr/local/bin/python Browsers: Chrome: 81.0.4044.138 Edge: 81.0.416.72 Firefox: 72.0.2 Safari: 13.1 npmPackages: gatsby: ^2.21.16 => 2.21.16 gatsby-plugin-eslint: ^2.0.5 => 2.0.8 gatsby-plugin-manifest: ^2.4.2 => 2.4.2 gatsby-plugin-offline: ^3.2.1 => 3.2.1 gatsby-plugin-postcss: ^2.3.1 => 2.3.1 gatsby-plugin-react-helmet: ^3.3.1 => 3.3.1 gatsby-plugin-sharp: ^2.6.3 => 2.6.3 gatsby-plugin-styled-components: ^3.3.1 => 3.3.1 gatsby-source-apiserver: ^2.1.4 => 2.1.4 gatsby-source-filesystem: ^2.3.1 => 2.3.1 gatsby-source-graphql: ^2.5.1 => 2.5.1 npmGlobalPackages: gatsby-cli: 2.12.11

[herecydev]

Sounds like your missing a promise being returned (and therefore the gatsby api runner continues without waiting). I might be wrong but this looks problematic:

 ...
  Promise.resolve(getSchemas()).then(schemas => getContents(schemas));
};

I think that needs to be returned:

 ...
  return Promise.resolve(getSchemas()).then(schemas => getContents(schemas));
};

I personally don't find promises "thens" very easy to read, so just a suggestion but it might be easier to use async/await

[Hannah-goodridge]

Ran this a few times and it seems to be working, thanks ever so much!
I think you are right about the promises point too, I'll take it in to consideration!