Gatsby: Dynamic and programmatic gatsby-config

[thibautRe]

Summary

I have a use-case for which I can't find a way forward. It involves multiple gatsby-source plugins and how to have a dynamic configuration for them.

I'm in the process of writing a source plugin for Adyen (payment provider) to expose nodes about the Adyen data for a given merchant configuration. This plugin is intended to be open-sourced - so not specific to our own technical choices.

Because of Adyen's API limitations, it's impossible to export the whole Adyen data like it's done in the Contentful plugin for instance. So you need to be quite explicit in the source plugin config and specify a lot of things - merchant account, storeCodes, etc.

We don't want to hardcode the list of all merchant accounts and storecodes in the codebase as we already have 40+ combinations, instead want to store this type of information somewhere else - in Contentful in our case.

Now my problem is: it's impossible to dynamically (asynchronously) configure the options of a Gatsby plugin. At first I was thinking about doing something like this in gatsby-config.js:

// Returns the Adyen config for a given market
const getAdyenConfig = async market => {
  // Do a request to Contentful's content API
  const contentfulMarket = await makeContentfulRequest(market)
  return contentfulMarket.adyenConfig
};

module.exports = {
  plugins: [
    {
      resolve: 'plugin-source-adyen'
      options: {
        adyenConfig: '???' // Heyyyy I can't `await getAdyenConfig()` here, so I can't give a proper config.
      }
    }
  ]
}

The issue here is the await - since the module.exports needs to export an object, it's not possible to do an asynchronous action before exporting the Gatsby config.

Possible solutions

Using Gatsby Nodes

I thought about using Nodes (in Gatsby's terms) to pass data to the plugin. The flow for instance would be:

1. gatsby-source-contentful create all the nodes
2. A custom plugin, gatsby-adapter-contentful-adyen would adapt some Contentful Nodes for the Adyen plugin to use. These intermediate nodes could be standardized, and not Contentful-specific.
3. gatsby-source-adyen would use these nodes.

I don't think this architecture is a viable solution because that's not how most source plugins work - they use gatsby-config as their way of configuring themselves, not nodes. Besides, that makes the plugin more complex as it could (should?) be able to read config from both these special nodes and the gastby-config as it's the common way to do it.

Allow gatsby-config to asynchronously generate the configuration object

This means that gatsby-config could be written like this:

// gatsby-config.js
module.exports = async () => ({
  plugins: [
    {
      resolve: 'gatsby-source-adyen',
      options: await getAdyenConfig(), // it works fine now
    },
  ],
});

In Gatsby's internals, the file requiring the config would check if the exported value is a Promise or an object, and await it if it's a Promise.

Some other way?

There might be other ways to do it, but introducing APIs to allow plugins to communicate or change each other's configs for instance.

[sidharthachatterjee]

Thank you for opening this @thibautRe

We do allow config for the upcoming Gatsby themes for be functions so we could look at this
https://github.com/gatsbyjs/gatsby/blob/a11f79abcfe7f0390dedc5b10c7d0b79848595c2/packages/gatsby/src/bootstrap/index.js#L86

@pieh What do you think?

[wardpeet]

Allow gatsby-config to asynchronously generate the configuration object

This one is fairly simple and we can introduce this one without doing much work. I'm not sure we will prioritize this feature as there is a workaround to make this work.

This is the line you probably need to do some magic:
https://github.com/gatsbyjs/gatsby/blob/e8a41020d42fe400f647776e5e0c92a03de40f93/packages/gatsby/src/bootstrap/load-plugins/load.js#L122

it should be as simple as plugin.options = await Promise.resolve(plugin.options || {}). You'll have to do some other changes like making the function async and awaiting in the callers function.

Using Gatsby Nodes

I can't help you with that so it's good to wait for the other @gatsbyjs/core members

[thibautRe]

@wardpeet I was actually thinking about allowing async functions on the module.exports of the config, not on individual plugin's options. Your solution would indeed work in my use-case. But allowing async on the top-level would allow more things like asynchronous siteMetadata configuration, and maybe other relevant things. In that case, it would be just before https://github.com/gatsbyjs/gatsby/blob/a11f79abcfe7f0390dedc5b10c7d0b79848595c2/packages/gatsby/src/bootstrap/index.js#L121 that something like

config = await Promise.resolve(config)

[wardpeet]

that sounds even better 👍 thanks for thinking out of the box

[thibautRe]

Just following up on this one - is the API I mention (allow exports of Promise from gatsby-config) something that would be accepted by the maintainers? I can work on a PR

[KyleAMathews]

Couldn't the plugin itself go fetch the config?

[pieh]

As @KyleAMathews mentioned you can let your plugin accept functions:

module.exports = {
  plugins: [
    {
      resolve: 'plugin-source-adyen'
      options: {
        adyenConfig: async market => {
          // Do a request to Contentful's content API
          const contentfulMarket = await makeContentfulRequest(market)
          return contentfulMarket.adyenConfig
        }
      }
    }
  ]
}

and you could invoke that and await inside your plugin

[thibautRe]

That would work for my use-case indeed. I was just wondering if there was maybe a more idiomatic way of doing these kind of stuff but now that I think of it this solution might be the best.

Thanks for the help 👍

[michaelpumo]

I could really do with this feature...has it been implemented at all in any way? I can't get the plugin author to change their plugin, so it would be great to have control of this natively.