Gatsby: How to use async with gatsby-plugin-manifest options?

[jrestall]

Summary

I would like to query my CMS for web manifest data within the gatsby-config before passing it to the gatsby-plugin-manifest plugin. I have hundreds of websites all with different branding and languages.

It doesn't work as the theme-loader doesn't expect a Promise to be returned.

https://github.com/gatsbyjs/gatsby/blob/ed105ecfb05fb757a3daf667a382301275f8fd42/packages/gatsby/src/bootstrap/load-themes/index.js#L65

Potential Fix

    themeConfig = await theme(themeSpec.options || {});

File contents

gatsby-config.js:

module.exports = async () => {
  const manifestOptions = await resolveManifestOptions(process.env.CMS_URL);
  return {
    plugins: [
      {
        resolve: `gatsby-plugin-manifest`,
        options: manifestOptions,
      },
      `gatsby-plugin-offline`
    ]
  }
}

Is there some other way I can load theme options dynamically from a server or would you accept the PR with the above fix?

[sidharthachatterjee]

Hey @jrestall

Thanks for opening this (and the associated PR as well)

We've considered enabling dynamic gatsby config generation in the past but have since decided against it because it potentially exposes a tonne of edge cases and seems like an easy way for users to shoot themselves in the foot. Themes _do_ currently accept a function (as you've mentioned) but we expect it to the sync.

Is there some other way I can load theme options dynamically from a server

I suppose you could add a pre build npm script of sorts that fetches your config and writes it to gatsby-config before running gatsby.

I'll be closing this and the PR for now. But thank you once again for bringing this up! 🙌

[jrestall]

Okay thanks @sidharthachatterjee, I can't get the following working from within a theme, is there something I'm missing? Maybe @ChristopherBiscardi has come across this requirement for themes.

    "scripts": {
        "prebuild": "echo Test"
    }

I also couldn't find any previous discussion on dynamic configs except at https://github.com/gatsbyjs/gatsby/issues/11128 where @wardpeet and the team was receptive initially but the change wasn't made as a workaround for that specific case was found. I guess the discussion and decision for this was made internally? Seems a shame as it opens up a lot of possibilities, I'd liked to have been able to read about the edge cases.

[jrestall]

For those coming across this issue, I ended up working around gatsby's lack of async theme configuration support by using the asynchronous createPages API and then overwriting the gatsby-plugin-manifest's pluginOptions before it uses it in onPostBootstrap.

E.g.

exports.createPages = async ({ graphql, store }) => {
  const state = store.getState()

  const plugin = state.flattenedPlugins.find(plugin => plugin.name === "gatsby-plugin-manifest")
  if (plugin) {
    const manifestOptions = await resolveManifestOptions(graphql)
    plugin.pluginOptions = {...plugin.pluginOptions, ...manifestOptions}
  }
}

I couldn't find how to use a pre-build script as gatsby themes need to be modular and not require separate changes to the site's build process.

It works but I'd still prefer gatsby supports async config loading as it seems like a very hacky workaround.

[joewood]

Nice workaround.
I simple solution for Gatsby to support this could be to allow for gatsby-config to export a Promise object for the entire options object. This avoids any potential of race conditions etc...

[webbugt]

We've considered enabling dynamic gatsby config generation in the past but have since decided against it because it potentially exposes a tonne of edge cases and seems like an easy way for users to shoot themselves in the foot. Themes _do_ currently accept a function (as you've mentioned) but we expect it to the sync.

I'm sorry for reactivating this issue, but why is this disallowed? I mean, using async config is by itself a far-fetched concept for 90% of use cases. As such, most people wouldn't even try it. In the remaining cases, it should be expected that the dev is competent enough to deliver a proper async function that would return a proper config.

Basically we're trying to generalize our builder to work for 2 different languages and on 2 different domains. This includes different facebookPixelIDs and googleTagManagerIDs. Since the IDs are stored in the project on the API, we need to fetch those first.

[wardpeet]

We're aware of the inconveniences it may cause. Gatsby config can be used in gatsby-themes as well so there are different use cases we should consider. We would love to support this in a future release but at the moment of writing, we haven't felt the need to prioritize this.

You can look into a prebuild script that writes these API requests to disk and consume them inside your build script as a workaround.

[webbugt]

Thanks for the answer, I get your point, but the use case that requires
async config is quite different from the use case that uses a prebuilt
gatsby-theme. They're completely opposite approach-wise. We don't use any
theme. Everything comes from the API. The rest is plugins plus some
hardcoded components that we still have to merge into the main system.

Maybe make the async config an .env flag? Eg. GATSBY_ASYNC_CONFIG=true.
Also when the flag is on the reporter could warn the user to check if their
downloaded theme supports async config, or they should modify it
themselves. Should be easy to implement considering it is just 1 line
changed (+logic for .env flag and reporter warning regarding the same).
Basically by using the .env flag, the dev is declaring that he's
responsible for the other possible compatibility issues that might surface.

Also, do you mind explaining in more detail how the async config impacts
gatsby-themes? Aren't they supposed to get the config from gatsby-cli
instead of directly parsing gatsby-config.js?

We are currently implementing the prebuild script approach, but it's really
messy since all our API calls are already neatly integrated into normal
gatsby processes.

If we went with such an approach from the start, we'd integrate all the API
calls and file download during prebuild... But since you provided us with
all these neat functions for remoteFiles and async page creation based on
API requests, it's really illogical that the config isn't async as well...
Basically, half of the build process is suggesting that users integrate
their APIs with Gatsby, while the other half requires integration around it.

Hopefully you understand our (and many other users') frustration with this
situation.

On Thu, 23 Jul 2020 at 23:08, Ward Peeters [e-mail] wrote:

We're aware of the inconveniences it may cause. Gatsby config can be used
in gatsby-themes as well so there are different use cases we should
consider. We would love to support this in a future release but at the
moment of writing, we haven't felt the need to prioritize this.

You can look into a prebuild script that writes these API requests to disk
and consume them inside your build script as a workaround.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/19644#issuecomment-663233243,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHB3H6YCWXQYML2AOYRXNBLR5CREVANCNFSM4JPLQFKA
.

[wardpeet]

I do understand. Gatsby-themes and gatsby-config inside the root follow the same code path to load in their config. This would mean we have to split it up or make sure we handle it gracefully in gatsby-themes. Opening it up means more maintenance/potential bugs, ... even if we add it behind a flag, it still needs to be maintained and handled correctly.

You can always try out scripts like https://github.com/alexeypetrushin/synchronize to make it work.

[webbugt]

That should solve the issue if it works, I will try that tomorrow, thanks
for the tip!

If it does work as intended, should we add this temporary work-around to
documentation?

I'll check back in once I've tried this approach.

On Thu, 23 Jul 2020 at 23:41, Ward Peeters [e-mail] wrote:

I do understand. Gatsby-themes and gatsby-config inside the root follow
the same code path to load in their config. This would mean we have to
split it up or make sure we handle it gracefully in gatsby-themes. Opening
it up means more maintenance/potential bugs, ... even if we add it behind a
flag, it still needs to be maintained and handled correctly.

You can always try out scripts like
https://github.com/alexeypetrushin/synchronize to make it work.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/19644#issuecomment-663246425,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHB3H64EP23TDUWL4NQXLZ3R5CVATANCNFSM4JPLQFKA
.

[webbugt]

Hey, back with some info, tried integrating both desync and synchronize. Synchronize doesn't work on Node 12 while desync just gets stuck.
For now the only working approach (that allows reuse of api code that we already have) seems to be via .env variables. We solved it by dividing the config into separate blocks (eg gtmPlugin) which get pieced together into the final config inside gatsby-config.js (since you can still run synchronous functions inside it). For now we're connecting each property that we need to be dynamic, but plan on testing JSON as .env variable, that would allow us to pass the whole (or most of) the config as an object.