Gatsby: "mixed" dynamic route not working

[BernardA]

Summary

Let me explain my use case and what I mean by "mixed" dynamic route.

It's a small ads app, currently available at https://musing-davinci-538143.netlify.app/ - minus this attempt for "mixed" dynamic routes.

The /search route can have different configurations:
/search is the overall search page whence drill down searches can be performed
/search/{country}/{root-category}/{parent-category}, country level searches
/search/{place}/{root-category}/{parent-category/{category} are the drill down routes.

There are about 200 categories and place can be any number. In the case of US, that would be just short of 20,000 cities, towns and villages.

You can see that this can become a huge number if one attempts to create pages for all possible combinations of categories and places.

Initially I chose to create pages only for the places that have an actual ad active, and that works fine as you can see in the link above. Those were created using the createPages API ( see code below).

Now, once an user creates an ad he/she should be able to immediately see its ad published and, of course, that ad should be available for all visitors to see.

If the user creates an ad that belongs to one of the pre-built /search/... pages there would be no issue as I fetch current information on componentDidMount on the /src/templates/search.js.

But for the not previously built pages I need to create routes on the fly - dynamic - to show that newly created ad.

Of course those routes should properly handle any valid search still not having a pre-built page and either show the ads or a proper message to the visitor otherwise. IT SHOULD ABSOLUTELY NOT END UP IN 404

It's certainly impractical and very expensive to re-build the app each time an ad is posted, so I was attempting to generate dynamic routes for the all routes not generated by createPages.

The, once a day or any other time interval, I would rebuild the app, generating new pages if applicable. ( I do have a question on that, as well, but let's leave it for later)

But these "mixed" dynamic route are not being generated as it seems. It will just send any URL that does not match one generates by createPages to 404.

Relevant information

gatsby.node.js

const path = require('path');
const { urlWriter } = require('./src/tools/urlWriter');
require('dotenv').config({
    path: `.env.${process.env.NODE_ENV}`,
 });

module.exports = { 
//  ***** THIS SECTION IS WORKING
createPages: async ({ graphql, actions }) => {
    const { createPage } = actions;
    const component = path.resolve('./src/templates/search.js');

    const pages = await graphql(`
        {
            myApi {
                categories {
                    id
                    title
                    parent {
                        id
                        title
                    }
                    root {
                        id
                        title
                    }
                }
                collectionQueryAddresses {
                    city
                }
            }
        }
    `);
    const categories = pages.data.myApi.categories;
    const addresses = pages.data.myApi.collectionQueryAddresses;
    const main = process.env.ROOT_CATEGORIES.split(',');
    main.forEach((type) => {
        categories.forEach((row) => {
            if (row.root && row.root.title === type &&
                row.parent.title === type) {
                const parentTitle = urlWriter(row.title);
                createPage({
                    path: `/search/france/${urlWriter(type)}/${parentTitle}`,
                    component,
                    context: {
                        categoryParent: row.id,
                        categories,
                        addresses,
                        city: null,
                        categoryId: null,
                        first: 10,
                        after: null,
                        isActive: true,
                        isDeleted: false,
                    },
                });
                categories.forEach((row1) => {
                    if (row1.parent && row1.parent.title === row.title) {
                        createPage({
                            path: `/search/france/${urlWriter(type)}/${parentTitle}/${
                                urlWriter(row1.title)}`,
                            component,
                            context: {
                                categoryParent: null,
                                categories,
                                addresses,
                                city: null,
                                categoryId: row1.id,
                                first: 10,
                                after: null,
                                isActive: true,
                                isDeleted: false,
                            },
                        });
                    }
                });
            }
        });
    });
    addresses.forEach((address) => {
        const place = address.city.replace(' ', '-').toLowerCase();
        categories.forEach((cat) => {
            if (cat.root && cat.root.title === cat.parent.title) {
                const catTitle = urlWriter(cat.title);
                createPage({
                    path: `/search/${urlWriter(place)}/${urlWriter(cat.root.title)}/${catTitle}`,
                    component,
                    context: {
                        categoryParent: cat.id,
                        categories,
                        addresses,
                        city: address.city,
                        categoryId: null,
                        first: 10,
                        after: null,
                        isActive: true,
                        isDeleted: false,
                    },
                });
            }
            if (cat.root && cat.root.title !== cat.parent.title) {
                const catParentTitle = urlWriter(cat.parent.title);
                createPage({
                    path: `/search/${urlWriter(place)}/${urlWriter(cat.root.title)}/${catParentTitle}/${urlWriter(cat.title)}`,
                    component,
                    context: {
                        categoryParent: null,
                        categories,
                        addresses,
                        city: address.city,
                        categoryId: cat.id,
                        fist: 10,
                        after: null,
                        isActive: true,
                        isDeleted: false,
                    },
                });
            }
        });
    });
},
// ***** BELOW DOES NOT WORK *****
onCreatePage: async ({ page, actions }) => {
    const { createPage } = actions;
    const component = path.resolve('./src/templates/search.js');
    if (page.path.match(/^\/search\/[-a-z]+\/[-a-z]+\/[-a-z]+\/[-a-z]+/)) {
        page.matchPath = '^\/search\/[-a-z]+\/[-a-z]+\/[-a-z]+\/[-a-z]+';
       page.component = component;
        // Update the page.
        createPage(page);
    }
},
};

Environment (if relevant)

System:
OS: macOS 10.15.5
CPU: (4) x64 Intel(R) Core(TM) i5-4570S CPU @ 2.90GHz
Shell: 5.7.1 - /bin/zsh
Binaries:
Node: 12.16.0 - /usr/local/bin/node
npm: 6.13.4 - /usr/local/bin/npm
Languages:
Python: 2.7.16 - /usr/bin/python
Browsers:
Chrome: 83.0.4103.106
Safari: 13.1.1
npmPackages:
gatsby: ^2.20.7 => 2.20.28
gatsby-cli: ^2.12.34 => 2.12.34
gatsby-image: ^2.2.33 => 2.3.4
gatsby-link: ^2.2.24 => 2.3.4
gatsby-plugin-create-client-paths: ^2.1.17 => 2.2.3
gatsby-plugin-eslint: ^2.0.8 => 2.0.8
gatsby-plugin-manifest: ^2.3.3 => 2.3.6
gatsby-plugin-material-ui: ^2.1.6 => 2.1.6
gatsby-plugin-offline: ^3.0.22 => 3.1.4
gatsby-plugin-react-helmet: ^3.1.15 => 3.2.4
gatsby-plugin-robots-txt: ^1.5.0 => 1.5.0
gatsby-plugin-sass: ^2.1.23 => 2.2.3
gatsby-plugin-sharp: ^2.5.3 => 2.5.6
gatsby-source-filesystem: ^2.1.38 => 2.2.4
gatsby-source-graphql: ^2.1.24 => 2.4.2
gatsby-transformer-sharp: ^2.4.2 => 2.4.6
npmGlobalPackages:
gatsby-cli: 2.12.34

File contents (if changed)

gatsby-config.js: N/A
package.json: N/A
gatsby-node.js: N/A
gatsby-browser.js: N/A
gatsby-ssr.js: N/A

[freiksenet]

You can create a catch-all client-only route that will match /search/*. This route will always be built and gatsby will choose existing routes as more important (as they have higher precedence), but it will fallback to this route in case the page wasn't there before.

See this doc https://www.gatsbyjs.org/docs/client-only-routes-and-user-authentication/

[BernardA]

Thanks for the prompt response.
I have implemented your solution as below, but there are a few issues with that. Some minor but a major one as well.

Here is the code that replaces the section after // BELOW DOES NOT WORK above.

    onCreatePage: async ({ page, actions }) => {
    const { createPage } = actions;
    // const component = path.resolve('./src/templates/searchWild.js');
    if (page.path.match(/^\/search\/*/)) {
        // page.component = component;
        // page.matchPath = '/search/[-a-z]+/[-a-z]+/[-a-z]+/[-a-z]+';
        page.matchPath = '/search/*';
        console.log('**PAGE**', page);
        createPage(page);
    }
},

Initially I thought I would be able to have those wild card pages created from a template. I attempted to do that with the commented out searchWild.js above. That works in a way, but then the actual page /pages/search is no longer reachable and an error is thrown on gatsby develop.

Then I attempted to weed out some of the garbage that could end up with that wild card with
page.matchPath = '/search/[-a-z]+/[-a-z]+/[-a-z]+/[-a-z]+'; ,
That does not work either as it seems that matchPath needs to be same as actual page.

So I ended up with above that works in a way with a big BUT.

You alluded to the precedence when Gatsby chooses the routes, given priority to existing ones, but that does not seem to be the case.

The main /search page seems to fall into the same onCreatePages and then is generated fully client side, as are all other routes created with the existing code in gatsby-node.js meaning they becomes useless for SEO, which is a big no no.
Looking at page source shows an empty <body> tag.

The pages generated with the above code, with the format /search/[-a-z]+/[-a-z]+/[-a-z]+/[-a-z]+ are meant to be temporary, just till a new build that will incorporate them as pre-built pages. So, not being visible for SEO for a day or so is not a huge problem.

But that's not the case for all /search/* page. They should be very much SEO friendly.

Am I missing something?

[wardpeet]

You'll have to create 1 search page that will be used as the root search page. All other pages will be client-side only. You can still create static pages for them but you should drop the matchPath for those pages.

To talk about SEO, how would Google be able to reach those search pages? If they are linked by other pages google will still find them. Google is crawling SPA and waits for them to complete. More info on this can be found at https://developers.google.com/search/docs/guides/javascript-seo-basics

We're marking this issue as answered and closing it for now but please feel free to reopen this and comment if you would like to continue this discussion. We hope we managed to help and thank you for using Gatsby! 💜

[BernardA]

@wardpeet, thanks but I do not feel that you answered the question or resolved the issue. So closing it is premature.

My needs are clear:
1 - /search pages should be SEO ready, easily available to search engines.
2- is not practical to pre-build 100% of the /search routes due to sheer volume, so some need to be generated on the fly and, for those only, SEO is not a real concern as they are temporary and will be replaced on the next build.

I may have chosen the wrong tool for the job, Gatsby in the case. If that's the case I expect you to be clear about that.

Directing me to a Google video about generics on the crawling is not an answer. I am aware of the limitations of Google crawling concerning javascript. Otherwise I would have used plain React to build the app.

[utoppo]

I'm in the same boat as @BernardA and would be really happy about a solution to this issue and think the issue should be opened again.

In my view, the scenario of temporary dynamic routes which can be built to static ones occasionally, should be a very common one. I would expect, that a lot of projects would benefit from a proper way of achieving this.

@BernardA
Did you find a solution or workaround?

[BernardA]

Yeah, I found a solution in a way. I moved over to Nextjs.
Gatsby does seem to be the wrong tool for the job and, unfortunately folks at Gatsby seem to prefer to sweep this under the rug - as they did by prematurely closing this issue - rather than either just assume it or trying to solve it.

Just to add that this is actually a pity, as Gatsby is way easier to work with than Nextjs.
Also, let me know if you do find a solution.

[utoppo]

Glad to hear, you found a working solution with the move to Nextjs! Seems like I should have a closer look at it for future projects.

Sad to hear you had to come to the conclusion, that Gatsby isn't suited for that kind of requirement. It would be great if this drawback could be clearly communicated in the Gatsby documentation, so one can decide upfront if Gatsby is the right tool for the job, or if one should look for other possible solutions.

BernardA [e-mail] schrieb am Mi., 19. Aug. 2020, 13:07:

Yeah, I found a solution in a way. I moved over to Nextjs.
Gatsby does seem to be the wrong tool for the job and, unfortunately folks
at Gatsby seem to prefer to sweep this under the rug - as they did by
prematurely closing this issue - rather than either just assume it or
trying to solve it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/25171#issuecomment-676158374,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIZFIAP7XMAUE66NRARXM3SBOXAZANCNFSM4OD4B6VQ
.

[utoppo]

@wardpeet Would it be possible for you to elaborate on why this kind of mixed dynamic/static routing-behavior can't be implemented with Gatsby? Or could you come up with steps that would be necessary to achieve this? Thank you very much in advance!

[mcclaskiem]

I just posted something in Discord regarding this exact issue on my project. We've been in development for over a year with hopes that a "real" solution would come forward but since this issue was just closed with no real explanation, Im assuming it has not been addressed. Maybe I am crazy but this seems like a pretty common flow for applications that generate pages based on data provided by users.

The last time around I was told to use the solution in this issue, however hacking the 404 page doesn't seem like a scalable solution, especially with Gatsby trying to pave a way for websites built with React. Millions in funding raised, countless promises in "re-defining" web development and hacking a 404 page is the best solution? I can only imagine that tons of developers have evaluated Gatsby and decided against it because of this exact reason.

Essentially our app allows users to create public profiles each with their own slug. What ends up happening is as soon as they create it, they are given the unique URL, but if they try to access it a 404 follows since that page does not exist in the Gatsby lifecycle. After implementing the "solution" above, we now have to use our 404 page to match a URL and decide whether or not to serve the public profile. My concern is that the browser still gets 404 code from the server and in the event a Search Engine bot hitting this page before a build has occurred they too would get a 404 and most likely abort crawling the page.

Any help/guidance on this issue would be greatly appreciated. We are about to move into production and I would like to have a working, scalable solution in place as soon as possible.

[LekoArts]

@mcclaskiem I think you should give the File System Route API a chance. If you run the example and watch this file you can navigate to a non-existent theme park:

The logic of the file prints out this then:

Using client-only routes like my colleagues already suggested is exactly this, only in a cleaner way of doing it. There's no need to hijack 404 pages.

[mcclaskiem]

@LekoArts Is this to be used in place of createPages and generating pages during build or does Gatsby know in it's lifecycle that there is a SSR version of the page and allow both to co-exist?

[LekoArts]

This is a replacement for the explicit createPages API, yes.

or does Gatsby know in it's lifecycle that there is a SSR version of the page and allow both to co-exist

That was always the case, already before the File System Route API, as my colleague pointed out here:

This route will always be built and gatsby will choose existing routes as more important (as they have higher precedence), but it will fallback to this route in case the page wasn't there before.

So while with the Route API this construct is more clear (generating pages and having a client-only fallback for when a page wasn't created yet):

- {Model.id}
--- {Model.name}.js
--- [name].js

This is also possible with the APIs inside gatsby-node.js

[mcclaskiem]

Using the below files:

Profile/index.js
Profile/[id].js

Could I make one GET request inside of gatsby-node.js at build time and call createPages on each profile using Profile/index.js as the component to mount. Then also have Profile/[id].js with a useEffect to fetch the required data on the client side as a fallback?

Just wanna make sure this isn't dependent on the default query that Gatsby looks for and also that things done in gatsby-node are aware of the File System Route API.

[LekoArts]

I'm not sure I completely follow. If you can, please create a discussion here so that you can share more detail: https://github.com/gatsbyjs/gatsby/discussions?discussions_q=category%3AHelp

[mcclaskiem]

@LekoArts Didn't even see this feature was rolled out yet, Thanks. I have linked the discussion thread I have created below. Really appreciate your help on this.

https://github.com/gatsbyjs/gatsby/discussions/28115