Next.js: getStaticPaths re-imports node modules preventing caching remote data in memory

[smnh]

Bug report

Describe the bug

When running next dev and requesting a page having the getStaticPaths method, node modules required by the page are re-imported, thus preventing caching headless CMS remote data in memory.

Same happens when running next build - modules required by page component that have getStaticPaths method are re-imported for every pre-rendered page. Making it impossible to fetch the whole remote data in a single API request and use it for all pre-rendered pages.

Important Note:
Headless CMS services may limit number of API requests per month and may apply charges when this limit is passed. The development and build of statically generated sites should minimize the usage of headless CMS API and re-fetch the data only when it is changed. The caching and data invalidation logic might be implemented by CMS clients. Additionally headless CMS services have endpoints to fetch the whole data in a single request. Therefore, to decrease the API usage and support caching, I think Next.js should allow importing modules that cache their data in memory and not re-import them every time page is pre-rendered, while running dev server or when building the site.

To Reproduce

Create simple page pages/[...slug].js

import React from 'react';
import pageLayouts from '../layouts';
import cmsClient from '../ssg/cms-client';

class Page extends React.Component {
    render() {
        // every page can have different layout, pick the layout based
        // on the model of the page (_type in Sanity CMS)
        const PageLayout = pageLayouts[this.props.page._type];
        return <PageLayout {...this.props}/>;
    }
}

export async function getStaticPaths() {
    console.log('Page [...slug].js getStaticPaths');
    const paths = await cmsClient.getStaticPaths();
    return { paths, fallback: false };
}

export async function getStaticProps({ params }) {
    console.log('Page [...slug].js getStaticProps, params: ', params);
    const pagePath = '/' + params.slug.join('/');
    const props = await cmsClient.getStaticPropsForPageAtPath(pagePath);
    // If not using JSON.parse(JSON.stringify(props)), next.js throws following error when running "next build"
    // Error occurred prerendering page "/blog/design-team-collaborates". Read more: https://err.sh/next.js/prerender-error:
    // Error: Error serializing `.posts[4]` returned from `getStaticProps` in "/[...slug]".
    // Reason: Circular references cannot be expressed in JSON.
    return { props: JSON.parse(JSON.stringify(props)) };
}

export default Page;

Implement simple singleton CMS client that fetches CMS data and caches it in memory:

class CMSClient {

    constructor() {
        console.log('CMSClient constructor');
        this.data = null;
    }

    async getData() {
        if (this.data) {
            console.log('CMSClient getData, has cached data, return it');
            return this.data;
        }
        console.log('CMSClient getData, has no cached data, fetch data from CMS');
        this.data = await this.fetchDataFromCMS();
        return this.data;
    }

    async getStaticPaths() {
        console.log('CMSClient getStaticPaths');
        const data = await this.getData();
        return this.getPathsFromCMSData(data);
    }

    async getStaticPropsForPageAtPath(pagePath) {
        console.log('CMSClient getStaticPropsForPath');
        const data = await this.getData();
        return this.getPropsFromCMSDataForPagePath(data, pagePath);
    }

    async fetchDataFromCMS() { ... }
    getPathsFromCMSData(data) { ... }
    getPropsFromCMSDataForPagePath(data, pagePath) { ... }
}

module.exports = new Client();

Navigate to any page rendered by [...slug].js, for example /about.
Following logs will be printed on server::

CMSClient constructor
Page [...slug].js getStaticPaths
CMSClient getStaticPaths
CMSClient getData, has no cached data, fetch data from CMS
Page [...slug].js getStaticProps, params:  { slug: [ 'about' ] }
CMSClient getStaticPropsForPath
CMSClient getData, has cached data, return it

• constructor is invoked - assuming [...slug].js module was loaded for the first time it is OK.
• [...slug].js calls getStaticPaths - OK according to Runs on every request in development
• getStaticPaths of the CMS client is invoked, it does not have the cached data because the client was just constructed therefore the getData is called for the first time - OK.
• [...slug].js calls getStaticProps - OK according to Runs on every request in development
• getStaticPropsForPath of the CMS client is invoked, it already has cached data so getData returns early returning the cached data - OK

Refresh the page or click a link <Link href="/[...slug]" as="/about"><a>About</a></Link>.
Following logs will be printed on server:

CMSClient constructor
Page [...slug].js getStaticPaths
CMSClient getStaticPaths
CMSClient getData, has no cached data, fetch data from CMS
Page [...slug].js getStaticProps, params:  { slug: [ 'about' ] }
CMSClient getStaticPropsForPath
CMSClient getData, has cached data, return it

As it can be seen the CMS client is constructed again, and every time a page is requested (even thought it uses the same page module), and the same steps related to fetching and caching the data are repeated. This behavior suggest that when page is requested and getStaticPaths is called, it re-imports all modules.

Note: When using getStaticProps without getStaticPaths, the client is not constructed on every request and therefore cached data is used as expected. See link to demo repository below.

Expected behavior

When running next dev server (or next build), the modules imported by a page component should be imported only once and reused to allow them cache remote data in memory.

Page [...slug].js getStaticPaths
CMSClient getStaticPaths
CMSClient getData, has cached data, return it
Page [...slug].js getStaticProps, params:  { slug: [ 'about' ] }
CMSClient getStaticPropsForPath
CMSClient getData, has cached data, return it

System information

• OS: macOS
• Browser: Chrome
• Version of Next.js: 9.3.0

Additional context

I've setup an example repository that I've used to reproduce this issue. It uses Sanity as Headless CMS. The README file has all the info needed to setup Sanity account and import the initial data used by this example site.

https://github.com/stackbithq/azimuth-nextjs-sanity/tree/nextjs-ssg-api
(use nextjs-ssg-api branch)

Note, when loading the root page '/' (pages/index.js) which has only the getStaticProps method and does not have getStaticPaths, the CMS client is not constructed on every request and therefore data cached in memory of the CMS client module is used as expected.

[ijjk]

Hi, we run getStaticPaths and getStaticProps in separate workers to ensure we're prerendering pages as fast as possible. In development, we make sure this separation is honored by running getStaticPaths in a separate worker also so that you don't see differing behavior between development and a production build.

If you would like to cache data between calls to getStaticPaths and getStaticProps you can use various strategies to achieve this like writing the cache to the filesystem in getStaticPaths and reading the data from the filesystem in getStaticProps or using something like Redis to store the data and query it from there.

The filesystem cache approach works pretty well on the Notion blog example and helps prevent re-fetching of data that has already been fetched.

[tomduncalf]

Oh man, this is a shame, makes my use case much more complicated (assumed state in a "store" class would be preserved in a single client side session as you navigate between pages, but actually the store gets recreated when you navigate to a page generated by getStaticPaths). Any way to disable the worker thread, or approach this in a different way? Having to serialize the client state into LocalStorage or whatever feels like overkill for my use case (its just UI state that should not persist beyond a single session)

[tomduncalf]

Actually, am I doing something wrong here? My site can be entirely run as SSR, and if I browse the exported site (just served from a web server, no Next.js server), I can see that when I navigate between non-getStaticPaths-generated pages, the browser just loads the extra bits of JS, as expected, but if I navigate to a getStaticPaths-generated page, the browser does a full reload of the page, which also makes the load noticably slower. Seems like strange behaviour is this is correct, so perhaps I have set something up wrong?

[tomduncalf]

I was doing something wrong heh, dynamic links need both href and as