Next.js: Initialize code at server startup and make it available in ctx

[c100k]

Feature request

Is your feature request related to a problem? Please describe.

In almost all the applications, we need to write code that needs to be ran only once when the server starts in order to initialize "global" elements:

• Connecting to a database
• Starting some jobs
• etc.

In the current version, we don't see how to do it. There are some hacky ways of initializing these but they do not seem 100% reliable. (i.e. : https://www.codeoftheprogrammer.com/2020/01/16/postgresql-from-nextjs-api-route/)

Describe the solution you'd like

It would be very handy to be able to define a file where all this initialization code can be added following the example of _app.js or _document.js.

It could be called _init.(js|ts). When present, the next command would execute it before starting the actual server. Here is what it would look like:

// Here is the signature of the function, it should not accept any arguments
// It returns a Promise with custom context that next will set in `ctx`
export default async function _init(): Promise<ICustomContext> {
    // Connect to a PostgreSQL database
    const dbClient = await connectToDatabase('psql://...');

    // Start some jobs
    setInterval(() => {
        // eslint-disable-next-line no-console
        console.log('This is a global setInterval that would run while the server is running');
    }, 1000);

    // The returned object will be available in pages through `ctx`
    return {
        dbClient,
        someOtherValue: 'GlobalSetting',
    };
}

Describe alternatives you've considered

As an alternative, we can implement global code initialization using NodeJS.Global but it seems too hacky : https://www.codeoftheprogrammer.com/2020/01/16/postgresql-from-nextjs-api-route/.

Additional context

Might be related in some way to #8969

[mattcarlotta]

You may want to use a custom server if you need to initialize custom jobs/services vital to a connection; otherwise, off-load them to a microservice. However, if you're trying to avoid a custom server, another approach I've found is to store the pg-promise connection into a runtime configuration variable. While this also a bit hacky and also won't work with serverless, it at least avoids polluting the global/window spaces.

This will also work if you're using pg-monitor as it can only monitor one connection at a time without calling detach.

Click to expand example

database/index.js

const bluebird = require("bluebird");
const monitor = require("pg-monitor");
const pgPromise = require("pg-promise");

const initOptions = { promiseLib: bluebird, noWarnings: false }; // Database options
const pgp = pgPromise(initOptions); // initialize pg-promise with options

const { NODE_ENV, DB, DBHOST, DBPASSWORD, DBPORT, DBUSER } = process.env;

if (NODE_ENV === "development") {
  monitor.attach(initOptions, ["query", "error"]);
} else {
  monitor.attach(initOptions, ["error"]);
}

module.exports = pgp({
  database: DB,
  host: DBHOST,
  password: DBPASSWORD,
  port: DBPORT,
  user: DBUSER,
});

database/connection.js

const getConfig = require("next/config");

const { db } = getConfig().publicRuntimeConfig;

module.exports = db;

next.config.js

require('dotenv').config()
const db = require("./database");

module.exports = { 
  publicRuntimeConfig: { 
    db 
  }
};

[c100k]

Mmmh thanks @mattcarlotta, I hadn't seen this next/config solution. It looks better than polluting global indeed.

I've had a look at the docs. Is there a reason why you would put your db in publicRuntimeConfig instead of serverRuntimeConfig and therefore making it available in the client ?

[mattcarlotta]

Depends on how you're using db. If you're using getInitialProps, and calling the db directly, then you'll want publicRuntimeConfig. If you're using getServerSideProps, then you'll want to use serverRuntimeConfig. Otherwise, if you're just making AJAX requests to the api pages, and the db logic is used within the api page, then you'll want to use serverRuntimeConfig.

[c100k]

So after some testing, it turns out one cannot easily do what you recommend @mattcarlotta because it is impossible to have a next.config.ts if you develop in TypeScript. Therefore, you need your whole chain to be JavaScript otherwise you can't require/import your code in next.config.js. 😕

[mattcarlotta]

Looks like Zeit doesn't recommend Typescript (nor do I, but for more personal reasons) because of the extra transpile/compile overhead. That said, I don't know much about Typescript, but you may be able to find a workaround here.

Otherwise, if you want type checking, use prop-types or use flow.

[adorsk]

Some other use cases for something like an 'preServerStart' hook:

1. Setting up a store or DB that uses the filesystem. For example, a leveldb instance or a sqlite instance.
2. Reading data from a file.
3. Setting up a local 'scratch' dir, relative to the build file.

These use cases sometimes come up when making small tools or hack projects. For cases like these it would be nice to not have to use a custom server, but to instead hook into the server start-up.

[jmoo]

So after some testing, it turns out one cannot easily do what you recommend @mattcarlotta because it is impossible to have a next.config.ts if you develop in TypeScript. Therefore, you need your whole chain to be JavaScript otherwise you can't require/import your code in next.config.js. 😕

I was able to hack together a solution for this that works extremely well at the moment but due to relying on nextjs internals it may break in the future. It works with typescript and hot module reloading.

Here's what you'll have to do to get this working:

1. Create a ./src/bootstrap.ts that will handle everything you want to do on server start.
2. Get your server-side typescript code into the webpack toolchain and externalize all of your imported modules.(Generalizing this to automatically import all of your server-side code is left as an exercise for the reader)
   // next.config.js example module.exports = { webpack: (config) => { if (config.output.path.match(/server$/i)) { const entryConfigFunction = config.entry; config.entry = (context) => entryConfigFunction(context).then((entry) => { entry['foo-service'] = './src/foo-service.ts'; entry['bootstrap'] = './src/bootstrap.ts'; return entry; }); } // you will need to replace this regex with a function that fixes the import paths once you have modules in nested directories config.externals.push(/foo-service|bootstrap/); } }
3. Create a custom server.js that will require your compiled application bootstrap.

// server.js example
const nextToolchain = next(...);
const handler = nextToolchain.getRequestHandler();

const boot = () => {
   const app = require('./.next/server/bootstrap');
   app.boot();
}

nextToolchain.prepare().then(() => {
   boot();
   server = createServer(handle);
   server.listen(parseInt(process.env.PORT, 10), process.env.HOST);
});

1. At this point production builds should work but to get the dev server working, you will need to hook into the next internals and reload your application bootstrap after every compilation.
   ```
   // server.js
   nextToolchain.prepare().then(() => {
   boot();
   server = createServer(handle);
   if (dev) {
   require('next/dist/build/output/store').store.subscribe(async (state) => {
   if (!state.loading && !state.typeChecking && !state.errors && !state.warnings) {
   boot();
   }
   }
   }
   server.listen(parseInt(process.env.PORT, 10), process.env.HOST);
   });