Next-i18next: Let user define their own data source and caching stategy

[BigZ]

Is your feature request related to a problem? Please describe.

In my context, i need my translations to be coming from an external api. As I18next offer it, i would be necessary for me to have my data coming from somewhere else than a file. and i would like to define a caching strategy, as offered by i18next

Describe the solution you'd like

let the user define loadPath. let him use a custom i18next instance ?

[isaachinman]

First, please search issues before opening a new one. We just discussed this same request in #163.

Second, have you seen how we create the i18next instance? It ties directly into the config options, and is pretty core to this package. If you want to use your own i18next instance, you might as well roll your own solution.

[BigZ]

thanks for your quick answer.
as seen here you can't really override loadPath, neither can't you add caching strategies.
Even if it's not ideal in your point of view, why not leave a choice to the users ? i can start to work on that to offer you a PR but i would prefer you to let me know that this work is welcome or not

[isaachinman]

Hi @BigZ. What would you like to override loadPath with? If you explain what you want, I can try to help you achieve it.

I am very open to all suggestions and sensible ideas, but I don't have a clear idea what you are trying to do.

Did you see @capellini's idea here?

[BigZ]

hello @isaachinman
i want to override loadPath with an URL.
Also, i saw @capellini 's idea, but it looks like a workaround, and from what i understand from it, will only work client side. I (and i believe many people) need to fetch data from an api, then cache it as file/redis/etc...
eventually, the user want to be able to choose the provider for fetching and caching data.
look how many different backend options you have in 18next: https://www.i18next.com/overview/plugins-and-utils
I think people want to use next-i18next to simplify the serverside work, not to get a super-opinionatted way to get translations and a very narrow set of feature compared to what the original librabry has to offer, and ultimately i believe we want to have the same freedom with just some additional wrappers to make it work easily in nextjs.
So, where do we start ?

[isaachinman]

So, where do we start ?

You tell me, it's your request 🙂

This is an open source project, and driven by community contribution. Crucially, this project is "super" opinionated because it solves a _very hard problem_. I welcome you to take a look at the source and offer any ideas you might have.

The working assumption is that next-i18next offers an opinionated zero-config way for less-experienced users to achieve something previously not within their reach, and that more experienced users would be rolling their own implementations if they needed more complex solutions.

[capellini]

So, where do we start ?

@BigZ - are working on #151 which I think will address some of your concerns? This PR should allow you to define your own config.backend and, if you do, we won't touch it.

But #151 won't solve the issue of how/when to set preload and ns server-side. Would it be safe to assume that a user who has defined their own config.backend wouldn't want preload and ns defined for them? Or perhaps if a user has set preload and/or ns themselves, we should not touch it, but if they didn't we should set them?

If we can hash out a plan here, I can modify #151 to accommodate.

[isaachinman]

Would it be safe to assume that a user who has defined their own config.backend wouldn't want preload and ns defined for them?

So, the SSR process relies upon the fact that the i18n instance already has all namespaces in all languages loaded into memory. If users want to change preload and ns, they will need to be aware of this and be responsible for making that happen. They will very quickly run into edge case bugs and behaviour which is not obvious at first glance.

[BigZ]

thans @isaachinman for re-opening, and @capellini for pointing me to your PR
i think it is a first good step, but i don't see how we use something else than the default here as the use of node-fs-backend is hardcoded here: https://github.com/isaachinman/next-i18next/blob/master/src/create-i18next-client.js#L13

[isaachinman]

Again, I really do not think it's going to be possible to support other sorts of backends, and it's not something I am going to dedicate my personal time to.

This project is inherently opinionated and will most likely continue to be so for the duration of its life. If you have different opinions, you should most likely consume react-i18next and the various bits of the i18next ecosystem yourself.

That said, I am open to all reasonable suggestions.

[isaachinman]

I will leave this issue open until either a solution is proposed, or discussion dies out.

[isaachinman]

@BigZ Any further thoughts on this?

[capellini]

i don't see how we use something else than the default here as the use of node-fs-backend is hardcoded

To prevent the hardcoding, I suppose one thing we could do is let the user define a backend module (e.g. config.backendModule) in their configuration as a string, which we would then import (we'd probably want to have a whitelist of modules). if a user specifies a module, then we do not configure the backend at all for them.

[lucasfeliciano]

There are some points that concern me about this approach.

1. If next-i18next add support to an endpoint to fetch translations, the API should be opinionated to i18next standards to make it work properly, I don't know if this is a good idea.

2. As was mentioned before on the server we need to load all namespaces to bundle and serve only the namespaces that are needed for the respective page. So, if the translations come from an API when starting the server you should make a request to the API to fetch all translations and manage it yourself writing on disk or in memory

3. Caching should not be a responsibility of next-i18next, because if you change your translations in the external service next-i18next doesn't know how to invalidate the cache.

4. If your API for some reason goes down, your site won't have translations

With this in mind, let's say that we don't go with the opinionated solution, on your translation service ( API ) you manage the cache and you have an endpoint that returns a JSON with i18n.initialStore's format then we could implement something like this:

1. On getInitialProps, you make a request to the translation endpoint passing whatever you need to get the translation object, probably props.namespacesRequired, i18n.language and i18n.defaultLanguage

2. Once we have the JSON we use i18n.addResourceBundle(lng, ns, resources, deep, overwrite) to load the fetched resources into next-i18next ( https://www.i18next.com/overview/api#addresourcebundle )

[isaachinman]

@lucasfeliciano There are many things about this suggestion that concern me as well!

That said, your ideas here are very clear and articulate. It would indeed be sensible from a next-i18next perspective to require an initialStore override for "custom backends".

Let me just copy paste my response from #163 here:

My response to this the first time around is basically the same now: why are you dead set on doing this? It's poor design, in my opinion. If your locales live externally and are served by an API, you should add a build step that compiles the resources from your remote to the local filesystem. Then you can trigger a rebuild/deployment on your app whenever translations change (via a webhook, for example).

Otherwise: (1) you're going to be hitting a service over and over when the content presumably has not changed (in most cases), and (2) you have no control over when localisation changes go live, and lose the concept of versioning.

If you _really_ want this functionality, I would be open to any sensible PRs, but this is not something I am going to spend any of my own time working on. In short, fileSystemLoader _already exists_ in that you can just set up a build step on your own.

Basically, setting up this external backend with caching and cache invalidation and so on is essentially over-engineering something that should be in the fs. What happens if you invalidate your cache due to new translations, but then your translation-API goes offline? Does your entire frontend go offline?

I don't understand why some devs are opposed to redeploying their frontends when new translations become available. If you have a proper pipeline in place, this is the safest and most reliable way to do things. I would assume your localisations are being tracked via version control somewhere, anyways.

All of that being said, I am again open to merging anything reasonable, but this effort will need to be led by someone who is very clear on how next-i18next works, and has a simple/logical approach of how we can support this.

Take a look at these three props. These three things are, more or less, what we need to be able to perform SSR. If you want a custom backend solution, you must be able to provide these three props inside AppWithTranslation's getInitialProps.

[isaachinman]

Closing due to lack of activity - happy to reopen if desired.

[mehmetnyarar]

I’m working on a monorepo which has the following structure:

• server (express + GraphQL)
• web (nextjs)
• mobile (expo)

In order not to duplicate translations and manage them in a single place, I have I18next setup on my server which then returns translations via url /locale/:lng/:ns.

Below is the code for web/lib/i18n.ts:

import { I18NextXhrBackend } from 'i18next-xhr-backend'
import fetch from 'isomorphic-unfetch'
import I18Next from 'next-i18next'

const backend: I18NextXhrBackend.BackendOptions = {
  loadPath: '{{lng}}/{{ns}}',
  addPath: '{{lng}}/{{ns}}',
  ajax: async (url: string, _: any, callback: any) => {
    const backendUrl = `http://localhost:8090/locale/${url}`
    const res = await fetch(backendUrl)
    const locales = await res.json()

    console.log('i18n:backend', {
      backendUrl,
      res,
      locales
    })

    callback(locales, { status: '200' })
  },
  parse: data => {
    console.log('backend:parse', { data })
    return data
  },
  allowMultiLoading: false
}

const i18next = new I18Next({
  defaultLanguage: 'en',
  otherLanguages: ['ru'],
  localePath: 'static/locales',
  localeStructure: '{{lng}}/{{ns}}',
  localeSubpaths: 'none',
  backend
})

export default i18next
export const {
  appWithTranslation,
  withTranslation,
  config,
  Link,
  Router,
  Trans,
  useTranslation,
  i18n
} = i18next

I successfully get translations from the API but they aren’t loaded somehow. What should I do?

Also, as far as I’ve understood, it’s obligatory to create static/locales directory and json files. Could it be optional when the translations are loaded from an API?

[isaachinman]

it’s obligatory to create static/locales

No.

Could it be optional when the translations are loaded from an API?

Yes, that's what this issue is about.

[mehmetnyarar]

When I do below and delete the static/locales directory

const i18next = new I18Next({
  defaultLanguage: 'en',
  otherLanguages: ['ru'],
  backend
})

I get the following error:

Error: ENOENT: no such file or directory, scandir '/packages/web/static/locales/en'
at Object.readdirSync (fs.js:790:3)
at getAllNamespaces (/node_modules/next-i18next/dist/commonjs/config/create-config.js:49:19)
at _default (/node_modules/next-i18next/dist/commonjs/config/create-config.js:54:27)
at new NextI18Next (/node_modules/next-i18next/dist/commonjs/index.js:30:43)
at Object. (/packages/web/lib/i18n.ts:28:17)

So, what should I do so that I don’t have to create static/locales directory?

Could you also respond my first question?

I successfully get translations from the API but they aren’t loaded somehow. What should I do?

[isaachinman]

Did you read the source code which is throwing that error? By default, we don't require users to pass ns into their config object, and we scan the directory of translations to gather the list of namespaces.

Just pass your own array of namespaces and the filesystem should be ignored.

I successfully get translations from the API but they aren’t loaded somehow. What should I do?

Sorry, but I have no idea what this means or how to help. Feel free to provide a reproducible example and I will debug for you.