Next-i18next: Support slashes in namespaces (locale subdirectory)

[haggen]

Describe the bug

When there's a slash character (/) in the namespace the pre-loading of the translations (SSR, via namespacesRequired, fs-backend) stops working. The client translations (http-backend) will still work though. There's no warnings and it's inconsistent, so I'd consider it a bug.

Occurs in next-i18next version

v6.0.3

Steps to reproduce

1. Have at least two languages configured.
2. Create a locale file at ${localePath}/${defaultLanguage}/a/b.json and add a sample translation. Do the same for the other language.
3. Set namespacesRequired of a page to ["a/b"].
4. Use the sample translation in this page and visit the page.
5. You'll see the initial server-side loaded translation breaks (doesn't get translated, simply prints out the strings ID), but changing the language in the client will load the new translation correctly.

Expected behaviour

At step 5 above, the initial server-side loaded translation should have worked.

Screenshots

n/a

OS (please complete the following information)

• Device: Windows 10 PC
• Browser: Chrome 86.0.4240.111 (Official Build) (64-bit)

Additional context

I understand that the fs-backend would still break with slashes in the namespace, there's still a way to make it work using a loadPath function, as suggested here: https://github.com/i18next/i18next-fs-backend/issues/13.

I also noticed that when there's a / in the namespace, everything from that character on gets yanked but it happens before the loadPath call. i.e. If the namespace is pages/login loadPath will get just pages. Here's my test:

const localePath = "./public/locales";
const localeExtension = "json";

const i18n = new NextI18Next({
  // Rest of the configuration...

  backend: {
    loadPath: (lang: string, ns: string) => {
      if (typeof window === "undefined") { // a.k.a. if isServer
        console.log(ns); // => Will print out the broken namespace.

        return path.resolve(
          `${localePath}/${lang}/${ns}.${localeExtension}`
        );
      }
      return `/${localePath.substr(9)}/${lang}/${ns}.${localeExtension}`;
    },
  },

[isaachinman]

Hi @haggen – that sounds like an entirely upstream feature request (or bug). As in, once it's fixed in i18next-fs-backend, it's fixed in next-i18next. Let's track this discussion in that repo. Thanks!

[haggen]

Hi @isaachinman thanks for the reply. As I mentioned in my report, there's strange behavior in both parts.

1. i18next-fs-backend encodes slashes (and I'm assuming other characters).
2. next-i18next deletes everything after the slash.

Also from the looks of it it's a wontfix for fs-backend, but there's an alternative using a loadPath function, which next-i18next is breaking by deleting part of the namespace.

[isaachinman]

Would you be interested in working on a fix?

[haggen]

@isaachinman I tried to find where the namespace get changed but couldn't find it. Any pointers from the top of your head? If not it's cool I'll figure it out.

[haggen]

I'll document my progress here in case I have to stop mid things.

I'm suspecting it's this line: https://github.com/isaachinman/next-i18next/blob/abdf06545410f340b0529e3448f8b102ab840249/src/config/create-config.ts#L99

It seems we pre-collect all locale files based on the default language and use that as namespaces, but it's a shallow read, so, following my example above, only a get read from a/b. First instinct is to change that to something like glob to get all locale files with config.localeExtension within that directory. I'm testing it out...

The fact that the test is so artificial (we mock fs.readdirSync to return an array) make things a bit more complicated.

[isaachinman]

Yes it's a shallow read, mostly as a benefit to users, but you can provide your own array of namespaces into the config to override that. Really not sure this is a next-i18next issue.

[haggen]

@isaachinman Okay cool I got it working by providing my own ns array. But why do we have to provide the ns before hand when it's the server but not when it's the client? If you take a look at the line mentioned above (./src/config/create-config.ts:99) you'll see the collecting of namespaces happen inside an if(isServer()) but in the else bracket it's just ns = [defaultNS].

Here's my proof-of-concept in the context of my application:

const defaultLanguage = "pt-BR";
const localeSubpaths = {
  "en-US": "en-us",
  "pt-BR": "pt-br",
};
const localePath = path.resolve("./public/locales");
const localeExtension = "json";

/**
 * Main instance of i18n.
 */
const i18n = new NextI18Next({
  defaultLanguage,
  otherLanguages: ["en-US"],
  localeSubpaths,
  localePath,
  localeExtension,

  ns:
    typeof window === "undefined"
      ? require("glob")
          .sync(`${localePath}/${defaultLanguage}/**/*.${localeExtension}`)
          .map((file) => {
            return file.replace(
              new RegExp(
                `${localePath}/${defaultLanguage}/(.+?).${localeExtension}`
              ),
              "$1"
            );
          })
      : ["common"],

  backend: {
    loadPath: (lang, ns) => {
      if (typeof window === "undefined") {
        return `${localePath}/${lang}/${ns}.${localeExtension}`;
      }
      return `${localePath.substr(8)}/${lang}/${ns}.${localeExtension}`;
    },
  },
});

[isaachinman]

Because on the server, we load _all namespaces_. Then, for each client request, we only send down the necessary namespaces to render that page, based on namespacesRequired. Hope that makes sense.

[haggen]

First of all, huge thanks for the quick feedback @isaachinman, please don't let me disturb you too much! 🍻

Well then I guess my point is, if we collect the namespaces (shallowly) beforehand due to a next-i18next requirement and it's breaking a usage that would otherwise be viable it's a bug. I must say I find this "viable usage" quirky, but it still a viable option, and one that was suggested to me on a i18next-fs-backend's issue: https://github.com/i18next/i18next-fs-backend/issues/13.

The way I see it, we have three options:

1. Change the namespace collection to a deep traverse of the locales path.
2. Detect a slash in the namespace and display a friendly warning.
3. Do nothing in the code, and leave a note in the docs about slashes in the namespace.

I confess none of them are particular ideal but I still think it's an odd behavior and it caused a lot of confusion. I guess option 1 would be great if they end up deciding to support deep locale files upstream.

[isaachinman]

Do you have a proof of concept for deeply traversing locale paths? Unless it's dead simple, I would prefer to simply add a note to the docs, as this is basically just user configuration.

[haggen]

We could use a package for that, which should be safer, or we could it manually.

Package options:

1. glob · 36m downloads/week · 10kb gzipped · 10 deps
2. @nodelib/fs.walk · 7m download/week · 3kb gzipped · 5 deps

Better comparison: https://npmcompare.com/compare/@nodelib/fs.walk,glob

Should look like this:

glob
  .sync(`${localePath}/${defaultLanguage}/**/*.${localeExtension}`)
  .map((file) => {
    return file.replace(
      new RegExp(
        `${localePath}/${defaultLanguage}/(.+?).${localeExtension}`
      ),
      "$1"
    );
  })

Manual option:

This would involve making a recursive function that walks over the locale path of the default language, tests each item to see if it's a directory and walk over those or if it is a file and has the configured extension, and save the path segment corresponding to the namespace (full path minus locale path and extension). Sounds larger than it is, should be about 20 short lines.

[isaachinman]

My preference is to simply add documentation, as this issue has rarely/never come up in the 2+ years this package has been around.

[haggen]

It surprises me that nobody needed better locale files organization (perhaps people end up using other backends) but your decision makes sense to me. Thanks again for the quick feedback @isaachinman, I'll look into the docs and leave a PR soon.

[Cedric-Delacombaz]

@haggen Do you have a working repository you could share? I would love to have such a file organization but only by reading the docs and previous issues about this, it was too complicated and I just gave up on it. I even went to look for other libraries because of that missing feature, but finally came back as I also couldn't find this feature anywhere.

[haggen]

@Cedric-Delacombaz I can whip something up later today. I'll update you as soon as I can.

[haggen]

@Cedric-Delacombaz Here's a working example: https://codesandbox.io/s/next-i18next-locale-subdirectories-5ys9t

It's based of the example provided by next-i18next (https://github.com/isaachinman/next-i18next/tree/master/examples/simple). The important part is in i18n.js.

[cdelacombaz]

Thanks a lot haggen!