Js-lingui: Compiling the catalog at runtime

[edorivai]

First off, thanks for the awesome library. The day-to-day dev experience seems a lot nicer than react-intl - where I've migrated from :).

The problem

We're fetching translations from a translation platform at runtime - we want our translators to be able to see their changes in (near) real time on our staging environment. I could be wrong, but AFAIK the lingui compile workflow is optimized for compile time. What I mean by this, is that it outputs a JS file, which can then be imported by Node (in case of SSR) or Webpack (for the frontend).

Given this background I haven't found a straightforward way to set this up. My current solution works like this:

• For the frontend we expose an endpoint like /api/lang/nl.js. This endpoint uses @lingui/cli/api#createCompiledCatalog to build a JS file which exposes the catalog on the window. This is actually the least painful step.
• For the backend we use createCompiledCatalog to compile a CJS bundle, which we are then force to run through require-from-string. I mean, I guess it works pretty well. It just feels kinda weird to start from a plain JSON catalog, then run this through a compiler that turns the plain JSON into valid JS, which we then compile to get a working catalog at runtime. I feel like it should be possible to go JSON :arrow_right: runtime catalog without the transpilation/compilation steps.

The solution I'd like

I'd like an API which generates a runtime catalog - meaning an in-process Object which I can pass directly into the <I18nProvider>.

Alternatives I've considered

Not sure if this is helpful, but I've considered writing the CJS catalog to disk, and then requireing it in Node. This has an obvious problem: Node's module caching. Given that we want to be able to update the translations at runtime, we would have to clear the cache for the module, either by explicitly doing something like delete require.cache[require.resolve('./catalog.js')], or by hashing the catalog's content, and writing to ./{hash}-catalog.js, which would bust the cache every time the contents change.

Additional context

If this is something that you'd consider adding, I think this might be a good candidate for v3 #334.

@tricoder42 mentioned something related here: https://github.com/lingui/js-lingui/issues/293#issuecomment-416272667

I'm trying to say that my original proposal isn't about loading locales, but only message definitions. No matter if messages are extracted from source files or remote API, first you always need just message definitions and then you create locale files by merging with existing catalogs. Yes, we could also add loading translations from API, but that's another step.

I'm not sure if what you're thinking would solve my problem, or whether "loading translations from API" would remain at compile time.

[tricoder42]

Thanks for detailed description of your usecase.

Does your staging environment runs with NODE_ENV=development? If so, the core library loads required localeData (plurals) and compiles messages at runtime:

https://github.com/lingui/js-lingui/blob/d33f9b3fc19197d916fb42254bbdb78ac664a819/packages/core/src/i18n.js#L204-L208

All you need to do is to load catalog as a plain JSON:

import { setupI18n } from "@lingui/core"
// load initial catalog, just JSON
import messagesEn from "./locale/en/messages.json"

const i18n = setupI18n()
i18n.load({
  en: messagesEn
})

// I don't know your localization provider, but if they have an API which
// notifies you about updates in catalog, it might look like this:
const localizationProvider = new LocalizationProvider()
localizationProvider.onChange = (locale, catalog) => {
  // when the catalog changes, load it
  i18n.load({ [locale]: catalog })
}

Now, the only problem is that you need to run it in React component and save catalogs to state so you can pass it to I18nProvider. This will be changed in #334 when i18n core object is the single source of truth for locales and catalogs.

---

If you don't run staging with NODE_ENV=development then it might be a bit tricky, because we need to expose dev/compile.js in the compiled source and then you need to add it to i18n._dev...

Anyway, this will definitely become easier in v3. The core API is more flexible and I was thinking about exposing runtime compiler.

---

For the backend we use createCompiledCatalog to compile a CJS bundle, which we are then force to run through require-from-string. I mean, I guess it works pretty well. It just feels kinda weird to start from a plain JSON catalog, then run this through a compiler that turns the plain JSON into valid JS, which we then compile to get a working catalog at runtime. I feel like it should be possible to go JSON ➡️ runtime catalog without the transpilation/compilation steps.

lingui compile does more than compile messages to JS functions:

• it resolves default and fallback messages
• it bundles localeData (plurals) to compiled catalog
• finally, it compiles messages with formatting to functions. If you don't use formatting, you'll probably see just plain strings, but it's not that simple in general case.

I want to create a detailed benchmark about using compiled message catalog and compile them at runtime, but first I want to finish refactoring in #334 before adding more features.

[edorivai]

Wow, that was really quick.

Our staging environment is not running with NODE_ENV=development, as we want staging to mimic production as much as possible. Moreover - even though production might not sync as often as staging, we eventually want the translations to end up in production as well, ideally without requiring a build.

I want to create a detailed benchmark about using compiled message catalog and compile them at runtime

This would be really awesome. I was just assuming compiling them at runtime would be a significant performance hit, but we can actually not be sure before we measure I guess.

Even though I like Lingui a lot more that react-intl so far, the fact that react-intl expected a plain JSON catalog did simplify our workflow a bit. If the performance of runtime compilation turns out acceptable, I think it would be a nice solution :rocket:

[tricoder42]

Well, the performance can't be worse than react-intl or other message format libraries.

My initial motivation was reduction of bundle size, because the parser is large. However, stripping runtime parser isn't completely free and I want to run few benchmarks to know the real costs and benefits.

[edorivai]

FWIW, if the bundle size increase _with_ runtime parser turns out to be significant, I would consider doing compilation serverside. As mentioned in the first post, our frontend workflow is quite alright. The major pain point is in the backend. If this would make implementation easier, having some kind of flag that I could set on <I18nProvider> to trigger the NODE_ENV=dev behavior, I guess that would work for us as well.

[edorivai]

Or now that I think of it, maybe an API like:

// Compiled
<I18nProvider catalogs={{...}}>

// Raw, catalogs will be parsed at runtime
<I18nProvider rawCatalogs={{...}}>

[tricoder42]

Most likely it would be something like this:

import { setupI18n } from "@lingui/core"
import messageFormat from "@lingui/messageformat"

const i18n = setupI18n()
i18n.runtime = messageFormat

// and later in React

<I18nProvider i18n={i18n}>...</I18nProvider>

I want to keep door open for Fluent syntax.

This could be actually done in minor release. We just need to rename _dev property and export runtime parser as a separate package. Unfortunately I want to focus on #334 to start testing it asap.

[edorivai]

Yeah no rush, you're going at an amazing pace already :sweat_smile:

That API looks fine by me :+1:

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.