Keystone: Multi-lingual support (translated fields)

[molomby]

For supporting multilingual sites and apps, it will help if Keystone itself has some concept of multiple languages. The conversation around this so far is based on a few ideas, namely:

• The languages supported by a single app will be consistent across the app. That is, different lists or area of the app or site, will support the same set of languages.
• That in a typical multilingual system, only a subset of fields (those that are user-facing) require translation. Internal labels, statuses, descriptions, etc. and user entered data (names, posts, etc.) do not.
• If a user/API-consumer wishes to operated within a specific language, the system should automatically (by default) supply them the correct version of any multilingual fields they request.

This starts to take the shape of a technical design in which:

• A developer can configure a set of languages (and a default) at the Keystone level.
• Within a list, fields (which support it? eg. text, HTML, Markdown?) can be nominate as "multilingual". This causes:
  
  
  
  • Multiple columns/fields to be created for them at the DB layer
  
  
  • The GraphQL layer will continue to return a single field value but this can be controlled by field argument (or a header or cookie?)
  
  
  • In the Admin UI, multilingual fields present tabs (or similar) so their different translations can be edited/populated
  
  
  • Maybe some additional filters are added? (Eg. give me a list of all items where the blahBody field has no German translation.)
  
  

This feature likely interacts with:

• Full-text search (#320) and filtering (#319), both in terms of limiting the translation to search and using indexing appropriate for different languages. Mongo only supports a single text index per collection and, as far as I can tell, only a single language per _document_ 🙁. This has significant storage implications if we need to support multilingual content and full-text search concurrently (which seems likely).
• Default formatting of fields (eg. dates and times). It's part of a larger i18n and l10n discussion.
• Item versioning (#321)
• Schema building and migrations (#298). Eg. altering the list of supported languages requires schema changes/possible data migration.

We should examine how these ideas are managed in other, related systems like Contentful.

[molomby]

Related conversations in KS 4:

• Multiple language support
• Internationalization

[camcnz]

Here is how I did it on a recent KS 4 project with multiple language and field text search.
I had translatable wysiwyg fields in there too.

locales.js

exports = module.exports = {
    languageListObj: {
        en: { enLabel: 'English', naLabel: 'English' },
        fr: { enLabel: 'French', naLabel: 'Français' },
        de: { enLabel: 'German', naLabel: 'Deutsch' },
        it: { enLabel: 'Italian', naLabel: 'Italiano' },
        es: { enLabel: 'Spanish', naLabel: 'Español' },
        ja: { enLabel: 'Japanese', naLabel: '日本語' },
        zh: { enLabel: 'Chinese', naLabel: '简体中文' },
    },
};

Helper_StandardContent.js

const _ = require('lodash');
const keystone = require('keystone');
const Types = keystone.Field.Types;
const { languageListObj } = require('../locales');

let translation = {};
let name = {};

for (const key in languageListObj) {
    if (languageListObj.hasOwnProperty(key)) {
        const element = languageListObj[key];
        let fieldDepends = `translation.${key}`;
        if (key !== 'en') {
            translation[key] = { type: Types.Boolean, label: `${element.enLabel}` };
        }
        if (key !== 'en') {
            name[key] = {
                type: String,
                label: `${element.enLabel} name`,
                dependsOn: { [fieldDepends]: true },
                index: true,
                validate: {
                    validator: function (v) {
                        return (_.get(this, fieldDepends)) ? (v && v.length > 0) : true;
                    },
                    message: `Please provide a name!`,
                },
            };
} else {
            name[key] = {
                type: Types.Text,
                label: `Default name`,
                initial: true,
                index: true,
                validate: {
                    validator: function (v) {
                        return (v && v.length > 0);
                    },
                    message: `Please provide a name!`,
                },
            };
exports = module.exports = {
    translation,
    name
};

ShopProduct.js

const { languageListObj } = require('../locales');
const { translation, name } = require('./Helper_StandardContent');

const ShopProduct = new keystone.List('ShopProduct', {
    label: 'Products',
    map: { name: 'name.en' },
    autokey: { path: 'slug', from: 'name.en', unique: true },
    track: { createdAt: true, updatedAt: true },
});

ShopProduct.add(
    'Translation',
    {
        translation,
    },
    {
        name,
                brand: { type: Types.Relationship, ref: 'ShopProductBrand', createInline: true, index: true },
        brandTitle: { type: Types.Text, hidden: true },
})


ShopProduct.schema.pre('save', async function (next) {
    let item = await new Promise((resolve, reject) => {
        keystone.list('ShopProductBrand').model.findOne({ _id: this.brand }, (err, data) => {
            if (err) reject(err);
            else resolve(data);
        }).lean();
    });
    if (item) {
        this.brandTitle = item.brandTitle;
    }
    return next();
});


ShopProduct.schema.plugin(mongoosePaginate);

ShopProduct.defaultColumns = 'name.en, brand, categories, gender, homepageHero, state, publishedDate';
ShopProduct.register();

// ShopProduct.model.collection.dropIndexes();

// Create text search index.
ShopProduct.model.collection.createIndex({ 'brandTitle': 'text', 'name.en': 'text', 'name.de': 'text', 'name.fr': 'text', 'name.it': 'text', 'name.es': 'text', 'name.ja': 'text', 'name.zh': 'text' }, {
    name: "TextIndex"
    }, function (error, res) {
    if (error) {
        return console.error('failed ensureIndex with error', error);
    }
    console.log('ensureIndex succeeded with response', res);
});

Product.js

exports.getAllProducts = async (req, res) => {
    try {
        let query = (req.user && req.user.isAdmin) ? {} : { state: "published", publishedDate: { "$lte": new Date(Date.now()) } };

        const locale = req.query.locale || 'en';

        if (req.query.text) {
            query.$text = { $search: req.query.text, $language: locale };
        }

        if (req.query.gender) {
            query.gender = req.query.gender;
        }

        if (req.query.categories) {
            query.categories = req.query.categories;
        }

        let options = {
            page: req.query.page || 1,
            limit: 20,
        };

        if (req.query.text) {
            options.sort = {
                score: {
                    $meta: 'textScore',
                },
            };
            options.select = {
                score: {
                    $meta: 'textScore',
                },
            };
        } else {
            options.sort = {
                publishedDate: -1,
            };
        }

        const items = await ShopProduct.model.paginate(query, options);

        res.apiResponse({
            items,
        });
    } catch (err) {
        res.apiError('database error', err);
    }
};

[stale[bot]]

It looks like you haven't had a response in over 3 months. Sorry about that! We've flagged this issue for special attention. It wil be manually reviewed by maintainers, not automatically closed. If you have any additional information please leave us a comment. It really helps! Thank you for you contributions. :)

[GasimGasimzada]

I have been researching this idea for some time now and I have a proposal. Firstly, in terms of releasing, I think it will be easier to do this in two steps: Localization for the Core/API and Localization for the Admin.

Now, let's get into it.

Localization Core

Initially, we need a way to define localization for "static messages." Things like validation messages, field labels, etc. So, I suggest adding a locales parameter in Keystone instance:

const keystone = new Keystone({
  name: 'New Project',
  adapter: new MongooseAdapter(),
  locales: [localeEnUs(), localeEnUk(), localeDeutsch(), ...]
});

Locale functions

As you can see, these locales do not store string value. They are functions with specific values (I have not thought of the implementation details but just the design):

// I am using momentjs formatting here but it really doesn't matter

const localeEnUs = custom => ({
  locale: 'en-us',
  languageCode: 'en',
  languageInNative: 'English',
  defaultLongDateFormat: 'dddd, MMM Do YYYY',
  defaultShortDateFormat: 'MM/DD/YYYY',
  messages: {
      api: {
        'inbox.available.one': 'You have 1 item in your inbox',
        'inbox.available.many': 'You have %d items in your inbox',
      },
      admin: {
          ...
      },
      ...custom
   }
});

This way, all the language information is contained in a single object. If a translation does not exist in a specific language, default translation will be used.

Installing Locales via NPM

Secondly, this type of containment with objects has an advantage. Since, a lot of messages are based on internal values (validation errors, admin translations), these messages should be provided by Keystone. Since, loading all locales can increase the bundle size significantly, we can separate them into separate NPM packages:

yarn add @keystonejs/locale-en-us @keystonejs/locale-de-de

# or

yarn add @keystonejs-locales/en-us @keystonejs-locales/de-de

Then store these locales in a separate monorepo (let's call it keystonejs/locales). Then, people can fork and create PRs for their own localization files and push PRs to have their localization files added to the main repo.

Custom translation messages

What if we need custom translation messages that is available to our frontend app or to a custom controller? We can just pass a JS object to the locale function to have custom variables available. Example:

const customDeDe = require('./locales/de-de.json');
const customEnUk = require('@my-own-project/localizations');
const customEnUs = { ... };

const keystone = new Keystone({
  name: 'New Project',
  adapter: new MongooseAdapter(),
  locales: [localeEnUs(customEnUs), localeEnUk(customEnUk), localeDeutsch(customDeDe), ...]
});

This allows for developer do whatever they want with their locales: they can store them in the same file as Keystone instance, store them in a separate module or just a JSON file, or publish their own localization module and import the object. Sky is the limit as long as an object is passed.

Using translation strings

In keystone, we provide a simple "translate" function. Typically, these functions have a very short name. For example, we can call it t(domain, name, default, params). Here is how to use it:

const myNumber = getNumber();
if (myNumber === 1) {
  const msg = keystone.t('api', 'inbox.available.one', 'You have 1 item in your inbox'); 
} else if (myNumber > 1) {
  const msg = keystone.t('api', 'inbox.available.many', 'You have %d items in your inbox', [myNumber]); 
}

Default value is implemented during function call. This way, if there is no value in the messages object for the current language, default one will be used. If you want to print custom values, you just need to use the domain name and the key of the custom message:

const msg = keystone.t('myCustomModule', 'hello.world', 'Hello World);

Migration

We need to use keystone.t in all our validation messages instead of normal messages.

API

Now, there needs to be a way for apps to request for two things based on locale: Localized messages and localized field values (we will discuss this one later).

Language Detector

There are multiple ways to detect languages in APIs: Cookies, query strings, Accept-Language header. My understanding is that, keystone uses express server. We can use a middleware that detects the language (or write our own) and then match the found locale with accepted locales that is based on locales parameter. However, if no locale is sent to the user, we might need a fallback / default locale. We can pass a fallbackLocale / defaultLocale parameter to Keystone instance:

const keystone = new Keystone({
  name: 'New Project',
  adapter: new MongooseAdapter(),
  locales: [localeEnUs(), localeEnUk(), localeDe(), ...],
  defaultLocale: 'de-de' // default is always the first item in the `locales` array
});

Setting language

Once the language is detected, we can store it in request context; thus, use the active language withing keystone.t or custom fields to identify what to show.

Getting language info (if needed)

We can also add a field in GraphQL to retrieve locale info and messages:

{
  localeInfo {
    nativeName
    code
    locale
    messages {
       admin {
         key
         value
       }
    }
  }
}

---

Admin Localization

Admin localization consists of couple of parts: Fields, View localization, and Language Switcher.

Fields

To make fields translatable, I suggest adding a Localization field that accepts a field parameter:

const { Text, Localized } = require('@keystonejs/fields');
keystone.createList('Post', {
  fields: {
    title: { type: Localized, field: Text },
  },
});

This field uses implementation of an existing field and adds localized functionality to it. This functionality consists of storing field values in a localized manner, getting localized and all versions of the field (needed for admin), and showing localized field in admin.

Database

For Mongoose adapter, I suggest storing field in the following way:

{
  "field": {
    "en-us": "...",
    "en-uk": "..."
  }
}

For SQL adapter, I suggesting storing the field in a related table:

// Sorry, not using the SQL syntax here. Just a simple demonstration of the database

Table: posts
 - id
 - stuff here...

Table: post_localized_fields
 - post_id: FK -> posts
 - locale
 - field
 - value
UNIQUE(post_id, locale, field)

Retrieving all localized fields from GQL

We need a way to get these values from GQL. When dealing with GQL, we can add an additional field that is only available for admins (through Access Control). Example:

{
   posts {
      titleLocalized {
        locale
        value
      }
   }
}

Viewing Localized Fields

I think the easiest way to manage views for a localized field is to add Tabs on top of the field. Each tab shows the locale code the the fields are shown within the tab. When preparing the request for the view, a schema like this will be created:

{
   "en-us": "...",
   "de-de": "..."
}

However, view code has one minor issue. Labels for each input field is always visible. I suggest hiding them (screen reader friendly) when they localized and change the labels by adding localization in parenthesis. Example (not using arch-ui, just basic, static JSX to show what I mean):

<h4 className="label">Title</h4> {/* Visible Label */}
<Tab name="English">
  <label className="sr-only" htmlFor="title-en-us">Title (English)</label
  <input type="text" id="title-en-us" ... />
</Tab>
<Tab name="Deutsche">
  <label className="sr-only" htmlFor="title-de-de">Title (Deutsche)</label>
  <input type="text" id="title-de-de" ... />
</Tab>

View Localization

We might need a way to localize, not just custom fields but the entire template (all the texts etc). I have an idea to do this in an optimized fashion. Firstly, we add a field to GQL that shows a "hash" of the all localizations. This value is generated only and stored once -- when the server is initialized and in keystone instance. Let's call this field a "localizationHash."

This field can be used by all static apps. When admin app is opened, "localizationHash" is requested from the API. Then, this hash is compared with a localizationHash key in localStorage. If values are equal, it means that we have the latest localization messages. If they are not equal (update or if value does not exist), admin app requests all the messages for the current locale and stores the values in localStorage with the new localizationHash. Then, these values will be displayed immediately. This will work great because content editors will not be changing their locales all the time. They will use the locale that they prefer (e.g French content creator might like the entire view to be in French) while also edit values from other locales since these values are custom fields that are available in all locales.

---

That's All

I know it is a long post but I wanted to suggest my proposal in detail here. If you have any questions on this or don't like something, let me know. Once a specific design is agreed upon, I am willing to implement this functionality step-by-step.

Future and Other Consideration

Once there is a localization system in place, we can use Access Controls for localized content. Developer should be able to restrict certain locales of certain fields. For example, developers should be possible for restrict a French content creator to only view and update fields that are for French localization.

[anjoalre]

Any development on this!?

[GasimGasimzada]

I haven’t done anything on this because I wrote this as a proposal to be discussed. This is a big change to the framework; so, I wrote to this to get to an agreement before going into implementation.

[justintemps]

Enthusiastically support @GasimGasimzada's idea as I'm going to have to bootstrap a multilingual support for my next project and I don't think my solution will be as good.

[lucatk]

Any update on the proposal status? @GasimGasimzada
I'd be willing to contribute on this, as localization functionality is a major requirement for a project of mine intended to be migrated to KeystoneJS currently.

[justintemps]

I'd be happy to help out as well.

[stale[bot]]

It looks like there hasn't been any activity here in over 6 months. Sorry about that! We've flagged this issue for special attention. It wil be manually reviewed by maintainers, not automatically closed. If you have any additional information please leave us a comment. It really helps! Thank you for you contribution. :)

[phoniks]

I could use this on a project I'm working on, it sounds like there's some enthusiasm for the proposal. @GasimGasimzada - it's been a while since you originally proposed this, would you still be willing to lead? I would also be willing to contribute where I can. I think this is a pretty worthwhile addition and I'll wind up doing something less complete for the project I'm currently working on if this doesn't happen, so I'd much rather contribute the hours that I'm going to spend on the problem to the implementation of this proposal as well. Any comments from the maintainers? Any reason why it's a bad idea for us to start working on this?

[stale[bot]]

It looks like there hasn't been any activity here in over 6 months. Sorry about that! We've flagged this issue for special attention. It wil be manually reviewed by maintainers, not automatically closed. If you have any additional information please leave us a comment. It really helps! Thank you for you contribution. :)