Kibana: Kibana Globalization

[rashidkpc]

    <div class="sidebar-list-header">
        <h5>{{ FIELDS_SELECTEDFIELDS | translate }}</h5>
    </div>

Is this expecting FIELDS_SELECTEDFIELDS to be attached to $scope of the controller? It would make more sense to implement this as an attribute directive similar to ng-bind

eg:

     <div class="sidebar-list-header">
         <h5 kbn-bind="FIELDS_SELECTEDFIELDS"></h5>
     </div>

Then kbn-bind would know how to handle FIELDS_SELECTEDFIELDS as a string, retrieve the appropriate replacement text, and fill it. In addition it would allow us to use one time binding and not suffer the performance consequences of two way binding with a filter.

[rashidkpc]

    {
        "FIELDS_SELECTEDFIELDS": "Selected Fields",
        "METRIC_COUNT": "Count"
    }

My initial thought is that a massive master list doesn't seem very elegant. It seems like this sort of thing would be better implemented the way we do .less files, scoped to the current app?

[rashidkpc]

There would also need to be a test framework around this to ensure that all of the constants that must be defined, are. Potential translation volunteers need a simple way of discovering what needs translation and a simple way of confirming that they've defined everything required

[rashidkpc]

    $translate('METRIC_COUNT').then( function(count) {
        return new MetricAggType({
    title: count
    }}

This seems to imply that any functions that currently contain strings would need to be updated to return a promise. I don't see the need for the async behavior here.

[rashidkpc]

• Use angular-translate

Given the need to translate both angular and non-angular components, why was a framework specific translation implementation chosen? And why this one specifically?

[srl295]

@rashidkpc thanks. Comments on a couple of items:

• translation scoping - yes, the translations could definitely be scoped to smaller units. I'll take a look at how the .less files are scoped.
• async behavior / angular-translate - this was one option. There may be a simpler option. The non-angular components might be able to use a different mechanism for loading content. We picked angular-translate to try and use something pre existing for the angular side.

[DTownSMR]

Rashid, maybe I should take these out of order.

• Why a framework & angular-translate? - The angularjs side of things looked to us like a
  good place to start a first iteration of efforts of adding g11n to Kibana. As Steven
  mentioned we picked angular-translate since we didn't want to reinvent the wheel and it
  is feature rich, good community support and seems to be gaining momentum within the
  angularjs community. We're not married to that solution of if there is a different
  preference within the Kibana community we can certainly look at that.
• angular-translate supports a "translate" directive in addition to it's filter mechanism.
  We were just showing an example of what translation can look like.
• angular-translate does not require a single resource bundle. It supports a number of
  options. Their partialLoader that will handle breaking up the resources at a component
  level.
• The idea of a test framework is interesting, especially for on-going efforts.

[shikhasriva]

Rashid, regarding test framework, we suggest having that as a separate issue. And definitely agree with the need and usage of it... what do you think ?

[shikhasriva]

Rashid and all, DO you all agree with the phased approach we suggested ...thanks

[w33ble]

Given the need to translate both angular and non-angular components, why was a framework specific translation implementation chosen?

While I agree that angular-translate looks pleasant to use, and seems to be well maintained, using something that's explicitly bound to angular for something as far-reaching as i18n concerns me.

As @rashidkpc points out, we have non-angular components in our code, and that's likely to become increasingly common in the future. I'm also concerned about server-side use as we push things into the node server.

Is there something more generic that fits the bill, and also has an existing angular wrapper?

[rashidkpc]

I haven't looked deeply into i18n, but I'm hesitant to tie to angular-translate for the full implementation. It could be used in places, but there's plenty of places it can't be used. Like joe said, I think it would be good to investigate something lower level that could be wrapped up for the angular bits.

On the topic of the phases. I don't see any problem with those stages, but i think more discussion is needed on the approach before beginning.

I would like discussion of the testing framework to happen here. We can always split it off if needed, but it has direct influence on the approach taken.

[srl295]

The translated files are just JSON and could certainly be consumed by the server side or the widgets, etc without use of the angular-specific framework. So this approach doesn't preclude using angular-translate where it makes sense, and something else (or no framework) elsewhere.

Re testing - yes, we can definitely discuss it here. We were wanting to split the test itself into a separate phase as above. (I added _1AA_) One issue would be that I would recommend having a missing translation be more of a warning than a failure from a CI perspective, otherwise a feature couldn't be added in the source language (English) without all translations being done, nor could a translation be partial.

[w33ble]

The translated files are just JSON and could certainly be consumed by the server side or the widgets, etc without use of the angular-specific framework. So this approach doesn't preclude using angular-translate where it makes sense, and something else (or no framework) elsewhere.

That's a fair point. It does seem like the format is pretty common, and the more or less the same regardless is which library/framework you use. Getting hung up on angular-translate specifically is probably a waste of time.

Given that, I think it's worth adding something in one of the earlier phases about ensuring that whatever translation format we go with works both with and without Angular somehow, most likely by specifically targeting a part of the application that isn't in Angular. Server-side code is probably an easy candidate.

[srl295]

@w33ble OK. Perhaps something like - "a simple JSON format will be specified in the documentation, and the proof of concept will use angular-translate. Non-angular portions (such as the server) would not use angular-translate, and the specific plugin technology used for translating on the client side is subject to discussion"

[w33ble]

I think Rashid and I just want to make sure the plan specifically takes into account using the i18n definitions outside of Angular. Adding the above sounds good to me.

[srl295]

@w33ble — thanks, that makes sense.
Edit I have updated the text above to address the feedback from yourself and @rashidkpc .

[srl295]

@rashidkpc @w33ble — any comments on the updates?

[rashidkpc]

I'm happier with the new proposition.

    var uiStrings = …; // loading TBD
    return new MetricAggType({
      title: uiStrings.METRIC_COUNT,

Either webpack/import or though a Private module, which would make it easy to inject into other parts of the application.

[srl295]

great. We will investigate using webpack.

It seems like the proof of concept should be able to proceed for phase 1A if there are no issues on that part.

[Bargs]

Can we create a new ticket specifically for Phase 1A and turn this ticket into more of a meta ticket? There's a lot of good discussion and context here so I wouldn't want it to get closed when #7247 gets merged.

[Bargs]

Someone was asking in chat today about currency field formatters, which brought my attention to https://github.com/elastic/kibana/issues/4392

It's probably a long way down the road if we tackle something like that, but food for thought perhaps.

[srl295]

@Bargs @epixa given the updated discussion, I'm going to propose the following steps— not assigning 'Phase' numbers for now, just trying to capture a way forward.

First steps will be to have the i18n plugin, and to make use of it from CLI - not even from the front end to start.

Low level i18n plugin

• Manages "what languages are available" (maximal set), responsible for loading translated content at the granularity of a plugin. Example: "plugin: Kibana, language: French" or "plugin: Status Page, language: German"
• API:
  
  
  
  • Register Translation Bundle
  
  
  • ex: kibana.Plugin({ i18n_bundle: 'some/path' }) similar to table_vis -- "look in some/path/<pluginname…>/<language>/… for a bunch of translated content to consume"
  
  
  • Get the maximal list of currently-installed languages
  
  
  • Perform content-language negotiation against maximal list
  
  
  • Fetch a specific plugin's translated content
  
  

Translation Plugin(s)

• A plugin can register one language for one plugin, multiple languages for one plugin, one language for multiple plugins, or multiple languages for multiple plugins
• core plugins (kibana, status_page) could come with their own English translations bundled in
• a plugin could add, say "Swedish support" or multiple languages at once
• Translation plugins have a minimal of actual code in them, again following the model of https://github.com/elastic/kibana/blob/HEAD/src/plugins/table_vis/index.js

Translation Bundle-Serving Plugin

• This plugin is responsible for pulling together all translated content by language and serving it in one HTTP request, such as webpack does currently.
• It could use webpack APIs if it makes sense, but it's really just a bunch of JSON

Angular Translation Plugin

• This is another layer which implements what we put into #7247 but using the i18n plugin or the Translation Bundle-Serving Plugin to access translated content on the client side. It makes the data available to angular-translate

[Bargs]

@srl295 at first glance this looks really great! I'll have to give a few of the points some thought, so I'll try to get you some more detailed feedback next week.

One note - I chatted with @hickeyma on IRC yesterday about using the translations from the CLI. I think that's going to be a tricky place to start because the plugins don't get loaded up until the Kibana server actually starts booting up. We talked about starting with some simple API tests instead. I'll keep trying to think of other good places to start testing the i18n plugin without involving the frontend.

[srl295]

@Bargs Thanks. Yeah, lots of handwavy here, just trying to capture the general direction. I didn't actually look at the wiki yet, @hickeyma did and was going to ground this firmly in reality.

on CLI, perhaps there's a way it could be used for error messages- so at least, once booted, they are localized, even if not before hand. hm.

Thanks, let's keep discussing.

[hickeyma]

@srl295 Looks good! :-) Like @Bargs I will give the points some more feedback as I go along with implementing. BTW, tried loading translations in CLI from plugin, Having issues with timings as seems to be returning request and not the response back from the plugin. After chatting with @Bargs, using mocha tests instead for the moment.

[shikhasriva]

@srl295 @Bargs @hickeyma, I would propose to start from the Ui side as that will be more prevalent use case. We can take the cli as a later phase. Enable for Ui first and then CLI.
Also, lets revisit the rendering (agular, react etc) plugins. I am thinking we need a protocol setup that can be called by react or angular or anything else.

[shikhasriva]

@Bargs, One point that we need to discuss is correlation between views and translation bundle. So we have an algorithm to load the right bundle.

[hickeyma]

@shikhasriva: This is a good point to raise! As we discussed today, I am currently PoCing with known localized string file but the plugin would need to know which file(s) to load per client/view.

[srl295]

@shikhasriva that makes sense from a usage point of view. @Bargs I think “simple API tests” may suffice for a phased bringup. Our prior “Phase 1A” had the translation mechanism _in_ the UI area, and moving it to a plugin moves it out into non-UI territory.

[hickeyma]

The following questions have come to mind while thinking how best to implement the "Low level i18n plugin":

1. How will the plugin retrieve the currently-installed languages? Does this need to be a configuration setting?
2. How will the i18n plugin decide which translation files to load for a plugin? Does it traverse all translation files (i.e. any i18n) directories in a plugin directory? What about plugins where some of its languages are in different plugins?
3. Should we use a custom HTTP header to inform the i18n plugin which plugin resources it needs to load?
4. Does the i18n plugin load all translations (all translation file content) for a plugin and return as a JSON object in the payload?
5. Do we need to define the JSON structure for all translation files so that there is uniqueness between plugins and in plugins? For example:
   { <plugin> { <view> { <operation> { <ID>: <string> } } } }
   i18n plugin could define the 'plugin' as passed in the request and 'view' as retrieved from translation files traversal.
6. Should "accept-language" header be used instead of "content-language" header? "accept-language" header would provide all languages loaded in an user's browser.
7. Could we use the default language from "accept-language" header as the fallback language? Do we need a server configuration as well?

[Bargs]

These are just some thoughts off the top of my head, so take them as guidance rather than hard requirements.

How will the plugin retrieve the currently-installed languages? Does this need to be a configuration setting?

I think the i18n plugin will keep track of which languages are available as they're registered by other plugins, similar to how vis_types are added by plugins (see the table_vis plugin as an example). That's just my initial thought though, I'm open to other ideas.

How will the i18n plugin decide which translation files to load for a plugin? Does it traverse all translation files (i.e. any i18n) directories in a plugin directory? What about plugins where some of its languages are in different plugins?

That might be one way to handle it, but it would require us to force a convention on plugin authors (not necessarily a bad thing). We could also leave it up to the individual translation plugin to register the complete bundled translations with the i18n plugin. But this would probably need to be done in such a way that the translation bundles aren't all being loaded into memory on the Kibana server. We could also lean on webpack to bundle up all the translation files, if it makes sense.

Should we use a custom HTTP header to inform the i18n plugin which plugin resources it needs to load?

Not sure what you mean here.

Does the i18n plugin load all translations (all translation file content) for a plugin and return as a JSON object in the payload?

I think the biggest determinant here will be memory usage on the Kibana server. I like the simplicity of serving the translations as JSON payloads from a REST api, but it's probably too much info to hold in memory, and if we're just streaming a flat file there might not be much advantage in wrapping it in an API vs simply serving that file directly. My intuition might also be totally off on how much memory these will consume though.

Do we need to define the JSON structure for all translation files so that there is uniqueness between plugins and in plugins?

They definitely need to be namespaced in some way. Plugin id might be the natural candidate. But how do we distinguish between which plugin provided the original version of a string, and which are just providing translations? For instance, say the table_vis plugin defines a string resource keyed by table_vis.foo.bar. Another plugin called table_vis_es might provide a Spanish translation for the string, but that doesn't mean there should be a table_vis_es.foo.bar key.

Should "accept-language" header be used instead of "content-language" header? "accept-language" header would provide all languages loaded in an user's browser.

I believe these headers are for two different purposes, accept-language is a request header whereas content-language is set by the server in its response. So for the purpose of the client telling us which language they want, we should definitely use accept-language.

Could we use the default language from "accept-language" header as the fallback language? Do we need a server configuration as well?

I imagine we need some sort of fallback if the client provides no accept-language header at all.

[Bargs]

I forgot, another idea @epixa mentioned is that we could use the default language string itself as the key for each string resource, rather than coming up with a separate unique namespaced key. This might also side step the question of matching resource bundles to views, as the strings would be view agnostic.

[epixa]

Plugins should have to explicitly register their translations rather than Kibana traversing the file system to find them. We could just lean on the hapi plugin system for this:

// synchronously
server.plugins.i18n.registerTranslation('en', {
  somekey: 'Some Value'
});

// asynchronously
import { promisify } from 'bluebird';
import { readFile } from 'fs';
const readFileAsync = promisify(readFile);
const loadTranslation = readFileAsync('mytranslations.json').then(JSON.parse);
server.plugins.i18n.registerTranslationAsync('en', loadTranslation);

The i18n core plugin in Kibana could then write registered translations to disk on startup so it's not all sitting in memory.

The "available languages" would be inferred from all of the plugin register calls at startup.

Also, we shouldn't rely on webpack for any of this. We want to get webpack out of our release builds entirely.

[hickeyma]

@Bargs, @epixa: Thanks for the feedback.

@epixa: You are suggesting that all plugins register their translations with the i18n core plugin? This can be 1..m files and for multiple languages supported. If different languages of a plugin are in different plugins, then maybe they need to register with the parent plugin ID?

This registration could be one off registration on first start of the server or if new language added or if new plugin added? There could also be the option to re-register.

The i18n core plugin can then store internally each plugin language translations in file as you mentioned. It can be structured to best suit how to load data as needed. Do we consider breaking translation from one bundle file into multiple bundles?

As you said, it makes sense to get available languages from registration.

It might be prudent to put a convention on plugins that all translation IDs in a plugin are unique. This removes unnecessary multi-structured JSON and less complexity from client.

When the plugin need its translations, how best to load it from the i18n core plugin? Does it do a REST call at start of plugin, to load a language translation bundle which covers all translations for the plugin into memory? Or is it best to serve the file directly?

[Bargs]

I can't speak for @epixa but here are my thoughts:

This registration could be one off registration on first start of the server or if new language added or if new plugin added? There could also be the option to re-register.

My initial thought was that the translations would just get dynamically registered on every server start up. But that strategy would make build time verifications difficult/impossible. Maybe registration (and the subsequent build of the translation file by the i18n plugin) should happen at plugin installation time? This is probably something we'll have to experiment with a bit.

Do we consider breaking translation from one bundle file into multiple bundles?

I was thinking we'd have one bundle per language, maybe per app? It's possible the i18n plugin's internal representation of the data, and the API it exposes to the frontend will be different. I kind of liked @srl295's idea of a separate "Bundle Serving Plugin".

It might be prudent to put a convention on plugins that all translation IDs in a plugin are unique. This removes unnecessary multi-structured JSON and less complexity from client.

The more I think about it the more I like the idea of using the text strings as the resources keys themselves. It removes the need for any namespacing and even opens up the possibility of plugins sharing translations of common strings in a pretty frictionless manner. For instance I don't see any reason why two plugins should both need to translate some common words like "Submit" or "Click for more info". There's some precedence for this approach in other libs: https://github.com/jeresig/i18n-node-2. Can you think of any downsides?

When the plugin need its translations, how best to load it from the i18n core plugin? Does it do a REST call at start of plugin, to load a language translation bundle which covers all translations for the plugin into memory? Or is it best to serve the file directly?

We can probably figure that out as we go. As long as the content is coming from an endpoint that Hapi exposes, we can always change how that content gets stored/loaded. For instance, this is how we serve the static js and css bundles: https://github.com/elastic/kibana/blob/f379074ebd9c73fa64fc999f6cb82ca548e54350/src/server/http/index.js#L17. As you can see they currently get loaded up from the filesystem by Hapi, but that's an implementation detail of the server.

The translations probably shouldn't be loaded per plugin though. We don't want the user to have to wait for translation bundles to download every time they click to a new page in the app. Unless we can show that downloading all of the translations for a given language in one big bundle at page load time is too slow, we're probably optimizing prematurely.

[srl295]

Be back from two days off tomorrow, but lots of comments. Briefly
• having plugins have too much code and not data means harder for toolchain to work with such plugins.

• using English text as the key is not a good idea - example, "back" could mean to return or could mean the part of your body. Such reuse is actually not a good idea. Again tooling that can see the English side can reuse content using more context. You also lose out on history and context. If the English changes from "hello, world" to "Hello, world!" the translation may need minimal/ or even no change. Only having the English as the key throws away any information and starts from scratch.

[Bargs]

Good points on english text keys @srl295, I knew it sounded too good to be true. @epixa you mentioned you've seen this approach before, was there any way of mitigating the issues @srl295 brought up?

[epixa]

Nope, I never thought of that. We just lucked out.

[srl295]

Some clarifications/responses to the discussion above.

@epixa I realized something, when you said:

rather than… traversing the file system

I didn't expect that the i18n plugin would look in all possible places for translations (such as the …/i18n/en.json files in our old proposal). What I had in mind was the following:

• a plugin (whether it has other code or not) can register translations by adding something like a UiExport Type - perhaps there would be part of the yo generator that says "Does this plugin have translations?" … but the plugin would provide an explicit path within the plugin as to where translations were located. Maybe just uiExports: { translations: './translations' }
• Then, the i18n plugin would basically just collect all of the translations: paths together whenever this registration happens. I wouldn't expect the translations would all be "read into memory" at that point necessarily. Mostly it would just need to know which languages are available.
• The i18n plugin would (at its option) lazily scan for the set of languages and translations available.
• note that the "Bundle Serving Plugin" may need to collect the actual translations into some sort of optimized bundle (equivalent to the "optimize" step. But again, lazy and optional.)
• For example, a plugin might have the following files under its ./translations/ directory (to use the above example). Consider three different use cases:

1) a plugin which contains just the English content for (perhaps its own) plugin:

<plugin>/translations/status_page/en.json

2) a plugin which contains English contents for multiple plugins:

<plugin>/translations/status_page/en.json
<plugin>/translations/tableVis/en.json

3) a plugin which contains German contents for multiple plugins:

<plugin>/translations/status_page/de.json
<plugin>/translations/tableVis/de.json

• yes: there could be an API on the i18n plugin for adding translations by callback or otherwise programmatically - this could enable interesting things such as Pseudolocalization or plugging into translation management things like what i work on. But I don't think programmatic should be the preferred mechanism for a couple reasons:
  
  
  
  • it adds complexity for everyone who wants to help translate - let's keep the boiler plate simple
  
  
  • it adds complexity for tooling that wants to work with the translations. Having the source be flat json files as listed above, means they can be used by existing tooling. You can copy en.json to de.json to get started on the German format.
  
  
• Contents of the .json files: I would opt for a flat keyspace, but the keys could have delimiters by convention for each plugin. So for example, en.json above could have;

{
   "create.CONFIGURE_INDEX_PATTERN": "Configure an index pattern", 
   "create.MUST_CONFIGURE_INDEX_PATTERN":"In order to use Kibana you must configure at least one index pattern. Index patterns are used to identify the Elasticsearch index to run search and analytics against. They are also used to configure fields.",
   …
   "edit.DESCRIPTION": "This page lists every field in the <strong>{{indexPatternId}}</strong> index and the field's associated core type as recorded by Elasticsearch. While this list allows you to view the core type of each field, changing field types must be done using Elasticsearch's",
   "edit.TIME_BASED_INDEX_ALERT":"This index uses a <strong>Time-based index pattern</strong> which repeats"
}

Limiting the contents to a simple map makes it easier for tools to work with. In this example the UI uses the view name (create or edit) followed by a dot and then a local suffix.

• I discussed not using the English translation as the Key in some detail above. One option might be to use auto generated UUIDs (or something shorter), or having some sort of tooling which would put sequential numbers in place as messages are needed. Even better would be to put some descriptive name as in the example above.
• in terms of the consumer side i18n plugin API, the simplest example is just i18n.getTranslation( 'status_page', 'de') which would return the entire map for the status_page plugin in that language. A _convenience_ API could be added to fetch a specific string, such as i18n.getTranslation('status_page', 'de', 'edit.DESCRIPTION' )
• In terms of loading, right now kibana.bundle.js loads ALL of the views together. What I would imagine is that the UI bundle loader would calculate a table_vis.en.json or a status_page.de.json containing all of the aggregated translations needed for the plugin level, and send that over on a plugin-by-plugin basis.
• I'd imagine the UI bundle loader having a REST endpoint such as:
  
  
  
  • GET / i18n/status_page ==> returns the English (or German or whatever) bundle for the status_page plugin - content negotiated with the browser against available languages
  
  
  • GET /i18n/status_page?de ==> same, but with a specified language (if the UI has a specific request)
  
  

(This from discussion hashing things out with @hickeyma … any errors or omissions are mine of course…)

[hickeyma]

@Bargs @epixa following on from our discussions and investigations, we are going to propose the following for Kibana Globalization. Any errors or omissions are mine!

Archiving the proposal as now added to the main document above https://github.com/elastic/kibana/issues/6515:

[Bargs]

@hickeyma This is a great summary. I think phase 1 and 2 probably need to happen in tandem, in fact I should probably get phase 2 merged prior to merging phase 1 so that we'll have a working pluggable translation system once the initial i18n PR gets merged (The jade template translation should be able to come from a translation plugin, not just the baked in english version).

One language for one Kibana view/plugin

I think this is outdated, we're no longer dividing translations across view/plugin boundaries right?

Sub-directories containing translation file(s)

Let's just support a single directory of translation files, one per language for now.

Otherwise I think this is a great summary of our current thinking. @srl295 could you update the main ticket description?

[Bargs]

@hickeyma @srl295 I chatted with @epixa today. I think we just have a few more items to take care of before we open this up to a wider audience. Hopefully the checklist below provides a clear path.

Open questions and blocker tasks:

• [x] @epixa mentioned he would prefer the i18n plugin's registerTranslation method take individual file paths, rather than a directory name. We debated about this a bit - I prefer the simplicity of the directory approach whereas he prefers the explicitness of a single file approach. I was wondering what you guys think?
• [x] If we add translations to the Jade template, we really need the tool for verifying the translations exist. This could be a simple grunt task we run during the build step. This needs to be solved for phase 1 since we're adding translations for the loading screen in phase 1.
• [ ] How do core plugins register their translations since they're never "installed"? How does this work in dev mode?
• [x] Solidify the plan for verifying angular translations. We don't have to actually write this utility at the moment, but we should have a pretty clear idea of how we'd like to implement it. Again, at the very least this should be something we can run at build time to ensure we don't cut a Kibana release with broken translation keys.
• [x] Phase 3 - "The bundle produced will be of the following format: i18n_.bundle.js" - since we have a different strategy for translating the loading page content now, I think these can be simple json payloads instead of javascript files?

Documentation clean up:

• [x] Add jade template verification with description of proposed approach to phase 1.
• [x] Update info about translation plugin directory structure once we make a decision there.
• [x] Not sure what the "Translation Loader Plugin" is?
• [x] "REST API" and translation bundle serving plugin are one in the same in my mind at the moment. I don't think we need separate sections for them. I think to begin with we'll have one endpoint that serves translations per language, whether we call them bundles or not isn't a hugely important detail.
• [x] Update the description of this issue with the cleaned up docs. The description should be very clear and up to date before we start asking for feedback from a wider audience.
• [x] Need to fill out the "Translation identifiers added to Kibana UI" section in phase4. How will we translate the UI? Are we going to use the angular translate module?
• [x] Description of the translation file json structure should go under phase 1 "register translations" section (currently under phase two section for boilerplate)

Other items, thoughts, non-blockers:

• [ ] I need to look into doing some refactoring to support the plugin install time hook. Our current strategy of booting up an actual kibana server instance during plugin install is less than ideal, if we can implement the install hook outside of that context it would be ideal. This shouldn't block gathering feedback from a wider audience, but it would be a blocker for phase 1.

[hickeyma]

@Bargs, @epixa Thanks for the great feedback. I will initially respond to the points here and tackle them subsequently:

Open questions and blocker tasks:

@epixa mentioned he would prefer the i18n plugin's registerTranslation method take individual file paths, rather than a directory name

What is the use case for this that is not covered by directory name?

If we add translations to the Jade template, we really need the tool for verifying the translations exist.

OK. As mentioned previously , it could possibly be some pattern which would help identify that this is a string identifier. This might be something as simple as defining that the variable used for getting the translations is a certain name (for example, 'i18nTranslation') and then could search for all instances in code files and extract the identifier from it. Then you could check the translations files.
It can be JS code called by a grunt task.

Solidify the plan for verifying angular translations.

I will investigate this further and come up with some plan/PoC.

Documentation clean up:

Update info about translation plugin directory structure once we make a decision there.

Sure, will do.

Phase 1 and 2 really need to go together, so we should update the summary to reflect that.

Sure, will flatten into one phase.

Not sure what the "Translation Loader Plugin" is?

On second thoughts, this plugin is probably superfluous. I will remove it during update.

"REST API" and translation bundle serving plugin are one in the same in my mind at the moment. I don't think we need separate sections for them.

OK. They are in separate sections because addition of "REST API" is to the i18n plugin which the translation bundle serving plugin uses.

Update the description of this issue with the cleaned up docs.

Sure, will do.

Other items, thoughts, non-blockers:

I need to look into doing some refactoring to support the plugin install time hook. This shouldn't block gathering feedback from a wider audience, but it would be a blocker for phase 1.

OK. My only concern here is that doing this may have the potential to block phase 1 for a substantial period of time? Or is it quite doable to get it merged?

[Bargs]

What is the use case for this that is not covered by directory name?

@epixa's argument isn't that it will support additional use cases, but that it will be less brittle since it is more explicit. My feeling was that it would complicate things for the translator without much benefit. However, just while typing this I realized we could do the single file registrations, and still maintain a good translator experience if we just move the default file gathering behavior into the translation plugin boilerplate. What do you think?

it could possibly be some pattern which would help identify that this is a string identifier. This might be something as simple as defining that the variable used for getting the translations is a certain name

It would be nice if we could enforce the pattern somehow, rather than relying on all devs using the same variable name. Maybe we could expose a translate('<key>') function in the Jade template instead of a variable. That makes the strategy for accessing translations pretty explicit.

They are in separate sections because addition of "REST API" is to the i18n plugin which the translation bundle serving plugin uses.

I'm not sure what you mean by this. What does the bundle serving plugin do?

OK. My only concern here is that doing this may have the potential to block phase 1 for a substantial period of time? Or is it quite doable to get it merged?

It's possible, I'll need to spend some more time with the existing code to know for sure. If it looks like it will be a huge effort, maybe we can consider a phase 1 without the install hook. Here's another question that just occurred to me: how do core plugins register their translations, since they don't get "installed"? I'll add this to my list above.

[hickeyma]

@Bargs I have updated the description in https://github.com/elastic/kibana/issues/6515#issuecomment-231400097 accordingly following our discussions.

@epixa's argument isn't that it will support additional use cases, but that it will be less brittle since it is more explicit.

OK. _registerTranslations_ updated to take absolute path to file instead of directory argument.

It would be nice if we could enforce the pattern somehow, rather than relying on all devs using the same variable name. Maybe we could expose a translate(<key>) function in the Jade template instead of a variable. That makes the strategy for accessing translations pretty explicit.

Sure, agreed.

What does the bundle serving plugin do?

Again, in retrospect this is now unnecessary. Removed.

How do core plugins register their translations, since they don't get "installed"?

Added "Tool for Registering Translations" section above to phase 1. By adding this section, it now seems to decouple phase 1 and phase 2 dependencies as you will now have workable translations for the welcome message.

[srl295]

@Bargs I've updated the heading of this issue w/ @hickeyma 's latest… also added a History section with links to gists of prior versions of the issue heading.

[shikhasriva]

Thanks, Looks good. @Bargs, lets us know if you think of any open Qs or comments

[Bargs]

@hickeyma I think the "Tool for registering translations" section might give the wrong impression, it sounds like this is an external tool we'll create. I think the more important questions to answer are:

• How/when do core_plugin translations get registered?
• How does this tie into the development workflow? At the moment we don't have any sort of build step in development, you just npm start and the optimizer does all the work behind the scenes. Ideally we could rebuild the translation bundles on the fly in dev mode.

[hickeyma]

@Bargs: Sure, probably need an integrated means in the Kibana infrastructure to register the core plugin translations. I will investigate this further and in the meantime I will add a reference in the design doc and flesh it out once we have an approach unearthed.

[hickeyma]

I have updated the design doc kibana-proposal-update

@Bargs This should hopefully tick off some more items in your checklist! :-)

@Bargs, @srl295 could you update the main ticket description?

[Bargs]

Looking pretty good. Issue description updated, as well as my checklist. Just a couple of open items left.

[hickeyma]

@Bargs thanks!

[hickeyma]

@Bargs WRT "registering the core plugin translations" in delivery section of phase 1 of #6515, this could be updated as follows:

• Welcome message and start-up error message are registered with the i18n plugin. This will require separate mechanisms for the build and development process:
  
  
  
  • Build process:
  
  
  • Grunt task which calls the registerTranslations API and make them available to the build package
  
  
  • Dev process:
  
  
  • Registration of translations during dev start-up of the Kibana server (npm start). Hook is added to the 'optimize' module to call the registerTranslations API.
  
  

Additionally, maybe the "Blockers" section can now be cleaned up?

[hickeyma]

@Bargs WRT Jade template verification in delivery section of phase 1 of #6515, this could be updated as follows:

• Jade template verification of translation strings:
  
  
  
  • Enforce a pattern to be used. For example a translate(<key>) function in the Jade template. A tool can then be used to find such pattern and extract the keys to file
  
  
  • The keys in the key file(s) would then be checked against the language translation files registered
  
  

[Bargs]

@hickeyma lgtm - description updated.

[Bargs]

@hickeyma so I was thinking more about the core plugin registration problem and I'm starting to question whether we even need to build this intermediate json bundle per language. Due to the simple, flat translation file structure we've chosen we could easily build a single unified JSON object from multiple file streams on the fly. The i18n plugin would only need to keep track of the translation file paths per language. This would simplify a number of things:

• No need for a plugin install time hook. Translation file paths would simply be registered at server start up time.
• Core plugins and third party plugins would be treated equally, registering themselves inside the existing plugin init hook.
• If (if!) we run into performance issues, we can always just add a layer of caching.
• We get live reload in dev mode for free, because there is no build step.
• This is actually more in line with the direction we're trying to head with css/js resources as well. Ideally we'd like to remove "compilation" from production environments entirely.

What do you think?

[hickeyma]

@Bargs good feedback, thanks.

Are you saying that we now just register the translation files names per plugin when the plugin is being initialized?
Does this mean the plugins still have to register with the i18n plugin but not during the install phase of the plugin?
That would definitely simplify the infrastructure hooks difficulty. It probably doesn't matter then if you register just the filename or the actual file?

[srl295]

(I'm coming off of NodeSummit, just getting to some more detailed review.)
@Bargs this makes some sense.

[Bargs]

@hickeyma yeah, basically the i18n plugin just needs a reference to each translation file and knowledge of which language it contains. I think each plugin would still be responsible for explicitly registering its translation files with the i18n plugin during init time, just like how plugins currently register their routes with Hapi at init time.

[shikhasriva]

@Matt and @hickeyma , this will simplify the approach for sure. And removes the install dependency.
I do like this approach

[LeeDr]

I'm not clear if the initial phases are providing a translation at the server level and all users see only that one language, and then in Phase 3 it uses the browser HTTP header “accept-language” to translate per users browser language? Can we add notes to the phase descriptions to make it clear what the functionality is?

We might also want to start a discussion about which phases might be release candidates. It can happen later, but I wouldn't want a bunch of contributors to have an expectation about getting a release with some phase functionality and Elastic deciding not to release until some other phase. For example, translations for the whole server and not on a per-user basis.

[srl295]

a translation at the server level and all users see only that one language

I don't think this "one language" step is part of the plan. The current PR (phase 1) already translates, using accept-language, on a per-browser basis— it's just that only the "kibana is starting up" and one other error message are translated. Phase 4 has the "Translation Identifiers Added to Kibana UI" step which actually allows the rest of the UI to be translated. Hope this helps!

[hickeyma]

@LeeDr thanks for the feedback.

a translation at the server level and all users see only that one language

To elaborate on @srl295 description above, the idea here is to deliver in a phased approach. The phases 1-3 is to put the i18n infrastructure in place and show it working end-to-end in Kibana. The key part is the i18n plugin which provides means to supply translations in a standard format that it not dependent on a localization framework. The proof of this is to translate the welcome string in Kibana.
Phase 4 is about translating the rest of the UI using the underlying i18n plugin to get the strings.

Can we add notes to the phase descriptions to make it clear what the functionality is?

Sure, can do. That is a good idea.

We might also want to start a discussion about which phases might be release candidates.

This is on the TODO list with @Bargs and @epixa.

[hickeyma]

@Bargs, @LeeDr I have updated the design doc kibana-proposal-update as per your feedback.
Let me know if I have covered all the angles.

@Bargs, @srl295 could you update the main ticket description?

[srl295]

@hickeyma done!

[hickeyma]

Thanks @srl295

[hickeyma]

@Bargs, @srl295, could you also add this line to i18n Plugin (update) in Phase 2 of #6515:

• The algorithm which decides the translation language to use for retrieving the translations works as follows:
  
  
  
  • First, do a direct comparison with the HTTP header “accept-language” priority list against the registered translation languages.
  
  
  • Next, do a partial comparison whereby you try and get a language code comparison. For example, "fr" or "fr-FR" is used for all regions of French if comparison applies.
  
  
  • Finally, the default language as configured per Kibana instance is used. Default configuration is "en". This is also used if no HTTP header “accept-language” is set.
  
  

[hickeyma]

@Bargs, @epixa, @shikhasriva, @spalger, @ycombinator, @cjcenizal, @thomasneirynck, @tylersmalley, @bevacqua, @simianhacker, @BigFunger, @jbudz, @ppisljar Thank you all for attending the red-team review yesterday and for your feedback. Let me know if I forgotten anyone.

Here are notes from the review (thanks to @Bargs for taking notes!):

• [ ] Handling requested language for translations:
  
  
  
  • [ ] Server side set language (ignore the client's requested language)
  
  
  • [ ] Ability to define a set of supported languages on server instead of relying on list of languages auto generated from available translations (when some might be partial)
  
  
  • [x] In the "partial match" stage of the accepted language matching algorithm (Phase 2 'Decide Language for Translation Algorithm'). What would happen if a user requested 'fr-FR', but the system only supports other regional dialects. Should we have capability to set a default region for all language codes (e.g. fr, de etc.) requests?
  
  
  • [ ] Provide mechanism in UI for user to manually switch their language from list of supported languages
  
  
  • [x] What do we do when the requested language exists, but it doesn't cover all translations keys (for instance, there are French translations for a plugin but not core Kibana)? Do we fill in the missing keys with the default language?
  
  
• [ ] What about bi-directional language support? This affects language, charts and UI content support.
• [ ] UI views should be able to handle different languages (This means when switching from one language to another (for example, English to German) that the look and feel is maintained) :
  
  
  
  • [ ] Different languages can have variable content lengths. This means having adequate spacing to handle the strings in each language.
  
  
  • [ ] Need to consider different fonts. Some web fonts won't support all possible languages.
  
  
• [x] Use YAML instead of JSON for the translations? Easier to write strings than in JSON, less escape chars, multi-line, etc.
• [ ] Should we embed the translations in the initial HTML payload? Reduces complexity and removes a round trip. May eliminate the need for a HTTP API? Probably worth a PoC to see if it is performs and does not create a big payload?
• [ ] Need to consider how to translate and verify xplugins (React) (separate meeting w/ xplugin devs) Shouldn't treat x-plugins as a second class citizen after core Kibana.

Feel free to comment and give feedback especially if anything has been forgotten!

[hickeyma]

I have updated the design doc kibana-proposal-update with the feedback leaving unresolved items as open or blockers.

Maybe for the moment, we use it as an intermediate document and only update the main document when we are ready?

[shikhasriva]

Thats works fine. @hickeyma , Thanks for aggregating the comments

[srl295]

responding to https://github.com/elastic/kibana/issues/6515#issuecomment-237287195 very late

yaml instead of JSON…

I'd recommend JSON as it is more commonly used in translation systems.

[Bargs]

@ycombinator you had some compelling reasons about using YAML, could you share them here?

[ycombinator]

Sure, I don't feel strongly about either choice but, in a past i18n project YAML was chosen over JSON for a couple of reasons:

• Its a tiny bit more human-readable than JSON. This might lower the barrier to entry a bit for folks who'd like to supply translations but may not be familiar with JSON.
• YAML supports a few heredoc formats which make it more natural for expressing multi-line translations, as opposed to JSON where the newlines would have to be escaped. That being said, I imagine most translations for Kibana will be quite short so this is probably not a huge factor.

Those are the two reasons I recall at the moment. If I think of others I'll add them to the comment. But again, I don't personally feel strongly about one being strictly better than the other.

Also, we could go with JSON initially and start accepting YAML at a later point in time if we start to run into the above reasons in practice.

[hickeyma]

@ycombinator Thanks for the feedback.
For now, we will use JSON. We can always provide a YAML to JSON converter if that is the preference for localization engineers.

[hickeyma]

What do we do when the requested language exists, but it doesn't cover all translations keys (for instance, there are French translations for a plugin but not core Kibana)? Do we fill in the missing keys with the default language?

Refer to i18n plugin pr comment for more details.

[hickeyma]

Decide the phases to be delivered in the Elastic release candidates

Discussed with @Bargs. Not necessary for the feature. More for Elastic internal QA.

[hickeyma]

@srl295, @Bargs I have updated the design doc kibana-proposal-update. Could you update the design doc when you get a chance?

[Bargs]

Updated

[hickeyma]

@Bargs thanks

[shikhasriva]

thanks ... great collaboration and awesome progress.

[hickeyma]

@srl295, @Bargs, @spalger I have updated the design doc kibana-proposal-update following some changes in Phase 1. Could you update the design doc when you get a chance pls.?

[hickeyma]

@Bargs, @spalger I wonder should we re-examine the design again following the merge of "Phase 1"? I started a Phase 2 PR (https://github.com/elastic/kibana/pull/8766) and @posijon is working on it. Maybe we should see if current design still applies for the phase?

[spalger]

It might be worth taking a look. There are things, like pulling the translations from a rest api, that I would push back on.

[Bargs]

Yeah we should definitely take a look and make sure we're on the same page. The REST API bit, for instance, I don't think has been updated since our "red team" meeting. At that time I believe we said we'd try embedding the translations in the initial page load first.

Also, issue description updated.

[hickeyma]

Great stuff @Bargs and @spalger. Lets try and sync-up on IRC and work out how to progress for phase 2 and 3.

[hickeyma]

@Bargs, @epixa, @shikhasriva, @spalger, @posijon Thank you all for attending the sync-up yesterday and for your feedback.

Here are notes from the meeting (let me know if I have misrepresented anything):

• It is good that we landed Phase 1. We want to now drive on with Phase 2 and 3.
• Phase 2 is to get the angular template right for localization engineers to use.
• Phase 3 is to add translation bundles for various languages.
• Updates to Phase 2:
  
  
  
  • In "Deliverable" section, "making the translations to client side" needs to be updated. The proposal is to now implement it by embedding the translations in the initial HTML payload. If discovered that this implementation is non performant then we will consider other approaches like: using current bundle mechanism or then using REST APIs to return the translations.
  
  
  • Add to "Deliverable" section, Elastic needs to handle how to translate and verify xplugins with React
  
  
• "Tool for verification of translations" needs to be moved from Phase 3 to Phase 2.
• "Implement a tool to generate the translation plugin" is to be moved from Phase 3 to Phase 4.

[hickeyma]

@Bargs, @spalger Updated the design doc after yesterday's meeting as kibana-proposal-update. Could you update the design doc when you get a chance pls.?

[Bargs]

Updated

[yuwtennis]

+1

[hickeyma]

@spalger Updated the design doc kibana-proposal-update. Could you update the design doc when you get a chance pls.?

[spalger]

Could you update the design doc when you get a chance pls.?

Sorry, which design doc are you referring to?

[srl295]

@spalger he's asking for this issue ^^ ’s text to be updated from the above gist. As issue creator I was able to do it - so Done

[hickeyma]

Thanks @srl295

[hickeyma]

@spalger unfortunately no rights on the doc to update!

[srl295]

FYI: 🎦 posted blog+movie showing off g11n progress so far https://srl295.github.io/2017/03/17/translating-kibana/

[tangyong]

Great work so far! @srl295

[elkusr]

Thanks for all the great work! When can we expect phase 3 to be available?

[epixa]

@elkusr There's no target yet for phase 3, and there's still some work in phase 2 to complete first now that Kibana is using react so much.

[zzvara]

How are these efforts progressing now?

Is that correct, that the tasks presented in (https://srl295.github.io/2017/03/17/translating-kibana/) can be used to add any language to a Kibana instance (for example 5.6)? In other words, adding another language to Kibana is feasible as of this writing, it just takes a "little" effort, and that Phase 3 is about to ease this effort?

[epixa]

@zzvara It isn't possible to translate Kibana right now. Phases 2 and 3 need to be completed before you can translate templates across Kibana.

[sunneyoy]

Looking forward to the Chinese version.

[16beer]

Have you already supported Chinese?
config/kibana.yml
i18n.defaultLocale: "zh_CN"
This configuration does not work, so I use this tool now.
Kibana_Hanization

[kimjoar]

@sunmaolin Sorry, we currently only support English.

[epixa]

I'm going to close this issue out in favor of #17201. While that is a much newer issue than this one, it contains a lot of information about our current localization effort, and it's best if we don't fragment the discussion across multiple issues.

Many of the details in that other ticket are based on all the things we learned from this effort in particular, so thank you to the great folks at IBM and everyone else that has been involved in getting us to this point!