Botbuilder-dotnet: [DCR] Make Language Policy configurable in Adaptive

[boydc2014]

Scenario

In composer (as well as in VA) scenario, to support a multi-language experience, we create lg file for dialog per locale. For example, we will have main.en-us.lg and main.fr-fr.lg if en-us and fr-fr is the locale to support.

And assume the user want en-us to be default locale that bot serves. So

Expected VS actual behavior

The expected behavior for this design, is if the incoming locale is fr-fr, then use fr-fr.lg, otherwise use en-us.lg, but actually when incoming locale is not en-us or fr-fr, or the incoming locale is empty, the bot will failed to locate a lg file.

So, given main.en-us.lg and main.fr-fr.lg

| incoming locale | expected locale | actual locale | comments |
| :--- | :--- | :---| :---|
| en-us | en-us | en-us| 👍 |
| fr-fr | fr-fr | fr-fr | 👍 |
| zh-cn (any locale not en-us or fr-fr) | en-us | failure | 👎 |
| "" | en-us | failure | 👎 |

Cause

The root cause of this behavior is that the built-in language fallback policy don't work in this scenario, the default behavior is when incoming locale is en-us, the searching order of lg file is "en-us" -> "en" -> "".

to be specific, in main.dialog case,

• If the incoming locale is en-us, the lg file it will be looking for is "main.en-us.lg" -> "main.en.lg" -> "main.lg"
• If the incoming locale is zh-cn, it will look for "main.zh-cn.lg", "main.zh.lg", "main.lg"
• If the incoming locale is ""(empty), it will look for "main.lg",

Since there is no main.lg, it fails eventually. And it don't make sense for composer to generate a main.lg and use as default, because user may switch default language, have a flexible configuration would be better to allow the behavior to be flexible without any extra content manipulation.

this language policy, at it's core is a string => list dictionary that driving the actually behavior like this:
{
"en-us": ["en-us", "en", ""],
"fr-fr": ["fr-fr", "fr", ""],
"“, [""]
}

Solution

So, a simpler solution to fit composer's scenario (and other scenarios i can think of), is to have this language policy configurable. In this specific case, let the dict to be
{
"en-us": ["en-us"],
"fr-fr": ["fr-fr", "en-us"],
"": ["en-us"]
}
Will solve the problem.

The proposal here, is have this language policy configurable by include it into schema and been loadable from dialog file.

Other considerations

This language policy is now only a property in Generator itself, we may consider to put root dialog's language policy into turnContext and shared as default if other dialog is not configured with a customized language policy.

[vishwacsena]

@tomlm will look at this and mark approved if the proposal looks right.

[cleemullins]

@vishwacsena, @tomlm, this is a P0 bug that's not moved forward in 26 days. How can we push this forward?

[cwhitten]

I like to consider an update to the DCR.

Some context:

Today the language policy looks like this:

"en-US":["en-US", "en", ""],
"fr-FR":["fr-FR", "fr", ""]

When the locale on the input matches key "en-US" for resource "foo", the progression will be: foo.en-US, foo.en, foo. If no resource is found it's an error.

In multi-lingual first authoring environment, we're not generating non-locale resource files--there is always a language serialized into the filename. Additionally, some channels like facebook messenger don't always provide the locale with the input, so a fallback is necessary. This is called out as a blocker from the Skills team, and I believe we've heard from the community that this is breaking some bots on facebook messenger.

I suggest a modified proposal: Accept a language policy extension that will support and allow users to define the behavior of an empty input locale resolution.

What this would look like:

language policy:

"en-US":["en-US", "en", ""],
"fr-FR":["fr-FR", "fr", ""]

bot injected language policy:

"br": ["br-BR", "br"]
"": ["de-DE", "br"]

the complete language policy will be:

"en-US":["en-US", "en", ""],
"fr-FR":["fr-FR", "fr", ""],
"br-BR": ["br-BR", "br"],
"": ["de-DE", "br"]

The resource progression for foo and input locale "en-US" would be: foo.en-US, foo.en, foo.de-DE, foo.br, where when you find the empty string in the array, it looks for the existence of "" in the language policy, and respects its value to continue the progression. If it does not exist, behavior today does not change.

The resource progression for foo and input locale "" would be: foo.de-DE, foo.br

cc @boydc2014 @luhan2017

[boydc2014]

I like the update to this @cwhitten, this would make it easier to configure a customized policy. most of time, people just override the entry of "".

Also link to issues blocked by this:
https://github.com/microsoft/BotFramework-Composer/issues/2427
https://github.com/microsoft/BotFramework-Composer/issues/2426

And any facebook related issues will be blocked by this, because FB is not sending locale

[vishwacsena]

I like this. @cwhitten am I correct in understanding this?

1. Composer will never write out resource files without a lang code.
2. It is possible that the incoming activity from a channel might not include a locale. To handle these cases we want the bot author to be able to specify a fall back locale to handle empty locale. E.g. I want to default this to en-us.
3. It is possible that for the locale set on the incoming activity to not have a corresponding lang x locale resource file. E.g. incoming locale was pt-br but there is no resource for pt-br or pt. Today we then fall back to "" which should continue to re-map and evaluate #2 (today we would not do this).

[vishwacsena]

@boydc2014 can this be marked as closed via #3624 ?