Botframework-webchat: Specific Locale passed into webchat is not being sent to Bot service. (HealthBot)

[ggetty]

Screenshots

Version

https://cdn.botframework.com/botframework-webchat/latest/webchat-es5.gzip.js

To determine what version of Web Chat you are running, open your browser's development tools, and paste the following line of code into the console.

[].map.call(document.head.querySelectorAll('meta[name^="botframework-"]'), function (meta) { return meta.outerHTML; }).join('\n')

If you are using Web Chat outside of a browser, please specify your hosting environment. For example, React Native on iOS, Cordova on Android, SharePoint, PowerApps, etc.

Describe the bug

Looks as though if I pass in a locale "es-US", it will send "es-ES" to the HealthBot service when posting activities. It seems the chatbot js only supports es-ES as a locale for it's strings and then therefore sending on this locale to the service. It'd be ideal that the locale passed in to the chatbot can translate to es-ES for the chatbot strings however respect the intention and forward on the locale that was passed in.

Steps to reproduce

1. Pass in locale "es-US" to store.dispatch({
   type: 'DIRECT_LINE/POST_ACTIVITY',
   meta: { method: 'keyboard' },
   payload: {
   activity: {
   type: "invoke",
   name: "InitConversation",
   locale: "es-US" ...
2. Bot text is based on "es-ES" locale.

Expected behavior

Would like to see that the culture passed in is respected to forward on to botsvc.

Additional context

This was found using the HealthBot service as we created localization files for "es-US".

[Bug]

[compulim]

@ggetty makes sense!

Today, we have a list of supported languages (total 47), and we do the fallback (es-US -> es-ES) as soon as in the constructor.

I think we should:

• (Optional) Verify the xx-YY from the Unicode database, which consists of 542 combos (instead of 47, which we supported)
  
  
  
  • If it's not in the combo, we give a warning and optionally fallback
  
  
• We should pass es-US as-is to the bot
• We should only do the fallback when we need it
  
  
  
  • E.g. we try to format the date in es-US, but since we don't support it, we fallback to es-ES instead
  
  

I will let @cwhitten decide the priority of this work.

[compulim]

Before the fix, can you steal some code in this sample and workaround with the locale field in the activity payload?

https://github.com/microsoft/BotFramework-WebChat/tree/master/samples/04.api/b.piggyback-on-outgoing-activities

const store = window.WebChat.createStore({}, ({ dispatch }) => next => action => {
  if (action.type === 'DIRECT_LINE/POST_ACTIVITY') {
    action = window.simpleUpdateIn(
      action,
      ['payload', 'activity', 'locale'],
      () => 'es-US'
    );
  }

  return next(action);
});

You will also need to load a script in your HTML. It is primarily for simplifying the code.

<script
  crossorigin="anonymous"
  src="https://unpkg.com/simple-update-in/dist/simple-update-in.production.min.js"
></script>

[amir-microsoft]

Hi @compulim, I think xx-YY won't cover all cases, and combinations like XX-yy should also be accepted.
Take a look at this answer from stackoverflow:
https://stackoverflow.com/a/48300605

[compulim]

Hi @amir-microsoft. Sorry, for xx-YY, I mean, it is one of the below, official from Unicode consortium, including multiple formats, e.g. zh-Hans-HK, agq, vai-Latn.

af
af-NA
agq
ak
am
ar
ar-AE
ar-BH
ar-DJ
ar-DZ
ar-EG
ar-EH
ar-ER
ar-IL
ar-IQ
ar-JO
ar-KM
ar-KW
ar-LB
ar-LY
ar-MA
ar-MR
ar-OM
ar-PS
ar-QA
ar-SA
ar-SD
ar-SO
ar-SS
ar-SY
ar-TD
ar-TN
ar-YE
as
asa
ast
az
az-Cyrl
az-Latn
bas
be
bem
bez
bg
bm
bn
bn-IN
bo
bo-IN
br
brx
bs
bs-Cyrl
bs-Latn
ca
ca-AD
ca-ES-VALENCIA
ca-FR
ca-IT
ccp
ccp-IN
ce
ceb
cgg
chr
ckb
ckb-IR
cs
cu
cy
da
da-GL
dav
de
de-AT
de-BE
de-CH
de-IT
de-LI
de-LU
dje
dsb
dua
dyo
dz
ebu
ee
ee-TG
el
el-CY
en
en-001
en-150
en-AE
en-AG
en-AI
en-AS
en-AT
en-AU
en-BB
en-BE
en-BI
en-BM
en-BS
en-BW
en-BZ
en-CA
en-CC
en-CH
en-CK
en-CM
en-CX
en-CY
en-DE
en-DG
en-DK
en-DM
en-ER
en-FI
en-FJ
en-FK
en-FM
en-GB
en-GD
en-GG
en-GH
en-GI
en-GM
en-GU
en-GY
en-HK
en-IE
en-IL
en-IM
en-IN
en-IO
en-JE
en-JM
en-KE
en-KI
en-KN
en-KY
en-LC
en-LR
en-LS
en-MG
en-MH
en-MO
en-MP
en-MS
en-MT
en-MU
en-MW
en-MY
en-NA
en-NF
en-NG
en-NL
en-NR
en-NU
en-NZ
en-PG
en-PH
en-PK
en-PN
en-PR
en-PW
en-RW
en-SB
en-SC
en-SD
en-SE
en-SG
en-SH
en-SI
en-SL
en-SS
en-SX
en-SZ
en-TC
en-TK
en-TO
en-TT
en-TV
en-TZ
en-UG
en-UM
en-US-POSIX
en-VC
en-VG
en-VI
en-VU
en-WS
en-ZA
en-ZM
en-ZW
eo
es
es-419
es-AR
es-BO
es-BR
es-BZ
es-CL
es-CO
es-CR
es-CU
es-DO
es-EA
es-EC
es-GQ
es-GT
es-HN
es-IC
es-MX
es-NI
es-PA
es-PE
es-PH
es-PR
es-PY
es-SV
es-US
es-UY
es-VE
et
eu
ewo
fa
fa-AF
ff
ff-Latn
ff-Latn-BF
ff-Latn-CM
ff-Latn-GH
ff-Latn-GM
ff-Latn-GN
ff-Latn-GW
ff-Latn-LR
ff-Latn-MR
ff-Latn-NE
ff-Latn-NG
ff-Latn-SL
fi
fil
fo
fo-DK
fr
fr-BE
fr-BF
fr-BI
fr-BJ
fr-BL
fr-CA
fr-CD
fr-CF
fr-CG
fr-CH
fr-CI
fr-CM
fr-DJ
fr-DZ
fr-GA
fr-GF
fr-GN
fr-GP
fr-GQ
fr-HT
fr-KM
fr-LU
fr-MA
fr-MC
fr-MF
fr-MG
fr-ML
fr-MQ
fr-MR
fr-MU
fr-NC
fr-NE
fr-PF
fr-PM
fr-RE
fr-RW
fr-SC
fr-SN
fr-SY
fr-TD
fr-TG
fr-TN
fr-VU
fr-WF
fr-YT
fur
fy
ga
ga-GB
gd
gl
gsw
gsw-FR
gsw-LI
gu
guz
gv
ha
ha-GH
ha-NE
haw
he
hi
hr
hr-BA
hsb
hu
hy
ia
id
ig
ii
is
it
it-CH
it-SM
it-VA
ja
jgo
jmc
jv
ka
kab
kam
kde
kea
khq
ki
kk
kkj
kl
kln
km
kn
ko
ko-KP
kok
ks
ksb
ksf
ksh
ku
kw
ky
lag
lb
lg
lkt
ln
ln-AO
ln-CF
ln-CG
lo
lrc
lrc-IQ
lt
lu
luo
luy
lv
mas
mas-TZ
mer
mfe
mg
mgh
mgo
mi
mk
ml
mn
mr
ms
ms-BN
ms-SG
mt
mua
my
mzn
naq
nb
nb-SJ
nd
nds
nds-NL
ne
ne-IN
nl
nl-AW
nl-BE
nl-BQ
nl-CW
nl-SR
nl-SX
nmg
nn
nnh
nus
nyn
om
om-KE
or
os
os-RU
pa
pa-Arab
pa-Guru
pl
prg
ps
ps-PK
pt
pt-AO
pt-CH
pt-CV
pt-GQ
pt-GW
pt-LU
pt-MO
pt-MZ
pt-PT
pt-ST
pt-TL
qu
qu-BO
qu-EC
rm
rn
ro
ro-MD
rof
root
ru
ru-BY
ru-KG
ru-KZ
ru-MD
ru-UA
rw
rwk
sah
saq
sbp
sd
se
se-FI
se-SE
seh
ses
sg
shi
shi-Latn
shi-Tfng
si
sk
sl
smn
sn
so
so-DJ
so-ET
so-KE
sq
sq-MK
sq-XK
sr
sr-Cyrl
sr-Cyrl-BA
sr-Cyrl-ME
sr-Cyrl-XK
sr-Latn
sr-Latn-BA
sr-Latn-ME
sr-Latn-XK
sv
sv-AX
sv-FI
sw
sw-CD
sw-KE
sw-UG
ta
ta-LK
ta-MY
ta-SG
te
teo
teo-KE
tg
th
ti
ti-ER
tk
to
tr
tr-CY
tt
twq
tzm
ug
uk
ur
ur-IN
uz
uz-Arab
uz-Cyrl
uz-Latn
vai
vai-Latn
vai-Vaii
vi
vo
vun
wae
wo
xh
xog
yav
yi
yo
yo-BJ
yue
yue-Hans
yue-Hant
zgh
zh
zh-Hans
zh-Hans-HK
zh-Hans-MO
zh-Hans-SG
zh-Hant
zh-Hant-HK
zh-Hant-MO
zu

[amir-microsoft]

Thanks @compulim ,
Can you share where this list is coming from?
looks like common locales are missing (like he-IL).
Also, we see cases of using lower case for the country code, and from the docs, it seems to be permitted - https://stackoverflow.com/a/48300605
any reason why lower case for country code won't be allowed?

[compulim]

In Unicode CLDR, he is he-IL. Similar to en is in the list but not en-US.

When you pass he-IL, we will pass he-IL to the bot. But when we try to format date and units using he-IL, we will normalize it into he before resolve.

This list downloaded by cldr-data package of version 36.

[beyackle]

I think the standard way to do a fallback is to just drop the country code entirely. If we don't support es-US, then the next language we should look for is just es, not necessarily es-ES. If it can't find es without a country code, then that's a problem - I don't know if there's a policy about this, but I'd expect every language to have a sensible default when the country code is unspecified. Similarly, he-IL isn't in the list, but the fallback should find he, which is.

[compulim]

We have this fallback logic in-place already in normalizeLanguage.js.

But we do this fallback too early, so the bot is not receiving he-IL, but only he. I have a PR up, so we are passing the locale as-is to the bot, even it is an invalid locale.