Runtime: How can I add an unsupported culture/locale?

[lemmaa]

I need to support several cultures/locales currently unsupported in .NET Core. Is there any way?

| Culture | Description |
|:-:|:-:|
| 'ckb-IQ' | Which is 'ku-arab-IQ' in .NET's culture notation, but only working on Windows platform. https://github.com/dotnet/coreclr/issues/12759 |
| 'mai-IN' and 'sat-IN' | Which are in ISO639-2 specification, but not supported in .NET Core. http://www.loc.gov/standards/iso639-2/php/langcodes_name.php?code_ID=278 http://www.loc.gov/standards/iso639-2/php/langcodes_name.php?code_ID=388|
| 'tu-IN' | Which are not in ISO639 specification, but need to support. |

[tarekgh]

We pick the cultures from the OS. on Windows we get it from Windows. starting from Windows 10, Windows picks up the cultures data from CLDR. on Linux we get it from ICU which is also picking the culture data from CLDR too. so I suggest you just send your request to the CLDR

http://cldr.unicode.org/

[lemmaa]

Thank you @tarekgh. However, even though I confirmed that all three languages, mai-(Maithili), sat-(Santali) and tu-(Tulu) were supported by CLDR, I got the result as dotnet/coreclr#12759.
What should I do?

% Could you reopen this issue until it's resolved?

[tarekgh]

However, even though I confirmed that all three languages, mai-(Maithili), sat-(Santali) and tu-(Tulu) were supported by CLDR, I got the result as dotnet/coreclr#12759.
What should I do?

It depends on the OS version you are running on. Windows 10 for example snap a specific version of CLDR and use it. if you are using Windows 10 please let me know what exactly the version you have and I can tell you when Windows will pick the updated CLDR data.

CC @jefgen from Windows team to help if you have any request here.

If you are using Linux based system, you'll need to check the icu version installed there to know if it is up to date one or you need to upgrade to newer version of the ICU.

I want to clarify that net core doesn't really carry any locale data so the support of the data always come from the OS. so there is no action needed from net core. but feel free to reply here or send any question and I'll be happy to help with that.

[lemmaa]

It depends on the OS version you are running on. Windows 10 for example snap a specific version of CLDR and use it. if you are using Windows 10 please let me know what exactly the version you have and I can tell you when Windows will pick the updated CLDR data.

I'm using 10.0.15063
````
C:\dotnet>systeminfo

Host Name: DESKTOP-OFAH79H
OS Name: Microsoft Windows 10 Enterprise N
OS Version: 10.0.15063 N/A Build 15063
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: Windows User
Registered Organization:
Product ID: 00330-00182-43236-AA709
Original Install Date: 5/16/2017, 3:26:21 AM
System Boot Time: 7/11/2017, 5:33:29 PM
System Manufacturer: SAMSUNG ELECTRONICS CO., LTD.
System Model: Samsung DeskTop System
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~1800 Mhz
BIOS Version: American Megatrends Inc. P04LBS.073.140702.XJ, 7/2/2014
Windows Directory: C:\Windows
System Directory: C:\Windows\system32
Boot Device: \Device\HarddiskVolume1
System Locale: en-us;English (United States)
Input Locale: ko;Korean
Time Zone: (UTC+09:00) Seoul
Total Physical Memory: 8,147 MB
Available Physical Memory: 5,287 MB
Virtual Memory: Max Size: 9,427 MB
Virtual Memory: Available: 6,553 MB
Virtual Memory: In Use: 2,874 MB
Page File Location(s): C:\pagefile.sys
...
````

FYI, I'm using libicu 55.1 and 57.1 on each linux system.

I want to clarify that net core doesn't really carry any locale data so the support of the data always come from the OS. so there is no action needed from net core. but feel free to reply here or send any question and I'll be happy to help with that.

However, I can find various codes that contain locale data in the coreclr.

• https://github.com/dotnet/coreclr/blob/master/src/mscorlib/shared/System/Globalization/LocaleData.Unix.cs#L25
• https://github.com/dotnet/coreclr/blob/master/src/mscorlib/src/System/Globalization/CultureData.cs#L164
• https://github.com/dotnet/coreclr/blob/master/tests/src/Common/CoreCLRTestLibrary/GlobLocHelper.cs#L2009
• ...

% Could you reopen this issue until it's resolved? I'm still hungry.

[tarekgh]

I have checked the latest Windows data and last ICU 5.9.1 and both don't have the locales mai, sat, tu.

I have looked the latest CLDR data http://www.unicode.org/Public/cldr/31.0.1/ and I am not seeing these locales there either. how did you know these locales are part of CLDR?

@jefgen do yo have any more info regardign such locales?

However, I can find various codes that contain locale data in the coreclr.

The files you referenced in your list are not the data we get the list of locales we use in the net core. these data in the referenced files just to help mapping some data on Linux that we cannot get it from the OS. for example mapping the locale name to region name. We used this data also to populate the latest LCIDs values used on Windows when running on Linux. The bottom line, these are not really the locale data we use for cultures and it is just helpers to enable some functionality we cannot do on Linux.

If you are curious to see where we call the OS to get the locale data, you may look at https://github.com/dotnet/coreclr/blob/master/src/corefx/System.Globalization.Native/locale.cpp for Linux. and https://github.com/dotnet/coreclr/blob/master/src/mscorlib/shared/Interop/Windows/Kernel32/Interop.Globalization.cs for Windows

% Could you reopen this issue until it's resolved? I'm still hungry.

Please clarify what action you are asking from net core to do? I'll reopen the issue temporary and will wait you clarify how we can help you here more than directing you how to request adding more locales on the OS's

[jefgen]

@lemmaa I think the result in https://github.com/dotnet/coreclr/issues/12759 that is causing confusion is this:

message 'mai-IN' Hello (en)
message 'sat-IN' Hello (en)
message 'tu-IN' Hello (en)

_(The key thing is the "(en)" at the end of the line.)_

Note that CLDR does not actually have support at all for any locales for mai-*, sat-*, or tu-* in either _main_ or _seed_, and by default, ICU v55 doesn't either.

In your results, it would appear that .NET is actually somehow falling back to English (en) when asked for these locales, which is likely the source of confusion.

[jefgen]

@lemmaa I'd also like to add that CLDR is an open project which accepts feedback and requests from external people. You can request that CLDR add support for these three locales by filing a ticket on the CLDR project here:

http://cldr.unicode.org/index/bug-reports

Additionally, if you have some locale data to contribute for these locales, then you can also help in that regard as well. :)

[tarekgh]

@lemmaa based on the latest replies I am closing the issue again. feel free to reply or ask to reopen if you think there is anything need to be done from net core side. thanks.

[lemmaa]

@tarekgh , @jefgen I got it. I have confirmed that the locale is not included in the CLDR official release package. However, I still do not know why it is in the charts, Maithili for example, and not in the release package. Do you know why, as an experienced person? (The only difference I figured out on the charts is that the number of unicode strings resource is very small compared to other languages. For example, ko has more than 150 string resources, andmai is still around 50.)

By the way, given the release cycle of the CLDR, we need to take some quick action. Fortunately, we can control the our platform directly, but the case of tools is different. How quickly can a new CLDR be applied to Visual Studio? For example, when is the CLDR released in September usually applied to Visual Studio? Or is there another way to successfully compile a *.resx file that contains the locale currently not supported in Visual Studio?

One other thing, Is there any possibility that .NET Standard will support System.Globalization.CultureAndRegionInfoBuilder? I think this is the most appropriate solution for various situations. What do you think about?

P.S. In the meantime, I think I omit the background explanation. Our product supports about 100 locales, which currently has difficulties with 9 in .NET. This is a very important issue for us in applying .NET to our products. So I need help. Thank you in advance for your help.

• Kashmiri(kas/ks-deva-in)
• Sindhi(Sin/sd-deva-in)
• Burmese(bur/my-mm)
• Sorani(sor/irc)
• Kurmanji(kur/ku-arab-in)
• Bhilodi(bhi)
• Maithili(mai)
• Santali(san)
• Tulu(tul)

[tarekgh]

Do you know why, as an experienced person?

I don't know either but I think you can send your question to CLDR folks

How quickly can a new CLDR be applied to Visual Studio?

CLDR is not applied to VS, it will get applied to the OS. So Windows every release picks the new changes from CLDR and include it with that new release. Linux depends on ICU. so the new ICU version will pick the CLDR and then you have to install such new ICU version on that Linux box.

Or is there another way to successfully compile a *.resx file that contains the locale currently not supported in Visual Studio?

If you are running on Windows 10 I expect these locales will work because Windows allows you to use any locale name (in BCP-47 form) even if Windows doesn't have data for such locale. but I didn't really try VS in that scenario. it is worth trying it.

One other thing, Is there any possibility that .NET Standard will support System.Globalization.CultureAndRegionInfoBuilder? I think this is the most appropriate solution for various situations. What do you think about?

Custom cultures are Windows specific concept and not supported on Linux. As I mentioned before, the net core doesn't read any locale specific files and instead, we just call the OS to get the locale data. however, you can still create a custom locale on Windows using the tool https://www.microsoft.com/en-us/download/details.aspx?id=41158 and then this locale will be supported on .Net core as it is supported on the OS. this will be Windows specific solution.

which currently has difficulties with 9 in .NET.

I would suggest sending your request to support such locales to CLDR from now so can try to get support in the near future.

[lemmaa]

however, you can still create a custom locale on Windows using the tool https://www.microsoft.com/en-us/download/details.aspx?id=41158 and then this locale will be supported on .Net core as it is supported on the OS. this will be Windows specific solution.

Thanks, @tarekgh . I'll try it with your guide. I expect that if I can install a custom locale on Windows 10, and Visual Studio works well on it, I can temporarily solve almost all the problems.

[tarekgh]

@lemmaa just to mention, using custom cultures will work on pre-Windows 10 too.

[lemmaa]

Thank you @tarekgh. After installing the custom locale through the guide you gave me, I confirmed that *.resx was successfully built in VS2017. Now I just need to make sure I can build custom locale using dotnet cli on linux for CI. I will share the results. Thanks a lot for your help.

[tarekgh]

Now I just need to make sure I can build custom locale using dotnet cli on linux for CI. I will share the results. Thanks a lot for your help.

I am not sure I understand this. As I mentioned previously custom cultures is Windows specific feature and not supported on Linux. This is OS feature and not .net framework feature.

[tarekgh]

one thing I came across for Linux, is the link http://userguide.icu-project.org/locale#TOC-Usage:-Creating-Locales which it looks ICU can allow creating a locale which is not supported. I didn't look at the details but you take a look and figure out if this can help in your scenario

[jefgen]

Thanks Tarek. Note that if you do "create" a locale using the approach in that document, ICU will end up falling back to using some sort of 'default' data, so you might not get the results that you expect. (Date formats will not necessarily match what you might expect, etc.)

@lemmaa As for the previous question regarding the 'charts' in CLDR, note that CLDR does collect data for all sorts of languages that may not have official "locale" level of support. They also collect translations for many of the language names.
In the link you provided, they are noting that they have localized strings associated with the ISO-639 tag "mai". (For example, the Arabic string for the language "mai" is "المايثيلي".)
Just because something appears in the CLDR charts does not mean that there is actually a locale for it.

[tarekgh]

Thanks @jefgen.

so you might not get the results that you expect.

I think @lemmaa scenario is mainly resource lookup which I think don't care about the locale data more than using the name in the resource lookup.

[lemmaa]

Thanks @tarekgh, don't worry, I know that the tool, Locale Builder 2.0, is for windows only. I know there is a similar approach to linux, and I want to add a custom locale to the linux system in the same way as https://askubuntu.com/questions/21316/how-can-i-customize-a-system-locale. For linux, there is a glibc locale system in addition to the libicu locale, so I can not be sure before trying. I will check with the team's linux locale expert.

Thank you, @jefgen , detailed explanation. I was expecting the same reason. I become more convinced through your explanation. I am glad to know about locale with your help through this opportunity. Thank you very much.

I think @lemmaa scenario is mainly resource lookup which I think don't care about the locale data more than using the name in the resource lookup.

Your guess is absolutely right. :)

[tarekgh]

just ensure the locale name on Linux using underscore in the name and not dashes :-)

[lemmaa]

Yes, so coreclr automatically converts this as in https://github.com/dotnet/coreclr/blob/master/src/corefx/System.Globalization.Native/locale.cpp#L98