Ultimate-geography: NB vs NO language code for Norwegian (bokmål)

[ukanuk]

In PR #147, the Norwegian (bokmål) translation was added using language code nb. The translations ought to be based on https://no.wikipedia.org, which uses a language code no. I'm guessing it came from nb being the acronym of Norsk bokmål. Can @Vegas or someone else confirm there are no other significant reasons for preferring nb?

My reason for wanting the change is because I started working on a script, and am finding it very inconvenient having to map between no on Wikipedia and nb in the deck data.csv.

FWIW, my still-hacky script is intended to verify English capitals (this could catch if capitals changed in the future), and verify/generate translations for Country and Capital fields. @GrimPixel suggested the generation part in #239, and I'm finding that while a simple script cannot deal with info fields or confusing capitals like in Bolivia, it can hopefully translate/verify 100% of countries and capitals for simple countries.

I looked a little into where Wikipedia gets its language codes, and I think it's ISO 639-1, but I couldn't find explicit verification of this. https://en.wikipedia.org/wiki/ISO_639.

[ukanuk]

Of course, I do one last search after posting and find that Wikipedia gets its language codes from a mix of ISO 639-1, ISO 639-3, and a few non-standard codes as explained at https://en.wikipedia.org/wiki/List_of_Wikipedias.

Turns out no is not ISO-defined, it's just for Wikipedia:

https://en.wikipedia.org/wiki/List_of_Wikipedias#Wikipedia_edition_codes

WP code | WP edition name | ISO 639 code for this language | Notes
-- | -- | -- | --
no | Norwegian Bokmål | nb, nob | ISO uses no for Norwegian in general. (Norwegian Nynorsk is at 'nn' in both ISO and Wikipedia.)

[axelboc]

I have no objection with changing the language code to match Wikipedia. It makes sense since we follow it so closely, and NO is just as clear, if not clearer, than NB.

---

I'm very much looking forward to your script. It sounds like it will make our lives a lot easier!

[aplaice]

I think that Wikipedia is doing "the wrong thing" here. It seems that the current situation is mostly a historical accident. The initial Norwegian Wikipedia chose no while not mandating which Norwegian (Bokmål (NB) or Nynorsk (NO)) should be used. It ended up mostly using Bokmål, due to the 90 %/10 % split in real-world use, in writing, so a new Wikipedia was created for Nynorsk, while the original Norwegian Wikipedia chose to prescribe Bokmål. Currently, Wikipedia does redirect from the nb subdomain to the no one, though.

---

I'm very much looking forward to your script. It sounds like it will make our lives a lot easier!

So am I! :)

Another idea would be to use Wikidata's data, though it has the disadvantage that it's language independent, so it doesn't reflect the conflicts between language versions (e.g. Montenegro only has one capital).

[ukanuk]

I think that Wikipedia is doing "the wrong thing" here. It [seems[(https://en.wikipedia.org/wiki/Norwegian_Wikipedia#History) that the current situation is mostly a historical accident.

This is a great reason to use the ISO code rather than the Wikipedia code. Another reason to prefer the ISO code is that the Norwegian translation sources don't actually use Wikipedia where possible, according to CONTRIBUTING.md#translation-sources. With some more searching I found it's been a large discussion about whether no should change to nb on Wikipedia, and presumably there's even more discussion in Norwegian as in this 2017 discussion. The question then is what is the "right thing" for UG, to follow ISO or to follow Wikipedia's current conventions whatever they may be. We want to be as correct as possible of course, but also want to avoid repeating discussions already completed on Wikipedia.

I'm very much looking forward to your script. It sounds like it will make our lives a lot easier!

So am I! :)

Please temper your expectations! For now my hope is to run it once before each release, and provide a list of countries whose Wikipedia info may have changed since the last release. The (hopefully short-ish) list of results would then be checked manually 😞. Now that I've realized the Norwegian deck doesn't even depend on Wikipedia, this tool may not be so useful for the Norwegian translation. I won't know until I'm able to run it on all the countries and get some summary results. Or maybe this tool can alert us to Norwegian Wikipedia articles needing updates to match Anki Ultimate Geography's NB/NO translation 😄 And if someone wants to translate a new language, I hope this tool can automatically populate Country and Capital fields with mostly-correct data to help kick-start the process.

Another idea would be to use Wikidata's data, though it has the disadvantage that it's language independent, so it doesn't reflect the conflicts between language versions (e.g. Montenegro only has one capital).

Wikidata would absolutely be easier to use programmatically, but it's Wikidata does not always match Wikipedia (as in your example) and the source of truth for this deck is Wikipedia, not Wikidata. Besides, my limited tests so far lead me to believe Wikipedia can be directly parsed well enough often enough to be useful. Ideally in the future, the Wikipedia page data will come from Wikidata directly as has already been done with Category:Templates using data from Wikidata, but that day is not today. And this only applies to capitals -- for translations, these should be much easier since we hopefully only need to grab the interwiki language links provided in JSON from their API, like this for London.

[GrimPixel]

Either in Wikipedia or Wikidata examples above, only one script for a language is available, e.g. Chinese, Serbian, Kazakh. Can more scripts show up with the API?

[ukanuk]

@GrimPixel can you clarify? The interwiki langlinks for London include these results:

{"lang":"ar","*":"\u0644\u0646\u062f\u0646"}
{"lang":"gan","*":"\u502b\u6566"}
{"lang":"zh","*":"\u4f26\u6566"}
{"lang":"zh-classical","*":"\u502b\u6566"}
{"lang":"zh-min-nan","*":"L\u00fbn-tun"}
{"lang":"zh-yue","*":"\u502b\u6566"}
{"lang":"sr","*":"\u041b\u043e\u043d\u0434\u043e\u043d"}
{"lang":"kk","*":"\u041b\u043e\u043d\u0434\u043e\u043d"}

The first character under zh is \u4f26, which refers to Unicode Codepoint 4F26, which is 伦.

The rest of these map out (I think I've done it right...) to:

'ar', 'gan',  'zh',  'zh-classical', 'zh-min-nan', 'zh-yue', 'sr',     'kk':
'لندن', '倫敦', '伦敦', '倫敦',         'Lûn-tun',    '倫敦',    'Лондон', 'Лондон'

I am aware that Chinese Wikipedia automatically converts between five orthographies so that's absolutely something I'm not sure how to fix. But I'm not aware of other languages where that's an issue.

[GrimPixel]

On Chinese Wikipedia, there is a drop-down list at the upper-left corner
https://zh.wikipedia.org/wiki/Wikipedia:%E9%A6%96%E9%A1%B5

You can try to select versions of different regions and see their locale codes: zh-cn, zh-hk, zh-mo, zh-my, zh-sg, zh-tw. They don't appear in your list because they are all inside one Wikipedia site: the Chinese Wikipedia.
Notice: “zh” doesn't mean “Chinese” here, but “Mandarin”. As you can see, Min Nan (Taiwanese) and Yue (Cantonese) are represented with “zh-min-nan” and “zh-yue”, which are different sites from Chinese Wikipedia.

[GrimPixel]

I would suggest that when your script for fetching those titles is completed, you can add some more features and release it as a language learning tool. I will recommend it to other people then.

[axelboc]

From what I gather, we're all on board with keeping the ISO code nb, so I'm closing this issue.

If there's confusion as to which Wikipedia site a translation of the deck is based on, it can easily be clarified under _Translation sources_ (which is the case for the Norwegian translation).