Zotero-better-bibtex: Japanese rendered as Chinese in Citekey

[bokamm]

Fields with Japanese characters (author names, titles etc.) are transliterated as Chinese in the citation keys. Is there an option I missed or can I change this behavior somehow so that Japanese is correctly rendered?
See the attached picture for the problem. The kanji are rendered as Chinese, the phonetic syllables (e.g. the めぐる/meguru) are recognized correctly. I am using Juris-M 5.0.47m37 with BBT-plugin 5.0.145.

[duncdrum]

Yes I noticed the same just yesterday. It would be great if the transliteration rule for the citekey generation was based on the language set in the language field.

[retorquere]

Turns out this is a known problem in the module I use for transliteration. I'm looking into https://github.com/hexenq/kuroshiro.js.

[retorquere]

@bokamm can you right-click that reference and submit a BBT error report? That will give me a copy of this reference I can use for testing and that I don't have to retype from the screenshot (I wouldn't know how anyway). The issue template asks to do this for this specific reason.

[bokamm]

Thank you for the quick response. I just submitted the report: W4JBSUEQ

From what I can surmise, the other module should do the correct rendering.

[retorquere]

I'm just not sure whether I can safely stack calls on these transliterators. I'll run tests tonight.

[bokamm]

This would not work with normal zotero but only Juris-M, but can you set it up that it calls the kuroshiro module if the field language is set to "ja"?
E.g., in the item I sent via error reporting, the fields have that label:
"multi": {
"main": {
"place": "ja",
"publisher": "ja",
"title": "ja"
},

I don't know if this helps...

[retorquere]

Why wouldn't this work in Zotero?

[retorquere]

And is there a particular reason why I just wouldn't do

transliteration.transliterate(kuroshiro.convert(str, {to:'romaji'}))

[retorquere]

kuroshiro doesn't work. It wants to load data dynamically, and it is convinced that Firefox is a node environment, so it tries to use to use node's require.

[bokamm]

That's unfortunate...
With my previous comment I had meant that only Juris-M allows you to set the language of fields, Zotero does not.

[retorquere]

I'm looking at other modules to see if one of them will work.

Zotero does allow you to set the language for the whole reference, I could use that.

[retorquere]

Is

感じ取れたら手を繋ごう、重なるのは人生のライン and レミリア最高！

all Japanese? When I pass it through https://github.com/lovell/hepburn (fromKana) I get

感JI取RETARA手WO繋GOU、重NARUNOHA人生NORAIN and REMIRIA最高！

[retorquere]

https://github.com/hakatashi/japanese.js strips everything it doesn't recognize, so that's also not an option.

[duncdrum]

I think kuroshiro is the way to go, could you internally take the whole string, transliterate into kana, and then construct the citekey from the romaji transliteration of the kana string?
And yes the whole OP whole string is japanese, its just that they have three separate scripts kanji, hiragana, katakana.

[retorquere]

kuroshuro has a really awkward system to load the data sets. I'd have to create a private bastardized copy of it to get it to work in Zotero.

[duncdrum]

from npm.js
{mode:'okurigana', to:'hiragana'}

and then
{mode:'hiragana', to:'romaji'}
?

sorry traveling, i was thinking along the lines of:
transliterationtransliterate(let $temp = kuroshiro.convert(str, {to:'hiragana'}) return kuroshiro.convert($temp, {to:'Romaji'}) )

[retorquere]

That's not the problem. The problem is it uses kuromoji.js, which dynamically loads it's dictionaries either from disk (when it assumes it's running in node, so it uses node functionality) or online from the same server that hosts the calling script (when it assumes it's running in the browser). Neither of these will work in Zotero (not until the electron port any way).

[retorquere]

kuroshiro.convert(str, {to:'romaji'}) works without problems in node, but does not in Zotero, even if webpacked.

[retorquere]

The problem is going to be similar to https://github.com/takuyaa/kuromoji.js/issues/15 ... that issue a year and a half old with no activity.

[duncdrum]

ah i see now, sorry i misunderstood.
Could you use what you have right now to get a hiragana string, and then use hepurn to go to romaji?

[retorquere]

How would I get a hiragana string?

[retorquere]

https://github.com/Pomax/node-jp-conversion just straight up errors out.

[duncdrum]

In that case change of tactics, if the record is in japanese, see if there is a romaji field in Juris-M, if there is use that for citekey, otherwise use hepburn, kuroshiro…and pray for the best?

[retorquere]

https://github.com/WaniKani/WanaKana gets similar results to hepburn

[retorquere]

kuroshiro is a non-starter because it uses kuromoji.

[retorquere]

How do I request language variants inside Juris-M (so not in the translator)?

[duncdrum]

yes its already in their descriptions, non support kanji transliteration, except for the non-starter.
the json should have IANA language tags, no?

[retorquere]

I don't know what that last part means. What JSON?

[duncdrum]

I meant the actual record, I thought BBL is using the JSON version of that to do its thing? Or do you run a SQL for reading the key data?

[retorquere]

Oh you mean the json version of the reference. Keys are generated ahead of time, so I use item.getField

[duncdrum]

i see, if this includes the juris-m extra fields, you could scan the internal lang tags for any jpa-Latn. This would make nice keys for juris-m users, but wouldn't help regular zotero users i m afraid.

[retorquere]

It wouldn't help zotero users, and I don't know how to scan the internal lang tags.

[duncdrum]

in which case I m out of ideas, short of waiting for electron.

[retorquere]

There must be an internal api to get language variants in Juris-M, right?

[duncdrum]

@fbennett would be in the know, is there an internal api to get the language variants?
or a good js library for transliterating mixed kanji / kana contents ?

[retorquere]

What does this say? https://www.npmjs.com/package/jaconv

[duncdrum]

nope, doesn't process kanji either. I mean i know that kanji are hard, but i m quite surprised that there aren't more options out there.

[retorquere]

Is kanji prevalent? Or would hepburn generally do the job?

I've taken another look at kuromoji but to get that to work I'd have to get fancy with webpack and graspjs. It'd be fragile and I don't have the time right now.

[duncdrum]

I m afraid kanji are very prevalent in titles, names are 100% kanji, ...

[duncdrum]

this promises to do the trick, https://www.npmjs.com/package/kakasi not sure how it plays with FF though

[bokamm]

dear both, thank you for the continued work put into this and please excuse my silence (time zone difference). Using any internal api for a field's language setting is what I had in mind — but a solution that works for Juirs-M _and_ Zotero would be more ideal).

[retorquere]

this promises to do the trick, https://www.npmjs.com/package/kakasi not sure how it plays with FF though

https://github.com/loretoparisi/kakasi.js/blob/master/kakasi/index.js#L10

Perhaps this one is easier to graspjs into submission, but as it stands, it's node-only.

[retorquere]

kakasi is another non-starter; all the node "library" does is start this (platform-dependent) executable for each separate transliteration.

[retorquere]

Just to make clear, if Kanji is hard, and kuroshuro+kuromoji has a bug a year and a half old (which makes me think it's not a trivial fix), I am not going to try my hand at creating my own library to do Japanese romanization, or to fix kuromoji. Just putting that out there. I don't have time to get acquainted with a whole new field of study right now.

[fbennett]

Sorry for the delay in responding. There are methods in Juris-M for pulling out field variants. There is also a node module for un-remangling data delivered by Zotero but that was saved in Juris-M. I need to check the details again, and will post in a few hours after I get home.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5.0.147.7053 ("only escape and for two-part names").

[retorquere]

Can someone get me a new sample? W4JBSUEQ has expired and I forgot to save it.

Also, give 7053 a go. You'll have to go into the hidden prefs and enable kuroshiro; I managed to fool it into loading its dictionaries, but they're big, and I don't want to add 100M+ memory consumption just for romajization by default. You need to restart to have a change in this setting to take effect.

[bokamm]

I just sent an error report (BM2YLC4F). And will give 7053 a try! Thank you so much for all the hard work!

[bokamm]

I am sorry but where do I find the hidden prefs?

[retorquere]

I just sent an error report (BM2YLC4F)

I should have been more specific: can you send an error report by right-clicking a reference with Japanese characters in the author name?

I am sorry but where do I find the hidden prefs?

https://retorque.re/zotero-better-bibtex/configuration/#hidden-preferences

[bokamm]

I resent the report (WMTN2LNN). This time the reference should be included (do not know why it wasn't the first time).
Seems to work after enabling kuroshiro!

[retorquere]

That reference gets key higuchinippongatahaigaishugiwomegurutoi2014, is that correct?

[retorquere]

I've added WMTN2LNN as a test case and it now generates higuchinippon2014 with your citekey pattern.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5.0.147.7069 ("#979 test case for juris-m").

[retorquere]

I'd appreciate one more confirmation on 7069 before I merge.

[retorquere]

Can I just say here that for someone raised sheltered in US-ASCII, Japanese is weird in a whole lot of ways?

[bokamm]

Please excuse the late response (time zones...). I just checked 7069 and refreshed several Japanese entries: Looks perfect to me! Thank you so much!
Oh, and weird: Yes! :)

[retorquere]

I'm getting in a wave of improvements from @bwiernik, but when that subsides I'll cut a new release.

[bokamm]

Understood, thank you!

[retorquere]

(in the meantime you can just use 7096. The kuroshiro changes are the same here and in the release, and the changes by bwiernik are largely cosmetic. You will be upgraded automatically from 7096)

[retorquere]

Kuroshiro has just gone async. I can stay on the older version for a while, but making the key formatter async to accommodate kuroshiro would be a major undertaking.

[retorquere]

It also has a broken node detection, so it erroneously determines it's running in node when run in Firefox chrome code, and because of that uses node-only features. This is the end of kuroshiro for BBT until the Electron port happens.

[bokamm]

Thank you for letting us know!
Just to understand: If I update to 5.0.183 the rendition of Japanese will no longer work, correct? How can I prevent that existing keys are rendered as Chinese again? Should I pin all keys or would this break the feature that new keys are generated by taking existing keys into account? Sorry, if these are noob questions, I just want to make sure that this won't break any articles or papers using the keys as they are now.
(again thank you for the work put into this)

[retorquere]

It's not a noob question at all! Thanks for asking it.

For the time being, I am keeping korushiro at the old version that I could still patch into submission. The problem with the javascript ecosystem is that sooner or later I'm going to update another library that has conflicting dependencies, and the I'll need to either update kuroshiro or drop it.

Even then all is not lost, as citekeys don't update unless the reference updates, and don't update at all if you've pinned them, so you can just update to 183; nothing will change. I will try to talk to the kuroshiro author to see if something can be worked out.

[bokamm]

Thank you again for the quick response!
Ok, understood. And - repeating myself - thank you also for continuing to deal with kuroshiro!

[retorquere]

Oh this is just great, the new kuroshiro also exhibits new behavior. Is it OK if

         author = {樋口, 直人},
         booktitle = {日本型排外主義: 在特会・外国人参政権・東アジア地政学},
         date = {2014},

generates the citekey higutinitupon2014 rather than what it generated before (higuchinippon2014)?

[bokamm]

Hallo. Er... no, that would not be correct. There are different transliterations for Japanese but that one is wrong.

[retorquere]

I've opened an issue for it over at kuroshiro (https://github.com/hexenq/kuroshiro/issues/41), maybe it's an error.

[retorquere]

Where are you from? Hallo points to a germanic language I think.

[bokamm]

Yup, from Germany but living in Japan.

[retorquere]

Cool. I'm from the Netherlands mostly. I have a patched kuroshiro that removes the (unnecessary and bothersome) async path, but that doesn't resolve https://github.com/hexenq/kuroshiro/issues/41, so we'll have to wait for that; I may need your expertise because I don't know anything about Japanese. In the mean time, 183 will work as usual.

[bokamm]

Thank you! Yes, will definitely help with the Japanese (and the kuroshiro author is correct, hepburn would be the romanization of choice).

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5.0.183.7984 ("kuroshiro-patched 1.1.0").

[retorquere]

OK, so the patched kuroshiro does the job. That should give me some breathing room.

[bokamm]

Great!

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.