Vcpkg: gettext doesn't convert charsets

[375gnu]

Describe the bug
I have an application which keeps translations in UTF-8, to display them in a user's native encoding I use bind_textdomain_codeset(const char * domainname, const char * codeset ) but calls to gettext() return strings in UTF-8.

Environment

• OS: Windows
• Compiler: MSVC2015u3

Expected behavior
Strings returned by gettext() are converted to local encoding.

[vpaladino778]

Is this a vcpkg issue?

[375gnu]

Is this a vcpkg issue?

Vcpkg provides unusable gettext (or libiconv). So it's definitely a vcpkg issue.

[JackBoosY]

Hi @375gnu, thanks for reporting this issue.
Can you provide a demo project to repro this issue?

Thanks.

[375gnu]

Hi @JackBoosY, I pushed it to https://github.com/375gnu/vcpkg_bug/tree/master/gettext_8484. output.png shows the issue, correct output should have 8 letters only, actually there are 16.

[JackBoosY]

@375gnu I still cannot repro it. It's my result:

The only difference is that I use the msvc project instead of the cmake project.

[375gnu]

@JackBoosY it looks like you don't have a translation. You need to translate test.po to your language and put it to %YOUR_TWO_LETTER_LANGUAGE_CODE%/LC_MESSAGES/test.mo

[JackBoosY]

@375gnu Sorry, I have not used this port experience.
Which label needs to be changed?

[375gnu]

Which label needs to be changed?

@JackBoosY didn't get. Are you asking how to create translation file or about something else?

[JackBoosY]

@375gnu How to translate "test.po" file? Should I use a tool to change the character set to utf-8 / unicode or something else? Which label should I change in this file? Can you give some templates?

Thanks.

[375gnu]

@JackBoosY

Step 1, edit test.po, this file should be in UTF-8 (and it already is):

1. Replace be with your language code in "Language: be\n" and "Language-Team: be\n"
2. Replace праверка with your language translation for test in msgstr "праверка\n"
3. You can also update "Plural-Forms:" field but we don't care for it now.

Step 2:

1. download gettext for windows from https://mlocati.github.io/articles/gettext-iconv-windows.html
2. create directory for your translation %LANG%\LC_MESSAGES where LANG has the same value as on step 1
3. convert test.po to binary form: msgfmt -o %LANG%\LC_MESSAGES\test.mo test.po

Step 3:

1. Replace CP1251 in test.cpp with your non Unicode encoding.
2. Compile and run the test application.

Alternative way:
Temporary set regional settings to be_BY (Belarusian, Belarus in the Control Panel) (Windows' interface should stay in English) and run the test application.

[375gnu]

Or you can tell me your language and i will create a test file, according to your profile it should be Chinese.

[JackBoosY]

@375gnu My machine is en-US, but I hope you can make a Chinese files. : )

[375gnu]

@JackBoosY I've pushed updated file. With locale set to zh_CN.UTF-8 it works file on my GNU/Linux.

[JackBoosY]

@375gnu Sadly, I still can't reproduce it on my zh-CN machine. This is my result (complete use of your provided file):

What did I miss?

[375gnu]

@JackBoosY I forgot to push all changes, please try now. Btw, instead of changing locale settings you can try to define environment variable LANGUAGE=zh, this is a GNU gettext extension and should be supported on all platforms.

[JackBoosY]

@375gnu Thanks, confirmed.

[JackBoosY]

Then, I found that the actual character set of the output string is UTF-8, but it is printed as ANSI.
In the absence of knowledge of gettext, we need official support.

[375gnu]

i'd say that the problem is not in gettext itself but in libiconv used by gettext

[JackBoosY]

@375gnu I think gettext doesn't really use iconv right now.
Because only DEPENDS_ON_LIBICONV is defined in CMakeLists.txt and HAVE_ICONV is not defined, but in the code:

#if DEPENDS_ON_LIBICONV && HAVE_ICONV

Rebuilt after adding -DHAVE_ICONV, an error occurred:

gettext-runtime\intl\dcigettext.c(1316): error C2065: 'ICONV_CONST': undeclared identifier

ICONV_CONST definition not found after looking up the header file of libiconv.

After set -DICONV_CONST=\"\", error shows:

relocatable.c.obj : error LNK2019: unresolved external symbol __imp__libiconv_set_relocation_prefix referenced in function _libintl_set_relocation_prefix
libintl.dll : fatal error LNK1120: 1 unresolved externals

Searching for libintl_set_relocation_prefix in libcharset.lib/libiconv.lib... not found.

I think the problem is in libiconv, and the definitions in CMakeLists.txt in libiconv is missing something.

@ethanhs @vicroms Can you have a look? Thanks.

[375gnu]

the problem is that libiconv on windows is built without ENABLE_RELOCATABLE, while gettext with.
I tried to build gettext without ENABLE_RELOCATABLE, and it seems that it works more or less good but too slow. So to fix the issue we need to build libiconv with ENABLE_RELOCATABLE