Suitecrm: 7.9.3 charset problem

[ghost]

Issue

After upgradeing suitecrm to version 7.9.3 with polish language pack all special characters ąśźć, etc. are changed to "?" and stored in database this way.

Expected Behavior

Special characters shouldn't change to "?"

Only error i get:
Tue Jul 18 13:55:18 2017 [19165][1][FATAL] Exception handling in /var/www/crm.fortecait.pl/include/MVC/Controller/SugarController.php:397
Tue Jul 18 13:55:18 2017 [19165][1][FATAL] Exception in Controller: Akcja o takiej nazwie nie istnieje.
Tue Jul 18 13:55:18 2017 [19165][1][FATAL] backtrace:

0 /var/www/crm.fortecait.pl/include/MVC/Controller/SugarController.php(548): sugar_die('Akcja o takiej ...')

1 /var/www/crm.fortecait.pl/include/MVC/Controller/SugarController.php(1083): SugarController->no_action()

2 /var/www/crm.fortecait.pl/include/MVC/Controller/SugarController.php(460): SugarController->handleActionMaps()

3 /var/www/crm.fortecait.pl/include/MVC/Controller/SugarController.php(368): SugarController->process()

4 /var/www/crm.fortecait.pl/include/MVC/SugarApplication.php(105): SugarController->execute()

5 /var/www/crm.fortecait.pl/index.php(52): SugarApplication->execute()

6 {main}

Your Environment

• SuiteCRM Version used: 7.9.3
• Browser name and version Chrome Version 51.0.2704.63 (64-bit)
• Operating System and version Win 10 Home all latest update installed

[horus68]

Please confirm: this only apply to words in fields, not to the actual translated interface of SuiteCRM, right?

Also reported here for Korean language
https://suitecrm.com/forum/suitecrm-7-0-discussion/15088-korean-language-destroyed-when-save-on-any-text-relative-fields-after-upgraded-to-7-9-3

[ghost]

Yes, only word fields.

[pgorod]

For reference, those error messages in English are these

include/MVC/Controller/SugarController.php:548: 
    sugar_die($GLOBALS['app_strings']['LBL_NO_ACTION']);

include/language/en_us.lang.php:1088: 
   'LBL_NO_ACTION' => 'There is no action by that name.',

@AdminPL when you say these values "are stored in the database this way", did you really go into MySQL and check? Or do you just mean when you retrieve them from SuiteCRM app they show wrong?

[ghost]

I checked in database, thought it may be comparing charset problem

[pgorod]

And did you upgrade PHP at the same time? Or MySQL? Or other Linux packages?

[ghost]

No other upgrade's same time, we turned off all auto upgrade after 3 weeks ago php went to latest version an we needed to install version 7 again.

[ghost]

Temporary i fixed issue by replacing include folder from version 7.9.2 on testing installation

[pgorod]

Interesting, that fix... can you narrow it down to just a single file?

Compare the two directories and replace only a single file that fixes the problem?

[ghost]

I tried to switch only files which i found in error log but no effect, so at best i can compare and get back to you with list which files did changed after upgrade, but not sure if it will truly resolve the problem.

Also what i did may affect another functions so i don't recomend it to anyone unless absolutely necessary.

[pgorod]

@horus68 in the upgrade package from 7.9.x to 7.9.3 I see that string defined like this:

'LBL_NO_ACTION' => 'There is no action by that name: %s',

This is the commit merged in 7.9.1 that touches this part of the code. It was something @gymad was working on.

[horus68]

@pgorod crowdin languages are for version 7.9.2
It will take some hours to update to 7.9.3 as this is a massive language update (version 7.9.3 includes some big PR for copyright and unused language strings) but I also need to compare for changes on my local "advanced" language master branch to not remove crowdin strings from PR not merged yet.

[pgorod]

@horus68 I just corrected my post above, I was mistaken. The string is updated on my 7.9.3. Sorry.

Maybe I'm also wrong about this being present since 7.9.1. Anyway, once you have time to update the translations on Crowdin, that %s will have to be present for any system that has the new code in gymad's commit, otherwise the error will come up.

We still have to determine whether the error in the logs is indeed related to the charset bug described. They might be different things.

AdminPL can you please try reverting to the include directory contents that were giving the error, and instead change your Polish language file that contains LBL_NO_ACTION so that it has a %s in it?

'LBL_NO_ACTION' => 'Akcja o takiej nazwie nie istnieje: %s',

[ghost]

Can't do it right now, VM with test system is down, i'll do it ASAP, propably tommorow.

[horus68]

I found that the actual string on Crowdin already includes the % in that string. So if someone uses a language pack version 7.9.2.0 and up) will not have that issue.

[pgorod]

Ok everyone I have some not very good news about this...

• this has nothing to do with language packs, it can be reproduced on the live demo and on my system without any language pack
• the error we've been discussing above is misleading - forget about it, if it's a problem, it's a different problem. This problem does not leave traces in the logs (at least I couldn't find any).

The simplest steps to reproduce:

1. click create Contact
2. fill the Last name field (which is the only mandatory field) with some values
3. hit Save
4. the values are not correctly stored in the database, and so show incorrectly also on screen

The values that produce problems are basically anything that is not the standard english character:

• I tried the name Gonçalves, it turned to Gonlves. Other portuguese characters like ã get the same treatment, the character disappears, and takes the following character along with it...
• I tried the polish ąśźć listed above, it turns to ????

So basically 7.9.3 somehow broke SuiteCRM's Unicode capabilities in every field of every screen...

[samus-aran]

Hi Everyone.

We'll raise this as a high priority and deal with this and produce a patch immediately. I believe the issue has came about dealing with a Security issue, But when we know more we'll update the issue and provide a resolve.

Be in touch soon.

[gymad]

Hi @pgorod, I have a commit with a possible fix. Can you please confirm that it solves the issue?

[pgorod]

@gymad I have a meeting starting now, but in a five minute test, it did seem to solve everything. Those two tests above (with the portuguese and the polish characters) seem to be ok with that code change 👍
Text was showing correctly in phpMyAdmin and on screen.

[ghost]

Sam situation on my instance. Checked also with german character's seems to be working fine.

[halest]

Seems to be working