Prestashop: Email validation are not checked cyrillic characters before the @ sign. Also there is no support for Cyrillic domains.

[ilyasovivan]

Describe the bug
At the php level, the email validation form skips the cyrillic characters before the @ sign. Also, the form has no support for cyrillistic domains.

How it should work
Cyrillistic domains are supported by some email providers, such as Yandex or Google (Gmail).
Cyrillic characters should not be before the @ sign, only after it in domain, if the domain is converted to punycode.
For example:
[email protected] - this is a valid email address.
test@сайт.рф - this is a valid email address ( after in conversion to pinycode: [email protected] )
иван@сайт.рф - this is not a valid email address.
иван@site.com - this is not a valid email address too.
It should work that way.

To Reproduce
Steps to reproduce the behavior:

1. Go to the Create Account page
2. First, we need to disable html5 validation (because we will test only server validation).
   For that we need to add 'novalidate' atribute in <form> tag.
3. On the page in the Email field we write: test@сайт.рф
4. Click "Save" button. And we will see the message:
   > Invalid format.
   This is the first problem. The second is cyrillic characters before the @ sign.
5. Now repeat step 2 and enter another email address: тест@site.com
6. After that we will see the message:
   > Fatal error: Uncaught exception 'Swift_RfcComplianceException' with message 'Address in mailbox given [тест@test.site] does not comply with RFC 2822, 3.6.2.'

Screenshots
Step2:

Step3:

Step6:

Additionnal information
PrestaShop version: from 1.6 to 1.7
PHP version: all

[khouloudbelguith]

Hi @ilyasovivan,

This issue is fixed in the PrestaShop version 1.7.4.0.
Here's the PR: https://github.com/PrestaShop/PrestaShop/pull/9080 to fix the issue.
Also, you need to check that the php-intl extension is installed on your server.
Your ticket is a duplicate of #9080
So, I close it.
Thanks!

[khouloudbelguith]

Duplicate of #9080

[ilyasovivan]

@khouloudbelguith NO!!! THIS IS NOT FIXED.
I spent about 1.5 hours of my time to describe the problem, and you did not even want to read it!
JUST SEE THIS VIDEOSCREEN: https://www.youtube.com/watch?v=jkIJd_y8yoI
AND OPEN THIS ISSUE.

Tested on Windows 10, Open Server 5.2.2
PHP: 7.0 x64
Apache for PHP 7.0
MariaDB 10.3 x64

I tested stable 1.7.4.4 and beta version 1.7.5.0

[khouloudbelguith]

@ilyasovivan, I tried with the email test@домен.com in the PS1.7.4.4 / PS1.7.5.0beta1 & I did not manage to reproduce the issue.
I attached a video record
https://drive.google.com/file/d/1lKPrKuPe4vl4mXdw_eDOYKuBULtx-CD4/view
Thanks to check & feedback.

[khouloudbelguith]

Hi @ilyasovivan,

You can take a look at https://en.wikipedia.org/wiki/Punycode
In this situation we can't do anything about Chrome, it's a normal behaviour when you send anything in input type="email" everything store in hostname is converted in PunyCode.

This is what we received after chrome conversion:

/home/got/projects/prestashop/prestashop/index.php:27:
array (size=8)
'id_customer' => string '' (length=0)
'id_gender' => string '2' (length=1)
'firstname' => string 'azdvv' (length=5)
'lastname' => string 'azdvv' (length=5)
'email' => string '[email protected]' (length=23)
'password' => string 'test@домен.com' (length=19)
'birthday' => string '' (length=0)
'submitCreate' => string '1' (length=1)

Thanks!

[ilyasovivan]

@khouloudbelguith
More info about cyrillic domains like .рф .срб .укр .москва .онлайн .рус .орг .дети: https://www.rnids.rs/en/domains/cyrillic-on-internet

For test you can use it vaild email addresses:
test@тест.рф or test@украина.укр
And NOT vaild (because email addresses cannot contain cyrillic characters before the @ sign):
россия@тест.рф or украина@тест.укр or россия@test.ru or ла-ла-ла@test.ru

[ilyasovivan]

'email => string [email protected] (length=23)'

Yes.
test@тест.рф = [email protected]
Because all cyrillic domains are converted to Punycode

[ilyasovivan]

'I did not manage to reproduce the issue.'

OK. Maybe test@домен.com is worked. But now can you check test@домен.рф ?
Or you can use real site: погода.рф and real email address info@погода.рф
I test it and I see "Invalid format." - it works wrong.

[ilyasovivan]

@khouloudbelguith I enable Debug in defines.inc.php and test with test@домен.com

Chrome is the most popular browser, and if it does, then Prestashop should work with it.

[khouloudbelguith]

Hi @ilyasovivan,

This issue is occurred due to php-intl extension as I said in my first comment.
Could you please try to install it, it is required.
Thanks to check and feedback.

[ilyasovivan]

@khouloudbelguith ok.

test@погода.com - works good
test@погода.рф - Invalid format Error
погода@test.ru - With novalidate atribute in <form> tag for disable html5 validation (because we test only server validation, because in some modules it can used without html5 form):

[ilyasovivan]

@khouloudbelguith now you are understood what I mean?

test@погода.рф > need to work (because it real valid email address)
погода@test.ru > need to be "Invalid format" (because email addresses cannot contain cyrillic characters before the @ sign)
That's all. Thank for your support, @khouloudbelguith

[khouloudbelguith]

@ilyasovivan,

test@погода.com - works good
test@погода.рф - Invalid format Error
погода@test.ru - when I tried with this email, I have this error => The part followed by the symbol @ must not contain п

Thanks to check & feedbak.

[ilyasovivan]

The part followed by the symbol @ must not contain п

You have not disabled the html5 validation to test the validation using the server.

Add novalidate atribute to <form> tag

html5 validation is validation via browser, not Prestashop. Some browsers may not have html5 validation support. Or some modules or themes for Prestashop may not have this structure of registration form. And in this case, you need to check email via PrestaShop.

[khouloudbelguith]

@ilyasovivan, thanks for this clarifications.
When I added the novalidate atribute to<form>tag, I manage to reproduce the issue with PS1.7.5.0beta1.

@PierreRambaud, what do you think?
Thanks!

[ilyasovivan]

Thank you @khouloudbelguith !
@PierreRambaud if you will fix it, can you add fix for PS 1.6 too?

[PierreRambaud]

Hi @khouloudbelguith Can you try with the latest 1.7.5.x please?

[khouloudbelguith]

@PierreRambaud, I tried with PS 1.7.5.x.
When I added the novalidate atribute to <form> tag, I manage to reproduce the issue.
Thanks!

[ilyasovivan]

@khouloudbelguith @PierreRambaud
And don't forget about this issue:
test@погода.рф - "Invalid format" Error
Need to add support for cyrillistic domains.

Cyrillistic domains using in countries: Russia, Ukraine, Bulgaria, Mongolia, Kazakhstan, Serbia and more.
We need it fix for Prestashop.

And you need to learn more about converting these domains. For this using Punycode.
And some browsers can convert cyrillistic domains before sending and some other browsers may not.

[ilyasovivan]

@khouloudbelguith add 1.6.1.23 and 1.7.5.0 labels, please.

[khouloudbelguith]

@ilyasovivan, As you can see here : the maintenance of PrestaShop 1.6 is extended to June 2018 for security and critical bugs.
Thanks!

[TemKaSD]

Does not work (((

[ilyasovivan]

Does not work (((

Здравствуй, земляк))
Данный баг с приоритетом Standart и в списке To Do на будущее.
На данный момент есть около 70 нерешенных багов с приоритетом повыше, я думаю, пока эти баги не будут решены, команда Prestashop не возьмётся за исправление багов с приоритетом Standart.
Остается только ждать пока это случится или пока какой-нибудь толковый кодер столкнется с такой же проблемой и захочет поправить этот баг на Гитхабе.
Со своей стороны вы можете ускорить решение вопроса придавая данной проблеме бОльшую огласку, например на форуме https://forum.prestashop.com

Но конкретно в вашем случае нужно перевести домен в punycode:
manager@растения-кубани.рф >> [email protected]

[TemKaSD]

@ilyasovivan Прив. Я пробовал в настройках магазина, там где адрес указывается, ошибку выдало, домен в punycode не принимает при сохранении.

[Com3t]

RFC 6532 allows Cyrillic characters in the mailbox part of an email address. (e.g. https://cctld.ru/en/news/news_detail.php?ID=11042)

[NeOMakinG]

It looks like it's a backend issue, by the way this also needs to be fixed asap don't you think @colinegin ? It's blocking many shops using cyrillic characters

[PierreRambaud]

This issue should be fix with #13761

[colinegin]

Thanks @PierreRambaud , i remove this issue from the 178 kanban then.
We will test it when the PR #13761 will be merged !

[PierreRambaud]

Should be ok now :)

[khouloudbelguith]

Hi @ilyasovivan,

I did not manage to reproduce the issue with the branch 1.7.7.x (next release PS1.7.7.0)
https://drive.google.com/file/d/1osWiN2CyUOr2T9L-YTnaO6wnuU3U2yRo/view
Since it is fixed, I will close the issue, feel free to open a new one when needed.

Thanks!

[ilyasovivan]

Hi @ilyasovivan,

I did not manage to reproduce the issue with the branch 1.7.7.x (next release PS1.7.7.0)
https://drive.google.com/file/d/1osWiN2CyUOr2T9L-YTnaO6wnuU3U2yRo/view
Since it is fixed, I will close the issue, feel free to open a new one when needed.

Thanks!

Oh no...
The creation and use of such addresses (погода@test.ru) is not permitted by the RFC 3696 standard for creating email addresses. Only the part of the address after the @ symbol, that is, the domain name of email server, can consist of cyrillic characters (because only this part can be converted to punycode).

Current state:
test@погода.рф - not valid but must be. (should work, because this address must be converted to punycode and must be valid. _test@погода.рф > [email protected]_)
погода@test.ru - valid but must not be.

Thank you for your tests and excuse me for my English (I use Google Translate prompts), apparently that's why you did not understand the essence of the issue

[ilyasovivan]

@khouloudbelguith, correct validation works only in browser validation now, like in this comment: https://github.com/PrestaShop/PrestaShop/issues/11588#issuecomment-444107509
but does not work with Prestashop validation (you checked it in your video using atribute 'novalidate').

[khouloudbelguith]

@ilyasovivan,

but does not work with Prestashop validation (you checked it in your video using the tag 'novalidate').

In branch 1.7.7.x, after adding novalidate atribute to<form>tag, I don't have any exception as described in my last screen record.

Thanks!

[khouloudbelguith]

Hi @ilyasovivan,

I will re-open the issue.

Thanks!

[khouloudbelguith]

Ping @colinegin what do you think?
I followed these steps with 1.7.7.x branch

test@погода.com - works good
test@погода.рф - Invalid format Error
погода@test.ru - With novalidate atribute in <form> tag for disable html5 validation (because we test only server validation, because in some modules it can used without html5 form) => there is no exception => fixed in the branch 1.7.7.x.

But according to this comment => there is no Prestashop validation.

Thanks!

[ilyasovivan]

test@погода.com - works good
test@погода.рф - Invalid format Error
погода@test.ru - With novalidate atribute in <form> tag for disable html5 validation (because we test only server validation, because in some modules it can used without html5 form) => there is no exception => fixed in the branch 1.7.7.x.

test@погода.com - it's valid now (it's right).
test@погода.рф - it's not valid now, "Invalid format Error" (it's not right, must be valid).
погода@test.ru - it's valid now (with novalidate atribute in <form> tag. And it's not right, because it must not be a valid address).

[colinegin]

@khouloudbelguith @ilyasovivan thanks for your feedbacks.
So if I understand we still have 2 different issues with address formats with cyrillic characters.
Do you know if these issues occurred before already ?

[khouloudbelguith]

@colinegin, in fact,
The issue reproduced is that there is no Prestashop validation with address formats with Cyrillic characters before @ for example погода@test.ru.
There is only browser validation now.
In the branch 1.7.7.x, if we remove the browser validation => there is no exception displayed comparing previous versions, but always there is no Prestashop validation because the customer with Cyrillic characters before @ is well-created which is incorrect.

Thanks!

[colinegin]

So I guess we should fix what's left to be done.
@PierreRambaud @NeOMakinG do you have an idea on how to fix it ?

[Com3t]

@ilyasovivan RFC 6531, SMTP Extension for Internationalized Email allows Cyrillic (and other non-ASCII) mailboxes.

An SMTP client that receives the SMTPUTF8 extension keyword in response to the EHLO command MAY transmit mailbox names within SMTP commands as internationalized strings in UTF-8 form. It MAY send a UTF-8 header [RFC6532] (which may also include mailbox names in UTF-8). It MAY transmit the domain parts of mailbox names within SMTP commands or the message header as A-labels or U-labels [RFC5890]. The presence of the SMTPUTF8 extension does not change the server-relaying behaviors described in RFC 5321.

The Wikipedia article on International email gives the following test cases:

The following are all valid international email addresses:

用户@例子.广告               (Chinese, Unicode)
अजय@डाटा.भारत               (Hindi, Unicode)
квіточка@пошта.укр          (Ukrainian, Unicode)
θσερ@εχαμπλε.ψομ            (Greek, Unicode)
Dörte@Sörensen.example.com  (German, Unicode)
коля@пример.рф              (Russian, Unicode)

[matks]

FYI on this issue we are also highly constrained by external systems: we depend on server config, we depend on dependency config (for example we use swiftmailer so we depend on what emails they accept/validate for the version we use), we depend on character encoding config ...

We try to do our best to workaround all these constraints but it's not easy 😅

[ilyasovivan]

@ilyasovivan RFC 6531, SMTP Extension for Internationalized Email allows Cyrillic (and other non-ASCII) mailboxes.

The Wikipedia article on International email gives the following test cases:

The following are all valid international email addresses:

用户@例子.广告               (Chinese, Unicode)
अजय@डाटा.भारत               (Hindi, Unicode)
квіточка@пошта.укр          (Ukrainian, Unicode)
θσερ@εχαμπλε.ψομ            (Greek, Unicode)
Dörte@Sörensen.example.com  (German, Unicode)
коля@пример.рф              (Russian, Unicode)

So, how many providers support this format? Very few. In the future there will be more. But at the moment this is not very common.
But I don’t mind that developers add support for this format to use it in the future.

[matks]

@ilyasovivan RFC 6531, SMTP Extension for Internationalized Email allows Cyrillic (and other non-ASCII) mailboxes.
The Wikipedia article on International email gives the following test cases:

The following are all valid international email addresses:

用户@例子.广告               (Chinese, Unicode)
अजय@डाटा.भारत               (Hindi, Unicode)
квіточка@пошта.укр          (Ukrainian, Unicode)
θσερ@εχαμπλε.ψομ            (Greek, Unicode)
Dörte@Sörensen.example.com  (German, Unicode)
коля@пример.рф              (Russian, Unicode)

So, how many providers support this format? Very few. In the future there will be more. But at the moment this is not very common.
But I don’t mind that developers add support for this format to use it in the future.

The numbers might be low but one of the core values of PrestaShop is to be usable worldwide. This is why we put so much energy into features such as Right-To-Left display, localization packs, large currencies management. The PrestaShop company was created with the following idea : "Allowing anybody in the world to open its shop online".

So if we can find a nice way* to support these formats, we'll try to do it 😊.

*Nice = not too complex (both for the code and the user), does not take weeks of work, does not create issues (in term of shop capabilities, code behaviors ...) ...

[Progi1984]

Fixed by #17561