Triplea: MARTI diceserver (at least sometimes) fails to validate email-addresses

[panther2]

As reported on axisandallies.org:

https://www.axisandallies.org/forums/index.php?topic=40884

Here we have two cases where MARTI answered

after the users clicked the validation link provided by the MARTI-mail after registration.

We have one hotmail- and one gmail-account involved. I was able to reproduce the issue with one of the affected addresses/links (not having the other one).

Personally I have tested registration (using other mail-addresses) without issues, so it does not appear to be a general issue.

[RoiEXLab]

Weird. This really should not be the case, and since this seems to be a very recent problem I don't know the cause either. I haven't been messing with the server config lately (other than https, which has been reverted anyways).

Might increase the priority of https://github.com/triplea-game/dice-server-js/issues/1 further...

[ssoloff]

The only thing that can cause that error message is if there is no row in the MARTI pending_validations table with the specified email and validation key:

https://github.com/triplea-game/dice-server/blob/master/dice/validate.php#L15-L31

It would be interesting if someone could run a query against this table for the affected users' emails to see what validation keys are present. Then compare those to what the users received. Just in case there's some weird URL encoding issue happening somewhere in the pipeline.

One thing I noticed is that the register.php page URL encodes the user's email, but the validate.php page does not URL decode it (at least I don't see any explicit decoding; maybe it happens automatically as part of another function). If the user's email happened to contain some reserved character that was altered by URL encoding, the email used by validate.php in the database lookup would be different, which would cause the query to return no records and produce the reported error message. I'll see if I can reproduce this with a local MARTI server.

[ssoloff]

I'll see if I can reproduce this with a local MARTI server.

It looks like PHP itself is URL decoding the values, so that theory, in general, is probably invalid.

However, there still may be something special about the user's email that could trigger a difference to what's stored in the pending_validations table. Have the users disclosed their emails? Maybe they can be shared via Gitter?

[RoiEXLab]

@ssoloff I sent you the relevant SQL data via gitter

[ssoloff]

Got it. Thanks, @RoiEXLab. Nothing looks suspicious (although there appears to be one fat-finger with a gmail.comg domain, but I'm assuming the user realized that and subsequently submitted the correct email).

I was able to reproduce the issue with one of the affected addresses/links (not having the other one).

The next thing to verify is the actual URL received by one of the users reporting the problem against the content of the pending_validations table. @panther2, would it be ok to forward the validation email for the account you could reproduce the issue on? Can we set up a private Gitter room for you, me and @RoiEXLab so we can easily share this information while we're debugging without disclosing it publicly?

[panther2]

Good idea, @ssoloff

[panther2]

So, I have created the room Issue2603 and added you two @RoiEXLab and @ssoloff .

Please take a look, thank you.

[ssoloff]

Per Gitter, @RoiEXLab confirmed that at least one of the users in question has no record in the pending_validations table.

The code in register.php responsible for inserting/updating records in the pending_validations table does not check if the insert/update was successful. Therefore, it's possible that the user will receive a validation email even if no record is added to the pending_validations table. In that case, clicking the validation link will have no chance at succeeding.

We should probably add appropriate error handling to register.php for the insert/update. It sounds like at least one of the affected users was able to reproduce this consistently. Once the error handling code is in place, we should ask them to try again and see what error the DB is raising.

[ssoloff]

Per Gitter, @RoiEXLab also confirmed the user in question has no record in the dice_emails table. So that pretty much rules out that the validation failures are due to the user somehow already being validated.

At this point, I think our leading theory is that there was a DB error inserting the record into the pending_validations table, but the validation email was sent out anyway.

[panther2]

@ssoloff @RoiEXLab

I have just added a new case to Issue2603.

[panther2]

... and another one today.

[ron-murhammer]

@RoiEXLab @ssoloff Are we able to see any kind of logs for MARTI?

[ssoloff]

@ron-murhammer I added you to the Gitter room where we've been discussing this out-of-band.

[ssoloff]

Per Gitter, a user reported that they had successfully registered in the past but are no longer receiving dice roll emails. Examining the syslogs, it was observed that Postfix refused to deliver mails to the user's MX because the MX did not support TLS. We determined the root cause was due to a recent Postfix configuration change (approximately 30 days ago) whereby TLS is required between nodes. That explains why the user received their initial registration verification email but was no longer receiving dice roll emails.

It was decided to not disable the TLS requirement at this time but to suggest to the user that they use a different mail service that supports TLS. For reference, the mail service that does not support TLS is charter.net.

NB: It does not appear this has anything to do with the original problem described in this issue. It is a separate problem that happened to arise while investigating this issue.

[panther2]

It was decided ... to suggest to the user that they use a different mail service that supports TLS.

That has been done, but using the new/alternative email address the user ran into the #2603 issue again.

[ssoloff]

I have a pretty good theory why the DB inserts are failing during registration. In addition to the registrant's email, validation key, and timestamp, we also store their IP address. We use the function ip2long() to convert the IP address as a string to an integer for storage in the pending_validations table. Unfortunately, ip2long() only handles IPv4 address. If it is passed an IPv6 address, it returns FALSE. When you try to bind FALSE to the SQL INSERT or UPDATE statement we're using, the insert or update will fail, but since we don't do any error detection here, the validation email is still sent.

I tested this locally and was able to reproduce the behavior the affected users have reported (i.e. receive the validation email but no record exists in the pending_validations table).

I verified that the nginx configuration allows both IPv4 and IPv6 connections. I also scanned the nginx access logs and saw at least one client using IPv6. From that I conclude that the above theory is at least possible.

In the long term, we need to decide how to handle IPv6 addresses. I'm not sure why the pending_validations table does not simply store the IP address as a string, since it doesn't appear to be used for anything. Storing it as a string would solve the issue because no conversion is required.

In the short term, we should ask one of the affected users if they can try to register from an IPv4 host. This may be a bit tricky for them because it could involve changing router configuration or possibly some other NAT issues outside of their control. @panther2 Did you say you had tried to submit a registration request on behalf of one of the affected users? I would assume that worked since you said you have never had problems registering from your machine previously. Please advise if otherwise.

[RoiEXLab]

@ssoloff I think your theory is pretty likely to be true.
Ip2long is not able to handle ipv6 according to the docs
So therefore the email can't be stored in the DB.
Because of the bad design of the PHP language, ip2long doesn't throw an exception if the given argument is invalid, but simply returns false instead.
False can't be stored as an 10byte Integer, and so the insertion fails.

I can't find a use case for the ip address either, so the most obvious solution would be to just not store it, and to change the schema accordingly?

[RoiEXLab]

And looking at the PHP code of register.php again:
For the latter insertions in the table, which are actually supposed to store the email the error messages are swallowed which means not explicitly catched/logged and so the user can't see any error message.

[ssoloff]

I can't find a use case for the ip address either, so the most obvious solution would be to just not store it, and to change the schema accordingly?

Yes, this is probably the right solution.

For the latter insertions in the table, which are actually supposed to store the email the error messages are swallowed which means not explicitly catched/logged and so the user can't see any error message.

Actually, I was a bit surprised when I added some DB error handling code to my local MARTI. The calls to prepare(), bind_param(), and execute() are _not_ failing when $IP is FALSE instead of an integer. I would have expected at least one of those (most likely execute()) to return FALSE, but that wasn't what I observed during my tests. The only way I could detect the failure to insert the record with an IPv6 address was by checking if $affected_rows is not equal to one. That seems a bit obtuse.

[panther2]

Did you say you had tried to submit a registration request on behalf of one of the affected users? I would assume that worked since you said you have never had problems registering from your machine previously. Please advise if otherwise.

@ssoloff It was a bit different. The user registered his email address and immediately forwarded me the confirmation mail. I inspected the mail and was able to reproduce this issue klicking the link then. I never had access to the user's mail account.
Indeed, personally I have never experienced any problems registering mail addresses from different mail providers using my machine(s). (Also see Gitter please.)

[ssoloff]

Thanks for the information, @panther2. Then just repeating what was said in Gitter: you should be able to submit a registration request for the user from your machine (which apparently only has a public IPv4 address). The user will once again receive the validation email, but this time when they click on the link, the registration should complete successfully.

I don't believe anything related to sending dice emails involves storing the user's IP address, so having only an IPv6 address should not stop the user from using MARTI after registration is complete.

In the meantime, we're still working on a process for updating MARTI in a reproducible manner until the new dice server code is ready for production. Hopefully we'll get this done soon so you won't be called upon to register other IPv6 users in the future. :smile:

[panther2]

Thanks a lot, indeed my ISP always distributes public IPv4 addresses.
The affected user and me will go through the procedure in about 6 hours.
I will report here.

[panther2]

@RoiEXLab @ssoloff
I can now confirm that the discussed workaround has worked. I entered the user's mail-address into the diceserver's form. The user was then able to successfully finish the registration by clicking the validation link. Thank you both for identifying the problem and working on a solution. :smile:

[davelarson79]

I am experiencing a similar issue. My operating system is Windows 10 I downloaded the 64bit version of the game. My plan was to try some PBEM games but I can't get my email registered with MARTI. I keep getting a "Could not verify the data. Please check the link you have received in your email." I have tried also registering from my cell phone as well as using both a @comcast.net and @gmail.com addresses. Any help getting this registered would be appreciated.

[panther2]

@davelarson79
Are you on www.axisandallies.org/forums/ or https://forums.triplea-game.org ?
If yes what is your username there? I will contact you then...

[ssoloff]

I have tried also registering from my cell phone

Another workaround may be to use a cell phone, but switch to mobile data (i.e. 3G, 4G etc.) instead of wi-fi. That may assign a public IPv4 address to the phone instead of the phone being behind the router's IPv6 address on the wi-fi network. That's at least been my experience with my carrier.

[ssoloff]

Note that, while the fix for this issue has been merged, it hasn't been deployed to production yet. I'll try to get to that tomorrow and will post back here once that's done.

[ssoloff]

@panther2 I'm unable to test the fix for this issue on the Internet-facing server because I don't have access to an ISP that will assign me a public IPv6 address. Could you please reach out to the player who originally had this issue and ask them to register on the staging server (http://dice-staging.tripleawarclub.org/)? They should no longer receive an error when they click the email validation link.

I'm going to go ahead and promote this fix into production in a few hours. I'm fairly confident that it will fix the issue, so I'm not going to wait for the IPv6 player's confirmation. But that independent confirmation will still be helpful to prove that we really fixed the root cause of that player's registration failures.

[RoiEXLab]

@ssoloff I could do the testing later today, my ISP does provide both, IPv6 and IPv4

[ssoloff]

Thanks, @RoiEXLab. However, note one thing... I ran into an issue testing this locally because my host has both IPv4 and IPv6 enabled. Basically, MARTI was always seeing my IPv4 address even when I forced the connection to the IPv6 endpoint. The only way I could get MARTI to see my IPv6 address is by disabling IPv4.

What was weird is even when I disabled MARTI's IPv4 endpoint (in Nginx, not at the OS level in the Docker container), the code was still seeing my remote address as IPv4. Not sure if I'm missing something obvious about IPv6 networking. Maybe there's some kind of 6-to-4-to-6 tunneling going on?

I read someplace that many networking stacks auto-choose between IPv4 and IPv6 on a hybrid network depending on which protocol the stack thinks will perform better at that time. I couldn't find any way to disable this optimization except by disabling IPv4 outright.

[ssoloff]

@panther2 @RoiEXLab Nevermind, I found an alternate host from which I can connect to MARTI via IPv6. I verified that I could reproduce the original issue in production, but registration was successful in staging. So it looks like we're good. I'll re-verify production once I've promoted staging in less than an hour.

[RoiEXLab]

@ssoloff Just when I was able to succesfully unregister myself from the production MARTI ^^
However there's another Issue that's bugging me...
Because of the tripleawarclub.org redirects to the new forum, the browser is being redirected when you accidentally use HTTPS to connect to dice-staging.tripleawarclub.org, so point 1 of https://github.com/triplea-game/dice-server/issues/18 should get a higher priority

[ssoloff]

@RoiEXLab Agreed. I'm going to try to complete that task this weekend.

[ssoloff]

This fix has been deployed into production. I confirmed I could successfully register using https://dice.tripleawarclub.org via IPv6 (IPv6 was confirmed by viewing my connection entry in _/var/log/nginx/dice.tripleawarclub.org-access.log_ and observing an IPv6 address as the source address).

For my reference, this is the cURL command I used to test registration:

curl -6 -X POST -d 'email=steven.soloff%[email protected]' 'https://dice.tripleawarclub.org/register.php'

[RoiEXLab]

@ssoloff Can confirm as well ^^