Triplea: [Mitigated] Prod DNS Not resolving

[DanVanAtta]

dan@dan-desk:~$ nslookup lobby.triplea-game.org
Server:     127.0.0.53
Address:    127.0.0.53#53

** server can't find lobby.triplea-game.org: SERVFAIL


dan@dan-desk:~$ dig lobby.triplea-game.org

; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> lobby.triplea-game.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 21284
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;lobby.triplea-game.org.        IN  A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Fri Jun 14 16:34:30 PDT 2019
;; MSG SIZE  rcvd: 51

[DanVanAtta]

Uptime robot shows lobby as up, and is using the DNS address. I wonder what the secret sauce is for it.

[DanVanAtta]

https://forums.triplea-game.org/ is also not resolving.

Looks like the NGINX redirect in front of the forums is re-writing to forums.triplea-game.org when accessing that machine via IP address:45.33.80.67, so there does not seem to be a work-around for forums currently.

[DanVanAtta]

Just saw a user join the lobby, located in USA . So this is not a global DNS problem, regional.

[DanVanAtta]

Switch to use IP address is deployed, can access lobby again.
Keeping open as forums.triplea-game.org is still dead (regional problem)

[prastle]

Ya suffering same problem on any triple-game.org
lobby/forum/or downloads

[prastle]

Interesting that my cell works to join forum = rogers but my comp doesn't also = rogers but diff isp

[DanVanAtta]

A gap in escalation process here is having phone numbers to contact people.
@RoiEXLab @ron-murhammer Please add your numbers to the new emergency contact section in the Triplea secrets document. @prastle FYI

[prastle]

Used Gitter server but ya ill add np

[DanVanAtta]

@ssoloff would be great have you as an emergency contact as well if you're willing.

[prastle]

It's working again for my comp now

[DanVanAtta]

FWIW, I verified that the empty error message box is fixed in 1.10:

[prastle]

Interesting that the forum still fails to find DNS

[ron-murhammer]

Everything is fine for me. It was a bit spotty earlier today though.

[DanVanAtta]

Looks like everything is resolving once again here.

dan@dan-desk:~$ nslookup forums.triplea-game.org
Server:     127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:   forums.triplea-game.org
Address: 45.33.80.67
Name:   forums.triplea-game.org
Address: 2600:3c03::f03c:91ff:fee7:fcb6

Take-aways:
1) Escalation process could have used refinement, I think we'll have improved this by adding our contact number to the administrator secret document.
2) Likely should keep using IP address in the lobby properties file just so we can cut out the DNS dependency (already done in PR)

[RoiEXLab]

@DanVanAtta yeah, had this problem yesterday as well with the forum, (I verified it was a DNS problem) but since my ISP had the same problem (even for PayPal.com and similar big services) I didn't think much of it.
I think keeping DNS is important to be able to have the optional ipv6 support, however having an hardcoded ip fallback might be a good option.

Note however that hardcoded IPs don't work with TLS encryption, because certificates (with let's encrypt or other free options at least) can't be issued to IPs directly, just to hostnames. Thus having a different nginx conf for the forum wouldn't have solved anything, users would have gotten a big yellow warning instead of the site.

[DanVanAtta]

Good points. Does lack of IPv6 support cause a material impact to TripleA, or just theoretical?

Problems with certs I see as a problem for forums and will become one for http server.

I wonder if the thing to do is to have clients cache IP address and use that as a fallback.a

[RoiEXLab]

Well AFAIK there are some ISPs out there that offer IPv6 only plans, because they don't have enough IPv4 addresses, so there's definitely a real downside.
I just don't know the exact numbers

[DanVanAtta]

We do have a mechanism to cache the lobby address fetched from properties:
https://github.com/triplea-game/triplea/blob/master/game-core/src/main/java/games/strategy/engine/lobby/client/login/LobbyServerPropertiesFetcher.java#L107

If github.com did not resolve, that hopefully would have kicked in. It's interesting that the actual host address, needing the resolved IP address needs to also be cached. It seems like that level of caching could have provided a fallback to avoid this problem.

[DanVanAtta]

Created a follow-up task for the caching of resolved host address: https://github.com/triplea-game/triplea/issues/4907

I think that is the only outstanding post-mortem item, resolving this issue now.