Jormungandr: Dynamic public IP handling

[gufmar]

Some countries (eg Germany) have rules for ISPs to disconnect and change public assigned IP address once a day for broadband internet connections

cron jobs can watch the public IP

The feature request is to update (inject) the new public IP via jcli without restarting the node.

[mmahut]

I think this should be system's operator duty to mediate such as specific case and should not build into the software directly.

[mzabaluev]

By design (or at least by intent), restarting a node should not be a very dramatic event, both the node and the network should continue with minimal disruption. When the public IP changes, this means there is a time window when the node is unavailable, and it needs to reconnect to some peers and gossip about its new address to continue participation, so the process might as well restart as a systemd unit or whichever way the service is deployed that can be also used to provide it with an updated network configuration.

[gufmar]

@mmahut to clarify: Yes definitively a sysop's duty to detect a dynamic changed IP (for those who want to run it in such an environment)
Based on such an event they need to decide if and how to react.
Current only possible option: rewrite the config and restart the node.

The feature request is to have a smooth, instant and smart way to update (inject) what effectively changed (the public gossiping IP) without restarting the whole node.

Today a restart is fast but will take more and more time with a growing DB. And with the current implementation, a restarted node has no idea and also no fresh (instant) information about the current epochs leader schedules.
So I wonder if it's possible - without any noteworthy node-internal complexity - to inject the new setting through jcli, similar to what is possible with jcli rest v0 leaders post -f?

[mmahut]

@mmahut to clarify: Yes definitively a sysop's duty to detect a dynamic changed IP (for those who want to run it in such an environment)
Based on such an event they need to decide if and how to react.
Current only possible option: rewrite the config and restart the node.

Yes, I would suggest using systemd and BindsTo using sys-subsystem-net-devices, whenever the IP changes, systemd would rewrite the config line and restart the service.

Today a restart is fast but will take more and more time with a growing DB. And with the current implementation, a restarted node has no idea and also no fresh (instant) information about the current epochs leader schedules.

Good point, so I guess this issue is about implementing some kind of a configuration reload without restarting the entire node?

@mzabaluev has a valid point, that given the IP changed, your node is already down for the network itself.

[mzabaluev]

Today a restart is fast but will take more and more time with a growing DB.

With a good journaling DB engine, it does not have to be so. sqlite that we use now is well tested (as opposed to how we use it), and in the future we may switch to something more lightweight, like TiKV, as a backend.

[rickymac68]

I agree with @gufmar we should not have to mess with public IPs and restarting nodes. I have run other crypto full nodes and never had to mess with my dynamic public IP which changes monthly. Automate as much as possible.

[mzabaluev]

It should be possible to run a node without a public IP; in fact, this may be preferred for nodes behind firewalls/NATs. The only functionality disabled by not having a public IP address is being a trusted peer and having subscribers that connect to the node rather than the other way around. So for users without a statically mapped public IP address, a non-public node may be the preferred configuration until we implement NAT traversal automation and are sure that it works reliably.

Having said that, the network's strength depends on the number of publicly reachable nodes.

[mark-stopka]

Already discussed in #839; automatic IP discovery, and NAT traversal should be implemented within the node itself. It's beneficial for the network to have many relay nodes, not only stake-pool nodes, it may very well be expected for those to run behind NAT with a UPnP enabled,...

@mmahut, the systemd workaround is like scratching left ear with a right arm if not worse; it's commonplace for p2p node to handle dynamic network topology changes (including a change of your own IP address).

[khwerhahn]

From a point of view of making it as easy as possible for less-techie personas to spin up a node and grow the network, it would definitely a beneficial feature. The less configuration there is for the average user, the better it is. If the next generation of Blockchain (Cardano) still has the same level of bad usability as the older Blockchains out there, it might not deter the developer but will deter the user side. It would fall into the "Delighter Feature" area from the Kano Model.

[JamesRobertKelley]

Will a node lose the slots won in the leader lottery if it is restarted?
Since uptime is so important we also need to consider server maintenance for security. The ability to change IP addresses could allow me to shutdown one node and fire up another with the same certificate to take over for the node in maintenance without losing slots. I don't know about those processes yet, so I could be missing something.

Another Issue is to add DNS resolution to the trusted nodes list. This could be part of a solution to this problem. When the IP address changes the node's DNS entry could be updated.

[mzabaluev]

When the IP address changes the node's DNS entry could be updated.

That's even more prone to be applied with delay as DNS entries are usually cached.

[JamesRobertKelley]

When the IP address changes the node's DNS entry could be updated.

That's even more prone to be applied with delay as DNS entries are usually cached.

You can set the Time-to-Live on the DNS entry. I often lower it to 5 minutes and then 5 seconds before a server migration, which also offers a quick way to reverse the migration.

Round-robin DNS entries can also be used by setting two IP addresses with the same name. It is often used for load-balancing, but a client can automatically try the 2nd IP address if the first does not respond.

With a little more work Jormungandr could issue a "302 redirect" if it is not the current master for a certificate. The client would use DNS round-robin to pick up an IP, and if the wrong node was contacted a redirect message would point the client to the correct node. If the client does not receive a response it can try the other IP in the round-robin.

[mark-stopka]

DNS resolution typically happens at startup, besides poldercast advertises
IP addresses / port pairs, not FQDNs, DNS is not a way forward for this one.

Besides that, the issue you're describing is non-issue because the Cardano
blockchain does not care from what node the valid block came so you can
migrate the slot leader node to a different IP and not loose your slot
production right during maintenance.

On Tue, Nov 26, 2019, 09:23 James Kelley [e-mail] wrote:

When the IP address changes the node's DNS entry could be updated.

That's even more prone to be applied with delay as DNS entries are usually
cached.

You can set the Time-to-Live on the DNS entry. I often lower it to 5
minutes and then 5 seconds before a server migration, which also offers a
quick way to reverse the migration.

Round-robin DNS entries can also be used by setting two IP addresses with
the same name. It is often used for load-balancing, but a client can
automatically try the 2nd IP address if the first does not respond.

With a little more work Jormungandr could issue a "302 redirect" if it is
not the current master for a certificate. The client would use DNS
round-robin to pick up an IP, and if the wrong node was contacted a
redirect message would point the client to the correct node. If the client
does not receive a response it can try the other IP in the round-robin.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/input-output-hk/jormungandr/issues/1072?email_source=notifications&email_token=AEKMP3EN5ZXWGVA7BY2MYC3QVTMGLA5CNFSM4JIZVPWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFFEQ7Q#issuecomment-558516350,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEKMP3CHWGB2DW7MIWN5NKLQVTMGLANCNFSM4JIZVPWA
.

[TheRealMrBojangles]

On a wireless LTE or 5G network you have 2 or 3 layers of dynamic IP's, at cell tower level, and at ISP level. These changes are quite frequent. You could be restarting the node every 2 min if you live in a high traffic area, with lots of mobile devices coming and going all the time...

[mark-stopka]

@TheRealMrBojangles my LTE provider offers dynamic globally routable IPv4 that changes only upon link-reestablishment free of charge or fully static globally routable IPv4 for 4 EUR / month surcharge.

Don't extrapolate; every network is different. In networks behind CG-NAT you hardly going to configure any TCP port forwarding; which is also a situation Jormungandr should be able to deal with using other Jormungandr nodes as reflectors.

[TheRealMrBojangles]

@mark-stopka here in South Africa ISP's don't issue static Public IP on wireless networks only offered on fixed line fibre networks at an additional charge. Most areas don't have fibre connectivity. The whole of the African continent is pretty much on wireless. Our CBD runs on a copper network still, with fibre as a backup.

[bitnado]

is there any documentation on what happens when a jormungandr slot leader node re-connects to the network with a different ip address? is there any risk of being quarantined?