Zerotierone: Is there a way to totally bypass ZeroTier's servers?

[apognu]

Hi,

I'm trying to wrap my head around the way ZeroTier's addresses are resolved to peers. As far I understood it, networks and members are referenced on ZeroTier's servers for it to initiate the first connection.

I set up my own moon and controller, by following the doc, which, I believe, allows me to create my own networks outside of my.zerotier.com, and to have my own root servers for my clients to resolve my network.

That's actually what happens when I copy the signed moon file to my clients, they can connect to the network by communicating directory with my moon, and then controller without, as far as I know, needing the root servers.

The whole thing still works when I don't copy the moon file. I guess this works by going through the Planet's root servers, which then refers to my moon, which then refers to my controller. I didn't find anything in the documentation that explains how that works. Does my controller register its networks with the Planet, so my client can find it through the official root servers? Does my moon do it?

I would like to understand exactly what happens in this resolution scheme and if there is any way to prevent the root servers from learning about my networks? I know this would prevent global resolution without a manual reference to my moon, but in some implementation scenarios, I believe it would be preferred).

Don't hesitate to point it out if there is anything I did not understand in how ZeroTier works, or if I haven't been clean enough.

Thanks!

[adamierymenko]

This isn't easy right now but it can be done. You could rebuild it without the default 'planet' definition.

Better support for air-gapped operation is slated for the future. As it stands when you add your own roots it uses them alongside ours, not in place of them.

There is a rationale behind this. The goal here is zero-configuration operation everywhere and having a core root set enables this with minimal overhead. The roots don't really do anything other than introduce peers, cache public keys, and relay traffic when direct links are not possible.

[apognu]

I understand, and I agree the defaults should be set for zero-config (that's one feature that is remarkable in ZeroTier!). Glad to know the idea of detaching totally from the Planet is considered. I'll wait until then.

I presume my understanding of the resolution process when the client has no reference to the moon was right? Which is:

• my moon ID is extracted by the root servers from the network ID
• the location(s) of my moon(s) is resolved from the root server's cache
• my custom roots know which controller manages the network
• profit?

Thanks for your answer.

[adamierymenko]

We're thinking a lot about how to enable more independence without breaking zero-conf too badly or introducing other headaches. We'd ideally like it to be pretty transparent. It's not an easy problem.

[StrikerTwo]

+1 for an option to ONLY use the defined moons.
A command line or config option to disable the default planet hosts would be fine for me. People who use this probably know what they are doing ;)

[apognu]

From what I've seen, the only file containing the planet definition is /var/lib/zerotier-one/planet which is generated on first run. It can also be generated manually from mkworld.cpp.

I replaced lines 87-116 with parameters from my moon and distributed the file to my nodes and all went smoothly, they only connected to my moons (which were the new planet) to establish connection.

It's still a bit of a pain to configure (generate a binary planet file from code and distribute it to all nodes), it might be easier, as @StrikerTwo said, to provide a CLI option to use a custom planet definition from a JSON file on disk (even if it generates the binary planet definition on first run).

The second pain point I see here is there is not way to override the planet definition for ZeroTier One mobile apps. So I guess if your moon/controller does not use Alice and Bob to register itself, you'll have no way to connect your mobile app to your network (since it'll try and find it through the main planet).

[adamierymenko]

This is both a pain point and a feature. As I said we're thinking about ways to make this easier without sacrificing the feature part.

A major thing we are trying to do here is to do away with manual network configuration as much as possible. We want networks that are like chat rooms.

"Oh, join my network it's (network ID.)"

"Ok."

If you don't have some common basis then you need additional information beyond the network ID. We don't want to require that.

[apognu]

I do understand, and I would never suggest removing that feature or make it non-default. The beauty of this project rests on its zero-configuration capabilities.

Just giving some hints at what you could do to provide an optional way of overriding the planet's definition when you decide to take this on. ;)

[StrikerTwo]

This is both a pain point and a feature. As I said we're thinking about ways to make this easier without sacrificing the feature part.

I'm not asking you to sacrifice a feature. I'm asking you to add one ;) Like I said, the majority of users are fine with it the way it is. Just adding an option for the 0.01% of users who would like something different won't hurt.

That said, if I had to design this feature, I would probably do something like

• Network X is registered at ZeroTier with moons Y and Z and a flag "use only those moons" (maybe an extension of the existing switch public / private)
• Host wants to join network X, contacts Zerotier root servers, learns about moons and flag
• Host contacts moons and waits for authorization
• Moon authorizes host
• Host never asks ZeroTier root servers again until it wants to join another network

Easy configuration, completely transparent to the user, but minimal metadata leaking outside my control.

Edit: I do realize this is not exactly what the OP wanted, but maybe a step in his direction while still maintaining the principle of ZeroTier ;)

[apognu]

In some scenario, even having a single request home which would leak the client ID and IP address, the moons IDs and IP addresses would be too much.

In my mind, the easiest solution would be to be able to override the planet's definition through a configuration file (still optional).

[StrikerTwo]

Agreed. Personally, I'd be happy with a "don't use the planet" option. Just wanted to suggest ideas for more decentralization as well.

[andrew-d]

Worth noting that this is something I'm interested in both from a decentralization perspective ("what if ZeroTier the company goes away?") but also a security perspective ("what if someone compromises the ZeroTier root servers?"). It's likely true that if you can compromise the ZeroTier servers you can also do so to any roots hosted by others, but it's an additional layer of defense if you can host literally everything yourself.

If it would be helpful, I'm also happy to contribute code towards achieving this, but I'm new-ish to this project so I'm not sure what would be helpful (and also, the mobile apps don't appear to be open-source?). Feel free to let me know!

[apognu]

I forked the client (not yet on GitHub, purely as a proof of concept) so it picks up a custom planet if and only if /etc/zerotier-one/planet exists (this file has to be generated beforehand but can then be distributed through CM).

As far as I know, once this is enabled, the client does not have a single reference to ZTO's root servers. But all properly configured clients can join the networks managed by the controllers referenced by the moon. You cannot join global networks anymore, but that's not what we are looking for, here.

[fauust]

Hi @apognu,
could I look at your fork? I am also interested in finding a way to totally bypass ZeroTier's servers.
Should I just rebuild the zerotier-one binary with my specific options in the mkworld.cpp file?

[adamierymenko]

We will be making this easier in the 1.4 tree, but right now this requires a custom 'planet' file or a fork. Closing since it's already on the road map.

[jstasiak]

The 1.4 tree with the changes mentioned is not available yet, right? I could really use this, as I have two endpoints behind NAT and they have to communicate over a relay – as long as the connection goes through my moon all is relatively well, but when zerotier thinks (incorrectly), that one of the roots is faster than my moon and starts using it instead of my moon everything slows down to a crawl.

PS. It may not be the best place to say this, but thank you for the project.

[Fastidious]

1.4 is out, is there a way to run planets now? If so, how?

[ziyifeihu]

Will the project continue?

[abclution]

I have a simple, somewhat related question regarding moons and planets.

I have several multi-gateway locations, one with a fast (4g 15dl/5up) and slow (adsl 15dl/0.8up), I run several other self run vpns and I initiate the VPNS by forcing them through the fastest gateway availiable. But in order to do that, I need to know the IP/DNS of the destination servers to setup my route mangling rules.

I want to make sure my ZT clients all communicate over the 4g
If I run my own controller, this would be easy as I would know the controller + planet/moons ip.

But using the free tier of the ZT service, where is the list of helper planet/moons IPs or DNS so I can adjust my firewall rules? Is there a single reference source?

[StrikerTwo]

The main VPN traffic with Zerotier is peer to peer. Knowing the planet/moon IPs and setting extra routes for them will not help you (unless you force your nodes to relay through them).
Running your own controller won't change things (again, unless you force every node to relay).

You can configure all nodes to use a specific port (UDP/9993 is default) and use policy based routing to send those packets over the fast links.

[JocPelletier]

The main VPN traffic with Zerotier is peer to peer. Knowing the planet/moon IPs and setting extra routes for them will not help you (unless you force your nodes to relay through them).
Running your own controller won't change things (again, unless you force every node to relay).

You can configure all nodes to use a specific port (UDP/9993 is default) and use policy based routing to send those packets over the fast links.

Do you have an example on how I can configure/achieve that? Thanks a lot

[StrikerTwo]

Depends on your operating system and/or router.
I use OPNsense virtual machines on every site and can just set firewall rules for UDP/9993 and force them to use a specific gateway.

I have used similar rules (again, with OPNsense) to force already encrypted traffic over fast, but unencrypted links and push the rest through the VPN.

[catalystsitaly]

Since all default roots of zerotier very slow in China, I am expecting an easy way to customize own root very much.
Great project thanks.

[qiuwei]

Since all default roots of zerotier very slow in China, I am expecting an easy way to customize own root very much.
Great project thanks.

For the speed issue, you can set up your own moon, but still use the official root.