Teleport: Reverse tunnels for individual nodes?

[zbuttram]

Are there any plans for support of individual nodes creating reverse tunnels to a proxy server without creating a new cluster? We have a case where we would like to have a single node setup at multiple different sites, but currently it looks like we would need to configure a cluster for each site just for a single node to use a reverse tunnel.

[klizhentas]

@zbuttram I think it's an interesting idea, I will defer to @kontsevoy to brainstorm this

[TuomasKiviaho]

The teleport Web UI search for instance becomes useless when there is close to 1:1 relation between clusters and nodes and number of clusters is high. It would be nice at least to have an opinion to group these clusters somehow together so that the dropdown menu becomes more manageable.

[kontsevoy]

Gents, this indeed would be pretty awesome, but it's a heavy change to how Teleport is currently designed. We definitely see the need for this and will keep this in mind.

[burdiyan]

We would also like to see something similar in Teleport. We have a use-case where some machines behind firewall need to be accessible from the Teleport Web UI by creating a reverse tunnel to a publicly accessible server. It seems like it is not easy to do currently with Teleport, without creating a separate cluster for behind-firewall machine. Am I right?

[kontsevoy]

@burdiyan you're right. the current advice is to start each behind-firewall node as proxy+server and they'll all become single-node clusters.

[burdiyan]

@kontsevoy Thanks! Shouldn't they also run auth role? As far as I can see you can only specify reverse tunnel connections for in auth part of the configuration.

On the other hand, is there any way to make a reverse tunnel, without specifying trusted clusters configuration on the "main" auth server? Maybe common CA cert?

[kontsevoy]

@burdiyan you're correct, they have to be auth too. BTW the upcoming 2.3 will make this process A LOT easier. Should be out in August.

[burdiyan]

As far as I understand, in Teleport 2.3 it is still required to create a separate cluster for behind-firewall nodes. Indeed it is more simple with resources but anyway somehow tough to work with.

I guess, what we are actually trying to achieve is similar to how Teleconsole work. Does Teleconsole act as a one node cluster when you join a session (or when you start a session?). It seems like both teleconsole "server" (the one who shares the session) and client (the one who connects to the session) are able to use the same remote "proxy" (the one running by you).

Can you describe briefly which roles in terms of Teleport each of the Teleconsole actors (server, client, proxy) are handling?

[kontsevoy]

@burdiyan yes, Teleconsole creates a single-node cluster. The latest approach of creating trusted clusters on the fly in Teleport you can achieve similar result.

[kontsevoy]

The importance of this ticket is now elevated a little, some customers see this as an additional security feature, see #982

[tomberek]

I have a use case where the nodes are behind firewalls or NAT. Having every single one run proxy, node, and auth seems like overkill. ( this also seems to use 100% of the CPU when run on an arm V7, I am still debugging this, will create another issue when I know more. )

I have also experimented with having my node just be a user of the central proxy. This gives them to anything on the cluster, but does not provide themselves to the cluster. This is all doable with normal ssh with a reverse tunnel dropping a socket. But I can’t seem to make it work when trying to use an openSSH client with teleport. ( other forms of using openSSH work well though, thank you) Is there an SSH library setting that prevents this, or a certificate restriction, or something else? Using the TSH client does not expose the -R capability.

[kontsevoy]

Having every single one run proxy, node, and auth seems like overkill

@tomberek I am surprised to hear you're having issues with ARM v7. I actually have a small cluster of 5 year old Raspberry Pis (IIRC they're v6) running at home and Teleport works really well there, if by "overkill" you mean resource usage.

These "services" aren't additional processes and can be configured to be quite lightweight. For example you don't need to run a proxy, just auth+node and I would recommend turning off session recording for auth, which makes the result just negligibly heavier than just a node running alone.

LMK how it goes, I like your use case!

[siepkes]

For an IoT scenario where you have a lot of devices with no public IP which incidentally require some remote debugging / troubleshooting this would be ideal.

EDIT: It would provide a more elegant solution to whats described here: https://blogs.wcode.org/2015/04/howto-ssh-to-your-iot-device-when-its-behind-a-firewall-or-on-a-3g-dongle/

[tomberek]

Yes, for now I've implemented reverse forwarding into a unix domain socket (rather than ports as suggested by the link in @siepkes 's comment). The one pain point here is that SSH does not expose as fine grained control on reverse forwarding as local forwarding - you can't restrict what ports or what paths (other than Chrooting) the sockets are made in.

[samouds]

Hello @kontsevoy

I trier to run auth/node only, but I did not see the new cluster in the web ui. I had to turn the proxy on to see it.

teleport config :

teleport:
  nodename: mynode
auth_service:
  enabled: yes
  session_recording: "off"
ssh_service:
  enabled: yes
proxy_service:
  enabled: no

cluster config :

kind: trusted_cluster
version: v2
metadata:
  name: x.x.x.x
spec:
  enabled: true
  token: "tokeeeeeeen"
  tunnel_addr: x.x.x.x:3024
  web_proxy_addr: x.x.x.x

And it uses all the CPU.

[kontsevoy]

@samouds we're aware, see #1886 thanks!

[cfoust]

Are there any plans to introduce this functionality? It seems like a pretty common use case for devices behind NAT. Having each node be its own cluster is rather cumbersome.

[kontsevoy]

@cfoust we're not closing this ticket, i.e. we're considering it. No question, it would be very useful. The trick is to implement it without making it hard to configure and without confusing regular Teleport users who use multi-node clusters.

[cfoust]

For what it's worth, this is a prerequisite for using Teleport in IOT/robotics contexts. That being said, I think that use case has a corollary of requiring a lot of effort to improve the resilience of Teleport's connections since internet access may be unreliable. That's something we've encountered. Just some food for thought -- I understand this would be a big shift in development priorities.

[burdiyan]

Maybe we could explain here what exactly needs to be changed in Teleport to support this feature? Also whether or not it can be implemented in a backward-compatible way. I guess having a bit more clear design decision, someone could submit a PR and help to make this feature a reality.

[burdiyan]

BTW, is it possible to set this up by running all Teleport services on each behind the firewall node, and connect them to the same remote DynamoDB or Etcd? Will they act as a single cluster?

[MrSaints]

I don't think it is necessarily a prerequisite for use in IOT / robotics context. It is definitely a UX problem in those cases, but I don't believe it was ever designed for those cases. That said, I'd argue that the biggest "blockers" for those cases are the resource overhead which comes with running an entire cluster on a single node, and resilience when it comes to network connectivity (though, I personally have not encountered this, it tends to recover from disconnections fairly quickly).

[cfoust]

It would be helpful to have better UX around exposing individual nodes to a remote cluster. Having each IoT device host its own "cluster" and then have that tunnel to another one is completely unscalable if you have hundreds of devices, in the sense that it's a pain to navigate on the internal web page. You're also forced to check the contents of each cluster to see whether the device is even up.

I doubt making this functionality better would would involve big architectural changes. Not only that, but it makes Teleport that more enticing to use for cases where you might have hundreds or thousands of devices that you want to give access to. The current model is just clunky is all.

[klizhentas]

We have discussed this use case and have a plan that will allow individual nodes to register using reverse tunnel channel, we will try to roll this out soon

[samouds]

@klizhentas great news. Do you have any idea when you can release it ?

[kontsevoy]

@samouds see above. The next release will have it.

[samouds]

@kontsevoy thanks ! Do you have any date for the next release ?

Thanks for all the great work !

[kontsevoy]

Well... it should be in master already, i.e. you can build from source and play with it. To enable, give nodes the proxy address instead of the auth server address, i.e. teleport start --auth-server=proxy.example.com

Code freeze will happen around June 1st, with an official release within few weeks after that.

[speeltronics]

Has anyone got this to work?

I have tried multiple configurations and all I can ever get to happen is the node will show up on the remote server device web panel and when I click to login, it logs in to the remote server device instead of the node device.

This command is doing the same thing: teleport start --auth-server=proxy.example.com

Also, the nodename doesn't seem to be used when I use a config file for this. It does change the name when using this: teleport start --nodename=mynode --auth-server=proxy.example.com

I compiled the master branch for both my remote server and the node device when this was pushed into it when russjones merged commit 6d1c16f into master 24 days ago

[kontsevoy]

@speeltronics The log messages currently aren't helpful. We're making this more user-friendly as I write this. Most likely, you didn't have certain ports open. See below.

If you're starting the node like this:

$ teleport start --auth-server=proxy.example.com

... it means that proxy.example.com must have 3 ports open:

1. Port 3080 for HTTP/TLS, i.e. the connection used to SSO, issue user certs, serve the web UI, etc.
2. Port 3023 for client SSH connections
3. Port 3024 for SSH tunnels (i.e. the facility used by this new feature)

More:
https://gravitational.com/teleport/docs/admin-guide/#ports

LMK if opening these 3 ports worked.

[speeltronics]

Yes those ports are all open. When the node shows up on the web interface it has a label with the correct hostname and a label with the correct os. My main server running the web interface is also there and it has the correct hostname label as well as os. The address for both of them shows 0.0.0.0:3022. The user names are the same on both machines and I log into the web interface using the same username. When I click the login as button for either of the nodes they both go to my main server runnning the web interface.

[speeltronics]

Here is my client config file:

#
# Client Teleport configuration file.
#
teleport:
  data_dir: /var/lib/teleport
  auth_token: myauthtoken
  auth_servers:
  - proxy.example.com
  connection_limits:
    max_connections: 1000
    max_users: 250
  log:
    output: stdout
    severity: INFO
#    output: stderr
#    severity: WARN
auth_service:
  enabled: "no"
  listen_addr: 0.0.0.0:3025
  tokens:
  - proxy,node:authtoken
  session_recording: "node"
ssh_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3022
  labels:
    db_role: master
    db_type: postgres
  commands:
  - name: hostname
    command: [/bin/hostname]
    period: 1m0s
  - name: arch
    command: [/bin/uname, -p]
    period: 1h0m0s
proxy_service:
  enabled: "no"
  listen_addr: 0.0.0.0:3023
  web_listen_addr: 0.0.0.0:3080
  tunnel_listen_addr: 0.0.0.0:3024

I start the client in a systemd service file with the command:
ExecStart=/usr/bin/teleport start --nodename=%I --config=/home/myuser/teleport/teleport.yaml --pid-file=/var/run/teleport.pid

Here is my main server config file:

#
# Server Teleport configuration file.
#
teleport:
  nodename: example.com
  data_dir: /var/lib/teleport
  pid_file: /var/run/teleport.pid
  auth_token: myauthtoken
  auth_servers:
  - 0.0.0.0:3025
  connection_limits:
    max_connections: 1000
    max_users: 250
  log:
    output: stdout
    severity: INFO
#    output: stderr
#    severity: ERROR
auth_service:
  enabled: "yes"
  cluster_name: example.com
  listen_addr: 0.0.0.0:3025
  tokens:
  - proxy,node:myauthtoken
  - trusted_cluster:truestedtoken
  session_recording: "off"
ssh_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3022
  labels:
    db_role: master
    db_type: postgres
  commands:
  - name: hostname
    command: [/bin/hostname]
    period: 1m0s
  - name: arch
    command: [/bin/uname, -p]
    period: 1h0m0s
proxy_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3023
  web_listen_addr: 0.0.0.0:3080
  tunnel_listen_addr: 0.0.0.0:3024
  public_addr: proxy.example.com:3080
  ssh_public_addr: proxy.example.com:3023
  https_key_file: /etc/letsencrypt/live/example.com/privkey.pem
  https_cert_file: /etc/letsencrypt/live/example.com/fullchain.pem

I start the server in a systemd service file with the command:
ExecStart=/usr/bin/teleport start --config=/home/myuser/teleport/teleport.yaml --pid-file=/var/run/teleport.pid --debug

[speeltronics]

That didn't get formatted like I thought it would using the insert code button for the config files

[speeltronics]

Here's what the web ui looks like when the node connects, but both buttons log into the example.com node.

[kontsevoy]

@speeltronics I would suggest removing listen_addr entirely because it overrides the default connection mechanism (you are effectively instructing the server to _always_ use 0.0.0.0 address to connect to those nodes, which of course resolves to 'localhost` from a server's point of view, i.e. it connects to itself)

EDIT: nevermind, this is apparently an earlier bug. see below.

[kontsevoy]

@speeltronics are you building from master? Our engineers are saying that this bug was indeed present during development (and it was present in one of the earlier alpha builds) but it should be gone by now.

[speeltronics]

I built from master the last time I tried it before I posted 5 days ago. Both my client and server have the following version.
Teleport v4.0.0-rc.2 git:v4.0.0-rc.2-0-g00a7ad9f go1.12.5

I commented out the line on the client: listen_addr: 0.0.0.0:3022

It still does the same thing and logs into the server.

Also, commented out that line on the server and tried it. Then tried commenting out the server proxy tunnel and listen addresses. Still does the same thing.

[kontsevoy]

@speeltronics Thanks for helping with troubleshooting. We're working on this as part of #2751 We'll be posting updates there (let's move the convo there).

Also, I would highly recommend trying the latest master (build by typing make docker). Looks like the bug you're experiencing was fixed in b0aa4a940dc248f839dc272001e91258d4ff3c