Docker-gitlab: Sharing SSH port between host and the container

[djelenc]

What would be a good way of setting up a system where the host and the container could share port 22? Is that even possible?

I'm thinking of a proxy that would allow certain users to log into the host, while remaining ssh connection attempts would be forwarded to the ssh daemon inside the container. (Related to #33, I guess.)

[adrianmalacoda]

This is what I do:

• create git user on host
• as git run a daemon which listens to key_create and key_destroy events from gitlab, you'd need to set up a webhook in gitlab for this.
• this daemon will pull the authorized_keys from git@container user whenever such an event is received, this means that git@host will need to be in git@container's authorized_keys.
• a script is placed in gitlab-shell/bin/gitlab-shell which forwards all sessions on git@host to git@container

Implementation is here: https://gitlab.monarch-pass.net/monarch-pass/gitlab-ssh-proxy

some notes:

• See also this question https://serverfault.com/questions/569236/username-based-ssh-proxy. As the top answer there notes, having sshd itself be the proxy is probably infeasible since the ssh connection is encrypted and you can't get at the username with a proxy in the middle. Some of the answers suggest using ForceCommand + nc which may be sufficient for our use case.
• instead of grabbing the authorized_keys from the container using ssh, the gitlab-listener (daemon) could probably just insert or delete the key from authorized_keys given that gitlab provides the key in the webhook event payload. This would obviate the need for git@container to need to know git@host's key.

[r2evans]

@djelenc, you suggest writing a proxy that intercepts all SSH connection attempts, determines if the connection is meant for gitlab or not, and then forwarding the connection to the "real" sshd? Though novel, I suggest this just opens an injection avenue, whereas instead of just SSH security channels, now I need to be worried about this ssh-proxy for security holes. (Suggestion: don't intervene on SSH channels.)

I propose an alternative solution would be to use restricted commands in the non-shell-users' ~/.ssh/authorized_keys file to forward the connection to the gitlab's sshd. This would benefit from (but not strictly require) agent-forwarding. This assumes that the "real" sshd is on the main port (22), and the gitlab sshd is configured on a non-standard port.

A start would be to use something like https://superuser.com/a/343675/402193, but changing ./gitserve to something like exec ssh -p otherport localhost git-shell or something like that.

[r2evans]

I just tested this locally, and it worked.

Assuming a topology similar to:

                          . . .+ +..+  - . .
                     -. .%.+..+%m..m.-%---- -.
                    --+#m*+m%+#%%##+##*#m+%--+.
+----------+        +#*####################%#.%-
|          |      .m ###########-#=##-######m#%.
| client   |======= +##%#####           ###%*#-
+----------+       .--+####   internet   ###*m+.
                   -.+-+#####          ###%%#  .
                     -- mm#############m+m*%-+
                      - .+#+#*##+%++%#--.m.%.
                       .  .+-.-m. .m .++...
                         = .+.
                         =
                        =
                   -----=---------
              ----/     =         \----
          ---/          SSH(22)        \---
        -/         ===='                   \-
       /       nc =                          \
      /           =   +---------------+       \
     /             =  |               |        \
     |              === SSH(11122)    |        |
     \                |               |        /
      \               |               |       /
       \              |  [gitlabhost] |      /
        -\            +---------------+    /-
          ---\                         /---
              ----\   [mainhost]  /----
                   ---------------

This assumes that a user logged in to mainhost can do git clone ssh://mainhost:11122/user/repo.git and the repo on gitlabhost is cloned. (I say this only to say that gitlabhost must be reachable on mainhost through port 11122. Whether port 11122 is reachable outside of mainhost is not required.)

Client's ~/.ssh/config (use of agent-forwarding not required):

Host somename
  Hostname mainhost
  ProxyCommand ssh -p 22 mainhost nc mainhost 11122

(Requires nc installed on mainhost.)

Then, on their laptop/computer:

$ git clone ssh://git@somename:22/user/repo.git

One problem: since you have to configure gitlab to listen on port 11122 and cannot change the port that it advertises to users, the clone field on gitlab repos will still show ssh://git@gitlabhost:11122/user/repo.git (or ssh://git@mainhost:11122/user/repo.git if you configured GITLAB_HOST=mainhost).

User configuration:

• only gitlab access: (for security, you might want to use restricted commands)

  Host mainhost
    Hostname mainhost
    ProxyCommand ssh -p 22 mainhost nc mainhost 11122
  

• both shell and gitlab access, using gitlab as default; this has the advantage that the user has less to change on the gitlab-provided URI

  Host mainhost
    Hostname mainhost
    ProxyCommand ssh -p 22 mainhost nc mainhost 11122
  Host mainhostshell
    Hostname mainhost
  

• both shell and gitlab access, but shell access is the default; not sure why you'd do this if you're already having to add blocks to the ~/.ssh/config, but perhaps there's rationale; this requires the user to change both the host and port in the gitlab-provided URI to something like ssh://git@mainhostgitlab:22/user/repo.git (depending on the ssh config recipe name):

  Host mainhost
    Hostname mainhost
  Host mainhostgitlab
    Hostname mainhost
    ProxyCommand ssh -p 22 mainhost nc mainhost 11122
  

Without a workaround (I know of none), the user will always need to change the gitlab-provided URI a little: at least the port number. It might be possible to do this without nc on mainhost (possibly by calling ssh again) but it's a common-enough tool that this seems to work just fine.

Motivated by https://stackoverflow.com/a/28201217/3358272.

[xiaket]

I have documented a solution here in my blog: https://blog.xiaket.org/2017/exposing.ssh.port.in.dockerized.gitlab-ce.html

Hope you'll find it helpful.

[s1113950]

@r2evans that's the most awesome picture in a comment I've ever seen ٩(͡๏_๏)۶

[harobed]

@xiaket

I have documented a solution here in my blog: https://blog.xiaket.org/2017/exposing.ssh.port.in.dockerized.gitlab-ce.html

Hope you'll find it helpful.

Yes, it is helpful.

My note, be careful, with this detail:

Luckily, around half a year ago, someone added this to the sshd_config file used by the sshd in
the container, and it was later renamed to /gitlab-data/ssh/authorized_keys.

This isn't available for sameersbn/docker-gitlab Docker image.

[harobed]

Implementation proposal: https://github.com/sameersbn/docker-gitlab/pull/1731

[duckbrain]

I have a solution that doesn't require nc or copying/templating files to keep things in sync.

setup_git_user.sh

# Create the git user if not already created
id -u git &>/dev/null || useradd git
# Set git's home directory to where I have the gitlab data directory mounted on the host 
# (this is the home directory for git in the container as well)
usermod -d /var/lib/gitlab/data git
# Give the git user permission to run docker commands without authentication.
usermod -a -G docker git
# Copy my fake gitalb-shell binary where the `authorized_keys` file expects it.
install -D -m 755 \
        dist/gitlab-shell \
        /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell

And my fake gitlab-shell binary looks like this:

#!/bin/sh

exec docker-compose \
    --file /srv/docker/gitlab/docker-compose.yml \
    --project-directory /srv/docker/gitlab/ \
    exec -T \
    --env SSH_CONNECTION="$SSH_CONNECTION" \
    --env SSH_CLIENT="$SSH_CLIENT" \
    --env SSH_TTY="$SSH_TTY" \
    --env GIT_SSH="$GIT_SSH" \
    --env SSH_ORIGINAL_COMMAND="$SSH_ORIGINAL_COMMAND" \
    gitlab \
    "$0" "$@"

This just makes it so SSH into git on the host will used the authorized_keys file created by the image and try to run the gitlab-shell binary, which just forwards all the relevant variables and arguments to gitlab-shell in the container.

This requires a volume mount, instead of a named volume, but that can probably be worked around. You can probably use fast lookups to prevent needing to modify the authorized_key file anyway.

To avoid adding the git user to the docker group, you could create a sudoers entry with a modified script.

[neuhausf]

@duckbrain I had the problem, that the user git on the docker host couldn't access the authorized_keys file in the named docker volume (no permission). If you give access to this file, the sshd isn't happy anymore.
As a workaround you can use this procedure:
https://github.com/sameersbn/docker-gitlab/pull/737
Howto:
https://github.com/IlyaSemenov/docker-gitlab/blob/fb42c3a7dcfdccfa9b924062eb3af83174d1825a/docs/docker_host_ssh.md

[WqyJh]

Here is an simple and safe solution.
Assume your hostname of the host is host1 and container docker1.

1. Edit /etc/ssh/sshd_config on host1 and reload the sshd

   Match User git
     PermitEmptyPasswords yes
     ForceCommand ssh -T git@docker1 $SSH_ORIGINAL_COMMAND
   

2. Create an git user on host1

   sudo useradd -m -s /bin/bash -p '' git
   sudo -u git ssh -T git@docker1 # Accept the key then ctrl-c
   

3. Edit /etc/ssh/ssh_config on client

   Host host1
      User git
      ForwardAgent yes
   

4. Use it with ssh-agent running

   eval `ssh-agent`
   ssh-add ~/.ssh/id_rsa
   
   ssh -T git@host1
   

This solution is not only suitable for docker-gitlab, it's designed for expose internal gitlab to internet.

[ohmyhusky]

I have documented a solution here in my blog: https://blog.xiaket.org/2017/exposing.ssh.port.in.dockerized.gitlab-ce.html

Hope you'll find it helpful.

Thank you very much xiaket. I've tried your solution but the gitlab would ask me for the git account's password, so I opened the authorized_keys generated by gitlab-shell, turns out the commands in authorized_keys is changed to /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-1 instead of key-3, is that the reason why user can not authenticate? If so, do you have a solution for this?

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.

[ccoenen]

I, personally, would really like if there was a canonical way. I can report that the things described in here work.