Ddev: As a ddev user, I want to be able to use ssh/rsync from within the web container

[rickmanelius]

What happened (or feature request):

• Feature request

What you expected to happen:

In a conversation with @schnitzel, he mentioned a workflow within his organization where they leverage an SSH agent docker container to make it easier for team members to communicate with an API endpoint without having to re-authenticate by typing in their SSH passphrases.

Given that ddev already allows the ability to extend/add other services as described here (https://ddev.readthedocs.io/en/latest/users/extend/additional-services/), this feature could be provided by essentially creating a docker compose file into the ddev library. Example, we did the same with Apache Solr here https://ddev.readthedocs.io/en/latest/users/extend/additional-services/#apache-solr.

Anything else do we need to know:

Ideally, we can get access to the docker compose file used by @schnitzel along with any other business logic needed to connect in their workflow.

Related source links or issues:

• The ability to add custom containers #121

[rfay]

Note that MacOS already does all of this - you just do ssh-add and it adds your keys and prompts for key passwords and keeps them live for the duration (until you reboot).

[Schnitzel]

@rfay correct, I tried to leverage the ssh-agent that is already existing on mac inside docker containers, but unfortunately I have never been able to do so.
This is what I tried:

docker run --rm -it -v $SSH_AUTH_SOCK:/tmp/sock -e SSH_AUTH_SOCK=/tmp/sock -u 501 amazeeio/ssh-agent ssh-add -l

The ssh-agent Dockerimage is here: https://github.com/amazeeio/docker-ssh-agent

So we ended up creating our own ssh-agent within a new container, allow clients to add individual ssh-keys to that container and mount the socket into the drupal containers.

[rfay]

I guess the problem I was missing is that we don't use ssh for access to the container, we use docker exec, so the keys don't get carried. Sorry for my short-circuit there. We'd love a full description of your solution.

The basic idea is that you want to carry out ssh actions from within the container that access external remotes, right? So a git pull from within the container. I didn't understand that fully from the OP, and also short-circuited on the "we get into container with docker exec not ssh" thing.

In my own workflow, I would carry out most git activities on the host rather than in the container. Can you help us understand what the circumstances are where it's better to do it within the container? (I note that it's quite a bit slower to do git activities inside the container due to file access times, as well.)

[rickmanelius]

I'm pulling this from memory, but I believe a primary use case is less about git and more about being able to make calls over ssh to other endpoints, specifically the amazee.io hosting API. So this might not be resolvable by a preference of working within versus outside the container.

@Schnitzel can correct me if that isn't accurate :)

[Schnitzel]

@rickmanelius yes!

so the idea is not to use SSH to connect TO the container (that's what docker exec is for), but to use SSH within the container to connect to the amazee.io API. (the idea of the amazee.io api is that you make an ssh connection with your ssh key to an auth-server, that auth server gives you a JWT Token that then can be used to authenticate against the api, with that we never need to ask the user for any username or password, but we can still identify them perfectly).
So it's not to run like git pull within the container (even thought this would maybe work), it's for things like drush @dev ssh which basically tells drush to connect to the dev branch environment via ssh. (which itself needs to make an api call to figure out which project you are in and then also figure out if you have access to the dev branch of your project)

[rickmanelius]

Repeating this back with high level criteria, it sounds like we could meet this use case in the following way:

1. Docker Compose + Container: Similar to the "Additional Services" instructions for Apache Solr, a docker-compose.ssh-agent.yml example is created leveraging the docker image here https://github.com/amazeeio/docker-ssh-agent (or an image feature compatible).
2. Mount + Keys: As per the Readme.md, essentially we have the long term volume mount used to hold the sockets. There is also a secondary, temporary mount used to pull select keys from the host OS and into the ssh-agent container.
3. Expose to Drupal Containers: Once the above steps are completed, the other containers (notablye web and db) can access the sockers on the ssh-agent mount through setting the appropriate environment variable on these images to use this mount.

Or repeating even more succinctly from the end-user's perspective.

A. Provide ssh-agent docker-compose file.
B. Provide a means to specify which keys to add.
C. Provide a way to specify known hosts to add.

Additional implications from ddev's perspective.

• Include test coverage to Linux and Windows.
• Ensure that solution above could easily extend to all containers/services (e.g. Solr container, Varnish container, etc).

[rickmanelius]

Crap! I forgot to copy over the intro paragraph from my local markdown editor :)

"Thanks, @Schnitzel! That's what I remembered and this will help us move this from a high-level discussion to a more detailed implementation strategy."

[rfay]

Looking around at the history of this problem. It seems that it has been chronically beset by instability, apparently from both Apple's end and Docker's end.

• gist with ongoing workarounds
• docker feature promised for Nov 2016 but never acted on and related docker-for-mac issue
• nardeas/docker-ssh-agent
• avsm/docker-ssh-agent-forward, which was recently popular but now looks unworkable and abandoned based on issue queue and lack of responses.
• and of course Amazee's own entry, amazeeio/docker-ssh-agent

So there are some resources. Bottom line is this has been very unstable territory, so we need to do something that is sustainable and maintainable.

[rfay]

I have this working fine in docker-compose. I happened to use nardeas/docker-ssh-agent instead of amazeeio/docker-ssh-agent but they do the exact same thing. There are some config changes in docker-compose v3 config that caused some pain, but it should work lots of ways.

I'm not clear whether you'd want an ssh-agent per site or per workstation, so we'd need to sort that out.

There are some potential pain points though.

• Normally this would only work for root, and although if a user will be root if they do a ddev ssh into the web container, but the nginx and php processes run as user nginx. If the goal here is for php to be able to use the ssh credentials, then this approach won't be enough. There are workarounds for tricking the ssh-agent into allowing an alternate user.
• I don't know how to test this approach in any realistic way, so would appreciate your help on that @Schnitzel.
• Auth is lost when the ssh-agent container restarts or gets rebuilt. In ddev this might happen a lot.
• The user's actual private keys get copied onto the container. This isn't normally the care we'd want to take with a private key.

version: '3'

volumes:
  dot_ssh:
  socket_dir:

services:
  ssh-agent:
    container_name: ssh-agent
    image: nardeas/ssh-agent:latest
    volumes:
      - "dot_ssh:/root/.ssh"
      - "socket_dir:/.ssh-agent"
    environment:
      - SSH_AUTH_SOCK=/.ssh-agent/socket
  ssh1:
    container_name: ssh1
    image: kroniak/ssh-client:latest
    command: 'sh -c "tail -f /dev/null"'
    volumes:
      - "dot_ssh:/root/.ssh"
      - "socket_dir:/.ssh-agent"
    environment:
      - SSH_AUTH_SOCK=/.ssh-agent/socket
  ssh2:
    container_name: ssh2
    image: kroniak/ssh-client:latest
    command: 'sh -c "tail -f /dev/null"'
    volumes:
      - "dot_ssh:/root/.ssh"
      - "socket_dir:/.ssh-agent"
    environment:
      - SSH_AUTH_SOCK=/.ssh-agent/socket

[rickmanelius]

A follow-up after we had a productive team meeting this morning to discuss. Part of what we want to accomplish from a workflow perspective is ensuring that adding this ssh-agent functionality can be done without introducing too many dependencies or make things fragile. However, we're hopeful that we have a path forward to do just that.

Previously (see https://github.com/drud/ddev/issues/414#issuecomment-329192447), I had suggested these 3 main activities fromm an end-user perspective:

A. Provide ssh-agent docker-compose file.
B. Provide a means to specify which keys to add.
C. Provide a way to specify known hosts to add.

Randy's comment below gives an example docker-compose.SERVICE.yml file that is useful for testing and proving out the concept. To make this more usable with ddev out of the box, we would want this docker-compose.ssh-agent.yml file to provide the appropriate mounts and environment variable entries essentially as an override/addition to the core docker-compose.yml file, which by default names db and web. Then, for common commands to run after (essentially step 2 https://github.com/amazeeio/docker-ssh-agent#2-add-your-ssh-keys and 3 https://github.com/amazeeio/docker-ssh-agent#3-optional-add-known_hosts), we could leverage post command hooks https://ddev.readthedocs.io/en/latest/users/extending-commands/. This would allow devs to have those actions happening automatically as well as shareable.

In short, end-users that needed ssh-agent could easily pull down a singular docker-compose.ssh-agent.yml file per project and then copy/paste whatever typical commands they specify to load the keys and/or known hosts.

Now that still puts us into a one ssh-agent container per site model instead of one ssh-agent container shared by all sites. We believe, similar to our use of a singular nginx proxy container, that we could meet that use case but that this would potentially require this functionality/business logic to be pulled into ddev's binary instead of extending ddev through known and existing paths (i.e. docker-compose file stacking per project and post command hooks).

[rickmanelius]

Hey @Schnitzel. One thing that is currently blocking us was this question from @rfay.

"I don't know how to test this approach in any realistic way, so would appreciate your help on that"

Any thoughts on this?

[rickmanelius]

Just wanted to bump this up because we have a proposed path forward and just need a means of getting an agreed upon test in place.

[Schnitzel]

Normally this would only work for root, and although if a user will be root if they do a ddev ssh into the web container, but the nginx and php processes run as user nginx. If the goal here is for php to be able to use the ssh credentials, then this approach won't be enough. There are workarounds for tricking the ssh-agent into allowing an alternate user.

Yes, that's what we do in our setup and it works good, we know the user that the nginx, php, etc. containers are running in are user 1000 and we start the ssh-agent also with that user.

I don't know how to test this approach in any realistic way, so would appreciate your help on that @Schnitzel.

Mhh test what exactly? Basically if you can run an ssh-add -L within the nginx container and you see an ssh key that has been injected it is all good?

Auth is lost when the ssh-agent container restarts or gets rebuilt. In ddev this might happen a lot.

correct, you just need to restart the nginx container and then it works again. It's docker, so restarting a container should not be of pain.

The user's actual private keys get copied onto the container. This isn't normally the care we'd want to take with a private key.

that's why you want the container that has actual access to the key only short lived, basically only for the couple of seconds that the user enters the passphrase (if there is one even). Afterwards you only want the ssh-agent container running, from which it is impossible to generate the ssh key again.

Now that still puts us into a one ssh-agent container per site model instead of one ssh-agent container shared by all sites. We believe, similar to our use of a singular nginx proxy container, that we could meet that use case but that this would potentially require this functionality/business logic to be pulled into ddev's binary instead of extending ddev through known and existing paths (i.e. docker-compose file stacking per project and post command hooks).

aehm yes please, not per site, or the users will end up entering their ssh keys again every time they start a site and then we could directly mount the ssh keys into the nginx container and don't need a system where we can share containers.

[rickmanelius]

Our next actions on this are as follows.

Research:

• Which approach provides 1) the lowest LOE and 2) the lowest impact to users that don’t need this setup.

Current Options

• Global. Spin up and down in conjunction with the nginx router, which makes this available for any other container to use.
• App type specific (Amazee.io). When a container of a specific type is spun up (how we key off of this is still TBD), spin up this container.

[BrianGilbert]

any updates on this issue?

[rfay]

No @BrianGilbert.. but we thought this was a specialty request from @Schnitzel - are you on the same team, or do you have a different reason for requesting this? If you do, please help us understand your use-case.

[BrianGilbert]

@rfay Not on the same team.. and on the other side of the world :) it's just something that we've needed at least once in the past.

[rfay]

On docker-for-mac, the problem with not being able to mount $SSH_AUTH_SOCK is described in https://github.com/docker/for-mac/issues/410

The usage (when it works) is demonstrated very nicely in the official composer container, https://hub.docker.com/r/library/composer/

[rfay]

Related request, https://github.com/drud/ddev/issues/867 - ability to actually ssh into container.

[rfay]

Thinking about this in the context of #867 (rsync utility, ssh into container), there's another approach we haven't considered I don't think, which could have a lot of usability value for the user:

• Run sshd inside the container
• Allow the user to specify public key in config.yaml and place that in the container's ~/.ssh/authorized_keys on container start
• Optionally set the username used in the container to the same as the username on the host at container start.
• On start, provide ssh instructions (probably to ephemeral port, or we can come up with a ddev realssh command.

If the user used ssh -A to get into the container, it would carry their keys into the container in a pretty safe and normal way, and rsync and ssh would work from the container, including access to private repositories.

[Schnitzel]

That sounds like a good idea, just with one flaw (in my opinion)

Run sshd inside the container

That goes for me against everything that containers are here for :)
not only would it require to run two ore more services per container and to keep track of them via some kind of initd system.
But it also doesn't really make sense to me to have a service running inside a container that basically just does what docker exec -it already does.
Also having two commands ddev ssh and ddev realssh is confusing.

[rfay]

• We already run supervisord in the web container to keep track of nginx and fpm services, so it's not hard to start and manage the service.
• Unfortunately, ddev exec doesn't support the very things that this issue is about, having a secure user key management system inside the container. That would sure be a nice feature from docker though.
• I was mostly joking about ddev realssh, and also hate the idea of a competing command. But it might be necessary to do something along that line.

[Schnitzel]

We already run supervisord in the web container to keep track of nginx and fpm services, so it's not hard to start and manage the service.

mhh okay, the original ddev images maybe do, but other docker images don't do that (as they shouldn't) and having the requirement to have sshd running within the Docker Images so that they work with ddev is a requirement I wouldn't suggest or support.

Unfortunately, ddev exec doesn't support the very things that this issue is about

mhh why not? We have it already running at amazee.io/lagoon and there has never been an issue?

[rfay]

the original ddev images maybe do

Well, we don't really have any current plans of directly supporting anything except the images built to work with ddev. So "the original ddev images" are plenty good enough at this time :)

We have it already running at amazee.io/lagoon

I know you have a technique that works for you for managing ssh keys, but it's not the same as docker exec or ddev exec. docker exec (inside ddev exec) doesn't support any ssh client activities, and even has problems with stdin/stdout. I know that you have another way of getting the keys in there with lagoon, but it's not via docker exec, as we've talked about earlier in this issue. Many users, including you, have the need to have their keys available in a traditional way (for drush rsync, for example, or for composer access to private repos). ssh does this trivially with ssh -A, leaving that choice to the user, not automatically bringing the keys across.

[rfay]

A hacky way to provide ssh, not suggested in some security environments, definitely not recommended if your private key is not password-protected:

1. Create a .ddev/docker-compose.ssh.yaml

version: '3'
services:
  web:
    volumes:
      - "$HOME/.ssh:/home/.ssh"

1. ddev start
2. ddev ssh
3. eval $(ssh-agent)
4. ssh-add (to provide the password to your key) (If mounting from Windows, you may need to cat ~/.ssh/id_rsa | ssh-add -)
5. Do what you need to do.

It's awkward, but surprisingly useful when required.

[DaleTrexel]

On the Drupal #ddev Slack channel, I mentioned a similar need/use case, and was pointed here. I just tried @rfay 's suggestion and can confirm it works! Adding this process to the DDEV documentation would be useful (perhaps with security caveats), if you decide this isn't something you want to provide as a more integrated, configurable option.

For my use case, I was attempting to use Composer within the web container to build a demo site based on a configuration that included private git repositories. The conversation on Slack got me as far as setting up a docker-compose.override.yaml file with the same contents as the docker-compose.ssh.yaml example above. This allowed me to run "composer install" and download all the dependencies without having to create an SSH key on the container and register that with the private github account. However, because the SSH key was password protected, I was prompted for the password for each repository that was private. (Actually, I was prompted 2x for each repo.) That was not fun! It would have ultimately been less work to create a new SSH key in the web container and register it, rather than all the password re-typing I ended up doing.

I just tried re-doing the process with the addition of "eval $(ssh-agent)" and "ssh-add" (on a Mac) as suggested above. It worked perfectly! If you're looking for an easy way to "borrow" credentials from the host system in a container, and if you understand and accept the security implications, this does the job with relatively few steps.

[rfay]

Added a Stack Overflow question/answer at https://stackoverflow.com/questions/51065054/how-can-i-get-my-ssh-keys-and-identity-into-ddevs-web-container

[rfay]

I think the time has probably come to try to solve this for non-linux users (Linux works fine with the SSH_AUTH_SOCK mount). It's becoming obvious that composer users should do their work in the environment they're going to deploy on (which means not running composer on the host, but instead running it in the web container). And lots of users have private repos. So it's probably just time to solve this one.

[frleb]

The openssh-client package was installed before 1.2.0. I could use composer with private repositories with ssh. Since 1.2.0, and also 1.3.0, I can no longer clone private repositories, and I can not even install the openssh-client package with a post-start / exec. Is there a reason why the package was removed?

If I want to upgrade from 1.0 to 1.2, or 1.3, I will have to create my own webserver image; what I would like to avoid.

If ddev is used exclusively for development, and the size of the docker image is a critical issue, why not create docker images by major version of php? So it would be possible to add all php packages in the same image?

[KaffDaddy]

I have the same problem, too. Because I am using private repositories with composer inside the docker container. And without a ssh client I can't deploy with TYPO3 Surf from within the docker container anymore. The open-ssh client was removed in this commit https://github.com/drud/ddev/commit/b992f6f0c45f41f9a119aceca6bbf481ecf7d274

[KaffDaddy]

Workaround is, that you set webimage: drud/ddev-webserver:v1.2.0 in your config.yaml. But I have still not evaluated, if that causes any unwanted side effects.

[cjanody]

Same problem here. Installing package manually return dpkg error which results in a "Failed to start" when installed via post-start hook.

addgroup: No GID is available in the range 100-999 (FIRST_SYS_GID - LAST_SYS_GID).###############################################################################################################################################..............................] addgroup: The group 'ssh' was not created. dpkg: error processing package openssh-client (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: openssh-client W: No sandbox user '_apt' on the system, can not drop privileges E: Sub-process /usr/bin/dpkg returned an error code (1)

[rfay]

Sorry, I guess openssh-client must have been default in debian jessie, and not on stretch. We'll get that in there.

In the meantime, you should be able to add to your post-start hooks:

hooks:
  post-start:
    - exec: sudo bash -c "apt-get update && apt-get install -y openssh-client || true"

It slows down start, but it's a workaround. As soon as we have a PR ready (super easy) we'll post a link to the image that you can use.

[frleb]

@rfay , I tested what you suggested to me, but there is an error while installing the package. 

This is the same problem that @cjanody mentioned.

Failed to restart fsg: post-start exec failed: Failed to run docker-compose [-f /home/flebel/repositories/fsg-instance/.ddev/docker-compose.yaml -f /home/flebel/repositories/fsg-instance/.ddev/docker-compose.assets.yaml -f /home/flebel/repositories/fsg-instance/.ddev/docker-compose.hosts.yaml -f /home/flebel/repositories/fsg-instance/.ddev/docker-compose.solr.yaml -f /home/flebel/repositories/fsg-instance/.ddev/docker-compose.ssh.yaml -f /home/flebel/repositories/fsg-instance/.ddev/docker-compose.typo3.yaml exec -e DDEV_EXEC=true -T web sudo apt install php7.1-ldap -y], err='exit status 100', stdout='Reading package lists...
Building dependency tree...
Reading state information...
php7.1-ldap is already the newest version (7.1.23-2+0~20181017082622.9+stretch~1.gbpab65a0).
0 upgraded, 0 newly installed, 0 to remove and 66 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up openssh-client (1:7.4p1-10+deb9u4) ...
addgroup: No GID is available in the range 100-999 (FIRST_SYS_GID - LAST_SYS_GID).
addgroup: The group `ssh' was not created.
dpkg: error processing package openssh-client (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
openssh-client
', stderr='WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

W: No sandbox user '_apt' on the system, can not drop privileges
E: Sub-process /usr/bin/dpkg returned an error code (1)', stderr='WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

W: No sandbox user '_apt' on the system, can not drop privileges
E: Sub-process /usr/bin/dpkg returned an error code (1)'

[rfay]

Yeah, that's not fatal at all. I tested the hook I posted and it did fine.

But much simpler, please use the pushed image from #1217 by adding webimage: drud/ddev-webserver:20181017_add_ssh to your .ddev/config.yaml. Sorry I forgot to post that here, was waiting until it was through pushing.

[rfay]

Note that at least on docker desktop for mac, docker itself now knows allows access to $SSH_AUTH_SOCK, see https://github.com/docker/for-mac/issues/410

It may be that as a result we can stop using the ssh auth container. New issue to review that opened in https://github.com/drud/ddev/issues/1904