Machine: Export / import machines

[bfirsh]

Something like this should work for transferring machines:

 $ machine export test | ssh anotherhost machine import

(Maybe we could use machine inspect?)

[arminc]

What's the gain on this one? The goal of using machine from my perspective is to set up a simple machine with docker on it (no fancy stuff or custom things). With that in mind creating or export->importing a machine is exactly the same and does not give me any benefit.

[bfirsh]

When you create a machine, you can only use it and manage it from a single computer. You might want to:

1) backup your hosts
2) transfer them to another computer
3) share them with a team member

[waitingkuo]

Should we have a drop command so that we can remove the machine from the list without terminating it? I think it might be useful after some machines are exported and transferred to another computer.

[thaJeztah]

What exactly will the saved file contain? i.e.; Will it contain the token for connecting to the daemon?

Ideally, there should be something to manage computers/people to have access to the host. Thinking along the lines of a one-time token that can be to connect a new computer to an existing host.

[cjellick]

@bfirsh @waitingkuo This seems like a really useful feature that we'd use at Rancher (we'd like to use machine server-side).

Any reason the implementation (#29) seems to have stalled. Anything I could do to attempt to hep?

[gastonmorixe]

+1 would be awesome

[Starefossen]

+1

[urish]

+1

[kevin-coyle]

+1

[kevinsimper]

We have create a tool that we use ourselves, to make it easy to export docker-machines. It basicly exports all the certificates ssh keys. Feedback is welcome!

https://github.com/blackbeardapp/docker-machine-export

[uptownhr]

might be an overkill but something like a docker-machine registry would be awesome!

[cusspvz]

Currently we are using a git repo for sharing docker settings, problem is that some of us have virtualbox machines and thats the down-side

[docteurklein]

@kevinsimper how do you import them back?

[saada]

Does this issue include the idea of sharing swarm clusters between developers?

[docteurklein]

@saada I think it only covers credentials sharing, not cluster config.
It would be awesome tho to have this ability in machine.
Currently I only know of https://github.com/efrecon/machinery and https://github.com/nathanleclaire/moby

[micheletedeschi]

+1

[tonysickpony]

I am looking after the capability to share machine between users in the same hosts.
And is there a workaround for this ? I am copying the whole ~/.docker/machine/ folder for now

[fuse]

+1

[ffissore]

+1

[shanewow]

+1

[montytorr]

+1

[toh2526]

+1

[lukaville]

+1

[arpagon]

+1

[gomex]

+1

[mhubig]

+1

[leandromoreira]

I think a better way to show your intention is to click on Subscribe (at right side Notifications )

[schickling]

I've written an import/export script which you can use in the meanwhile until this feature gets implemented natively. Hope this helps :)

https://gist.github.com/schickling/2c48da462a7def0a577e

[thaJeztah]

@schickling perhaps something to be added to the "contrib" directory in this repository? https://github.com/docker/machine/tree/master/contrib/completion

[schickling]

Unfortunately I don't have the time atm to do that but please feel free to pull it in there!

[geovanisouza92]

+1

[grinnery]

+1

[ian-bartholomew]

+1

[gregkeys]

I wrote yet another little utility to export/import docker machines. We are using this for our CI deployment. https://www.npmjs.com/package/@mumbacloud/dmport

Thank you to kevinsimper for https://github.com/blackbeardapp/docker-machine-export I used your idea to export as JSON, which makes it quite portable for CI environments.

[mxab]

I also started working on some export/import script, what I really don't understand why there some duplicated files in the certs and the specific machine folder machines/<myMachine>. Also after adjusting the paths in the config.json to some custom paths outside the .docker/machine/machines... I noted that docker still expects some certs/pems/keys to be located in that folder

Can anyone recommend some documentation on what should be where actually?

[StefanScherer]

Looks like Node.js developers are actively using this (and rewriting own solutions). This one I found via Twitter

https://www.npmjs.com/package/machine-share

Sharing your Docker Machine config:

npm install -g machine-share
machine-export <machine-name>
machine-import <machine-name>.zip

[Nepoxx]

That same node package ( as mentionned by @StefanScherer ) has shell scripts as well if you don't have Node installed.

https://github.com/bhurlow/machine-share/blob/master/export.sh
https://github.com/bhurlow/machine-share/blob/master/import.sh

This is a workaround however, docker-machine should handle this and ideally provide a unique certificate per developer/user (so revocation is possible).

[mxab]

the problem I'm having with the current export/import scripts is that they will overwrite existing files in ~/.docker/machine/certs , won't they?

[gregkeys]

i can only speak for dmport, it will overwrite files if they exist.

its not a problem for us though because we use it in a docker ci that
doesn't have any thing to overwrite since it's a new container on each
deploy.

I'm planning on cleaning up dmport soon i can add a flag which will prevent
overwriting existing files if it will help
On Jun 1, 2016 1:54 AM, "Max Bruchmann" [email protected] wrote:

the problem I'm having with the current export/import scripts is that they
will overwrite existing files in ~/.docker/machine/certs , won't they?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/docker/machine/issues/23#issuecomment-222932695, or mute
the thread
https://github.com/notifications/unsubscribe/ADIV-3T7FEa9CJvn_UdTb9AeQfwI3lkCks5qHUjegaJpZM4DFGT5
.

[opHASnoNAME]

+1

[schmunk42]

We're using a docker container for this task - what else 😄

The idea is also to have the same environment (eg. cert configuration paths) for all users, mounted as a host-volume. You can use docker-machine, docker-compose and docker in containers created from roj. And you can pass the data-folder as a zip via email or put it into a private git repo.

[ricardodantas]

+1

[jcrben]

I'm not sure my use-case is the same as others - I've created a couple different machines for different purposes and I want to back up their existence and (minor) configuration. I'd rather just have some simple text files which are used to recreate the boxes.

https://www.npmjs.com/package/machine-share doesn't seem to respect my MACHINE_STORAGE_PATH. schickling's script backs up couple large binary files (e.g., boot2docker.iso). https://www.npmjs.com/package/@mumbacloud/dmport threw a couple errors.

[montanaflynn]

We also want this functionality, we've created a machine on gcloud using docker-machine for development and I want to allow my coworker to exec into the container and view the logs.

[bmmathe]

I'm actually surprised this isn't supported already. My example is I created a docker swarm cluster in Azure from my laptop (without thinking this all the way through) and now I want other devs to be able to manage the cluster so I created a shared VM in Azure that everyone can share. I installed docker-machine and now I need a way to import the machines from my laptop to our jump box in Azure.

[colthreepv]

@jcrben Hi, what about posting a full issue on machine-share, regarding the MACHINE_STORAGE_PATH?

I recently reviewed bhurlow's work, and I am curious to see if something breaks

[schmunk42]

@jcrben Did you also try https://github.com/dmstr/docker-roj? ... I am just curios.

[ntwrkguru]

People, for the love of all things proper, please 👍 the OP and not as a discrete comment. "+1" spam distracts from the conversation.

[lnshi]

FYI: #3212

[pedrodevoto]

@lnshi I think that the problem with importing an existing docker server with docker-machine create -d generic ..., is that it recreates certificates and restarts Docker, so not appropriate for a production/teamwork scenario.

[lnshi]

@pedrodevoto I know, I just mentioned that issues, since ppl have discussed there a lot.
For now I don't think there is a proper solution yet.

[Fedcomp]

@ntwrkguru still, it makes issue relevant and alive. There's no notification on emojis.

[ntwrkguru]

This issue is nearly 4 years old, I doubt that a few more +1 comments will suddenly cause it to become a priority.

[exhuma]

@ntwrkguru As long as the docker devs sort issues by 👍, it should be on their radar. It's the second-most upvoted issue at the moment. So keeping the 👍 coming will make sure that it stays on top.

Now, I don't know how the docker team prioritises issues, but I bet they also have paying customers, which I bet will always come first before looking at the public community feature requests.

[adilinden]

The same: https://github.com/docker/machine/issues/1328 ?

I am just getting used to docker and built my first host using docker-machine. Now one day later, I am trying to connect to my docker host using a different workstation... Surprised to see that that's no possible, at least not without some major headaches. Seems like an essential feature to allow teams to manage hosts...

[BretFisher]

That's because docker-machine was never designed to be a team tool. It's mean for an individual to easily create VM's that auto-install docker and setup dockerd TLS certs for remote cli management.

If you're trying to build and deploy docker environments for teams, you likely want to use a different tool.

[adilinden]

@BretFisher and what tools would that be? Not that I am wanting to use it for teams, just from 3 different workstations...

[BretFisher]

One of the big benefits of docker-machine is that it sets up TLS on the docker engine for remote cli management.

Today Docker Engine 18.09 was released and with it a great new feature that allows remote cli management using SSH tunneling (of sorts) and is a much better approach IMO for those of us without the need for full of docker engine RBAC. For those using docker-machine simply to make remote management easier, I recommend checking this out in 18.09. Both your docker server and client need to be on 18.09, and as long as you can SSH in, you can use the cli remotely.

Docker Captain Luc Juggery has a quick write-up on this. I expect it'll become my default way to use docker remotely.

[ntwrkguru]

I get what you're saying @BretFisher, but it would be nice to have it in docker-machine, since it keeps track of the remotes anyway. If i'm going to manage them over SSH, then I likely won't even bother using docker-machine to deploy them, and instead use Ansible. Docker-machine is yet another tool that could be great, but is relegated to mediocre due to lack of development.

[BretFisher]

@adilinden for a single user options include:

1. sync ~/.docker/machine/machines between those machines in a private and/or encrypted git repo. You may also need ~/.docker/machine/certs too, I'm not sure. It needs to be reasonably secure since it's storing private certs.

2. Pick a AMI, Droplet, etc. that already has docker installed and deploy those. Then use the new 18.09 SSH feature to remotely control the docker engine.

3. Try one of the many tools that export/import the machine data directories I listed above.

[adilinden]

@BretFisher I really like #2, it seems the best option by far for my scenario as all the knobs and buttons are in place for that already. Switching hosts looks trivial as well, as it is simply setting a single environment variable.

Now just to figure out how to get Docker for Mac to 18.09 instead 18.06.1-ce-mac73 (26764).

[BretFisher]

@ntwrkguru docker-machine is great for one-off machines or personal setups or someone who maybe only needs ~10 servers max, but if you've ever tried to manage 10+ servers with it, you'll see its limits right away. I wouldn't say that's due to lack of development, but rather "not within scope" of the tools original goals.

On the SSH, you wouldn't use ansible in this case, what I'm saying is once you've deployed any docker server (ansible or otherwise) that runs 18.09+, as long as you have SSH permission to that server, you can now do something like:

docker -H ssh://[email protected] run -p 80:80 nginx
or
DOCKER_HOST=ssh://[email protected] docker system prune

For controlling that engine remotely without any special config or setup. Being able to export an envvar and then run a bunch of commands against that engine (without wrapping them in ansible, etc.) is very handy.

[BretFisher]

@adilinden the Linux release just dropped today, so you'll need to give the teams a few days or week to roll out all the downstream products that use it :). If you want it today, use the edge release which has the 18.09 beta and works fine for me on Mac.

[ntwrkguru]

Admittedly, I haven't messed with docker-machine in several years. I have 4 servers left running plain Docker engine.

So, what's the benefit between docker -H ssh://[email protected] run -p 80:80 nginx and ssh://[email protected] docker run -p 80:80 nginx?

[edit] Not being argumentative; I'm wondering if there's a clear benefit.

[adilinden]

@ntwrkguru I can build from a Docker file on my workstation using the docker -H ssh://[email protected] . command. Whereas the same fails (as expected) using ssh -luser 10.10.10.10 docker build .

[adilinden]

Just for information, can also use ssh for socket forwarding. Per https://medium.com/@dperny/forwarding-the-docker-socket-over-ssh-e6567cfab160, but with a slight modification.

In one terminal session run

ssh -nNT -L $(pwd)/docker.sock:/var/run/docker.sock -l user 10.10.10.10

In another run

docker -H "unix:///$(pwd)/docker.sock" run -p 80:80 nginx

or

export DOCKER_HOST="unix:///$(pwd)/docker.sock"
docker run -p 80:80 nginx

[usr42]

I had the same issue and I created shell scripts to do the backup and the restore:
https://github.com/usr42/docker-machine-backup

Only the really needed data is backed up. For instance no iso files are inside of the backup.