Kitematic: How to increase the disk size of the VM Kitematic creates

[atbrew]

I can see I can manually update the size of the boot2docker VM, but tbh I seem to have to delete the VM quite regularly so manual adjustment of the initial size is unpalatable. I would like to specify a profile that just say create a 60G disk on creation, and possibly lets me specify where that disk image is stored (eg on a external drive).

How can I use a boot2docker profile with kitematic?

[FrenchBen]

The VM created is actually a dynamic disk VM which will grow as needed - The default location will be what ever you have selected within VirtualBox for all of your VM creations, which can be found inside the VirtualBox app under:
VirtualBox > Preferences.. > Default Machine Folder

[atbrew]

It will grow up to the maximum specified, which is 20G (for the Kitematic docker-machine default).

I did try to work around this with docker-machine creating the box for me before launching kitematic, with mixed sucess. I needed to use the docker-machine that was bundled with KiteMatic (as opposed to the one I have installed separate to kitematic) to create the machine manually. After this Kitematic was able to use that virtual machine

# create an alias to kitematics docker-machine
alias dm='/Applications/Kitematic\ (Beta).app/Contents/Resources/resources/docker-machine'

# create the dev docker machine with a 40G (dynamically grown) disk
dm create -d virtualbox --virtualbox-disk-size "40000" dev

[FrenchBen]

@atbrew If you desire to do so, you can install the same docker-machine version via the File -> Install Docker Commands menu.
Having more granularity on the VM size is on the roadmap, as seen in the wiki:
https://github.com/kitematic/kitematic/wiki

[psychemedia]

+1 on there being an easier way to manage the upper size limit on the VM from somewhere within Kitematic.

One of the powerful features of Kitematic for me is that it makes it possible for relative novices to make use of container run apps/demos in a GUI environment - but expecting them to work on the command line before launching Kitematic is too much of a stretch...

[FrenchBen]

That's the Goal of kitematic - Although going over 20GB in containers/images I feel isn't the typical setup.
Out of curiosity, how many containers/images do you have?

[psychemedia]

I have a set-up for a course on databases and data management that runs an extended scipystack plus mongodb and postgresql, as well as an openrefine container (some explanatory context here). One of the activities uses quite a large dataset (a national road accident dataset and a whole bund of shapefiles) spread across a sharded mongo db.

(Finding effective ways of managing prepopulated databases and configurations that can be distributed efficiently to students is another issue, eg see https://github.com/kitematic/kitematic/issues/513 in this context also. A second issue is the ability to easily create and run docker-compose configs from within the UI environment. I was going to blog some thoughts on that when I get a chance.)

[FrenchBen]

Why not map volumes for the dataset instead of adding them to the VM, shouldn't that help reduce the size? Great blog post and explanation btw.

Adding support for Dockerfile + compose is in the Feature request list - Just need some dev time to get it implemented.

Ps: I created that 'portable app' branch and now realize the link between all this ;)

[psychemedia]

@FrenchBen re: mapping the volumes - I seem to recall I had a problem with that on a Mac? (I'm right at the limit of my skills/knowledge with all this - an optimistic technologist, rather a sysadmin/devops person!;-)

Also - how would that work for a distribution? If I have a large database, I don't really want students to have import the data, index it etc? The route I am taking is to build a virtualbox box with prebuilt images, and some prebuilt data volumes. Students then use vagrant and vagrant-docker-compose to fire up the containers and link directly to the prebuilt volumes.

I guess an alternative distribution route would be for me to build the databases, seed them, export a backup, distribute the backup and then let students fire up db containers, create new linked/mapped volumes and then restore the data from the backup (eg on a USB stick) into a mapped data volume? I haven't tried that route yet... (things not helped by the fact that I'm not a db admin either so don't know how to do that basis dbadmin stuff anyway, let alone in a container context!;-)

Being able to make use of docker-compose in kitematic context would be really helpful, I think. I'd quite like a GUI option so I could create a new composition, drag images from the Kitematic sidebar into it to specify a container of that image type, name the container on that canvas, and then use a graphical tool to wire them together:-) I'd also want to be able to open a panel that lets me specify properties etc of each container. In addition, split the left hand sidebar into available images and available compositions.

[FrenchBen]

@psychemedia The idea is for them to map the volume and just fire up the container without having to import anything.
Think of it like this, your db folder within your VM would be the same as the db folder within the user's desktop - Which means that the container would start up and see the data as it left it, just on the user desktop instead of within the VM. (to keep the VM lean)

Otherwise you can have it all in one place, only suggesting an alternate route so that you're not carrying a 20+GB VM

[psychemedia]

@FrenchBen I tried that but couldn't seem to mount database directories onto host though (on a mac; building the machine via vagrant-docker-compose). This is odd, though, because I can mount the host notebook folder for my IPython notebook server container and also a host directory for an OpenRefine server container's project files...

Just checked the Dockerfile again and it seems I ran the successfully mounted container as privileged. Setting the db containers as privileged won't let me mount them against host though, nor can I get linked data containers to mount against host?

[FrenchBen]

I forgot what DB you're using, but it could be the dreaded UID issue, where MongoDB (for example) expects the DB files to be MongoDB owned.

[psychemedia]

@FrenchBen I'm using mongo and postgresql... Which I guess means... sigh..?!;-)

Do you know if there is likely to be a fix to this to this any time soon (4-6 weeks), or mid-term (2-4 months)?

[FrenchBen]

@psychemedia see the following post: http://www.ybrikman.com/writing/2015/05/19/docker-osx-dev/
Where the files are simply rsync'd from one location to the VM.

[psychemedia]

@FrenchBen Thanks for that; my requiirements are rather more general though, I need a solution that works in a distance education setting where students essentially bring their own devices whilst working (largely unsupported) from home. Cloud would be one solution, but we need to try to make s/w available for arbitrary desktops - which is partly where the promise of Kitematic comes in:-)

[FrenchBen]

Yup, and why rsync would work, since you would be rsync'ing from the folder on your USB drive (or whatever) to the VirtualBox VM - But this would increase the size of your VM quite a bit, which is what we tried to avoid in the first place.

[leebenson]

That's the Goal of kitematic - Although going over 20GB in containers/images I feel isn't the typical setup. Out of curiosity, how many containers/images do you have?

I blitzed through 20GB in the first day of using Kitematic. My dev environment re-builds huge caches (about 3GB each), and each new build is versioned. It's not uncommon for my team to go through several versions a day. That's on top of the 10-15GB of extra tooling we bring in through 5-7 different Dockerfiles via Docker Compose.

We delete all active containers and kill any images, but the artefacts left behind by switching versions all the time easily pushes us over this number. SSH'ing into the box and manually culling is arduous and doesn't always prevent the 'no space left on device' warnings.

Applying a 20GB limit is too opinionated and arbitrary and clearly doesn't work for all users, hence this issue.

[FrenchBen]

@leebenson I think for 'most' users, 20GB is usually enough - In your case, you're working with a lot of data and could re-create the default VM to have a larger disk size that fits your purpose.

[leebenson]

@FrenchBen Defaults are fine if there's an easy way to change them. Is there someone I could look for instructions? thanks.

[FrenchBen]

@leebenson Unfortunately VirtualBox doesn't provide a simple 'click' option to resize the disk image, but take a look at: http://stackoverflow.com/questions/11659005/how-to-resize-a-virtualbox-vmdk-file for suggestions

[leebenson]

@FrenchBen thanks. I understand this is VirtualBox's problem, but since Kitematic is the front-facing entry point to OS X Docker (and docker-machine in turn), it'd be nice if one of the Docker tools took ownership of the underlying boot2docker set-up / machine resizing with a simpler GUI, or at least some kind of CLI flag to abstract it.

I think it'd go a long way to support Docker's goal of making Docker Toolbox a friendly front-end.

Sorry if I'm airing opinions in the wrong place. It's just because Kitematic is the first layer we use.

[kieranblight]

@FrenchBen I completely support @leebenson argument! I have 5 containers running and I am out of space! I have tried all of the above and more with no clear way to achieve more disk space for docker!! In my view dicker now becomes a no use tool for development.

[leebenson]

Thanks for sharing, @kieranblight. Good to hear this isn't just me. I'm having to delete ~/.docker/machine/machines/default almost daily now. Even deleting images / stopping and removing active containers doesn't seem to prevent the "out of space" message. I guarantee this will trip a lot of people up who don't want/know to go looking at fiddling with VirtualBox's CLI.

[FrenchBen]

@leebenson @kieranblight If you feel the disk space is an issue, simply re-create a new 'default' vm with more room:
Create 100GB VM:
docker-machine -D create -d virtualbox --virtualbox-disk-size "100000" default

[kag0]

@FrenchBen I've found that's the best option, however did you mean docker-machine create -d virtualbox --virtualbox-disk-size "100000" default?

[FrenchBen]

I did thanks for noticing - _fixed_

[impanda-cookie]

@kag0 thanks, it's working for me !