Kitematic: Allow for volumes without host-mapping them to VirtualBox

[rmb938]

Databases like mongo, mysql and rabbitmq get permission denied when trying to start. This has to do with the volume permissions. Being that having access to that data isn't really necessary I would remove the volume but there doesn't seem to be a way to do that either.

[jmorganca]

:+1: Thanks for reporting this!

Seems like many of the official hub repositories such as mysql or redis run the db process as a separate user, which _won't_ have access to a mounted volume.

It seems like a quick fix here is just to add the ability to disable a container's volume. Keep an eye out for a fix!

[rmb938]

Alright and thanks for the quick reply

[thaJeztah]

Currently, it's not possible to "disable" a volume if an image defines one. There's work in progress to allow UNSET-ing properties. See https://github.com/docker/docker/pull/8177 for more information on that.

_However_, volumes are used deliberately in those images for a number of reasons;

• For persistent data; volumes "outlive" the lifespan of a container; if the container "dies", the data is not lost
• Being able to "share" data (ie, --volumes-from), for example to use a one-off container to _initialize_ the data/database. And another container to actually _use_ or _run_ the data(base)
• Being able to "upgrade" a container (stop the old database container, start a new one, using --volumes-from=old-container, and remove the old container)
• For performance; and this is an important reason, the layered/copy-on-write filesystem in the container has a huge performance penalty for data that's changing often, which is the case with databases.

Using a "regular" volume in stead of a bind-mounted volume could solve the permissions issue. Another solution is to chown the directory/files using the uid:gid of the database user, but if VBox shared directories are used, performance and permissions can still be tricky.

[rmb938]

I know you can't disable a volume, but if you run a container without giving it volume information it will create a random volume inside the docker storage directory. This can be an easy and quick fix for now especially if the user does not care about having an accessible volume

[thaJeztah]

@rmb938 :+1: alright! This put me on the "wrong foot";

It seems like a quick fix here is just to add the ability to disable a container's volume.

[jmorganca]

:+1 Wrong wording on my part! Sorry! I meant adding away to disable the host binding altogether.

Sent from my iPhone

On Tue, Mar 3, 2015 at 12:33 PM, Sebastiaan van Stijn
[email protected] wrote:

@rmb938 :+1: alright! This put me on the "wrong foot";

It seems like a quick fix here is just to add the ability to disable a container's volume.

Reply to this email directly or view it on GitHub:
https://github.com/kitematic/kitematic/issues/209#issuecomment-76993212

[thaJeztah]

No problem, and no harm done! my comment may be useful for people finding this issue and are new to Docker and not familiar with all aspects of volumes :smile:

[jmorganca]

@thaJeztah Hugely useful! That comment should be a blog post post somewhere :smiley:

[g12n]

Another example for this Problem is the elsdoerfer / thumbor container.
It automatically exposes 3 volumes, but cant write into it. This way you cant run the tool at all.

[tslater]

+1 this is a pretty important thing here...I have the same issue with many containers
ex. mkdir: cannot create directory `/var/lib/rabbitmq/mnesia': Permission denied

[ghost]

+1 for the comment above , Getting no love from the rabbit container setup ,

Cannot create directory `/var/lib/rabbitmq/mnesia': Permission denied.

permissions error occurring on many other containers , making most unuseable

[jmorganca]

Sorry for the delay in getting a patch out and thanks for all the feedback
on this. This is priority for 0.5.11

On Saturday, March 14, 2015, Paul Mortimer [email protected] wrote:

• for the comment above , Getting no love from the rabbit container
  setup ,

Cannot create directory `/var/lib/rabbitmq/mnesia': Permission denied.

permissions error occurring on many other containers , making most
unuseable

—
Reply to this email directly or view it on GitHub
https://github.com/kitematic/kitematic/issues/209#issuecomment-80801832.

[sebgoa]

+1 for a fix to this. This is a petty big blocker.

[stereosupersonic]

+1 please fix this, i want to use mongodb

[IBMRob]

+1 please fix so I can try mqadvanced

[g12n]

@boot2docker ist struggeling with the same Problem :unamused:
compare: https://github.com/boot2docker/boot2docker/issues/581

[ezegolub]

+1. This is a huge blocker :(

[adamgering]

Does Kitematic use Boot2Docker underneath or replace it?

This thread has several suggestions for fixing the issue with Boot2Docker. A promising suggestion is to use NFS.

https://github.com/boot2docker/boot2docker/issues/581

[adamgering]

What are the solid work-arounds to date?

Was going to try the NFS solution, but not sure how to create the Kitematic-VM equivalent for:
/var/lib/boot2docker/bootlocal.sh

[adamgering]

What's the release ETA for 0.5.11?

[danielchatfield]

This is a massive blocker, I think virtualbox share should be ditched and NFS (or other) used instead (NFS even performs better).

[kipelovets]

+1 official/mysql doesn't work because of permission problems

[DominicBoettger]

+1 Mongodb official does not work

[amrox]

+1 Mongodb official or postgres official does not seem to work

[DominicBoettger]

Solved it for me with another image which does not export the data volume.
https://registry.hub.docker.com/u/inspirationlabs/mongodb/

[jmorganca]

ETA for 0.5.11 is Friday. Sorry again in a delay with the update

[jmorganca]

Fixed in #340. Cutting a release for 0.5.11 today

[danielchatfield]

It looks like the fix just lets you disable volumes. Not sure I would call that a fix, more of a workaround. For me, the primary benefit of using kitematic was the volume sharing with the host - without it, wordpress development is not really possible.

[jmorganca]

Sorry I wasn't more specific: it's more of a fix for databases where you don't really need access to the volumes. It defaults the volumes to be 'non-host mounted' and if you click the directory icon then it mounts them and lets you unmount them in the settings.

@danielchatfield which docker image are you using? I feel like this should still work for you. E.g. only enable the wordpress volume and don't enable the mysql one.

[danielchatfield]

@jeffdm I'll give it another go and report back. I seem to remember ngninx/apache getting access denied errors to the wordpress volume but I may be remembering that wrong.

Thanks :)

[nickcarenza]

Current feature disabling volume as of 0.5.12 working great for me. Thanks for the feature!

[g12n]

Though, to be honest: turning off the volume doesn‘t actually fixt Volume Permissions at all.
In most cases you actuall want to shave a volume to persist data.

[DominicBoettger]

Any ideas how i can work with mongodb an exported volumes? It's really frustrating....
Worked with internal storage and the database hung up. After restarting the container i was not able to remove the lock file.

[jmorganca]

@g12n it was a quick fix so they at least work out of the box :smiley:. It's still work in progress. That said, this is obviously not a fixed issue so I'll re-open it.

@DominicBoettger Sorry about this. The volumes won't work if you open them in Kitematic or choose/mount a Mac directory for now because of VirtualBox shared folders. Still working on a fix.

[dbeach24]

+1

Just tried kitematic for the first time and ran smack into this issue in the first 30 minutes. This is a major blocker.

[dstroot]

+1 can't use redis...

[freeatnet]

+1, redis persistence is broken :(

[cboden]

+1 RabbitMQ is broken :(

[jmorganca]

If you run a container in the latest version (0.5.13) and do not click the edit files folder icons or map any volumes in the container settings it should work.

Sorry the UX isn't the greatest - we're still working out a better solution to this.

[freeatnet]

@jeffdm That worked (while I have no access to data, it at the very least survives across container restarts). Thank you!

[chasegawa]

+1 For Postgres

[almathie]

@jeffdm With Kitematic 0.5.18 using image Redis 2.8.19

• If the volume is left unmounted => persistence is OK
• If the volume is mapped to an OSX Folder (~/Kitematic/redis/data) => Persistence fails (Permissions errors)

[jamestenglish]

@jeffdm Kitematic 0.5.23 with official Postgres 9.4 image. Same issue as @almathie
Not being able to have persistent database data is a huge blocker.

[DominicBoettger]

I had a look at that problem and as far as i understand is that the FS/Volume driver of Virtualbox is causing these problems. So there will only be the option to implement something like a network share with the MacOSX host or rewrite the Virtualbox driver. I think mapping a network share CAN also cause some problems with some filesystem operations.

[gpapin]

I experienced the same issue. I use the official mongo image and I wanted to persist/ being able to migrate data to another container easily but I cant map the data folder :(

[bastiao]

I'm also faced these issues with mongo and postgres. Moreover, I also tried create my own container but without success.

[jmorganca]

Thanks for the feedback everyone - seems like right now Kitematic only offers full VBox host mounted-volumes or none at all. We should have a way to enable the volume without host mapping it.

[andrewluetgers]

@jeffdm any update on the progress with this one?

[johnykov]

Hi, I have a problem starting postgres, I see this cause:
initdb: could not create directory "/var/lib/postgresql/data/global": Permission denied

[solisoft]

+1 for postgres and mariadb

[AlexKvazos]

+1 for mongodb

[timbrandin]

+1 for NFS

[starkovv]

+1 RabbitMQ is broken

[lypht]

+1 for mongodb. No issues with Rabbitmq in the same Kitematic instance, though.

[lypht]

Well this is interesting...

By defaulting the advanced options to TTY and STDIN open I was able to get all of my containers to stay running. I deselected both and restarted without issue. Can anyone else replicate this behavior? I've changed no permissions or otherwise altered these containers (beyond renaming them for dev purposes). The screenshot is useless other than to demonstrate that the containers are renamed and running.

[otype]

+1 for elasticsearch 1.4.5, mysql 5.6.26, rabbitmq 3.5.4-management ... all of these won't start up if the volume has been enabled.

Redis 3.0.3, though, seems to be able to run with an enabled volume ...

[fotisp]

+1

[marksomething]

Is there a work around?

[solisoft]

My work around : install Linux on your laptop and use terminal ;)

[ghostrocket]

+1 kind of makes this exercise futile

[pierre-b]

+1 also ;)

[alexbeletsky]

+1 blocks rabbitmq, will be happy to see workaround.

[alexbeletsky]

That issue exists for quite long time, basically preventing to use docker-compose for devepment environments. It seems to me, that it's not related to Kitematic itself, but rather docker (docker-machine) problem.

Are there any connected issues?

[sahin]

+1

I am using latest version in Kitematic. After we are setting "Enable Volumes", we are getting the following the errors.

Postgres

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... initdb: could not create directory "/var/lib/postgresql/data/global": Permission denied
initdb: removing contents of data directory "/var/lib/postgresql/data"
The files belonging to this database system will be owned by user "postgres".

Mysql

Initializing database
2015-12-28T01:24:37.592834Z 0 [Warning] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2015-12-28T01:24:37.644847Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2015-12-28T01:24:37.645642Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2015-12-28T01:24:37.646569Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2015-12-28T01:24:37.647002Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2015-12-28T01:24:37.647438Z 0 [ERROR] InnoDB: Cannot open datafile './ibdata1'
2015-12-28T01:24:37.647820Z 0 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2015-12-28T01:24:37.648064Z 0 [ERROR] InnoDB: InnoDB Database creation was aborted with error Cannot open a file. You may need to delete the ibdata1 file before trying to start up again.
2015-12-28T01:24:38.252569Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2015-12-28T01:24:38.252688Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-12-28T01:24:38.253262Z 0 [ERROR] Failed to initialize plugins.
2015-12-28T01:24:38.253551Z 0 [ERROR] Aborting

Mongo

2015-12-28T01:26:54.678+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=a484d3b2a3e0
2015-12-28T01:26:54.680+0000 I CONTROL  [initandlisten] db version v3.2.0
2015-12-28T01:26:54.682+0000 I CONTROL  [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] modules: none
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] build environment:
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten]     distmod: debian71
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten]     distarch: x86_64
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] options: {}
2015-12-28T01:26:54.702+0000 I STORAGE  [initandlisten] exception in initAndListen: 98 Unable to create/open lock file: /data/db/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
2015-12-28T01:26:54.702+0000 I CONTROL  [initandlisten] dbexit:  rc: 100

[sahin]

is there any ETA on this issue?, this is very simple and serious hassle for average dev to use kinematic.

@jeffdm @elesant

[gildor]

Really big issue, you just can't use Mac as host with dir shared with container (really useful for development).

[eturino]

+1, I can't really use this with mariadb without persisting the data...

[FrenchBen]

@eturino I understand the issue at hand, but you should be able to create a data container and simply map mariadb to it - You can copy your data to the container and back it up pretty easily as well.
If you need a new mariadb container, just scale it and allow it to grab the volume from the above data container.

[soosap]

+1

[koushikckm]

+1 Struggling a lot with mysql on docker :( due to this.
Any work around pls?

[agborkowski]

its funny one solution to solve this issue is get back to one vm image and dont waste time for trying change this behavior ;)

[Lenzcrafter]

ya... this is a joke. done wasting hours with docker... 5 minutes with vmware and im good to go...

[FrenchBen]

@Lenzcrafter I'd be interested in knowing how 5 minutes with vmware and im good to go solved your issue - Could you clarify?
Also there's an on-going discussion regarding mounting and its complexity here: https://github.com/docker/kitematic/pull/1202

[emad-elsaid]

I'm stuck with Postgres for 2 days, is there any workaround that can work with both development (OSX) and production (Linux ubuntu) machines?

[FrenchBen]

@blazeeboy You can setup a data volume and have Postgres write to it. The same volume can be used on a production machine as well.

[emad-elsaid]

@FrenchBen How can we do that? because as for now the official Postgres image doesn't boot and produces this output

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
initdb: could not create directory "/var/lib/postgresql/data/pg_xlog": Permission denied
initdb: removing contents of data directory "/var/lib/postgresql/data"

On OSX 10.11.4, Docker tools installed 24 hours ago, so it's the latest I think.

[FrenchBen]

@blazeeboy Something like this would work:

# Create a volume for all of the Postgres Data
docker run -d --name pgsql-volume -v /var/lib/postgresql/data tianon/true
# Create your postgres container using the volume created above
docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword --volumes-from pgsql-volume -d postgres

If you need to backup/restore/migrate your data, follow the following:
https://docs.docker.com/engine/userguide/containers/dockervolumes/#backup-restore-or-migrate-data-volumes

container logs:

...
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
...

If you remove the postgres container and create a new one, the container logs will be (notice no DB creation) 🎉 :

LOG:  database system was shut down at 2016-04-13 00:50:46 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

[thaJeztah]

Instead of a data-only container, you can also use a named volume;

# create a volume for the postgres data
docker volume create --name postgres-data

# and use that volume when starting the container
docker run --name some-postgres -v postgres-data:/var/lib/postgresql/data/ -e POSTGRES_PASSWORD=mysecretpassword -d postgres

[jchilders]

For mongodb, upgrading to the Docker Beta (1.13.0-beta39 (15140)) fixed the issues I was having and others have reported above. Docker Beta for Mac