Docker-sync: docker-compose.override.yml support

[evserykh]

Me with teammates use Docker to run a project. Some of us use Linux, others - Mac OS (Docker For Mac + docker-sync). Therefore we need to have different docker-compose configuration for different systems (I mean volumes configuration mostly).

Right now I'm trying to do the following:

1. docker-compose.yml - here is nothing about volumes and it's stored in repo
2. docker-compose.overrride.yml (https://docs.docker.com/compose/extends) - here is different volumes configuration depending on particular system and it isn't stored in repo

I'm able to run docker-sync and docker-compose up separately and it works fine with same approach. But docker-sync-stack doesn't respect docker-compose.overrride.yml file (as I understood) and some services can't be started because of missing data.

I read about docker-compose-dev.yml to split configuration and it might work (I didn't try), but on other hand docker-compose doesn't use this file by default and all Linux users will be forced to run docker-compose with -f docker-compose-dev.yml.

It turns out docker-sync-stack change default docker-compose's behaviour? Should it have a docker-compose.override.yml support or am I doing something wrong?

[EugenMayer]

options:
# default: docker-compose.yml if you like, you can set a custom location (path) of your compose file like ~/app/compose.yml
compose-file-path: 'docker-compose.yml'

# optional, default: docker-compose-dev.yml if you like, you can set a custom location (path) of your compose file. Do not set it, if you do not want to use it at all

# if its there, it gets used, if you name it explicitly, it HAS to exist
compose-dev-file-path: 'docker-compose-dev.yml'

Please read the documentation of docker-compose for that. Docker-Sync does not change anything, it rather helps you with the convenience of not using -f <> -f <> all the time, which you would need to to with docker-compose all the time.

You can define which docker-compose*.yml file are used, see the options above. To have custom scaffolding, you need to a little wrapper based on docker-sync https://github.com/EugenMayer/docker-sync/wiki/7.-Scripting-with-docker-sync

[evserykh]

Thanks for reply.

I don't think it'll work for my case because I need to have common configuration for both systems and different volumes configuration for Linux and Mac OS accordingly.

As I understand you offer:

1. to have completely different docker-compose file for Mac OS where all docker-sync stuff live
2. use compose-dev-file-path option for custom volume configuration

The first option is working but leads to redundancy (I did so formerly).
The second gives usage's impossibility because it will add volumes rather than replace. As result I'll have Linux and Mac OS volumes configuration both at the same time when running docker-sync-stack.

Actually I run manually docker-sync start and docker-compose up. My Linux teammates just run docker-compose up. We just have different docker-compose.override.yml with proper volumes configuration. There are no problems with it.

But I find quite strange running docker-sync + docker-compose up differs from docker-sync-stack start. docker-compose.override.yml is the part of docker-compose so it's support is expected behaviour in my opinion.

[EugenMayer]

@evserykh interesting, reading https://docs.docker.com/compose/extends/#example-use-case docker-compose added the .override.yml more recently.

I understand you issue, you can define multiple docker-compose files https://github.com/EugenMayer/docker-sync/pull/222 this might be the quick solution.

The other is, as i have linked, scripting/scaffolding with docker-sync as a lib https://github.com/EugenMayer/docker-sync/wiki/7.-Scripting-with-docker-sync .. that is what we do to leverage about 15 docker-compose-x.yml files for our project, we call them variants.
Does the first one help you already?

[EugenMayer]

maybe .env could help you too https://github.com/EugenMayer/docker-sync/issues/472#issuecomment-337504662

[evserykh]

@EugenMayer I stopped with docker-compose.yml + docker-compose.override.yml. Don't forget about Linux compatibility for my case. docker-compose.override.yml allows Linux and Mac users decide how to arrange volumes. It limits docker-sync-stack's usage but it's not a big deal I guess

[EugenMayer]

I guess we can close the issue then, right?

[ignatiusreza]

@EugenMayer I would keep this issue open, or to at least list it in #467 as potential breaking change for v1.0

@evserykh I'm curious to see what kind of issue you're having with compatibility between Linux and Mac users, and how different the overrides are between them... I'm also using Linux (Ubuntu) within a team of OSX users, and we solve the compatibility issue by using the default sync strategy (which would use native docker mount in linux, and native_osx - #465 - in OSX), without needing to use different compose files..

with that said though, you should be able to regain the default behavior of docker-compose in docker-sync-stack by having the following configuration in your docker-sync.yml file:

options:
  compose-file-path: 'docker-compose.yml'
  compose-dev-file-path: 'docker-compose.override.yml'

[evserykh]

@ignatiusreza sure. The docker-compose.yml doesn't know anything about volumes and it's common for both systems:

...
services:
  web:                                                                                                                     
    image: evserykh/rails:ruby2.3                                                                                          
    entrypoint: ./docker-entrypoint.sh                                                                                     
    command: /bin/bash -c 'rm -rf ./tmp/pids/server.pid && bundle exec rails s -b 0.0.0.0'                                 
    working_dir: /app                                                                                                      
    depends_on:                                                                                                            
      - sidekiq                                                                                                            
      - webpack-dev-server                                                                                                 
    ports:                                                                                                                 
      - 3000:3000  
...

Here is docker-compose.override.yml for Linux users. It just mount some directories from host to container:

...
services:
  web:                                                                                                                  
    environment:                                                                                                        
      SSH_AUTH_SOCK: /ssh-agent                                                                                         
    volumes:                                                                                                            
      - .:/app                                                                                                          
      - .bundle:/usr/local/bundle                                                                                       
      - ${SSH_AUTH_SOCK}:/ssh-agent
...   

Here is docker-compose.override.yml for Mac OS users. It does docker-sync trick:

...
volumes:                                                                                                                
  code_sync:                                                                                                      
    external: true                                                                                                      
  ruby2.3_bundle_sync:                                                                                                  
    external: true   

services:
  web:                                                                                                                  
    environment:                                                                                                        
      USER_ID: 501                                                                                                      
    volumes:                                                                                                            
      - code_sync:/app:nocopy                                                                                     
      - ruby2.3_bundle_sync:/usr/local/bundle:nocopy 
...

We add docker-compose.override.yml.example to our repo with Linux configuration and docker-compose.override.yml add to .gitignore. Therefore any new teammate can just copy docker-compose.override.yml.example -> docker-compose.override.yml and set up volumes as he wish.

[EugenMayer]

@ignatiusreza how do you consider this as breaking change / why a issue like this is kept to be open - i am not sure whay you would like to extract out of this in particular, could you elaborate?

@evserykh @ignatiusreza was making the point, so if you do not define a strategy at all in the docker-sync, you gain a cross-platform compatibitly with docker-sync. That means, under OSX there will be native_osx, under linux plain bind mounts ( host mounts ) in charge - automatically. so you could use the same -dev version for both systems

[ignatiusreza]

@EugenMayer since docker compose now natively support automatic overrides, don't you think it's a chance for us to retire both compose-file-path & compose-dev-file-path options? and people can just either use .override or COMPOSE_FILE.. and get feature parity with or without docker-sync-stack.. retiring both options will break existing codes which depends on them though, which is why it should be considered breaking change..

@evserykh not sure why SSH_AUTH_SOCK only needed for linux users while USER_ID is only needed by osx users.. but other then that.. if we only consider the app and bundle volumes, then following should be cross-compatible between linux and osx:

```yml...

docker-compose.yml

volumes:
code_sync:
external: true
ruby2.3_bundle_sync:
external: true

services:
web:
volumes:
- code_sync:/app:nocopy
- ruby2.3_bundle_sync:/usr/local/bundle:nocopy

docker-sync.yml

version: '2'
syncs:
code_sync:
src: '.'
ruby2.3_bundle_sync:
src: '.bundle'
```

granted that, linux users will need to run docker-sync start once to create the mount volumes..

[evserykh]

@ignatiusreza SSH_AUTH_SOCK is only actual for LInux because of docs says:

Socket files and named pipes only transmit between containers and between OS X processes – no transmission across the hypervisor is supported, yet.

More here

Your approach seems working, but in this case Linux users will be forces to create volumes manually before starting services while they don't need to have volumes at all. They can just mount desired folders.

[EugenMayer]

@ignatiusreza point given, but that needs to be an extra issue, not this one in particular - also this will need a firm discussion. Even if .override is there now, its by far not adopted and by far not as flexible as defining the compose files yourself, also several. So i see a pro/con - but lets move this to a different issue - it should not taken to lightly, a you said, it would brake existing code

[evserykh]

@EugenMayer maybe we can check docker-compose.override.yml here and use is it if file exists:
https://github.com/EugenMayer/docker-sync/blob/5de1a85aba13b199dfc0275519e7f3e7b01e7d33/lib/docker-sync/compose.rb#L7-L34

[Inglorion-G]

My organization decided to use docker-compose.override.yml to set ENV vars in our containers, and it took some experimentation to figure out that compose-file-path was the correct option for using the override file with docker-sync-stack start.

Because docker-compose.override.yml is inferred automatically by docker-compose, at first we expected docker-sync-stack start to work similarly - for example how docker-sync will automatically infer a docker-compose-dev.yml file. I do see the difficulty in accommodating all the different types of compose files that can exist, and maybe a bit more explicit documentation would be sufficient.