Docker-sync: Files not sync in containers created using `docker-compose run`

[zedtux]

Error/Feature Requestion/Docs

I have my entire set of applications working, with the file synchronisation working too.
I'm now trying to run a command using docker-compose run --rm <container-name> <command> but in this case I'm missing files while with docker-compose exec <container-name> <command> it's working fine.

I don't remember having read anything in the documentation about this case so that why I'm opening this issue.

Docker Driver

d4m

Sync strategy

default

your docker-sync.yml

# Find the documentation at https://github.com/EugenMayer/docker-sync/wiki/2.-Configuration
version: "2"

options:
  # optional, activate this if you need to debug something, default is false
  # IMPORTANT: do not run stable with this, it creates a memory leak, turn off verbose when you are done testing
  verbose: false

syncs:
  app-sync:
    src: './my-application'
    sync_excludes: ['.git']

OS

macOS 10.12.6

[EugenMayer]

docker-compose run is simply not supported - you need to understand what you are actually doing there to understand, why this is not possible

[zedtux]

Well your comment is not really helpful to understand the issue but I will figure it out myself ...

[EugenMayer]

I understand that, but i am not able to be more descriptive here because docker-sync is not really a docker workshop .. that just would take way to much time. I just try to be docker-sync specific so i can cover all the issues here, otherwise docker-sync related issues will not be worked on ,and thats even worse

[zedtux]

In the end, I think the only peace I'm missing is the way to link the docker data container that docker-sync created to the run command, or is it much more complex than that?

[EugenMayer]

you neither understand what run does and what exec is in comparence to that - so thats you starting point i would say.

What ever you want, running commands on started containers of the stack is either docker exec <conainer> command or docker-compose exec <service> command

in no way its run.

[zedtux]

OK, whatever, thank you.

[EugenMayer]

just remove docker-sync from your stack entirely and try out things you currently do - it will not work differently in terms of the concept. Thats why this is not a docker-sync question. Anyway.

[zedtux]

Well I was using it this way without docker-sync and it was working perfectly, why would I change my way of working by adding a new tool?

Anyway, I will find by myself.

[zedtux]

I knew it was nothing hard :)

So for the people looking to this, as I said in my comment it's nothing more than adding the volume, so you just need this:

docker-compose run --rm --volume app-sync:/application:nocopy <container-name> <command>

I did it using watch -n 1 ls -al as command, and changed files and it's working.

[EugenMayer]

and now you understand what run is :)

Since it creates a new container and you did not define any volume mounts, you obviously will not have any.

the whole point here is, docker-compose makes no sense at all, since you are not referencing a service, you are rather referencing an image - very confusing solution by docker-compose here, it rather matches a docker run here, and then the volume option would be far more obvious.

Still, not sure why you ever would use docker-composer run here. Start the stack and use docker exec - that is what it is designed for.

[zedtux]

and now you understand what run is :)

Actually I knew it before already, I didn't want to go to fast in my comments because I was thinking that something was missing in my knowledge that was explaining your first comment.

To me, docker-compose run has nothing confusing here. As you say, you need to know what is the difference between run and exec but you don't want, all the time, to do exec.

Still, not sure why you ever would use docker-composer run here. Start the stack and use docker exec - that is what it is designed for.

I have another point of view than you. I need an instance of the image to run a command, and I don't want to touch the running one, that's all.
Of course, I could do the same with exec but my goal here is just to say "I'm doing something temporary, I don't need to keep anything generated/created in this container", while exec would keep it.

For instance, I want to generate an export file, a one shot. While would I go with exec? I just want to run once my export, get the file out of this and I don't care about the container (I always use run --rm).

[motin]

TL;DR this is how it is accomplished:

docker-compose -f docker-compose.yml -f docker-compose-dev.yml run <container-name> <command>

This will make docker-compose run use the same volumes as the docker-sync'd stack uses. It will however not be up-to-date / synced unless docker-sync is also running.

@EugenMayer Just fyi, I very well know the difference between run and exec, docker-compose and docker, how docker-sync works etc. Still, I was looking to achieve the same thing as @zedtux - ie to use docker-compose run while still utilizing the docker-sync'd volume for the locally shared/synced app source code.

As @zedtux explains, it is important that the commands executes in a clean environment based on an image, not touching a service/container running in the stack. By using run, one can also have multiple commands executing in parallel.

[EugenMayer]

@motin basically everything you said, makes it clear, that you 100% know what you are doing. I

By using run, one can also have multiple commands executing in parallel.

I understand that fully, but this is a 100% special case. And if you are building special cases, you should be perfectly aware of your toolset.

And this is what it boils down to. This is a 100% docker-compose topic, has nothing to do with docker-sync at all.

You "fix" is nothing else then ensuring that you actually use the same docker-compose.yml "merge" used with the docker-sync-stack start command, thus the one with the prod+dev.

If you use docker-compose run with -f .. -f -dev you end up having the prod env, without the volume mounts, and thats absolutely obvious what will now fail and be slow. That said, you can very much make this a lot more convinient using a .env file and doing

COMPOSE_FILE=docker-compose.yml:docker-compose-dev.yml

Now you can use docker-compose run without -f .. -f -dev since that has the same result/

All that said, i stay with my initial statement, docker-sync is no docker school, and especially not for people doing specialized things. I fully understand the use case, how its useful and how this stateless run ( no leftovers ) can be useful for e.g. tests.

But keep it honest, this is just vampiring help for a total different topic, which has 0 relation to docker-sync. Its basic docker-compose understanding, the different between exec and run was for sure not clear to the initial author of this issue.

If you would not use docker-sync but have a dev which opens port or mounts ./src and @zedtux would have done the exact same thing with run, not using -f -f, and then tried to access the port or expect the code mount, he would failed equally. No docker-sync involved, plain docker-compose.

Again, pretending to understand all this beforehand did not help, it made me even more reluctant to help.

Since exec connects to a running container, its volume definition, the one form the initial docker-sync-stack start are intact - its already started.

run starts a _fresh_ container_ from the image, using the exact configuration from the "applied" docker-compose in that very moment - and by default, its using -f docker-compose.yml

Why those can be different now, is obvious but its simply out of scope that i would help with those here.

I am glad @motin you shared your idea and solution with the community - but this topic was not about sharing initially, it was about "please help me with docker". So a very different kind of thing.

---

so you can thumb down whatever you want there - if you actually know what you are doing, why even ask for help in the first place at all? @zedtux said he knows that all - nothing new- there is nothing he does not know. so whats the issue for then? Why involving docker-sync into this at all?

This issue has not been started as "i share my issue and my answer for people to find and use it".

[motin]

This is a 100% docker-compose topic, has nothing to do with docker-sync at all.

Actually, this issue is actually about both docker-compose and docker-sync. It is about how to be able to take advantage of the genius sync logic in docker-sync, while still be able to use docker-compose "as usual".

Since neither ordinary docker volumes and https://github.com/IFSight/d4m-nfs etc does not break docker-compose commands, it is fair to expect that docker-sync wouldn't neither.

For all other comments, let's just say we have different ideas of how to respond to issues on github and leave it at that :)

[EugenMayer]

@motin we do not agree, but let me explain.

docker-sync start does not imply any of the above magic. So you then have to run

docker-compose -f .yml -f -dev.yml up to actually get the stack up and running.
All explained here https://github.com/EugenMayer/docker-sync/wiki/2.-Configuration#docker-composeyml--docker-compose-devyml.

So beside the convenience of docker-sync-stack start there is nothing docker magic here.

So if you are going to utilize docker-compose outside of docker-sync-stack you have to know what you are doing - no strings attached.

docker-sync has not been build to remove and wrap the burden of understanding docker - not at all. And there are quiet some people want it to be that way.

Its only a tool to get across some inconvenience of d4m performance. Anything else is docker and actually, reading a dev.yml and knowing what a volume is - its very transparent.

[metaskills]

FWIW I found this thread really helpful. Just a reminder that there are people like me capable of searching issues and asking questions like @zedtux but maybe (also like me) are not super knowledgable on the topics here. So maybe what felt like open source maintainer vampirism for lateral support was as @motin said, a merger of two topics and has value.

I would also like to point out that if Docker did have a good Mac file system that docker-compose run would be (IMO) a very sane default way to interface with the project and mirror how we treat Rails locally.