I am doing something pretty simple. I have a docker-compose.yml file:
tsdb:
build:
context: ./tsdb_prod
....
and in the Dockerfile in that directory I have:
FROM influxdb:0.13
COPY file_prod /etc/file_prod
and I have the file_prod file in that directory.
running docker-compose build with --no-cache to rebuild my image after changes:
0s ๐ $> docker-compose build --no-cache
[snip]
Building tsdb
Step 1 : FROM influxdb:0.13
---> c7a58aa7ce3b
Step 2 : COPY file_prod /etc/file_prod
---> ff2aa86452fd
Removing intermediate container 91c3eafd328c
Successfully built ff2aa86452fd
2s ๐ $> docker run --rm -it ff2aa86452fd /bin/bash
root@0e660d6a1784:/# ls /etc/file_prod
/etc/file_prod
But when I do my docker-compose up I see
2s ๐ $> docker-compose up
Recreating task_tsdb_1
[snip]
tsdb_1 | run: open server: open service: open "/etc/file_prod": no such file or directory
I'm pretty new to Docker. I'm sure I'm missing something stupid here. Is it building one image but starting another? I do see
2s ๐ $> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f742363da3aa 58fcf34dd924 "/entrypoint.sh influ" 44 minutes ago Exited (1) 34 minutes ago task_tsdb_1
[snip]
Not sure what's going on with that. I would appreciate anyone's insight.
OK. I found something last night that seemed to explain and fix this, and now I have the same problem again this morning. Grr.
more of my docker-compose.yml:
version: '2'
services:
tsdata:
build:
context: ./tsdb_prod
volumes:
- /var/lib/influxdb
command: "true"
tsdb:
build:
context: ./tsdb_prod
restart: always
environment:
[snip]
volumes_from:
- tsdata
ports:
[snip]
My docker-compose build output:
3s ๐ $> docker-compose build --no-cache
Building tsdata
Step 1 : FROM influxdb:0.13
---> c7a58aa7ce3b
Step 2 : COPY forelight_data.pem /etc/file_prod
---> 22d88415e029
Removing intermediate container 115d46c1e419
Successfully built 22d88415e029
Building tsdb
Step 1 : FROM influxdb:0.13
---> c7a58aa7ce3b
Step 2 : COPY forelight_data.pem /etc/file_prod
---> 78a43a107f4e
Removing intermediate container fa2aba9694e4
Successfully built 78a43a107f4e
35s ๐ $> docker run --rm -it 78a43a107f4e /bin/bash
root@ea6546277008:/# ls /etc/file_prod
/etc/file_prod
My docker-compose up output:
1s ๐ $> docker-compose up
Creating task_tsdata_1
Creating task_tsdb_1
Attaching to task_tsdata_1, task_tsdb_1
[snip]
tsdb_1 | run: open server: open service: open "/etc/file_prod": no such file or directory
same dockerfile as above.
so the file is there in the image, but the container docker-compose produces doesn't have it... ?
Can you try running docker-compose down -v --rmi
then run build
/ up
again?
I can and I will, after I deal with a couple other things. Though that
sounds like it will destroy my data volume. Which is fine... now. Any
solution that will involve that is pretty much out after this week, though.
Does compose get wedged this way often? If that's the case I'm inclined to
go with a host directory instead of a data volume.
On Wed, May 18, 2016 at 4:35 PM Joffrey F [email protected] wrote:
Can you try running docker-compose down -v --rmi then run build / up
again?โ
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/3472#issuecomment-220165334
No - sorry about the misunderstanding. I'm not proposing it as a solution _per se_, just as a debug step.
On somewhat related note, your config could potentially be made quite simpler by using named volumes instead of the volumes_from
trickery you're currently using. And it would allow you to manage volumes independently of containers, too.
HTH
Nope. Same message.
I went back and deleted all my images except my copy of the base influxdb image. Then I did docker-compose build and up and got the same message. I list my images now and get
4s ๐ $> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
task_tsdata latest 2fba08d97ba7 24 seconds ago 234.7 MB
task_tsdb latest 2fba08d97ba7 24 seconds ago 234.7 MB
influxdb 0.13 c7a58aa7ce3b 7 days ago 234.7 MB
so it's finding my dockerfile from the context, finding the InfluxDB image, but not copying the file... ?
Looking at the named volumes doc... the volumes_from 'trickery' is suggested by a lot of sources. Not that I'm attached to it (this week, anyway). Having an explicit way to declare a volume does look cleaner.
Though I will say, the 'external' parameter is confusing. If they mean it mounts on the Docker host, then where? It looks like it puts it in the same place specified in the referring service. But then if there are multiple services referring to the same volume trying to locate it in different places does it error out? That seems inconsistent with Docker's design in general. I suspect I'm misunderstanding something.
ok closer...
0s ๐ฉ $> docker-compose build
Building tsdata
Step 1 : FROM influxdb:0.13
---> c7a58aa7ce3b
Step 2 : COPY file_prod /etc/file_prod
---> Using cache
---> 0914f84be4a1
Successfully built 0914f84be4a1
Building tsdb
Step 1 : FROM influxdb:0.13
---> c7a58aa7ce3b
Step 2 : COPY file_prod /etc/file_prod
---> Using cache
---> 0914f84be4a1
Successfully built 0914f84be4a1
0s ๐ $> docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
task_tsdata latest 0914f84be4a1 About a minute ago 234.7 MB
task_tsdb latest 0914f84be4a1 About a minute ago 234.7 MB
influxdb 0.13 c7a58aa7ce3b 7 days ago 234.7 MB
so it's copying the file (and I can make it bomb if I introduce a misspelling in the filename in my dockerfile).
But the file isn't showing up in my container.
And also I don't see a clear way to answer the question: 'what image is container x based on?'
aha, inspect. No surprises... but no explanations either.
0s ๐ $> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
01bf4a397710 task_tsdb "/entrypoint.sh influ" 52 minutes ago Restarting (1) 25 minutes ago 8083/tcp, 8086/tcp task_tsdb_run_3
0s ๐ $> docker inspect 01bf4a397710 | grep Image
"Image": "sha256:0914f84be4a19ba5ad25f7404ede741c06afe63ddb06374d855df438c33a925a",
0s ๐ $> docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
task_tsdata latest 0914f84be4a1 About an hour ago 234.7 MB
task_tsdb latest 0914f84be4a1 About an hour ago 234.7 MB
influxdb 0.13 c7a58aa7ce3b 7 days ago 234.7 MB
and when I
0s ๐ $> docker run --rm -it 0914f84be4a1 /bin/bash
root@6e93d6b39419:/# ls /etc/file_prod
/etc/file_prod
root@6e93d6b39419:/# exit
and then I do docker-compose up and see the same message.
That's weird. I'll take a closer look and update when I have more info.
Looking at the named volumes doc... the volumes_from 'trickery' is suggested by a lot of sources. Not that I'm attached to it (this week, anyway). Having an explicit way to declare a volume does look cleaner.
Yeah, named volumes didn't exist and volume management through the API is a pretty recent feature, so that was indeed the conventional solution for a long time.
Though I will say, the 'external' parameter is confusing.
In that context, external
just means it's not managed by Compose (i.e. it will not attempt to create or destroy it at any point, and its name will not be namespaced).
actually, that's kind of ideal. I can create a volume with docker volume and use that, and then when I get compose working I can just use external. Nice.
More messing around:
I started messing with docker volume, and found I had a huge list of volumes I had created in the course of my messing around. So I l deleted all of those, and cleaned out all my containers and images, except the base one I got from Docker Hub, redid build and up, and... got the same result.
Then I switched to your recommended volume configuration, and upgraded my Docker client from 1.10 to 1.11.1, and cleaned everything out again, containers, images, volumes, and... got the same result.
Then I deleted everything including my base image and started again and... got the same result.
So I am out of bullets. I'm going back to my direct docker invocations (which work) for now and using those. If you want I can send you my dockerfile directly--I'd rather not post it here. See if you can reproduce it.
Sure, you can send it to [redacted]. I've tried to create a simple reproduction case but everything worked fine.
I'm having the same problem here.
Further, if I docker run image
, the file is there. If I docker-compose up
, the file is not there.
Exact same behavior on the latest beta on OSX and on Windows.
PS C:\Users\pedro> docker-compose version
docker-compose version 1.7.1, build 0a9ab35
docker-py version: 1.8.1
CPython version: 2.7.11
OpenSSL version: OpenSSL 1.0.2d 9 Jul 2015
@pedrovanzella InfluxDB as well?
Nope, it's a rails app. Base image is rails:latest.
@shin- found it, in my case. In his words from an offline email
The quotes around the path weren't being parsed out, and as such influxdb was trying to access
"/etc/...
, which, unsurprisingly, isn't a directory.
I had a bunch of fields quoted that didn't want to be quoted in docker-compose.yml. That was the one that was bombing for me, but there were others.
Anything to add, @shin- ?
@pedrovanzella Make sure the paths are correct - that also means checking for quotes that aren't parsed out, like John pointed out. If it's a different issue, please create a new ticket with more details so we can look into it.
I'm going to close this as the original issue has been solved.
Weird thing is, if I run the app with docker run
, it's fine. If I run it with docker-compose up
, it's not. It's almost as if docker-compose were picking the wrong image to spin the container from - an intermediary image, maybe?
that also means checking for quotes that aren't parsed out
In the Dockerfile or in docker-compose.yml? There's nothing quoted there.
I also tested on a simpler app, by copying first the Gemfile and then the rest of the app. On Windows, running with docker-compose, the image fails to run because Rails can't find the Gemfile. Running with docker run it runs fine. On OSX, it runs fine either way.
I think you need to post something that we could look at to help diagnose
the problem. Describing it isn't enough. (maybe anonymize some stuff for
security's sake).
On Wed, May 25, 2016 at 9:16 AM Pedro Vanzella [email protected]
wrote:
that also means checking for quotes that aren't parsed out
In the Dockerfile or in docker-compose.yml? There's nothing quoted there.
I also tested on a simpler app, by copying first the Gemfile and then the
rest of the app. On Windows, running with docker-compose, the image fails
to run because Rails can't find the Gemfile. Running with docker run it
runs fine. On OSX, it runs fine either way.โ
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/3472#issuecomment-221589627
I've got the same issue with docker-compose 1.7.1
. The server has two websites that are supposed to be behind the dynamic nginx-proxy. They are in /var/www/production-site/
and /var/www/development-site/
. Both are the same git repo just checked out at different commits. Each has docker/docker-compose.yml
and docker/.env
. The name of a container in docker-compose
is dynamic:
version: "2"
services:
web:
build:
context: .
dockerfile: Dockerfile
container_name: app-${MODE}
...
i.e app-dev
and app-prod
; the suffix being taken from .env
.
I have two ssh terminals open in each folder and I'm trying to docker-compose
up
and down
these two websites independently. docker-compose config
gives different results in different folders as expected, $pwd
assures me that I'm in the right folders.
Surprisingly when I docker-compose up -d --build
I'm getting Recreating app-dev
or Recreating app-prod
randomly regardless of what folder I'm in. Can this be to do with some caching?
Tried supplementing my docker-compose.yml
with image name:
version: "2"
services:
web:
image: my-project/my-app-${MODE}
build:
context: .
dockerfile: Dockerfile
container_name: app-${MODE}
...
Result of docker images
after running docker-compose build --no cache
in both folders (/var/www/production-site/docker
/var/www/development-site/docker
); the only difference is $MODE
in .env
:
REPOSITORY TAG IMAGE ID ...
my-project/my-app-dev latest 920eae94d96a ...
my-project/my-app-prod latest bd6d87daa129 ...
However running docker-compose up
in the two dirs does not let me see both containers in docker ps
: it is either app-dev
or app-prod
, starting one kills another. What else can it be if not over-caching by docker-compose up
?
Similar complaints here: https://github.com/docker/compose/issues/1883
@kachkaev This isn't a bug, and has nothing to do with caching.
Both your directories are named the same thing (docker
), so when you run docker-compose up
in either directory you're working on the same project. Each time you're updating the configuration of the same service (web
) to change both the image name and container name.
It looks like what you actually want is two separate apps, each with its own image and container. In that case, I suggest doing the following:
--project-name
option or COMPOSE_PROJECT_NAME
environment variable to set a different project name for each app. For example, myappdev
and myappprod
.container_name
option from both Compose files - it's unnecessary.If you run docker-compose up
in both directories, you should now have two containers: myappdev_web
and myappprod_web
.
Thanks for your reply @aanand!
Both your directories are named the same thing (docker), so when you run docker-compose up in either directory you're working on the same project.
This is quite strange. I was expecting docker to figure out the name of the project using the full path, which is different (/var/www/production-site/docker
&& /var/www/development-site/docker
) - just looking at the name of one parent dir seems a bit naive. What is the reason for not considering the full path here?
I'll try --project-name
tonight when get to the laptop, thanks for the hint! Is there a way to define project name inside docker-compose.yml
as a derivative of a variable in .env
? E.g. myapp${MODE}
. The main point of using docker-compose for me is to reduce the length of commands to type (I could just go for docker-run
otherwise in this case).
I was expecting docker to figure out the name of the project using the full path, which is different (
/var/www/production-site/docker
&&/var/www/development-site/docker
) - just looking at the name of one parent dir seems a bit naive. What is the reason for not considering the full path here?
Mostly historical. We used to depend entirely on container names for keeping track of what containers belonged to what project, and so we had to squash a reasonably unique identifier into each container name - hence, a munged directory basename. It's a very difficult thing to change while preserving backwards compatibility.
Is there a way to define project name inside
docker-compose.yml
as a derivative of a variable in.env
?
No, though it's a very old discussion - see https://github.com/docker/compose/issues/745 - but putting a COMPOSE_PROJECT_NAME=myappdev
line in .env
will work.
@aanand awesome, it worked! Summary of changes:
.env
:```
# /var/www/development-site/docker
COMPOSE_PROJECT_NAME=development-site
# /var/www/production-site/docker
COMPOSE_PROJECT_NAME=production-site
```
image
and container_name
from docker-compose.yml
Images and containers no longer conflict and I see developmentsite_web_1
and productionsite_web_1
in docker ps
. Awesome!
I am having the same problem, my image is a simple golang image. I can run docker-compose build
without error, but the docker run
works well while docker-compose up
fails all the time.
I have read all your comments above, it seems not useful to me though.
Error message
โ gray-test git:(master) โ docker-compose up
Starting graytest_server_1
Attaching to graytest_server_1
server_1 | sh: 1: /go/bin/app: not found
It tell me that /go/bin/app
is not found, however the file exists if I run docker run -it [IMAGE NAME] /bin/bash
and check the directory.
docker-compose.yml
server:
build: './src'
ports:
- 8988:8988
volumes:
- .:/go
By the way, the volumes
configuration is not making any sense here either. I think this roots for the same cause of failure of docker-compose up
.
Dockerfile
FROM golang:1.8
WORKDIR /go/src/
COPY . .
RUN go-wrapper download graypublish/* # "go get -d -v ./..."
RUN go-wrapper install graypublish/* # "go install -v ./..."
RUN ls /go/bin
CMD ["sh", "-c", "/go/bin/app"] # ["app"]
EXPOSE 8988
Directory structure
โโโ README.md
โโโ docker-compose.yml
โโโ src
โโโ Dockerfile
โโโ graypublish
โโโ api
โย ย โโโ graypublish.go
โโโ app
โย ย โโโ main.go
โโโ db
โโโ db.go
โโโ tracks.go
Can you please give some suggestion?
I am having the exact same issue. But I think it is the 'docker' version. I am using Docker toolbox under OSX. When the VM is a boot2docker.vm image, it works fine. When the VM is a coreos, then the exact same build fails. The files are missing from the /usr/src folder, expect for the node_modules directory.
Here is the Dockerfile:
FROM node:10.5.0
RUN npm install --global yarn
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
ADD package.json /usr/src/app
RUN yarn install
ADD . /usr/src/app
If I use 'docker run image -it /bin/bash' then all of the files are there, if I use 'docker-compose run server /bin/bash' then the only directory under /usr/src/app is node_modules.
Again, this is only an issue when the VM is coreos which is using
docker -v
Docker version 18.03.1-ce, build 9ee9f40
core@my_vm01 ~ $ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.4
Git commit: 9ee9f40
Built: Thu Apr 26 04:27:49 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.4
Git commit: 9ee9f40
Built: Thu Apr 26 04:27:49 2018
OS/Arch: linux/amd64
Experimental: false
I experienced a similar issue, and found this discussion. In my case, I was using a busybox image to persist some data. The issue was resolved by force removing this image and docker-compose build
'ing anew.
I had this issue. For anybody getting a similar thing, I realised I was accidentally binding the local folder (using docker-compose) as a volume into the docker container like this:
volumes:
- ".:/home/node"
Which I must have done early-on, trying to get live-reload working or something. This was confusing the heck out of docker-compose and was showing a "container" version of the filesystem when putting RUN ls -al
statements in Dockerfile
and a "host" version of the filesystem when using CMD ls -al
resulting in completely different outputs. Check to make sure you aren't doing something similar.
Leaving a comment here in case it helps someone else. docker-compose stop
followed by docker-compose build
and docker-compose start
isn't good enough, you have to docker-compose down
, docker-compose build
and docker-compose up
because down/up is when docker actually maps images to containers. If you don't use down/up docker-compose will use the old pre-existing containers.
This is probably obvious to seasoned docker users, but a casual like me didn't understand this nuance and it very well could be the trap many of the commenters on this thread are accidentally falling into.
Most helpful comment
Weird thing is, if I run the app with
docker run
, it's fine. If I run it withdocker-compose up
, it's not. It's almost as if docker-compose were picking the wrong image to spin the container from - an intermediary image, maybe?