Boulder: oci runtime error: no such file or directory

[frankienicoletti]

I am trying to run the Boulder containers locally. After cloning a fresh copy (I'd also deleted all docker containers and images from previous attempts), and running docker-compose up this is the output:

Pulling bhsm (letsencrypt/boulder-tools:2016-08-03)...
2016-08-03: Pulling from letsencrypt/boulder-tools
357ea8c3d80b: Already exists
52befadefd24: Pull complete
3c0732d5313c: Pull complete
fee55c622298: Pull complete
85155ee2fbc2: Pull complete
c51febe84798: Pull complete
52609aaab90b: Pull complete
6bce3cb224fd: Pull complete
2bd147de227f: Pull complete
Digest: sha256:df628ba1235b551c4451da0e4d796f3f8dd590a5ce1b43d1357fa6a6d23e1e10
Status: Downloaded newer image for letsencrypt/boulder-tools:2016-08-03
Building boulder
Step 1 : FROM letsencrypt/boulder-tools:2016-08-03
 ---> 7515d7ed729e
Step 2 : EXPOSE 4000 4002 4003 8053 8055
 ---> Running in 7bafcc9257ad
 ---> f079f46d3fde
Removing intermediate container 7bafcc9257ad
Step 3 : ENV PATH /go/bin:/usr/local/go/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/
 ---> Running in a5749b003134
 ---> 35ad0752ffb9
Removing intermediate container a5749b003134
Step 4 : ENV GOPATH /go
 ---> Running in c705a56f5a49
 ---> d85b3e3c79b7
Removing intermediate container c705a56f5a49
Step 5 : RUN adduser --disabled-password --gecos "" --home /go/src/github.com/letsencrypt/boulder -q buser
 ---> Running in df2249616e55
 ---> d0b2d654d7cd
Removing intermediate container df2249616e55
Step 6 : RUN chown -R buser /go/
 ---> Running in 560462ab0ee9
 ---> db1267983d2f
Removing intermediate container 560462ab0ee9
Step 7 : WORKDIR /go/src/github.com/letsencrypt/boulder
 ---> Running in 27556b1afe30
 ---> 815f7d66d4a8
Removing intermediate container 27556b1afe30
Step 8 : COPY . .
 ---> 21eb7d3030b8
Removing intermediate container 23d3fadf9e6b
Step 9 : RUN mkdir bin
 ---> Running in 59db70efdc04
 ---> f56714c62fa4
Removing intermediate container 59db70efdc04
Step 10 : RUN GOBIN=/usr/local/bin go install ./cmd/rabbitmq-setup
 ---> Running in d4c4eb63ffc1
 ---> e3fa939dcc45
Removing intermediate container d4c4eb63ffc1
Step 11 : RUN chown -R buser /go/
 ---> Running in cb39e5a73744
 ---> a1feac83c4c3
Removing intermediate container cb39e5a73744
Step 12 : ENTRYPOINT ./test/entrypoint.sh
 ---> Running in 38290363865e
 ---> 66d216b54744
Removing intermediate container 38290363865e
Successfully built 66d216b54744
WARNING: Image for service boulder was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating boulder_bhsm_1
Creating boulder_bmysql_1
Creating boulder_brabbitmq_1
Creating boulder_boulder_1

ERROR: for boulder  Cannot start service boulder: oci runtime error: no such file or directory
ERROR: Encountered errors while bringing up the project.

As suggested, I also included the versions for docker and docker-compose:

docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 21:15:28 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 21:15:28 2016
 OS/Arch:      linux/amd64

docker-compose version
docker-compose version 1.8.0, build f3628c7
docker-py version: 1.9.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.2h  3 May 2016

Thanks for your help!

[jsha]

Googling around, it sounds like one cause may be apparmor profiles: https://github.com/docker/docker/issues/21702.

Closer to home, I think we have sometimes run into this when the file that docker-compose is supposed to run doesn't exist. It's supposed to run /go/src/github.com/letsencrypt/boulder/test/entrypoint.sh, where /go/ is mounted from $GOPATH on the host. Do you have GOPATH on the host set to a directory with the Boulder code checked out (we should document this)?

[cpu]

Googling around, it sounds like one cause may be apparmor profiles: docker/docker#21702.

I don't think this could be the culprit on an OSX host.

[rolandshoemaker]

I've also had the exact same issue on a Ubuntu 16.04 machine running Docker 1.12.0. $GOPATH was properly set, and I retried with a hard coded directory path and that didn't fix anything.

Not really sure _what is going on_.

[frankienicoletti]

Follow up: It is working now! My GOPATH was malformed and I think that was the main problem. I deleted everything and ran a go get to grab the repo again, now it's working. Checking for the entrypoint.sh file helped me figure that out. Thank you!

[jsha]

Excellent!

[rolandshoemaker]

This can be caused by having broken symlinks in your GOPATH. It seems like Docker trys to follow symlinks when mounting volumes and will choke if it encounters something that doesn't point to anything real.

It looks like a number of Golang tools include broken symlinks in their repos as part of their test suites (i.e. https://github.com/FiloSottile/gvt/blob/master/fileutils/_testdata/copyfile/a/rick) which can trigger this issue. Finding and deleting the offending symlinks will fix the issue, but this seems like a subpar solution.

[rolandshoemaker]

Possibly related to docker/docker#17944, limited local testing seems to indicate that pre-1.12.0 versions work fine.

[cpu]

@rolandshoemaker That would make sense, I'm on 1.10.3 and haven't been able to reproduce.

@katejefferson Glad your environment is working now. I'm going to close this issue since you're unblocked and it seems to be Docker/$GOPATH related. Please open more issues if you run into any other trouble!

[hanneshapke]

@katejefferson Thank you for posting your question. I ran into the same issue and discovered that I was linking to the wrong volume.