Apollo: Can not start any of the Docker images

[yosuah]

I would like to build and try the Apollo Docker image, but I keep receiving the same error message no matter how I try:

Unable to find image 'users:latest' locally
docker: Error response from daemon: pull access denied for users, repository does not exist or may require 'docker login'.
See 'docker run --help'.

I have set up Docker and I am logged in to Docker Hub. I tried starting Apollo three different ways:

• using the docker/scripts/release_start.sh script
• using the docker/scripts/dev_start.sh script
• by building a local Docker image using docker/build/build_dev.sh ./dev.x86_64.dockerfile script and then trying to start it using docker/scripts/dev_start.sh --local

However in each case the process fails with the error message Unable to find image users:latest. This is strange and I can not figure out where this reference to users:latest comes from. I am not even convinced this is a real image and I suspect this may be the result of an incorrect parameter somewhere, but I can not find the source.
I am running an up-to-date Ubuntu 18.04 LTS, a fresh install of Docker, and had the same issue with both the current master and the r2.5.0 branches of Apollo.

Any help would be much appreciated.

$ docker/scripts/dev_start.sh --local
map_volume-sunnyvale_big_loop-latest: Pulling from apolloauto/apollo
Digest: sha256:68ab11db001d315fcd896a06d9100e6fa6ac0f2d578b6988a4e194b02cee62da
Status: Image is up to date for apolloauto/apollo:map_volume-sunnyvale_big_loop-latest
9d569bb50e2f7e8db35704b74087a5469875943a2b07b82fdfb5228aca3f1688
map_volume-sunnyvale_loop-latest: Pulling from apolloauto/apollo
Digest: sha256:1337ca5d357f33318e38368b6218d343f6138c63a9bb491175ca7cdd39c1a789
Status: Image is up to date for apolloauto/apollo:map_volume-sunnyvale_loop-latest
1a8dfa208ec93164813e8bf5c9fc8c884cbda43c67db2ddd2ed2ab04bee73555
[INFO] Start docker container based on local image : apolloauto/apollo:local_dev
localization_volume-x86_64-latest: Pulling from apolloauto/apollo
Digest: sha256:c9d98694aa3ff68d069d05308ad1562b96c2f069bf17e949e3b26772fb2023b9
Status: Image is up to date for apolloauto/apollo:localization_volume-x86_64-latest
89df31d36a97dc8d99d7d6f449796c234251ac17df22586f5b06ab83f7cba992
yolo3d_volume-x86_64-latest: Pulling from apolloauto/apollo
Digest: sha256:96a4fdbac89a738568ba186f0c8dc6b1245e533b8e12cb20ebce64445530f1f6
Status: Image is up to date for apolloauto/apollo:yolo3d_volume-x86_64-latest
68649bec73e0a7d09feb680750ad0be354d51ccf516beb0aa9bb85551131c3ea
[INFO] Starting docker container "apollo_dev" ...
Unable to find image 'users:latest' locally
docker: Error response from daemon: pull access denied for users, repository does not exist or may require 'docker login'.
See 'docker run --help'.
[ERROR] Failed to start docker container "apollo_dev" based on image: apolloauto/apollo:local_dev

$ docker ps
CONTAINER ID        IMAGE                                                    COMMAND             CREATED             STATUS              PORTS               NAMES
bb75ef8c3d2f        apolloauto/apollo:yolo3d_volume-x86_64-latest            "/bin/sh"           23 minutes ago      Up 22 minutes                           apollo_yolo3d_volume
5dc015215662        apolloauto/apollo:localization_volume-x86_64-latest      "/bin/sh"           23 minutes ago      Up 23 minutes                           apollo_localization_volume
eab9e274e2f9        apolloauto/apollo:map_volume-sunnyvale_loop-latest       "/bin/bash"         23 minutes ago      Up 23 minutes                           apollo_map_volume-sunnyvale_loop
4d4933095ed0        apolloauto/apollo:map_volume-sunnyvale_big_loop-latest   "/bin/sh"           23 minutes ago      Up 23 minutes                           apollo_map_volume-sunnyvale_big_loop



md5-27e03b4fd11a528ca1fbf10e25dbdfa0



$ docker images
REPOSITORY          TAG                                    IMAGE ID            CREATED             SIZE
apolloauto/apollo   dev-x86_64-20180604_1620               92c9630a574e        About an hour ago   5.54GB
apolloauto/apollo   local_dev                              92c9630a574e        About an hour ago   5.54GB
<none>              <none>                                 bc618ad0387b        5 hours ago         3.18GB
apolloauto/apollo   map_volume-sunnyvale_big_loop-latest   3eb7011c5a63        2 days ago          994MB
apolloauto/apollo   dev-x86_64-20180530_1312               0f85ef416c93        4 days ago          7.32GB
nvidia/cuda         8.0-cudnn7-devel-ubuntu14.04           49e1500149ca        2 weeks ago         2.14GB
apolloauto/apollo   dev-x86_64-20180413_2000               2d338b6e1942        7 weeks ago         7.17GB
hello-world         latest                                 e38bc07ac18e        7 weeks ago         1.85kB
apolloauto/apollo   localization_volume-x86_64-latest      75cf8c77134c        2 months ago        6.94MB
apolloauto/apollo   yolo3d_volume-x86_64-latest            562d2b2b5a71        2 months ago        70.6MB
apolloauto/apollo   map_volume-sunnyvale_loop-latest       36dc0d1c2551        4 months ago        906MB
apolloauto/apollo   release-x86_64-v2.0.1                  d778cd0bdb67        4 months ago        7.19GB



md5-27e03b4fd11a528ca1fbf10e25dbdfa0



$ docker --version
Docker version 18.05.0-ce, build f150324



md5-27e03b4fd11a528ca1fbf10e25dbdfa0



$ docker login
Authenticating with existing credentials...
Login Succeeded



md5-27e03b4fd11a528ca1fbf10e25dbdfa0



$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"



md5-27e03b4fd11a528ca1fbf10e25dbdfa0



$ uname -a
Linux adas326linux 4.15.0-22-generic #24-Ubuntu SMP Wed May 16 12:15:17 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[gengqx]

Just docker/scripts/dev_start.sh or if in China docker/scripts/dev_start.sh -C

[Xiaoyang-Rebecca]

Your apollo_dev container is not open yet
If success, should have 5 containers rather than 4
also try exit from the current docker

[yosuah]

Thanks for the quick answers. I removed all containeres and images from docker using docker rm -f $(docker ps -a -q) and docker rmi -f $(docker images -q), restarted the docker service and ran docker/scripts/dev_start.sh and got the same error. (I am not in China, so I did not use the -C option.) I tried the same procedure both on master and r2.5.0 branches with the same result. Do you have any other ideas? Do you know what users:latest could be referring to in the error message?

Here is the full error log for the fresh pull:

$ docker/scripts/dev_start.sh 
map_volume-sunnyvale_big_loop-latest: Pulling from apolloauto/apollo
ff3a5c916c92: Pull complete 
35d9a7f5de96: Pull complete 
Digest: sha256:68ab11db001d315fcd896a06d9100e6fa6ac0f2d578b6988a4e194b02cee62da
Status: Downloaded newer image for apolloauto/apollo:map_volume-sunnyvale_big_loop-latest
8faf65c310bf045c30aaa66f34773ca22f495e8cd1dfc974e81e16c97404df13
map_volume-sunnyvale_loop-latest: Pulling from apolloauto/apollo
7ee37f181318: Pull complete 
df5ffabe5e97: Pull complete 
ae2040ed51a1: Pull complete 
3ce7010d244b: Pull complete 
2538b201d2a6: Pull complete 
de601066203a: Pull complete 
Digest: sha256:1337ca5d357f33318e38368b6218d343f6138c63a9bb491175ca7cdd39c1a789
Status: Downloaded newer image for apolloauto/apollo:map_volume-sunnyvale_loop-latest
b9c6065e47726b4e81ee51a9b3368a483dcb0475f627909dc5283fc61a9e4139
[INFO] Start pulling docker image apolloauto/apollo:dev-x86_64-20180530_1312 ...
dev-x86_64-20180530_1312: Pulling from apolloauto/apollo
286b175c283f: Pulling fs layer 
01ff24a32906: Pulling fs layer 
a1a601a05071: Pulling fs layer 
baa11bb80080: Waiting 
481b7b6a0535: Pull complete 
39bb3ff41a5b: Pull complete 
133c02592fd3: Pull complete 
d0dd998b15f8: Pull complete 
812877fefe39: Pull complete 
b4d443b5ade3: Pull complete 
cd27e0317998: Pull complete 
87f0ec12eef9: Pull complete 
fe5a872a7f1b: Pull complete 
922a475dfe27: Pull complete 
8057ed981ae1: Pull complete 
133cdb28630d: Pull complete 
02b265bf90b4: Pull complete 
940f1805e4f0: Pull complete 
9d46c6ce503b: Pull complete 
ee63b043510d: Pull complete 
b75f1cff3401: Pull complete 
420883ba84f2: Pull complete 
8afecd8a5b64: Pull complete 
f6581db08701: Pull complete 
151610eb1e1f: Pull complete 
2cd3bb011f3d: Pull complete 
bf5c8ac23d3d: Pull complete 
d14f758b1e97: Pull complete 
82e68e5a0f55: Pull complete 
21759bbf577b: Pull complete 
7fab12d9471e: Pull complete 
b1a72f26c56a: Pull complete 
8d03371a8ecc: Pull complete 
ac1e2b360f10: Pull complete 
3bd498af354b: Pull complete 
864311fb9c50: Pull complete 
ab0ee1921042: Pull complete 
28a5e7535a2b: Pull complete 
1a10fa6d4ef3: Pull complete 
b0bfe53e52cf: Pull complete 
e1431f7430ad: Pull complete 
af9bf420d52a: Pull complete 
f63af21a3306: Pull complete 
bfa87fcdbeb7: Pull complete 
315b5c3dbfe1: Pull complete 
9dbde3c8fb68: Pull complete 
Digest: sha256:081be280c464eb3bf5f41f3c170c61708b68d90abf765101c6c3cf5814e03d74
Status: Downloaded newer image for apolloauto/apollo:dev-x86_64-20180530_1312
localization_volume-x86_64-latest: Pulling from apolloauto/apollo
ff3a5c916c92: Already exists 
d259d37176a8: Pull complete 
Digest: sha256:c9d98694aa3ff68d069d05308ad1562b96c2f069bf17e949e3b26772fb2023b9
Status: Downloaded newer image for apolloauto/apollo:localization_volume-x86_64-latest
ef1282caeb224a55fb740b2f3c342aa1ed3ed5ca8ced4da25947d8832f2940ad
yolo3d_volume-x86_64-latest: Pulling from apolloauto/apollo
ff3a5c916c92: Already exists 
d65535c2c0d5: Pull complete 
Digest: sha256:96a4fdbac89a738568ba186f0c8dc6b1245e533b8e12cb20ebce64445530f1f6
Status: Downloaded newer image for apolloauto/apollo:yolo3d_volume-x86_64-latest
a1e7a66ca4abf589f2efa697b55ddd6c55edf0c2425bf65270f4512defac7339
[INFO] Starting docker container "apollo_dev" ...
Unable to find image 'users:latest' locally
docker: Error response from daemon: pull access denied for users, repository does not exist or may require 'docker login'.
See 'docker run --help'.
[ERROR] Failed to start docker container "apollo_dev" based on image: apolloauto/apollo:dev-x86_64-20180530_1312

[jinghaomiao]

Please delete all local containers and docker images and do it again.

[yosuah]

As described above, that was exactly what I tried and it did not help, however I did find the solution eventually.

The issue was found to be within the dev_start.sh script: the user I am logged in with is part of multiple groups, and the script passes the output of $(id -g -n) unescaped to the Docker command, so it created an invalid command trying to run the final container. I fixed the problem by changing line 269 of dev_start.sh to -e DOCKER_GRP="$GRP" (note the added quotes around the variable) and now the container started correctly. (Apart from the fact that I have a dot in my username, which caused further errors in other scripts).

I continued running sudo docker/scripts/dev_into.sh, ./apollo.sh build and ./apollo.sh release to build everything and I could open Dreamview on localhost:8888.

I tried replaying an existing demo recording, as described in the demo docs, but it failed as the rosbag command could not be found. Does is mean ROS is not installed correctly on my image?

# rosbag play demo_2.0.bag --loop
bash: rosbag: command not found

I also tried following the quick start guide but could not get it to work - sometimes the only thing I see is Please send car initial position and map data, and I could somehow randomly get a car on screen, but I could never start navigation or do anything meaningful with it, as GPS and CAN was always marked as NOT_READY on the Module Controller page. I know these are unrelated problems, but if you have any ideas on how to get it to work finally, I would still appreciated it.

[yosuah]

I had to run source scripts/bootstrap.sh on top of all the other commands to get rosbag in the current namespace and be able to replay bag files. I do not know why it does not work after a simple call to bash scripts/bootstrap.sh, but I have seen other people having the same issue.
I will close this issue now, thanks.