Docker-gitlab: docker-compose up = Cannot start container, failed to create endpoint gitlab_gitlab_1

[burkeazbill]

Any idea why I would be getting the following when attempting to issue the docker-compose up command?

ERROR: Cannot start container b85e95d2b2680c199d448a65713ba60285da48b85a3ce67de4bc4512ec944f70: failed to create endpoint gitlab_gitlab_1 on network bridge: COMMAND_FAILED: '/sbin/iptables -t nat -A DOCKER -p tcp -d 0/0 --dport 10080 -j DNAT --to-destination 172.17.0.4:80 ! -i docker0' failed: iptables: No chain/target/match by that name.

I first tried with customizing the docker-compose.yml and thought I had messed something up. I then tried with the provided yml file and got same error.

[sameersbn]

This is a docker/iptables/host issue. Not related to this container image in particular. From the logs is appears that iptables cannot find the DOCKER chain or it does not exist. Please see https://github.com/docker/docker/issues/10218 for some ideas or possible solutions on this issue.

[sameersbn]

p.s. If you are able to resolve the issue, then please let us know the resolution for the benefit of other users that may face the same issue. Thanks/

[burkeazbill]

That certainly helped since I'm also running CentOS 7.
Okay, took a bit of digging around since I'm new to docker and CentOS 7 way of managing stuff.

My first path of resolution had me disabling firewalld AND adding a command line option of --iptables=false to /etc/sysconfig/docker, but I didn't like this approach so I took a step back and re-evaluated the error message - time to learn a tiny bit more about iptables. The command essentially tells us exactly what we need to do in order for the command to work: we need to pre-create the DOCKER chain, and it needs to be in the "nat" table.

Before running docker-compose up, run the following:
iptables -t nat -N DOCKER

Ideally, if your scripting can be updated to check for the existence of the DOCKER chain in the nat table and create it if not found, that would save an extra manual step. Sample script:

!/bin/bash

if ! iptables -t nat -n --list DOCKER 2>&1 >/dev/null; then
echo "DOCKER chain not found on nat table, creating now..."
iptables -t nat -N DOCKER
fi

An additional issue (and resolution) I ran into was this:
gitlab_1 | sed: -e expression #1, char 31: unknown option to `s'
gitlab_gitlab_1 exited with code 1

I saw a reply to this in another thread here, while I had the correctly formatted string: America/Los_Angeles, specifying Los_Angeles on the second entry in the docker-compose.yml did not work, neither did America/Los_Angeles. I had to specify as: America/Los_Angeles
That is a in front of the / to escape it..

[sameersbn]

Ideally, if your scripting can be updated to check for the existence of the DOCKER chain in the nat table and create it if not found, that would save an extra manual step

This is a configuration on the host. We do not and cannot alter any host configuration directly.
If I understand correctly, when docker is installed it automatically creates the DOCKER chain, I don't know why this was not created in your case, maybe because you are using centos (i am not sure).

I had to specify as: America/Los_Angeles

GITLAB_TIMEZONE is used to set the timezone on the rails app. This is a mapped value as listed here http://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html. In your case if you wanted to set the timezone to America/Los_Angeles you should set this value to Pacific Time (US & Canada) while escaping the & character so that sed is not confused. In short GITLAB_TIMEZONE=Pacific Time (US \& Canada) is what you should set.

As an aside, the TZ variable in the docker-compose.yml is a bash environment variable. This should be set to America/Los_Angeles like you normally would. This basically effects the schedule of the cron jobs so that the jobs execute relative to your local time, which otherwise is as per the GMT time.

[sameersbn]

The & character is now automatically escaped. If you have manually escaped the & character, then you need to remove it in versions > 8.1.x.

[angeloh]

I got a similar error in my docker setup. It is not related to this issue. But if someone sees this error when you run docker run ..., you can try sudo service docker restart.

[sakhinov]

I just had the same issue on CentOS 7. I figured out that docker service is configured that way (/lib/systemd/system/docker.service):

After=network.target docker.socket
Requires=docker.socket

It means, that docker service starts right after network.target started, and doesn't wait for network.target finished (including firewalld.service). Therefore firewalld.service actually started in parallel or after docker.service rewriting its nat rules.
Changing docker.service settings to these solved the problem (/lib/systemd/system/docker.service):

After=network.target docker.socket
Requires=network.target docker.socket