Harbor: docker-compose stop/start failing to start mysql container

[biolounge]

Restarting of Harbor always failing to start MySQL container. Followed the procedure in https://github.com/vmware/harbor/blob/master/docs/installation_guide.md#managing-harbors-lifecycle

$docker-compose stop
Stopping deploy_proxy_1 ... done
Stopping deploy_ui_1 ... done
Stopping deploy_registry_1 ... done
Stopping deploy_mysql_1 ... done
Stopping deploy_log_1 ... done
Stopping deploy_jobservice_1 ... done

Successfully stopped all containers then rebooted the server.

$docker-compose start
Starting log ... done
Starting ui ... done
Starting mysql ... error
Starting jobservice ... done
Starting registry ... done
Starting proxy ... error

ERROR: for mysql Cannot start service mysql: mkdir /var/run/docker/libcontainerd/containerd/545880b0e8e6e0512543b36486446606d40ee588170c4f795472dff98654a054: file exists

Also, can't remove mysql container.

$docker-compose down
Removing deploy_proxy_1 ... done
Removing deploy_jobservice_1 ... done
Removing deploy_registry_1 ... done
Removing deploy_ui_1 ... done
Removing deploy_mysql_1 ... error
Removing deploy_log_1 ... done

ERROR: for deploy_mysql_1 Driver devicemapper failed to remove root filesystem 545880b0e8e6e0512543b36486446606d40ee588170c4f795472dff98654a054: Device is Busy
Removing network deploy_default
WARNING: Network deploy_default not found.

---

$ docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:
OS/Arch: linux/amd64

Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:
OS/Arch: linux/amd64

docker-compose --version
docker-compose version 1.8.0, build f3628c7

[yhua123]

Hi Biolounge,
You install harbor first time or already have harbor, upgrade to new code?
I guess this maybe caused you have old mysql data struct, you can mv /data/database to /data/database.bak and try again.

[yhua123]

Hi Biolounge,
It's ok?

[biolounge]

Hi yhua123,

Actually, I didn't upgrade the code. Just stopped containers using docker-compose and restarted the server. After server restart, mysql container is not starting and shows the below error.

ERROR: for mysql Cannot start service mysql: mkdir /var/run/docker/libcontainerd/containerd/545880b0e8e6e0512543b36486446606d40ee588170c4f795472dff98654a054: file exists

[levchik]

Same problem here +1.

$ docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:22:43 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:22:43 2016
OS/Arch: linux/amd64

$ docker-compose --version
docker-compose version 1.6.0, build d99cad6

[dejwsz]

Hi, the same for me on Centos 7.2, after VM restart mysql and registry do not want to get up:

docker-compose start
Starting log ... done
Starting ui ... done
Starting mysql ... error
Starting jobservice ... done
Starting registry ... error
Starting proxy ... error

ERROR: for registry Cannot start service registry: mkdir /var/run/docker/libcontainerd/containerd/20e0b358e43e5824319661ad3494a140e3cc87f270a4fa72f1cb7b690990426a: file exists

ERROR: for mysql Cannot start service mysql: mkdir /var/run/docker/libcontainerd/containerd/21d0bf0065c81d4c3a4142d77fe6c843fa2b72dee79626299976b1dad3053237: file exists

While trying to remove dead mysql container:

Error response from daemon: Driver devicemapper failed to remove root filesystem 21d0bf0065c81d4c3a4142d77fe6c843fa2b72dee79626299976b1dad3053237: Device is Busy

[dejwsz]

Generally I can see there are problems with restarting things. Mysql and rsyslog containers often do not start after previous stop. In case of log container I could see such error after restart: "rsyslogd: pidfile '/var/run/rsyslogd.pid' and pid 8 already exist". In case of mysql: [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
Nov 16 12:39:14 172.18.0.1 mysql[7957]: 2016-11-16 11:39:14 1 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.

[dejwsz]

Interestingly in case of problems after restart I needed to do such things:

• manually remove all Harbor containers
• in the directory which is mounted as volume I did such thing for Mysql (looks like some problem with permissions?):
  mkdir bak
  mv ibdata1 bak/.
  mv ib_logfile* bak/.
  mv mysql bak
  cp -a bak/ibdata1 ibdata1
  cp -a bak/ib_logfile* .
  cp -a bak/mysql mysql
  And then I did reinstall with install.sh in harbor directory what brings new containers up and running.

[dejwsz]

In case of rsyslog container it is worth to change CMD line in Dockerfile to such one:

CMD crond && rm -f /var/run/rsyslogd.pid && rsyslogd -n

Before rsyslog is started the preexisting PID file is removed. So after restarting the log container there is no problem with existing PID file and rsyslogd starts without any problems.

[davidstack]

the same proplem. with this porblem ,the harbor can not be used.

[yhua123]

Please post your log information.
Or check your data directory. Maybe caused by previous data structure.
solution:
1 migration data
https://github.com/vmware/harbor/blob/master/docs/migration_guide.md
2 remove data
mv /data/database /data/database.bak

[dejwsz]

I had investigated it a little bit more. In my case it turned out that the mysql user which is created by the mysql container image had the same UID and GID as the user of systemd on my host. So the mysql data files placed in the directory which was used as a volume for database container was owned by systemd and that fact messed up a little. Harbor was not able to stand up or restart automatically and so on.
The solution for me was such: I took the mysql container out of docker hub and changed the line with user creation to:

ENV GROUP_UID 1020
ENV USER_UID 1020
RUN groupadd -r mysql --gid ${GROUP_UID} && useradd -r -g mysql mysql --uid ${USER_UID}

You can pass some custom UID and GID if needed to have no overlaps. Later I have created a local copy of harbor-db container definition where I changed the base to the modified mysql container plus I needed to add a line "ADD registry.sql /docker-entrypoint-initdb.d/" to have it initialized properly. After these changes Harbor is restarting very well and I don't have any problems with permissions with mysql data on attached volume.

[yhua123]

Thanks for your contribution~

[yhua123]

1163 follow rsyslogd.pid issue