Docker-gitlab: after docker-compose restart gitlab exited with error

[flashermaster]

Hi,
After a successful docker-compose up, I'd like to reboot the container.
I tried docker-compose down/stop, then I'd like to start it again I received errors like this:

/volume1/docker/gitlab# docker-compose up
Starting gitlab_postgresql_1 ... 
Starting gitlab_redis_1 ... 
Starting gitlab_postgresql_1
Starting gitlab_redis_1 ... done
Creating gitlab_gitlab_1 ... 
Creating gitlab_gitlab_1 ... done
Attaching to gitlab_postgresql_1, gitlab_redis_1, gitlab_gitlab_1
postgresql_1  | WARNING: no logs are available with the 'db' log driver
redis_1       | WARNING: no logs are available with the 'db' log driver
gitlab_1      | WARNING: no logs are available with the 'db' log driver
gitlab_gitlab_1 exited with code 143
gitlab_1      | WARNING: no logs are available with the 'db' log driver
gitlab_gitlab_1 exited with code 143
......

I watched the log and the last unusual log is:
/sbin/entrypoint.sh: line 22: 558 Terminated /usr/bin/supervisord -nc /etc/supervisor/supervisord.conf

Don't understand the meaning.
My yml file is just the standard info and the first up works well.

Could someone help me with this issue?
Thank you so much.

[adamlc]

I'm getting a similar error too after an upgrade

mysql: [Warning] Using a password on the command line interface can be insecure.
/sbin/entrypoint.sh: line 22:   657 Terminated              /usr/bin/supervisord -nc /etc/supervisor/supervisord.conf

[flashermaster]

@adamlc There is only endless reboot. I have to remove the container and also the data in my volume to make next clean docker-compose up work.

[adamlc]

I've managed to get it working by commenting out the database migration stuff in entrypoint.sh on line 22 and rebuilding the docker image.

Obviously this isn't good because there may well be migrations in 10.8.3. But at least its running now! If there are migrations I doubt it'll be anything major in a point release.

[flashermaster]

@adamlc Thanks.Would you please give me any suggestion of which former version of sameersbn/gitlab would be pretty stable?

[adamlc]

@flashermaster well as of this morning I had been running 10.8.1 until I tried to upgrade 😞

[HsuNick]

I'm also getting the same error ...

[derphilipp]

Downgrading is not really supported afaik, so I will also go for the "deactivate migrations" solutions for now

[flashermaster]

@derphilipp I'm not upgrading from former version to 10.8.3. I've just made a clean installation and a single reboot try. TOT

[derphilipp]

@flashermaster Ah, sorry; I got that wrong

[flashermaster]

@adamlc So now I'll try 10.8.1 to find whether it suits me...

[GytisT]

Anybody willing to share a solution, please?

[flashermaster]

Trying 10.8.1 and I got exit 1 error. Solving....

[solidnerd]

Mhmm strange. I tested it before I was gonna merge it. This could be happen through the introduced lines in last release. https://github.com/sameersbn/docker-gitlab/blob/master/entrypoint.sh#L18 . So we need to find a solution that covers everyone uses case. Could you post the exact steps. From VERSION X to 10.8.3. So I can gonna try to reproduce it.

[LatinCatin]

Hi !

Same problem here for a migration from 10.6.3 to 10.8.3.
I just stop my docker (docker-compose down), upgrade the docker-compose.yml file to 10.8.3 and then run docker-compose up.
After that, the docker container was stuck inside a bootloop with the error "/sbin/entrypoint.sh: line 22: 558 Terminated /usr/bin/supervisord -nc /etc/supervisor/supervisord.conf"

Edit : a migration from 9.5.2 to 10.8.3 works perfectly... Strange.

@solidnerd Thank you for looking into that. Good luck !

[adferrand]

Hello @solidnerd, for me, it was when upgrading from 10.8.1 to 10.8.3. No modifications done, just recreate and start the container.

I commented out lines in /sbin/entrypoint.sh from lines 18 to 23. Made temporary fix by mounting the "sanitized" entrypoint.sh from host to the docker in place.

Error on log is the following (line 5):

juin 04 11:19:32 nuuk b977f18f05f6[801]: Configuring nginx::gitlab...                                                                                              
juin 04 11:19:32 nuuk b977f18f05f6[801]: Configuring nginx::gitlab::hsts...                                                                                        
juin 04 11:19:32 nuuk b977f18f05f6[801]: Configuring nginx::gitlab-registry...                                                                                     
juin 04 11:19:32 nuuk b977f18f05f6[801]: Configuring nginx::gitlab-pages...                                                                                        
juin 04 11:19:32 nuuk b977f18f05f6[801]: /sbin/entrypoint.sh: line 22:  2887 Terminated              /usr/bin/supervisord -nc /etc/supervisor/supervisord.conf     
juin 04 11:19:33 nuuk 9c7bde9c4ff7[801]: Loading /etc/docker-gitlab/runtime/env-defaults                                                                           
juin 04 11:19:33 nuuk 9c7bde9c4ff7[801]: Initializing logdir...                                                                                                    
juin 04 11:19:33 nuuk 9c7bde9c4ff7[801]: Initializing datadir...                                                                                                   
juin 04 11:19:34 nuuk 9c7bde9c4ff7[801]: Updating CA certificates...                                                                                               
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Installing configuration templates...                                                                                     
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: SSL Key, SSL Certificate and DHParam were not found.                                                                      
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Assuming that the container is running behind a HTTPS enabled load balancer.                                              
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: SSL Key, SSL Certificate were not found.                                                                                  
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Assuming that the container is running behind a HTTPS enabled load balancer.                                              
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Configuring gitlab...                                                                                                     
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::database                                                                                              
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::redis                                                                                                 
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::secrets...                                                                                            
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::sidekiq...                                                                                            
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::gitaly...                                                                                             
juin 04 11:19:39 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::monitoring...                                                                                         
juin 04 11:19:40 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::gitlab-workhorse...                                                                                   
juin 04 11:19:40 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::unicorn...                                                                                            
juin 04 11:19:40 nuuk 9c7bde9c4ff7[801]: Configuring gitlab::timezone...                                                                                           

Failing line appears in red color in my log, meaning the execution is considered as failing in the bash environment, and the container restarts immediatly after that. It makes me think that the wait called on line 22 uses an invalid PID, leading to a non 0 exit status.

[adamlc]

@solidnerd I did actually remove those lines from the entrypoint that were added for 10.8.3 and still got the same issue with the migrations.

I think the issue may lie in the migrations. I went from 10.8.1 to 10.8.3 too.

[flashermaster]

@solidnerd Thank you for your support. I'm not upgrading. I just made a clean install in my synology docker. I followed the instructions in https://hub.docker.com/r/sameersbn/gitlab/. Wget the yml and fix it with my own info. Then I made the first docker-compose up and it worked. Then I tried to make some date fix(it doesn't matter at all cause I tried the whole steps another time without doing this) so I made a docker-compose stop. This time the up got error code 143.
Unfortunately now I'm tring 10.8.1 for clean installation and get failure info:
Missing Rails.application.secrets.openid_connect_signing_key for production environment. The secret will be generated and stored in config/secrets.yml.

[spritle]

@adamlc There is only endless reboot. I have to remove the container and also the data in my volume to make next clean docker-compose up work.

@flashermaster Could you please give the commands you used for removing the container and also the data volume?

[flashermaster]

@spritle Well I tried 2 ways.

1. The synology nas provide the docker UI to manage the images and containers though it doesn't support docker-compose through WEB UI.
2. I also tried docker rm/rmi commands to remove the containers&images.
   To clean the data volume I pointed in yml such as:
   volumes: <ul> <li>/volume1/docker/gitlab/gitlab:/home/git/<a href="data:Z">data:Z</a>
   
   I just rm -rf * the directory 🤣

[adferrand]

@flashermaster It is not the real error, the real one is below.

In fact you did not specified explicit signing key on your docker with env variables GITLAB_SECRETS_*. So at first start, the docker generated it for you, put it in config/secrets.yml and started to sign data using it. Then you deleted this file somehow (certainly with your data fix in persistent storage). Finally, at second start, the docker generated a new key, which is obviously incompatible with existing data.

So your data is messed up, you need to start with a completely clean state, meaning with nothing in /volume1/docker/gitlab/gitlab. And avoid to use 10.8.3 for now, and put explicit signing keys in env variables.

[flashermaster]

@adferrand I made a mistake. Actually I'm trying 10.8.1 and faced with the error. I used the same yml and up succeeded with 10.8.3. And also GITLAB_SECRETS_* was filled as the instruction mentioned.
Now the problem is:

• 10.8.3 reboot failure.
• 10.8.1 install failure.
  😢

[HsuNick]

i try from 10.5.6 to 10.8.3 upgrade
when appear

Missing Rails.application.secrets.openid_connect_signing_key for production environment. The secret will be generated and stored in config/secrets.yml.

I try enter cmd line.

docker exec -it gitlab bash
vi config/secrets.yml

left without making any changes.
the magic happened, It's working...

[adferrand]

Ok @flashermaster, I think that now, if you try to start in 10.8.1, it will fail because your data is already in 10.8.3. But 10.8.3 will bootloop as we saw in this issue.

So now, mount a custom entrypoint.sh from your host on path /sbin/entrypoint.sh (like --volume /your/host/path/entrypoint.sh:/sbin/entrypoint.sh), with the content of the script as in the repo sources (entrypoint.sh in on the root of sameersbn/docker-gitlab). Comment lines 18-23, and restart the docker in version 10.8.3. Do not forget chmod +x on entrypoint.sh

For me it worked. Also for @adamlc with a slightly different approach (he modified entrypoint.sh and built his own docker image).

For everyone, it is a working temporary solution, until the repo sources are fixed and a new image is deployed. But it skips database migration, so you should already be on a 10.8.X gitlab version to avoid weird issues.

[flashermaster]

@adferrand Thank you for your solution. Now I'll try it.
But before I try the 10.8.1, I have deleted all the containers and images(includes postresql&redis). And the issue is not the same of 10.8.3. 10.8.3 can boot the first time while 10.8.1 failed to init.

[hungnguyentps]

@adferrand Awesome, it's worked. You save my life. Thanks very much.

[solidnerd]

Could someone provide me the logs from the failing supervisord ?

[solidnerd]

I tested it again 10.8.2 to 10.8.3 and it's working for me.

[flashermaster]

Where can I get the failing supervisord logs?
In docker logs there are only error logs mentioned above.

[lucianomagrao]

I did the update from 10.8.1 to 10.8.3 everything fine, until I needed to remove 1 pod, then the pod started to reboot over and over. @solidnerd did you tried restart the instance after migration finish?

[flashermaster]

@HsuNick I tried. Before the reboot, I managed to vi the config/secrets.yml:
production:
db_key_base: some_long_random_codes1
secret_key_base: some_long_random_codes2
otp_key_base: some_long_random_codes3
development:
db_key_base: development
test:
db_key_base: test

seems all fine.

[adferrand]

@solidnerd, I have some debug data. So I put some output (echo and ps aux) before each commands on line 20, 21 and 22. Basically, before migrate_database, kill supervisord and wait PID.

Here is the result:

juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: Configuring nginx::gitlab-registry...
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: Configuring nginx::gitlab-pages...
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: Supervisor PID is:2887
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: root         1  5.2  0.0  19108  3816 ?        Ss   20:59   0:00 /bin/bash /sbin/entrypoint.sh app:start
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: root      2887  0.0  0.0  20124  3040 ?        R    20:59   0:00 /usr/bin/python /usr/bin/supervisord -nc /etc/supervisor/supervisord.conf
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: root      2888  0.0  0.0  34424  2840 ?        R    20:59   0:00 ps aux
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: Before kill
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: root         1  5.2  0.0  19108  3816 ?        Ss   20:59   0:00 /bin/bash /sbin/entrypoint.sh app:start
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: root      2887  0.0  0.0  38376 14032 ?        R    20:59   0:00 /usr/bin/python /usr/bin/supervisord -nc /etc/supervisor/supervisord.conf
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: root      2899  0.0  0.0  34424  2808 ?        R    20:59   0:00 ps aux
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: Before wait
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: root         1  5.3  0.0  19108  3816 ?        Ss   20:59   0:00 /bin/bash /sbin/entrypoint.sh app:start
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: root      2900  0.0  0.0  34424  2800 ?        R    20:59   0:00 ps aux
juin 04 20:59:25 nuuk 9c7bde9c4ff7[801]: /sbin/entrypoint.sh: line 16:  2887 Terminated              /usr/bin/supervisord -nc /etc/supervisor/supervisord.conf
juin 04 20:59:26 nuuk 9c7bde9c4ff7[801]: Loading /etc/docker-gitlab/runtime/env-defaults
juin 04 20:59:26 nuuk 9c7bde9c4ff7[801]: Initializing logdir...

This log shows that supervisord is started with PID = 2887, and shown in the ps aux. Before kill, just after migrate_database, supervisord is still up and shown in ps aux. So migrate_database did not break anything, everything is OK so far.

Where it becomes interesting is after the kill and before the wait. No process of PID = 2887 anymore in ps aux ! The process has been killed before wait even occur.

So @solidnerd, you do not see the error and some of us see it because our hardware is fast enough to "instantly" kill supervisord, and when wait kicks in, the process does not exist anymore and creates the error.

[solidnerd]

Release is underway if your errror still exists please reopen this issue. You can check https://hub.docker.com/r/sameersbn/gitlab/builds/ when the latest version is ready.

And thanks for @adferrand for the work ;)

[flashermaster]

@adferrand Nice work! The temporary solution works. 👍

[adamlc]

@solidnerd works perfectly! Also thanks @adferrand