Invidious: Docker image gets stuck after installing dependencies/Docker image is slow to compile
I have the problem that on my current version of master docker-compose build seems to stuck when trying to Install sqlite3:
Step 5/6 : RUN sed -i 's/host: localhost/host: postgres/' config/config.yml && shards update && shards install && crystal build src/invidious.cr
---> Running in 42b32bad10db
Fetching https://github.com/kemalcr/kemal.git
Fetching https://github.com/luislavena/radix.git
Fetching https://github.com/jeromegn/kilt.git
Fetching https://github.com/crystal-loot/exception_page.git
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
Fetching https://github.com/crystal-lang/crystal-sqlite3.git
Installing kemal (0.25.2)
Installing radix (0.3.9)
Installing kilt (0.4.0)
Installing exception_page (0.1.2)
Installing pg (0.16.1)
Installing db (0.5.1)
Installing sqlite3 (0.11.0)
Fetching https://github.com/kemalcr/kemal.git
Fetching https://github.com/luislavena/radix.git
Fetching https://github.com/jeromegn/kilt.git
Fetching https://github.com/crystal-loot/exception_page.git
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
Fetching https://github.com/crystal-lang/crystal-sqlite3.git
Using kemal (0.25.2)
Using radix (0.3.9)
Using kilt (0.4.0)
Using exception_page (0.1.2)
Using pg (0.16.1)
Using db (0.5.1)
Using sqlite3 (0.11.0)
.-/+oossssoo+/-. mymachine
`:+ssssssssssssssssss+:` -------------------
-+ssssssssssssssssssyyssss+- OS: Ubuntu 18.04.2 LTS x86_64
.ossssssssssssssssssdMMMNysssso. Host: vServer 20171111
/ssssssssssshdmmNNmmyNMMMMhssssss/ Kernel: 4.15.0-47-generic
+ssssssssshmydMMMMMMMNddddyssssssss+ Uptime: 18 days, 7 hours, 5 mins
/sssssssshNMMMyhhyyyyhmNMMMNhssssssss/ Packages: 589
.ssssssssdMMMNhsssssssssshNMMMdssssssss. Shell: bash 4.4.19
+sssshhhyNMMNyssssssssssssyNMMMysssssss+ Terminal: /dev/pts/0
ossyNMMMNyMMhsssssssssssssshmmmhssssssso CPU: Intel Xeon (Skylake, IBRS) (1) @ 2.100GHz
ossyNMMMNyMMhsssssssssssssshmmmhssssssso GPU: Vendor 1234 Device 1111
+sssshhhyNMMNyssssssssssssyNMMMysssssss+ Memory: 544MiB / 1945MiB
.ssssssssdMMMNhsssssssssshNMMMdssssssss.
/sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
+sssssssssdmydMMMMMMMMddddyssssssss+
/ssssssssssshdmNNNNmyNMMMMhssssss/
.ossssssssssssssssssdMMMNysssso.
-+sssssssssssssssssyyyssss+-
`:+ssssssssssssssssss+:`
.-/+oossssoo+/-.
$ docker -v
Docker version 18.09.5, build e8ff056
theScrabi
All 20 comments
What do you mean by "my current version of master"?
The RUN-line of the Dockerfile in master looks different from the one in your code snippet. It looks more similar to the RUN-line in of the alpine docker proposed in https://github.com/omarroth/invidious/issues/361#issuecomment-463247424
Would you care to share the Dockerfile that causes that issue?
For me the current master Dockerfile (archlinux) works on:
$ uname -v
#1 SMP Debian 4.19.37-3 (2019-05-15)
$ docker --version
Docker version 18.09.6, build 481bc77
m0se
on 26 May 2019
What do you mean by "my current version of master"?
The repository from which I am running the docker-compose build is at the latest commit of the master branch. So it is the same Dockerfile as in commit 267bf289c494b405ef6da81dd754731b3df0a709.
The RUN-line of the Dockerfile in master looks different from the one in your code snippet. It looks more similar to the RUN-line in of the alpine docker proposed in #361 (comment)
Sorry my fault, this is the wrong output. I fixed it.
theScrabi
on 26 May 2019
Unfortunately I'm having trouble reproducing on Ubuntu 16.04, Ubuntu 18.04 or locally.
./+o+- mymachine
yyyyy- -yyyyyy+ OS: Ubuntu 16.04 xenial
://+//////-yyyyyyo Kernel: x86_64 Linux 4.4.0-143-generic
.++ .:/++++++/-.+sss/` Uptime: 56d 17h 44m
.:++o: /++++++++/:--:/- Packages: 849
o:+o+:++.`..```.-/oo+++++/ Shell: bash 4.3.48
.:+o:+o/. `+sssoo+/ CPU: Intel Xeon Gold 6140 CPU @ 2.295GHz
.++/+:+oo+o:` /sssooo. RAM: 169MiB / 992MiB
/+++//+:`oo+o /::--:.
\+/+o+++`o++o ++////.
.++.o+++oo+:` /dddhhh.
.+.o+oo:. `oddhhhh+
\+.++o+o``-````.:ohdhhhhh+
`:o+++ `ohhhhhhhhyo++os:
.o:`.syhhhhhhh/.oo++o`
/osyyyyyyo++ooo+++/
````` +oo+++o\:
`oo++.
$ docker -v
Docker version 18.09.2, build 6247962
./+o+- mymachine
yyyyy- -yyyyyy+ OS: Ubuntu 18.04 bionic
://+//////-yyyyyyo Kernel: x86_64 Linux 4.15.0-50-generic
.++ .:/++++++/-.+sss/` Uptime: 6m
.:++o: /++++++++/:--:/- Packages: 564
o:+o+:++.`..```.-/oo+++++/ Shell: bash 4.4.19
.:+o:+o/. `+sssoo+/ CPU: Intel Xeon Gold 6140 @ 2.295GHz
.++/+:+oo+o:` /sssooo. GPU:
/+++//+:`oo+o /::--:. RAM: 128MiB / 985MiB
\+/+o+++`o++o ++////.
.++.o+++oo+:` /dddhhh.
.+.o+oo:. `oddhhhh+
\+.++o+o``-````.:ohdhhhhh+
`:o+++ `ohhhhhhhhyo++os:
.o:`.syhhhhhhh/.oo++o`
/osyyyyyyo++ooo+++/
````` +oo+++o\:
`oo++.
$ docker -v
Docker version 18.09.5, build e8ff056
I'm a bit stumped. Could there be any sort of cap on CPU for docker on the target machine?
omarroth
on 26 May 2019
Might also be worth enabling progress reports to see where the compiler is getting stuck:
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 043d950..0e88851 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -13 +13 @@ RUN sed -i 's/host: localhost/host: postgres/' config/config.yml && \
- crystal build src/invidious.cr
+ crystal build -s src/invidious.cr
Seems like the same output. Is this right, or am I doing something wrong?
Step 5/6 : RUN sed -i 's/host: localhost/host: postgres/' config/config.yml && shards update && shards install && crystal build -s src/invidious.cr
---> Running in 3801a63fafd1
Fetching https://github.com/kemalcr/kemal.git
Fetching https://github.com/luislavena/radix.git
Fetching https://github.com/jeromegn/kilt.git
Fetching https://github.com/crystal-loot/exception_page.git
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
Fetching https://github.com/crystal-lang/crystal-sqlite3.git
Installing kemal (0.25.2)
Installing radix (0.3.9)
Installing kilt (0.4.0)
Installing exception_page (0.1.2)
Installing pg (0.16.1)
Installing db (0.5.1)
Installing sqlite3 (0.11.0)
Fetching https://github.com/kemalcr/kemal.git
Fetching https://github.com/luislavena/radix.git
Fetching https://github.com/jeromegn/kilt.git
Fetching https://github.com/crystal-loot/exception_page.git
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
Fetching https://github.com/crystal-lang/crystal-sqlite3.git
Using kemal (0.25.2)
Using radix (0.3.9)
Using kilt (0.4.0)
Using exception_page (0.1.2)
Using pg (0.16.1)
Using db (0.5.1)
Using sqlite3 (0.11.0)
Maybe I have messed up volumes, I may check on a fresh vm if I can reproduce the issue there.
theScrabi
on 26 May 2019
Hm, in that case it looks like the compiler is having trouble even starting up, or shards isn't exiting cleanly. Mind trying:
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 043d950..8941781 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -10,6 +10,7 @@ WORKDIR /invidious
RUN sed -i 's/host: localhost/host: postgres/' config/config.yml && \
shards update && shards install && \
- crystal build src/invidious.cr
+ echo 'Target reached: compiling...' && \
+ crystal build -s -p src/invidious.cr
CMD [ "/invidious/invidious" ]
Step 5/6 : RUN sed -i 's/host: localhost/host: postgres/' config/config.yml && shards update && shards install && echo "target reached: compiling..." && crystal build -s -p src/invidious.cr
---> Running in 5998b6c0a10c
Fetching https://github.com/kemalcr/kemal.git
Fetching https://github.com/luislavena/radix.git
Fetching https://github.com/jeromegn/kilt.git
Fetching https://github.com/crystal-loot/exception_page.git
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
Fetching https://github.com/crystal-lang/crystal-sqlite3.git
Installing kemal (0.25.2)
Installing radix (0.3.9)
Installing kilt (0.4.0)
Installing exception_page (0.1.2)
Installing pg (0.16.1)
Installing db (0.5.1)
Installing sqlite3 (0.11.0)
Fetching https://github.com/kemalcr/kemal.git
Fetching https://github.com/luislavena/radix.git
Fetching https://github.com/jeromegn/kilt.git
Fetching https://github.com/crystal-loot/exception_page.git
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
Fetching https://github.com/crystal-lang/crystal-sqlite3.git
Using kemal (0.25.2)
Using radix (0.3.9)
Using kilt (0.4.0)
Using exception_page (0.1.2)
Using pg (0.16.1)
Using db (0.5.1)
Using sqlite3 (0.11.0)
target reached: compiling...
The next thing that came in my mind was strace...
theScrabi
on 27 May 2019
Nope docker does not let me run strace under no circumstances. -.-
theScrabi
on 27 May 2019
Might be worth trying to clean everything and build from scratch: docker-compose down --rmi all -v --remove-orphans, then building with docker-compose up --force-recreate --build. Unfortunately I'm a bit out of ideas as to what the problem might be.
Does it occur with only the above mentioned VM, or do all similarly created VMs have this issue?
omarroth
on 29 May 2019
I went even further, and just clicked a new fresh vserver. However same problem on the new host. I guess the issue is on hetzer site. On their systems its not possible to compile. It would just be interesting to find out why this happens.
theScrabi
on 31 May 2019
Which server do you use? CX11?
On May 30, 2019 11:41:56 PM UTC, Christian Schabesberger notifications@github.com wrote:
I went even further, and just clicked a new fresh vserver. However same
problem on the new host. I guess the issue is on
hetzer site. On their systems its not
possible to compile. It would just be interesting to find out why this
happens.--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/omarroth/invidious/issues/551#issuecomment-497524121
Perflyst
on 31 May 2019
Yes The CX11 in Nuremberg with Ubuntu 18.04.
theScrabi
on 31 May 2019
Seems to work with the CCX11 (those with a dedicated CPU). Tried with an instance in Helsinki and Nuremberg.
and0x000
on 31 May 2019
After a bit of testing, it appears that Docker is not correctly allocating enough memory for the container, even if there's enough to compile normally.
Adding >256M swap space (dd if=/dev/zero of=swap bs=1M count=256 ; mkswap swap ; swapon swap) fixes the issue on CX11. I'm still not sure why docker should be getting stuck, from strace there may be a mutex somewhere that isn't being released, but unfortunately haven't looked into it much more than that.
omarroth
on 31 May 2019
I'm still not sure why docker should be getting stuck, from strace there may be a mutex somewhere that isn't being released, but unfortunately haven't looked into it much more than that.
I didn't get docker permissions configured to be able to run strace. I'm pretty sure that was a simple layer 8 problem.
theScrabi
on 3 Jun 2019
We were able to run a few straces but there was no helpful output. It just stuck
On June 3, 2019 10:37:43 AM UTC, Christian Schabesberger notifications@github.com wrote:
I'm still not sure why docker should be getting stuck, from strace
there may be a mutex somewhere that isn't being released, but
unfortunately haven't looked into it much more than that.I didn't get docker permissions configured to be able to run
strace.
I'm pretty sure that was a simple layer 8 problem.--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/omarroth/invidious/issues/551#issuecomment-498205123
Perflyst
on 3 Jun 2019
@omarroth does this swap statement have to be inside the docker container or outside, because I enabled 256 mb swap but it still gets stuck for me.
theScrabi
on 4 Jun 2019
Should be run outside Docker. If it's still getting stuck then you might try setting aside more space for swap (512MB-1GB). If it's still getting stuck you might try running strace outside Docker, although I think you mentioned earlier that you were running into issues doing this.
omarroth
on 4 Jun 2019
@omarroth yea i had a layer 8/PEBKAC issue. @and0x000 told me that I can strace processes from outside a container, which I didn't know was possible. I'll try 512mb, then test strace and tell you the outcome afterwards :)
theScrabi
on 4 Jun 2019
Closing since this appears to be resolved. If you encounter this issue again @theScrabi or have more info please mention here and I'll be happy to re-open.
omarroth
on 6 Jul 2019
Related issues
arch-btw
路
4Comments
Uffje
路
3Comments
elypter
路
4Comments
SebKranz
路
3Comments
atahanacar
路
3Comments