Invidious: "Invalid memory access (signal 11) at address 0x10000001b" on Docker build
After updating to 05988c1c49851b7d0094fca16aeaf6382a7f64ab and successfully building, every time I load a page, images and videos don't work. I get the following in the logs for every page load:
Invalid memory access (signal 11) at address 0x10000001b
[0x56416bc179a6] ???
[0x56416b6c319f] __crystal_sigfault_handler +479
[0x7f941f76d28f] ???
There are no other errors. I tried removing the image and rebuilding, but the issue still persists.
Thanks.
atahanacar
All 30 comments
I'm getting this error when docker-compose up -d
root@invi-master-1572340487917-s-1vcpu-1gb-sgp1-03:/home/invidious-api-only# docker-compose up -d
Building invidious
Step 1/16 : FROM alpine:edge
---> 7eacb6761fa1
Step 2/16 : RUN apk add --no-cache crystal shards libc-dev yaml-dev libxml2-dev sqlite-dev zlib-dev curl && curl -Lo /etc/apk/keys/omarroth.rsa.pub https://github.com/omarroth/boringssl-alpine/releases/download/1.1.0-r0/omarroth.rsa.pub && curl -Lo boringssl-dev.apk https://github.com/omarroth/boringssl-alpine/releases/download/1.1.0-r0/boringssl-dev-1.1.0-r0.apk && curl -Lo lsquic.apk https://github.com/omarroth/lsquic-alpine/releases/download/2.6.3-r0/lsquic-2.6.3-r0.apk && apk update && apk add boringssl-dev.apk lsquic.apk && rm -rf /var/cache/apk/* boringssl-dev.apk lsquic.apk
---> Using cache
---> 4a5dafee1333
Step 3/16 : WORKDIR /invidious
---> Using cache
---> cfd64197aaca
Step 4/16 : COPY ./shard.yml ./shard.yml
---> Using cache
---> 135026f7094b
Step 5/16 : RUN shards update && shards install
---> Using cache
---> 829bd97a19fe
Step 6/16 : RUN cp /usr/lib/libcrypto.a ./lib/lsquic/src/lsquic/ext/libcrypto.a && cp /usr/lib/libssl.a ./lib/lsquic/src/lsquic/ext/libssl.a && cp /usr/lib/liblsquic.a ./lib/lsquic/src/lsquic/ext/liblsquic.a
---> Using cache
---> 176f76b46894
Step 7/16 : COPY ./src/ ./src/
---> Using cache
---> 3490fc9cd52e
Step 8/16 : COPY ./.git/ ./.git/
---> Using cache
---> 51c513043e7b
Step 9/16 : RUN crystal build --release --warnings all --error-on-warnings -Dmusl ./src/invidious.cr
---> Running in 14c25a16f5dc
Failed to raise an exception: END_OF_STACK
[0x55830a34fa36] ???
[0x5583099e4259] __crystal_raise +153
[0x5583099e564c] ???
[0x5583099f14af] ???
[0x5583099ea72a] ???
[0x5583099e99fd] ???
[0x558309d6662d] ???
[0x5583099fc0e0] ???
[0x5583099e7b47] main +55
[0x7f02a153f226] ???
ERROR: Service 'invidious' failed to build: The command '/bin/sh -c crystal build --release --warnings all --error-on-warnings -Dmusl ./src/invidious.cr' returned a non-zero code: 5
greentornado
on 20 Nov 2019
@greentornado That's unrelated to this issue. You should open a new issue so it can be addressed quicker.
atahanacar
on 20 Nov 2019
Hello you can fix this problems ?
elekber123
on 23 Nov 2019
0e3a48ff76aa0d5ea5f3f135c01ddde02b217dff fixes this issue.
atahanacar
on 25 Nov 2019
I am still having this issue. I tried running docker system prune then rebuild the image, but now I'm getting this error:
```
Invalid memory access (signal 11) at address 0x7f0f5fa7e518
[0x555caf362a36] ???
[0x555caf2b17f9] __crystal_sigfault_handler +457
[0x7f0f8c50728f] ???
avion540
on 25 Nov 2019
After some testing, I get the following error along with the previous one when I open a channel page, and images are still not working.
Invalid memory access (signal 11) at address 0x1c
[0x55d5875c8bb6] ???
[0x55d58708221f] __crystal_sigfault_handler +479
[0x7f20e093928f] ???
Unhandled exception in spawn: Write not allowed on read side (Exception)
from lib/lsquic/src/lsquic/channeled_pipe.cr:54:5 in 'unbuffered_write'
from /usr/lib/crystal/core/io/buffered.cr:194:5 in 'close'
from lib/lsquic/src/lsquic/client.cr:102:7 in '->'
from ???
from ???
from ???
from ???
from lsquic_engine_process_conns
from lib/lsquic/src/lsquic/client.cr:213:9 in '->'
from /usr/lib/crystal/core/fiber.cr:255:3 in 'run'
from ???
Building also results in invalid memory accesses
strace:
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f930fcf4000 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f930748c000 munmap(0x7f930fcf4000, 266240) = 0 munmap(0x7f930748c000, 1052672) = 0 munmap(0x7f930fd35000, 266240) = 0 munmap(0x7f930f8e8000, 528384) = 0 munmap(0x7f930f867000, 528384) = 0 munmap(0x7f9305364000, 2101248) = 0 munmap(0x7f93077f4000, 266240) = 0 munmap(0x7f9305565000, 1052672) = 0 munmap(0x7f930758d000, 790528) = 0 mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f930fd35000 madvise(0x7f93542c0000, 217088, MADV_DONTNEED) = 0 munmap(0x7f930fd35000, 266240) = 0 madvise(0x7f9354999000, 212992, MADV_DONTNEED) = 0 mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f930fd35000 munmap(0x7f930fd35000, 266240) = 0 mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f930fcf5000 munmap(0x7f930fed9000, 266240) = 0 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7f930fee81f8} --- writev(2, [{iov_base="Invalid memory access (signal 11"..., iov_len=60}, {iov_base=NULL, iov_len=0}], 2) = 60 writev(2, [{iov_base="[0x7f933bfc2fc6] ???\n", iov_len=21}, {iov_base=NULL, iov_len=0}], 2) = 21 writev(2, [{iov_base="[0x7f933bf18f4e] ???\n", iov_len=21}, {iov_base=NULL, iov_len=0}], 2) = 21 writev(2, [{iov_base="[0x7f933cd90461] ???\n", iov_len=21}, {iov_base=NULL, iov_len=0}], 2) = 21 exit_group(11) = ? +++ exited with 11 +++
Perflyst
on 27 Nov 2019
Also looks like something is tracked here https://github.com/crystal-lang/crystal/issues/7558#issuecomment-520154896
Perflyst
on 27 Nov 2019
Building without any cache results in the following
invidious@invidious:~/invidious$ ./invidious
Invalid memory access (signal 11) at address 0x0
[0x55ba9dddc846] ???
[0x55ba9d8c96cf] __crystal_sigfault_handler +479
[0x7f13387e5730] ???
[0x55ba9d8eaf97] ???
[0x55ba9d8eaf7d] ???
[0x55ba9de3904e] lsquic_stream_readf +238
[0x55ba9d922010] ???
[0x55ba9de3b5fe] lsquic_stream_dispatch_read_events +126
[0x55ba9ddeb904] ???
[0x55ba9ddede00] ???
[0x55ba9dddfa50] ???
[0x55ba9dde22bb] lsquic_engine_process_conns +875
[0x55ba9d9673a2] ???
[0x55ba9d8c4f3d] ???
[0x0] ???
Strangely enough it seems to affect only Alpine and Ubuntu 18.04 because I tried to use this Docker image of invidious which is based on ArchLinux and it's working perfectly without any crash whatsoever.
unixfox
on 28 Nov 2019
Debian 9 using Git master has been giving me this problem for at least a week as well. Sometimes it'll be stable for a few hours but mostly it just crashes over and over (service keeps trying to restart it).
I'm pretty sure some of the dependencies updated around that time.
ncaos
on 30 Nov 2019
Debian 9 using Git master has been giving me this problem for at least a week as well. Sometimes it'll be stable for a few hours but mostly it just crashes over and over (service keeps trying to restart it).
I'm pretty sure some of the dependencies updated around that time.
Same thing happen to me. I also use Debian 9 and cannot update to the latest Git master.
haizrul
on 5 Dec 2019
I can reproduce this bug on fresh installs of Debian 8, 9 and 10 (without docker). Also tried to downgrade crystal, but that wont work at all.
Since the branch use-quic was dropped (was not working properly anyways, #939) please give this issue some more prio. Only 1 instance and invidio.us is up right now...
Perflyst
on 5 Dec 2019
What command are you using to build?
omarroth
on 5 Dec 2019
Personally I'm using these commands:
git clone https://github.com/omarroth/invidious.git
cd invidious
docker-compose build
PS: Maybe reopen the issue because it's not fixed?
unixfox
on 5 Dec 2019
On 1499ce43bf1cd6a47e8c1b0df4c915feb7ca8fb4 I am able to build and run by removing the --release switch.
'crystal build src/invidious.cr' instead of 'crystal build src/invidious.cr --release'
if using the --release, invidious crashes with the same output as https://github.com/omarroth/invidious/issues/917#issuecomment-558951199.
To build with --release, I need to go back to using both lsquic 0.1.3 and invidious 276bf09238103aaabe64be09b9ac10f976510ff7
gled-rs
on 5 Dec 2019
That's pretty much why the image flourgaz/invidious is working, it doesn't specify the --release in the Dockerfile so that's why it's not crashing, not because it's using ArchLinux.
Interesting...
unixfox
on 5 Dec 2019
shards install && shards update && crystal build src/invidious.cr --release
Please note, the first build crashes here with invalid memory access but the second one succeeds. But the invidious binary cannot be executed as always the invalid memory error occurs.
Perflyst
on 5 Dec 2019
Seems to be working without docker if it is builded without --release
So --release is broken? :D
Perflyst
on 5 Dec 2019
28669d940a363bfb504a930184f9da27d08daa30 works around the issue. Does this mean I should close the issue, or wait for a proper fix?
atahanacar
on 6 Dec 2019
I'd say keep this around until it's fixed, --release break things clearly means there's still problems.
tleydxdy
on 6 Dec 2019
I get the same error despite removing --release from the Dockerfile
seonwoolee
on 6 Dec 2019
try removing everything before building, I do this by
docker-compose stop
docker-compose rm
docker system prune
That did the trick. Thanks
seonwoolee
on 6 Dec 2019
$ ./invidious
Invalid memory access (signal 11) at address 0x1
[0xdbbbe6] *CallStack::print_backtrace:Int32 +118
[0x5f8279] __crystal_sigfault_handler +217
[0x7fa90c2b35f0] ???
[0x7fa90ca9445d] ???
[0x131fd7b] ???
[0x1328742] ???
[0x13276a6] ???
[0x131ff2d] ???
[0x103a931] *OpenSSL::SSL::Context::Client +17
[0x103a8f2] *OpenSSL::SSL::Context::Client#initialize:Nil +34
[0x103a8b9] *OpenSSL::SSL::Context::Client::new:OpenSSL::SSL::Context::Client +73
[0x10642d8] *QUIC::Client#initialize<String, (Int32 | Nil), Bool>:Bool +104
[0x1064255] *QUIC::Client::new<String, (Int32 | Nil), Bool>:QUIC::Client +213
[0x1063eab] *QUIC::Client::new<URI, Nil>:QUIC::Client +107
[0x1063e33] *QUIC::Client::new<URI>:QUIC::Client +19
[0x655393] ~procProc(QUIC::Client) +35
[0x106c5f8] *ConnectionPool(QUIC::Client) +104
[0x106c4ea] *ConnectionPool(QUIC::Client) +58
[0x6c7c7b] *fetch_decrypt_function<String>:Array(NamedTuple(name: String, value: Int32)) +123
[0x6c7bf5] *fetch_decrypt_function:Array(NamedTuple(name: String, value: Int32)) +21
[0x6c7ac9] ~procProc(Nil) +41
[0xe52d63] *Fiber#run:(IO::FileDescriptor | Nil) +115
[0x5f7776] ~proc2Proc(Fiber, (IO::FileDescriptor | Nil)) +6
[0x0] ???
Does not work on CentOS 7.7.1908 even without --release
Perflyst
on 7 Dec 2019
Commits before quic was added are working (d46b26e3bc87fbbdfa9550820410041b861bb393) but of course there are other things broken, like search and watching videos..
Perflyst
on 7 Dec 2019
strace: invidious-strace.log
Perflyst
on 7 Dec 2019
Should be fixed with 588fc6df85e4bdcd1caff3c3d544a55a234362ee, (specifically omarroth/lsquic.cr@d8b1b5077f469f79a12f4f2cb12e984f20353575).
omarroth
on 16 Dec 2019
I can confirm the issue is fixed.
atahanacar
on 17 Dec 2019
@omarroth can we revert 28669d940a363bfb504a930184f9da27d08daa30 now that the underlying issue has been resolved?
leonklingele
on 4 May 2020
Related issues
tmiland
路
4Comments
stodge
路
4Comments
Rb4eogrl
路
3Comments
tmiland
路
4Comments
SebKranz
路
3Comments
Most helpful comment
On 1499ce43bf1cd6a47e8c1b0df4c915feb7ca8fb4 I am able to build and run by removing the --release switch.
'crystal build src/invidious.cr' instead of 'crystal build src/invidious.cr --release'
if using the --release, invidious crashes with the same output as https://github.com/omarroth/invidious/issues/917#issuecomment-558951199.
To build with --release, I need to go back to using both lsquic 0.1.3 and invidious 276bf09238103aaabe64be09b9ac10f976510ff7