Crystal: Possible cache race condition

[Sija]

Refs:

• https://github.com/crystal-lang/crystal/pull/5447#issuecomment-354624371
• https://github.com/crystal-lang/crystal/pull/5882#issuecomment-377703015

/cc @asterite @RX14

[Sija]

And again - https://travis-ci.org/crystal-lang/crystal/jobs/373199400#L540-L546:

C-har.o: In function `*Char#to_s:String':
Char:(.text+0xea7): undefined reference to `*Pointer(UInt8)@Pointer(T)#appender:Pointer::Appender(UInt8)'
Char:(.text+0x12c2): undefined reference to `*Pointer(UInt8)@Pointer(T)#[]=<Int64, UInt8>:UInt8'
/usr/bin/ld: /tmp/crystal/crystal-run-crystal-spec-output.tmp: No symbol version section for versioned symbol `*Pointer(UInt8)@Pointer(T)#[]=<Int64, UInt8>:UInt8'
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc "${@}" -o '/tmp/crystal/crystal-run-crystal-spec-output.tmp'  -rdynamic  -lpcre -lgc -lpthread /mnt/src/ext/libcrystal.a -levent -lrt -ldl -L/usr/lib -L/usr/local/lib`

[RX14]

We've never ever had any reports of this outside CI, which is extremely strange considering how often it happens on CI. Very interesting.

[j8r]

It seems to only occurs in Travis? Interesting, very likely a system issue.
Everything won't probably be possible on Travis, but we may try to change some system parameters of the underlying OS.

For example increasing the value of /proc/sys/vm/max_map_count(maximum number of memory map areas per process).
Not sure about previous errors of this type, but here it occurs at the end - maybe no more memory map, memory cache or something else?

I've also read this, not sure if it's related but we may also try to clear the caches before compiling:

echo 1 >/proc/sys/vm/drop_caches
echo 2 >/proc/sys/vm/drop_caches
echo 3 >/proc/sys/vm/drop_caches
sync

[asterite]

I'm not sure how to fix this. One can reproduce for example by going to the cache and changing one o file for another (that is, do cp some-file.o some-other-file.o with two random .o files inside the same cache dir). Then you will get this error, but I don't know how it can be detected...

[anamba]

Just today at mid-day, I started getting this error consistently when trying to deploy (morning deploys were fine). I feel like I've seen it in the past, too, but at that time, it wasn't happening often enough for me to go out and look for solutions.

Since this issue mentioned a race condition, I tried adding a sleep 5 between two compile steps and that was enough to get my deploy working.

Oh, and in my case, it's not one undefined reference error, I get pages and pages and pages and pages of these errors...

[jhass]

Since this issue mentioned a race condition, I tried adding a sleep 5 between two compile steps and that was enough to get my deploy working.

That sounds weird. Are you sure they're not running concurrently? Crystal currently doesn't support concurrent invocations of the compiler (with the same cache directory).

[anamba]

Since this issue mentioned a race condition, I tried adding a sleep 5 between two compile steps and that was enough to get my deploy working.

That sounds weird. Are you sure they're not running concurrently? Crystal currently doesn't support concurrent invocations of the compiler (with the same cache directory).

It does sound weird, I know. But they are definitely running sequentially, unless bash and/or docker are doing something crazy.

Here are the exact commands being run:

    ...
    docker-compose exec -u app web crystal build -Dpreview_mt --release --error-trace --progress --stats --time src/faktory-worker.cr && \
    sleep 5 && \
    docker-compose exec -u app web crystal build -Dpreview_mt --release --error-trace --progress --stats --time src/hostingdash.cr && \
    ...

One thing I thought of was that Docker might not have finished writing something to disk before the second invocation starts, but the container is not stopping between those two execs, and this particular invocation was on a Linux host, so the :delegated on the mount can't be the issue either.