Truffleruby: malloc(): smallbin double linked list corrupted

[ioquatix]

* Error in `ruby /home/travis/.rvm/rubies/truffleruby-1.0.0-rc16/bin/bundle install --jobs=3 --retry=3 --path=vendor/bundle': malloc(): smallbin double linked list corrupted: 0x000000000a00a3e0 *

Full details:

log.txt

[eregon]

Thanks for the report, I just saw it this morning as well trying to debug why Bundler fails with OpenSSL errors in TravisCI for nio4r.
For example, almost all 10 jobs of https://travis-ci.org/eregon/nio4r/builds/525265563 failed due to this error (each trying 3 times to run Bundler).

[ioquatix]

Let me know if I can do anything to help.

[eregon]

Recompiling the OpenSSL C extension against's TravisCI's libssl seems to solve this problem, as well as #1627 (nested asn1 error) and #1632 (segfaults and memory corruption).

Specifically, all 10 builds ran bundle install --without development successfully from the first try in
https://travis-ci.org/eregon/nio4r/builds/525277863 :tada:

This probably means slightly different versions of libssl such as 1.0.1 and 1.0.2 or 1.0.2g and 1.0.2l are actually highly incompatible and cause segfaults without recompiling the C extension. This suggests that binary compatibility is not enough for libssl, and every single version tends to modify things in non-trivial and segfault-causing ways. Maybe due to different definitions for macros, which of course get embedded ("harcoded") in the C extension.

The good news is we already have a mechanism to conveniently recompile the OpenSSL C extension, by running truffleruby/lib/truffle/post_install_hook.sh. This is already run by Ruby installers (rvm, ruby-build, ruby-install), but is currently a no-op if it appears require "openssl" works. That's not a good enough check as we see with this issue.
The safest would simply be to always recompile, which takes around 35 seconds on TravisCI Trusty (compare to rvm install truffleruby, which takes ~80s on Trusty, ~150s on Xenial).

An alternative would be to ship TruffleRuby with a fixed, vendored and pre-compiled version of libssl, but I'm not sure if that's advisable. It's what quite a few softwares do however, for instance node. I think RVM and ruby-build also sometimes install and build a libssl just for installing MRI, which is basically the same but pushed to Ruby installers.

An immediate workaround for now is to set TRUFFLERUBY_RECOMPILE_OPENSSL to true before installing TruffleRuby.
For instance, in TravisCI this looks like:

env:
  - TRUFFLERUBY_RECOMPILE_OPENSSL=true

and if you have multiple env entries then something like (add it once per entry):

env:
  - TRUFFLERUBY_RECOMPILE_OPENSSL=true MYVAR=first
  - TRUFFLERUBY_RECOMPILE_OPENSSL=true MYVAR=second

Then rvm will recompile the OpenSSL C extension no matter what.

I'll work on fixing this in TruffleRuby directly.

[eregon]

I did some further testing and found that it reliably fails without recompiling the OpenSSL C extension:
https://travis-ci.org/eregon/test-travisci-truffleruby/builds/525338120
While it reliably works when recompiling it:
https://travis-ci.org/eregon/test-travisci-truffleruby/builds/525338259

Somewhat worryingly, neither the OpenSSL specs nor MRI tests catch the issue, only bundle install (e.g., in nio4r here) does.

[nirvdrum]

I'm sorry for the slow update. This was addressed in f08dd1ac65f22da9bb976ee8792429bfce6d04c5, which updates the post-install hook to always recompile OpenSSL for the system it's running on. The fix will be available in the next release.