Truffleruby: Support the openssl standard library

[chrisseaton]

We are implementing support for openssl using Sulong as described in our RubyConf talk Ruby's C Extension Problem and How We're Fixing It.

I'll comment on progress in this issue. Feel free to ask questions or contribute in any other way.

[ylluminate]

sulong looks tremendous. Where can we follow along the progress of getting OpenSSL to work via Sulong? From what I quickly gathered, and perhaps I'm wrong, there will be some kind of a fork of or commits to OpenSSL that will get it prepped to compile properly with sulong and this is where your effort is going to come in in terms of what you're saying here. Ultimately it would seem that this will be a support step for rvm installation as it will have to grab this version of OpenSSL or compile it itself via sulong during the install process...

This is quite a selfish inquiry as I'd love to follow along (and probably many others) so that we could get a feel for how to convert other programs or libraries over with sulong and this seems like a perfect opportunity! :)

[chrisseaton]

Where can we follow along the progress of getting OpenSSL to work via Sulong?

That's what this issue is for.

From what I quickly gathered, and perhaps I'm wrong, there will be some kind of a fork of or commits to OpenSSL that will get it prepped to compile properly with sulong and this is where your effort is going to come in in terms of what you're saying here.

Yes I may modify openssl in place here

https://github.com/graalvm/truffleruby/tree/master/truffleruby/src/main/c/openssl

But I then hope to go back and fix the need for any modifications later on.

Ultimately it would seem that this will be a support step for rvm installation as it will have to grab this version of OpenSSL or compile it itself via sulong during the install process...

It should come ready to use in the GraalVM bundle.

[chrisseaton]

Where can we follow along the progress of getting OpenSSL to work via Sulong?

If you mean how to build it yourself, see

https://github.com/graalvm/truffleruby/blob/master/doc/contributor/cexts.md

[ylluminate]

Yes, wanting to follow along with your code changes in order to get an idea of the workflow at converting our own C, etc. projects over to run via Sulong. Thanks!

[chrisseaton]

I'll post things here next time I do some commits so you can see concretely what I'm doing.

[eregon]

From what I quickly gathered, and perhaps I'm wrong, there will be some kind of a fork of or commits to OpenSSL that will get it prepped to compile properly with sulong and this is where your effort is going to come in in terms of what you're saying here

Just to clarify, we might need to modify the code of the openssl C extension, but we do not plan on touching the OpenSSL library itself (but rather use the system-installed library).

[chrisseaton]

Today I'm looking at how to implement rb_scan_args. It's a simple function but happens to trip up a few things that Sulong doesn't implement as well as we'd like.

https://github.com/graalvm/truffleruby/pull/72

[chrisseaton]

Trying to implement rb_scan_args as a macro https://github.com/graalvm/truffleruby/pull/74.

[chrisseaton]

Related to C extensions, I'm looking at implementing fiddle in https://github.com/graalvm/truffleruby/pull/85.

[chrisseaton]

We've been working on C extension specs for the last week, to hopefully make it easier to run openssl without encountering so many bugs.

[ylluminate]

It really looks like you're mowing down https://github.com/graalvm/truffleruby/tree/master/spec/tags/optional/capi rather quickly.

[chrisseaton]

We can now load the openssl C extension and the associated Ruby code that sets it up. We pass most of the specs for it, but can't even successfully run the setup for the MRI tests for it. And actually trying to use it doesn't work. Progress is pretty quick now though.

[ylluminate]

You're doing fantastic. Been tracking your updates and commits. Several of us have been here in the background silently and cautiously cheering you on and are very excited!

[chrisseaton]

We can now run the setup for the MRI tests, so we've got a large number of fine-grained tests we can now work through.

[chrisseaton]

All the openssl specs now pass. There's still lots of tests, and it still doesn't actually work.

I'm having some trouble with what I think is undefined behaviour in openssl which Clang happens to accept but Sulong is more strict about and detects as undefined.

https://github.com/graalvm/truffleruby/commit/fad82d0e7295727601c457a39cc044f6ea6281b2

[chrisseaton]

openssl seems to basically work now for making requests that use HTTPS. We still need to integrate so it's available by default.

[DemiMarie]

@eregon I am glad that the system-provided OpenSSL library is being used. A lot of the crypto stuff makes assumptions regarding timing that might break on a JIT, and besides much of it is in assembly anyway. And none of it accesses Ruby objects.

[chrisseaton]

@DemiMarie yes definitely right for OpenSSL, but I should just also mention that Sulong can actually handle inline assembly as well!

[DemiMarie]

@chrisseaton Nice! Does it treat it the same way as ex. GCC or Clang?

[chrisseaton]

I think it still interprets (and just-in-time compiles) the assembly instructions, rather than copying them into machine code. It all has to work on a standard JVM where you can't just emit machine code whenever you want.

[chrisseaton]

openssl is now enabled by default on master. We're now merging this into the next GraalVM release.

[ylluminate]

Great work guys. Really excited to see you forging ahead here.

Giving you a little love over on Reddit.

I think Rails users will be most interested in this at the moment for server purposes and I think that's your primary target right now from listening to your talks, etc. Please correct me if I'm wrong.

[chrisseaton]

The next version of GraalVM will include this by default.