Puma: Spike an implementation of replacing minissl w/openssl

[eregon]

What's the purpose of Puma having its own MiniSSL, instead of using the OpenSSL standard library?

When seeing things like https://github.com/puma/puma/pull/2181 it seems to me this choice has a significant maintenance cost.
cc @nateberkopec @fzakaria

In other words, I'd prefer if everyone optimized the standard library SSL vs doing per-gem specific optimizations for SSL.

[HoneyryderChuck]

+1, I'd like to know it as well.

However, the linked PR is introducing the Netty OpenSSL variant instead of the JRuby Bouncy Castle-based openssl, which is massively outdated, so it's technically not the same thing.

[dentarg]

https://github.com/puma/puma/pull/1836, https://github.com/puma/puma/pull/1832, https://github.com/puma/puma/pull/1562 might be better (recent) examples of the maintenance cost

[eregon]

However, the linked PR is introducing the Netty OpenSSL variant instead of the JRuby Bouncy Castle-based openssl, which is massively outdated, so it's technically not the same thing.

Sounds like something worth fixing in JRuby itself then. I would think almost the entire Ruby ecosystem is using the OpenSSL stdlib for SSL.

[HoneyryderChuck]

I would think almost the entire Ruby ecosystem is using the OpenSSL stdlib for SSL.

It should. JRuby lags massively behind, however. The java ecosystem is mostly to blame, as they chose to ignore the TLS elephant in the room. AFAIK there's only one HTTP/2-enabled server in Java, Netty, which maintains their own TLS stack, which FFI's openssl (the one being integrated in the linked PR), and although I don't know all the reasons, I'd guess it's because all other alternatives don't support TLS 1.2 features such as ALPN negotiation, which you need to have an HTTP/2 server.

There are also API discrepancies between CRuby's openssl and JRuby's, an example being the ASN API. All in all, JRuby would be better off FFI'ing to openssl, but that would be a massive change.

None of the reasons above seem to apply to puma, however. So, looking forward to hear about it from the maintainers, as that's been hanging on my head for a while.

[MSP-Greg]

First of all, two somewhat connected issues are being discussed here. First, why is JRuby's SSL support lagging, and second, why is Puma essentially bypassing Ruby OpenSSL.

As to JRuby, from what I've seen, some of the issues are due to not fully implementing the Ruby methods defined in the Ruby OpenSSL extension. Some of the methods are private, and hence, only (normally) called by Ruby OpenSSL script files.

I don't believe that Puma uses JRuby's OpenSSL, but it is used by JRuby's net/http, and that is used as a client for testing. So JRuby's OpenSSL broke some tests due to its OpenSSL stdlib...

As to Puma, it was implemented a long time ago. Ruby's OpenSSL is designed to met the needs of a wide range of applications, Puma's needs may be narrower, and also focused on handling a large number of clients.

There is certainly a maintenance cost. But, many of the changes are relatively simple ones required by OpenSSL API updates/revisions. Some of those changes are due to the fact that a few years ago there were just three protocols, then three more were added, then insecure protocols started being deprecated, etc.

Lastly, Puma needs someone fluent in Java that can update the JRuby code. As to Puma and the Ruby system in general, the main issue is http/2...

[fzakaria]

I an 👍 on this discussion although I believe it's better targeted primarily on a JRuby issue.

I found Puma to be using a proxy to OpenSSL through _MIniSSL_; which creates an "engine" likely created to help create an abstract interface for Java/C extension.

The current Puma source code creates Java's default SSL engine which is historically slow; especially on JDK8.

(I don't believe it's even using _jruby-openssl_ for TLS; defaulting to just the JRE native implementation)

A common idiom is to simply replace it with Netty's JNI (Java Native Interface) to OpenSSL; which is what the patch I put forward does.

There are some interesting benefits I like about distribution with Netty; you have the option to include a statically linked BoringSSL binary which is incredibly easy to distribute rather than relying on environment setup.

---

JRuby does offer an OpenSSL package (https://github.com/jruby/jruby-openssl) however that uses BouncyCastle; another Java implementation that is also slow.

A worthwhile much larger change would be for Ruby's OpenSSL package to leverage FFI to natively call OpenSSL; that theoretically should work also on JRuby which supports FFI.
(that is out of scope for Puma & this change)

---

I am fairly fluent in Java + Java ecosystem and open to additional contributions that make sense.

[nateberkopec]

From my perspective as a maintainer:

1) I have no attachment to any extension code. If we can do something in Ruby, I would prefer to do that. Especially for anything outside the parser.

1. I have no attachment to our own implementation of anything. So, if we can swap MiniSSL for someone else's implementation, I would be more than happy to do that and let someone else take up the maintenance burden. SSL is hard and we should be doing as little of it ourselves as possible.
2. For any change in implementation, we'd like to see the performance remain roughly the same for now. Roughly_ our worst-case latency target would be about 1 millisecond per request, but ideal range is more like 0.1-0.2 milliseconds. I believe the benchmark on #2181 shows we're currently in the 3ms range on JRuby w/ssl... not acceptable.
3. Puma has no active maintainer for it's Java code. We have a few people that can do basic C, but no Java experts.
4. JRuby is our most widely-used truly-parallel platform, so I want to support it as best we can, and "support" means "roughly as fast as it can be" in this context.

So, I think Puma should support the solution that a) uses the most Ruby code possible b) uses the most of other people's code w re: to SSL as possible and c) Maintains at maximum 1 ms of overhead per request on reasonable hardware/config.

[fzakaria]

Speaking to your questions:

1. @nateberkopec a little offtopic; but why does Puma use Ragel?
   HTTP parsers are a dime-a-dozen per language; generally even included in the standard library. (I find that to bee an odd choice)
2. Puma has already largely swapped out MiniSSL for other's implementation by either relying on JRE, OpenSSL (or Netty+OpenSSL bindings) through pretty minimal extension code.
3. I agree! I started the linked PR due to some complaints at my work about slow TLS connections using Puma with JRuby.
4. I can help review future Java code; there is not much of it in Puma

Further even expanding scope:
A nice change would be to consolidate the interface between Java & C MiniSSL and limit the use of private methods. In my PR I've added some remaining missing methods, but they don't seem to have proper analogous implementions in all cases and should be rethought.

Furthermore, the setup between the two could be consolidated.
For MRI Puma a PEM key/cert is needed but for JRuby you provide a keystore; it adds unneeded divergence between the codepaths.

I love to see this discussion.
(I hope this doesn't detract at all from the change I put forward too; I think given the benchmark it's still a worthwhile change 😉)

[HoneyryderChuck]

@nateberkopec I think that the stdlib openssl package is good enough for puma. Starting an OpenSSL::SSLServer just works(tm). The limitations I mentioned above do not apply for puma. I'd be very interested in seeing benchmarks, although my gut feeling is that the miniSSL extension isn't doing less or anything faster than openssl.

Also, the main use-case for puma is running with TLS disabled behind a proxy that does all the TLS, so we're discussing here about an "edge case" where some people might find convenient to run TLS in puma. I don't think that the maintainers should be burdened with maintaining so much C/Java extensions to get a marginal benefit for a subset of users.

These are just my 2 cents. I really think that, unless there's a really good reason (other than performance) to keep maintaining these extensions, puma should be using stdlib openssl.

In case anyone's interested, I've used ruby-tls to build a CRuby/JRuby-compliant HTTP/2-enabled server, and that's also a valid option for any server.

[nateberkopec]

why does Puma use Ragel?

Because that's what Mongrel used. Puma and Unicorn were both born from Mongrel, which you can see here.

I can help review future Java code; there is not much of it in Puma

This would be amazing. All relevant issues/PRs to jruby are already tagged if you want to take a look.

they don't seem to have proper analogous implementions in all cases and should be rethought.
it adds unneeded divergence between the codepaths.

Yes and yes, please do PR.

the main use-case for puma is running with TLS disabled behind a proxy that does all the TLS

Hard to know. I do think Puma's TLS support is a factor in its popularity. The ship has more or less sailed, and its a feature now IMO.

puma should be using stdlib openssl.
I've used ruby-tls

Would very much appreciate any investigation/PRs/draft PRs to this effect.

[fzakaria]

@HoneyryderChuck the discussion here mentions that Ruby's openssl gem in JRuby is implemented via https://github.com/jruby/jruby-openssl which is a Java implementation; that is slow -- you can see benchmarks in https://github.com/puma/puma/pull/2181

My two-cents; any web-server should offer a reasonable TLS mode.
Plenty of compliance requires TLS everywhere and rely on loadbalancers in TCP passthru or backends with TLS enabled.

_ruby-tls_ is interesting because it does use FFI;
I like the starting point but the largest question here is why doesn't Ruby's openssl gem use FFI;
that would make all this transparent
(again that's a much larger scope)

[eregon]

jruby-openssl ... is slow

I'm not sure that's the case, and IMHO it would be much better by having the JRuby openssl stdlib use Netty instead of BouncyCastle if that was a large perf gain.
The numbers in https://github.com/puma/puma/pull/2181 seem to compare JDK SSL (which is not jruby-openssl) and the new approach, but not JRuby's openssl stdlib.

OpenSSL is a C extension for historical reasons but also I suspect because it uses a few macros which are inconvenient in FFI. Besides, MRI doesn't ship with FFI.
OTOH TruffleRuby uses the OpenSSL C extension and passes 100% of the OpenSSL tests ands specs. So I think it's up to JRuby to find a compatible solution here, and it seems jruby-openssl is reasonably compatible.

---

So what if we would just use the OpenSSL stdlib? I think that would improve things:

• No need to maintain custom SSL C code and custom SSL Java code. Anything openssl/security-related would be handled by the Ruby implementation/OS. Only Ruby code for SSL stuff in the Puma codebase.
• Speed up SSL on JRuby, by using jruby-openssl which from what I read here is faster than JDK SSL.
• I would think same perf on MRI and other implementations.

[eregon]

cc @headius since this is related to JRuby SSL performance

[fzakaria]

@eregon -- thank you for the history of OpenSSL with respect to FFI;
I'm a long time Java developer but a shorter time Ruby (JRuby specifically) dev.

My intuition is that the BouncyCastle implementation will lag the Netty OpenSSL JNI.

I haven't run a benchmark myself of Netty OpenSSL vs BouncyCastle but I find this JMH benchmark worthwhile: https://github.com/msdousti/jmh-benchmark

JDK 11 (with AES instructions) > BouncyCastle (30x)
raw OpenSSL > JDK11 (2x)

tl;dr; OpenSSL ~60x BouncyCastle
(Unclear the overhead of JNI though)

The longterm path forward would be for jruby-openssl to make use of FFI so that it actually uses OpenSSL -- BouncyCastle is a _huge_ dependency anyways that is a PITA

[HoneyryderChuck]

@fzakaria you mention that puma needs to have "reasonable TLS mode", but your main complaint seems to be performance. I think that using jruby-openssl qualifies as "reasonable TLS mode", because it satisfies all TLS modes that puma requires to run today. it ships with the runtime as well. It does not require any extra dependencies.

I'm not going to argue against the merits of Netty's TLS stack, however, given that puma doesn't need any of the features that that stack provides (such as ALPN negotiation), this would be better explored as an extension to an otherwise stable default stack.

[nateberkopec]

the BouncyCastle implementation will lag the Netty OpenSSL JNI.

FWIW I set a specific threshold in my previous post. We should do what it takes to get an average JRuby SSL request below 1ms. I'm going to be less interested in adding more extension code if it gets us an additional 0.5ms ("2x faster") and even less about adding more extension code to get us from 0.2 to 0.1ms, for example. Relative benchmarks can be confusing when talking about code, because what we really care about at the end of the day is latency. My hunch is that for 99.9% of Puma users, 1ms of app server overhead is acceptable, so I want to get us to that point first.

I think the best path forward would probably be spiking an openssl stdlib implementation and seeing what that does to MRI and JRuby perf.

[headius]

A few clarifications from the JRuby side...

jruby-openssl uses BouncyCastle because the built-in SSL (and other crypto) support on JDK was lacking... but that was almost 15 years ago. Much of what we do today could probably just be moved back to using standard Java security features.

Our openssl is also an epic port of a lot of code from CRuby and another blob of code that actually mimicks OpenSSL's logic. It's a very large maintenance hassle and we are not happy about that.

The option of using OpenSSL via FFI is tempting, if only to get full feature-level compatibility. However this will be difficult to make work on Windows (how does CRuby do OpenSSL on Windows? Ship its own version?) and will make JRuby depend on the host system to have OpenSSL installed (CRuby does, but JRuby currently can run without any SSL library on the host system). It will also complicate JRuby deployment on Java servers, since we may not be able to load FFI libraries there at all. We have a large number of users that deploy this way.

We need help improving the jruby-openssl wrapper, either making it depend on BouncyCastle less (by using more built-in JDK features) or by having it use the Netty OpenSSL binding (assuming that exposes enough of the library to implement the openssl Ruby extension.

To be clear: we don't like jruby-openssl (and its dependency on Bouncy Castle) any more than you do, but with only a couple FTE maintaining the entirety of the JRuby ecosystem we don't have the resources to solve this quickly.

TruffleRuby uses the OpenSSL C extension

Can you provide some numbers on how well this performs? It may inform us whether a native OpenSSL approach is worth exploring, since TruffleRuby's C extension interface should have similar overhead to FFI or JNI.

I think the best path forward would probably be spiking an openssl stdlib implementation

Agreed. Regardless of how we proceed long-term with jruby-openssl, it should at least behave like CRuby's openssl. Don't concern yourselves with performance when it comes to the JRuby openssl for now... performance can be dealt with once we know this actually works.

[headius]

ruby-tls is interesting because it does use FFI

This is also interesting to us.

For the record, the openssl extension is a nightmare to port or maintain. It depends very heavily on the raw C API of OpenSSL and does very little to hide that fact. There have been many attempts to replace it with something easier to support and more portable across implementations (see krypt, now a dead project) but inertia has kept it alive.

[nateberkopec]

Thanks for weighing in @headius!

To sum up, I think someone taking a stab at getting us on openssl stdlib would be a good "epic project" for someone and I'm happy to help in any way I can for someone that wants to take this up. Personally I'm focused on the 5.0 milestone so I don't have bandwidth in the immediate term.

[MSP-Greg]

I took a brief look at the stdlib, and there may be some things missing. Or, given the current functions called in the OpenSSL libraries from Puma, equivalents do not exist for all in the Ruby OpenSSL extension files. I maybe wrong, as it was a brief look...

@headius

Is there a better place to discuss JRuby & Windows issues?

[headius]

@MSP-Greg The official JRuby chat is on Matrix here: https://matrix.to/#/#jruby:matrix.org

There's also a very low-traffic mailing list for JRuby on ruby-lang.org here: https://lists.ruby-lang.org/cgi-bin/mailman/listinfo/jruby

[fzakaria]

Very informative all this information.

@MSP-Greg its likely that some of the calls by the Puma extension are not needed or even have equivalent in OpenSSL stdlib; I think though what you've called out is that it's not a simple change.

How do people then feel about the Netty change in the original PR ?
I'm biased; but I think it is minimal change for a decent gain.

[MSP-Greg]

its likely that some of the calls by the Puma extension are not needed

Part of the reason I said 'brief look'. Some things could have been added for old SSL protocols, and may not be needed for newer ones, etc. I know TLSv1.3 changed some things compared to TLSv1.2 and earlier.

is that it's not a simple change.

That would be the 'bottom line' conclusion.

How do people then feel about the Netty change in the original PR ?

Sorry for not looking into Netty more. I did look into the standard JRuby, and there were a few gaps that I wasn't keen on.

I try to look at Puma (and similar software) in terms of proper resource handling. Are connections being closed that are misbehaved, are connections/sockets always properly handled on restarts, etc. I believe that some of the functions currently used to do so are not available in standard JRuby or the OpenSSL stdlib. I think...

[nateberkopec]

@fzakaria I haven't had time to go back and review it again yet. I'll keep discussion on that PR.

[MSP-Greg]

On the topic of OpenSSL, 1.1.1e was released yesterday, and it caused one failure in the Ruby MinGW OpenSSL test suite.

It seems to be related to closing the SSL connection when the TCP socket 'behind' it has already been closed. I haven't checked around, but the Windows MinGW platform is the only one where 1.1.1e is being used, most others are still using 1.1.1d. More later, and this may not affect Puma CI...