Netty: OpenSslEngine should not call SSL_set_state

[Scottmitch]

OpenSslEngine currently calls SSL_set_state(SSL_ST_ACCEPT) during the renegotiation process. The OpenSSL team has advised against this [1]. We should investigate and avoid calling SSL_ST_ACCEPT if possible.

[1] https://github.com/openssl/openssl/issues/2551#issuecomment-277644153

[snazy]

OpenSSL 1.1.0e cannot be used in netty-tcnative, because SSL_set_state no longer exists in OpenSSL since 1.1.0 (https://github.com/netty/netty-tcnative/issues/263).
Given that OpenSSL 1.1.1 (hopefully) supports TLS 1.3, it seems worth to remove usage of SSL_set_state.

[nename0]

I don't know if this helps, but this is how apache tcnative does the renegotiation (in an older version): https://github.com/apache/tomcat-native/blob/043f95087f712595598901b7c5f915ca5fbf86d5/native/src/sslnetwork.c#L650
In the trunk they removed the SSL_set_state call completely: https://github.com/apache/tomcat-native/blob/trunk/native/src/sslnetwork.c#L632

[normanmaurer]

@Scottmitch I wonder if we should just remove the support for renegotiation as its risky business anyway. This would also allow us to just drop set_state usage and so support openssl 1.1.x.

WDYT ?

@carl-mastrangelo @ejona86 @nmittler WDYT as well?

[floragunn]

+1

[ejona86]

I honestly don't know enough to determine whether renegotiation is necessary or alternative approaches.