Openssl: SSL_accept memory leak

[Gilks]

OpenSSL Version: 1.1.1 Stable and 3.0.0 Dev (both versions came from the github repo)
Host OS: Windows 10

Details:
When calling SSL_accept() roughly 1 MB of memory is used. This can be observed through task manager. When calling SSL_free() the memory is not returned. The issue appears to be almost exactly like this stackoverflow post. Here's some code to demonstrate the issue.

High level code overview:
I've stripped down my project to demonstrate the issue, so I apologize if there's any dead code. The code below binds to port 80 and waits for new connections. When a connection is received a new thread is started and the function handle_connection() is called. The function begins by applying TLS to the incoming client connection before calling SSL_accept().

If shut_it_all_down() is called _before_ SSL_accept() there is no memory increase. If shut_it_all_down() is called _after_ SSL_accept() memory is used but never returned after freeing it.

The reason this is important is because a large volume of network connections leads to a spike in memory usage that never returns. shut_it_all_down() contains some commented commands that I have tried in various orders but cannot seem to nail down the root cause.

Is this a bug within SSL_accept() or am I freeing incorrectly?

#include <stdio.h>
#include <winsock2.h>
#include <WS2tcpip.h>
#include <ctype.h>
#include <openssl\ssl.h>
#include <openssl\err.h>
#include <openssl\sha.h>
#include <openssl\pem.h>
#include <openssl\x509_vfy.h>

#pragma comment(lib,"ws2_32.lib")

struct client {
    SOCKET Client_sockt;
    SSL *tls_client_sockt;
    SSL_CTX *ctx;
    int in_use;
} theclient;

SSL_CTX *create_context() {
    SSL_CTX *ctx;
    ctx = SSL_CTX_new(TLSv1_2_server_method());
    if (!ctx) {
        perror("[-] Unable to create SSL context");
        ERR_print_errors_fp(stderr);
        return EXIT_FAILURE;
    }

    return ctx;
}

DWORD WINAPI handle_connection(struct client *r_client) {
    puts("[+] Connection accepted");
    (*r_client).tls_client_sockt = SSL_new((*r_client).ctx);
    SSL_set_fd((*r_client).tls_client_sockt, (*r_client).Client_sockt);

    // Shutdown everything before the call to SSL_accept and there's no leak
    //shut_it_all_down(r_client);
    int ret = SSL_accept((*r_client).tls_client_sockt);
    // Shutdown everything AFTER the call to SSL_accept and the leak occurs
    shut_it_all_down(r_client);
    puts("This code is never reached because of shut_it_all_down(). Using this line as a breakpoint.");
}

DWORD shut_it_all_down(struct client *r_client) {
    printf("[-] Shut it all down!\n");

    if ((*r_client).tls_client_sockt != NULL) {
        SSL_free((*r_client).tls_client_sockt);
        //SSL_shutdown((*r_client).tls_Client_sockt);
        //SSL_clear((*r_client).tls_Client_sockt);
        //SSL_CTX_flush_sessions((*r_client).ctx, 0);
    }

    closesocket((*r_client).tls_client_sockt);
    closesocket((*r_client).Client_sockt);
    ExitThread(0);
}

void start_server(unsigned long port) {
    // Setup socket, bind and listen
    SOCKET binded_sockt;
    WSADATA wsa;
    struct sockaddr_in server;
    server.sin_addr.s_addr = 0;
    server.sin_family = AF_INET;
    server.sin_port = htons(port);
    WSAStartup(MAKEWORD(2, 2), &wsa);
    binded_sockt = socket(AF_INET, SOCK_STREAM, 0);
    bind(binded_sockt, &server, sizeof(server));
    listen(binded_sockt, SOMAXCONN);
    struct sockaddr_in client;
    int client_size = sizeof(client);

    // Maximum connections set arbitrarily
    struct client r_client[2000] = { 0 };
    int i = 0;

    /* declare SSL objects */
    SSL_CTX *ctx;
    ctx = create_context();

    // Keep listening for new connections
    while (TRUE) {
        for (i = 0; i <= sizeof(r_client) / sizeof(r_client[0]); i++) {
            r_client[i].in_use = 1;
            r_client[i].Client_sockt = accept(binded_sockt, &client, &client_size);
            r_client[i].ctx = ctx;
            u_long mode = 1; // Disable blocking socket
            ioctlsocket(r_client[i].Client_sockt, FIONBIO, &mode);
            CloseHandle(CreateThread(NULL, 0, handle_connection, &r_client[i], 0, NULL));
        }
    }

    WSACleanup();
}

int main(int argc, char *argv[]) {
    start_server(80);
    return 0;
}

[mattcaswell]

Are you using static linking? If you are then you are missing a call to OPENSSL_thread_stop(). See:

https://www.openssl.org/docs/man1.1.1/man3/OPENSSL_thread_stop.html

[Gilks]

Yes, I am static linking. I did not realize that static linking would mean additional steps in a threaded environment. I've spent weeks pouring over my code looking for the root cause and to find out it's as easy as OPENSSL_thread_stop() seems unreal.

Thanks again for the support. If I could, I'd buy you a beer.