Sdk: HttpClient leaks on Darwin (iOS/macOS)

[dnfield]

Dart program:

import 'dart:io';

Future<void> main() async {
  HttpClient client = HttpClient();
  final req = await client.getUrl(Uri.parse('https://google.com'));
  final HttpClientResponse resp = await req.close();

  await for (final bytes in resp) {
    print('Got some bytes ${bytes.length}');
  }
  client.close();
  print('done');
}

Leaks a bunch of stuff all seems to be rooted in SecCertificate:

It gets particularly bad if you end up downloading a bunch of things over HTTPS. Noticed this while working on https://github.com/flutter/flutter/issues/32143, and would see hundreds of these floating around in memory dumps as I scroll back and forth in a gridview with 100+ network images.

[dnfield]

/cc @bkonyi since you seem like you might know about https://github.com/dart-lang/sdk/blame/master/runtime/bin/security_context_macos.cc based on the blame in it

[bkonyi]

I'll assign this to myself for now just so I don't forget. Not sure what the exact issue is, but I wouldn't be surprised if there's leaks.

[dnfield]

Opened https://dart-review.googlesource.com/c/sdk/+/102560 - there's still some leaks in open SSL for this but they're smaller now (some random malloc blocks that trace back to OPENSSL_malloc).

[dnfield]

The remaining leaks all seem to be related to a call to status = SSL_do_handshake(ssl_); and likely affect platforms other than Darwin.

Not clear to me at this point if there's something Dart needs to do to free resources after that call, or if there's a leak in Boring SSL.

[dnfield]

Example backtrace of a malloc block that isn't freed after making an SSL connection:

0 malloc_zone_malloc
1 malloc
2 OPENSSL_malloc
3 ASN1_OBJECT_new
4 c2i_ASN1_OBJECT
5 asn1_ex_c2i
6 asn1_d2i_ex_primitive
7 asn1_item_exd2i
8 asn1_template_noexp_d2i
9 asn1_template_ex_d2i
10 asn1_item_ex_d2i
11 asn1_template_noexp_d2i
12 asn1_template_ex_d2i
13 asn1_item_ex_d2i
14 asn1_template_noexp_d2i
15 asn1_template_ex_d2i
16 asn1_item_ex_d2i
17 ASN1_item_ex_d2i
18 ASN1_item_d2i
19 d2i_X509
20 X509_parse_from_buffer
21 bssl:ssl_crypto_x509_session_cache_objects(ssl_sesison_st*)
22 bssl:do_read_server_certificate(bssl::SSL_HANDSHAKE*)
23 bssl::ssl_client_handshake(bssl:SSL_HANDSHAKE*)
24 bssl::ssl_run_handshake(bssl:SSL_HANDSHAKE*, bool*)
25 SSL_do_handshake
26 dart::bin::SSLFilter::Handshake()
27 dart::bin::Builtin_SecureSocket_Handshake(_Dart_NativeArguments*)
...

[dnfield]

This seems like a leak in Boring SSL - did verify that we're calling SSL_free after all of this. Someone from Boring SSL is looking at it now.

[bkonyi]

FYI, this is likely the same as https://github.com/flutter/flutter/issues/20409

[dnfield]

Someone with more knowledge of boring SSL than I says this isn't reproducing in Boring SSL and is probably a problem with the way Dart holds it.

It's not obvious to me what's wrong right now, but I'll try to poke at it a little more when I have time.

[dnfield]

From what I can tell, all our calls to SSL_new are paired with an SSL_free, and even after the SSL_free we're leaking. I'm not sure what else I can do here - it would really help probably sit with someone from BSSL or more familiar with BSSL on this.

For what it's worth, I am able to reproduce this on a local macOS VM build without using Flutter now.

[dnfield]

I'm not actively working on this and not sure when I'll get back to it. A lot of the leak has been cleaned up but there's still something in there.

[dnfield]

Also, if it's not clear, the program in the first comment creates the leak without any Flutter dependencies - running it through dart will create the leak, which can be observed by attaching Xcode.

[dnfield]

Here's what it looks like now:

We leak (2* 400) + (4 * 288) + (2 * 272) + (2 * 144) + (2 * 128) + (2 * 112) + (2 * 96) + (4 * 80) + (4 * 64) + (51 * 48) + (32 * 116) + (34 * 16) = about 11kb. It doesn't seem to matter how many clients or requests we make.

[mkustermann]

/cc @zichangg