Envoy: perf: add memory-usage integration test

[jmarantz]

Envoy has a large number of maintainers and contributors, and we've seen that memory usage can increase significantly as features are added, or optimizations are made for other criteria (e.g. speed, readability). In some cases, the right decision is to consume more memory to realize other benefits, but this should be an explicit decision made during PR review, and should not be a silently bloating footprint.

During the time since https://github.com/envoyproxy/envoy/issues/4196 was issued, the zero-stat per-cluster memory consumption increased from 14557 bytes to about 16.5K bytes, and is now at 14742 bytes. We probably want to avoid a severe increase in the future.

Whenever tcmalloc is used, the exact memory byte-count appears to be very consistent, as observed withe the admin /memory endpoint. So it should be possible to write an integration test that measures the memory usage across some operation, e.g. initializing 10k clusters with and without stats. #4196 has details on how to do this with a bash script, but a C++ integration test should be much easier, following the pattern in HeapStatsThreadLocalStoreTest.MemoryWithoutTls : https://github.com/envoyproxy/envoy/blob/c97eb9ff7874bac62f5b75385b5ced0e89bdd74b/test/common/stats/thread_local_store_test.cc#L679

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[jmarantz]

/wait

[cmluciano]

@jmarantz Just got back today from holiday break, I'm working on this today

[cmluciano]

FYI based on the manual run of Piotr's script, I see the following on master as of Jan 3rd 2019.

Overhead per cluster: 57058 + 2031 for each extra worker thread

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[cmluciano]

not stale, pushing new changes to PR today

[jmarantz]

/wait

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[jmarantz]

/wait

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[jmarantz]

/wait