Envoy: add memory filesystem for making tests faster, more hermetic, and more robust

[jmarantz]

Currently real files are used for file i/o during tests. An alternative I've used in the past is to have an in-memory representation of a file-system: https://github.com/apache/incubator-pagespeed-mod/blob/master/pagespeed/kernel/base/mem_file_system.h

This has these advantages:

• completely hermetic, with a clean slate between tests
• very fast (no I/O), speeding up the running of massive numbers of tests
• you can inject notifiers etc, so you can for example get a callback or condvar wakeup or similar when a file is written

Envoy already has a filesystem abstraction (yay!) and a way to squirrel that into code that needs to read/write files (Api::Api) just no in-memory impl for it.

I think this would be a good intern project. The example I gave above would not exactly meet Envoy's needs but it could be a reference point.

[jmarantz]

@alyssawilk FYI

[mattklein123]

cc @junr03. This would help a ton with Envoy Mobile as we would be able to run more existing tests. I think there is an open issue on this in that repo.

[junr03]

This would be awesome! I have https://github.com/lyft/envoy-mobile/issues/124

[lpan]

@jmarantz I am interested in taking on this! It seems that the task is to author an in-memory implementation for the Envoy::Filesystem::File interface :slightly_smiling_face:

I am wondering if you could point me to one of these unit tests---currently uses real files for File I/O and could potentially benefit from this in-memory implementation of the interface. So that I could use it as a reference to come up with the implementation.

Thanks so much!

[alyssawilk]

Thanks for being up for tackling this!

It came up in the context of waitForAccessLog, which (without simtime) has to poll disk to see when a log entry has been added. If we had memfiles, the test memfile could explicitly notify condition variables when a file was tweaked, so we'd have far less looping and sleeping. I think if we had it we'd also swap all of the writeStringToFileForTest and friends over, so that for example all of the integration tests could load from memory rather than writing out and reading in. It'll be a really nice speed improvement, especially on mobile!