Hypothesis: Long-running fuzzing mode to discover bugs

[DRMacIver]

Currently Hypothesis is intended for being run as part of a small test suite of reasonably fast tests. This is great and all, but particularly given the coverage based discovery features what you really want is to be able to part Hypothesis on a server somewhere and just have it sitting there 24/7 looking for bugs in your code, then run only the corpus of examples that it built up there every time you do a CI run.

[bukzor]

This is almost exactly my use case. Is there a "next step" that I can contribute, please?

[DRMacIver]

This isn't really a feature I'm up for someone else taking the lead on I'm afraid. It's big, touches on a lot of issues, and is pretty central to my long-term plans for the project.

The only real next step that someone who isn't me can take on for this is finding a way to pay me to work on it it so I can move it up my priority queue, and I'm overcommitted enough right now that even that will have a fairly long lead time.

[bukzor]

:broken_heart:

[DRMacIver]

💔

The suggestion of funding it is a serious offer. If this is really something that you need, Google isn't exactly short on cash...

[bukzor]

I'm attempting a hypothesis project that would justify such a thing, but I can't before it's successful. Given that I really want such a feature, what should I do?

[Zac-HD]

1567 makes Hypothesis suitable for long (x hours per test) running processes, and the standard database workflow will ensure that discovered bugs can be re-run quickly. There's plenty more to do though!

[Zac-HD]

There was a short student project at Imperial College to implement a fuzzing mode for Hypothesis. This did not result in production-ready code, but did suggest the following approach:

1. Always keep running until every known bug has been fully minimised,
   (simplifying the standard life-cycle for consistency across modes)
2. Have an internal flag to restart the cycle after bugs are fully minimised,
   (i.e. the "keep fuzzing" mode)
3. Provide some way for test cases from multiple tests to run interleaved or in parallel.
   (so we can fuzz better than 'one test per process')

It's still not the proverbial "simple matter of code", but we at least have some idea what to do.

[Zac-HD]

Before writing our own fuzzing engine, we should start by exposing a way to treat Hypothesis tests as fuzz targets - i.e. a wrapper which takes bytes and invokes the test function once - which can be executed by any engine. This is much easier to support, a better match for the scope of our project, and allows both us and users to experiement with a variety of different approaches and tools.

python-afl would only need a small shim, and pythonfuzz or services like fuzzbuzz wouldn't even need that, so IMO we should keep such an interface even if we do end up developing our own engine.

Post-MVP we'll want to think about e.g whether we need a new Phase (or similar) for this, how it integrates with settings such as timeouts and healthchecks, and especially how the example database should work. Definitely good to get an MVP first though...