Hypothesis: Allow settings to be set via Pytest CLI

[SeanDS]

First of all, great job on Hypothesis - I'm using it for the first time for a project and love it! What I'm missing though is some way to run loads of tests on a single function via the Pytest plugin. I'd like to run Hypothesis for a long time on one test rather than the default (100 ms?) timeout. I've found that I can set the max_examples setting via the @settings decorator but it would be great if I could do this in a way that doesn't touch the source code, e.g.

pytest /path/to/tests --hypothesis-setting max_examples=10000

I could then target the test or tests that I want to run using the normal Pytest means, e.g. pytest /path/to/tests -k some_pattern or pytest /path/to/some/test::test_some_function. The overridden setting could then just be used for that session.

Is this something that could be considered for Hypothesis? (Apologies if it's already possible - but I searched the docs and didn't find anything).

[Zac-HD]

I think you probably want settings profiles plus a little bit of logic in your conftest.py, e.g. like this - you can check an environment variable or whatever else suits your project.

We could provide more detailed pytest integration, but then we'd also have to maintain it, ensure that all the interacting features made sense and worked togther, etc. Generally we prefer to provide a clean integration point and make that someone else's problem :stuck_out_tongue_winking_eye:

[SeanDS]

Thanks, I managed to get it to work this way:

def pytest_addoption(parser):
    # Add an option to change the Hypothesis max_examples setting.
    parser.addoption(
        "--hypothesis-max-examples",
        action="store",
        default=None,
        help="set the Hypothesis max_examples setting",
    )


def pytest_configure(config):
    # Set Hypothesis max_examples.
    hypothesis_max_examples = config.getoption("--hypothesis-max-examples")
    if hypothesis_max_examples is not None:
        import hypothesis

        hypothesis.settings.register_profile(
            "hypothesis-overridden", max_examples=int(hypothesis_max_examples)
        )

        hypothesis.settings.load_profile("hypothesis-overridden")

[Zac-HD]

Nice!

[asmeurer]

I'd like to second the request to have --hypothesis-max-examples as a standard flag in the pytest plugin. I very often find myself running pytest over and over again to check if hypothesis will find an error.