Hypothesis: On the possibility of renaming SearchStrategy

[DRMacIver]

One of my least favourite things about Hypothesis is that it uses the word 'strategy' to describe its data generators. This was an internal name that I unthinkingly propagated into the public API, and was at best vaguely descriptive of what it did back in Hypothesis < 3.0 and is completely irrelevant to its function now if not actively misleading.

Additionally, it's looking increasingly likely that Hypothesis for other languages is going to start being a thing. If that's the case, I will not be propagating this mistake to them, and it would be good to have Hypothesis-for-Python use the common terminology.

This feels like a huge big scary change, and it is but that's mostly updating the documentation, articles, and user perception. In terms of updating the code:

1. One of the oddities around how Hypothesis is structured is that the type SearchStrategy is not actually part of the public API. There's nowhere public that users can actually import it from, and users are not allowed to subclass it - all creation of strategies goes through hypothesis.strategies. So nothing really needs to be done there.
2. The hypothesis.strategies module can easily be renamed and have a module that just imports and re-exports everything in its __all__.

Set against that relative ease is that we're renaming what is effectively the main entry point to the Hypothesis API. There's a lot of documentation to be updated and we'll be tripping over the terminology for a long time. So if we do this it comes with a significant cognitive burden and a moderate amount of user ill-will.

So, two questions:

1. Should we do this? My vote is yes, but I'm sufficiently nervous about the idea that I could easily be talked down.
2. If so, what should we call it?

Possible alternative names

I'm not currently flush with good names for this. I'd like generator or something along those lines, but that would be horrendously confusing in a Python context. Alternatives:

• Recipe
• Schema
• Kitten (we probably shouldn't go with this one)
• Provider

Suggestions very much welcome.

[DRMacIver]

See also #212 where I say:

Arguably SearchStrategy is a rubbish name that doesn't tell you anything about what it does, but I'm not sure this ever causes problems and I think it would be too annoying for users to change it at this late date.

As you can see I've about faced on this a bit.

[alexwlchan]

So you want to pull a Python 3 in Hypothesis 4? 😛

I’m a tentative 👍 on this change, if (and only if!) we can find a new name that we have unanimous (or near-unanimous) agreement is clearly better. If we’re going to take this pain, let’s switch to a name that is an obvious improvement on the old one, o/w we’re setting ourselves up to do it again when the next better name comes along.

“Recipe” is probably the best alternative I’ve heard so far (and how I’ve described strategies in the past) – but I’m not sure it’s better enough.

if we do this it comes with a significant cognitive burden and a moderate amount of user ill-will

Cognitive burden falls mostly on us (and if we can think of a clearly better name, I think is worth it). As for user ill-will, I think we mitigate that by committing not to break existing code. Something like:

• v4.0 All existing imports from hypothesis.strategies continue to work as before. All documentation points to hypothesis.newnane. New strategies are added to hypothesis.newname only. Only carrot.

  Some people will migrate now either because we’ve told them too, or to pick up a new shiny in _newname_.

• v5.0 As before, but now importing from hypothesis.strategies gives you a DeprecationWarning. Slightly more of a stick to get people migrating away, because nobody wants a test suite full of warnings.

• v6.0 We finally throw away the old names. Stick is now a large pointed one, because your code is broken.

[Zac-HD]

Proposed name: "provider" - as in, thing that provides examples. I like that it emphasises function rather than mechanism, and avoids the calling-it-a-generator problem.

Should we do this? I think yes, if we're about to bump the major version (within say six months) or Hypothesis-for-$language exists and is using the new name. My migration strategy would looks very like @alexwlchan's, except that I'd do 4+5 at the same time on a minor version (again, preferably close to a major version bump) and save the semver 4.0 for actually breaking changes. We did something similar with the datetimes extra, FWIW.

[alexwlchan]

except that I'd do 4+5 at the same time on a minor version (again, preferably close to a major version bump) and save the semver 4.0 for actually breaking changes

While semver major is only compulsory for breaking changes, I think it’s justified here – symbolically, this is a massive change to the public API, even if old code paths continue to work. I wouldn’t bundle it in a minor release (although I could be persuaded to skip the bit where we don’t emit any DeprecationWarnings).

The datetimes extra was a smaller change to the API, and the bit that was being deprecated was in the extras package – whereas here, we’re changing the core API.

[Zac-HD]

I was thinking that (especially without the deprecation warning) adding a providers module is certainly a backwards-compatible change, and implementation shouldn't be reflected in semver above the patch level. My biggest objection to the 4/5/6 plan is simply that the deprecated stuff will be around for too long, annoying contributors and confusing new users.

I also think the deprecation pathway is mostly a matter of taste though, and as mine has now been explained I'll leave it there. (I remain open to bikeshedding on the name, of course 😜)

[DRMacIver]

So you want to pull a Python 3 in Hypothesis 4? 😛

Yes but with actually thinking about migration paths. 😜

I agree with basically everything you said about migration paths and think your plan is sensible (and better than the one I had in mind). I'd be tentatively in favour of merging steps 4+5 but am unsure and am happy to go along with consensus. Definitely think that the version where hypothesis.strategies starts emitting deprecation warnings has to be a major version bump even if semver doesn't require it - this is a big change which will affect literally all code written using Hypothesis.

Regardless of exact migration path, I would want peoples' old code to keep working for a minimum of a year, and that a migration would be achievable with a simple 's/hypothesis.strategies/hypothesis.newname/'.

Proposed name: "provider" - as in, thing that provides examples. I like that it emphasises function rather than mechanism, and avoids the calling-it-a-generator problem.

I think provider might be the best option yet, thanks (it's certainly on par with recipe, which was my previous favoured option). I don't love it, but it has precedent and I'd be more than happy to settle for it.

My only nagging complaint with it is that it is technically incorrect: A strategy/recipe/provider/kitten doesn't actually provide the value. Instead it describes to Conjecture how it should build/provide the value.

ETA: But given the internal nature of conjecture, that doesn't really matter to the end user.

[Zac-HD]

My only nagging complaint with it is that it is technically incorrect: A strategy/recipe/provider/kitten doesn't actually provide the value. Instead it describes to Conjecture how it should build/provide the value.

Conjecture is an implementation detail 😄

[DRMacIver]

Conjecture is an implementation detail 😄

It is, but given @alexwlchan's point that the main cognitive burden falls on us, it's nice to be consistent with our implementation details!

[DRMacIver]

My biggest objection to the 4/5/6 plan is simply that the deprecated stuff will be around for too long

I don't think this is true. We're not on any fixed schedule for major releases, and can do them whenever we like. The amount of time things remain around for is completely unconnected to the release schedule.

(Related: one of the reasons why I've been making deprecation messages say "go away in future" rather than "Go away in 4.0" is that we don't have to get rid of everything that's currently deprecated in 4.0/5.0/whatever).

All that being said, I think the deprecated stuff should stay around for a while. I think a year is probably a good amount of time to leave deprecated APIs hanging around for in general, especially ones of this magnitude.

[Zac-HD]

Ah, the Hypothesis 2 schedule 😛

[DRMacIver]

How do people feel about "source"? As in "source of examples".

[alexwlchan]

How do people feel about "source"? As in "source of examples".

👎

I don’t think it really explains what they do, and it already has a common meaning in CS circles.

[hwayne]

blueprint?

[tgs]

I have a feature request for the new name :grin:

I never import strategies directly, I always import hypothesis.strategies as st. This is to avoid importing each individual strategy because that's annoying, while also avoiding import * because it annoys my friends with IDEs (it turns off some static analyses). It would be nice for the new name to be much shorter than 'strategies', or have an abbreviation that's more meaningful. As David pointed out, _generator_ means too many things and isn't accurate, but @given(username=gen.text()) is just so nice...

some.lists()
spec.lists() - meaningful although it doesn't flow in a the sentence @given(junk=spec.lists())
make.lists()
descr.lists() - it does describe them, although it hints at "descriptor" which is different
req.lists() - a request/requirement is kind of what these things actually are?

[DRMacIver]

Yeah, that's a good point. I mean the name you import it as doesn't have to be the same as the name of the type, which doesn't have to be the same as the name of the module, but it's nice if they all play well together.

[DRMacIver]

Another thing to think about naming wise is that we use the verb 'draw' in several places (composite, data) and I think we need to do one or more of:

• come up with a better verb than draw
• make the type name something where we can naturally apply the verb to the noun (either "verb a noun" or "verb from a noun".

So for example:

• We could change "draw" to "introduce" and "strategy" to "premise", so you introduce a premise.
• We could keep draw and change strategy to urn (you draw from an urn. Literally every probability textbook can't have the wrong idea, right? OK maybe this isn't a good idea)

I kinda like introduce + premise, and it ties in well with the "Hypothesis/Conjecture" naming scheme, but I feel like it's probably a bad idea.

You could then "import hypothesis.premises as posit"

from hypothesis import given, strategies as posit

@given(posit.lists(posit.integers())
def test(ls):
    ...

[DRMacIver]

I guess another thing to consider is that our naming convention of having strategies be lower-case plural is a little weird with some of these nouns. e.g. it doesn't quite fit premise.

[Zac-HD]

I don't like abbreviations as module names. Introduce, premise, and posit are all way too algebraic - we're aiming for wide popularity, not the stereotype of Haskell. I do really like the verb draw, as it's the standard for getting a single value from any kind of distribution or collection.

Another proposal: what about hypothesis.data?

from hypothesis import data, given

@given(data.integers())
def test(n): pass

The import is short, the name is reasonably descriptive of the module, and makes conceptual sense to me. It's also a reasonable name for other languages - examples are data, even if you're passing code objects as data - so I think it would work in Java and is already used in Haskell QuickCheck (OK, that's a neutral point 😜).

[wordaligned]

How about "arguments"?
I think this plays well with hypothesis, since you can argue about a hypothesis. And the module essentially generates function arguments. It abbreviates nicely to "args".

[alexwlchan]

One reason I’m not a fan of names like source or data is because they have associations with static things. I might load data from a file, and it never changes – but the values from Hypothesis strategies change all the time, and that’s a good thing!

I quite like arguments, both thematically and as an accurate phrase. The collision with f(*args) could be problematic, but I wonder if that’s less of an issue in non-Python languages?

Otherwise my favourites are still strategies or recipes.

[alexwlchan]

What do other libraries use for this name? Is there a consistent term elsewhere we could use?

[DRMacIver]

What do other libraries use for this name?

"Gen" is pretty universal in property-based testing land, though I'd rather not use it.

In similar tooling: Faker calls their equivalent providers. Factory is also recently common (in e.g. factoryboy). Model mommy calls them recipes.

I think "recipes" is probably my current favourite, but I haven't quite convinced myself it works for me yet.

[hwayne]

What about "fact" as "fact" but _not_ short for "factory"? So you'd have

@given(fact.integers())
def test(n): pass

"It is a fact that given integers, test holds".

[moreati]

producers? emitters? seeders? expanders? growers? forgers? guessers? walkers? spawners? breeders? feeders? feedstocks? improbabilitydrivers?

[Zac-HD]

FWIW Hypothesis-Ruby is using provider, so if or when Hypothesis-Python switches that's what they'll be called.

[DRMacIver]

FWIW Hypothesis-Ruby is using provider, so if or when Hypothesis-Python switches that's what they'll be called.

Though hypothesis-ruby is thoroughly in alpha, so it's still able to change that. I only went with providers because they were favourite of the suggestions and I needed something, but if someone gave me a new name I loved there would still be time to switch.

[ghost]

Entirely off-topic re. premises: In the overcrowded London of their time, Johnson and Boswell saw two women arguing from their upper windows, only a few feet apart. Johnson quips "They will never agree, because they are arguing from different premises".

I can only apologise.

[DRMacIver]

I have a possible suggestion to consider:

How would we feel about Domain?

Reasoning: Comes from "the domain of a function", or "the domain of valid values" and captures the intuition that a née Strategy specifies a range of valid values. Works particularly well with the inline style's use of given in hypothesis-ruby.

Arguments against:

• Maybe too mathsy?
• It's "technically wrong" in that a née Strategy defines more than just a set, but given that punning between a mathematical object and its base set is almost universal in mathematics that's not intrinsically a problem.
• Arguably the domain is the argument to given, but that doesn't mean that the type should be called Domain.

[DRMacIver]

Maybe this is best treated as a fill in the blanks:

• A single execution of a Hypothesis block/function is a test case
• A test case consists of givens, assumptions and assertions.
• The argument to an assumption is a condition (which is a boolean)
• The argument to a given is its domain (which is a X) and its return value is a Y.

An X defines (though possibly not as part of its public API):

• A range of valid values (its base set)
• A naive distribution of values (what you would get if the underlying PRNG were uniform)
• A reduction process (if a value u is provided as a Y from this X, we will attempt to replace it with v)

What are the values of X and Y?

Currently we have no term for Y. It's not really a premise? We could just call it "a given value" but that isn't ideal. We could call it an "argument" but it's not really one in the programming sense. "A component of the test case" might be the best I've got so far, but it's quite long winded.

ETA: This could of course be fixed by not calling the method given! I think that's probably the way to go, but it's hard to come up with good names for that.

Current terms for X:

• SearchStrategy in hypothesis-python (essentially nonsensical at this point)
• Provider in the hypothesis-ruby prototype.
• Domain in the above suggestion.

[ghost]

I like domain, I don't problem with it having extra structure. May be too mathsy; I guess non-mathsy folk would visualise the "domain of an animal" or something to do with Game of Thrones ("dare you enter my domain?") which would not be too far from a reasonable metaphor?

[Zac-HD]

IMO "too mathsy" is a sufficient argument against it - the biggest problem with Hypothesis is that people aren't using it.

---

On that note - I suggested changing the repo description a while ago, as "Advanced property-based (QuickCheck-like) testing for Python" probably scares some people off - and by design Hypothesis is quite unlike QuickCheck! The best current contender is "Hypothesis is a powerful, flexible, and easy to use library for property-based testing." - not cute, but a clear description in one sentence.

We should also add tags! I suggest python, testing, property-based-testing, coverage, and fuzzing - two generic tags, and three specific topics. We would rank page seven, page two, first, fourth, and second, which seems pretty good to me (ranking is by stars on the repo). Importantly, these tags would also serve as a reasonable overview of what the project is for: testing Python code, with fancy techniques that many developers have never heard of we make easy to use.

(and @DRMacIver, you have to do this personally - you're the only person who can edit the description and tags!)

[DRMacIver]

Ah yes! Done, RE tags and description.

I don't think we should necessarily treat mathematical terms as taboo. The biggest problem with Hypothesis may be that people aren't using it (though it's increasingly not a problem), but the biggest reason people aren't using it is that people haven't heard of it! The second biggest reason is "I'm not sure what I would test with it". I don't think "I looked at it and there was a maths word in the documentation" is ever even going to make it onto the top 5. Maybe the top ten.

If the name was something like LinearProbabilitySpace then yeah I'd agree that would probably be a door slam and people would run away screaming. But the "too mathsy" aspect of domain is largely "People won't get why it's a sensible word for it", and we have plenty of evidence that people don't actually care much whether it's a sensible word for it because they've been happily using SearchStrategy for the last three years!

[Zac-HD]

Missed the coverage tag, which is both descriptive and discoverable 😄

I don't disagree with your other points - I just think that we can do better and find a name that helps people understand rather than being neutral. My favorite is still "Provider" because the natural assumption 'a thing which provides [values]' is entirely accurate; "Domain" requires somewhat more explanation.

In other words "this is obvious, I don't need to write docs" is terrible; "this is obvious, I don't need to read the docs" is great 😜

[DRMacIver]

Missed the coverage tag, which is both descriptive and discoverable 😄

Actually a deliberate omission! I don't think Hypothesis is good enough at coverage-guided testing to brag about it right now, and I'm not really convinced that coverage is what people would look for for that anyway.

My favorite is still "Provider" because the natural assumption 'a thing which provides [values]' is entirely accurate; "Domain" requires somewhat more explanation.

So explaining individual Domviders is actually what got me to this: Provider might not require explanation, but each individual provider does, and it results in a fairly unnatural documentation structure. You might start with a provider, but each individual documentation immediately switches to the provider's domain (technically codomain *cough*).

When documenting a Promaintegy we want to specify:

1. What a valid value looks like
2. What a reduction looks like
3. (Optionally and rarely) the naive distribution

So the natural description is is value centric, while a Provider as a concept is process centric: It makes the name about what the thing does rather than what values it contains.

[ghost]

Provider would also risk verby -er classes, which make me sad.

[DRMacIver]

Bikeshedding continues!

How about the possibility of renaming SearchStrategy to... Possible?

Example documentation:

def integers(min_value=None, max_value=None):
    """A possible integer.

    If min_value is not None, only values >= min_value are possible.
    If max_value is not None, only values <= max_value are possible.
    """

class Possible(object):
    ....

    def map(self, f):
        """Returns a Possible where the possible values are f(x)
        for any x that is possible for self."""

I think I kinda like it? I'm not sure it's quite right, but it's a lot closer than Domain for sure and has a lot of things I like about it that I dislike about Provider.

ETA: This might also work better with Possibility instead of Possible. I'm unsure.

def integers(min_value=None, max_value=None):
    """A Possibility of an integer.

    If min_value is not None, only values >= min_value are possible.
    If max_value is not None, only values <= max_value are possible.
    """

class Possibility(object):
    ....

    def map(self, f):
        """Returns a Possibility where the possible values are f(x)
        for any x that is possible for self."""

[ghost]

Hmmm, adjectives for mixins, sure, but for classes? Possibility smells a bit like the -er deverbification of those told that they shouldn't use verbs for classes, and do so only grudgingly. If only deadjectivication were a word. Possibility.integers() just sounds clumsy to me. This could all be old-man grumpiness of course ...

[DRMacIver]

Hmmm, adjectives for mixins, sure, but for classes?

I'm not sure the distinction is meaningful, but to the degree that it is, a Stratible is much more like a mixin than a class: It's a type with a single abstract method and some convenience methods built on top of that.

Also the usage would be much more like:

import hypothesis.possibilities as p

@given(p.integers())
def test_foo(i):
    ...

Or in Ruby:

include Hypothesis::Possibilities

hypothesis do
    x = given integers
    ...
end

You'd probably never find yourself writing Possib{le|ility}.integers