Typing: Should we change PEP 484 to disable implicit Optional when default = None?

[gvanrossum]

CC: @ddfisher @JukkaL @markshannon

PEP 484 currently says:

An optional type is also automatically assumed when the default value is
None, for example::

  def handle_employee(e: Employee = None): ...

This is equivalent to::

  def handle_employee(e: Optional[Employee] = None) -> None: ...

This was intended as saving some typing in a common case, but I've received strong feedback from some quarters that this is not consistent and a bad idea. There are other places where None is allowed but none of them automatically add Optional (to the contrary).

So far it hasn't mattered much for mypy users because mypy doesn't have Optional support, but that's soon going to change (the --strict-optional flag is becoming more reliable) so if we're going to change this, now would be a good time. Thoughts? If we don't change this soon it will probably be too late.

[ddfisher]

I'm for this change. If I recall correctly, this is the only place where the PEP suggests typecheckers understand types to be something other than what's written as the type, so I think this change would be a nice simplification.

I've also started to notice this cause a bit of user confusion. See https://github.com/python/typeshed/pull/500, for example, for all the arguments to builtin functions that were accidentally marked Optional due to this.

[ilevkivskyi]

I have seen some confusions between optional arguments and optional types, and the convention in question amplifies such confusions. Therefore I am in favour of removing it.

[gvanrossum]

OK, let's bring this up on python-dev once Python 3.6 beta 1 is released
(i.e. after next weekend).

On Sat, Sep 3, 2016 at 7:32 AM, Ivan Levkivskyi [email protected]
wrote:

I have seen some confusions between optional arguments and optional types,
and the convention in question amplifies such confusions. Therefore I am in
favour of removing it.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/python/typing/issues/275#issuecomment-244549699, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMpUgUvnteOgZfJvI0g_D50N3TOthks5qmYTugaJpZM4J0K87
.

--Guido van Rossum (python.org/~guido)

[dmoisset]

IMO most of the confusion comes from the name being wrong. An argument is «optional» when the parameter has a default in python. An «optional» type is a type that you can choose to add or not (as in gradual typing). If we had Nullable[T] instead of Optional many of these confusions wold go away (and it's the name used mostly everywhere else, other than Maybe T in Haskell).

Other than that, there is an inconsistency that may require a change (I just wanted to clarify that the confusions shouldn't be part of the argument)

[prechelt]

@dmoisset: I agree that Nullable or Nonable (1 char shorter) or Maybe (3 chars shorter) would be better solutions than Optional.

On the other hand, even the short Maybe _is_ cumbersome. Nonable parameters are a frequent case and I believe the =None is sufficiently explicit to warrant introducing an additional PEP rule to cope with it, even if that rule has the character of an exception.

[markshannon]

I like that the Optional[] can be omitted if a None default is provided. It is less typing, reads naturally (to me) and is unambiguous.
If users want to use Optional they can since Optional[Optional[T]] == Optional[T].
As syntactic sugar goes, it seems quite sweet :)

OOI, which quarters?

[JukkaL]

I haven't had enough experience with using mypy strict optional checking to decide whether the extra verbosity would become awkward. Maybe we should wait until we have a substantial codebase that can be type checked using strict optional checking.

[markshannon]

Why do you use the word "strict"? That suggests that there is something "loose" about the syntactic sugar. There isn't; it is precisely defined.

[JukkaL]

Strict optional checking means that mypy treats None as a regular type, and mypy enforces that you add guards like if x: foo(x + 1) for values with an Optional[] type. Without strict optional checking, None is implicitly compatible with every type, kind of like null in Java, and an Optional[X] type is equivalent to just X. It has nothing to do with syntax.

[markshannon]

So how is it relevant to default values for parameters making the type Optional?

[JukkaL]

The reason for adopting the syntactic sugar was the worry that otherwise type annotations would be too verbose and thus maybe also harder to read. I think that the main arguments for not having the syntactic sugar are consistency and conceptual simplicity.

If we have a codebase that uses Optional[] consistently, which can be enforced by the strict optional mode of mypy, we could evaluate, at least for that codebase, whether the arguments hold water. We can get a realistic of estimate of the actual verbosity of annotating code with or without the syntactic sugar. We can perhaps also get a better understanding of whether the inconsistency is an actual problem for people who are working on the code.

It may all boil down to personal preference, but having some empirical data could be useful for the discussion. Also, it doesn't seem to me that this is an urgent thing to resolve, as this doesn't require major typing implementation changes (though get_type_hints may be affected) and it's going to be easy for a static checker to support both modes via a configuration option. I don't see a need to rush the decision, and we can first wait for tools that can perform useful "strict optional" checking to be built.

[markshannon]

Whether it is easier to read is, as you say, personal preference, which is rarely influenced by empirical evidence :smile:

Happy to wait and see what you come up with.

[ilevkivskyi]

@JukkaL I ma not sure what kinds of inconsistency you mean, but this one

x = None # type: int # here the type stays int, unlike for function params

will go away after PEP 526. I think there it is better to assume implicit Optional or not for both of these

def f(x: int = None): ...
x: int = None

but not separately for each of these.

[markshannon]

x = None # type: int

Ah, that thing. I think that should never have been added. It is just self-contradictory.

[JukkaL]

@ilevkivskyi There's still the question about annotating instance variables with mutable types in the class body in pre-3.6 code. This is what mypy users can write today:

class A:
    x = None  # type: List[int]  # Can't use x = [] (list is mutable) and don't want Optional

    def __init__(self) -> None:
        self.x = []

This is a common use case and I think we need to have a reasonable way to expressing this without PEP 526 syntax.

[markshannon]

That seems somewhat off topic, want to open a new issue?

[JukkaL]

I'm not proposing to change that, I just brought it up as I think that it's related to this consistency discussion and one of the more common cases where things currently feel a bit clumsy. PEP 526 would resolve that inconsistency for new code that can assume Python 3.6+.

[ambv]

Internal feedback at Facebook also suggests we should make this change.

[ddfisher]

There's been some internal feedback at Dropbox to that effect as well.

I think if the status quo was the other way, it's very unlikely that we'd want to add this syntactic sugar. I think the main reason we still have it is due to status quo bias. This change should still be low cost to make -- let's make it now, before that changes.

[gvanrossum]

I agree we should not have done this. I'm more worried about the cost of updating 600K LoC internal code (plus numerous stubs) than @ddfisher. It's probably a simple PR so we can try it out really cheaply.

[ddfisher]

On our internal repo that uses None Checking more, a run with this change causes about 20 errors in Python 2 stubs (13 of which are in builtins) and 120 errors in the rest of the codebase. Thankfully, these errors are all fixable mechanistically -- I'd be fine with making the fixes to our internal repos.

[gvanrossum]

What should be the next step? Maybe a simple thing would be to add a flag to mypy that disables the default Optional. That will allow us to track down all the stubs that rely on this; once we have those fixed (and our internal repo) we can make it the default.

At the same time we should propose a diff to PEP 484 and discuss it. At this point I expect that would be pretty uncontroversial.

[Tinche]

Coming here via Guido's Python-Dev post.

I'm a strong +1 on explicit Optionals. I use them heavily in our internal codebase and I consider them very helpful. They are verbose though, I'd be totally on board if we could get a shorter name for them and deprecate Optional.

[markshannon]

To clarify, the proposal is that this becomes a type error:
def handle_employee(e: Employee = None): ...
Is that correct?

[ilevkivskyi]

To clarify, the proposal is that this becomes a type error:
def handle_employee(e: Employee = None): ...
Is that correct?

Yes.

[JukkaL]

I also prefer requiring explicit Optional[...] types, and I'd like a shorter syntax. However, the shorter syntax is a separate issue.

[matthiaskramm]

To clarify, the proposal is that this becomes a type error:
def handle_employee(e: Employee = None): ...

Only in type-checkers that consider

def handle_employee(e: Employee):
  e = None

a type error. pytype doesn't, for example.

[gvanrossum]

Correct (and neither does mypy unless you pass --strict-optional).

[ambv]

True, but only --strict-optional conforms to PEP 484. Quoting the PEP:

By default, None is an invalid value for any type, unless a default value of None has been provided in the function definition.

[gvanrossum]

So then pytype doesn't conform to the PEP either. I wonder if we should actually give implementations more freedom in this area, given that nobody implements this correctly by default?

[matthiaskramm]

mypy and pytype accept this example because of different reasons.

• mypy accepts it because without --strict-optional, it allows all types to be None.
• pytype accepts it for the same reason it accepts x = 42; x = "foo"

That said, I'm +1 on deleting the sentence Łukasz found, from the PEP. Given the role None plays, as a placeholder in typical late-initialization patterns, type-checkers do need some leeway in this area.

[gvanrossum]

If we change this we should do it in a more explicit way than just deleting
the sentence. For example, we could say something like

At the type checker's discretion, None may be considered a valid value

for any type that's not a union containing None.

But this is separate from the nominal topic of this issue, which is about
what to do with

def foo(a: int = None): ...

Even if the type checker treats None as an invalid value by default, does
this specific pattern (an argument with a default value of None) modify the
type to become a union containing None? The original PEP 484 says yes.
We're proposing to change this to No. But maybe this should also be at the
discretion of the type checker (and separately from the other choice, since
they are now separate flags for mypy).

[matthiaskramm]

At the type checker's discretion, None may be considered a valid value
for any type that's not a union containing None.

None is always a valid value for Union[None, ...], so you can simplify that sentence? (Did you mean "not containing None"?)

But maybe this should also be at the
discretion of the type checker (and separately from the other choice, since
they are now separate flags for mypy).

The definition of def foo(a: int = None) should not be up to the type-checker. At the very least because that would make interpretation of the typeshed stubs ambiguous.

[gvanrossum]

... you can simplify that sentence? ...

Sure, I meant to say that for Union[None, ...] the value None is always acceptable (not at the checker's discretion) but let's rephrase it to

At the type checker's discretion, `None` may be considered a valid value for any type.

The definition of def foo(a: int = None) should not be up to the type-checker. At the very least because that would make interpretation of the typeshed stubs ambiguous.

Well, the whole point is that we would like to change the meaning. Maybe we should just fix all of typeshed to add explicit Optional[] in those cases and then add the --no-implicit-optional flag to the mypy tests for typeshed? There's about 400 of them, maybe we can automate this?

[gvanrossum]

(Also the stub issue goes away if we just declare that typeshed shouldn't depend on this behavior and always have explicit Optional in these cases.

[gvanrossum]

BTW, I think this was accidentally closed.

[matthiaskramm]

Well, the whole point is that we would like to change the meaning. Maybe we should just fix all of typeshed to add explicit Optional[] in those cases and then add the --no-implicit-optional flag to the mypy tests for typeshed? There's about 400 of them, maybe we can automate this?

We could just disallow "x: t = None" altogether in typeshed, and require "x: t = ...".

Btw. we probably have a number of stubs in typeshed that say "x: t = None" where they should say "x: t = ..." (i.e., x is declared as optional, but wasn't meant to be declared Optional). Automatic fixup will convert all those to an (explicit) Optional, as well, making it slightly harder to spot these kinds of bugs. (Not that it's easy to spot them now, but right now the handful of stubs that use Optional aren't candidates)

[matthiaskramm]

Also, re-reading the "optional vs Optional" in my last paragraph, it wouldn't be the worst idea to make typeshed do

from typing import Optional as Noneable

[prechelt]

Regarding what to do with

def foo(a: int = None): ...

I suggest the original PEP view that this should imply Optional is "right":

• it is a very useful and common shorthand
• in particular for the cases int and str, adding Optional[] is bulky while Python syntax is supposed to be lean
• the shorthand can be expressed as a syntactic rewriting and hence is conceptually straightforward
• there is only this one case = None, not a multitude of them.

[gvanrossum]

@prechelt

I suggest the original PEP view that this should imply Optional is "right": [...]

Those are all the original reasons why this was added to the PEP. But against it:

• In all other contexts, a: int = None is a type error
• It leads to user confusion, see e.g. python/typeshed#500

@matthiaskramm

Btw. we probably have a number of stubs in typeshed that say "x: t = None" where they should say "x: t = ..." (i.e., x is declared as optional, but wasn't meant to be declared Optional). Automatic fixup will convert all those to an (explicit) Optional, as well, making it slightly harder to spot these kinds of bugs. (Not that it's easy to spot them now, but right now the handful of stubs that use Optional aren't candidates)
Hm, I don't expect it to be a rampant problem, and like all other stub defects these should just be fixed when we find them.

it wouldn't be the worst idea to make typeshed do
from typing import Optional as Noneable
...and then 's/Optional/Noneable/' everywhere in typeshed? I think that's just causing churn for no good reason. Also it ends up exporting Noneable from every module containing it (per PEP 484). And "Noneable" is not a word.

[gvanrossum]

BTW https://github.com/python/typeshed/pull/1421 addresses some of this. It adds a way to run mypy from tests/mypy_test.py with the -no-implicit-optionals flag, which complains about things like def f(a: int = None). The PR also fixes the majority of such issues. (There's one round to go yet.)

[JelleZijlstra]

We fixed all instances of implicit Optional in typeshed a while ago.

@gvanrossum is the next step here just to change PEP 484?

[gvanrossum]

We're nearly there, I think we have a few steps to go first:

• Add --no-strict-optional somewhere so that typeshed stays clean
• Add words to PEP 484 allowing (maybe even encourage) type checkers to do this
• Discuss on python-dev when type checkers should do this

[JelleZijlstra]

The first one is done: https://github.com/python/typeshed/blob/master/tests/mypy_test.py#L134.

I submitted a patch to PEP 484 in python/peps#689.

[ilevkivskyi]

I assume we will keep a flag in mypy, but will just invert the default. Right?

[gvanrossum]

I assume we will keep a flag in mypy, but will just invert the default. Right?

Yes we should do that.

[gvanrossum]

Closed by https://github.com/python/peps/pull/689.

[sproshev]

What if I'm annotating function that does not accept None but has it as a marker of missed argument? E.g.:

def foo(p: int = None):
    if p is None:
        print("p not specified")

Should I use some another value to achieve this? If so, should it have type int or other one? E.g.:

class _MISSING:
    pass

def f(p: int = _MISSING):
    ...

[ewjoachim]

I'd say it's not desirable to put a default value if you consider that default value is not an acceptable value. In this case, just don't put a default value. If you don't pass anything, mypy will tell you and you'll get a type error, and if you explicitely pass None, mypy will tell you it's wrong.

def foo(p: int):
    # in the function body, don't try to defend against p being not an int
    ...

[sproshev]

What about other cases such as dataclasses.field? Default value for default and default_factory has incompatible type. See stub and specification.

[keu]

@ewjoachim there are these different scenarios:

# None is not one of possible values, func1 can't be called with explicit `param = None`
def func1(param: Class = None):
   param = param or Class("somethin")

# None is one of the possible values, funct2 can be called with explicit `param = None`
def func2(param: Optional[Class] = None):
   pass

# None is one of the possible values, funct3 requires param and can be called with explicit `param = None`
def func3(param: Optional[Class]):
   pass

# None is one of the possible values, funct3 can be called with `param = None`
def func4(param: Optional[Class] = obj):
   pass

i.e. default values has nothing to do with type annotation i.e. public contract of a function

[ewjoachim]

Not sure I get your point, and I don't agree with "default values has nothing to do with type annotation":

This function has wrong annotations which we can tell BECAUSE the annotations and the default value are clashing:

def func(param: int = "hello"):
   pass

---

I was answering to:

What if I'm annotating function that does not accept None but has it as a marker of missed argument? E.g.:

def foo(p: int = None):
    if p is None:
        print("p not specified")

and I understood "does not accept None" as "if you call the function with None (or without an argument), the call is going to be rejected. And I stand by my (... 2-year-old-)opinion: if a function is written to fail if an argument is missing, it's best to not have that argument have a default value.

If you want a default value and you want to be able to detect that the default value was used, either use : Optional[...] = None if None doesn't already have another meaning or do something like:

missing = object()
def func(a: Any = missing):
    if a is missing:
        print("default")

or

class Missing:
    pass
missing = Missing()
def func(json: Union[None, list, dict, str, int, float, Missing] = missing):
    if json is missing:
        print("default")

Maybe I missed an important point ?