Mypy: Allow __init__ with signature but no return type

[JukkaL]

Code:

class Visitor:
    def __init__(self, a: int):
        pass

Error:

x.py: In member "__init__" of class "Visitor":
x.py, line 2: Cannot define return type for "__init__"

The return type is Any (implicitly), and Any should be a valid return type for __init__.

This was reported by Guido.

[gvanrossum]

Hm, I think the return type for init should be None. A "return " inside an init is almost certainly caused by a misunderstanding. (Maybe you're thinking of new, which _does_ return the object it created?)

[JukkaL]

Ah, the issue is that the error message is wrong. The message should say that the return type must be None? The message may be a relict from times before Python-compatible syntax.

[gvanrossum]

But why do I have to specify the return type at all? Can't it default to
None here?

On Wednesday, March 18, 2015, Jukka Lehtosalo [email protected]
wrote:

Closed #604 https://github.com/JukkaL/mypy/issues/604 via 182a70b
https://github.com/JukkaL/mypy/commit/182a70b80b162b28c74f2b67a2b83ea6f78c8480
.

—
Reply to this email directly or view it on GitHub
https://github.com/JukkaL/mypy/issues/604#event-258931926.

--Guido van Rossum (on iPad)

[JukkaL]

Because of consistency. Consider __init__ that takes no arguments, other than self:

def __init__(self):
    ...

Would this be dynamically typed or statically typed? The current approach is to treat any function that has no annotation (no argument or return types) as dynamically typed, and statically typed (i.e. type checked) otherwise. The None return type marks the function as type checked.

Same goes for functions in general:

def f():   # Dynamically typed
    ...
def f() -> None:   # Statically typed
    ...
def g(x: int):  # Statically typed, Any return type
    ...

[gvanrossum]

Hm. I think it'd suck if we'd have to develop the habit of adding explicit
"-> None" to all init methods just to make mypy happy.

On Thu, Mar 19, 2015 at 8:11 PM, Jukka Lehtosalo [email protected]
wrote:

Because of consistency. Consider init that takes no arguments, other
than self:

def init(self):
...

Would this be dynamically typed or statically typed? The current approach
is to treat any function that has no annotation (no argument or return
types) as dynamically typed, and statically typed (i.e. type checked)
otherwise. The None return type marks the function as type checked.

Same goes for functions in general:

def f(): # Dynamically typed
...
def f() -> None: # Statically typed
...
def g(x: int): # Statically typed, Any return type
...

—
Reply to this email directly or view it on GitHub
https://github.com/JukkaL/mypy/issues/604#issuecomment-83869112.

--Guido van Rossum (python.org/~guido)

[JukkaL]

Okay, let's reopen this. Here's a proposed new spec. Let me know if it matches your thinking.

-> None would only be needed for type checked __init__ methods that take no arguments:

def __init__(self) -> None:
    ...

(In my Python corpus, these account for about 15% of all __init__ methods. __init__ methods seem to cover about 10% of all methods.)

An __init__ method with any argument types would be type checked, and the return type would always be None implicitly, but you can also give an explicit None return type if you want.

What about other functions and methods that have arguments with types but don't have an explicit return type? Should they always default to an Any return type? I see a few potential alternatives to this:

1. Default to a None return type, similar to __init__.
2. Default to a None return type if there is no return statement with an explicit value in the body. Give an error otherwise.
3. Default to a None return type if there is no return statement with an explicit value in the body. Default to an Any return type otherwise.
4. Try to infer the return type automatically. I don't like this much because it adds complexity, reduces consistency and "explicit is better than implicit" in this case, in my opinion.

[gvanrossum]

It's a tricky issue, since we need to balance the desire to catch errors in the body of the function (what if a return type was intended but accidentally omitted?) with the need to derive the most useful return type for the benefit of checking call sites, as well as making the system be pleasant to use. I think for methods in general the status quo is fine, but (2) and (3) have some slight advantage, and some even slighter downside.

However I still think __init__ is a special case -- its return value is determined by how Python uses it, not by what the user might want it to return. I think the absence of -> None should never result in an error message here (and in fact -> something_else should be considered an error, even -> Any). The heuristic over whether a function is considered type-checked or not is somewhat unfortunate but I can live with the requirement to add a dummy -> None to type-check an __init__ method that has no arguments besides self.

[JukkaL]

I'm leaning towards using the same convention everywhere (2) -- missing return type (if the signature has argument types) is the same as -> None. One reason is that in many examples of PEP 484 / mypy code I've seen, people tend to omit return types, even though their intention was probably not to have an Any return type -- they just did the easiest/shortest thing. Programmers are lazy, and we probably shouldn't make the rare case (Any return) easier to write than the common case (None return) when we have a choice.

Still, a convention of using an explicit -> None even when it's not needed (except for __init__, where it isn't useful, as you argued) may be a reasonable thing. However, perhaps it shouldn't be enforced.

Here's some more stats (these are actually pretty interesting): In my corpus about half of all methods have signature (self) -- but only a small fraction of these are __init__ methods. Approx. 2/3 of (self) methods seem to have None as the expected return type. Top-level functions that take no arguments but have a None return type are much less frequent.

[gvanrossum]

I'm assuming those argument-less None-returning methods are some kind of pattern to set/clear specific flags? E.g. set_debug(). Would be interesting to look at these a bit more, since the stats look a bit suspicious. OTOH it's understandable that there aren't many top-level functions like that -- changing global state is frowned upon more than changing instance state.

[gvanrossum]

I also realized that I'm not sure what your proposal is, exactly. Do you propose to assume ->None for all functions and methods that don't have an explicit return annotation? Or only for those that have at least one argument annotation? Or only for those that have no "return " in their body? (And how would the latter differ?)

[JukkaL]

Sorry for being vague. Here's a more explicit version of my proposal:

If a function or method has at least one argument annotation and no return type annotation, the return type would implicitly be -> None. If there is a return statement with a value within the function, it would be handled identically to a function with n explicit -> None return (it would typically result in an error).

The body of a function is not type checked, if it has no argument type annotations and no return type annotation. All the argument types and the return type are implicitly Any for a such function. This way the function can be called in statically typed code and the return value can be used in any way.

Here's a sample of 40 method names with only self as an argument from my corpus:

    def _writemarkers(self):
    def testFilterOnAnyCategory_OneUsedAndOneUnusedCategory(self):
    def getreply(self):
    def setUp(self):
    def test_source(self):
    def test_istitle(self):
    def ABD_method(self):
    def MakeCopiesButton_clicked(self):
    def NextButton_clicked(self):
    def __init__(self):
    def test_handle_expt(self):
    def getwidth(self):
    def testTest(self):
    def test_read_longlink(self):
    def addpackers(self):
    def test_using_mapping(self):
    def __do_layout(self):
    def __getstate__(self):
    def flush(self):
    def slotMapSelectionChanged(self):
    def tearDown(self):
    def test_unicode_filenames(self):
    def _find_swig_libdir(self):
    def tearDown(self):
    def testFloats(self):
    def sendObject(self):
    def test_failing_reload(self):
    def testReverseSortOrderWithGrandchildren(self):
    def __init__(self):
    def copy(self):
    def testChangeDueDate(self):
    def __len__(self):
    def testTwoAttachmentsCompat(self):
    def testEllipsis(self):
    def test_lineno(self):
    def IsExpandable(self):
    def modifiedFiles(self):
    def test_zones(self):
    def redo(self):
    def __init__(self):

I didn't look at the implementations, but I'd guess about 70% of the methods would have -> None as the return type. Note that mostly that is due to test methods. Whether test cases should be statically typed is a matter of taste, but type checking can verify that your type annotations are consistent with the way you use your code in tests.

[gvanrossum]

OK, the proposal looks good. Unit tests are a rather special case in a sense -- the TestCase class encourages having lots of parameter-less None-returning functions (all test functions, of course, but also setUp and tearDown, and often code shared between tests is also written in this style). Do you have a sample that excludes tests?

[JukkaL]

Here's a sample where test, setUp and tearDown methods are filtered out:

    def setlist(self):
    def GetMeasuringFont(self):
    def __unicode__(self):
    def isAnyItemCollapsable(self):
    def make_html_file(self):
    def setup_shlib_compiler(self):
    def __init__(self):
    def ShouldInheritColours(self):
    def __set_event(self):
    def InitMisc(self):
    def tagBody(self):
    def current_frames_with_threads(self):
    def __iter__(self):
    def _open(self):
    def makerepairinstructions(self):
    def __repr__(self):
    def __getMainWindow(self):
    def createRecurrence(self):
    def defaults(self):
    def nextCoverpageButton_clicked(self):
    def GetText(self):
    def createTests(self):
    def getWidget(self):
    def showcontents(self):
    def onFinish(self):
    def getfp(self):
    def __init__(self):
    def accept(self):
    def __repr__(self):
    def getparams(self):
    def _check_choice(self):
    def get_clock_seq_hi_variant(self):
    def is_namespace(self):
    def getChildNodes(self):
    def is_regression(self): return _ml.CvKNearest_is_regression(self)
    def _get_float_longitude(self):
    def get_yolks(self):
    def initUi(self):
    def GetType(self):
    def __init__(self):

Now it's actually a bit tricky to figure out which of these will return a value, but it seems that a fairly small fraction would have -> None return type, maybe ~25%. On average, the explicit -> None return type would be needed on roughly 1 line out of 200 in my corpus.

[nfergu]

As far as I can understand it, what was proposed here doesn't seem to have been implemented.

Consider this test case (from check-classes.test)

[case testConstructorWithImplicitReturnValueType]
import typing
class A:
    def __init__(self, x: int): pass
[out]
main: In member "__init__" of class "A":
main, line 3: The return type of "__init__" must be None

This suggests that an explicit return type of None must be specified, even though the __init__ method has arguments.

This appears to go against the proposal here which was (as far as I understand it) to only enforce this for __init__ methods that have not arguments.

As @gvanrossum points out above, it's a bit cumbersome to have to add "-> None" to all init methods just to make mypy happy.

Can anyone comment?

[JukkaL]

We decided to require the explicit -> None for consistency. It's a bit of extra work but fairly trivial for any significant codebase.

[henryJack]

It's a real shame to have to add -> None to every __init__ method. I'm not sure why this is needed for consistency as adding a return command will already give you a pylint error,

http://pylint-messages.wikidot.com/messages:e0101

and a TypeError at runtime,

https://docs.python.org/2/reference/datamodel.html#object.__init__

...Is there a way to, at least, suppress these errors?

EDIT: Actually, I think I can live with this and agree with the reasoning. Thanks for the awesome tool!

[henryJack]

You can also use the following regexp's to find and replace offending __init__ methods. Helpful if you have a large code base!

For init's on a single line --> find: (def __init__\(self,?.*\)), replace: $1 -> None

For init's accross multiple lines --> find: (def __init__\(self[^)]*)\)(?!\s->), replace: $1) -> None

[henryJack]

Has there been any discussion about an automypy tool analogous to the autopep8 tool?

[JukkaL]

There has been a lot of discussion about various tools for automatically generating annotations. A tool that would automatically improve annotations by inserting missing -> None types could be useful. Another somewhat related thing that could be nice would be automatic addition of string literal escapes when they are needed for forward references.

[rostislav-simonik]

@gvanrossum I'm little bit confused. Is it possible to return value by __init__ method ? Am I missing something.

[gvanrossum]

No, __init__ cannot return a value, hence you must specify -> None here.

[rostislav-simonik]

Oh, ok. I don't understand why mypy requires explicit type declaration for __init__ ?
I assume the goal is to provide type validation with minimal code burden. Shouldn't be mypy clever on this? Is there any risk mitigation that I don't see? Thanks for explanation.

[Michael0x2a]

@rostislav-simonik -- I would imagine because if you have an __init__ function that accepts no parameters, it's ambiguous if it should be typechecked or not without that annotation (and being able to clearly delineate between typed and untyped functions and mix the two in a sane way is one of the core features of PEP 484).

Basically, we add a little bit of redundancy in exchange for consistency -- as the Zen of Python says, "In the face of ambiguity, refuse the temptation to guess".

[MatthieuBizien]

I'm not sure what is the ambiguity for __init__, it should always return None. It's even a pre-defined error in PyCharm ("Cannot return a value from __init__).

The Zen of Python also says "Beautiful is better than ugly", "Readability counts" and "There should be one-- and preferably only one --obvious way to do it".

I think it's an important issue for Mypy because it's the kind of small change that can have a big impact on usability, without visible drawbacks. The reactions to the message of @JukkaL the 29 Jun are all negative. I can't push Mypy for our organization code if I have to justify things like that.

[FichteFoll]

"There should be one-- and preferably only one --obvious way to do it"

This doesn't really support your argument. Quite the opposite, actually.
I initially thought that this requirement was weird too, but now I understand the behavior of not checking definitions when there are no annotations (and the flag to check them anyway isn't set), in which case a parameterless init requires a return value annotation to be checked while init methods with at least one parameter do not. This situation would be inconsistent and having to specify None for every init is the besser choice imo.

[technology-studio-apiary]

Guys, Zen or non Zen, there is always simple rule how to decide: What's the added value, or what risk mitigation comes with explicitly defined return type None for parameterless constructor?

Yes there is an argument that engine need decide whether class is type-checked or not. But this can be deterministically decided even without this explicit definition of return type for parameterless constructor. Maybe it's too complex implement this logic and therefore we have explicit definition.

Maybe it's good to experience something other, e.g. flow-type for javascript, to see how easy life for developer can be.

[JukkaL]

I can see how the current situation can feel like a compromise. We can perhaps do a little better now that we have a larger set of strictness options, by not requiring None return type for __init__ with argument types. A missing return type would default to Any, not None, similar to other functions. The Any return type will have little effect since most __init__ methods don't return a value.

Various strictness options could be used if the above behavior is not desired:

• When using --disallow-incomplete-defs, the return type is required if there is a type annotation.
• When using --disallow-untyped-defs, an explicit return type will be always required.
• When using --check-untyped-defs, the __init__ return type can be omitted even if there are no argument types (assuming __init__ should be type checked). The return type will default to Any.

The rationale for my proposed change is current inconsistency between __init__ and ordinary methods:

class A:
    def __init__(self, x: int):  # Error, explicit return type required
        ...

    def f(self, x: int):  # Okay, return type not required here (implicit Any return)
        ...

[JukkaL]

Let's reopen this for discussion (see my comment above for my reasoning). It seems like many users are unhappy with the current situation.

[srittau]

I am unhappy about the situation as well, even if I have started using -> None for __init__ methods. I have two complaints about it:

• Instead of helping the developers, it creates a burden for no gain.
• It makes the signature harder to read and mentally process, again for no gain.

I understand the "consistency" argument, but I don't agree that consistency is more important than readability and ease of use in this case. I prefer the type checker to do as much work as it can for me as opposed to the other way round.

One possible solution I can see is a --check-untyped-init or even a --check-untyped-dunder flag and just have mypy assume default types for __init__ and potentially other dunder methods as well.

[gvanrossum]

A simple compromise would be that if absent the return type for __init__
is assumed None instead of Any. All other rules would remain the same. So
if you have a __init__ without args (only self) and you want the body
type-checked you still have to use -> None, but if you type any arguments,
you don't need to do that.

[JukkaL]

Should we implement @gvanrossum's suggestion above? We kind of lost track of this issue.

[gvanrossum]

I relabeled this as feature. It still needs discussion, I personally vote +1 on the compromise.

[idchlife]

Sooo, official resolution of this issue will be later or something?
mypy requires None type

[ethanhs]

We need decide if we want to move forward with Guido's suggestion (I'm +1 on it). Then someone needs to implement it. Since there seems to be overwhelming support for Guido's suggestion, I would say unless someone raises an objection soon, we should go ahead with it. @idchlife since you seem keen to have this perhaps you'd be interested in working on it?

[idchlife]

@ethanhs I'm kinda new to Python (2 months) and mypy, but I will try to look inside repository and grasp what's need to be done when I have time

[ethanhs]

@idchlife, no problem. I don't want to ask you to bite off more than you can chew. I'm sure someone will implement it :)

[JukkaL]

I'm happy to move forward with Guido's suggestion.

[mristin]

(In case somebody is looking for a script how to add None as return type to every __init__ in the codebase: https://gist.github.com/mristin/073a67f19e4133b9c97ed2269637b2f9

The script handles also cases when the return type has been already specified. I couldn't make regular expressions work, so I resorted to AST parsing with asttokens.)

[onlined]

What is the status? Is final decision made? Is somebody assigned with the implementation? I can work on it, if there is no one available.

[JukkaL]

@onlined We are happy to accept a PR that implements the proposal. Nobody is working on this yet.

[ethanhs]

This was resolved in #5677. Thank you @onlined!