Typing: Proposal: syntax for variable and attribute annotations (PEP 526)

[gvanrossum]

Introduction

This issue is reserved for substantive work on PEP 526, "Syntax for Variable and Attribute Annotations". For textual nits please comment directly on the latest PR for this PEP in the peps repo.

I sent a strawman proposal to python-ideas. The feedback was mixed but useful -- people tried to poke holes in it from many angles.

In this issue I want to arrive at a more solid specification. I'm out of time right now, but here are some notes:

• Class variables vs. instance variables
• Specify instance variables in class body vs. in __init__ or __new__
• Thinking with your runtime hat on vs. your type checking hat
• Importance of a: <type> vs. how it strikes people the wrong way
• Tuple unpacking is a mess, let's avoid it entirely
• Collecting the types in something similar to __annotations__
• Cost of doing that for locals
• Cost of introducing a new keywords

Work in progress here!

I'm updating the issue description to avoid spamming subscribers to this tracker. I'll keep doing this until we have reasonable discussion.

Basic proposal

My basic observation is that introducing a new keyword has two downsides: (a) choice of a good keyword is hard (e.g. it can't be 'var' because that is way too common a variable name, and it can't be 'local' if we want to use it for class variables or globals,) and (b) no matter what we choose, we'll still need a __future__ import.

So I'm proposing something keyword-free:

a: List[int] = []
b: Optional[int] = None

The idea is that this is pretty easy to explain to someone who's already familiar with function annotations.

Multiple types/variables

An obvious question is whether to allow combining type declarations with tuple unpacking (e.g. a, b, c = x). This leads to (real or perceived) ambiguity, and I propose _not_ to support this. If there's a type annotation there can only be one variable to its left, and one value to its right. This still allows tuple _packing_ (just put the tuple in parentheses) but it disallows tuple _unpacking_. (It's been proposed to allow multiple parenthesized variable names, or types inside parentheses, but none of these look attractive to me.)

There's a similar question about what to about the type of a = b = c = x. My answer to this is the same: Let's not go there; if you want to add a type you have to split it up.

Omitting the initial value

My next step is to observe that sometimes it's convenient to decouple the type declaration from the initialization. One example is a variable that is initialized in each branch of a big sequence of if/elif/etc. blocks, where you want to declare its type before entering the first if, and there's no convenient initial value (e.g. None is not valid because the type is not Optional[...]). So I propose to allow leaving out the assignment:

log: Logger
if develop_mode():
    log = heavy_logger()
elif production_mode():
    log = fatal_only_logger()
else:
    log = default_logger()
log.info("Server starting up...")

The line log: Logger looks a little odd at first but I believe you can get used to it easily. Also, it is again similar to what you can do in function annotations. (However, don't hyper-generalize. A line containing just log by itself means something different -- it's probably a NameError.)

Note that this is something that you currently can't do with # type comments -- you currently have to put the type on the (lexically) first assignment, like this:

if develop_mode():
    log = heavy_logger()  # type: Logger
elif production_mode():
    log = fatal_only_logger()  # (No type declaration here!)
# etc.

(In this particular example, a type declaration may be needed because heavy_logger() returns a subclass of Logger, while other branches produce different subclasses; in general the type checker shouldn't just compute the common superclass because then a type error would just infer the type object.)

What about runtime

Suppose we have a: int -- what should this do at runtime? Is it ignored, or does it initialize a to None, or should we perhaps introduce something new like JavaScript's undefined? I feel quite strongly that it should leave a uninitialized, just as if the line was not there at all.

Instance variables and class variables

Based on working with mypy since last December I feel strongly that it's very useful to be able to declare the types of instance variables in class bodies. In fact this is one place where I find the value-less notation (a: int) particularly useful, to declare instance variables that should always be initialized by __init__ (or __new__), e.g. variables whose type is mutable or cannot be None.

We still need a way to declare class variables, and here I propose some new syntax, prefixing the type with a class keyword:

class Starship:
    captain: str                      # instance variable without default
    damage: int = 0                   # instance variable with default (stored in class)
    stats: class Dict[str, int] = {}  # class variable with initialization

I do have to admit that this is entirely unproven. PEP 484 and mypy currently don't have a way to distinguish between instance and class variables, and it hasn't been a big problem (though I think I've seen a few mypy bug reports related to mypy's inability to tell the difference).

Capturing the declared types at runtime

For function annotations, the types are captured in the function's __annotations__ object. It would be an obvious extension of this idea to do the same thing for variable declarations. But where exactly would we store this info? A strawman proposal is to introduce __annotations__ dictionaries at various levels. At each level, the types would go into the __annotations__ dict at that same level. Examples:

Global variables

players: Dict[str, Player]
print(__annotations__)

This would print {'players': Dict[str, Player]} (where the value is the runtime representation of the type Dict[str, Player]).

Class and instance variables:

class Starship:
    # Class variables
    hitpoints: class int = 50
    stats: class Dict[str, int] = {}
    # Instance variables
    damage: int = 0
    shield: int = 100
    captain: str  # no initial value
print(Starship.__annotations__)

This would print a dict with five keys, and corresponding values:

{'hitpoints': ClassVar[int],  # I'm making this up as a runtime representation of "class int"
 'stats': ClassVar[Dict[str, int]],
 'damage': int,
 'shield': int,
 'captain': str
}

Finally, locals. Here I think we should not store the types -- the value of having the annotations available locally is just not enough to offset the cost of creating and populating the dictionary on each function call.

In fact, I don't even think that the type expression should be evaluated during the function execution. So for example:

def side_effect():
    print("Hello world")
def foo():
    a: side_effect()
    a = 12
    return a
foo()

should not print anything. (A type checker would also complain that side_effect() is not a valid type.)

This is inconsistent with the behavior of

def foo(a: side_effect()):
    a = 12
    return a

which _does_ print something (at function definition time). But there's a limit to how much consistency I am prepared to propose. (OTOH for globals and class/instance variables I think that there would be some cool use cases for having the information available.)

Effect of presence of a: <type>

The presence of a local variable declaration without initialization still has an effect: it ensures that the variable is considered to be a local variable, and it is given a "slot" as if it was assigned to. So, for example:

def foo():
    a: int
    print(a)
a = 42
foo()

will raise UnboundLocalError, not NameError. It's the same as if the code had read

def foo():
    if False: a = 0
    print(a)

Instance variables inside methods

Mypy currently supports # type comments on assignments to instance variables (and other things). At least for __init__ (and __new__, and functions called from either) this seems useful, in case you prefer a style where instance variables are declared in __init__ (etc.) rather than in the class body.

I'd like to support this, at least for cases that obviously refer to instance variables of self. In this case we should probably not update __annotations__.

What about global or nonlocal?

We should not change global and nonlocal. The reason is that those don't declare new variables, they declare that an existing variable is write-accessible in the current scope. Their type belongs in the scope where they are defined.

Redundant declarations

I propose that the Python compiler should ignore duplicate declarations of the same variable in the same scope. It should also not bother to validate the type expression (other than evaluating it when not in a local scope). It's up to the type checker to complain about this. The following nonsensical snippet should be allowed at runtime:

a: 2+2
b: int = 'hello'
if b:
    b: str
    a: str

[ilevkivskyi]

Very good proposal! I was thinking some time ago about how to introduce __annotations__ dictionaries at various levels using type declarations for variables and came to exactly the same conclusions.

There is something that could probably be discussed here: should it be possible to add annotations to variables in for and with? Examples:

for i: int in my_iter():
    print(i+42)

with open('/folder/file', 'rb') as f: IO[bytes]:
    print(f.read()[-1]) 

"Practical" questions:

• In the proposal __attributes__ are used sometimes in place of __annotations__, is it a typo, or am I missing something?
• Will part of discussion happen here, or should all discussions go to python-ideas?

[gvanrossum]

• I haven't thought about for and with. It can be done for for easily, but with would be ambiguous to the parser (requiring too much look-ahead).
• __attributes__ was a typo; fixed.
• Much of the discussion is still happening in python-ideas. I am open to doing more of it here.

[gvanrossum]

This comment is going to list things brought up in python-idea that need more thought...

• When the class defines speed: int = 0 (meaning an instance var with a class default) can we change the class variable (effectively changing the default for those instances that haven't overridden it yet), as long as the type matches? (I.e. is C.speed += 1 valid?)
• In a class, should a: class int (i.e. without initializer) be allowed?
• Explain why this is better than continuing with type comments.

[UPDATES:]

• Do we enforce that the class <type> syntax is only allowed in a class? (A: Yes, same as return or break.)
• Could we start evaluating annotations for locals in the future?
• The class <type> syntax is a bit awkward. Maybe it should be ClassVar[<type>] instead?

[gvanrossum]

Here are some thoughts about for and with. tl;dr: let's not do it.

PEP 484 support using type comments on a for-statement, to indicate the type of the control variables, e.g.

for x, y in points:  # type: float, float
    # Here x and y are floats
    ...

This isn't implemented by mypy and doesn't strike me as super useful (typically type inference from points should suffice) but we could perhaps support this with the proposal under discussion:

for x: float, y: float in points:
    ...

However, this makes it a bit hard to find the iterable (points). It also seems to violate my desire to avoid having to create syntax for tuple unpacking (which the for-statement supports). So maybe it's better to just write this as follows, in the rare case where it comes up:

x: float
y: float
for x, y in points:
    ...

It's even worse for a with-statement -- we'd have to somehow support

with foo() as tt: Tuple[str, int]:
    ...

which would be unreasonable to ask our parser to handle -- the expression after the first : could well be the body of the with-statement, and by the time the parser sees the second : it's too late to backtrack (at least for CPython's LR(1) parser).

[ilevkivskyi]

OK, I agree, also the syntax:

x: float
y: float
for x, y in points:
    ...

emphasizes the fact that for does not create its own scope and x and y are visible outside for.

Concerning why it is better than comments I have two typecheck-unrelated "practical" points to add (not sure if this was already mentioned):

• many text editors (and also here on GitHub) comments are shown in light grey or similar dim color. Maybe it is only me, but I think this is very inconvenient.
• type annotations are somewhat similar to doc-strings (I frequently use them in such way), for functions we have __doc__ and __annotations__, but for modules and classes we have only __doc__, it would be helpful to have __annotations__ also for classes and modules. This will give a systematic way to generate documentation.

[gvanrossum]

Nick just proposed a: ClassAttr[x] insread of a: class X, which simplifies the grammar and is close to what I proposed to put in annotations anyway. So let's run with this.

[gvanrossum]

OK, I want to make this into a PEP in time for 3.6. And I want to hack on the implementation at the Core Devs Sprint Sept. 6-9 at Facebook, and get it into 3.6 beta 1. That should just about be possible.

[srkunze]

It appears to me as if a: ClassAttr could signify a class attribute without any type, right?

[ilevkivskyi]

@srkunze There is a convention for generics: if a type variable is missing it is assumed to be Any, so that a: ClassAttr would be equivalent to a: ClassAttr[Any]

[srkunze]

• many text editors (and also here on GitHub) comments are shown in light grey or similar dim color. Maybe it is only me, but I think this is very inconvenient.

I actually think a dim color is a good thing here.

Moreover, it's already true for many editors with docstrings and annotations as those are barely relevant for code execution. I find this very convenient as it helps me focusing on the important pieces.

[ilevkivskyi]

@srkunze I understand your point, but nevertheless I would like to differentiate two levels of "relevance": docstrings and type annotations are in some sense parts of "public API" while comments are not such parts.
Anyway, I already mentioned that this comment is quite personal.

[srkunze]

@ilevkivskyi

There is a convention for generics: if a type variable is missing it is assumed to be Any [...]

Got it.

[elazarg]

How about allowing the use of parenthesized (var: type) anywhere slightly ambiguous? (this is the way it's done in COQ)

If I'm repeating old proposals, I still think it will be helpful to document the reasons to reject it.

[srkunze]

@ilevkivskyi

docstrings and type annotations are in some sense parts of "public API" while comments are not such parts.

That could make sense and can be another motivation of why local variables don't have __annotations__: they simply don't represent a "public API".

[ilevkivskyi]

That could make sense and can be another motivation why local variables don't have annotations: they simply don't represent a "public API".

Good point.

[gvanrossum]

@elazarg

... parenthesized (var: type) anywhere slightly ambiguous ...

It was brought up on python-ideas and I briefly mention it in the section "Multiple types/variables" in the original post above, but I expect that the syntax will be hairy, the benefits slight, and the readability poor.

[ilevkivskyi]

@gvanrossum I have few short questions about the new PEP:

• What workflow do you propose for the new PEP, will it be developed in this repo or elsewhere?
• If I submit a PR with simply more formal and structured formulation of your draft modified taking into account comments from python-ideas, will it be a good first step?
• Should any implementation details be listed in the PEP (for example, should we introduce a new AST node or modify Assign or Name, or add another context apart from Load and Store, etc)?

[gvanrossum]

Oops, @kirbyfan64 and @phouse512 are the lucky lottery winners, and they are writing a draft in a repo cloned from the peps repo by the latter. If you have text to submit best post it here so they can cherry-pick it. I don't think the PEP should go into details about the AST nodes, though if you want to work on a reference implementation separately just go ahead!

[markshannon]

What is the point of variable declaration? You haven't given any justification, and I don't see any.

[gvanrossum]

It's less verbose, and makes the meaning clear. The existing notation is
easily confused with cast(), because of its placement after the value. ( I
know because this happened to me long ago. :-)

On Tuesday, August 9, 2016, Mark Shannon [email protected] wrote:

What is the point of variable declaration? You haven't given any
justification, and I don't see any.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/python/typing/issues/258#issuecomment-238761267, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMh0Zbs6lSZY-kCaNU_klenn_98Zeks5qeU-CgaJpZM4Jb--Z
.

--Guido (mobile)

[markshannon]

Which existing notation?
Declaring the type of parameters, return values and generic types is covered by PEP 484.
The type of local variables can be perfectly inferred from the assigned values.
For attributes of a name spaces (classes and modules) inference may fail, but should be generally good enough.
For those values where the inferred type would be either too specific or general, then a type comment should suffice.

[gvanrossum]

The type comments. Even when types can be inferred, sometimes adding them
explicitly helps the reader. And there is currently no good solution for
declaring an instance variable without initializer in a class.

On Tuesday, August 9, 2016, Mark Shannon [email protected] wrote:

Which existing notation?
Declaring the type of parameters, return values and generic types is
covered by PEP 484.
The type of local variables can be perfectly inferred from the assigned
values.
For attributes of a name spaces (classes and modules) inference may fail,
but should be generally good enough.
For those values where the inferred type would be either too specific or
general, then a type comment should suffice.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/python/typing/issues/258#issuecomment-238768891, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMhDO0DsN4cMJoESsC8Lh4CXdTY1qks5qeWBzgaJpZM4Jb--Z
.

--Guido (mobile)

[NeilGirdhar]

Regarding typed instance variables with initializers, it might be worth comparing this proposal with the ipython traitlets project, which is a heavier version of that.

[ilevkivskyi]

@gvanrossum There is one important question: Currently, there is a statement in the draft

If there's a type annotation there can only be one variable to its left, and one value to its right

I think that the limitation for the right side is not necessary, and these could be allowed (both assignments currently work, if you omit the types):

a: Tuple[int, int] = 1, 2
b: Tuple[int, int, int, int] = *a, *a

If we are going to stick only to the left side limitation (one variable per type annotation), then it looks like there is a nice and simple way to change the grammar by adding a "declaration statement" (you choose the name). This requires only two small changes to Grammar/Grammar:

• small_stmt: (expr_stmt | decl_stmt | del_stmt | ... etc.)
• decl_stmt: NAME ':' test ['=' (yield_expr|testlist_star_expr)] (right hand side is the same as for normal assignment in expr_stmt, type annotation is the same as for functions). Note that this prohibits adding annotations for attributes _after_ class creation, i.e., a.x: int = 1 is prohibited.

The above change requires to add an AST node:

stmt = 
    ...
    | Delete(expr* targets)
    | Declare(identifier name, expr annotation, expr? value)
    | Assign(expr* targets, expr value)
    ...

If you agree with this, then I will proceed this way.

[ilevkivskyi]

@gvanrossum I tried my idea above and it turns out it requires a bit more work on grammar to avoid ambiguity, so that the actual implementation is a bit different form described above. I implemented the parse step (source to AST) with the rules described in previous post and nice syntax errors, here is a snippet:

>>> x: int = 5; y: float = 10
>>> a: Tuple[int, int] = 1, 2
>>> b: Tuple[int, int, int, int] = *a, *a
>>> x, y: int
  File "<stdin>", line 1
SyntaxError: only single variable can be type annotated
>>> 2+2: int 
  File "<stdin>", line 1
SyntaxError: can't type annotate operator
>>> True: bool = 0
  File "<stdin>", line 1
SyntaxError: can't type annotate keyword
>>> x, *y, z: int = range(5)
  File "<stdin>", line 1
SyntaxError: only single variable can be type annotated
>>> x:
  File "<stdin>", line 1
    x:
     ^
SyntaxError: invalid syntax

(note: set of allowed expressions for type annotations is exactly the same as for function annotations)

If you think this is OK, I will continue with the actual compilation.

[gvanrossum]

@NeilGirdhar Regarding traitlets, that seems focused on runtime enforcement -- here we're interested in signalling types to the type checker. We're also only interested in a notation that reuses PEP 484 as the way to "spell" types (e.g. List[int]).

[gvanrossum]

@ilevkivskyi I am not super committed to allowing only a single expression on the right, but I feel allowing it would just reopen the debate on why we are not allowing multiple variables on the left.

Regarding your syntax: I'm glad you've got something working, but IIRC the parser looks ahead only one token, which means that if your have a rule

decl_stmt: NAME ':' test # etc.

it will prevent the parser from also recognizing regular assignments

expr_stmt: testlist_star_expr # etc.

that start with a NAME. That part of the grammer would become ambiguous.

In addition I think we need to allow more than just a single name on the left -- I think we must allow at least self.NAME which would make the grammar even more ambiguous.

My own thoughts on how to update the grammar are more along the lines of changing the existing

expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
                     ('=' (yield_expr|testlist_star_expr))*)

to

expr_stmt: testlist_star_expr (declaration | augassign (yield_expr|testlist) |
                     ('=' (yield_expr|testlist_star_expr))*)
declaration: ':' test ['=' test]

There will then have to be some additional manual checks to ensure that when a declaration is detected, the testlist_star_expr is actually a simple lvalue -- such tests are already necessary for assignments, because the grammar actually allows things like

len(a) = 42

I guess we can quibble about whether the initializer should be test, testlist or more, and that's not ambiguous.

[ilevkivskyi]

@gvanrossum

That part of the grammer would become ambiguous

Exactly, I implemented something very similar to the version you propose, and indeed I added some checks to ast.c. I could allow arbitrary dotted names (so that a typecheker can decide is it OK or not).

Ah, you just didn't write it up that way in your previous comment.

I'd accept anything that's acceptable as a single lvalue, so foo().bar, foo[x], foo[x].bar but not foo() and not (x, y) (the latter so people won't get too clever and try to declare multiple variables in one declaration).

[ilevkivskyi]

@gvanrossum
OK, I allowed the dotted names on the left.

Actually, I have just checked that the grammar in my implementation is equivalent to what you propose expcept with declaration: ':' test ['=' (yield_expr|testlist_star_expr)]. The logic here is that the allowed type annotations are the same as for functions, and allowed right hand sides are the same as for normal assignment.

It looks like it does not lead to ambiguity, at least all the test suite runs OK and I tried to play with new syntax a bit without generating any errors in parser.
Do you think it is OK or do you prefer test for the right hand side?

[gvanrossum]

I really prefer the more restrictive version; we can allow more in the future, and users can always put parentheses around their yield_expr or star expression. We may end up loosening this again during the PEP feedback cycle, but right now I prefer the more restricted thing, for the reason I explained (not wanting to open up the discussion about commas on the LHS).

[cheery]

This is a fine syntax for types. But should Python have types annotations in the first place?

I come from reading typescript programs where they abuse classes and inheritance to get type declarations for everything. I saw inheritance at interfaces and thought myself "never again". So that led me to left type annotations out from Lever programming language.

Type annotations aren't really what I would approve. They lead you to import modules that you wouldn't otherwise import. Create dependencies between things you wouldn't otherwise create dependencies between. I think it's inflexible and retarded. It shows out when you read Haskell.

I think the async and await inclusion into Python3 was pretty dumb too.

[blakev]

This feels like a solution looking for a problem.

These variable declarations and the existing type annotations are essentially unused by the run-time. Since they're optional, pointless and clunky -- why would I want to use them? I can already convey this information in several docstring formats that already get picked up by the IDE, so what does this solve?

If you want to make these "optional" type hints the norm in Python 4.0 for performance benefits it would be advantageous to convey your intentions now instead of trying to redefine how docstrings are already used.

[jstasiak]

@markshannon

What is the point of variable declaration? You haven't given any justification, and I don't see any.

There's an issue I find myself working around much too often - we use flake8 on our codebase and one thing it'll complain about is unused imports. If something is imported and used only in a # type: something comment flake8 won't notice[1]. This has at least a couple of downsides:

• If you import multiple objects from a module and one of them is used only in a comment you either need to split imports into multiple import statements or decorate the whole block with # noqa
• A # type: Something comment may be removed from the code and the # noqad import may remain unnoticed

It's not necessarily an issue worth modifying the language but it's something.

[1] https://gitlab.com/pycqa/flake8/issues/118

[JukkaL]

This proposal looks good to me! Some comments:

• String literal escapes may be needed in module-level and class-level annotations. They wouldn't be needed for local variable annotations as these aren't evaluated.
• We'd almost certainly want to support ClassAttr in type comments (for example, x = 0 # type: ClassAttr[int] in a class body). This could be useful for older Python versions, assuming an updated typing from PyPI is used. ClassAttr even seems like a separate proposal from the new syntax and we could discuss it separately.
• If we don't support mutating the default value of an instance variable via the class object (C.a += 1), potentially a lot of existing annotated code that doesn't use ClassAttr would have to be updated.
  
  
  
  • In particular, we'd have to update existing typeshed stubs to use ClassAttr annotations. Many class attributes are likely constants and for them using ClassAttr consistently would be less important, assuming that reading the default value of an instance variable would be okay in any case.
  
  
  • I guess that making backward incompatible changes is still acceptable, but it would be good to estimate the impact before making a decision.
  
  
  • Code that needs to remain compatible with pre-3.6 stdlib typing implementations could be problematic, as they don't have ClassAttr and thus no way to define class attributes.
  
  
  • If the annotation is necessary, some trivial annotations such as x: ClassAttr[int] = 0 would be needed, as the types of class attributes couldn't be inferred.
  
  
• Type comments are used a lot in typeshed. Eventually we'd want to migrate typeshed to use the variable annotation syntax instead of type comments. This would improve readability of stubs.
• It's easier to detect mistyped type annotations than mistyped type comments. For example, if a programmer writes a comment like # Type: int it would likely be silently ignored by tools, whereas a mistyped annotation would usually result in an error.
• As the motivation seems unclear to some people, perhaps the PEP should explicitly state the intended use cases. Variable annotations are mainly useful for code that uses PEP 484 annotations together with static type checkers or other code analyzers. The main benefits over type comments are better readability and the ability to declare the type of a variable without initializing it with a value. There's usually no reason to use variable annotations in otherwise unannotated code.
• If variable type annotations are part of the language syntax, then AST implementations would likely preserve them. Currently processing type comments is not universally supported by Python parsers. This can be somewhat tricky to implement, as many parsers strip comments pretty early on and modifying them to preserve some comments can be a lot of work.

[wITTus]

I don't like this proposal either and totally agree with @blakev. All other popular statically typed languages try hard to reduce the amount of type annotations (Scala, Rust, C++, C# and even Java!). Furthermore I'd vote heavily against variable declarations without initialization, because this will lead to patterns like C++'s RAII. Also, I don't see the benefit of UnboundLocalError.

[jstasiak]

Not sure this has been mentioned - would it handle attributes having different types depending on whether they're accessed through a class or through an instance of it? (Custom descriptors[1] come to mind)

[1] https://docs.python.org/3/reference/datamodel.html#object.__get__

[guettli]

I quote Guido:

But I can dream, can't I?

Yes, dreaming and talking about dreams is fun and helps to develop better tools.

My dream is a programming language that is easy, feels like flying, executable pseudo code, ....

How many python interpreters exists? Please think your self before reading the next line.

There are much more. There is an interpreter in each human being calling himself python programmer :-)

You can put new fancy stuff into the real python interpreter easily. But is this topic (variable declaration syntax) really important to do an update into all those interpreters running in ours heads?

My eyes want to read less, not more if I look into my favourite IDE.

Please keep the language simple and stupid.

I think I read this dialogue several years ago:

Java-Guy: Java is better, we do checks at compile time.
Python-Guy: We have unittests.

Question: If "variable declaration syntax" is the strategy, then what is the goal?

In other words: what's the benefit? I see mostly discussions about the "how to do it".

[gvanrossum]

I have nearly a year of experience now of needing it. Of course I'm
focusing on the how -- the why is answered for me already.

[blakev]

@gvanrossum if the annotations and declarations are completely optional and add no performance gains, then what's the real benefit of these features that can't be added to docstrings?

[refi64]

They're prettier:

def f(x, y):
    ''''
    :param: x List[str]
    :param: y int
    '''

vs

def f(x: List[str], y: int):

[JukkaL]

@blakev Many projects already use various docstring conventions, often without much consistency and generally without conforming to the PEP 484 annotation syntax yet. We experimented with teaching mypy to parse docstrings, but gave up since there was just too much noise and ad-hoc conventions in legacy code. Since traditionally docstrings have had no particular enforced style, it would be hard to change this now if we want to support legacy code (which we do). Adding a totally new syntax that previously wasn't valid syntactically conveniently sidesteps this issue.

[cheery]

I have nearly a year of experience now of needing it. Of course I'm focusing on the how -- the why is answered for me already.

@gvanrossum I haven't followed your every step. Details would be fun. If you've already explained the why somewhere then link to it.

Now, I realize that my earlier reasoning is pretty bad. The NoFlo guys already wrote so darn ugly python that I had to toothbrush my eyes after that incident. But what I said about details hold. Details are indeed fun.

[blakev]

@kirbyfan64 I don't think so,

def f(x, y):
    """ Something useful about what this function does.

        Args:
            x (List[str]): something useful about x
            y (int): it's just a number after all
    """

Since traditionally docstrings have had no particular enforced style, it would be hard to change this now if we want to support legacy code (which we do). Adding a totally new syntax that previously wasn't valid syntactically conveniently sidesteps this issue.

@JukkaL how does adding new syntax support legacy code any more than new docstrings would solve the same problem? I think the .pyl format is the best way to support legacy code. (example from typeshed)

[ilevkivskyi]

@gvanrossum I have got a working implementation at https://github.com/ilevkivskyi/cpython
I am fixing some failing tests in stdlib due to unexpected __annotations__ in modules and classes.

It looks like what I did follows the draft at https://github.com/phouse512/peps/commits/pep-0526 by @kirbyfan64 and @phouse512 with one exception: I don't store __annotations__ in the interactive mode (still they are stored in modules and exec).

Maybe there are some corner cases that I am missing, could you please take a look? It will be very helpful to have some feedback at intermediate stage.

[ilevkivskyi]

Also I currently do not support x: class int = 5, however this could be added if necessary.

[gvanrossum]

Am on vacation, will look Monday. Not implementing 'class' is fine.

--Guido (mobile)

[fperez]

@gvanrossum, forgive me if I missed it, I did try to skim the entire python-ideas thread... This looks great, but I wonder if it's not worth asking once more about allowing the multiple-variable annotation

a, b, c: float

syntax, _without_ simultaneous value assignment. Maybe this is my C/scientific code experience overly biasing me, but I know that declaring a bunch of variables of the same type is pretty common at least in those code bases.

In the python-ideas thread, the only concern I see about this comes from this post of yours, specifically this part:

And that's why I really don't want to go there. What if someone wrote

T = Tuple[int, int, str]
a, b, c: T

Do we now have three Tuple[int, int, str] variables, or two ints and a str?

I think that as long as the a, b, c: T syntax does not allow initialization, thus avoiding other ambiguities, this can be fairly cleanly taught as having "C-style" semantics (in C obviously the type goes first, T a, b, c;, but I think the idea remains).

Basically the idea to teach becomes: "in type declarations, ":" if for types, "=" for assignment/unpacking". This still allows for clean multiple variable initialization if desired:

a, b, c, d : int
a, b, c, d = 1, 2, 3, 4

This manages to avoid ambiguities while being significantly more compact and readable than

a: int
b: int
c: int
d: int
a, b, c, d = 1, 2, 3, 4

or (ant this might not be viable if the initialization must come later for logical reasons, as was argued in the python-ideas thread):

a: int = 1
b: int = 2
c: int = 3
d: int = 4

As a potential illustration of this type of code in the real world, the __init__ method for the dopri5 class in scipy sets a bunch of instance attributes that could be declared with this syntax. I'm sure other, potentially better examples could be found, but at least wanted to show this kind of code does indeed exist in the real world (thx @Carreau for that example).

[gvanrossum]

@fperez: I may have to cave on this one, but I'm not quite emotionally ready for that yet.

My key argument (which may be hidden somewhere else in the megathread) is that if we support the a, b, c: int notation (even without initialization) for variable declarations, people will be mislead into believing that it also works that way in function declarations, where it means something different:

def foo(a, b, c: int) -> int:
    ...

means that a and b have no type (or type Any) and only c is an int.

The other thing is that when initializing a variable with a constant of the right type, the type information is redundant and typically better left out. So if you currently have

a, b, c = 1, 2, 3

there really is very little reason to add types. Types become more useful when they are more complex, like

a: Optional[Dict[str, int]] = None  # Line count per file

where there's much less pressure to try and combine a bunch of variables in one line (and there's also more room for a comment :-).

I do realize I may lose this one and I recommend that my PEP co-authors at least incorporate a discussion of this issue, but it's easier to add this later than it would be to remove if we found it was confusing.

[guettli]

@ JukkaL you said:

Since traditionally docstrings have had no particular enforced style, it would be hard to change this now if we want to support legacy code (which we do). Adding a totally new syntax that previously wasn't valid syntactically conveniently sidesteps this issue.

This reminds me of a joke: https://xkcd.com/927/

[srkunze]

@kirbyfan64 Forgive me that I find docstrings prettier. ;)

[ilevkivskyi]

@gvanrossum

Am on vacation, will look Monday. Not implementing 'class' is fine.

OK. In the meantime I changed the behavior of exec so that it stores __annotations__ in globals, not locals (this seems in line with the PEP draft), also I added more tests for grammar, parser, and updated Tools/unparse.

I fixed all remaining tests (except datetime failure on Casablanca time zone when run test with -u all; maybe this is because they have DST twice per year, but I think it is not my fault). I run bigmem, refcount, and memory leak tests (-M, --findleaks, -R), everything seems to work OK.

[fperez]

@gvanrossum thanks for considering. And if you end up rejecting it, at least this ensures the rationale is more fleshed out and can make it into the PEP...

I see the concern about confusion with function call syntax, though I think that's somewhat mitigated by the fact that is already a special area of the syntax. For example, f(x, y, z=3) is valid but a plain (x, y, z=3) is invalid syntax.

As for having the type information be redundant when there's a value to initialize with, I'm thinking a bit of IPython here: it's easier for us to use the type information to provide useful interactive support (tab completion, help, etc) if that data is available to us in some introspectable entity provided by the language rather than by having to do full-blown type inference ourselves.

In any case, very excited about how 3.6 is shaping up, and I really hope you can squeeze this ito the calendar!

[webknjaz]

So how the following is supposed to work?

a: int = 5
del a
a: str = 'hi'
print(a)

[Carreau]

This is a very long thread, and sorry if I missed some part, but what would be the style guide that extend pep8 ? There seem to be a convention of no space before colon, and space after. I think that the spacing can make some difference in the perceived grouping, example with python dicts:

x = {a : 1,b : 2,c : 3,d :  4,e : 5}`

and the classical c++ "goes to" operator:

#include <stdio.h>
int main(){
    int x = 10;
    while (x --> 0) // x goes to 0
    {
        printf("%d ", x);
    }
}

In the proposal of assigning a type to multiples variable, it might be simple a question of tweaking the recommended style guide for whitespace:

a, b, c, d:int
# vs
e, f, g, h : int
# vs extreem
x,y,z,y : int

The second one looks _at least to me_ much less ambiguous.

As for having the type information be redundant when there's a value to initialize with, I'm thinking a bit of IPython here: it's easier for us to use the type information to provide useful interactive support (tab completion, help, etc) if that data is available to us in some introspectable entity provided by the language rather than by having to do full-blown type inference ourselves.

Agreed with @fperez , I tried to get mypy working in an interactive session, as I work 80% of the time at the command line. Having the possibility to have type introspection capability would be tremendously useful, and is one of the features that might help push the adoption of Python 3 among our end end-users in the scientific comunity.

Out of curiosity about the following example, I'm wondering how many function declaration type annotate only some of their parameters.

def foo(a, b, c: int) -> int:
    ...

[gvanrossum]

@ilevkivskyi, for exec I think locals would be better, since that is where assignments go. The concerns about "local slots" don't apply to exec.

[ilevkivskyi]

@gvanrossum

for exec I think locals would be better, since that is where assignments go. The concerns about "local slots" don't apply to exec.

OK, will do, this is very easy to change.

Actually, I have another short question: what exactly to do with dotted expressions?
Currently, I don't store them, so that

x.y: print(5)

will print 5. But the result of evaluation (None in this case) will not go anywhere (also if x was not defined this raises NameError).

Also, f()[0].x: int is accepted as syntactically correct, but I think it is OK, since one can write

y = f()[0]
y.x: int

also f()[0].x = 5 is a valid statement, so that it _could_ be useful to write f()[0].x: int = 5.

What do you think?

[gvanrossum]

The question about what to do for more complex assignment targets is actually very interesting, even apart from the question about what to do with __annotations__ for those. For example, if we agree that

f().a: int

calls f(), does that mean that

d[f()]: int

also calls f()? But it can't call d.__setitem__(), nor can we call x.__setattr__() in

x.a: int

if x overrides __setattr__.

For all of these maybe the answer should be that if there's no value being assigned (no = ...) then nothing of the LHS should be evaluated (so in the first and second example, f() should _not_ be called). Although that's a little unsatisfying, since it means that if f is undefined this will not be caught at runtime.

Independent from that, I think that the rule ought to be that the type expression should always be evaluated at the global and class level (and by exec()), and never at the local level (inside def).

And independent from _that_, __annotations__ should only be updated if the LHS is a simple name (and it's not a local). Edge case: if the LHS is also named in a global or nonlocal statement we should still treat it as a local and not update __annotations__ (in particular, with global we should not update the global __annotations__).

A type checker should be much more picky and probably reject annotating variables named in global or nonlocal or if the LHS is more complex than self.something.

If someone really wants to introspect a class's variable declarations it makes sense to say that the code should put all those declarations at the class level and not in __init__().

Finally, I am now convinced that we should not to the a: class t syntax and instead use a: ClassVar[t]. [Attn @phouse512, @kirbyfan64]

[markshannon]

There seems to be a lot of implementation effort going on. What are you implementing?
I really think that we need a PEP first. The community needs time to discuss this.

[ilevkivskyi]

@gvanrossum Let us consider two cases: name annotations, and more complex annotations.

1) If I understand correctly, for _name_ annotations the rules are simple: in class or module scope (including exec) they are evaluated and stored, everywhere else (e.g. in function scope or interactive input) they are just ignored (but still available in AST). This is exactly what the latest version does, if I have a module testm.py:

x = 1
y: print('y')

def f():
    global y
    x: print('fx')
    y: print('fy') = 1
    z: print('fz') = y

class C:
    x: print('C.x')
    def __init__(self):
        self.x: print('self.x')
        self.y: print('sely.y') = 1

f()
C()

then I get:

>>> import testm
y
C.x
>>> testm.__annotations__
{'y': None}
>>> testm.C.__annotations__
{'x': None}

2) In current implementation d[f()]: int is prohibited. Only Name and Attribute AST nodes are allowed as target. I could allow Subscript if you think that it should be allowed (but I think it does not look very useful, also it would be more like a "value" annotation). For me, the main reason to evaluate expr in expr.name is to catch errors like this:

def __init__(self, x):
    slef.x: int # Note the typo

To be honest, after all the discussions I start to think maybe "There should be one-- and preferably only one --obvious way to do it", in the sense that all annotations should be in class definition, not in __init__ and we don't need anything except _name_ annotations. Or at least restrict LHS to only name.name, otherwise there are too many options. What do you think?

As a side note, I tried to benchmark my implementation just for fun, and I didn't find any visible slow-down, although it adds two opcodes to every class and every module.

[ilevkivskyi]

@markshannon The draft PEP is developed right now, and I think it is easier to try ideas and make design decisions, if there is something to play with (this is true at least for me).

[gvanrossum]

Ivan, agreed on the LHS.

--Guido (mobile)

[vedgar]

Just a simple question: this is still an upgrade to PEP 484, its preamble staying the same? More precisely, https://www.python.org/dev/peps/pep-0484/#non-goals are still non-goals? Ie, this is not "Python gets variable declarations", but "Python gets syntactic sugar for better interoperability with 3rd party tools which need variable declarations"?

Ned has a powerful rule on his website, "Names have no type, values have no scope.", which really helps a lot to explain to people how exactly are Python variables different. If names suddenly start having a type (from Python's perspective), it would really blur that line and generate a lot of confusion.

[markshannon]

This needs to be a new PEP, not an extension to PEP 484. I'll let others comment on the goals of said PEP.

You are absolutely right about names having no type when talking about local variables. For names in longer lived namespaces like classes and modules, it is less clear cut, but it is a good approximation.

[vedgar]

Well, unless you use custom metaclasses (__prepare__) or custom importers (exec_module), it _is_ so for all simple (non-dunder) names. And it is what I would like to preserve. I propose that an analogue of PEP 484 Non-goals section be added to (or linked from) this new PEP. Especially the bolded part. ;-)

The solution I would be most happy with, in fact, is adding the from __future__ import variable_declarations, having the same behavior as importing braces. It's probably a nice thing in itself, and if Guido designed a language today he would probably do it differently (https://mail.python.org/pipermail/python-dev/2011-December/114871.html), but that is definitely not Python (https://twitter.com/gvanrossum/status/731144798757851136).

[ilevkivskyi]

Guido, I changed the parser, so that only name and name.name are allowed for LHS. I still have a check for latter that the name on the left exist, so that:

def __init__(self, x):
    slef.x: int

gives an error at runtime, unless one has defined slef in an outer scope :-)

[gvanrossum]

Python gets syntactic sugar for better interoperability with 3rd party
tools which need variable declarations

Yes, that's the idea. Apart from being new syntax, the new PEP is not any
different in philosophy from PEP 484.

[ilevkivskyi]

@gvanrossum @phouse512 @kirbyfan64
I propose to change the word "declaration" to "annotation" in the new PEP title and body (also maybe change "variable" to "name" or "attribute" or both?)
The point is that I have seen many people who got confused by "declaration". Also it would be consistent with function parameters annotations, they have syntax NAME [':' test] ['=' test], while here we have _effectively_ (NAME | NAME.NAME) [':' test] ['=' test] that is very similar.

[gvanrossum]

+1 on "variable and attribute annotations".

--Guido (mobile)

[blakev]

I've lost a lot of respect from this discussion. All of the concerns were conveniently side-stepped and instead the "full steam ahead" portions are given ample time and attention.

Carry on with bloating the language.

/unsubscribe

[markshannon]

Remember that this is just a discussion about how to formulate the PEP.

There will be plenty of discussion of the PEP itself. And plenty of time to do that. This is too big a change to be discussed properly in time for 3.6 so it will be for 3.7.

[ilevkivskyi]

@gvanrossum @phouse512 @kirbyfan64
I think there is another point that causes confusion for many people. The title mentions _syntax_ of annotations, but the body quotes PEP 484 and discusses also "semantics" of type hints for variables. Of course the main and intended use case is type hints, but this is also valid _syntactically_:

alice: 'well done' = 'A+'
bob: 'what a shame' = 'F-'

I propose to shift focus more to analogy with PEP 3107, maybe mention that functions can store structured metadata in __annotations__, but not classes/modules, and we want to remove this "injustice", etc.
Maybe we could even move most of the discussion of "semantics" to PEP 484 (anyway ClassAttr will be added to typing.py and it is still provisional). On the practical side, making the new PEP compact (in both textual and conceptual senses) will increase chances that it will be ready for 3.6beta1.

[srkunze]

I really cannot see any value in this kind of artificial annotation:

alice: 'well done' = 'A+'
bob: 'what a shame' = 'F-'

IF there is any real-world use-case here, then the PEP should mentioned it. If it is not, the PEP should honestly tell us so.

Also type annotations like the following

a: int = 5

are completely useless. I don't want a PEP encourage bad variable names. Almost all examples in this discussion don't use sane variable names.

[srkunze]

@blakev

I've lost a lot of respect from this discussion. All of the concerns were conveniently side-stepped and instead the "full steam ahead" portions are given ample time and attention.

Let's see if @gvanrossum or @JukkaL will answer them eventually.

[gvanrossum]

FWIW I will address the reason I want this feature at length before the PEP goes to python-dev, and hopefully well before that happens, but basically, we've been using PEP 484 at Dropbox for almost a year now, and we see a lot of uses for variable annotations with type comments that would just look better with the syntax proposed here. The motivation is connected to PEP 484 -- we want type annotations to look consistent. I am looking forward to constructive feedback.

[gvanrossum]

This is too big a change to be discussed properly in time for 3.6 so it will be for 3.7.

@markshannon, I'm not sure I agree. The basic decision points seem to be the following:

• Do we want variable/attribute annotations at all? (Or are # type annotations sufficient?)
• What should the basic syntax look like? Options seem to be:
  
  
  
  • <keyword> <variable>: <type> = <value>
  
  
  • <variable>: <type> = <value>
  
  
  • Something else?
  
  
• What runtime effect should it have, if any?
• Syntactic nits, e.g. whether and how to annotate multiple variables on a single line.

Recent discussion on python-ideas made great progress on most of these
issues (the initial comment here was my attempt to summarize that
discussion) except for the go/no-go decision, which can wait. Details
of the design often affect how positive people think about a proposal,
so I want to get the design hammered out before asking the big
question.

Nevertheless my own _motivation_ comes directly from practice. At
Dropbox we've been using EPP 484 for nearly a year now, and I've found
the current way to annotate a variable lacking readability.

When I see

path = None  # type: Optional[str]  # Path to module source

I have a hard time visually separating the variable, the type, the
initial value, and the comment. Switching to the proposed notation

path: Optional[str] = None  # Path to module source

shortens the line, avoids two separate uses for #, and its
similarity to argument annotations reinforces its meaning to novice
readers. (This effect goes both ways, though argument annotations are
more common.)

In addition, PEP 484 currently doesn't have a way to annotate a
variable without giving it an initial value. This need is common for
instance variables (especially when the type is a non-optional mutable
type). The similarity to argument annotations suggest an obvious
solution: keep the type but leave out the value.

As for the basic syntax, once I convinced myself that CPython was able
to parse the keyword-less form (using no worse a hack than for parsing
existing assignments) there was no doubt in my mind that it was the
superior syntax.

We've mostly sorted out the runtime semantics (see comments by
@ilevkivskyi) and we're close to sorting out the syntactic nits. His
effort creating a demo implementation has also turned up one or two
issues that would otherwise have been missed in the design.

So honestly, for me the remaining issues are

• Writing the draft for the PEP
• Deciding how the nits should be handled
• Presenting to python-dev

In the meantime this issue can be used for all open questions.

[gvanrossum]

@ilevkivskyi In PEP 484 we are actually trying to shift away from the overly vague PEP 3107. For PEP 526 I want to be crystal clear about the intended use case and the syntax and semantics of the annotations! While Python with PEP 526 shouldn't object to

alice: 'well done' = 'A+'
bob: 'what a shame' = 'F-'

(since it shouldn't care about the type annotation beyond "it evaluates without raising"), a type checker that encounters it will flag it, unless disabled with # type: ignore or @no_type_check

However, since Python won't care what the "type" is, if this is at the global level or in a class, __annotations__ will include {'alice': 'well done', 'bob': 'what a shame'}.

All these things should be spelled out in the PEP draft.

[vedgar]

One big difference between parameter annotations and general annotations is that functions have @no_type_check (and even @no_type_check_decorator), a natural way to label their signature with the metadata saying that their annotations are not to be interpreted as types. I would very much like to extend that to general annotations, but I'm not sure of the required syntax in case of globals. Anyway, I'd like to propose these extensions to the semantics of current no_type_check meta-annotations:

1. @no_type_check on a function (either directly, or via @no_type_check_decorator-decorated decorator) tells that _all_ the annotations for names in its scope are not to be interpreted as expected types of values named by them. So, not only does it encompass parameters in the function's signature, but also local variables in its body. It also recursively encompasses signatures and bodies of local (inner) functions defined inside the decorated function.
2. (https://github.com/python/mypy/issues/607 must probably be fixed for this to work) A class can also be decorated with @no_type_check (and its decorator with @no_type_check_decorator), also with meaning that all the annotations for names in its scope are not meant to be interpreted as expected types of values named by them. So, it encompassess ClassAttrs, ordinary class variables, and effectively puts @no_type_check on all the class methods. Ideally, it would also apply to all the subclasses of a decorated class (be "contagious" like metaclass curently is), but that's probably too hard to do.
3. (Here I'm not sure about the syntax, but I'm sure I want the feature) A module can be "decorated" in the same way, by executing no_type_check(__name__) in its global context. It encompasses all globals annotations inside the module, and puts @no_type_check on all the classes and functions defined inside it, but (important!) does _not_ encompass modules imported from it - those are type_checked or not according to their own no_type_check(__name__) statement, or the lack of it.

You'll probably ask about use case: we are writing a banking software, and we're using SQL Alchemy for the interactions with the database. So our actions are methods, but not only that: for the sake of auditing, all their invocations are logged in a separate database (journal). To facilitate that and to enhance the readability of the journal, actions' parameters are annotated with their types - but those are types from the perspective of the database, not necessarily from the perspective of Python. So we are adhering to the spirit of PEP 484, but not to the letter of it. It helps us a lot, and we'll probably make use of PEP 526 annotations (at least for instance attributes of contracts) if we'll be able to. (Currently we use a mix of docstrings and comments, and, just as with you at Dropbox, we found that solution a bit untidy.:)

[ilevkivskyi]

@vedgar, quoting PEP 484:

"the current proposal defines a decorator @no_type_check which disables the default interpretation of annotations as type hints in a given class or function".

Please note the words "in a given class or function", so that your first two points are covered by this, if mypy behaves in a different way, this probably should be discussed at mypy tracker. Second quote:

"# type: ignore comment on a line by itself disables all type checking for the rest of the file"

so that you third point is also covered, at least if you have your imports at beginning of file. I think that mypy should also check the modules import after single #type: ignore comment on a line, but this discussion again rather belongs to the mypy tracker.

[ilevkivskyi]

@gvanrossum

All these things should be spelled out in the PEP draft.

OK.

I have another small question on implementation (btw, currently I have a fully functional implementation with several tests for the new syntax and semantics, if details will change in latest version of the PEP I will adjust accordingly). However, there is one thing that has not been discussed yet: what to do with annotations in interactive mode? Now I simply ignore them, of course if one defines a class in interactive mode, annotations are stored in its __annotations__, but x: int in interactive input does nothing. Now I start thinking maybe this is inconsistent, since if I execute exec('x: int') in interactive mode, then it puts __annotations__ = {'x': int} in locals(). Maybe we should create __annotations__ in locals() at start-up, and all annotations from interactive inputs will go there? What do you think?

Also I would like to mention, that __annotations__ is writeable in current implementation, one can put something there by hand __annotations__['s'] = str. I think this is OK, since __doc__ is also writeable. Also for functions __annotations__ is writeable (although one cannot fool inspect.getfullargspec).

[Carreau]

Now I start thinking maybe this is inconsistent, since if I execute exec('x: int') in interactive mode, then it puts annotations = {'x': int} in locals(). Maybe we should create annotations in locals() at start-up, and all annotations from interactive inputs will go there? What do you think?

If that was possible, it would be of great help, especially for teaching; In classes that walk student interactively through various algorithms, like for eaxmple AeroPython from Lorena Barba is is extremly common to define variable interactively and ask student to complete some of the answers. Being able to fetch the annotation of already existing variable in the global namespace, and use them to analyse the piece of code the student is writing would be extremely valuable.

I'm taking the example of student above, but even as a day-to-day interactive usage that would be of high interest as ~50% of my python code is not in .py files.

[gvanrossum]

Yeah, let's create and update annotations in interactive mode. It feels
more like globals than like locals. Making annotations mutable is fine
(in fact I'd say by design). (I'm surprised by your claim about
getfullargspec() but don't really care.)

BTW When does annotations on a class get created? (There are some trick
questions here, e.g. how is it inherited.)

[gvanrossum]

@vedgar https://github.com/vedgar: I don't think the effect of
@no_type_check should extend to subclasses. Like other ways of disabling
type checks, it has a local effect.

[srkunze]

Thanks for taking this on, Guido.

• Do we want variable/attribute annotations at all? (Or are # type annotations sufficient?)

...
At Dropbox we've been using EPP 484 for nearly a year now, and I've found the current way to annotate a variable lacking readability.
When I see
path = None # type: Optional[str] # Path to module source

I don't want to criticize the code quality here but maybe before taking such example as justification, I would like to go one step back and ask a different question.

I tend to quote Raymond here (https://www.youtube.com/watch?v=wf-BqAjZb8M). You don't see the gorilla. :(

At least to me, the problem is not the comment. There are of course multiple uses of the comment but these are just symptoms of the real issue.

1) Variable Name: What about module_source_path or source_path if it is part of a class "module"? This would remove the second comment.
2) Why cannot str not be inferred from somewhere else? That it is optional, we see by None. That is can be a str should be clear from its further usage (especially with function annotations around and with IDE support). And if it can not, one might need to ask the unpleasant question: "what the heck are you doing there?"
3) Can one live without the type annotation here? Can the type inferer be smarter? Paths are usually path-like and str-like.

One way or another, removing one usage of comments is definitely possible here (and as I can speak from my experience with our codebase - it always is. For us, comments are reserved for the absolute necessary which cannot be properly expressed by code).

The code on its own is fine, but using it as an argument FOR yet another syntax doesn't seem very neat. "Clean it up" would be a better way IMHO.

[srkunze]

A question about the difference between instance and class variables, using your initial example (changing class to ClassAttr) and adding normal attribute initialization there:

class Starship:
    captain: str                           # instance variable without default
    damage: int = 0                        # instance variable with default (stored in class)
    shuttles = 42                          # class? variable
    stats: ClassAttr[Dict[str, int]] = {}  # class variable with initialization

So, adding shuttles to the mix (which is clearly a class variable), I naively would expect damage to be a class variable as well.

Put it differently, variables declared on a class body's level are usually perceived class variables. So, correct me if I am wrong but I would the following this more logical:

class Starship:
    captain: InstAttr[str]         # instance variable without default
    damage: InstAttr[int] = 0      # instance variable with default (stored in class)
    shuttles = 42                  # class variable with initialization
    stats: Dict[str, int] = {}     # class variable with initialization and type

The mere existence of an annotation would convert a class variable to an instance variable if the developer forgets to use "ClassAttr".

Maybe, I missed it but what kind of error could a type checker detect by knowing whether a variable is a ClassAttr or a InstAttr?

[ilevkivskyi]

@gvanrossum @Carreau I added __annotations__ in interactive mode. At interactive input globals() is locals() is __main__.__dict__ so I just add it to __main__ namespace at spart-up.

BTW When does annotations on a class get created? (There are some trick questions here, e.g. how is it inherited.)

First, built-in classes (and all classes in extension modules and dynamically created classes) do not get __annotations__ at all, this is consistent with built-in functions:

>>> len.__annotations__
AttributeError: 'builtin_function_or_method' object has no attribute '__annotations__'

Second, in user defined classes it is created by adding two opcodes BUILD_MAP 0, STORE_NAME at beginning of class body code. This way it is never inherited (or rather every class overrides the base class __annotations__). This is almost exactly how __doc__ is created and behaves:

>>> class C:
...     "Base class"
...     x: int = 5
>>> class D(C):
...     pass
>>> print(D.__doc__, D.__annotations__)
None {}

One of the differences is that docstrings are not stored if c->c_optimize > 1. Maybe we should do the same for annotations, what do you think?

I'm surprised by your claim about getfullargspec() but don't really care.

I was also a bit surprised, when I found this

>>> def f(): ...
>>> f.__annotations__['x'] = int
>>> f.__annotations__
{'x': <class 'int'>}
>>> inspect.getfullargspec(f)
FullArgSpec(args=[], varargs=None, varkw=None, defaults=None, kwonlyargs=[],
kwonlydefaults=None, annotations={})

[vedgar]

@ilevkivskyi Well, of course getfullargspec will answer {}. It concerns function's parameters. Help says it maps their names to their annotations (up to a confusion between arguments and parameters). Since f has no parameters, it must have no annotations of them.

On the other hand,

>>> def f(x, y): ...
>>> f.__annotations__['x'] = int
>>> import inspect
>>> inspect.getfullargspect(f)
FullArgSpec(args=['x', 'y'], varargs=None, varkw=None, defaults=None, kwonlyargs=[],
kwonlydefaults=None, annotations={'x': <class 'int'>})

BTW, +1 on removing annotations on -OO.

[ilevkivskyi]

I tried the idea of discarding variable annotations if run with -OO. The annotations are not evaluated and not stored, all class and module __annotations__ are set to None, and annotation statements without assignment are just thrown away (of course all annotations are still available in AST).

I tested this by creating 100000 classes with only one line x: int = 5 (this is an _extreme_ case), and the memory consumption is reduced by up to almost 20% with -OO. Probably, in real life annotated code it will be not so significant, but still I think it is a good idea.

[gvanrossum]

Perhaps we should also do this for function annotations? Do you think it
would be controversial? It would be odd to be inconsistent.

[ilevkivskyi]

@gvanrossum

Perhaps we should also do this for function annotations?

I tend to think so. It will require some work, but probably it is worth doing it.

Another question: I think that it is better to implement __annotations__ as a descriptor that will prohibit setting it to something non-dictionary or non-None as it is done for functions. Do you agree?

Sorry for posting ideas so frequently, but I have one more. I was reading how inspect.getdoc and inspect.getfullargspec are implemented and I think that it would be nice to add inspect.getannotations that will work like this:

• for functions it will return the same dictionary as getfullargspec
• for modules it will return mod.__annotations__ filtered to contain only keys that appear in inspect.getmembers(mod)
• for classes, it will return (approximately) collections.ChaimMap(base.__annotations__ for base in cls.__mro__) filtered to contain only keys from inspect.getmembers(cls)

This would be useful, assuming that you agree that __annotations__ should not be inherited (see my post above). What do you think?

[gvanrossum]

After thinking about it, I think most work on -OO is misguided, and just
creates random bugs. The speed/memory gain is usually minuscule. I would
roll back this part, and not bother with function annotations.

implement annotations as a descriptor [...]

This is unnecessary as long as any C code that uses or modifies it is
careful not to crash when it's been set to something else. Even doc can
be set to a tuple! And for the module-level annotations you can't even
control it, it's just a plain variable. Let people shoot themselves in the
foot! It's fine to abstain from updating it if it's not a dict, though if
you could support a more general mapping if you wish. You could also just
raise an exception if you can't update it.

add inspect.getannotations that will work like this [...]

The inspect function sounds fine; especially the behavior for classes is
useful and complex enough that people will get it wrong otherwise. Maybe
something Lisa can help with to get started?

[ilevkivskyi]

@gvanrossum

The speed/memory gain is usually minuscule. I would roll back this part, and not bother with function annotations

OK, I rolled it back. This part is "disentangled" from the main part of implementation and could be added later, if necessary.

You could also just raise an exception if you can't update it.

OK, then I will leave this part as it is now (__annotations__ is just a normal attribute in __dict__ of a class/module). One will see something like this:

>>> class C:
...     __annotations__ = 42
...     x: int = 5
TypeError: 'int' object does not support item assignment

We should probably mention this in documentation.

There is another minor point to discuss: now a class created by C = type('C', (), {}) does not have __annotations__. Consequently new_class and __build_class__ if used manually could sometimes create classes without __annotations__. I think it is better to leave it this way. The problem is that if I patch it in PyType_Ready, then everything starts to have it int.__annotations__ == len.__annotations__ == {}, I think this is ugly. What do you think?

Maybe something Lisa can help with to get started?

You are reading my mind :-) this is exactly what I wanted to propose for her.

[gvanrossum]

a class created by C = type('C', (), {}) does not have __annotations__.

Hm... That would be slightly unfortunate if one of the base classes _does_ have __annotations__, since the simple-minded approach of walking the mro that you showed before might see some ghosts. But I think it's better to improve the mro-walking code than to always add __annotations__ in this case, so I agree.

Now I think of it, what do you think about not adding __annotations__ at all to classes that don't contain any type annotations statically? This would address the concern that most classes would grow an empty and unused __annotations__ dict that just takes up space (quite a bit, as you measured).

[ilevkivskyi]

@gvanrossum

what do you think about not adding __annotations__ at all to classes that don't contain any type annotations statically?

On one hand it makes sense, on the other hand this will be inconsistent with functions, that always have __annotations__ == {} if they were not used. Also a code like this

>>> class T:
...     exec('x: int = 5')
>>> T.x
5
>>> T.__annotations__
{'x': <class 'int'>}

will raise an AttributeError at class creation. This could be however solved easily either by creating the __annotations__ in the class body and deleting it later if it is empty, or by simply saying that dynamic annotations like above are not supported.

So, I am not sure about this question. Maybe we could ask others?

[gvanrossum]

For those following along at home, here's the PR with @ilevkivskyi's implementation effort so far.

[gvanrossum]

Re: the question of whether to add __annotations__ always or not: this was discussed previously (probably on python-ideas) and the objection was that it would create more memory bloat for code that doesn't use the feature.

If you look at func_get_annotations() in Objects/funcobject.c, you'll see that functions don't really get an __annotations__ attribute when they are created -- it is constructed (and stored) on the fly when you ask for it. So the memory bloat is contained.

You could use a similar trick for classes, but I don't recommend it -- I don't think you need it, and class getattr always descends down the MRO.

I really, really don't care about classes using exec() in their bodies.

[ilevkivskyi]

@gvanrossum
Concerning the idea with descriptor, I have noticed the code you mentioned, when I was proposing to make a descriptor that prohibits anything but dict, but now I agree with you that it is not needed.

Actually, I was wrong about exec. I implemented a simplistic version of your idea (without recursive search for annotations in substatements that don't create scope) and it turns out that the code with exec works. I am not yet sure why, but I will go this way (will not create annotations at all if they are not used, even if methods or inner classes use annotations).

[ilevkivskyi]

@gvanrossum
OK, now the __annotations__ attributes are not created if variable annotations are not found statically. Still it will be created for:

if False:
    while False:
        x: int

Now, if a base class have annotations, but a derived class doesn't, then they will be inherited. We could explain this in docs as __annotations__ gives the "latest update" of annotations in the class hierarchy, while inspect.getannotations gives the "full story".

Now I understand why exec('x: int') sometimes work. exec compiles the given string as a "module", and since it contains annotations, the __annotations__ attribute will be created in the module dict. If exec is called without arguments, then module dict will be the current locals(). This means that exec creates a _new_ __annotations__ and can override the previously created one. Indeed:

>>> __annotations__
{}
>>> exec('x: int'); exec('y: int')
>>> __annotations__
{'y': <class 'int'>}
>>> class C:
...     x: int
...     y: int
...     exec('if False: z: int')
>>> C.__annotations__
{}

I think this is _not_ OK. First, it will be quite confusing and hard to debug if someone puts an accidental exec somewhere that will "erase" all the existing annotations. Second, one will be not able to use __prepare__ to give a custom mapping to collect annotations, it will be overridden by newly created dict in the class body code.

I have an idea how to fix this. Instead of just creating __annotations__ if annotaions are found, we could add a bytecode using SETUP_EXCEPT, LOAD_FAST, and NameError (around 10 opcodes) that will check whether __annotatons__ is in locals, and if not, create it. I could not invent something at least slightly more elegant than this that will work in all cases. Do you have any thoughts?

[gvanrossum]

How about adding a new opcode to store a type annotation? Basically,

x: int = 0
y: List[str]

should compile to bytecode for

<load int>
STORE_ANNOTATION 'x'
<load List[str]>
STORE_ANNOTATION 'y'

where the STORE_ANNOTATION opcode takes a variable name (specified in the same way as STORE_NAME) and the top of the stack and does something like this

if '__annotations__' not in locals():
    __annotations__ = {}
__annotations__['x'] = int

Checking whether __annotations__ exists as a local is simpler in C!

[gvanrossum]

I'd like to clarify my use of the terms "class variable" and "instance variable", now that I've thought about it a bit more (and discussed it with @michael0x2a).

When we see

class Starship:
    captain = 'Picard'
    stats = {}
    def __init__(self, captain=None):
        if captain:
            self.captain = captain  # Else keep the default
    def hit(self):
        Starship.stats['hits'] = Starship.stats.get('hits', 0) + 1

we hopefully all agree that stats is a class variable (maybe keeping track of many different per-game statistics), while captain is an instance variable with a default value set in the class. We see this more from the usage than from the class body -- both get initialized in the class, but captain serves merely as a convenient default value for the instance variable, while stats is truly a class variable -- shared by all instances, and in fact it's mostly treated as a global with a dot in its name.

Since both variables happen to be initialized at the class level, I find it useful to distinguish them there when we add types to the class, and that's why I like to have a way to mark class variables. My original proposal was to use the syntax class <type> for this purpose, but I currently think that it's fine to just use ClassVar[<type>] instead -- slightly clunkier, but the grammar is simpler and it avoids confusing simple-minded searching and highlighting algorithms; and we need the ClassVar wrapper to store in __annotations__ anyway.

Why not also write InstanceVar[<type>] for instance variables? Mostly because instance variables are way more common than class variables (at least in most code that I read). The more common usage deserves to be the default.

Why not put the annotation on the assignment to self.captain? (Mypy actually allows this.) But many __init__ methods do a lot of things besides initializing instance variables, and it would be harder (for a _human_) to find all the instance variable declarations. And sometimes __init__ is factored into more helper methods so it's even harder to chase them down. Putting all the instance variable declarations together in the class makes it easier to find them, and helps a first-time reader of the code. (We might even consider a "strict instance variables" option that requires that all instance variables be declared in the class, but that's out of scope here.)

Why not forget about ClassVar altogether? After all mypy seems to be getting along fine without a way to distinguish between class and instance variables. But a type checker can do useful things with the extra information, for example flag accidental assignments to a class variable via the instance (e.g. self.stats = {}) which creates an instance variable shadowing the class variable. It could also flag instance variables with mutable defaults, a well-known hazard.

[vedgar]

We might even consider a "strict instance variables" option that requires that all instance variables be declared in the class, but that's out of scope here.

Just to clarify, here you're talking about option for mypy, not for Python, right? I think it would be very wrong if Python used annotations in this way, because it's incredibly useful to just tag some instances with some attributes on a case-by-case basis. Or, at least, (similarly to __slots__), it won't transfer to subclasses - so I'll be able to trivially subclass in order to break the prison of mandatory predeclarations, just like I sometimes do now to obtain a __dict__. But still, I think it's best for this to be strictly (pun half-intended:) mypy's business, not Python's.

I don't think the effect of @no_type_check should extend to subclasses.

If that means you agree with the rest of that post, I'm fine with it. :-)

What I really had in mind there is a bit more subtle: consider an ABC with an abstractmethod decorated with @no_type_check. That should probably mean that as far as that particular abstract method is concerned, its parameter annotations have a different semantics than expected types of arguments passed. To me it surely implies that _implementations_ of such an abstract method cannot suddenly "regain" the usual semantics for their parameter annotations, despite the fact that those are separate methods as far as Python is concerned, because (conceptually) those are the same parameters! And that means that implementations of @no_type_checked abstract method should also be automatically @no_type_checked. But I agree it's a little gain for a comparatively big effort.

[gvanrossum]

Just to clarify, here you're talking about option for mypy, not for Python, right?

Absolutely! Python will ignore these declarations in the same sense that it ignores function annotations.

If that means you agree with the rest of that post, I'm fine with it. :-)

Indeed I do.

Regarding ABCs decorated with @no_type_check, implementations that define concrete methods matching abstract methods in such an ABC will be checked as if the base class had no annotations. We may have to clarify this in PEP 484.

[ilevkivskyi]

@gvanrossum

How about adding a new opcode to store a type annotation?

Great idea. However, one thing bothers me a bit: _always_ creating __annotations__ if they don't exist seems to me like silencing a possible error. I think we should allow people to explicitly del __annotations__ (for example if someone wants to make a class with annotations that are "invisible" to runtime tools) and warn them if later they use annotations. Maybe a better way is to add a SETUP_ANNOTATIONS opcode that does

if '__annotations__' not in locals():
    __annotations__ = {}

and will be emitted only if annotations are seen statically, and STORE_ANNOTATION will do only:

__annotations__['x'] = int

What do you think?

I'd like to clarify my use of the terms "class variable" and "instance variable"

Allow me some bikeshedding here. I think that ClassVar[] is too similar to TypeVar(), while its meaning is completely different. Also Class[] it is too similar to Type[]. I think ClassAttr[] is better in this sense (there is also a fun option: Classy[] :-)

[ilevkivskyi]

@gvanrossum There is another point about creating __annotations__ dynamically. In such class

class C:
    if fun():
        x: int = 5
    else:
        print('no way for x')

__annotations__ could be, or could be not created, depending on runtime. I think that static creation of __annotations__ will be more predictable and easier to debug. So that I am tending more in favour of two-opcode option.

[gvanrossum]

OK on the two-opcode version.

[gvanrossum]

FWIW I am thinking about slowing down things a bit, and maybe we should accept that it's better not to rush this PEP into 3.6 after all. Few people will be able to take advantage anyways, and a mechanical converter from x = v # type: t to x: t = v is easy enough to write.

[ilevkivskyi]

@gvanrossum
I implemented the two-opcode version and it works well. There of course could be some hidden bugs and more tests are needed.

FWIW I am thinking about slowing down things a bit

What exactly do you mean by slowing down? And what do you think will we gain by waiting?
I think it is better to let it go as it goes and then we will see. There is still almost one month before 3.6 beta 1

[perkinslr]

How about initializing __annotations__ to None? If it is not included in locals(), I have to use either hasattr(myobj, '__annotations__') or '__annotations__' in locals() to determine if it is provided. If it is initialized to None, I can just check myobj.__annotations__ is not None.

[ilevkivskyi]

@perkinslr I don't think it is a good idea. In current implementation you could normally expect only two options: no __annotations__ at all, or it is a dict. There is probably no point to add __annotations__ to _all_ classes (object, int, etc). So that if we will set __annotations__ to None on user defined classes, there will be three options that one would normally expect.

As a side note, I think too much attention is now attracted to implementation details (maybe this is also my fault). It is much more important now to figure out the specification and finish the actual PEP. Implementation is already working and could be used as a proof of concept or just to play with it. After PEP will be ready, it would be quite easy to tweak the implementation details.

[gvanrossum]

What exactly do you mean by slowing down?

I expect a huge firestorm when the PEP hits python-ideas (despite the previous discussion there) and again on python-dev. I'm not sure I am prepared to fight that fight (it takes a lot to do that, and I have other things I need to do).

[vedgar]

I think that as long as you emphasize what I said above about non-goals (this is not a way to add variable declarations to Python, but a way to more nicely interoperate with external tools; if only Python reads your .py files, you don't have to worry about this), there shouldn't be much of a fight. You've already fought and won the __annotations__ fight for function signatures, this is just a natural extension.

Of course, had you introduced a new keyword, there'd be endless bikeshedding about it, but fortunately you hadn't. :-) Syntax (what little is there of it) is just as similar as it can be to existing parameters annotation, it helps to think about class attributes as __defaults__ for instance variables, and there's even a nice parallel of your refusal to handle unpacking to the rationale of https://www.python.org/dev/peps/pep-3113/#exception-to-the-rule (I only hope you won't remove Python't tuple unpacking totally by analogy with PEP 3113, since it's very handy:).

But I'm also +1 for slowing down, for a different reason. There are many technical details (e.g. what format of names is allowed) which could probably use the help of playing with working implementation. And the history has shown that the longer you think about some new feature of Python, the better it is in the end. :-)

[vedgar]

May I ask how annotations work with metaclasses? If I have

class Meta(type):
    def __new__(meta, name, bases, namespace):
        print(namespace['__annotations__']
        return super().__new__(meta, name, bases, namespace)

@no_type_check
class Spam(metaclass=Meta):
    x: 'something'
    y: 'something else'

Will this print annotations of x and y?

[ilevkivskyi]

@gvanrossum

I expect a huge firestorm when the PEP hits python-ideas (despite the previous discussion there) and again on python-dev

My experience is that most people who opposed PEP 484 and are opposing PEP 526 just do not completely understand the idea (sorry if this sounds harsh). They tend to compare this with Java, C++, C#, etc. While the idea is conceptually more similar to Clojure, TypeScript, Racket, Erlang, etc. (or even more permitting). People just see what they want to see. Or more precisely, they paradoxically see what they are _afraid_ to see (Python is becoming Java, etc). I hope we will manage to explain ideas behind this PEP, and it will be well perceived.

Allow me also another personal comment. I really like PEP 484 and PEP 526 because they "fix" a "mistake" of other languages that was made long time ago. There are two important concerns in programming: code reuse and debugging; and some other languages decided to solve them both with the same concept - a class system. I think this is one of the main reasons why that languages are so verbose and restrictive. Python, on the contrary, does not mix these two concerns (although of course they are partially overlapping): it has convenient class system that allows easy and effective code reuse and good testing tools that allow easy debugging. Now we are adding a new complementary way to debug - type hinting. I perceive type hints as kind of machine _and_ human readable code documentation (raw comments are not easy for machines, while something like JSON is not easy for humans). I am happy that Python has taken this way and want to help to continue.

I'm not sure I am prepared to fight that fight (it takes a lot to do that, and I have other things I need to do).

I will be glad to help with this. Also I would like to help with the PEP writing, will you accept PRs to the PEP development fork https://github.com/phouse512/peps ?

[ilevkivskyi]

@vedgar I just enter your code in the interpreter and it told me SyntaxError because there is missing closing ) in print, after fixing this, it told me:

{'x': 'something', 'y': 'something else'}

Looks like it is what you wanted. For the next time, you are _allowed_ to try it by yourself :-)

(side note: @no_type_check is not necessary at runtime, it is only used by mypy and others)

[rwe1234]

@gvanrossum
With PEP 526, large codebase maintainers will benefit, but community will suffer because when big companies start to use variable annotations, it will become a standard (everybody wants to get hired, right?). So, I already can imagine answers in stackoverflow for easy questions from beginners like "How to append to a list":

example: List[int] = []
example.append(3)

For small projects this is as ridiculously unnecessary as putting 2-line code into a function and then calling that function from "main" function, which we see a lot and which we all know where it came from.
Big part of Python's success is that it's very friendly to beginners and easy to read. The evil here is that you can't guarantee that variable annotations will not become a routinely everyday standard in Python community.
Please don't take Python away from us! It's a beautiful language. We don't want another C, we want Python!

[vedgar]

Thanks for your kind permission:-D, but it's much more complicated that you think. I'm on Windows, and the machine is not mine. Installing Mercurial, Tortoise and Visual Studio without admin rights is not how I imagine my vacation. :-/

[vedgar]

@rwe1234: How many people now use function parameter annotations on SO questions? I must say I don't see many examples, and they have been with us for a long time already. [About main functions: people will always cargocult. main functions are at least mostly harmless.] The key thing is to keep Python and mypy separate. As long as mypy isn't even in the standard library, people will not be incentivized enough to care.

[Michael0x2a]

@rwe1234 -- well, you could make the same argument about pretty much any new feature, Python might add, right? For example, something like async def/await could be challenging for a beginner to learn, but that doesn't mean we shouldn't have it.

In any case, it turns out that what makes a codebase readable or not can actually start changing as you scale the size of your codebase. Type annotations might not make much of a difference in small programs, but they can make your code significantly easier to understand for newcomers in larger codebases, even if they know almost nothing about type annotations. So, if we care about readability, and making Python a viable choice for all kinds of programs, big and small, type annotations are a good thing, not a bad thing!

Also, TBH I don't really see type annotations suddenly taking over the entirety of the Python world. There are some cases where dynamic typing would be more useful, and some cases where static typing would be useful, and the beauty of PEP 484 is that it'll let you pick and chose, allowing you to have the best of both worlds. It's tempting to make it an either-or kind of thing, but I don't really think it is.

In any case, the community doesn't seem to be "suffering", so I don't really think there's too much point worrying about a hypothetical that doesn't actually seem to be happening. If we see data indicating that hypothetical is becoming true, we can collectively decide what to do then.

Please don't take Python away from us! It's a beautiful language. We don't want another C, we want Python!

As ilevkivskyi said, I think it's more like Python is taking inspiration from languages like Clojure, TypeScript, Racket, Erlang, or Haskell more so then languages like C or Java :)

[rwe1234]

@vedgar
@Michael0x2a
Type annotations are not even 2 years old. Python is 25 years old. Let's see what will happen after O'Reilly releases a dozen of Python books with type annotations in default function structure.

If we see data indicating that hypothetical is becoming true, we can collectively decide what to do then.

Collectively decide to kill backward compatibility? That already happened once.
Do we want python 3 to 4 migrating problem? If not, we should strategically analyze with great care new features and their possible long-term outcomes.

[Carreau]

Type annotations are not even 2 years old. Python is 25 years old. Let's see what will happen after O'Reilly releases a dozen of Python books with type annotations in default function structure.

As annotations have been there for more than 2 years, it is easier to migrate to using type annotation directly once you stop supporting version of python for which type annotation is a syntax error.

And this new syntax for attribute and variable annotation will only be able to be use for project that only support 3.6 and above, so indeed I agree that having the syntax available soon might be useful, and I understand the concern or rushing too much.

FWIW I am thinking about slowing down things a bit, and maybe we should accept that it's better not to rush this PEP into 3.6 after all.

While I completely understand that the syntax and implementation detail want to be developed hand in hand, do @gvanrossum have considered having only the _syntax_ changes proposed for 3.6 in pep-0526 ? and polish the exact details of where/how __anotations__ are stored/interpreted/used for later?

That will allow people to develop packages that are syntactically correct for 3.6 even if the new syntax is no-op, but tinker with the actual annotations in 3.7.dev, without preventing the use of MyPy on 3.6 as well ! This should bring lots of feedback on the annotation themselves that can be used for the rest of the proposal in 3.7.

That can be compared to the same separation of concern that pep-3107 and pep-0484 have done in the past. Tools like mypy will still be able to interpret annotations as they wish and can more easily handle breaking changes if necessary, as well as bring even more informed decision on how these annotation are used.

I would also expect a _syntax addition only_ pep to be much easier to digest for many, simpler to write for the authors, and simpler to implement for all alternative parsers that ingest python syntax in the wild; Thus being ready when 3.7 bring actual python interpreter changes.

[jstasiak]

While I completely understand that the syntax and implementation detail want to be developed hand in hand, do @gvanrossum have considered having only the syntax changes proposed for 3.6 in pep-0526 ? and polish the exact details of where/how anotations are stored/interpreted/used for later?

That's an approach I've been thinking about too, glad it's been mentioned. I second everything you said.

[vedgar]

I don't think so. Some of main contention points (what exact form of names will be annotated; we probably need "a.b: int", do we need "a[1]: str", or even "a/b: Fraction"?) are syntactic. But if you mean just allowing basic (atomic) names in 3.6, with a provision to enable more in the future, then I'm +1.

[Carreau]

But if you mean just allowing basic (atomic) names in 3.6, with a provision to enable more in the future, then I'm +1.

That was also partially in the back of my mind in the comment I left, but not explicit. I think it's a good idea as well; though if still introduce a syntax difference, then it will postpone the usage of some syntax by at least a year as well.

If we look at the current status of pep-0526, I was thinking first of sections like "Capturing Types at Runtime" and in particular moving at least this point to a future pep:

It's impossible to retrieve the annotations at runtime outside of attempting to find the module's source code and parse it at runtime, which is inelegant, to say the least.

Having already annotation down to the ast level seem – to me – to be sufficient to already have plenty to play with, and as the ast module is known to change quite a bit, it still leaves python 3.7 quite some possibility to experiment with and refine.

In general, regardless of the proposition, would a reduction of scope of pep-0526 make it easier to chew-on, and easier to get in 3.6 ?

[ilevkivskyi]

@gvanrossum Sorry for not waiting for you answer, I have made a PR https://github.com/phouse512/peps/pull/1 just scrolling through this thread and python-ideas, and tried to collect all the things that should not be lost (see details in PR description).

[srkunze]

@gvanrossum

I'd like to clarify my use of the terms "class variable" and "instance variable", now that I've thought about it a bit more

Thanks for coming back to this one. It seems what you've written down is the classical separation of class and instance variables we know from statically typed languages. So far, it makes sense for this world, which this PEP tries to please.

I expected your inferences of what should preferred (ClassVar) and expected use-cases. I agree that instance variables are more frequently encountered. Moreover, I hope these theoretical use-cases will turn into real checkers.

[srkunze]

@vedgar

this is not a way to add variable declarations to Python, but a way to more nicely interoperate with external tools

That's how politicians talk and if different-minded people come the discussion (as seen above), they are ignored, their points are ignored and some well sounding theoretical/abstract benefits are repeated once more.

That's how politics work.

And make no mistake: this is a variable declaration. People will notice this, most programmers ain't fools.

there shouldn't be much of a fight.

That's your perception.

You've already fought and won the annotations fight for function signatures, this is just a natural extension.

Well, he's the boss. He has the ultimate veto and anti-veto. So, it's not as if he has "won" something.

So, if he want it in, he will get it in. It's just that he needs to justify it for himself enough by making the proposal as good as he possibly can.

[srkunze]

@ilevkivskyi

My experience is that most people who opposed PEP 484 and are opposing PEP 526 just do not completely understand the idea (sorry if this sounds harsh).

Wtf? Have you ever taken your time to actually read what all those commenters above had written?

I haven't seen serious comments trying to trying to address these concerns by providing solutions for these problems mentioned. Just repeating the same "all this is good and necessary" mantra is not a solution to their concerns.

[srkunze]

@rwe1234

Type annotations are not even 2 years old. Python is 25 years old. Let's see what will happen after O'Reilly releases a dozen of Python books with type annotations in default function structure.

@gvanrossum That's actually an interesting question on it own.

How is the broader community supposed to act on this?
Should tutorials promote the usage?
Should literature use it?
Is it for personal/internal usage only?

Some expressed that annotations have no benefit for small programs but some benefit for bigger ones? Can we draw a line? Some guidelines here?

From my personal preference, I would do 0 annotations. Why? Because I have no idea what these abstract benefits mean in concrete implementation. So, no system can be big enough for me to have them (and I work with really big ones). Some statically inclined would use them all the time, maybe because he lacks the inner voice telling him "don't do it now, but do it here".

It definitely is the logical extension of function annotations but it's quite different because local variables have mostly no outer API purpose etc. So, they would need a different motivation. This reinforces the idea that this whole business should be split into smaller pieces for the community to decide.

I could even imagine a first step introducing the new syntax for classes only.

If we see data indicating that hypothetical is becoming true, we can collectively decide what to do then.

@Michael0x2a It seems you actually don't want part of the collective to decide right now.

But when it's messed up, the collective can clean it up. ;) Sounds familiar.

Collectively decide to kill backward compatibility? That already happened once.
Do we want python 3 to 4 migrating problem? If not, we should strategically analyze with great care new features and their possible long-term outcomes.

I have no idea how we would handle this one. from __past__ import annotations? That will get messy. This is no argument for or against it. I agree with you, however, that the long-term outcome is not 100% clear to anybody in this discussion.

[vedgar]

Well, of course scandal-loving reporters are going to call this "variable declarations". It has already started all over the web. Those are the same people who called f-strings "eval in disguise", and people who called Enums "not bold enough". In most cases it just shows deep misunderstanding of the way Python works, and how these features are actually implemented.

The only thing remotely smelling of actual declaration semantics is that Python localizes locally annotated names. That part makes me a bit uneasy, too, but I understand why Guido has done it that way. I don't think it's such a big deal. As Guido says, would you consider

if False: x = None

a variable declaration of x? Yes, with this syntax, names get another way (besides assignment) to be localized. But moving in the direction of "local by default" is almost always a good thing: look at JavaScript for example. :-o In any case I can think about where you mess this up, you get a precise UnboundLocalError, and you know what you've done wrong.

Now, if you want to say that these are variable declarations _for mypy_, go ahead. But keep in mind that mypy is not Python. It is _not_ an alternative implementation of Python the language, it does not compete with CPython. Honestly, I have a feeling that someday Guido is going to want to push it into the stdlib, and I'll be against it (with strong arguments). But until then, it's just a third-party tool which reads your .py files.

After all, Python has many entrenched rules about how to write docstrings, for example, just so it can better interoperate with a certain OS that masquerades as an editor. ;-) Does this bother me, when I see """ on a line by itself? Yes, of course it does. Do I make a big fuss out of it? No, because it's really not a big deal. Do I use that convention in the codebases my team writes? No, I don't. And I'm thankful that I can make that decision without Python getting in the way.

Same thing will happen here. Yes, sometimes people (me included) will be bothered by seeing annotations in Python programs. But they will understand what they are for, and they will not _have_ to use them to have a meaningful interaction with Python the interpreter. (Some libraries will require them, but there are already libraries that require special docstring format, for example. I don't see this as very much different.) And it will stay that way. If Guido (or his successor) one day decides that Py4M will be a statically typed language, people will just stay with Py3K. Our community has a long tradition of staying with old versions way beyond what's reasonable anyway. ;-D

[gvanrossum]

OK, enough political speculation already. I'd like to refocus this tracker issue on getting the details of the PEP worked out. Good questions include whether we should take the presence of x: int in a local scope to be enough to make x a local. (I think yes, but it can be argued.) I don't want to hear any more questions about whether I see a future for mypy in the stdlib (I don't) or what the community at large will think of this (ultimately that's what python-dev is for).

[ilevkivskyi]

My vote goes to x: int having the same effect as if False: x = 1 (like it is mentioned in the PEP draft). The decision to make a name local is made statically, type annotations are also something that is intended for static use. I think this is consistent and an easy to remember rule.

[vedgar]

I agree that it's logical. The reason I feel uneasy about it is that I envision a future in which people cling to that behavior as a rationalization for "you should always declare all your variables, so you never have to worry about mutating your globals", similarly to what has happened to JavaScript in the last 10 years. But Python has got its scopes right much earlier (what is usually called "globals" are in fact module-locals), and it would be a shame for it to socially disappear just because people have irrational fears. Maybe this just goes into the "cargoculting" bucket, just like getters and setters, main fuctions, and hiding metaclasses.

The big problem is that I don't really have a sensible alternative. If we allow people to re-annotate globals inside functions, it's a good question where and whether we should track those annotations. If we report an "UnboundAnnotatedError", that is probably too harsh and belongs to linters, not Python. Maybe we really should have the rule that function-local annotations are completely ignored by Python (not "as an optimization", but "because semantics is much harder to define"), but that also seems weird.

I think I have a few more suggestions about good questions. Have we decided on the form of names we are going to allow to be annotated? Currently, if we use the "logger example", but in fact we don't set it under the name logger but self.logger (but not all objects of our class have logger attribute set), we don't have a way to annotate this unless we allow the annotation

self.logger: Logger

And if we allow that, it's a short road to allowing a[2]: int (currently decorator syntax allows dotted but not subscripted names, and some people still haven't forgiven Guido that:).

Also a potentially good question: who are the hints _for_ (primarily)? For humans, or for type checkers? "hint" might mean helping the narrowminded automatic tool, or it might mean giving the guideposts for forgetful humans along the way. I suppose the ambiguity is somewhat intentional, but there are few cases that should be cleared IMO. Let's say, for the sake of example, that mypy doesn't know that sum of two ints is an int (you can insert a more complicated operation if you want). In that case, it's prudent to annotate c in

def f(a:int, b:int) -> int:
    c: int = a + b
    ...  # do something with c that you want typechecked

Once mypy learns that fact (that int + int is int), should the annotation for c be removed? I know that in the real world it's probably going to stay anyway, but in the ideal world, is it superfluous information? For static checkers yes, for humans maybe not (again, imagine a complicated operation if it's more convincing).

Another point: maybe Optional qualifier should be redefined (or TrulyOptional be introduced:), for cases like this: I have a class A, whose instances might but needn't have a spam attribute, and we want to annotate that _if_ they have it, it is a float. Currently we don't have a way to do that. Optional[float] means "a float or None", not "a float or nonexistent". :-/

[srkunze]

we should take the presence of x: int in a local scope to be enough to make x a local

I think so. It's the most frequent usecase anyway. Everything else seems counter-intuitive to me especially because of "Their type belongs in the scope where they are defined."

it would be a shame for it to socially disappear just because people have irrational fears.

Good thought but that might happen. Not because it's wrong but because linters would complain about it like pep8ifiers do about other things. And most people cannot resist to please them.

For static checkers yes, for humans maybe not (again, imagine a complicated operation if it's more convincing).

I don't buy that argument. That's what IDEs are for. People don't need to clutter source code if the type can be inferred. Furthermore, type hints are like comments. They age, change from true to almost wrong and give false sense of security/knowledge. So, less is better.

You said "imagine a more complicated operation", that will be done via an operator or a function, right? So, IDEs will then be able to infer the return type anyway (and display it somewhere automatically).

[gvanrossum]

My vote goes to x: int having the same effect as if False: x = 1

That was my original proposal and I haven't heard any compelling argument against it, so let's just keep this. (Irrational fears would have to go against my own, much more optimistic gut feelings, and re-annotating globals is just wrong, so I don't mind that we close that off for good.)

Have we decided on the form of names we are going to allow to be annotated?

IIUC Ivan thinks only simple names and names with one dot (e.g. self.foo) should be allowed, though the text of the PEP doesn't say it. My own preference is to be a little more lenient, and allow anything that's a valid (single) assignment target -- the type checker can deal with it (it already has to deal with self.foo). Only annotations for simple (un-dotted) names are stored in __annotations__.

who are the hints for (primarily)? For humans, or for type checkers?

Both, indeed, and it's intentionally somewhat ambiguous. When neither humans nor the type checker needs the information, leave it out. When either kind would be enlightened, add it. Requiring annotations for all variables would be absurd. (If everyone on your team uses an IDE that can show the types, well, then the annotation wouldn't be enlightening.)

I have a class A, whose instances might but needn't have a spam attribute, and we want to annotate that if they have it, it is a float. Currently we don't have a way to do that.

Well, Optional isn't for that. Type checkers in general don't concern themselves greatly with runtime exceptions -- when you access a dict by key, d[k], it's not up to the type checker to tell you that there's no such key. For _local_ variables I think it would be great if a type checker did a liveness check and warned you that you'd risk using an undefined variable -- but for attributes that's not a fair requirement. (Although honestly I think in most cases I much prefer a None check at runtime than a hasattr() check.)

[gvanrossum]

@ilevkivskyi You made the draft consistent about using ClassAttr[], but I still think ClassVar[] sounds better. (As I wrote earlier, many things are class attributes, but we're not interested in those -- we're interested in those class attributes that are conceptually class _variables_ (or maybe constants :-).

[vedgar]

This! That finally made it click in my head that everything becomes much easier if we think about different layers of "what" and "how".

What are we providing? Type hints.

How are we going to do it? By using annotations.

For what are we providing type hints?

(Among other things) for instance variables.

How exactly we do that?

By annotating the corresponding class attributes.

I know it's obvious in your head, but your comment https://github.com/python/typing/issues/258#issuecomment-240612866 has finally made a click in my head about the fine difference between how we think about objects and how it's actually implemented in Python. Maybe such points should be more prominent in the PEP?

[vedgar]

Type checkers in general don't concern themselves greatly with runtime exceptions -- when you access a dict by key, d[k], it's not up to the type checker to tell you that there's no such key.

If I read you right, you're saying that

class A:
    spam: float

_already_ has that semantics I wrote about (of TrulyOptional)? And _not_ "every instance of A has a spam that is a float"? It seems weird, but I think I like it. After all, the only obvious way we can ensure it's initialized, is to provide a default - so without the default (as above), it's TrulyOptional. Nice.

---

Restriction to one dot seems even more arbitrary than restrictions currently in place on @decorators. And it doesn't buy anything: things can still be side-effectfully evaluated (via descriptors). In fact, I agree with https://github.com/python/typing/issues/258#issuecomment-239580051 in that LHS shouldn't be evaluated ever. In fact, as far as evaluation is concerned, a: b is the same as

if False: a = None
else: b

Easy to remember, and does what we want in most cases.

[ilevkivskyi]

@gvanrossum

many things are class attributes, but we're not interested in those -- we're interested in those class attributes that are conceptually class variables

I have no strong preference for ClassAttr, ClassVar has its good points, so that I am fine with changing to ClassVar (I will do this in the next PR, or maybe already in https://github.com/phouse512/peps/pull/2 )

My own preference is to be a little more lenient, and allow anything that's a valid (single) assignment target

Here are some points why I don't like arbitrary (single expression) target:
1) The idea that left hand side is not evaluated could lead to errors that might not be caught by type checkers, and will occur late at runtime

f: Callable[[], Any]
# forgot to assign to f
f().x: int
# 1000 lines later
f().x = 5

2) Unclear use cases. Why one would write d[2]: int = 5? PEP 484 only has homogeneous lists, dictionaries etc., so that there is no sense to annotate a particular item. We have inhomogeneous tuples, but they are immutable.
3) I believe, this will decrease readability. If I see g(d[f(2)]): int, then I need to scroll up to find what is g, d, and what is f.
4) These will be no longer variable annotations, rather value annotations: 1: int or f(1): int.

[orent]

Finally, locals. Here I think we should not store the types -- the value of having the annotations available locally is just not enough to offset the cost of creating and populating the dictionary on each function call.

There is a third alternative to evaluating annotations on each call or not storing them at all: evaluate them at function definition time. Just like argument default values or argument type annotations. This means that the full annotations, including function-local, will be available as soon as a module is imported. No need to wait for the first execution of a function or incur the overhead of doing it every time it is executed.

This could support a form of type checking that lies somewhere between dynamic runtime checking at every call and full offline static analysis. I guess it can be called "early runtime" type checking.

Is anyone interested in such early runtime type checking? Is anyone going to devote the time and effort to write it? I don't know. My point is that type annotations should at least not preclude this possibility. The type annotation feature seems to be pretty agnostic about possible uses of annotation information and their implementations. If it can make this information available without incurring unreasonable overheads I think it should.

The early evaluation of things within the function body might strike some as surprising but I think it is very consistent - things after a ':' are evaluated early, whether in the function header or body. (If your annotations are actually affected by this you are probably doing things you shouldn't be doing anyway ;-)

[gvanrossum]

Keeping track...

• We'll switch to ClassVar
• @vedgar's "now it clicks" comment made me realize that I have no idea how to motivate this to someone who doesn't already have the right idea; HELP!

LHS syntax

There's still serious disagreement about what should be allowed on the LHS and to what extent it should be evaluated; options are:

1. Just a single variable name or a simple NAME '.' NAME; no evaluation
2. the same syntax as (single) assignment LHS; no evaluation
3. the same syntax as (single) assignment LHS; evaluate except for the final __setattr__ or __setitem__ call

In each case the type checker is ultimately responsible for deciding what it likes (it may like self.foo: int but not os.path.join: str :-). Alternatives (2) and (3) allow a type checker complete freedom. Some arguments have been made against these:

The idea that left hand side is not evaluated could lead to errors that might not be caught by type checkers, and will occur late at runtime

Which is why I'd prefer (3).

Unclear use cases

Agreed, beyond self.x: int. One possible use case (bot not seen in the wild as a use of # type comments):

shuttles = {}
...
shuttles['Enterprise']: Optional[SpaceShuttle] = None

This could be supported by some type checkers as an alternative way to spell

shuttles: Dict[str, Optional[SpaceShuttle]] = {}
...
shuttles['Enterprise'] = None

decrease readability

Nobody says you have to use it. Those abominations look just as ugly on the LHS of a regular assignment, but they still have their uses.

All in all I'm still wavering between (1) and (3), with (1) being what's provably useful and (3) giving type checkers all the rope they might need.

When to evaluate annotations

There is a third alternative to evaluating annotations on each call or not storing them at all: evaluate them at function definition time

Believe me, I've thought about this, and it may even have been brought up on python-ideas. I think it's a bad idea, because the placement of the annotation strongly suggests that it's in the same scope as the surrounding code.

I think it is very consistent - things after a ':' are evaluated early, whether in the function header or body

But it's still surprising. Consistency doesn't always make things easy to understand -- and you can't be 100% consistent with everything, so you've really just moved the inconsistency around (in your head you've given a different consistency priority).

All in all I really don't want to do this (early evaluation), but maybe it can be added to the PEP as a rejected idea.

[ilevkivskyi]

@gvanrossum
I have changed ClassAttr to ClassVar, added function definition evaluation of annotations to rejected ideas, removed complex expressions from that list, extended section on ClassVar with examples following your comment above. I just pushed all this changes on top of https://github.com/phouse512/peps/pull/2 (it contained also more minor changes), please take a look.

Between options (1) and (3) I am still more in favor of (1), but with one clarification. This code:

def __init__(self, x):
    sfel.x: int # note the typo

Should rise NameError at runtime. Maybe the decision between (1) and (3) is something that should be discussed by a broader audience of python-dev?

[drhagen]

I have a question about the purpose of ClassVar. A class currently looks like this:

class MyClass:
    spam = []
    def __init__(self):
        self.eggs = []

I would have expected an extension of PEP 484 typing to look like this:

class MyClass:
    spam: List[int] = []
    def __init__(self):
        self.eggs: List[int] = []

A static analyzer could eat this for breakfast. Is the reason for ClassVar solely to allow for (1) the annotations to be available at runtime but (2) not have them assigned every time __init__ runs and (3) not have them extracted statically from the __init__ body? Or are their more advantages?

[gvanrossum]

have a question about the purpose of ClassVar

This seems a repeat question. Can someone else explain it or copy/reference the place(s) where I explained it before?

[srkunze]

This seems a repeat question. Can someone else explain it or copy/reference the place(s) where I explained it before?

---

Why not forget about ClassVar altogether? After all mypy seems to be getting along fine without a way to distinguish between class and instance variables. But a type checker can do useful things with the extra information, for example flag accidental assignments to a class variable via the instance (e.g. self.stats = {}) which creates an instance variable shadowing the class variable. It could also flag instance variables with mutable defaults, a well-known hazard.

[vedgar]

Hm... (1) vs (3) seems to be a very important distinction, but I believe the answer lies in the middle. Look at your example ("This could be supported by some type checkers as an alternative way to spell") again. It's _obvious_ (at least to me) that "Enterprise" _value_ is not important at all. It is its _type_. To infer that shuttles is a Dict[str, Optional[SpaceShuttle]] (if we really want to infer that, of which I'm not quite sure, but I'm not an expert on type hints), all that matters is that 'Enterprise' is a str, and that shuttles[some string, thus any string] is an Optional[SpaceShuttle], which is what we need to infer what we want.

Yes, in this case the issue is blurred by 'Enterprise' being a constant, but imagine if we had

def long_running_function(x: int) -> str: ...
shuttles[long_running_function(5)]: Optional[SpaceShuttle] = None

... I'm pretty sure you'd agree on my points above: first, long_running_function(5) shouldn't be _evaluated_ (until runtime, of course) since its _value_ can't bring anything new into the decision (at least until we go into dependent types, and I _really_ don't want Python to go there;), but it should be _typed_: determined to be a str.

Now it's obvious that NameErrors really are errors of a different perspective. On "sfel.x: int" it's not _Python_ that should complain with "NameError: name 'sfel' is not defined", but it's _mypy_: "cannot infer the type of sfel".

I fully realize this is much harder to do. But I strongly believe this is the _correct_ answer. Python can, as it often has done before, give practicality a bigger baseball bat to beat purity with. :-)

[gvanrossum]

You've got me very confused. What the type checker should do with those things was never much in question, and there's nothing "hard" about it. Mypy already knows what to do with a # type: ... comment on any assignment (hint: in most cases it reports an error :-) and should just do the same for <variable>: <type>. It will certainly report that sfel is undefined! And mypy never "evaluates" anything (it never runs any code; type-checking os.system("sudo rm -rf /") is quite safe :-).

The question I was pondering was what should happen at _runtime_ when there it _only_ a type and no value. There's no precedent for that. When we choose (1), there are only two cases:

• name: int -- just evaluate the type (in this example, int); this will catch name: nit
• obj.attr: int -- evaluate obj (which must be a simple name) and the type; this will catch misspellings in obj and in the type

When we choose (3) there are three cases (here EXPR and KEY are truly arbitrary expressions):

• name: int -- same as above
• EXPR.attr: int -- evaluate EXPR and the type
• EXPR[KEY]: int -- evaluate EXPR, KEY and the type

When we choose (2) we would presumably only evaluate the type, which I find unpleasant (even though I know that mypy would catch errors in EXPR and KEY). The reason we evaluate the type in the name: int case is because we want to store it in __attributes__. In the other cases we can't store it in __attributes__ but for consistency I think the type should be evaluated regardless (so we consistently find the same errors in all types regardless of the LHS).

All this applies in the case where the scope is module-global or in a class body. In a function body, the type shouldn't be evaluated (because it would slow us down -- we've already established that!). Now if we were to pick (3), should EXPR and KEY be evaluated in a local scope? Regardless of the answer there's an inconsistency: if we say yes, the inconsistency is that at runtime a misspelled EXPR or KEY is found but a misspelled type goes unnoticed; if we choose no, the inconsistency is between the case where there's no value assigned to the LHS and the case where there _is_ a value (since if there's a value we definitely have to evaluate EXPR and KEY in order to carry out the assignment!).

I'd like to choose yes there, so that there is consistency between these two:

• EXPR.attr: int
• EXPR.attr: int = 0

in the sense that in either case EXPR is evaluated but the type isn't. Ditto for indexed assignments. But I could also be sensitive to the argument that, in local scopes, as long as we're not evaluating the type for performance reasons, we might as well not evaluate EXPR and KEY either. (However the same reasoning doesn't apply to class and global scopes, and there I more strongly prefer to evaluate them so that we get a runtime check that they make sense.)

[vedgar]

Ok, sorry for my part of the confusion. First, yeah, I was talking of (2) vs (3), not (1) vs (3). And of course, we're talking only of non-function-scope (but what about class nested inside def? Oh, that's too hard for me to think about, sorry). And in that case, the type _should_ be evaluated. Now on to your part of the confusion:

obj.attr: int -- evaluate obj (which must be a simple name) and the type; this will catch misspellings in obj

OMG, no! This is not the Python I know, love and want. Catching misspellings is a horrible misfeature (see Perl and Wolfram language), that way surely lies madness. Python has so far managed quite well to escape that particular path paved with good intentions. :-/ Look:

slef.x: int
...
if horribly_complicated_condition():
    slef = factory_of_objects_having_x_attribute()

Are you _sure_ it is a misspelling?

I stand firmly by "EXPR.attr: int" v. "EXPR.attr: int = 0" being a false analogy. Second is an assignment, first is not. After all, it's much like "EXPR.attr" v. "EXPR.attr = 0" (which is also a false analogy). And you surely wouldn't (would you? OMG, I'm not so sure:-/) want to evaluate the "simple_name" in "simple_name: int", in order to catch misspellings?

---

On rereading, I see that maybe I wasn't clear enough. I am convinced Python must allow me to annotate the attributes of not-yet-existing objects. And Python doesn't (and shouldn't) know at the time it's executing the annotation, what the name will refer to. By Heavens, someone can insert it from another file, via sys.modules[target].attr = whatever. (Ironically, the only place where Python _can_ know that, function-scope, is out of discussion.:) Besides, I suppose many people will write their annotations at the top of block, and there'll be a lot of questions "why do I have to initialize a to say a.b will be of type int?".

[gvanrossum]

Catching misspellings is a horrible misfeature (see Perl and Wolfram language), that way surely lies madness [...]

Stop being so dramatic (or are you being sarcastic? I can't tell).

I meant nothing more than evaluating the expression to the left of .attr and requiring that it is valid at that exact point in the code.

Recasting your forward-ref example:

foo.bar: int
if <cond>:
    foo = <expr>

I don't like this at all. In my head I treat something like

foo.bar: int

as "we've got foo, now we're talking about its bar attribute, its type is int". I want this to be valid if and only if assigning a value to foo.bar at that very point is valid. The fact that foo might not allow attribute assignment at all makes me slightly nervous, but that can't be helped. But if I have forgotten to initialize foo at this point, that _must_ be an error right there, when the program runs.

Python is a dynamic language, and statements are executed (or not) in a specific order. Type annotations without value assignments can talk about a simple name or an attribute that's not yet defined, but if an attribute, the thing it is an attribute _of_ must exist when the execution reaches the declaration.

This is not negotiable. Type annotations are not comments with a funny syntax.

If this really can't be resolved I propose to fall back on simple names _only_ as the LHS, in which case we deprecate annotating self attributes in __init__ (it could only be done using a type comment). I would be fine with that, you can declare your instance variables in the class body.

[JukkaL]

I think that supporting things like self.x: List[int] = [] is an important feature, and I'd be sad to see it go. At least it makes it easier to annotate existing code, even if for new code I might prefer to annotate attributes in the class body.

Also, I'd rather have type annotations cover all common use cases where we've previously used type comments. Having both annotations _and_ type comments still widely used would feel like too much of a compromise.

Supporting bare self.x: List[int] seems less important to me, since you can always move the annotation to the class body without extra typing (actually, there will be less typing in the class body, as you don't need self.). If we can't reach an agreement on the semantics of this case, i.e. whether self should be evaluated or not at runtime, here's my alternative proposal:

• If the LHS is not a simple name, only accept an annotation if there is an initializer (i.e., foo.bar: int = expr). Here we'll have to evaluate the LHS anyway to evaluate the assignment, so the semantics shouldn't be as controversial.
• Declaring the type of an attribute without initializing with a value is only supported in a class body, using attr: type.

[vedgar]

Ok, I'll try to be less dramatic. It's just that until now, I viewed annotations without assignments ("bare annotations") in the same vein as "nonlocal" or "global" statements. Yes, they appear at some concrete point in the code, but in fact they are just annotations. It seems that you view them as executable statements, something like "class" or "def" statements. But without assignment (without even __annotations__[...] assignment), it's really hard to imagine what are you executing them _for_.

I always imagined that when Python sees somedict[somefunction(3)] = 8, executing "somefunction(3)" is just a special case of "arguments are evaluated before function is called", somefunction(3) being the first argument of somedict.__setitem__. Yes, there is a special case of "expression statements" being evaluated for their side effects, but that case in this context makes me even more nervous. You really want

somedict: Dict[Optional[int], int] = {}
somedict[print('Boo!')]: int

this second line to print "Boo!"? I'm not saying it's wrong (it's right almost by definition:), but it's weird. It doesn't fit into my mental model of

if False: somedict[print('Boo!')]
else: int

so my mental model is wrong. Can you give me a better one? It seems you view bare annotations as "almost assignments": you do everything _as if_ you're going to assign to the LHS, but get cold feet at the last moment. It's not so bad, it's just that I viewed all of that from completely different angle. I thought conflating annotation with assignment is just a happy syntax synergy, not a deep conceptual thing. In other words, I thought a: b = c to be just a shortcut for a: b; a = c (or even a = c; a: b, which I perceived to be the same), but it's obvious now that to you, these are different (at least when a is not a simple name).

But in that case, I _really_ think that if you want to evaluate a in a.b, you have to evaluate it (and k) in a[k], and everything else short of actual assignment at the end ("cold feet semantics"). Otherwise Zen rule #17 says it is a bad idea. After all, for most purposes, a.b is just vars(a)['b']. :-)

---

Oh, wait a minute. If bare annotations are executable statements that are executed in order of appearing in the code, does that mean that re-annotating is fair game (as far as Python is concerned)?

x: int
x = 3
x: str
x = 'what?!'

And does it also mean that __annotations__ are updated in real time? (So, after a class is defined, its __annotations__ is just "the last snapshot", rather than a static image.) Again, not that it's bad, but it's _completely_ different from what I had in my head until today. Makes me wonder how many other things I understand completely wrong... :-|

[ilevkivskyi]

I like the proposal by @JukkaL (allow EXPR: EXPR = EXPR but only NAME: EXPR without right hand side value):

1) This is kind of a reasonable compromise
2) It is a good idea to put all "valueless" annotations like x: int in a class body, not as self.x: int somewhere in __init__. I was thinking about proposing to add a PEP 8 recommendation for this. With Jukka's suggestion, we don't need any recommendations, such syntax will suggest a more readable coding style.

[gvanrossum]

I really think that if you want to evaluate a in a.b, you have to evaluate k in a[k], and everything else short of actual assignment at the end ("cold feet semantics").

Yes, that's exactly what I was proposing.

The reason I was considering this at all was purely to leave a way open for future type checkers to give these a meaning, without having to go through the pain of changing the syntax again. (IOW if by the time Python 3.8 rolls around, mypy finds a use for a[k]: int, code using that syntax could be backward compatible with 3.6, and it can be introduced merely by rolling out a new version of mypy, without any changes to Python itself.)

I am fine with Jukka's compromise, if nobody thinks that's going to happen.

To summarize:

• NAME: TYPE
  
  
  
  • valid in local, class and global scope
  
  
  • in local scope, TYPE is not evaluated; NAME becomes a local variable
  
  
  • but disallowed if NAME is subject to global or nonlocal in the same scope
  
  
  • in class and global scope, TYPE is evaluated and __annotations__[NAME] = TYPE
  
  
• NAME: TYPE = VALUE
  
  
  
  • ditto, plus assigns NAME = VALUE
  
  
• EXPR: TYPE
  
  
  
  • never valid if EXPR is not a simple NAME (even (foo): int is invalid)
  
  
  • if valid, see the first top-level bullet
  
  
• EXPR: TYPE = VALUE
  
  
  
  • EXPR must be a valid assignment target syntactically (i.e. EXPR2.ATTR or EXPR2[KEY])
  
  
  • valid in local, class and global scope
  
  
  • in local scope, TYPE is not evaluated; assigns EXPR = VALUE
  
  
  • in class and global scope, TYPE is evaluated; assigns EXPR = VALUE
  
  
  • no assignment to __annotations__ unless EXPR is a simple NAME (then the second top-level bullet applies)
  
  

[ilevkivskyi]

@gvanrossum I completely agree with your summary.

Side note: from the point of view of implementation, it will be easy to allow this later if we go now with Jukka's idea (Grammar/Grammar will not change, only few lines in ast.c and compile.c will change), the main pain will be from writing and discussing another PEP, but that PEP could be quite short and straightforward.

[gvanrossum]

OK, it's decided. FWIW my concern is neither implementation nor PEP
process, it's the pain of not supporting older Python versions. But it's
unlikely to ever happen.

--Guido (mobile)

[vedgar]

If we're going to break backward compatibility, I have another idea/question.

IIRC, you said at some moment that titlecased names of builtin container types in typing.py are just a transient measure, to make it easier to play with, without changing too much of Python to enable list[int] or dict[str, int]. I think that if you haven't changed your mind and at some moment we are going to support such a syntax (with _true_ container types), now is the perfect time to do it.

(And I think we _should_ do it. Otherwise we have UserList and UserDict phenomenon all over again, with """hey, this here should be just a "list", but you can't really use "list" here because of Python limitations, so import List from typing instead""". It _could_ be that you really want those types to live in a separate namespace, but then we should have Int, Bool, Float and so on too.)

[gvanrossum]

I have another idea/question.

Please start another issue if you really want to discuss that.

[laranzu]

Saw the announcement on the python-dev list. Very late to the discussion, but I have done some crash reading of the PEP, previous PEPs, and rejected ideas so I hope this won't be too dumb.

Could a future version allow functions in place of types? These functions could take a value and return True if it is acceptable. This would allow duck typing, something like

def sheepish(obj):
    return hasattr(obj, "wool") and hasattr(obj, "baa")

def shear(subject: sheepish):
...

Functions could also provide more flexible and readable construction of complex types. Maybe something like

def audit(flock: MappableType(key=int, item=sheepish):
...

[refi64]

@laranzu Only problem would be type checking; the type checker would need to run the function in order to test if the type is valid. That's problematic since the types themselves only exist at runtime.

[gvanrossum]

@laranzu https://github.com/laranzu That idea is unrelated to PEP 526 --
_if_ we were to do that it'd be part of the general type "syntax" defined
in PEP 484 (or an extension thereof). I don't think what you propose is in
the cards, but if you want to preserve it for posterity, the place to do it
would be this tracker: https://github.com/python/typing/issues.

[NeilGirdhar]

I have been holding off commenting until there was a PEP. I like this proposal, I'd like to ask if static types are the only kind of annotation being considered and whether static type checks the only kinds of invariants considered?

One of the things about Python is known for is its flexibility. Is there any reason to set our sights only on matching the functionality of compiled languages that came before it?

Specifically, the actual types themselves are one kind of invariant — it happens to be the invariant that compiled languages check — but if you're writing your own checker, then it's possible to have a much more general set of checks that can be done statically. For a simple example, a static checker could verify that a variable is a nonnegative integer. More generally, the static checker is going to have some information, which in this proposal is a type, but could theoretically check things much more general than issubclass.

Another example is that I often add a list of variable names to my classes to indicate variables that should be pickled, and variables that should be passed as kwargs to the constructor in __deepcopy__. A base class automatically does both of these things for me based on these annotations.

My suggestion is to generalize the type annotations in this way. Add an annotations keyword, which functions as a block. Then have a syntax like:

class SomeClass:
    annotations:
        y: Tuple[int, int], DocString("The x and y coordinates of…") 
        x: bool, IsPickled(True)
        z: List[int], IsClassVariable(True)
        w: int, HasSupport(min=0)

This would still produce your __annotations__ class variable, but has the advantage that the annotations are, for each member (or class) variable, a list of arbitrary annotations. The type-checker would look at the type annotations and the IsClassVariable instances. Other checkers could look at other annotations.

This works everywhere the current syntax works except function definitions. So, I propose keeping the proposed syntax, but adding the general syntax for more complicated annotations.

Annotations that might be useful:

• types (of course)
• alignment and positioning (numpy does this with fields),
• member variable documentation (would be really nice!)
• whether a member variable should be pickled
• whether a member variable should be passed to the constructor in calls to copy
• a class default value of a member variable
• restrictions on the values of a member variable (e.g., greater than zero)

Functions could also benefit from general annotations to indicate overrides.

[refi64]

@NeilGirdhar Doesn't that work with the current syntax? I don't really see why a new keyword would need to be added...

[NeilGirdhar]

@kirbyfan64 How? The entire annotation is interpreted as a type whereas my proposal is to have lists of annotations whose elements are picked up by different checkers.

[gvanrossum]

@NeilGirdhar: Let's make that a separate PEP and aim for Python 3.7. The use case is more imaginative but there are no current tools that are begging for such a feature, and there are plenty of other ways to do it (e.g. just use a class).

Note that PEP 526 doesn't say anything about the syntax or semantics of a type -- it defers to PEP 484 for that, it just defines more ways of attaching types to names that don't happen to be function arguments. So any proposal that's about stretching the syntax for types should probably go to the python/typing tracker.

Also note that in a sense the _primary_ goal for PEP 526 is to ensure that types for variables and attributes end up in the parse tree rather than in comments, so they are easier to process for tools that only look at the ast.

[prechelt]

(This is more a PEP 484 topic than a 526 topic, so I made this a new thread.)

Iaranzu wrote:

Could a future version allow functions in place of types? These functions could take a value and return True if it is acceptable. This would allow duck typing, something like

def sheepish(obj):
return hasattr(obj, "wool") and hasattr(obj, "baa")

def shear(subject: sheepish):
...

Just a quick note:

Package typecheck-decorator has this functionality.

Those are purely dynamic checks and the 'typing' module support is incomplete.

But indeed this manner of typing is useful or convenient at times and very flexible.

Lutz

[NeilGirdhar]

What is the purpose of the initializer ("default value") of a member variable on a class? Where does that information get stored and how is it used?

[gvanrossum]

@prechelt: I don't know what you did, but you did not create a new thread and your comment in this thread looks ill-formatted.

@NeilGirdhar: That initializer gets stored as a class variable, just as it would be if you left out the ": TYPE" part. Due to the magic of instance attribute lookup this works, as long as you don't use a mutable default (and a type checker ought to check for mutable defaults anyway -- if not a type checker then some other linter).

[ilevkivskyi]

@NeilGirdhar I already have seen proposals similar to yours before, so that I could conclude it is popular (it was also discussed at time of PEP 484). Neither PEP 484 nor PEP 526 do not prohibit usage like this:

class C:
    lst: [List[int], some_machine_readable_metadata, 'some docstring']
    attr: [str, something_else_valid_at_runtime_here]

This is syntactically valid and all metadata will be stored at runtime in __annotations__. These PEPs do not provide any standards for such use, but there are some tools that are experimenting (one is mentioned above). I agree with Guido that this should be a separate PEP, because this proposal is more about semantics of annotations, not the syntax.

EDIT: Fixed a typo.

[Michael0x2a]

I'd like to propose an amendment to the Annotating expressions section.

Currently, the PEP states that annotations on expressions such as c.y must always be accompanied by a default value. So, doing c.x: int = 0 is allowed, but c.x: int is not. This is in contrast to annotations on variables and attributes, where the default value may be omitted. So, doing x: int is legal, for example.

I think it would be nice to add special case that allows you to omit the default value specifically for expressions of the form self.my_field: int, and perhaps only within an __init__ method. The net effect is that this would allow you to declare instance attributes within the __init__ method as opposed to within the class scope.

As an example, I think classes defined like this:

class BasicStarship:
    stats: ClassVar[Dict[str, int]] = {}

    def __init__(self, captain: Optional[str] = None) -> None:
        self.messages: List[Message] = []
        self.captain: str = 'Picard' if captain is None else captain
        self.damage: int  # allow a missing default!
        if ...:
            # Complex code here to initialize self.damage

...ought to be treated the same way as classes defined like this:

class BasicStarship:
    messages: List[Message] = []
    captain: str = 'Picard'
    damage: int                           # instance variable without default
    stats: ClassVar[Dict[str, int]] = {}

    def __init__(self, captain: Optional[str] = None) -> None:
        if captain is not None:
            self.captain = captain
        if ...:
            # Complex code here to initialize self.damage

I think an open question is whether or not annotations of the form self.x: type should be added to the class's __annotations__ dict or to the __init__ function's dict. In order to make the two examples above truly identical, the annotations would needed to be added to the class's __annotations__ dict, but then that would require more special casing.

(Actually, the special-casing you'd have to do in general is probably the main disadvantage of this proposal.)

---

The main reason why I'm bringing up this proposal is because I don't think the practice of declaring your instance attributes in the class scope or using class attributes as a way to define default values for your instance attributes is really a universal idiom within the Python community.

In particular, allowing self.value to be missing a default value would be helpful for Python users who prefer initializing all their fields within __init__, and for users who feel that mixing together class attributes and instance attributes in any way is a stylistic error.

Of course, neither example is actually mixing together class and instance attributes -- it just feels that way because everything in the second example is defined at the class scope level, and because adding a type annotation to a class attribute is enough to promote it to an instance attribute (which seems unexpected -- adding a type can change the scope?)

While I don't think an experienced Python dev or anybody who's read both PEP 484 and 526 would be confused by either of these issues, I can anticipate that somebody new to both Python and type annotations might be, especially if your codebase never ends up using the ClassVar type or provide other similar hints that there's something amiss.

Allowing the first example would allow typed code more closely resemble what untyped Python code looks like, and wouldn't require a user to subscribe to the practice of defining instance attributes at the class level.

[ilevkivskyi]

@Michael0x2a
This idea has been considered, but it was decided not do do this (at least for now). There are following reasons:

• The runtime rules for what is getting evaluated and what is not are already a bit complex, therefore adding even more exceptions is not good.
• It is problematic to add annotations found in __init__ to the class __annotations__ (at least, you need to make self a special name, like this in other languages).
• It is good (more readable) to have attribute annotations in one place (at the top of class body), at least in situations with complex initialization. If one needs an uninitialized self.x , then it means that one is doing something complex (not just self.x = x) and it is better to put type in the class body.

I would propose to add this idea to the list of rejected/postponed ideas.

[gvanrossum]

@Michael0x2a Thanks for reopening this one! This came out of "@JukkaL's compromise" above, https://github.com/python/typing/issues/258#issuecomment-242934936, which in turn came about from quite vocal pushback by @vedgar earlier (starting around https://github.com/python/typing/issues/258#issuecomment-242181631).

The crux of the issue is this: while it seems relatively uncontroversial to allow

self.x: int

the question is what other things to allow to the left of the dot -- can it be an arbitrary expression, e.g.

foo(0).bar[1].x: int

and if so, do we generate code to evaluate the foo(0).bar[1] part in that case?

Or should we only allow self to the left of the .? But what if it's a class method, or someone uses a non-standard name instead of self?

Perhaps we should only allow a single identifier to the left of the .? But still, do we generate a LOOKUP opcode for it (so it will fail if the name is undefined) or not?

These are the kinds of questions from which the restriction to a single name (if there's no initializer) was born. I don't really like any of the intermediary positions -- I think it should either be just a name, or the same syntax we support if there is an initializer. Also I think everything to the left of the last . should be evaluated (unless the compiler can prove that it's just a name lookup of a variable that definitely exists, like self if there are no del self statements in the function body). But @vedgar really didn't like that.

[gvanrossum]

@ilevkivskyi: I think if we don't do this now we'll never do it. I think the argument about __annotations__ is weak. The argument about the rules getting too complex is stronger IMO (it's why I want to go to either extreme but don't like intermediate positions).

@Michael0x2a: To counterbalance

the practice of declaring your instance attributes in the class scope or using class attributes as a way to define default values for your instance attributes is really a universal idiom within the Python community

I observe that in most _other_ languages instance variables _must_ be declared in the class body. (Though at least C++ forces you to put the initializers in the constructor, using horrible custom syntax.)

[vedgar]

But @vedgar really didn't like that.

I think I have recanted (repented?:) in the meantime (see https://github.com/python/typing/issues/258#issuecomment-242937143). If bare annotations are really executable statements, belonging to control flow and being executed in their place in the code, then I'm quite ok with them executing LHS according to the "cold feet semantics".

And while I still can't quite grasp such a revelation (I still think re-annotations and conditional annotations are really WTF material, and would still like to think of bare annotations as static, like global or nonlocal statements), it is quite clear from

Python is a dynamic language, and statements are executed (or not) in a specific order.

that you think of them that way.

[gvanrossum]

Well, then I'm ready for a reversal on this topic. It would make things simpler: in addition to type-less assignments (which don't change), instead of two separate forms

TARGET: TYPE = VALUE
NAME: TYPE

there is only

TARGET : TYPE [= VALUE]

(where TARGET is any syntactically valid single assignment target) and it's up to the type checker to decide which things it really likes. When there's no VALUE, the runtime uses the rules I previously proposed for this purpose:

• TYPE is evaluated or not, using the same rules as when there's a VALUE (i.e. not in the local scope),
• __attributes__ is updated using the same rules (i.e. not in the local scope and only if TARGET has the form of a single unparenthesized NAME), and
• TARGET is evaluated as if preparing for an assignment (just omitting the __setitem__ or __setattr__ call).

[laranzu]

I suggest that the syntax for assignments with type annotations be restricted to
TARGET : TYPE-NAME = VALUE
with no arbitrary type expressions. Or putting it another way, apply the one name only rule to the RHS of the colon as well as the LHS.

Type annotations are documentation, not code. But unlike docstrings and comments, they are intermingled with the code rather than being clearly delimited. If I write something like
foo : Dict[int, str] = bar
then to understand what the statement actually does it has to be read as
foo : _blah blah blah_ = bar

The less stuff you have to skip while reading, the better. I'd think this would be particularly valuable for new Python programmers.

Arguments and return values are not similarly limited? I'd be in favour of applying the same restriction to those as well, for the same reason of simple and understandable code. Even if it breaks backward compatibility for people currently using non-type expressions, because to me it seems clear that annotations are going to be used for nothing other than type checking going forward.

[Michael0x2a]

then to understand what the statement actually does it has to be read as
foo : _blah blah blah_ = bar

@laranzu -- I'm not convinced that's actually the case. In fact, I'd actually argue the opposite -- if a beginner wants to understand what that line of code is really doing, the type will probably be as valuable, if not more so, then the name of the variable/what's being assigned to it.

For example, suppose we had a variable foo = {} and followed that up with 20+ lines of (potentially very complex) code where you were mutating and adding things to foo in various ways, and using various advanced features of Python. In this case, a new programmer might be completely lost as to what foo is actually being used for, especially since the dict is empty and it's non-obvious what'll be eventually be going into it (for both the human reader and for type inference engines!)

In contrast, if you did:

foo : Dict[int, Dict[str, str]] = {}
# 20+ lines of code

...or perhaps:

UserId = int
Properties = Dict[str, str]
foo : Dict[UserId, Properties] = {}
# 20+ lines of code follows

...then that tells you exactly what the dict is used for, what types are permitted, and so forth without forcing the new beginner to have to first understand the rest of the code.

That is, the types have encoded the context and inherent assumptions about that variable, but unlike documentation/comments, have done so in machine-checkable form (and so are highly unlikely to go out of date, unless the developer has abandoned the notion of testing and type-checking their code altogether).

In any case, I suspect that your proposal will be a non-starter for other reasons, since it'll end up introducing unnecessary code bloat. For example, take the itertools.groupby. You'd have to change the type signature from:

_T = TypeVar('_T')

# ...snip...

@overload
def groupby(iterable: Iterable[_T]) -> Iterator[Tuple[_T, Iterator[_T]]]: ...
@overload
def groupby(iterable: Iterable[_T], 
            key: Callable[[_T], _S]) -> Iterator[Tuple[_S, Iterator[_T]]]: ...

...to:

_T = TypeVar('_T')

# ...snip...

_Iter = Iterable[_T]
_GroupbyCallback = Callable[[_T], _S]
_GroupbyOut = Iterator[Tuple[_T, Iterator[_T]]]

@overload
def groupby(iterable: _Iter) -> _GroupbyOut: ...
@overload
def groupby(iterable: _Iter, key: _GroupbyCallback) -> _GroupbyOut: ...

...which just seems excessive if taken to the extreme. (An additional downside is that info about what the function does is now no longer localized to the type signature and could require some scrolling around, which defeats the readability improvements type signatures can add to a codebase).

[ilevkivskyi]

@gvanrossum
I am still not sure. There are good arguments for both solutions. I am 60/40 towards the Jukka's solution. However, I would agree with your solution with one addition: It should be explicitly recommended to put annotations at the top of class body (of course we should not force this). I am a bit afraid that people might abuse annotations in __init__ sacrificing readability (and btw execution speed, since for f()[g()]): int functions f and g will be evaluated on every method call).

[Michael0x2a]

@ilevkivskyi -- well, it seems Guido ended up pre-empting me while I was trying to think through an actionable counter-proposal addressing your points + Guido's points.

That said, I do want to put it on the record that I agree with your first two bullet points, but disagree with the third. In particular, I don't think there's any particular difference in readability between doing this:

class BasicStarship:
    messages: List[Message] = []
    captain: str = 'Picard'
    damage: int                           # instance variable without default
    stats: ClassVar[Dict[str, int]] = {}

    def __init__(self, captain: Optional[str] = None) -> None:
        # ...snip...

...vs doing this:

class BasicStarship:
    stats: ClassVar[Dict[str, int]] = {}

    def __init__(self, captain: Optional[str] = None) -> None:
        self.messages: List[Message] = []        
        self.captain: str = 'Picard'
        self.damage: int                           # instance variable without default

        # ...snip...

In both examples, the field declarations are located near the top of the class, separated away from the initialization logic.

The main benefit the second form brings is that if a codebase consistently uses the second form, it wouldn't require somebody new to Python or to type annotations to have to acquire a new special-case to their mental model of how scoping works.

In particular, I can anticipate the first form being particularly confusing (especially if the reader isn't familiar with Java-style or C++-style class definitions) because the rule we're basically adding is "if you add a type annotation to an existing variable, that variable can sometimes change scope, but only when you're typechecking code". It seems to me that a beginner would find this a bit mysterious and potentially inconsistent.

So it seems that the tradeoff is whether we add make people learn a special case for how scoping works and introduce a distinction between "runtime scope" and "typecheck-time scope", or if we people learn a special case for how expressions of the form EXPR : TYPE works (if we end up going with the originally-announced proposal).

I guess it's not immediately clear to me which is better -- I'm personally leaning towards the latter, but as both you and @gvanrossum pointed out, that can be problematic perhaps unless we fully embrace the TARGET : TYPE [= NAME] syntax. (But even then, the new scoping rule wouldn't go away -- it would just be easier to hide)

[gvanrossum]

Nix on the proposal by @laranzu to only allow type names. That's up to a style guideline. Personally I find List[int] more readable than ListInt -- I'd still have to look up the definition of the latter to make sure it is what I think it is.

[gvanrossum]

@ilevkivskyi, @Michael0x2a: So the objections are threefold:

• More complex implementation
• No update to __attributes__
• Style preference for declarations in the class body

My responses:

• I think the implementation can be done and in this case should not be an reason to reject it.
• Update __attributes__ when TARGET is exactly a NAME and not in local scope.
• Style preferences differ. Using # type comments I've seen lots of code one way and lots of code the other way. Both feel acceptable. Maybe declarations in __init__ work better if the class already has lots of other stuff in the class body (e.g. lists of class-scoped constant definitions that wouldn't benefit from type annotations).

[ilevkivskyi]

@gvanrossum
A small clarification: When I was talking about difficult implementation, I meant the original proposal by @Michael0x2a (about special-casing self). It would be very easy to implement your latest proposal https://github.com/python/typing/issues/258#issuecomment-243966289

After some thinking about styles I agree that different styles have their pros (although I prefer the class body). In any case, we will need to develop some kind of style guide (as it is mentioned in the PEP now), but this should not slow down the PEP itself, we can fine tune the style guide later.

If you agree, I will go ahead and implement your proposal (if I will have time this afternoon I will also make a PR against master typing/peps repo).

A fun note: since annotations are stored only for simple names (without parens), there is an interesting way to hide annotations at runtime (x): int = 5 performs the actual assignment and puts annotation in AST, but throws it away at runtime.

[NeilGirdhar]

Are annotations done in the constructor available on the class? The way I understood it __annotations__ is available at runtime, so when you annotate a variable off of self, how is that assigned to the class's __annotations__? What if multiple objects annotate things differently through different branches within the constructor?

[ilevkivskyi]

@NeilGirdhar Currently and in Guido's proposal mentioned above only simple names get their annotations stored, annotations for self.x are ignored at runtime, they are only present in AST.

[gvanrossum]

Annotations done via self.NAME: TYPE [= VALUE] are _not_ stored in
__annotations__. The PEP is already clear about that for the case where
there is a VALUE.

[refi64]

So the PEP is getting updated so X: Y is allowed where X doesn't have to be a name, right? These mega-thread discussions can get kind of confusing...

[gvanrossum]

@kirbyfan64 Yes that's right. And sorry, I agree that mega-threads stink no matter which medium you use. :-(

[gvanrossum]

So who wants to write the PEP update?

[ilevkivskyi]

@gvanrossum @kirbyfan64

So who wants to write the PEP update?

I could do this in two-three hours, if someone is available sooner, please go ahead.

[refi64]

I'll try to do it now a sec.

[refi64]

Done. Note that I'm assuming (x): int is now valid...right?

[ilevkivskyi]

@kirbyfan64 Yes, it is valid, but it will not store an annotation in __annotations__ since it is not a simple name.

[ilevkivskyi]

@kirbyfan64 Your commit looks good. I have left three small comments on the commit.

[gvanrossum]

@ilevkivskyi Yes please go ahead and update the implementation. We're in
the final stretch!

[vedgar]

Can my idea (of treating bare annotations the same way as global or nonlocal declarations) be added to the PEP as rejected? The closest thing it currently says is

the placement of the annotation strongly suggests that it's in the same scope as the surrounding code.

but the same thing can be said for global and nonlocal, and that didn't stop you then. I'm sure you have valid reasons of how exactly is this fundamentally different, but I'd like to see them in the PEP.

At some moment Guido said

Python is a dynamic language, and statements are executed (or not) in a specific order.
This is not negotiable. Type annotations are not comments with a funny syntax.

and while this is a strong stance, I don't feel it's adequately explained - again, global and nonlocal statements are also "not comments with a funny syntax", but they _are_ evaluated statically, no matter _where_ they are put in the scope.

---

I'm very worried about what semantics should I assign to re-annotations and conditional annotations.

for tp in int, float, complex:
    x: Optional[tp] = None

if input():
    y: bool
else:
    y: int
y = True

z: str
z = 'Hi'
z: Sequence[str]

t: int = 8
t: str = 'What?'

In each of these cases, I'm extremely confused about what actually happens. And reading the PEP doesn't help very much. Are these illegal? What tool detects them (if any)? If the behavior is specified, where is the specification?

(Note that all (or most) of these would be trivially flagged as errors if annotations were treated statically.)

[ilevkivskyi]

@vedgar

I'm very worried about what semantics should I assign to re-annotations and conditional annotations.

Please don't worry. The title of the PEP contains "Syntax", not "Semantics" because we don't want to impose any new type semantics (apart from addition of ClassVar) beyond PEP 484. Moreover, PEP 484 is something like PEP 333, i.e. just a convention for tools that deal with type metadata. Your examples could be treated differently by different type checkers depending on what they see as safe/unsafe (or even a single type checker can have different modes like --sloppy-mode or --paranoid-mode).

[gvanrossum]

@vedgar I've added text to the PEP to explain (hopefully) your rejected proposal:

Treating bare annotations the same as global or nonlocal:
The rejected proposal would prefer that the presence of an
annotation without assignment in a function body should not involve
_any_ evaluation. In contrast, the PEP implies that if the target
is more complex than a single name, its "left-hand part" should be
evaluated at the point where it occurs in the function body, just to
enforce that it is defined. For example, in this example::

    def foo(self):
        slef.name: str

the name slef should be evaluated, just so that if it is not
defined (as is likely in this example :-), the error will be caught
at runtime. This is more in line with what happens when there _is_
an initial value, and thus is expected to lead to fewer surprises.
(Also note that if the target was self.name (this time correctly
spelled :-), an optimizing compiler has no obligation to evaluate
self as long as it can prove that it will definitely be
defined.)

I have nothing to add to what @ilevkivskyi said about the semantics of redefinition -- that is to be worked out between type checkers. (Much like PEP 484 doesn't specify how type checkers should behave -- while it gives examples of suggested behavior, those examples are not normative, and there are huge gray areas where the PEP doesn't give any guidance. It's the same for PEP 526.)

[vedgar]

Reading your explanation, it just occured to me that you _could_ detect misnamings and treat bare annotations statically. At least, Python already does this when nonlocal is used (not for global, of course, since the name can be inserted dynamically afterwards).

def f():
    def g():
        nonlocal x
        return x

If I end the input here with a blank line, I get "SyntaxError: no binding for nonlocal 'x' found". (But I _can_ introduce x later, and get no error.) Python obviously doesn't _evaluate_ x, but it somehow knows, statically, whether there is a variable it could refer to, even if it's assigned to later. Similar thing can be done here, I presume. "no binding for 'slef' found" could be reported without evaluating slef. But yeah, it gets hairy with more complicated names.

Ok, I'll stop here. I'm happy with the current solution. At least I think so. :-)

[SylvainCorlay]

@fperez @carreau I am 👍 on introspect-ability.

• Jupyter and Sage use introspection to do some simple automatic GUI generation to interact with a function with sliders / dropdowns / toggle buttons etc. At the moment, default values of function parameters are used to determine the type of widget that should be used for each parameter.
  It would be very nice to specify the widgets to use based on the type annotations.
• Encoding more information (like bounds) in an annotation would be very useful. I don't know what would be the preferred / recommended way to pass richer annotations.
• I can't help thinking of how this would play with traitlets and PEP 487. Would it be possible to have a metaclass which uses the type annotations of a class to create traitlets-like features?

[gvanrossum]

@SylvainCorlay @fperez @Carreau
Yes, there are 1000s of new uses for annotations in classes! You can indeed write a metaclass or a class decorator that uses the annotations as a better way to define traits. Let your imagination go wild!

Mypy already doesn't understand what's going on with libraries that use traitlets so I'm not particularly worried about how mypy should deal with this -- we'll cross that bridge when we get to it.

Personally I hope to be able to define named tuples like this:

class User(NamedTuple):
    username: str
    userid: int
    first_name: Optional[str]
    last_name: Optional[str]

[SylvainCorlay]

@SylvainCorlay @fperez @Carreau
Yes, there are 1000s of new uses for annotations in classes! You can indeed write a metaclass or a > class decorator that uses the annotations as a better way to define traits. Let your imagination go wild!

Mypy already doesn't understand what's going on with libraries that use traitlets so I'm not particularly worried about how mypy should deal with this -- we'll cross that bridge when we get to it.

Personally I hope to be able to define named tuples like this:

class User(NamedTuple):
   username: str
   userid: int
   first_name: Optional[str]
   last_name: Optional[str]

^ @ellisonbg @minrk.

[ilevkivskyi]

@gvanrossum

Personally I hope to be able to define named tuples like this:

class User(NamedTuple):
    username: str
    userid: int
    first_name: Optional[str]
    last_name: Optional[str]

This looks cool! I already want to implement this. Maybe this can even go into typing or future_typing instead of the current implementation of typed NamedTuple?

[NeilGirdhar]

@ilevkivskyi Can't you just add an __init_subclass__ method to NamedTuple that walks __annotations__?

[ilevkivskyi]

@NeilGirdhar As I understand __init_subclass__ is like __init__, not like __new__, so that if you want your subclass to be a real collections.namedtuple, then probably a bit of metaclass magic is needed.

[gvanrossum]

Feel free to get started on future_typing with this as the first feature!

[vedgar]

Yes, there are 1000s of new uses for annotations in classes. ... Let your imagination go wild!

Since you're in the mood of allowing expressivity expansion of the language, I have allowed my imagination to go wild a bit: what about Enums? Currently, there is a big discussion about how to declare Enum (and Flags and similar classes) whose member values "don't matter". I think it could be really cool if we could say

class Color(Enum):
    blue: 'Color'
    green: 'Color'
    red: 'Color'

and be done with it. Of course, lack of unpacking and unavaliability of class name are technical problems, and probably the best way would be just

class Color(Enum):
    blue, green, red: Color

but that's probably too big a change. However, the first variant is already possible to implement, and requires just a bit of philosophical adaptation, which I think is justified in this case. After all, blue, green and red _are_ of "type" (Enum, to be precise) Color. If we really want to strenghten the bond between annotation and assignment, it's not too great a stretch to envision a metaclass treating bare annotations as "I don't care about assigned values" than just ordinary "missing default assigned values". What do you say? For me, it reads _much_ better than the currently proposed abomination of assigning None, (), or some "magical object" like _auto_ to those members.

[gvanrossum]

I think it's up to the enum folks to come up with that. I think the first
form is too noisy (you have to say 'Color' for each value) -- maybe we'll
end up relenting and allow a, b, c: T in beta 2 or so. Personally I am
happy though to give my enums a value: red, green, blue = 1, 2, 3 or red = 1; green = 2 etc.

[vedgar]

I'm happy with explicit values too. But there are people who _really_ want to have the syntax for "don't care" values. I just think that having a class body where you have

class X(...):
    ...
    a = b

(a and b are simple identifiers) and after the class definition you have X.a != X.b (if b was previously assigned a "magical object") is infinitely worse, since it breaks a lot of assumptions of how Python assignments work. I think the "annotations way" is surely better, and of course allowing the unpacking semantics would be the best Christmas present to get after quite a few years;-), but let's first make clear whether you're ok with such a usage of bare annotations (_semantically_ - syntax can always be tweaked if there's a need, as you say).

[gvanrossum]

Enums are begging for syntactic support. They may yet get it, if someone
writes the PEP for Python 3.7.

[vedgar]

Hmm... by syntactic support, do you mean the "relenting" you have talked above (allowing unpacking on bare annotations, and maybe even a forward ("outward"?:) declaration without quotes), or you mean a full-fledged syntactic support like enum keyword? If we're going to go that route, I'm sure the general "make" keyword would be the better choice, and IIRC, you were against it.

Or maybe you mean something in between? Like

class Color(Enum, style='declarative'):
    red, green, blue

This we _can_ do right now, but it requires philosophical adjustment too.

[The _true_ solution is probably giving up the idea of the current type being "the root metaclass", and having different "grand metaclasses" for different purposes. They can be descended from one protometaclass having truly common behavior, but surely things like "__call__(T,...) calls T.__new__(...) and then maybe .__init__ on the result", or even descriptor semantics, shouldn't be there. We already had our share of problems implementing typing.py, ABCs, and other things (I think Enums are there too, they just feel it less because they are not _that_ different from normal classes) just because type does too many things. But that's probably too big a change even for Py4.:]

[gvanrossum]

Let's take this off this tracker, if you want to propose something,
python-ideas is the place to go.

[gvanrossum]

Shouldn't we close this issue? PEP 526 is accepted and marked Final.

[ilevkivskyi]

Shouldn't we close this issue? PEP 526 is accepted and marked Final.

I am glad to close this megathread. IIRC PEP 526 is accepted _provisionally_, but if there will be some ideas, it is better to open a new issue.