Flask: PEP484 static typing support

[jdemilledt]

Are there any plans to add type stubs for PEP484-compliant static typing?

[davidism]

I don't plan to add type annotations.

[ThiefMaster]

Sounds like a useful thing for someone to contribute though!

[jdemilledt]

Ok. I wanted to check if there were any ongoing efforts before I started working on it. I'll get to work when I have the chance!

[davidism]

If you're going to do this, that's great. Please do a "preview" on one file first, so I can review how it will work before modifying the entire project.

[ThiefMaster]

Aren't these type annotations usually in separate files, just like the ones for Jinja:
https://github.com/python/typeshed/tree/master/third_party/2and3/jinja2

[davidism]

I'm fairly sure typeshed is being deprecated. I don't understand the tradeoff between separate files versus comments, but comments in the code seem easier to follow.

[timster]

PEP484 type hints are not comments. They are an addition to the language. Type hints only work in 3.5 and later.

For example, an annotation of render_template might look like this:

from typing import Sequence, Any, Union

def render_template(template_name_or_list: Union[Sequence[str], str], **context: Any) -> str:
    ...

This says:

1. template_name_or_list must be a string or sequence of strings
2. context is a dict with any valid python key and Any type for the values
3. the function returns a string

Type hints are not used at runtime, but _can_ provide valuable hints in an editor when used properly.

[davidism]

I'm definitely not adding annotations, since Flask is compatible with 2.7 and 3.4+. Comments are, in fact, a supported format for specifying types in order to support older versions, that much I know. They're probably not introspectable at run time though.

[ThiefMaster]

I think that's the whole point of typeshed - being able to use the annotation syntax without making the project 3.5+-only

[timster]

Looks like you can use comments. But it's more verbose and not pretty (IMO) especially for function variables.

def render_template(
        template_name_or_list,  # type: Union[Sequence[str], str]
        **context  # type: Any
        ):
        # type: (...) -> str
    function goes here

[davidism]

There's another format, a comment of the signature with annotations on its own line. It's described right above the docs describing the multiline format. http://mypy.readthedocs.io/en/stable/python2.html

Typeshed is a way to bypass a project and add annotations for it externally. It's seems harder to maintain separate files, let alone a separate repository with the files. Use the comments if you're going to do this.

[ThiefMaster]

At least the example from @timster looks quite awful IMO...

[ThiefMaster]

The idea of typeshed isn't to "bypass a project" btw - the PEP explicitly requires permission from the package maintainers before stubs can be included there.

[davidism]

Yes, multiline doesn't look good. The other format is:

def render_template(template_name_or_list, **context):
    # type: (template_name_or_list: Union[Sequence[str], str], **context: Any) -> str
    """"docstring"""

At this rate I should have just said yes I'll implement all that. :joy:

I'm fairly sure I remember reading recently that GvR wanted to end typeshed in favor of adding the info to the projects directly, but I can't find that reference. In any case, if we're going to have it I don't want it in a remote repo that we don't maintain.

[jdemilledt]

It'd work best to just have pyi files. All type checkers are required to be
able to read the pyi stubs, so we should just implement the hints there (a
bit like C headers)

[davidism]

I do not want to deal with twice as many files, which devs have to remember to update whenever they make a change. Either add the comments or don't add anything. All this confusion is one of the reasons I wasn't enthusiastic about this to begin with.

[jdemilledt]

Unfortunately that's the best option if we want compatibility.

[tirkarthi]

Just to add to this discussion there are tools to automatically generate type hints from tests as comments that will preserve compatibility. pyannotate and pytest-annotate can be used for annotation of the source code. We have used this in our project and it worked fairly well. I tested it with Flask but it seems to generate type hints for the tests instead of the functions used. Maybe I am missing something here.

[jdemilledt]

I stand corrected. Apparently type checkers are required to read the comments in case of code straddling Python 2 and 3. Now we have to decide what form to use

[tuukkamustonen]

def render_template(template_name_or_list, **context):
    # type: (template_name_or_list: Union[Sequence[str], str], **context: Any) -> str
    """"docstring"""

You don't have to duplicate parameter names, the comment line can be just:

    # type: (Union[Sequence[str], str], Any) -> str

http://mypy.readthedocs.io/en/latest/python2.html

[lepture]

But it requires people to install typing.

[davidism]

I copied the previous annotations so I wouldn't have to write them out, and forgot to remove the names.

According to the docs, "Things like Any must be imported from typing, even if they are only used in comments." PyCharm complains about missing imports in type comments. I don't want to add a required dependency on typing.

[jdemilledt]

Correct me if I'm wrong, but wouldn't that only be an issue when either developing Flask or making Flask apps that check typing?

[davidism]

Given that I "develop Flask" pretty much every day, I do not want to see import errors flashing at me constantly. Also, mypy gives an error if the import is not present, even with only comments.

[jdemilledt]

Right. This is an interesting issue. People will want to use static typing (and thus need type hints), but some people don't, and they dislike extra dependencies for stuff they don't use. As much as typing is an extra dependency, it may be worth it to require it for Flask. Another option which could solve this is having the type stubs. Since they're Python 3 (IIRC, all type checkers that currently exist are Python 3, but support 2), typing is already included. We'll have to see how to continue from this point.

[jdemilledt]

Since this is an ongoing discussion, I'd like for this to be reopened until we figure out how we're going to approach this issue.

[jdemilledt]

@davidism There's another choice I just thought of. There can be code similar to the following such that typing is only needed in a type-checking environment. It assumes that anywhere that is checking types will have the package.:

try:
    import typing
except ImportError: pass

In that case, the development dependency list should include typing.

[jdemilledt]

Looks like we'll need type annotations for Werkzeug too. I can work on that at some point.

[neilsh]

Flask stubs were added to typeshed earlier this year: https://github.com/python/typeshed/pull/2740

[Halkcyon]

Since 2.7 is officially being deprecated this year, has there been any change of heart over the past two years on this topic? I can't express how much time is wasted by having to go back to documentation instead of just relying on my editor informing me or debugging through a console to find members.

[davidism]

From what I understand, the typing module has performance and memory issues in < 3.7 because it's eagerly evaluated and creates a ton of anonymous classes. Until we drop Python 3.6 and can take advantage of from __future__ import annotations in 3.7, I still don't plan to add annotations unless it can be proven that the startup time and memory footprint for 3.6 is not significant.

Annotations probably won't do as much as you expect for Flask. Most of the real stuff happens in the other 5 Pallets libraries (which are under the same 3.7+ constraint), and much of the interesting Flask stuff, like what a view function can return, is so generic that a type for it is fairly annoying to read in an IDE popup.

having to go back to documentation instead of just relying on my editor

PyCharm completions and parameter hints seem to be doing just fine for me without annotations, so I'm not sure where this is coming from. What functions have particularly confusing APIs that you're going back to the docs for? Note that many IDEs also have docs popups in the same place as param types popups, so you should be able to get them directly in your editor. And as already noted, you can already get the typing support that typeshed provides.

[Halkcyon]

@davidism I appreciate the feedback. I had not heard that annotations have performance impacts.

what a view function can return

I, personally, still refer to documentation on this especially with the change in 1.1 for automatic json conversion (which annoyingly doesn't work on lists).

functions have particularly confusing APIs

It's usually not functions themselves, but what they are returning, especially in tests. For example, the FlaskClient does not have auto-completion on the response from open in IntelliJ (documentation also doesn't say what the return type is, which required me to do some introspection).