Typing: Type hinting would generate a lot of import statements (possibly leading to circular imports). Use strings instead of imported names?

[vladiibine]

The need for the annotations to be valid Python expressions could generate a lot of otherwise unneeded (and ugly) import statements.

Two solutions seem appropriate on the first look:

1. The hinting could be done by using strings, if the referenced objects are in another module.

def foo(processor: 'myapp.processors.Processor',
        document: 'myapp.documents.Document')

1. Type comments could be used for type hinting, instead of actual function annotations

def foo(x,  # type: int
        y)  # type: float

Of course, the solutions I give are simple suggestions, I'm sure there is a better way to solve this problem.

Arguments against using actual imported names for type arguments:

1. It's ugly to have lots of extra imports, only used for hinting
2. The startup time of a lot of applications would become much slower - because importing lots of names from all over the place would require a lot more stuff to be loaded upfront.
3. Higher changes of circular imports - it's really not a problem one would expect to introduce. One could imagine people choosing not to specify type hints, out of fear of circular imports, or people introducing circular imports, by mistake, when creating "useful" type hints.

[gvanrossum]

The current rules for type hints in string literals or comments still require the imports. What makes you think you will need a lot of extra imports? Have you experienced this while trying to convert some code to use type hints? That would be very interesting feedback, I'd like to know more.

[vladiibine]

I've been using the kind of type hinting that Pycharm (and Sphinx I think) supports - :type myvar: SomeClass.
Although these hints are placed in comment strings, in order for them to "work", those symbols must be either proper imported names, fully qualified names, or relatively referenced names (with the relative import syntax).

The problems I had with the first approach (importing the names) are exactly the kind of problems this PEP would introduce.

So for the specific details: I'm working on a Django project (closed source), which has evolved over time, and includes about 50 models. We have methods on our models which accept as parameters instances of other models. In the current state of our project, to avoid circular references, we already use a pattern allowed by Django, whereby we specify the references (foreign key relationships) between these models as strings.

Now admittedly, many of the circular import problems could be solved by refactoring the models that reference each other, to live in the same module, but the effort for this would at this point would be non trivial and would require rethinking the architecture of the entire project.

This seems a little too much work to enjoy the simple benefits of type hinting.

To show however that not only the project I'm working with will have this problem, I cloned a popular django repository, namely django-cms.

I try to show how in django, the intrinsic relationship Model -> Manager -> QuerySet -> Model is so well spread that at least for the django community, the type hinting, as proposed, will cause a lot of chaos.

https://github.com/vladiibine/django-cms/tree/type-hinting-issue-105

In the first commit, I show how Django was already preparing for cyclic dependency problems, and solved this issue in exactly the way I'm proposing - using strings instead of names
https://github.com/vladiibine/django-cms/commit/553982f1a4553f059235fa9b7664f17d0e283e7c

In the second commit, I again break the application, merely by importin some names, and in the third commit, I fix the app by moving the classes around (I move the PageQuerySet in the same module as the model Page). However acceptable this would seem, as far as I understand Django, what I've done is not OK. Also, please notice that I had to fit more classes inside a module JUST to be able to document some function return types.
https://github.com/vladiibine/django-cms/commit/4e1a1b04abc834348ee80e77d661ff99600e2d9c

In the 4th commit I again try to specify types, and get to a functioning application state. This state unfortunally is quite sad, because I can not document function parameters with their proper types. That is, in Django, at least a part of a Manager's methods are expected to return QuerySets. It's quite unfortunate to have to yet again move the Manager class in the same module as the Model and the QuerySet class. Of course, we can imagine what happens if model methods take parameters or return instances of other models: All the Models, Managers and QuerySets will end up in 1 file, which would be crazy.
https://github.com/vladiibine/django-cms/commit/426c23ccaa91733f4841b1efb8f1b52949846630

Please note that DjangoCMS is not a model-rich application. I counted ~15 models here. As I mentioned the project I'm working on has around 50 models, which are more tightly connected. While I understand that this wouldn't be an example of Python project architecture, I'm concerned that projects like this (or ones even more coupled) would not be able to benefit much from the type hinting. These projects would however be in most need of it, to make them more readable, understandable and easier to refactor.

I mentioned that the loading time of applications would be affected, but have not yet tried to investigate this, though I'm hoping to sell that argument together with this one: If type hinting requires names to be imported, the coupling of modules will become more problematic.

[gvanrossum]

I see. This is indeed nothing to sneeze at. A style that limits imports to modules (instead of classes) would have much less of a problem here, because at the module level circular imports are not much of a problem, and then using strings for annotations could preserve the "must be imported" requirement. But this is clearly not the style that most Django projects use, so it would be kind of a hollow recommendation. Stubs could probably bypass the problem (the imports would still be there but a stub does not need to follow the strict rules of define-before-use when it comes to type hints). But then you lose the type checking of your model code.

The reason why we require the imports for type hints (even in strings or comments) is that otherwise the type checker would have to guess about the full name of a class referenced in a type hint -- many class names are not unique and even if they are the search space is unwieldy. "Fully-qualified names" might be considered an exception but there is still the ambiguity if a top-level module name matches a locally-defined name.

[JukkaL]

I agree that type annotations introducing circular module dependencies can be a pain. I've also seen that in the mypy codebase. Refactoring can help with this, but it can result in huge modules with dozens of classes which is not great for maintainability.

We had a related discussion (though without the Django context) before: https://github.com/ambv/typehinting/issues/34

Just supporting a non-imported class name in annotation isn't really practical, since we don't know where to look for the class definition, as Guido pointed out.

Currently the only reasonable approach that I've been able to come up with (and this is what I currently use in mypy) is to import a module without from ... import and use a fully-qualified string literal type. Example:

import app.model   # introduces circular dependency

def foo(t: 'app.model.Thing') -> 'app.model.Thing': ...

This is ugly and verbose, and it introduces circular dependencies, but it has worked for me without needing much refactoring.

Another idea (that I mentioned in the above issue) is to introduce a new primitive for introducing references to a class defined in another module, without the need to import anything. For example:

from typing import TypeAlias

Thing = TypeAlias('app.model.Thing')  # must be fully qualified name?

def foo(t: Thing) -> Thing: ...

This doesn't introduce a circular module dependency and the annotation is less ugly. However, the type alias definition is pretty verbose. Maybe we can also support relative module references (e.g., TypeAlias('.model.Thing')). A TypeAlias is basically a wrapper around str and doesn't have any interesting logic. It could define some methods such as .get() to actually find the target type but that's not important.

To make this more practical, we can define a module that only contains a bunch of type aliases (note that it doesn't import any model modules):

# app/types.py
from typing import TypeAlias

Thing = TypeAlias('app.model.Thing')
Manager = TypeAlias('app.manager.Manager')
... # etc. for other model classes

Now we can import the type aliases from this module and code looks cleaner. We just need to add an import -- however, now we can use from import without introducing a circular dependency since types doesn't import any other app modules:

from app.types import Thing

def foo(t : Thing) -> Thing: ...

The tradeoff is that we may need to add a definition to types.py for every new model object we define. I think that this is not a terrible burden, but it's not very nice compared to Java, for example.

If we'd have the freedom to modify the language, we could add a new kind of import that defines a (type) alias instead of importing a name. For example (straw man syntax proposal):

from app.model import alias Thing

# now Thing is an instance _Alias('app.model.Thing') and we didn't import app.model --
# there's no circular dependecy

def foo(t: Thing) -> Thing: ...

The above code would be equivalent to this code (I'm assuming that _Alias gets added to the builtins, but it can live anywhere):

Thing = _Alias('app.model.Thing')

def foo(t: Thing) -> Thing: ...

_Alias would be similar to TypeAlias above, but it wouldn't be specific to typing and could be used by Django, for example, as an alternative to string literal escaping. The problem with this is that mistyping an imported name wouldn't be an error, since we can't look into the target module when evaluating import alias. However, this is no worse than string literal escapes, and a type checker or linter would verify that the imported name is actually defined.

Another issue is that using an alias as a regular value outside a type annotation would probably result in an AttributeError or TypeError, which may be a little confusing. Having useful exception error messages would probably go a long way towards making this a non-issue.

[NYKevin]

I have some code which has an issue similar to this, though my type annotations are not (exactly) compatible with mypy's syntax. I solved it at runtime by basically building strings out of __name__ and calling str.split('.')/'.'.join(), as you can see here, since I don't want to repeat the full name of the module (that would defeat the purpose of relative imports). It's rather ugly and I doubt it can be parsed statically, but it does leave the annotations in a reasonable state for Sphinx and other tools which actually import my modules. Because the other module also imports this one, and uses it for non-type-hinting things, I feel it would be difficult to make this work correctly with circular imports. I suppose I could force the import order using __init__.py, but I dislike that kind of kludgery.

[gvanrossum]

I've written up a brief discussion of this issue with a suggestion how to go about it in the PEP: https://hg.python.org/peps/rev/06fbe54fcfe1

[164747]

I bump into this issue from time to time. Often the circular import statement only exists due to type hinting and are not used by python. In the (very) long term future, would the syntax

from my_module himport A

be a good solution? That is a statement ignored by python but read by the type-hinter as a normal import.

[gvanrossum]

On Thu, Dec 3, 2015 at 1:02 PM, 164747 [email protected] wrote:

I bump into this issue from time to time. Often the circular import
statement only exists due to type hinting and are not used by python. In
the (very) long term future, would the syntax

from my_module himport A

be a good solution? That is a statement ignored by python but read by the
type-hinter as a normal import.

I don't know about that particular syntax. Personally I would rather
experiment with these:

# type: import my_module
# type: from my_module import A

since they can be made to work in all versions of Python, today.

[NYKevin]

Personally, I like Guido's comment syntax better than inventing a new keyword (which at this point would be 3.6+ only at the absolute earliest). I mean, I suppose you could restrict the new keyword to stub files, but... eww.

[164747]

I wouldn't assume my syntax to be better than Guido's =)

[vladiibine]

I'm not sure if this is a good idea, but if python had some special comments where only metadata lives, this is how I imagine metadata could look like.

@# meta info about the code goes here
@# and then some more meta
@# return: mymodule.A
@# param x: othermodule.B
def foo(x):
  return something

This wouldn't mess up with the interpreter at run-time, but could be used to give static hints about stuff.

I'm sorry if I made Python look too much like something else. I also realize this isn't necessarily very compatible with the python 3.5 type hinting.

[gvanrossum]

Please take the more general discussion elsewhere (e.g. python-ideas).

[henryJack]

Sorry, but I wanted to check if the original issue has now been solved:

Question 1

"Type hinting would generate a lot of import statements" - Yes, this is true. My python code seems to have 2x as many import statements at the top of the scripts. There is no way to avoid this, right?

Question 2

"(possibly leading to circular imports). Use strings instead of imported names?" - Yes, this is also true. The current workaround is to use conditional imports. i.e.

from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from foo import Bar

[ilevkivskyi]

Sorry, but I wanted to check if the original issue has now been _solved_:

I think yes, it has been __solved__. TYPE_CHECKING is official and supported by mypy and probably other type checkers as well. If you don't like string literals, then vote for PEP 563.

[henryJack]

Okay great! How do I vote? Can I do it on the repo? https://github.com/ambv/static-annotations

[ilevkivskyi]

@henryJack Better send a message in this thread. (but you can of course star the repo as well)

[zsluedem]

@ilevkivskyi The TYPE_CHECKING works in python3.7 version? I can't make it work in 3.6 version.

[ethanhs]

@zsluedem what specific error are you seeing? It works fine on 3.6.4.

[gvanrossum]

What micro version? typing.TYPE_CHECKING exists in 3.6.5. It probably did
not exist in 3.6.0; I don't recall in which version it was introduced
(though a bit of git archaeology could surely tell you).

[ethanhs]

It seems to have been included in the changelog for 3.6.0 a2 https://docs.python.org/3.6/whatsnew/changelog.html#id110

[gvanrossum]

OK, @zsluedem please open a separate issue and show exactly what you did and what error(s) you got.

[zsluedem]

Here is the situation.
I have a a.py:

class A():
    pass

And I have a b.py:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from a import A

def foo(a:A):
    pass

The directory tree is like :

.
├── __init__.py
├── a.py
└── b.py

When I try to run b.py,I got the error below:

$ python3 ./b.py
Traceback (most recent call last):
  File "/Users/will/work/finance/untitled/b.py", line 7, in <module>
    def foo(a:A):
NameError: name 'A' is not defined

I am not sure if I use it right.

OS: MacOS 10.13.3
Python version: 3.6.1

[JelleZijlstra]

TYPE_CHECKING only affects static type checkers, not what python itself does at runtime. At runtime, the interpreter will see a reference to A that it cannot resolve and throw an error. You need to work around this by putting the annotation in quotes: def foo(a: 'A').

[zsluedem]

@JelleZijlstra Thanks for the help. I know it now.

[ilevkivskyi]

You need to work around this by putting the annotation in quotes

...or you can use the fancy new from __future__ import annotations

[zsluedem]

@ilevkivskyi I am on 3.6.4 version and I can't import from __future__ import annotations.

[ethanhs]

@zsluedem from __future__ import annotations is a 3.7 feature.

[laike9m]

Hi @gvanrossum, I really like the comment approach you mentioned:

# type: import my_module
# type: from my_module import A

I think it surpasses current solutions in that

1. It separates type imports from regular imports
2. One can put type import near the code it's used

1 and 2 both improves readability a lot. Is there a chance this gets added/proposed someday?

[gvanrossum]

No, we're going in a different direction.

[kfsone]

I find this breaks encapsulation: I have to list imports that are present solely for type annotation, and sometimes that means I have to include an explicit reference to the back-end a library is using.

c.f

from threading import Thread

class Worker(Thread):
    def __init__(
            self,
            args: argparse.Namespace,
            callback: Callable[[argparse.Namespace, data: any][]):
    #...

This module now has to import [from] argparse.

My concern is that this is the beginning of C-like include hell, but the precedent is already set by exceptions. You regularly have to import from a library for the singular reason that some other library/module doesn't repackage exceptions from a provider one or two layers down.

import newt
newt.from_peasant()  # may throw subprocess.CalledProcessError if not running on witchOS

In both cases, solvable with an import statement, it just always feels a tad clutzy to me.

"Hello, I've come to catch an exception"
"Oh yes, which one?"
"subprocess.CalledProcessError"
"Oh, I've never heard of that one, sir. Sounds foreign. Is it an import?"

Actually, a thought inspired by your surname:

    def ni(argv: Namespace from argparse):

    except CalledProcessError from subprocess as e:

[JelleZijlstra]

Protocols can potentially help in some cases. For example, instead of argparse.Namespace, you could expect something like

class Namespace(Protocol):
    def __getattr__(self, attr: str) -> Any: ...

[kfsone]

"it was a randomly chosen example" if (that's a workaround to passing argparse namespaces) else "if I wasn't fond of importing argparse just for an annotation or an except, then effectively forward-declaring the type would at least make me miss the days of importing modules I didn't need" :)

[JukkaL]

This is a trade-off between doing things in a modular way (defining a protocol or doing some other extra work) or the easy way (directly using a type from a library module). You arguably decided to go the easy way when implementing Worker, resulting in the need for extra imports. If you didn't use type checking, you'd probably still want to document args, and you'd have to make a reference to argparse in the docstring anyway, right? Besides, if you use if TYPE_CHECKING:, the import doesn't happen at runtime, so the dependency is for type checking and documentation only.

Other solutions that may help:

• Transform the object to instances of more primitive types, such as a dictionary for a namespace. This is not specific to the argparse example, but sometimes you'd have to invent custom data structures to hold the data in a useful form, which may defeat the purpose. This is analogous to translating an exception from subprocess to a library-specific exception class.
• Sometimes you only care about some aspects of a library type and you can represent this using built-in ABCs/protocols such as Iterable, Sequence and Mapping.
• If you need to refer to some library type in many places, you can add a corresponding protocol type to a type utility module. This way way you can reuse the protocol, instead of defining multiple one-shot protocols.