Pydantic: Mypy finds errors in the first example from the documentation

[andreymal]

This example

from datetime import datetime
from typing import List
from pydantic import BaseModel

class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: datetime = None
    friends: List[int] = []

external_data = {
    'id': '123',
    'signup_ts': '2019-06-01 12:22',
    'friends': [1, 2, '3']
}
user = User(**external_data)
print(user.id)
print(repr(user.signup_ts))
print(user.friends)
print(user.dict())

$ mypy --strict pydantic_example.py 
pydantic_example.py:3: error: Module 'pydantic' has no attribute 'BaseModel'
pydantic_example.py:5: error: Class cannot subclass 'BaseModel' (has type 'Any')
pydantic_example.py:8: error: Incompatible types in assignment (expression has type "None", variable has type "datetime")

This is not false positive. Pydantic __init__.py uses implicit reexport (this is an error, and I don't want to ignore it using --no-implicit-reexport), and this example uses None for a non-optional field (this is also an error).

In general, I can't trust a library that suggests putting None in a non-optional field in the very first example. This violates the mypy type system.

[dmontagu]

I agree that the implicit reexport is a nuisance; I think it is worth addressing. @samuelcolvin any ideas?

---

While mypy doesn’t like the following: signup_ts: datetime = None, there is an awkward inconsistency with mypy that this would be allowed as function arguments, and would imply the argument was optional. Given that it raises a mypy error, if you want to pass mypy, of course you should “correct” it. It is valid pydantic though if you choose not to care.

For what it’s worth, there may be times when you explicitly choose to override the annotations and have mypy assume the type is str, but allow it to be None for pydantic (eg, if you know a validator will always make it non-None). Obviously you would want to add # type: ignore but this doesn’t necessarily need to be a part of the documentation.

That said, I’m also not opposed to changing the docs here, PR welcome.

---

In general, I can't trust a library that suggests putting None in a non-optional field in the very first example.

Well, if you can’t trust the library, you are absolutely free to not use it.

This kind of language is not appreciated by the people who put much of their spare time into this project, and are offering it up to you for free. If you care enough to create an issue, please be courteous and try to phrase your misgivings in a less incendiary way.

[samuelcolvin]

Can we fix the implicit re-export in __init__ with __all__? It would be a little ugly but very explicit, we could add a test by extending the mypy tests. But very hard to implement a test that makes sure every new thing available to import from __init__.py gets added to __all__.

I'm less bothered by the Optional/None thing. Every python developer knows what this means, if they don't know what it means, adding optional won't help. Adding this would:

1. Add more fluff to the first example which tries to be as minimal as possible.
2. Perhaps confuse users as to whether Optional is required by pydantic. Since pydantic is very unusual in using annotations at run-time we have to be careful about this.

---

Couldn't have said it better myself.

When I see questions like this I'm always left asking myself the same question:

Why create the issue? - If you don't like the look of the library (I see lots of libraries that I instantly decide I don't want to use), why create the issue? If you do like it the library, surely you know starting off like that won't help get your problem fixed.

I guess the answer is either, to vent at an unjust world, or to demonstrate some new-found piece of knowledge by pointing out what you reprieve to be an error.

:sleeping:

[andreymal]

Every python developer knows what this means

Yes, every python developer and every mypy user knows this is an obvious error.

print(user.signup_ts.year)

Mypy will think this code is correct, but actually it will raise AttributeError: 'NoneType' object has no attribute 'year' when signup_ts is not set. This is a false negative because of the incorrect type hint.

Using # type: ignore is more awful.

[samuelcolvin]

python is not mypy. The standard library doesn't use mypy, many widely popular python packages don't use mypy. In fact a tiny percentage of python written or executed in the world everyday has passed mypy.

Using Optional[datetime] doesn't prevent the potential bug in python; just allows mypy (or other static analysis libraries) to warn about it more easily. Note, it would in theory be possible for static analysis tools to find this potential bug without the Optional[] decoration, as mypy does for keyword arguments.

I still think the example in the documentation is okay. If you'd like to create a PR to add a note linking from the first example to how to use pydantic with mypy I'd be happy to review it.

[andreymal]

I understand it, but it's strange that you don't respect the mypy type system. If you think that print(user.signup_ts.year) false negative is fine, it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design, and this means that python developers can't use mypy and pydantic together in their projects and libraries (they can use type:ignore, but it's equivalent to lack of mypy).

[samuelcolvin]

pydantic works absolutely fine with mypy indeed,

• @dmontagu put a massive amount of effort into building a mypy extension just for pydantic, see #708
• I put a fairly large amount of effort implement full type hint coverage in pydantic in #373
• We have two sections of the docs dedicated to mypy

That still doesn't mean our first example has to be fully type hinted. Just as most examples in django, requests, or the python standard library aren't type hinted.

[dmontagu]

it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design

It absolutely does not mean this, and it feels like you are being purposely obtuse for the sake of being contrarian.

All it means is that this one documentation example is not valid with mypy. You absolutely do not need to make use of any mypy-violating patterns in your use of pydantic, and we have done our best to make sure that pydantic is completely compatible with complete mypy coverage.

I personally use the same mypy config that pydantic uses in all of my projects, and while it is not --strict, it is also far from lenient. (And I think we should absolutely strive toward better supporting --strict mode.)

Yes, there are some patterns that work with pydantic that are not valid with mypy. There is also an enormous amount of non-pydantic python code that is valid python and not valid with mypy.

[euri10]

it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design

It absolutely does not mean this, and it feels like you are being purposely obtuse for the sake of being contrarian.

@dmontagu and @samuelcolvin it seems like you're both playing a new troll game :rofl:

[samuelcolvin]

@euri10 you've confused me and my entire office :confused: :confounded:. Shouldn't it be "is compatible"?

[euri10]

@euri10 you've confused me and my entire office . Shouldn't it be "is compatible"?

just for you ;)

[andreymal]

there are some patterns that work with pydantic that are not valid with mypy.

This is a problem. And you can't change my mind

django, requests, or the python standard library aren't type hinted.

This is also a problem, but I hope most libraries will be type hinted in the future. Note that typeshed provides type hints for the standard library and some other libraries. Although the standard library itself is not checked with mypy (it's probably also a problem but I'm not sure because the standard library has a lot of unit tests), typeshed allows me to check my code that interacts with the standard library. It doesn't save from bugs in the standard library, but it saves from bugs in my code.

There is also an enormous amount of non-pydantic python code that is valid python and not valid with mypy.

I prefer to use a python subset that is valid with mypy wherever possible. (Sometimes I use Any because I'm very lazy, but this is also a big problem in my discipline that I should fix someday.)

[dmontagu]

This is a problem.

All right, well once your PEP to change mypy errors to syntax errors is accepted, we’ll get right on converting the behavior of every edge case in the use of this library.

[samuelcolvin]

django, requests, or the python standard library aren't type hinted.

You've miss quoted me, I said

Just as most examples in django, requests, or the python standard library aren't type hinted.

Which is quite different. pydantic is type hinted.

I was talking about examples in documentation where it's common (and I would argue good practice) to partially omit type hints to keep the code as succinct and easy to read as possible.

[andreymal]

But the pydantic example is still type hinted. And these type hints are still not valid with mypy.

"missing type hints" ≠ "invalid type hints" (with mypy). Change my mind

In fact, many libraries have a bad documentation; sometimes even an API reference does not describe types, and I have to use print(type(something)) or even read a library source code to determine types. This is useless. I prefer type-hinted examples; they are actually easier to understand even if they less succinct. In general, explicit is better than implicit :)

[seansfkelley]

Just to chime in here, I also found the first example pretty weird for the same signup_ts: datetime = None reason.

My background is in Typescript, which has two null-checking modes: one where null is implicitly assignable to every type (like most languages), and one where it is not (--strictNullChecks, where you have to declare things like string | null). The root of the problem seems to be that mypy has a strict mode that people like to use (*raises hand*), whereas Python's type system (and, by extension, Pydantic) doesn't take a stance on strict versus lax null checking.

Notably, the example in question does not mention Optional or mypy anywhere, and I understand that it is valid Python, as Python operates in the first mode (noted above). So I guess what I'm trying to say is: there is an audience for Pydantic that is coming from other type checked languages (or strict-mypy codebases), rather than from un-type-checked Python, and the first example as written is jarring and unexpected to us.

The second line name = 'John Doe' also reads pretty strangely to me, since struct-like field declarations are a unusual place to have type inference (at least, coming from other typed languages).

I don't know what you want to do with that information, but hopefully it helps clear up why this example is odd to some people.

[samuelcolvin]

We could definitely improve the description of that example, explain better about types and mention/link to mypy. PR welcome.

I understand it's tempting to fully type the example, but I'm concerned that might imply to a casual viewer that all pydantic models have to be completely typed, e.g. for example you have to include Optional[] on all types with a None default, which currently you don't.

Similarly the name line is like that (even though I normally add annotations to all fields) to demonstrate type inference.

[aeldaly-tc]

Mypy fails on this for me as well:

pydantic version: 1.3
pydantic compiled: True
install path: /usr/local/lib/python3.7/site-packages/pydantic
python version: 3.7.4 (default, Sep 12 2019, 15:40:15)  [GCC 8.3.0]
platform: Linux-4.19.76-linuxkit-x86_64-with-debian-10.1
optional deps. installed: ['typing-extensions', 'email-validator']

from pydantic import BaseModel as PydanticBaseModel


class BaseModel(PydanticBaseModel):
    class Config:
        allow_mutation = False
        case_sensitive = True
        use_enum_values = True
        anystr_strip_whitespace = True

[mypy]
python_version = 3.7
mypy_path = ./src
show_error_codes = True
plugins = pydantic.mypy

follow_imports = normal
strict_optional = True
warn_redundant_casts = True
warn_unused_ignores = True
disallow_any_generics = True
check_untyped_defs = True
no_implicit_reexport = True

# for strict mypy: (this is the tricky one :-))
; disallow_untyped_defs = True

[pydantic-mypy]
init_forbid_extra = True
init_typed = True
warn_required_dynamic_aliases = True
warn_untyped_fields = True

mypy --config-file mypy.ini ./src

Outputs:

src/base_model.py:2: error: Module 'pydantic' has no attribute 'BaseModel' [attr-defined]