Pydantic: Class attributes starting with underscore do not support assignment

[dmfigol]

Bug

For bugs/questions:

• OS: macOS
• Python version import sys; print(sys.version): 3.6.6
• Pydantic version import pydantic; print(pydantic.VERSION): 0.14

In #184 it was suggested to use variables starting with the underscore, however this does not work.
The last comment in #184 referred to the same problem, but it is technically a separate issue.

from pydantic import BaseModel
m = BaseModel()
m._foo = "bar"

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dmfigol/projects/my/public/simple-smartsheet/.venv/lib/python3.6/site-packages/pydantic/main.py", line 176, in __setattr__
    raise ValueError(f'"{self.__class__.__name__}" object has no field "{name}"')
ValueError: "BaseModel" object has no field "_foo"

[samuelcolvin]

(I've updated the issue title to better reflect the issue).

Personally, I don't think it's that important, but if you'd like the feature, I'd be happy to accept a PR.

[asemic-horizon]

I'm having this same problem in a non-workaroundable context. Can someone maybe give some context into why there is this restriction and if there's any way to hack around it, even if ugly?

[samuelcolvin]

Have you tried using field aliases?

eg,

class MyModel(BaseModel):
    foobar: str
    class Config:
        fields = {'foobar': '_foobar'}

[asemic-horizon]

That works in a small example
```
import json, my_models

with open('test.json') as f: d = json.load(f)
model = my_models.MyModel(**d)
````

I continue to have problems from the context of FastAPI. That's not the beginning of a new question, it's for the next person to arrive here from Google.

Thanks for your help!

[edatpb]

That's not the beginning of a new question, it's for the next person to arrive here from Google.

👋 Oh hi!
@asemic-horizon Did you find a solution or a good workaround yet? I'll post again if I find one

[asemic-horizon]

Yeah sorry.

As I had said, field aliases worked in small toys examples but didn't seem to work in a larger app context, which made me suspect the problem was at FastAPI itself and therefore not pertinent to this issue.

After further rounds of research and bug squashing, minor and subtle errors on my end (my code on top of FastAPI) surfaced. By then I had forgotten I had gone to Pydantic support for it.

Thanks everyone!

[asemic-horizon]

(On my end I would "vote this issue closed" or its equivalent in the github issue tracker. I suspect I can't because I didn't actually open it.)

[samuelcolvin]

I'm pretty sure aliases work fine in fastapi.

Closing, but if someone really wants this, let me know and I can reopen it.

[tiangolo]

@asemic-horizon field aliases work normally in FastAPI and are even documented/recommended in several places.

If you still have issues with them, feel free to open an issue in FastAPI with the minimum sample to reproduce your issue: https://github.com/tiangolo/fastapi/issues/new/choose

[bartekbrak]

I read the above but saw no explanation for this limitation. From brief study of the source code I could not determine any danger in sunder fields.

This check is introduced in the first commit ever a8e844da and a check added in f9cf6b42 but no explanation given in commit messages.

Python itself by design does not enforce any special meaning to sunder and dunder fields - the "we are all consenting adults here" approach. I am used to a convention of marking fields private-like with sunder.

Using aliases is clumsy. I use Pydantic to save code lines and make code more readable. I would rather not add this magic. Also this prevents me from doing arguably dirty trick of having both _response and response (processed response).

I would very much like to know the motivation or have this restriction lifted.

Still, I adore Pydantic, thanks!

[pdonorio]

Hello there, I use pydantic in FastAPI as well, and I use mongodb as database.

Mongo sets Identifiers as the field _id in collections/query outputs and handling those via Pydantic models is quite confusing because of this issue. Let me try to show what I do to ask if I'm doing something wrong; situations I've met with my use case

1. when sending the id as output I have to create a Model with the alias as

class Foo(BaseModel):
    id: str = Field(..., alias='_id')
    ...

query = db.collection.find(...)
validated = Foo(**query).dict()
# where validated = {'id': MONGO_HASH, ...}
# send validated as response

Note: if I'd want to send the id like mongo does (if my API are used by another service that deals with queries and mongo for example) I need to use

validated = Foo(**query).dict(by_alias=True)
# where validated = {'_id': MONGO_HASH, ...}

1. when storing a PUT in the database the above require the input to be using id instead of _id like:

data = read_from_source(...)
# data = {'id': EXISTING_MONGO_HASH, ...}
db.collection.update(Foo(**data).dict(by_alias=True))

but if some other service is sending me raw data I would get the _id in input so I need to change the model to

class Foo(BaseModel):
    id: str = Field(..., alias='_id')
    ...
    class Config:
        allow_population_by_field_name = True

otherwise I get a validation error

---

There are other situations that I can't recall well enough to show with code,
but in general I'm currently forced by pydantic on taking care of these differences since the underscores are not accepted.
I wouldn't use the id translation at all, I'd go with _id always - because I'm risking non deterministic behaviour in wrapper methods for all models in the codebase and in the responses.

Hope this helps to reason on it, but thanks for this awesome library!

[gpakosz]

Hello @samuelcolvin 👋

I just started using pydantic along with FastAPI and pymongo. I used alias like @pdonorio mentions above. But I would like to understand why attributes starting with underscore are disallowed in the first place?

[samuelcolvin]

I still think most python users would not expect "private" attributes to be exposed as fields.

I know python doesn't formally prevent attributes who's names start with an underscore from being accessed externally, but it's a pretty solid convention that you're hacking if you have to access attributes with leading underscores. (Yes I know there are some exceptions, but in the grand scheme they're rare).

In the example quoted above regarding mongo, I would personally feel it's inelegant design to keep the mongo attribute _id in python with the same name. You should be able to use an alias generator and a shared custom base model to change the field name in all models without repeated code.

Still, if people really want this, I'd accept a PR to allow field names with underscores. I guess the best solution would be to add a new method to Config which replaces this logic, perhaps returning None to fallback to the default logic.

Then in v2 we can completely remove Config.keep_untouched it's a bad name and a confusing attribute.

---

None of my business, but: @gpakosz, unless i'm missing something, you're using a synchronous library for io (pymongo) while using an async web framework (starlette+fastapi). I would worry that your entire process will hang during every database call, thereby negating the point in asyncio.

[Garito]

I'm agree that, in python, underscore variables are considered private, so I understand your point
But
everyone that uses mongodb (perhaps others) must flow our code with by_alias=True which is not that bad (but talking about elegance seems very weird)
The problem becomes deeper when you use lists of pydantic models
Then you must add a good amount of extra fors to return the information in the correct way (for read and write)

What about a config attribute like always_by_alias = True?

[samuelcolvin]

You can just implement your own custom base model and overwrite .dict() to have by_alias=True default.

I don't want to add more config options unless absolutely necessary. Also it would be weird since I think .dict() and .schema() have different defaults.

Perhaps we can change the default to True in V2 once we have load and dump aliases.

[samuelcolvin]

Allowing fields with underscores by default would also conflict with #1139.

[mortbauer]

I'm very new to pydantic, however i'm really impressed and really like it, however with the current implementation filtering out any field starting with an underscore which is completely unconfigurable main.py 162
except monkey patching is not really acceptable. So I think it is really necessary to add an additional configuration parameter.

[SmadusankaB]

Same thing happens in connexion framework. I want to use default mongo _id as the id.
python class FileMetadata(BaseModel): _id: str created_date: str
Set id
python data = FileMetadata.parse_obj(info.to_dict()) data._id = my_custom_id
Error
ValueError: "FileMetadata" object has no field "_id"

Change summary explained here solved the problem.

````python
class FileMetadata(BaseModel):
_id: str
created_date: str

class Config:
    extra = Extra.allow

````

[samuelcolvin]

Better to use an alias for that field.

[StephenBrown2]

For the pymongo issue specifically I moved to mongoengine, which exposes _id as id. No more need for aliasing!

[iyedg]

The solution for me was using aliases and by_alias when exporting the model. Using the example from @Seshirantha that would look as follows:

class FileMetadata(BaseModel):
    id: str = Field(alias="_id") # can be done using the Config class
    created_date: str

input_data = {"_id": "some id", "created_date": today()}
FileMetadata(**input_data).dict(by_alias=True)

That should return {"_id": "some id", "created_date": "whatever today is"}

I am using Pydantic with pymongo therefore having access to _id and _meta on the validated model is very important.

[killswitch-GUI]

+1 on this, would be a nice feature. Ill use alias in the meantime, but this for sure caught me off-guard.

[mortbauer]

I earlier thought I needed that feature, and also found a quite easy way to have it via monkey patching:

# monkey patch to get underscore fields
def is_valid_field(name:str):
    if not name.startswith('__'):
        return True
    elif name == '__root__':
        return True

pydantic.main.is_valid_field = is_valid_field

However I have to say that using an alias works very well now for me!

[PrettyWood]

1679 would probably be a good solution

[aolefira]

+1, "allowing fields with underscores" is necessary for mongodb issue. Using aliases is just workaround and not very handsome

[153957]

+1 – Fields starting with _ are also useful for ArangoDB, each vertex object there has a _key, _id, and _rev, and edges also have _from and _to.

[PrettyWood]

Hello,
Should we close this issue now that private model attributes have been implemented?

from pydantic import BaseModel

class Model(BaseModel):
    _foo: str
    _id: str

    a: str

    class Config:
        underscore_attrs_are_private = True

m = Model(a='pika')
m._foo = 'foo'
m._id = 'id'
print(m.dict())

[AntonOvsyannikov]

Sorry, in which version underscore_attrs_are_private are implemented? I use 1.7.2 and still have error while using private attributes:

import pkg_resources
print(pkg_resources.get_distribution("pydantic").version)

from pydantic import BaseModel

class Model(BaseModel):
    _foo: int
    class Config:
        underscore_attrs_are_private = False

m = Model(_foo=2)
print(m, m._foo)

1.7.2
Traceback (most recent call last):
  File "/Users/aovsyannikov/Library/Application Support/JetBrains/PyCharm2020.2/scratches/scratch_139.py", line 14, in <module>
    print(m, m._foo)
AttributeError: 'Model' object has no attribute '_foo'

[153957]

Apparently with the new private (underscore_attrs_are_private) implementation, you can still not provide the values during initialization of the class. Moreover, the values are not included in the .dict() output.

class Model(BaseModel):
    _id: str = None

    class Config:
        underscore_attrs_are_private = True


x = Model(_id='a/123')
# Model()
x._id
# None
x._id = 'a/123'
x
# Model()
x._id
# 'a/123'
x.dict()
# {}

If I include extra = 'forbid' in the Config it forbids including _id when making the model, it can still be assigned after making the model, but is still not included in the dict output.

[AntonOvsyannikov]

Ok, thanks, clear. Just copy there my comment to #2101
Now, been having the Config for such attributes it seems quite artificial limitation to ignore sunder attributes (which is only private in Python world, and widely used outside), so it brings lot of inconvenience when work with models for MongoDB i.g. Suppose more flexible config is required, like sunder_attributes : Literal['normal', 'ignored', 'private']. I will risk opening a new feature request.

[MrMrRobat]

@153957, @PrettyWood, just to make it clear: underscore_attrs_are_private and private attributes themselves are not meant to be used as fields. They're meant to be internally implemented and used only inside of model class, not anywhere outside.

[153957]

Understood, however, for my usage with ArangoDB the issue remains that there are some 'normal/valid' fields which start with a single underscore that should be included in the output of dict(). For now I am using something similar to the monkey patch suggested by mortbauer.

[emremrah]

I have just tried a workaround like this to issue the data model to MongoDB using .dict() method:

class A(BaseModel):
    _id = 5

    def dict(self):
        d = super.dict()
        d['_id'] = d['id']
        del d['id']
        return d

When I call A().dict(), it gives an error: AttributeError: type object 'super' has no attribute 'dict'. But BaseModel has a dict method. I'm probably missing some Python feature. Can someone enlight me why this is not working? Thanks.

[PrettyWood]

Hey @emremrah
Probably because you want to call super() not super ;)

[emremrah]

Hey @emremrah
Probably because you want to call super() not super ;)

Ahh you are right @PrettyWood ,:

class A(BaseModel):
    id = 5

    def dict(self):
        d = super().dict()
        d['_id'] = d['id']
        del d['id']
        return d

returned {'_id': 5}. So if anyone is interested, I just have to call .dict() method when inserting to MongoDB. But initiating A with a MongoDB document like

doc = collection.find({})
A(**doc)

still won't work as we wanted. Any suggestions on that?

[PrettyWood]

@emremrah Using alias id = Field(5, alias='_id') should work 👍

[emremrah]

@PrettyWood Yep, I just have to be careful to use .id and not ._id in code. Thank you!

[samuelcolvin]

Thanks @PrettyWood, closed.