Pydantic: Required Optional fields

[boydgreenfield]

Question

Currently, adding an Optional field just removes the field from the generated required fields list in the schema/JSON schema. Is there a mechanism for having it instead generate an anyOf field requirement and requiring a field with a null value?

Please complete:

• OS: Mac OS X
• Python version import sys; print(sys.version): 3.7.4 (default, Oct 12 2019, 18:55:28)
• Pydantic version import pydantic; print(pydantic.VERSION): 1.1

I couldn't find info on this in the help manual, though the general documentation for Union (link) doesn't specifically call out that it behaves differently with None.

from typing import Optional
import pydantic

class Metadata(pydantic.BaseModel):
    nullable_field = Optional[str]

Would love a way to generate {"anyOf": [{"type": "string"}, {"type": null}]}, though I realize that passing {} and {"nullable_field": None} will generate equivalent Pydantic models.

Thanks for the help and the library!

[samuelcolvin]

Humm, the answer is "not really".

Currently the best solution might be allowing schema_extra to be a function as per #892. Then you could have a function that inspects required and modifies all non-required fields to add anyOf.

The reason for this is that although Optional[int] == Union[None, int], Optional generally has a different semantic meaning. I thought about changing it so that Fields marked nullable_field: Optional[int] e.g. without a default were required but could be None, however that would mean a field marked with the word "optional" was actually required, which is just too weird.

I guess in theory we could add a RequiredOptional type that meant you must supply a value to the field but it could be None. But I'm not sure how much work it would be.

@tiangolo or @dmontagu do you have opinion on this?

[dmontagu]

It seems a little unconventional/non-pythonic to me that Optional[X] gets a default value of None without that being explicitly specified, just because most related use cases would still treat it as required (eg dataclasses, regular function signatures, etc). I recognize the naming weirdness around the word Optional, but it just seems like the way python’s typing module works (and the way optionals work in most languages, as far as I’m aware).

That said, in practice having None as the default seems to be right in the vast majority of cases, so I don’t mind. It definitely removes a lot of = Nones.

Moreover, the schema currently generated by Optional is what I want in most cases, so if it is extra work to support both (and I suspect it would be), I’m glad it works the way it does now.

But I would be in favor of adding a RequiredOptional type that generated the schema described here. I’ve run into a few cases where I wanted this behavior in the past.

[samuelcolvin]

How about we do RequiredOptional now, and consider switching so that Optional fields are not implicitly optional in version 2?

Cue: someone comes along and abuses us with "Optional fields aren't optional! Rant Rant Rant". However, it is annoying not to have nullable fields which are required.

Other options:

• use Nullable instead of RequiredOptional
• have a config switch for whether Optional can be required

[dmontagu]

I think I’d prefer switching the behavior in v2 over adding a config setting. I’d also be okay leaving it as is; I’ll think about it. I just think config settings come with a lot of maintenance burden.

I’m not super worried about people saying Optional isn’t Optional, since, while yes that is maybe weird, for better or worse it’s also the convention everywhere else. It won’t surprise me if someone complains, but I think “convention” offers an easy response.

Nullable as a name sounds better to me than RequiredOptional 👍.

[tiangolo]

I think using Field(...) could be a way to specify that it's required, I guess it should do it, maybe we can implement it that way:

from typing import Optional
from pydantic import BaseModel, Field

class Metadata(BaseModel):
    nullable_field: Optional[str] = Field(...)

That would mean that the nullable_field can take str or None, but it has to have something (with the ...).

That's what I think I would imagine. (That independent of how we end up handling #1028).

[samuelcolvin]

Agreed, let's go with ... forcing a field to be required regardless of whether it's optional.

Then change the behaviour in v2 so that all fields without a default value are required regardless of whether they're optional.

[tiangolo]

About having Optional[x] with an implicit default of None, I'm OK with it. I'm OK with both options.

I can imagine how it could feel weird to have something without an explicit = None have a default None value.

But at the same time, I could imagine feeling weird about having something declared as Optional[x] that doesn't behave as "optional".

So, I guess it's kind of a fuzzy area... we'll get the rant anyway :joy:

Given that, I guess I would prefer to follow the convention in dataclasses, having Optional[x] still require a value. Just for convention, as @dmontagu says. But again, I'm OK with both.

---

About nullables / required-optional, we'll be able to achieve that with nullable: Optional[x] = ... or nullable: Optional[x] = Field(...) with #1031 .

---

That still leaves open the discussion for how Optional[x] vs Optional[x] = None should behave and if a Nullable[x]/RequiredOptional[x] is needed or not.

We'll still be able to achieve both functionalities with ... and None.

[bartv]

It seems that this change caused a regression in the latest release. The behavior between static and dynamic classes is different now.

from typing import Optional

from pydantic import create_model, BaseModel


def test_optional_value():
    class Test(BaseModel):
        field1: str
        field2: Optional[str]

    Test(field1="a", field2="b")
    Test(field1="a")


def test_dynamic_optional_value():
    fields = {'field1': (str, ...), 'field2': (Optional[str], ...)}
    Test = create_model("Test", __base__=BaseModel, **fields)

    Test(field1="a", field2="b")
    Test(field1="a")

With pydantic 1.1.1 both test cases succeed, with pydantic 1.2 the dynamic test case fails.

Is this expected? Based on the changelog and the documentation I am inclined to think it is not. Should I open a separate bug?

[samuelcolvin]

@bartv, thanks for reporting this. I've created a new issue #1047

[yurikhan]

I’m the kind of guy who is known to rant about optionals, nulls, zeros, empty strings, empty arrays and empty objects and how all of them are distinct cases. Would really like if all of the cases here and in various linked issues could be easily expressed in a Pydantic model and mapped to JSON Schema equivalents. I’ll try to summarize the current ways and workarounds to do it:

• A field is required and cannot be null. — Works out of the box

  class Foo(BaseModel):
      foo: int
  
  Foo()
  # ValidationError: field required
  Foo(foo=42)
  # Foo(foo=42)
  Foo(foo=None)
  # ValidationError: none is not an allowed value
  Foo.schema()
  # {'properties': {'foo': {'title': 'Foo', 'type': 'integer'}},
  #  'required': ['foo'],
  #  'title': 'Foo',
  #  'type': 'object'}
  

• A field is required and can be null. — Requires a schema patch

  class Foo(BaseModel):
      foo: Optional[int] = ...
  
      class Config:
          def schema_extra(schema, model):
              schema['properties']['foo'].update({'type': ['null', 'integer']})
  
  Foo()
  # ValidationError: field required
  Foo(foo=42)
  # Foo(foo=42)
  Foo(foo=None)
  # Foo(foo=None)
  Foo.schema()
  # {'properties': {'foo': {'title': 'Foo', 'type': ['null', 'integer']}},
  #  'required': ['foo'],
  #  'title': 'Foo',
  #  'type': 'object'}
  

• A field is optional but if present cannot be null. — Requires a validator

  class Foo(BaseModel):
      foo: Optional[int]
  
      @validator('foo')
      def not_null(cls, v):
          if v is None:
              raise ValueError
          return v
  
  Foo()
  # Foo(foo=None)
  Foo(foo=42)
  # Foo(foo=42)
  Foo(foo=None)
  # ValidationError: type_error
  Foo.schema()
  # {'properties': {'foo': {'title': 'Foo', 'type': 'integer'}},
  #  'title': 'Foo',
  #  'type': 'object'}
  

• A field is optional and may be null. — Requires a schema patch

  class Foo(BaseModel):
      foo: Optional[int]
  
      class Config:
          def schema_extra(schema, model):
              schema['properties']['foo'].update({'type': ['null', 'integer']})
  
  Foo()
  # Foo(foo=None)
  Foo(foo=42)
  # Foo(foo=42)
  Foo(foo=None)
  # Foo(foo=None)
  Foo.schema()
  # {'properties': {'foo': {'title': 'Foo', 'type': ['null', 'integer']}},
  #  'title': 'Foo',
  #  'type': 'object'}
  

Is that right, are those the easiest workarounds, and does the above cover all interesting cases?

(Cue OpenAPI users bickering that arrays in type will only be supported in 3.1+ while 3.0 uses a private extension "nullable": true.)

[lsorber]

@yurikhan How would you implement "a field that may be omitted, but may not be null if the field is supplied"?

class Foo(BaseModel):
    foo: int = Field(default=None)

Foo()  # OK: Don't expect to see a ValidationError because only supplied fields are validated
Foo(foo=None)  # Not OK: Expect to see a ValidationError because `foo` is not an `int`, but no error raised

[PrettyWood]

Hello @lsorber
One easy way is just to validate set values but not default one (so we don't use always=True in the validator)

class Foo(BaseModel):
    foo: int = Field(default=None)

    @validator('foo')
    def set_value_not_none(cls, v):
        assert v is not None
        return v

But the field is still set in the model. If we actually want the field not to be set at all it requires more work and probably use a custom Undefined class / object instead of None

[lsorber]

Thanks for the quick response @PrettyWood! I was heading the same direction with a validator-based workaround, but just submitted an issue (https://github.com/samuelcolvin/pydantic/issues/1761) with a different validator-less workaround if you're interested.

[yurikhan]

@lsorber As far as I understand, in the current version a validator is the way to go. It is a minor annoyance that the field behaves as if set explicitly, but, because it cannot validly be None, testing for foo is None works.

In some places in my team’s code that doesn’t use Pydantic (yet?), we use a different sentinel value to represent missing keys.