Marshmallow: Improve "required" validation; allow_none and allow_blank

[sloria]

Currently, setting required=True on Field only returns an error if the field's value is missing from the input data.

class UserSchema(Schema):
    name = fields.Str(required=True)

s = UserSchema()
s.validate({})  # {'name': ['Missing data for required field.']}
s.validate({'name': None})  #  No errors
s.validate({'name': ''})  # No errors

This may not meet the typical use case, where you want an error returned for None and the empty string (with the option to allow these values).

Solution

By default, return an error if None or '' are passed to a required field. Add allow_none and allow_blank options to allow these values to pass validation.

UserSchema().validate({'name': None})  #  {'name': ['Missing data for required field.']}

class LenientSchema(Schema):
    name = fields.Str(required=True, allow_none=True)

LenientSchema().validate({'name': None})  # No errors

Note: This idea was yoinked from Django Rest Framework. =)

[vesauimonen]

Not sure if this is related to this issue or a problem of its own. Consider the following example:

from marshmallow import fields, Schema


class User(object):
    def __init__(self, name, activated_at=None):
        self.name = name
        self.activated_at = activated_at


class UserSchema(Schema):
    name = fields.String(required=True)
    activated_at = fields.DateTime(required=False)


user = User(name='John')

s = UserSchema()

serialized_result, errors = s.dump(user)

print(errors)  # {}

deserialized_result, errors = s.load(serialized_result) 

print(errors)  # {'activated_at': ['Could not deserialize None to a datetime object.']}

I'd assume that a field would accept None value in deserialization, if the field is required=False. At least for fields.DateTime this is not the case. As you see in my example, this breaks the serialization -> deserialization process quite badly.

Am I misusing the API somehow or is this a real issue?

[sloria]

@vesauimonen The behavior in your example is expected and correct.

You are passing the value None for the the activated_at field (in serialized_result). None is not a valid datetime, so you get an error. required validation checks the _existence_ of a field in the input dict; None is not a special case.

[sloria]

Resolved by https://github.com/marshmallow-code/marshmallow/commit/a1308a3555f00ce8a29863d3d37bd925a237c479

[sloria]

@vesauimonen An update on this issue: I've added the allow_none parameter to field classes, which makes validation of None consistent across fields.

With a slight modification of your code above, validation works as expected.

from marshmallow import fields, Schema

class User(object):
    def __init__(self, name, activated_at=None):
        self.name = name
        self.activated_at = activated_at


class UserSchema(Schema):
    name = fields.String(required=True)
    activated_at = fields.DateTime(allow_none=True)


user = User(name='John')

s = UserSchema()

serialized_result, errors = s.dump(user)
# no errors
assert errors == {}

deserialized_result, errors = s.load(serialized_result) 
# no errors
assert errors == {}

[vesauimonen]

@sloria awesome :+1:

[cheburakshu]

@vesauimonen No errors reported when user field is all blank

```

from marshmallow import fields, Schema

class User(object):
... def __init__(self, name, activated_at=None):
... self.name = name
... self.activated_at = activated_at
...
class UserSchema(Schema):
... name = fields.String(required=True)
... activated_at = fields.DateTime(allow_none=True)
...
user = User(name='John')
s = UserSchema()
serialized_result, errors = s.dump(user)
assert errors == {}
user = User(name=' ') # I am a blank user
s = UserSchema()
serialized_result, errors = s.dump(user)
errors
{}

[sloria]

@cheburakshu Validation is only done on deserialization (calling .load), not on serialization.

[loic001]

@sloria Why ? Not useful ?

[deckar01]

@loic001 It is as standard marshalling design to load unstructured data and dump structured data. The schema should be designed to match the data structure you want your application to work with. Any data that does not comply with that structure needs to be loaded. Any data that does comply with that structure can be dumped.

[loic001]

Quick and efficient response! Thanks ;)

[vvksahoo]

class UserSchema(Schema):
name = fields.Str(required=True,allow_none=False)

s = UserSchema()
s.validate({}) # {'name': ['Missing data for required field.']}
s.validate({'name': None}) # No errors
s.validate({'name': ''})

In these all three case i want to get exception but i am not getting . How to solve it?
I have more number of field , so cant check every field has some value or not. Before i am making database call i want to make sure every required field has some data with them.

[LarryWachira]

A potential solution @vvksahoo would be adding custom validation via a function, for example, that checks no empty strings are allowed.