Marshmallow: Defining fields programmatically

[lafrech]

I have a list of field names as strings

FIELDS = ('field_1', 'field_2',..., 'field_42')

and I would like to create this schema programmatically:

class MySchema(Schema):

    field_1 = Float()
    field_2 = Float()
    ...
    field_42 = Float()

I managed to do that with a custom metaclass in which one can specify lists of dynamic fields. Each element of the list is a tuple containing both a list of field names and a Field specification (a dict containing field class + optional args and kwargs).

    class DynamicSchemaOpts(SchemaOpts):

        def __init__(self, meta):
            super().__init__(meta)
            self.auto_fields = getattr(meta, 'auto_fields', [])


    class DynamicSchemaMeta(SchemaMeta):

        @classmethod
        def get_declared_fields(mcs, klass, cls_fields, inherited_fields, dict_cls):

            fields = super().get_declared_fields(klass, cls_fields, inherited_fields, dict_cls)

            for auto_field_list in klass.opts.auto_fields:
                field_names, field = auto_field_list
                field_cls = field['cls']
                field_args = field.get('args', [])
                field_kwargs = field.get('kwargs', {})
                for field_name in field_names:
                    fields.update({field_name: field_cls(*field_args, **field_kwargs)})

            return fields


    class MySchema(Schema, metaclass=DynamicSchemaMeta):

        OPTIONS_CLASS = DynamicSchemaOpts

        class Meta:
            strict = True
            auto_fields = [
                (FIELDS,
                 {'cls': Float}),
            ]

See this SO question for the whole story and an example using a field with args and kwargs.

It seems to work.

I'd appreciate some feedback. Is this the right way to address this issue?

The code would be simpler if I could pass a field instance (especially for fields with args and kwargs)

class Meta:
    strict = True
    auto_fields = [
        (FIELDS, Float()),
    ]

but then all the fields in FIELDS would share the same Nested field instance. Could this be an issue? I peaked into the fields in Marshmallow and didn't identify any use case that would break, but it doesn't look like a good idea. A custom field could store some information in its instance that is not meant to be shared. I'd appreciate feedback about this as well.

I thought that would be a Python issue with a generic Python solution, but what I did is specific to Marshmallow. Would it make sense to add this feature to the library?

[maximkulkin]

What happened to plain old metaprogramming?!

MySchema = type('MySchema', (marshmallow.Schema,), {
    attr: marshmallow.fields.Float()
    for attr in FIELDS
})

You can even have different types of fields there:

fields = {}
fields['foo'] = marshmallow.fields.Float()
fields['bar'] = marshmallow.fields.String()
MySchema = type('MySchema', (marshmallow.Schema,), fields)

or as a base for your customizations:

class MySchema(type('_MySchema', (marshmallow.Schema,), fields)):
    @marshmallow.post_dump
    def update_something(self, data):
        pass

[justanr]

Adding to that, with Python 3 you can even pass keyword arguments to metaclasses. With 3.6 __init_subclass__ was introduced which isn't entirely helpful in this case (it's run after the class has been defined and ergo all the fields have been sucked up) but it's helpful to be aware of it as passing keyword arguments to a class constructor will break with an obscure-ish error: TypeError: __init_subclass__() takes no keyword arguments if that hook does not exist in the hierarchy after the current class but before object, there is not a base that discards all keyword arguments (e.g. it accepts **kwargs but does not pass them alone) OR a base that simply does not call super().__init_subclass__ (e.g. it's similar to __init_subclass__ = classmethod(lambda cls, **kwargs: None)

[lafrech]

Thank you guys for your answers.

This is what I'm doing now:

# Create MySchema programmatically knowing the list of fields
MySchema = type(
    'MySchema',
    (Schema,),
    {field: ma.fields.Nested(MyEmbeddedSchema)
     for field in FIELDS}
)

It works great and it is the Python-only solution I was looking for before engaging into this Marshmallow metaclass mess.

The only thing I miss is how to pass Meta class with strict=True.

Do I need to do this ?

MyDynamicSchema = type(
    'MyDynamicSchema',
    (Schema,),
    {field: ma.fields.Nested(MyEmbeddedSchema)
     for field in FIELDS}
)

class MySchema(MyDynamicSchema):

    class Meta:
        strict = True

or this ?

class Metaclass:
    strict = True

fields = {field: ma.fields.Nested(MyEmbeddedSchema)
             for field in FIELDS}
fields.update({'Meta': Metaclass})

MySchema = type(
    'MySchema',
    (Schema,),
    fields
)

or can I pass this metaclass somewhere? I'm a bit confused about how this Meta class works, and how it is inherited, so I put it in every Schema but maybe I should put it in a parent Schema somewhere.

[justanr]

The Meta field will be inherited like any other attribute. If it's not present, look for it in the MRO. If it is present, then use that value instead.

If you need to inherit some of the parent's configuration and override some, use inheritance on the Meta.

class ParentSchema(Schema):
    class Meta:
        strict = True

Now any schema that inherits from this schema and doesn't provide its own Meta will be strict.

class ChildSchema(ParentSchema):
    class Meta(ParentSchema.Meta):
        ordered = True

This schema is both strict and ordered because it explicitly inherits the parent's Meta class and added new attributes to it, similar to inheriting a regular class and adding our overriding attributes.

As a note, ordered looks like it's "infectious." If any parent schema is ordered, yours is too even if you set ordered=False. I'm unaware of the motivations for this so I can't say if it's a bug or not but be aware.

[lafrech]

Alright. So what I can do in my application is create a StrictSchema for all Schemas to inherit:

class StrictSchema(Schema):
    """Custom Marshmallow Schema with strict=True in Meta"""

    class Meta:
        """Base Meta class with strict=True

        To inherit in child Schema:

        class ChildSchema(ParentSchema):
            class Meta(ParentSchema.Meta):
                ordered = True
        """
        strict = True

then use it as a base here

# Create MySchema programmatically knowing the list of fields
MySchema = type(
    'MySchema',
    (StrictSchema,),  # <--- StrictSchema
    {field: ma.fields.Nested(MyEmbeddedSchema)
     for field in FIELDS}
)

And if I need to override Meta:

class MySuperSchema(StrictSchema):
    class Meta(StrictSchema.Meta):
        ordered = True

# Create MySchema programmatically knowing the list of fields
MySchema = type(
    'MySchema',
    (MySuperSchema,),
    {field: ma.fields.Nested(MyEmbeddedSchema)
     for field in FIELDS}
)

Thanks to you both.

[maximkulkin]

You could also do

MySchema = type(
    'MySchema',
    (marshmallow.Schema,),
    dict({
        field: ma.fields.Nested(MyEmbeddedSchema)
        for field in FIELDS
    }, Meta=type('Meta', tuple(), {'strict': True})
)

[luisincrespo]

What would you recommend if I want to extend the schema with a given field based on a specific condition, but keep all other fields intact?

[maximkulkin]

@luisincrespo I though it would be obvious when I showed this snippet:

fields = {}
fields['foo'] = marshmallow.fields.Float()
fields['bar'] = marshmallow.fields.String()
MySchema = type('MySchema', (marshmallow.Schema,), fields)

You can put an "If" statement in front of any line and thus make it conditional:

fields = {}
fields['base'] = marshmallow.Integer()
if foo_condition:
    fields['foo'] = marshmallow.fields.Float()
if bar_condition:
    fields['bar'] = marshmallow.fields.String()
MySchema = type('MySchema', (marshmallow.Schema,), fields)

[luisincrespo]

Thanks @maximkulkin but let me be clear. I have an existing Schema that inherits from marshmallow.Schema, and has its own fields. If I do this:

MySchema = type('MySchema', (marshmallow.MyParentSchema,), fields)

I'm not sure if it'll overwrite/ignore the existing fields from MyParentSchema.

[maximkulkin]

It will inherit fields from parent schema unless you override them in your fields dictionary.

[luisincrespo]

Perfect, thanks!

On Tue, Feb 21, 2017, 5:55 PM Maxim Kulkin [e-mail] wrote:

It will inherit fields from parent schema unless you override them in your
fields dictionary.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/marshmallow-code/marshmallow/issues/585#issuecomment-281509439,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEPycHiR-3LuMYdKUF4kZuM2EyTaLvtlks5re2tMgaJpZM4MBH8U
.