Yii2: Serialize deeply nested relations

[HriBB]

How can we serialize deeply nested relations?

I have a REST endpoint which you can call like this:

/api/locations?expand=types,files,locations,locations.types,locations.files 

Note: in my case Location model hasMany child Locations, hence the expand 'locations' on itself.

This is my controller actions() function:

public function actions()
{
    $actions = parent::actions();

    $actions['view']['findModel'] = function($id, $action) {
        $modelClass = $this->modelClass;

        $expand = $this->module->request->get('expand');
        $relations = preg_split('/\s*,\s*/', $expand, -1, PREG_SPLIT_NO_EMPTY);

        if (empty($relations)) {
            $model = $modelClass::findOne($id);
        } else {
            $model = $modelClass::find()->with($relations)->where(['id' => $id])->one();
        }

        if (isset($model)) {
            return $model;
        } else {
            throw new NotFoundHttpException("Object not found: $id");
        }
    };
}

If I set the breakpoint on 'return $model' I can see that all required relations and nested relations are populated.
I want to serialize everything into one big json, but the problem is that serializer does not understand nested relations beyond first level.
Location types, files and locations are successfully serialized, but locations.types and locations.files are not.

Model->toArray() calls ArrayHelper::toArray(), which runs a recursive loop and finally calls $model->toArray() again, but this time without the $expand parameter, which is needed to serialize relations.

At the moment I am overriding toArray() method on Location model:

public function toArray(array $fields = [], array $expand = [], $recursive = true)
{
    $data = [];

    foreach ($this->resolveFields($fields, $expand) as $field => $definition) {
        $data[$field] = is_string($definition) ? $this->$definition : call_user_func($definition, $this, $field);
    }

    if ($this instanceof Linkable) {
        $data['_links'] = Link::serialize($this->getLinks());
    }

    // start override
    $relations = [];

    // construct relation graph
    foreach ($expand as $relation) {
        if (strstr($relation, '.')) {
            list($relation,$child) = explode('.', $relation);
            $relations[$relation][] = $child;
        } else if (!isset($relations[$relation])) {
            $relations[$relation] = [];
        }
    }

    // serialize relations
    foreach ($relations as $relation => $children) {
        if (isset($data[$relation])) {
            $rel = $data[$relation];
            if (is_array($rel)) {
                foreach ($rel as $k => $v) {
                    $data[$relation][$k] = $v->toArray([], $children);
                }
            } else if (is_object($rel)) {
                $data[$relation] = $rel->toArray([], $children);
            } else {
                $data[$relation] = ArrayHelper::toArray($rel);
            }
        }
    }

    return $data;
}

It would be cool if rest Serializer would support this by default ;)

[ptheofan]

+1 for this one!

[fernandezekiel]

this will however make our toArray() have evaluations specific for fields that are relations, i don't know if that responsibility deserves to be there

[mdmunir]

:+1: Up

// ArrayableTrait
    public function toArray(array $fields = [], array $expand = [], $recursive = true)
    {
        $data = [];
        foreach ($this->resolveFields($fields, $expand) as $field => $definition) {
            $data[$field] = is_string($definition) ? $this->$definition : call_user_func($definition, $this, $field);
        }

        $expandRelation = [];
        foreach ($expand as $field) {
            if (($pos=  strpos($relation, '.'))!==false) {
                $relation = substr($string, 0, $pos);
                $child = substr($field, $pos+1);
                $expandRelation[$relation][] = $child;
            }
        }

        if ($this instanceof Linkable) {
            $data['_links'] = Link::serialize($this->getLinks());
        }

        return $recursive ? ArrayHelper::toArray($data,[],true,$expandRelation) : $data;
    }


// ArrayHelper
    public static function toArray($object, $properties = [], $recursive = true, $expandRelation=[])
    {
        if (is_array($object)) {
            if ($recursive) {
                foreach ($object as $key => $value) {
                    if (is_array($value) || is_object($value)) {
                        $expand = isset($expandRelation[$key])?$expandRelation[$key]:[];
                        $object[$key] = static::toArray($value, $properties, true, $expand);
                    }
                }
            }

            return $object;
        } elseif (is_object($object)) {
            ...
            ...
            if ($object instanceof Arrayable) {
                $result = $object->toArray([],$expandRelation);
            } else {
            ....

[martingeorg]

+1 from me too :]

[akaNightmare]

+1

[INHack20]

+1 +1 +1
Please integrate in core http://jmsyst.com/libs/serializer/master/usage

[cleippi]

+1 I'd love this too

[mdmunir]

I do it for my self

• Change yii\base\ArrayableTrait:

trait ArrayableTrait
{
    public function fields()
    {
        $fields = array_keys(Yii::getObjectVars($this));
        return array_combine($fields, $fields);
    }

    public function extraFields()
    {
        return [];
    }

    public function toArray(array $fields = [], array $expand = [], $recursive = true)
    {
        $data = [];
        foreach ($this->resolveFields($fields, $expand) as $field => $definition) {
            $data[$field] = is_string($definition) ? $this->$definition : call_user_func($definition, $this, $field);
        }

        if ($this instanceof Linkable) {
            $data['_links'] = Link::serialize($this->getLinks());
        }

        $expands = $this->expand($expand);
        return $recursive ? ArrayHelper::toArray($data, [], true, $expands) : $data;
    }

    protected function expand(array $expand) // <-- add this
    {
        $expands = [];
        foreach ($expand as $field) {
            $fields = explode('.', $field,2);
            $expands[$fields[0]][] = isset($fields[1]) ? $fields[1] : false;
        }
        foreach ($expands as $key => $value) {
            $expands[$key] = array_filter($value);
        }
        return $expands;
    }

    protected function resolveFields(array $fields, array $expand)
    {
        $result = [];

        foreach ($this->fields() as $field => $definition) {
            if (is_int($field)) {
                $field = $definition;
            }
            if (empty($fields) || in_array($field, $fields, true)) {
                $result[$field] = $definition;
            }
        }

        if (empty($expand)) {
            return $result;
        }

        $expands = $this->expand($expand);
        foreach ($this->extraFields() as $field => $definition) {
            if (is_int($field)) {
                $field = $definition;
            }
            if (isset($expands[$field])) {
                $result[$field] = $definition;
            }
        }

        return $result;
    }
}

• Override yii\helpers\ArrayHelper

class ArrayHelper extends \yii\helpers\BaseArrayHelper
{
    /**
     * @inheritdoc
     */
    public static function toArray($object, $properties = [], $recursive = true, $expands = [])
    {
        if (is_array($object)) {
            if ($recursive) {
                foreach ($object as $key => $value) {
                    if (is_array($value) || is_object($value)) {
                        if(is_int($key)){
                            $expand = $expands;
                        }elseif (isset ($expands[$key])) {
                            $expand = $expands[$key];
                        }  else {
                            $expand = [];
                        }
                        $object[$key] = static::toArray($value, $properties, true, $expand);
                    }
                }
            }

            return $object;
        } elseif (is_object($object)) {
            if (!empty($properties)) {
                $className = get_class($object);
                if (!empty($properties[$className])) {
                    $result = [];
                    foreach ($properties[$className] as $key => $name) {
                        if (is_int($key)) {
                            $result[$name] = $object->$name;
                        } else {
                            $result[$key] = static::getValue($object, $name);
                        }
                    }

                    return $recursive ? static::toArray($result, $properties) : $result;
                }
            }
            if ($object instanceof Arrayable) {
                $result = $object->toArray([], $expands, $recursive);
            } else {
                $result = [];
                foreach ($object as $key => $value) {
                    $result[$key] = $value;
                }
            }

            return $recursive ? static::toArray($result, [], true, $expands) : $result;
        } else {
            return [$object];
        }
    }
}

Now, in bootstrap.php i do like this:

        Yii::$classMap['yii\base\ArrayableTrait'] = 'path/to/my/ArrayableTrait.php';
        Yii::$classMap['yii\helpers\ArrayHelper'] = 'path/to/my/ArrayHelper.php';

Its work for me

http://dee-app.dev/api/movement/40?expand=items,reference.items.product,reference.items.uom

{
    "id": 40,
    "number": "IM150812.000001",
    "warehouse_id": 2,
    "date": "2015-08-11",
    "type": 10,
    "reff_type": 100,
    "reff_id": 8,
    "description": null,
    "status": 10,
    "trans_value": null,
    "total_invoiced": null,
    "created_at": "2015-08-12 09:15:05",
    "created_by": null,
    "updated_at": "2015-08-12 22:27:09",
    "updated_by": null,
    "nmStatus": "Draft",
    "branch_id": 1,
    "items": [
        {
            "movement_id": 40,
            "product_id": 835,
            "uom_id": 1,
            "qty": 3,
            "item_value": null,
            "trans_value": null
        },
        {
            "movement_id": 40,
            "product_id": 110,
            "uom_id": 1,
            "qty": 2,
            "item_value": null,
            "trans_value": null
        }
    ],
    "reference": {
        "id": 8,
        "number": "PU15.000002",
        "supplier_id": 4,
        "branch_id": 1,
        "date": "2015-01-26",
        "value": 1596000,
        "discount": null,
        "status": 20,
        "created_at": "2015-01-26 11:43:24",
        "created_by": 2,
        "updated_at": "2015-02-02 07:56:51",
        "updated_by": 2,
        "nmStatus": "Process",
        "items": [
            {
                "purchase_id": 8,
                "product_id": 835,
                "uom_id": 1,
                "qty": 12,
                "price": 73000,
                "discount": null,
                "total_receive": 12,
                "product": {
                    "id": 835,
                    "group_id": 12,
                    "category_id": 5,
                    "code": "7179675280710",
                    "name": "Kaos krah Salur WRN 74",
                    "status": 10,
                    "created_at": "2014-11-18 07:59:26",
                    "created_by": null,
                    "updated_at": "2014-11-18 07:59:26",
                    "updated_by": null
                },
                "uom": {
                    "id": 1,
                    "code": "Pcs",
                    "name": "Pieces",
                    "created_at": "2014-11-18 07:59:45",
                    "created_by": null,
                    "updated_at": "2014-11-18 07:59:45",
                    "updated_by": null
                }
            },
            {
                "purchase_id": 8,
                "product_id": 110,
                "uom_id": 1,
                "qty": 4,
                "price": 180000,
                "discount": null,
                "total_receive": 4,
                "product": {
                    "id": 110,
                    "group_id": 9,
                    "category_id": 1,
                    "code": "3724516640662",
                    "name": "Celana Pjg Jeans Onethree",
                    "status": 10,
                    "created_at": "2014-11-18 07:59:23",
                    "created_by": null,
                    "updated_at": "2014-11-18 07:59:23",
                    "updated_by": null
                },
                "uom": {
                    "id": 1,
                    "code": "Pcs",
                    "name": "Pieces",
                    "created_at": "2014-11-18 07:59:45",
                    "created_by": null,
                    "updated_at": "2014-11-18 07:59:45",
                    "updated_by": null
                }
            }
        ]
    }
}

[mdmunir]

Should i submit PR for this?

[sasha-x]

Thanks to @mdmunir ! It works.
+1 to merge feature to core.

[ghost]

Is this functionality already added to framework yiihelpersArrayHelper?

[sandinosaso]

Thanks @mdmunir

+1 to merge feature to core.

[rad8329]

Thanks @mdmunir
+1

[kazeno]

@HriBB I can't find enough words to thank you for a usable solution that does not require modifying the core Yii2 files. Just a quick observation for anyone else who ends up using it: you should also check if the first call to the toArray() method is made on an object, as it can throw an error otherwise. So I would change the following code snippet:

foreach ($rel as $k => $v) {
    $data[$relation][$k] = $v->toArray([], $children);
}

to this:

foreach ($rel as $k => $v) {
    if (is_object($v))
        $data[$relation][$k] = $v->toArray([], $children);
}

[Nemmo]

+1

[mramdhani]

cool +1

[supergithubo]

@HriBB Nice work! Any solution deeper than 2 levels? /?expand=item.product.category for instance.

[INHack20]

Hello, the best solution to serialize objects in php is http://jmsyst.com/libs/serializer, I implemented and works perfecion in all respects with yii2. Based on https://github.com/korotovsky/yii2-jms-serializer and updated in https://github.com/Tecnocreaciones/yii2-jms-serializer.

[sasha-x]

@supergithubo : @mdmunir 's solution works for any level.

[whyyk7]

+1

[slow-loris]

+1

[ivanmfan]

+1

[michaelarnauts]

14219 works fine for us. We use it in combination with a patch for yii\rest\Serializer that automatically fills in all loaded relations when passing the model to toArray(). (I can create a PR for that when #14219 gets merged, since it depends on that PR).

This makes it possible to do the following when using it in a yii\rest\Controller to automatically pass the relations as JSON.

    public function actionGetData()
    {
        return Service::find()
            ->with(['serviceItems', 'serviceItems.servicePrices'])
            ->all();
    }

[mikolajzieba]

@michaelarnauts Could you post your solution for yii\rest\Serializer ?