Yii2: API: Searching and filtering

[cebe]

follow up on #303

[samdark]

A common way filtering is achieved is by passing GET parameters like the following:

`/users?active=true&role=admin`

Searching is usually done like this:

`/users?q=alex%20-makarov`

The query is passed to SOLR, Sphinx or another search engine.

[Ragazzo]

also should implement related search and filtering user/2/posts, user/2/posts/5 and so on

[Ragazzo]

Could be useful too i think https://github.com/interagent/http-api-design

[fernandezekiel]

@Ragazzo yeah, i was thinking if posts will just have to have a PostController dedicated and be forwarded with the proper params, so users/2/posts would translate to posts?user_id=2

[Ragazzo]

@fernandezekiel could be as one of option to use Yii::$app->run('/posts',['id' => 2]), but we should be sure that second resource is created, however almost always it will be created, so yeah, could be an option to use )

[fernandezekiel]

yeah in this solution though
POST user/2/notes -> note?user_id=2
POST contact/3/notes -> note?contact_id=3

now the note controller will have to check in the query string parameter which of the query string params are passed and it will be created depending if it's a user_id or a contact_id,

so i was thinking if it's a viable option also to instead of having a NoteController, is that the UserController itself will have actions that will actually interact with /users/2/notes directly

[Ragazzo]

is that the UserController itself will have actions that will actually interact with /users/2/notes directly

i am not sure, i think request forwarding is fine here, however it is not true request forwarding since some GET and POST and other params are not separated, but since it is REST so for every resource it is should be some endpoint that should deal with it (including possible relations user_id , contact_id params in your case). So yes, it should be in NoteController and not in UserController

[fernandezekiel]

will the filtering part be something similar on what we do in our default CRUD where we create an extending UserSearch class (dedicated for handling the filtering parmaeters)

[githubjeka]

Filtering and searching can be done with the built-in functionality. It is individual for each app..

Filtering is implemented using $prepareDataProvider from IndexAction
Searching is implemented using ViewAction

I think it can be closed. Or do you want CMS? :)

[fernandezekiel]

i wouldn't mind if it would be closed, it'd be hard to make the filtering logic universal anyway

[Ragazzo]

fine with closing if there will be some good examples on how to do it

[fernandezekiel]

what i did so far is that my resource representation of models implements an interface Filterable,

interface Filterable
{
    const SCENARIO = 'filter';
    /**
     * @param \yii\db\ActiveQuery $query
     * @param array $params
     * @return \yii\db\ActiveQuery
     */
    public function search($query, $params);
}

my resource models (which are under api/modules/v1/models) will implement this and the IndexAction will have to evaluate if the obeject implements this interface,

also for startups, i also made a trait called Filtering

trait Filtering
{

    public function rules()
    {
        if ($this->getScenario() === Filterable::SCENARIO) {
            return [[array_keys($this->getAttributes()), 'safe']];
        } else {
            return parent::rules();
        }
    }

    public function scenarios()
    {
        $scenarios = parent::scenarios();
        $scenarios[Filterable::SCENARIO]  = array_keys($this->getAttributes());

        return $scenarios;
    }

    /**
     * @param \yii\db\ActiveQuery $query
     * @param array $params
     * @return \yii\db\ActiveQuery
     */
    public function search($query, $params = [])
    {
        $this->unsetAttributes();

        $this->setAttributes($params);

        $attributes = array_keys($this->getAttributes());

        $tableName = static::tableName();
        foreach($attributes as $attribute) {
            $query->andFilterWhere(['like', "$tableName.$attribute", $this->{$attribute}]);
        }

        return $query;
    }

    public function unsetAttributes()
    {
        $attributes = array_keys($this->getAttributes());

        foreach($attributes as $attribute) {
            $this->setAttribute($attribute, null);
        }
    }

    public static function searchForm()
    {
        $form = new static;
        $form->setScenario(Filterable::SCENARIO);

        return $form;
    }

}

this will enable you to filter all of your attributes, override the search() function if you want to change some of the filter logic

example of the prepareDataProvider:

$modelReference = new $this->modelClass;

$prepareDataProvider = function ($action) use ($modelReference){
    $query = call_user_func([$this->modelClass, 'find']);
    $params = Yii::$app->request->getQueryParams();

    $query = $this->applyYourCustomConstrainHere($query);
    if ($modelReference instanceof Filterable) {
        $modelReference->scenario = Filterable::SCENARIO;
        $query = $modelReference->search($query, $params);
    }

    return new ActiveDataProvider([
        'query' => $query,
        'pagination' => $this->getPagination(),
        'sort' => $this->getSort()
    ]);
}

[githubjeka]

I made a simple and specific. - https://github.com/githubjeka/yii2-rest/blob/master/rest/versions/v1/controllers/PostController.php#L27

[Ragazzo]

i think it can be adjusted , also why you creating model instance since it is not needed

[githubjeka]

:) new should be del ... thank

[fernandezekiel]

@Ragazzo sorry aobut that, yeah i suppose we can perform class_implements to the $this->modelClass instead of instanceof to the model instance i made,

[fernandezekiel]

anyway that's the general idea

[Ragazzo]

@fernandezekiel my comment was to @githubjeka )

[fernandezekiel]

the design however seems to promote creating a subclass that extending the base models

namespace api\modules\v1\models;

use api\modules\v1\components\Filterable;
use api\modules\v1\components\Filtering;

class Contact extends \common\models\Contact implements Filterable
{
    use Filtering;
} 

[githubjeka]

@Ragazzo I copy this code from @fernandezekiel )

[Ragazzo]

@fernandezekiel yes , it is a valid way , since it is different context , we should not mix functionality and roles of different contexts in one entity

[fernandezekiel]

alright ,i'm pretty sure my code could use a lot more improvements :) thanks

[tunecino]

Why not involve the DataProvider of the ModelSearch Class generated by Gii in this instead of building more DataProviders ? Why user will expect searching enabled if he didn't create one for the model ?

If you do this in a simple activeController :

public $serializer = [
    'class' => 'yii\rest\Serializer',
    'collectionEnvelope' => 'images',
];

public function actions() {
    $actions = parent::actions();
    $actions['index']['prepareDataProvider'] = [$this, 'prepareDataProvider'];
    return $actions;
}

public function prepareDataProvider() {
    $searchModel = new app\models\ImageSearch();    
    return $searchModel->search(\Yii::$app->request->queryParams);
}

by doing GET to :

/images?ImageSearch[title]=football&ImageSearch[status]=1 

you'll get filtered, paginated & serialized resources.

It may not be elegant as solution if used this way but what I mean is : if we already have a ModelSearch class where model logic most be handled then why not figure out a cleaner way to upgrade & reuse it ?

[tunecino]

This is how I added filtering support within Controller :

namespace app\api\modules\v1\controllers;

use yii\rest\ActiveController;
use yii\helpers\ArrayHelper;
use yii\web\BadRequestHttpException;

class TagController extends ActiveController
{
    public $modelClass = 'app\models\Tag';
    // Some reserved attributes like 'q' for searching 
    // or 'sort' which is already supported by Yii RESTful API
    public $reservedParams = ['sort','q'];

    public function actions() {
        $actions = parent::actions();
        // 'prepareDataProvider' is the only function that need to be overridden here
        $actions['index']['prepareDataProvider'] = [$this, 'indexDataProvider'];
        return $actions;
    }

    public function indexDataProvider() {
        $params = \Yii::$app->request->queryParams;

        $model = new $this->modelClass;
        // I'm using yii\base\Model::getAttributes() here
        // In a real app I'd rather properly assign 
        // $model->scenario then use $model->safeAttributes() instead
        $modelAttr = $model->attributes;

        // this will hold filtering attrs pairs ( 'name' => 'value' )
        $search = [];

        if (!empty($params)) {
            foreach ($params as $key => $value) {
                // In case if you don't want to allow wired requests
                // holding 'objects', 'arrays' or 'resources'
                if(!is_scalar($key) or !is_scalar($value)) {
                    throw new BadRequestHttpException('Bad Request');
                }
                // if the attr name is not a reserved Keyword like 'q' or 'sort' and 
                // is matching one of models attributes then we need it to filter results
                if (!in_array(strtolower($key), $this->reservedParams) 
                    && ArrayHelper::keyExists($key, $modelAttr, false)) {
                    $search[$key] = $value;
                }
            }
        }

        // you may implement and return your 'ActiveDataProvider' instance here.
        // in my case I prefer using the built in Search Class generated by Gii which is already 
        // performing validation and using 'like' whenever the attr is expecting a 'string' value.
        $searchByAttr['TagSearch'] = $search;
        $searchModel = new \app\models\TagSearch();    
        return $searchModel->search($searchByAttr);     
    }
}

Now my API URLs are looking like :

/tags?name=php&status=active&sort=id,-name 

The bonus I got: this will make building Nested resources much easier by configuring routes like https://github.com/yiisoft/yii2/issues/9474#issuecomment-141713074 and as explained here : https://github.com/yiisoft/yii2/issues/9474#issuecomment-141923415.

Now about searching within the reserved q param, In my opinion it will be hard to figure out a standard use case. It may deffer from a project to another. There is cases when you need to search all resource fields (or pre-selected fields) at once. There is other cases when you need to search also related model fields like in this example :

/apartments?q=petersburg

will return 0 results if apartment location is stored in a related belongTo model called _address_ or _location_. In that case I'll probably make the above URL return the same results as doing :

/apartments/location?q=petersburg

There is also more complex cases like when making searching global to return results related to all api resources or/and build indexation for it or/and group resources within categories like when searching q=xyw in _amazon.com_ you'll get _few suggested results_ + xyz IN BOOKS + xyz IN SALES + ...

[fernandezekiel]

we can always have a 'default' search implementation wherein it would perform a = comparisson against filter attributes...
however i like also having the ability to create your own search function just like how Yii does with its CRUD search models (it doesn't have to be a separate model maybe?)

[Faryshta]

How about a new option searchClass for the yii\rest\IndexAction class?

If that is empty it will be have as it does right now. Other wise class will be used to create a search model and the dataprovider. Its extremely useful specially if you want to return errors with your searchs.

Example: the url stores?lng=200&lat=200&radius=-10 returns http status 422

[
    {
        "field": "lat",
        "message": "lat can't be bigger than 180"
    },
    {
        "field": "lng",
        "message": "lat can't be bigger than 90"
    }
    {
        "field": "radius",
        "message": "radius can't be smaller than 0"
    }
]

[tunecino]

yes @Faryshta I think that is the missing thing. the searchClass already generated by gii is a great place to perform and maintain complex logic that may involve custom validation rules and scenarios so I totally agree with adding the optional choice of directly involving it in IndexAction the same way it is done with gridviews.

[tunecino]

@Faryshta will you do a PR ? or should I start ?

[fernandezekiel]

since we're already in this issue... i think that we can probably implement a way for the controller to actually modify the query object of the Index/View/Update/Delete without having to create its own actionIndex() override...
for example, add something like IndexAction.modifyQuery function that can be a callback

so that instead something like a UserController can just add a condition to the query like this:

public function modifyQuery(QueryInterface &$query)
{
    $query->andWhere(['created_by' => Yii::$app->user->id]);
}

this is the base feature that i think we can build on for the Searching feature

[tunecino]

@fernandezekiel I think that can be easily handled within a searchClass. it is about a returned ActiveDataProvider instance after all so IndexAction will just return it. it can be used to filter with related models like in this example. custom rules, validations and scenarios may also help to solve more advanced cases.

[niksamokhvalov]

A common way filtering is achieved is by passing GET parameters like the following:
/users?active=true&role=admin

I think that is wrong way. Names of fields can match with query parameters. I suggest used parameter ?filter in format JSON:

/api/v1/users?filter={"field": "value"}

Operators

• !=
• >
• <
• <=
• >=
• % (search by substring)

Settings
Configure a filter rule can be via ActiveDataProvider. For example:

return new ActiveDataProvider([
    'query' => $modelClass::find(),
    'filter' => [
        'attributes' => [
            '*', // All attributes are allowed to be filtered,
            '!bio' // but not the attribute "bio"
        ]
    ]
]);

Examples

/api/v1/users?filter={"id": [1,6,46,8], "active": 1)
/api/v1/users?filter={"!=id": 13, "age": null)
/api/v1/users?filter={"email": "[email protected])
/api/v1/users?filter={">=age": 18, "<age": 30, "gender": "w", "%name": "Natalie")

What do you say? I have this implementation of the filter, ready to send PR.

[darrencoutts118]

Sounds good Nik - do you have your filter hosted so I could take a look?

[samdark]

I'm not sure there's a conflict between filtering and parameters. Would be great to get use cases for that.

Substring search could be %something, something% or %something% at least. How do you plan to deal with that?

[niksamokhvalov]

Example of conflict. User model has fields:

• id
• name
• sort

How to filter on a field sort, if it ?sort parameter simultaneously set the sorting and is an attribute of the model?

Substring search could be %something, something% or %something% at least. How do you plan to deal with that?

Here it is:

/api/v1/users?filter={"%name": "Natalie")
/api/v1/users?filter={"name%": "Natalie")
/api/v1/users?filter={"%name%": "Natalie")

@darrencoutts118 unfortunately, it not public project. If the concept of filtering ideal for Yii, I will do PR.

[samdark]

Yes, indeed if model contains sort or limit fields, there's a conflict...

[samdark]

There's no way to prevent such conflict so the only way is to namespace it as @niksamokhvalov proposed. JSON is a good choice for value since there's no need to introduce special rules for separator escaping and there are means to parse/form it in every modern language.

The only thing I don't like is about LIKE matching. Name of the field should not be changed as proposed above. Instead, value should. Value isn't necessary a string so we can introduce something to specify type of the matching for this particular case.

So overall I agree that it's the best approach except, of course, LIKE matching syntax.

[niksamokhvalov]

OK, let use LIKE (%) in the values:

/api/v1/users?filter={"name": "%Natalie")
/api/v1/users?filter={"name": "Natalie%")
/api/v1/users?filter={"name": "%Natalie%")

It really is convenient for the client (frontend).

[samdark]

The issue here is about the meaning. It could be just exact match with 100% or LIKE match for 100%. Values are the same, meaning isn't. I think it worth implementing full match which is default first and then invent a syntax for more complicated match types such as

{
    "name": {
        "type": "like",
        "value": "Natalie%"
    }
}

[fernandezekiel]

it seems a nice syntax so far, not sure about how easy this is to implement for premade client-side frameworks like Emberjs

[samdark]

@fernandezekiel should be OK since URLs aren't hardcoded in Ember and JSON support is native in JavaScript.

[niksamokhvalov]

I made the admin panel in AngularJS that works with RESTful in Yii in the above-described standard. Works perfectly :)

[Faryshta]

@tunecino hi, sorry just saw your response to me.

i don't think it requires a pr, maybe just some doc. Wanna see a code example?

[tunecino]

@Faryshta thanks but that was before I made that PR for it: https://github.com/yiisoft/yii2/pull/11470. I'm already using a search class into my own code. Now I think the default solution to filtering will be this: https://github.com/yiisoft/yii2/pull/12251.

[tunecino]

@Faryshta I keep thinking that involving a search class is a good alternative for some specific cases (see https://github.com/yiisoft/yii2/pull/11470#issuecomment-239673434). I think that could be good for a cookbook article.

this is how I personally use it:
http://stackoverflow.com/questions/25522462/yii2-rest-query/30560912#30560912
and when throwing errors is required, simply returning the model itself instead of the data provider instance from the search class will solve it.

[klimov-paul]

The only acceptable solution I can see here is add a filterModel configuration for yii\rest\IndexAction, which should specify model responsible for the data accepting filtering and data provider composition.
I have implemented this approach for the regular CRUD here:
https://github.com/yii2tech/admin/blob/master/actions/Index.php

[klimov-paul]

Actually there is already IndexAction::prepareDataProvider callback, which can be used for the search feature:

public function actions()
    {
        return [
            'index' => [
                'class' => 'yii\rest\IndexAction',
                'modelClass' => $this->modelClass,
                'prepareDataProvider' => function () {
                    $searchModel = new SearchModel();
                    return $searchModel->search(Yii::$app->request->queryParams);
                },
            ],
        ];
    }

[klimov-paul]

Implementation proposed at #12641

[klimov-paul]

See also https://github.com/yiisoft/yii2/pull/12641#issuecomment-253789966

[Faryshta]

https://github.com/tecnocen-com/yii2-roa/blob/master/src/controllers/OAuth2Resource.php#L168 this was my approach on a library i am working. so far i have used it without any issue.

[samdark]

@Faryshta I did nearly the same thing for YiiPowered recently: https://github.com/samdark/yiipowered/blob/master/modules/api1/controllers/ProjectController.php#L13

[Faryshta]

very alike indeed, only meaningful difference is that https://github.com/samdark/yiipowered/blob/master/modules/api1/models/ProjectSearch.php you don't run a validation of the search params.

[samdark]

Good catch. Actually I think I should run validation :)

[Faryshta]

12641 i started a discussion about rfc's in the hopes to have a self-documented filters.