Hotchocolate: GraphQL Filters

[PascalSenn]

Hello there 👋

We are currently working with graphql.net. We miss some features that are proposed in this library and it seems like it would be a way better fit for our needs. Great work so far!

Anyway, for our graphql.net environment we developed a filter mechanism.
We were highly inspired by the work of prisma.
So that's how it looks like in our api:

You can also filte for nested objects:

 {
    servers(where:{disks{freeSpace_lt:$num}}) {
        principalName
    } 
  }

In graphql.net we define custom graph types that, with some DI magic, resolve the right input types.
So we define apart from ObjectType, MutationType and InputType also a filter type:

 public class SystemFilterType<T> : FilterGraphType<System> 
    {
        public SystemFilterType(IDependencyResolver resolver) : base(resolver)
        {
            FilterAndOrderBy(x => x.PrincipalName);
            FilterAndOrderBy(x => x.NetbiosName);

In our resolver the resolvers we get the arguments from the query and then just apply the filters on a IEnumerable.

     return arguments.Filter(context, result);

So a Filter Graph Type is on one hand a input type ( so defines arguments like name, name_contains, name_in ) and on the other hand applies that filter on a IEnumerable.
A filter polymorphically calls all nested filter.

That's a pretty powerfull tool. We enable user with no programming skills to query our api through graphiql or playground and give them the opportunity to quickly get insights for their demand.

We lift the filter process from the front end to the back end and will even in a later version use IQueryable to directly filter on the database (which hopefully delivers even better performance)

What would be the proper approach to implement something like this on hotchocolate?
Could we use directives?
Would that mean we could reduce the boilerplate for filter (SystemFilterType)?

[michaelstaib]

Funny, we just did something like this for our project.
We are currently working on version 0.7.0 which will allow for more extensibility.

So, I used a field middleware to solve it. We have a basically a database that is queried and returns a queryable which is than picked up by a sorting middleware which applies the filter arguments passed in. The next middleware handles then filtering and so on.

Defining the logic that transform the filter object and applies it to a queryable in a middleware makes it easier to reuse. Basically, in our project we can now say something like this:

protected override void Configure(IObjectTypeDescriptor<SomeObject> descriptor)
{
    descriptor.Field(t => t.SomeFieldReturningIQueryable) 
        .UsePaging<StringType, string>()
        .UseSorting()
        .UseFiltering();
}

The middleware extension methods are also adding the filter arguments that the middleware needs.

Moreover, to get something in place like generic object types you basically could look at out connection type:

 public class ConnectionType<T>
        : ObjectType<IConnection>
        where T : INamedOutputType, new()
{
    protected override void Configure(
        IObjectTypeDescriptor<IConnection> descriptor)
    {
        // TODO : Fix this with the new schema builder
        descriptor.Name($"{new T().Name}Connection");
        descriptor.Description("A connection to a list of items.");

        descriptor.Field(t => t.PageInfo)
            .Name("pageInfo")
            .Description("Information to aid in pagination.")
            .Type<NonNullType<PageInfoType>>();

        descriptor.Field(t => t.Edges)
            .Name("edges")
            .Description("A list of edges.")
            .Type<ListType<NonNullType<EdgeType<T>>>>();
    }

    protected override void OnRegisterDependencies(
        ITypeInitializationContext context)
    {
        base.OnRegisterDependencies(context);

        context.RegisterType(new TypeReference(typeof(T)));
        context.RegisterType(new TypeReference(typeof(EdgeType<T>)));
    }
}

We will make some changes how the schema is created thats why we still have the todo there ;)

But basically you can override OnRegisterDependencies and register the generic type arguments with the schema.

You have the following interfaces to express what you want to allow as type arguments:

• IType
  All types.

• IInputType
  All input types.

• IOutputType
  All input types.

• INamedType
  This basically means all types that have a name and excludes NonNullType and ListType.

• IOutputNamedType
  This basically means all output types that have a name and excludes NonNullType and ListType.

• INamedInputType
  This basically means all input types that have a name and excludes NonNullType and ListType.

The schema in _Hot Chocolate_ is immutable. This means that after the schema is created and completed it cannot be changed.
A type in the schema is created in three steps:

• Create - Instance is created and Configure is called
• Register dependencies - Depending types, resolvers etc are registerd.
• Complete type - The schema builder has completed all types and is injecting the missing parts like referenced types, compiled resolvers and so on.

Does that help you?

If your filtering solution is general enough we could built something like that into a HotChocolate.Types.Filtering package. I think many users would like that.

[michaelstaib]

Would you be willing to work with us to create a package that provides a prisma like filtering solution?

[michaelstaib]

Would that mean we could reduce the boilerplate for filter (SystemFilterType)?

Depends on your implementation. I cannot say that from the code that you have provided. Could you provide more code of the FilterGraphType?

[michaelstaib]

Could we use directives?

Depends on what you want to do. Directives can be powerful. A directive can essentially be used to change the execution behavior of queries. It also can be used to just annotate things.

[PascalSenn]

Would you be willing to work with us to create a package that provides a prisma like filtering solution?

I'm studying part time and work for the other part. I'm interested in the project and could kick it off with a PoC and see where that leads.

Depends on your implementation. I cannot say that from the code that you have provided. Could you provide more code of the FilterGraphType?

I'll give you a overview over what we currently have and how it behaves.

Overview

It's a pretty complex structure.
I try to give an overview.

Firstly, we highly rely on the dotnet core DI and work quite a lot with generics.

1. We define a FilterType

  public class UserFilterType : FilterGraphType<User>
        public UserFilterType (IDependencyResolver accessor) : base (accessor) { 
            ....
            Filter(x => x.Description); 
           ...
        }
    } 

1. We then resolve a object over the DI. Let's call it DefaultResolver.

       protected void Filter<TProcResult>(Type propertyType, Expression<Func<T, TProcResult>> proc, string attributeName)
        {
          .....
                AddSingle(propertyType, proc, attributeName);
        ....

        }

        private void AddSingle<TProcResult>(Type propertyType, Expression<Func<T, TProcResult>> proc, string attributeName)
        {
            var resolverType = typeof(IDefaultResolver<>).MakeGenericType(propertyType);
            var resolver = (IDefaultResolverMethod)this.Resolve(resolverType);
            resolver.Resolve(this, proc, attributeName);
        }

1. The DefaultResolver then resolves a more specific resolver for the generic type. Like StringResolver

public class DefaultResolver<TProcResult> : BaseResolver, IDefaultResolver<TProcResult>, IDefaultResolverMethod
    {
        private IDefaultResolverMethod _resolver;
        public DefaultResolver(IDependencyResolver dependencyResolver) : base(dependencyResolver)
        {
            Type resolverType;
            Type propertyType = typeof(TProcResult);
           .... 
            else if (propertyType == typeof(string))
            {
                resolverType = typeof(IStringResolver<TProcResult>);
            }
           ....
            _resolver = (IDefaultResolverMethod)this.dependencyResolver.Resolve(resolverType);

        }

1. The String Resolver then registers the different input types

  public class StringResolver<TProcResult> :....
    {
        ....
        public void Resolve<T, TResolveProcResult>(QueryFilterType<T> currentFilter, Expression<Func<T, TResolveProcResult>> proc, string attributeName)
        {  
                ...
                AddFilter<T, StringGraphType>(currentFilter, attributeName + "_contains", new StringContains<T>(stringProc, attributeName)); 
               ...
            }
        }
    }

1. This builds the schema of the "where" input type. Apart from being normal input graph types, these types store a predicate and also implement a method ApplyFilters. On a request we read the input types and then use the previously build filters types to execute them.

As the api grows the only thing we do ist adding new FilterTypes (like the UserFilterType above).

It's designed to just work on top of graphql.net so doesn't access any internal functionality. It literally just registers input fields.

Idea

Depends on what you want to do. Directives can be powerful. A directive can essentially be used to change the execution behavior of queries. It also can be used to just annotate things.

I guess interesting would be something like this (not sure if I places the directives right)

 {
    root {
         someObject(@filters) {
            property
            someNestedObject {
                 nestedProperty 
            }
           notFiltered
        } 
     }
}

protected override void Configure(IObjectTypeDescriptor<Root> descriptor)
{
    descriptor.Field(t => t.SomeObject).Directive(new FilterDirective());
}

protected override void Configure(IObjectTypeDescriptor<SomeObject> descriptor)
{
    descriptor.Field(t => t.Property)
        .UseFiltering();
    descriptor.Field(t => t.SomeNestedObject)
        .UseFiltering();
    descriptor.Field(t => t.notFiltered)
}

protected override void Configure(IObjectTypeDescriptor<SomeNestedObject> descriptor)
{
    descriptor.Field(t => t.NestedPropery)
        .UseFiltering();
}

what results in

```graphql 
 {
    root {
         someObjects({
                 property: "prop",
                 property_in:["prop"],
                 ...
                 someNestedObject_some: {
                       nestedProperty: "prop",
                       nestedProperty_in:["prop"],
                       ...
                 }
                 someNestedObject_all: {
                       nestedProperty: "prop",
                       nestedProperty_in:["prop"],
                       ...
                 }
                 someNestedObject_none: {
                       nestedProperty: "prop",
                       nestedProperty_in:["prop"],
                       ...
                 }
            }) {
            property
            someNestedObject {
                 nestedProperty 
            }
           notFiltered
        } 
     }
}

Is it possible to access the schema out of a directive? so could the directive, after the schema has been built, access the meta information (attached by UseFiltering) of the field and build a input type based on the tree?

[michaelstaib]

Ok,

now I get what you want to do. Basically you want to generate input types based on output types.
I will have to think on that one and talk to my colleagues about that.

We are introducing a new schema builder in version 0.8.0 which should be released end of January/beginning of February and I think this one would fit in there.

All types including directives can access the types of the schema during creation.

And I think we can make it much simpler to get this all working.

Give me the weekend to think about all the implications and then we will move this one forward.

[drowhunter]

Could this help?

https://join-monster.readthedocs.io/en/latest/

[michaelstaib]

Also, we have a slack channel if you want to join:
https://join.slack.com/t/hotchocolategraphql/shared_invite/enQtNTA4NjA0ODYwOTQ0LTBkZjNjZWIzMmNlZjQ5MDQyNDNjMmY3NzYzZjgyYTVmZDU2YjVmNDlhNjNlNTk2ZWRiYzIxMTkwYzA4ODA5Yzg

[PascalSenn]

@drowhunter
Definitely an interesting project I'll have a look at. The biggest differenceto the filter idea probably is thaty the filters are input type and are not generated by a graphql document body

@michaelstaib

Exactly, sounds good!
That's exactly what I was hoping to hear.

And I think we can make it much simpler to get this all working.

I'm well know for my overengineering skills 😄

We are introducing a new schema builder in version 0.8.0 which should be released end of January/beginning of February and I think this one would fit in there

Give me the tools and show me the borders, and I'll start building that wall a PoC.

And now for real: sounds good to me. The biggest problem I faced with graphql dotnet was the schema building. Not because the schema generation of graphql.net is bad. It's more related to the way I implemented it in the end. The filter for graphql.net started out as a side project and somehow ended up being efficient enough for production. The high reliability on .net cores DI was also the reason for not publishing it.

I do have a few more ideas for the filters and how they could be improved and also know the limits of what they are capable of.

I have a good feeling about this👌

But first, merry christmas to you all🎅

[drowhunter]

Yeah it would be great if you could just take the graphql query and translate it into an SQL query and execute it. Because the N+1 problem while solved by the dataloader is still a bit clumsy and ineffecient

eg. it still performs a select on all fields even thought the request specifically asks for certain fields.

this Join Monster is for NodeJS so we would just have to port it to c# for it to work.

Anyway I'm super interested in your approach , as right now i have to create specific revolvers to handle the filtering. Do you have a github somewhere with it implemented?

[PascalSenn]

Hm.. well filters are more like the part after there where statement of an SQL query.

The way of using iqueryable would be a very decent solution.

We do work a lot with document DBs and the performance impact isn't that big in our case. (No joins, always full document)

Unfortunately the source is nowhere on GitHub by now

[drowhunter]

Sorry a bit off topic, but i'm a SQL guy but am very curious about DocumentDB like mongo. But never understood what the difference is and why I would want to use it.

You say you always get the full document? Is that how mongo works?

[PascalSenn]

I started a repository as a Playground for the Filters.
Later this will be integrated into the main repository of hot chocolate.
It easier to develop in a test project. It's also great to verify the public api of the new schema builder.

This repo will ONLY be used as a PoC.
https://github.com/PascalSenn/HotChocolate.Filters

[michaelstaib]

@PascalSenn I started integrating some of your interfaces to the following branch https://github.com/ChilliCream/hotchocolate/tree/filter_types

[PascalSenn]

I'll start moving more parts over as soon as I got the time 🚀

We are also investigation on another layer of abstraction on top of IQueryable. Different Databases support different use cases.

[michaelstaib]

793

[adampaquette]

Hasura have implemented many powerful features like filtering / orderring / sorting for GraphQL queries. You could take example from their specifications :

• https://docs.hasura.io/1.0/graphql/manual/api-reference/graphql-api/query.html
• https://docs.hasura.io/1.0/graphql/manual/queries/query-filters.html

Examples

Filter nested objects:

query {
  author(where: {articles: {rating: {_gte: 4}}} order_by: [{id: desc}, {author: {id: asc}}]) {
    id
    name
  }
}

Distinct:

query {
  employee (
    distinct_on: [department]
    order_by: [
      {department: asc},
      {salary: desc}
    ]
  ) {
    id
    name
    department
    salary
  }
}

Sub query filter:

query {
  author {
    id
    name
    articles(
      where: {is_published: {_eq: true}},
      order_by: {published_on: desc},
      limit: 2
    ) {
      id
      title
      is_published
      published_on
    }
  }
}

Generic operators:

• _eq
• _neq
• _in
• _nin
• _gt
• _lt
• _gte
• _lte

Text related operators:

• _like
• _nlike
• _ilike
• _nilike
• _similar
• _nsimilar

Checking for NULL values:

• _is_null

[OneCyrus]

@adampaquette seems a bit verbose to have all operators as nested objects. IMHO the flattened prisma style looks nicer. though both can do pretty much the same.

for a quick overview of the prisma filters you can have a look here: https://www.prisma.io/blog/designing-powerful-apis-with-graphql-query-parameters-8c44a04658a9

[michaelstaib]

@PascalSenn @OneCyrus @adampaquette

TODOs:

• [ ] Graph Filter -> meaning we want o allow filters on object structures { a: { b_ne: “foo” } }
• [ ] List Filter -> meaning if your entity has a list of something then we want to be able to apply filter on that as well
• [ ] Filter conventions -> We want to add a conventions class that can be used to configure global behaviour if the type inference
• [ ] Filter provider -> the provider should influence which filters are possible… if you are using a stored procedure driver maybe not all filters are possible

[michaelstaib]

@PascalSenn This one is now really coming together... we are hitting 90% code coverage now on filters.

[michaelstaib]

OK, found a lot of issues in the implementation.

• we have to introduce some kind of validation. Contains does for instance not make sense to be empty. We need some kind of error handling for those kinds of cases.
• How do we secure the expressions against null errors without solution the expressions?

I also have fixed a couple of bugs with the current implementation.

[michaelstaib]

@PascalSenn really like the helper ... just made it so much easier to implement the null checks

[michaelstaib]

I am now thinking we should break up the expression generators... we could have a generator class for each case.... like string_equals and not equals could be one .... the string in and not in could be another and so on.... what do you thing @PascalSenn

[michaelstaib]

This would make it easier to handle errors....

[michaelstaib]

Should we rename UseFilter to UseFiltering?

[michaelstaib]

@PascalSenn and did you already think about sorting :)

[PascalSenn]

We can split up the expression generators. I guess this would make the whole thing even cleaner.

I prefer UseFilter. But I don't really have a strong opinion on this one.

Hm.. The sorting stuff is tricky
I don't like the way prisma is doing it

 feed(orderBy: createdAt_ASC) {
    id
    description
    url
  }

I do like it more this way:

 feed(orderBy: {created_at:DESC})) {
    id
    description
    url
  }

This would enable nested sorting like the following:

 feed(orderBy: {foo: { bar: DESC}})) {
    id
    description
    url
  }

I'm not sure what the performance implications are though

Sidenote: I'm quite busy the next two weeks. (UNI 🙃)

[michaelstaib]

OK, I have fixed the And/Or mixup and added some tests for that.

[michaelstaib]

We can split up the expression generators. I guess this would make the whole thing even cleaner.

lets do that

[michaelstaib]

I prefer UseFilter. But I don't really have a strong opinion on this one.

In general I prefer that too but it seems odd when putting UsePaging().UseFilter().UseSorting() together.

[michaelstaib]

How would we translate nested sorting to expressions?

[michaelstaib]

We should start splitting this task and work out what we want to deliver in 9.1 and what is coming next... I want to wrap up 9.1 over the next two weeks.

[PascalSenn]

We can split up the expression generators. I guess this would make the whole thing even cleaner.

Do you just want to separate just the expression extension class or also whole OperationHandlers?
like StringOperationHandler or ComparableOperationHandler

[PascalSenn]

general I prefer that too but it seems odd when putting UsePaging().UseFilter().UseSorting() together.

lets got for UseFiltering() then

[PascalSenn]

How would we translate nested sorting to expressions?

Not for Collections. But It is possible for nested objects
OrderBy(x => x.foo.bar.baz );

{
   foo {
       bar {
           baz 
       }
   }
}

[PascalSenn]

We should start splitting this task and work out what we want to deliver in 9.1 and what is coming next... I want to wrap up 9.1 over the next two weeks.

💪 = Stretched Goal

9.1 :

• [x] Scalar Filters
• [x] Queryable Visitor
• [ ] Object Filters
• [ ] IEnumerable Filters 💪

9.2 :

• [ ] Sorting
• [ ] IEnumerable Filters

9.X /10

• [ ] Aggregations

You did say the Object Filters would be easy right?
or also streched 💪?

[michaelstaib]

Do you just want to separate just the expression extension class or also whole OperationHandlers?
like StringOperationHandler or ComparableOperationHandler

I think we should have more ExpressionOperationHandler like:
StringEqualOperationHandler that is implementing Equal and NotEqual....
StringContainsOperationHandler and so on...

[michaelstaib]

OK, UseFiltering it is then :) always good to have consent :)

[michaelstaib]

Object Filters are quite simple.

I have some other issues still. The Filter type should be dependent on the actual output type on which the filter is applied. Also we need to be able to overwrite types in the filter. We are to dependent on the entity at the moment.

I will elaborate on this one later.

[michaelstaib]

@PascalSenn

just looked at @adampaquette posting again:

query {
  employee (
    distinct_on: [department]
    order_by: [
      {department: asc},
      {salary: desc}
    ]
  ) {
    id
    name
    department
    salary
  }
}

The sorting is not bad or what do you thing @PascalSenn

[michaelstaib]

And do you see that you can actually do a distinct with Hasura :)

[PascalSenn]

I think we should have more ExpressionOperationHandler like:
StringEqualOperationHandler that is implementing Equal and NotEqual....
StringContainsOperationHandler and so on...

I'll go for that one

[PascalSenn]

The sorting is not bad or what do you thing @PascalSenn

yup is not bad indeed :)
i just think this:
```graphql
order_by: [
{department: asc},
{salary: desc}
]

should actually be this
```graphql
  order_by:     {
     department: asc, 
      salary: desc
       }

I don't really see the reason for an array there.

And do you see that you can actually do a distinct with Hasura :)

I did see that. I was just wondering. Isn't that quite hard to implement on IQueryable ?

public static IQueryable<TSource> DistinctBy<TSource, TKey>  (this IQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector)
    {
        return source.GroupBy(keySelector).Select(x => x.FirstOrDefault());
    }
...
res = res.DistinctBy(x => x.Id);

[michaelstaib]

Lets move Object Filters and IEnumerable Filters to the next release. I think we should stabalize what we have now.

[michaelstaib]

I will create a follow up issue.

[michaelstaib]

Here are the follow up issues:

921 #922 #923 #924

They are at the moment just empty shells.

[PascalSenn]

I added some content to those empty shells

921 #922 #923 #924

[michaelstaib]

I will close this one since all the follow up work is tracked in different issues.

[kkliebersbach]

Hi I was just wondering whether support for a distinct_on filter was ever implemented? Thanks

[PascalSenn]

Hi @kkliebersbach

No I dont think so, can you open a feature request for this one?