Graphql-laravel: 500k records query perfomance

[drmax24]

rebing/graphql-laravel": "^5.1"
php 7.4
mysql 8

We are writing an API returning telemetry values for a mobile application to display.
250-500k records should be returned at once via a single graphql query.
I observe that server response is very slow. Around 40 seconds TTFB (Time to first byte) for 40 megabytes of json.
Any ideas how this can be improved?

Data from the database itself is fetched very quickly (20ms) for all 500k records (7 float and enum values per record)

public function resolve($root, $args, $context, ResolveInfo $resolveInfo, Closure $getSelectFields)
    {
    $microtime = microtime(true);
    ...

    $out = $model->select($fields->getSelect())->get()
    // dd(microtime(true) - $microtime);
    // 20 ms up until here.

    // It becomes very slow after I return data inside query resolver in the following line.
    return $out
}

and I have 40 sec TTFB for the API response

[mfn]

This is also a recurring topic on the underlying library itself: https://github.com/webonyx/graphql-php/issues?q=is%3Aissue+performance+is%3Aclosed

Then there's also a completely different issue of performance like https://github.com/rebing/graphql-laravel/issues/390

Whether "250-500k" is already a high number can't be easy said. You've to remember for each record (= mapping to a GraphQL type, I assume) and then _for each field_ a resolver is executed.

10 fields? That makes 2.5m to 5m resolver calls (just to paint a picture on the wall). For enums: each one is also validated. So these are a _lots_ of calls.

I _could imagine_ this taking some time.

There's no silver bullet here:

• benchmark: where exactly is the slow down: do you have custom resolvers, are they the slowdown? is it in rebing? in webonxy?
• can you reduce the number of returned entries? Can you split it up into multiple GraphQL calls? Pagination?
• …

[cichondev]

@drmax24 I'm sorry if I'm talking nonsense, but maybe you better process this data in the API and return only the final result.
But that definitely doesn't seem to be an issue with this package.

[mfn]

Unfortunately not more specifics were given yet, but as the author of the graphql-php also sometimes says: there are cases which might not appropriate for GraphQL due to this overhead. As with many things, it's a trade-off.

[drmax24]

Thank you for some effort to help me!
Sorry for the delay... I will investigate further and post in this thread as soon as i get more info

[drmax24]

(1)

do you have custom resolvers

No, i don't have custom resolvers within the type. Also, there are no relationships or linked objects.

(2)
Here is how i measure total execution time. So the problem is not caused by load balancer / compression and such.

<?php

require __DIR__ . '/../vendor/autoload.php';
$app = require_once __DIR__ . '/../bootstrap/app.php';

$kernel = $app->make(Illuminate\Contracts\Http\Kernel::class);

$response = $kernel->handle(
    $request = Illuminate\Http\Request::capture()
);

var_dump(microtime(true) - $_SERVER["REQUEST_TIME_FLOAT"]);

$response->send();

$kernel->terminate($request, $response);

(3)
There is no apparent performance difference if I request 500k records 6 fields each including one enum or 6 fields including integer instead of enum.

Will post more info soon...

[mfn]

I don't want to jump to conclusion, but even from your initial description it may just be "too much data to be serialized at once".

When https://github.com/rebing/graphql-laravel/issues/390 was reported, what helped the most was a repo which "simulated" the problem. lo' and behold, we truly were able to improve performance (AFAIK lazy types was the solution thanks to @crissi ❤️ ).

So something like this to play around would ofc help a lot.