Yii2: Check if REST API benchmark at phpbenchmarks.com is correct

[samdark]

Need to check if everything is OK with the benchmark:

http://www.phpbenchmarks.com/en/comparator/framework

https://github.com/phpbenchmarks/benchmark-kit

[kamarton]

• https://github.com/phpbenchmarks/yii
• https://github.com/phpbenchmarks/yii-common

[kamarton]

I can imagine a problem with UrlManager using cache as default. The FileCache configured in the configuration is not sure it will produce better results than simply running with CPU time all the time.

https://github.com/phpbenchmarks/yii/blob/ecfe4744711d796ac792ff1dcbed85ca0e476b90/config/web.php#L11-L13

[ailmanki]

• should add optimization flags to composer.json
  https://getcomposer.org/doc/articles/autoloader-optimization.md
  maybe phpbenchmark does that already?

• does not look optimal to use controllermap for this and alias. why not just use proper pathes which match defaults.
  https://github.com/phpbenchmarks/yii-common/blob/yii_2_hello-world/config/controller-map.php

[rob006]

@kamarton I would rather try ApcCache with disabled serializer. deserialize() may be really slow for big collections and deserialization of 500 objects may be a bottleneck here.

[bizley]

This might not pass since

To have equals chances for each tools, default installation is done.

and we've got FileCache as default everywhere.

Unfortunately we can not recreate the benchmarking process. I really don't like the way JSON is returned for Yii (via Users and Service) in this benchmark but I'm not sure if this causes the problem.

[rob006]

@bizley having 500 rules in Yii app is clearly wrong - URL manager in Yii is flexible enough to handle such scenarios with fewer rules, since you can handle more advanced logic in one rule. If cache is really a problem, then benchmark is testing performance of using Yii in incorrect way. If adjusting config for this pathological usage is not allowed, then this benchmark is completely unreliable ("look how slow/fast is this framework when you use it in a wrong way").

[bizley]

I fully agree.

I'll check the Symfony case and see how it's done there.

[bizley]

Looking through code of Symfony test case and ours I must admit I can not find anything that could help us, we are beaten fair and square. Symfony is also taking the 500 routes (which is no surprise since the test conditions say so) so we really can not call the argument of "misusing the framework" here because other frameworks are tested the same way. Symfony route matching is simply superior to ours and the benchmark is unfortunately created to test the route matching speed. This should rather open up the discussion how to improve our route matcher like they did.
So to summarize - the way it's tested is wrong but it's wrong for all testing subjects. Let's suck it up and focus on Yii 3.

[samdark]

Thanks for looking into it, everyone.

[rob006]

So to summarize - the way it's tested is wrong but it's wrong for all testing subjects.

No it's not. The problem with this test is that it does not define a problem ("handle this 500 routes") but proposes a solution ("create 500 rules to handle this 500 routes"). And this solution promotes specific approach - solutions based on many simple rules will get better results than solutions preferring fewer but more flexible/heavier rules.

and the benchmark is unfortunately created to test the route matching speed

Not really. In benchmark you always have match on first rule. So it is not even testing matching speed, but bootstrap overhead.

[kamarton]

This should rather open up the discussion how to improve our route matcher like they did.

That's the point.

[bizley]

@rob006 this arguing is pointless unless you can convince them to change the benchmarking methodology for all the frameworks.

[rob006]

@bizley I'm arguing that at least we should try APC cache to check if deserialization is the problem. If this is true, it makes perfect sense to optimize app config, especially if test is using inefficient approach. It is not like we're cheating here - there still will be 500 rules, and we're still using standard cache driver. This change does not even require writing any custom code.

[bizley]

I'm looking forward to see the results of this. If they go with the change I'll be very happy.

[rob006]

OK, I've done some tests, and the results are:

1. As I suspected, the main problem is with deserialization of 500 rules in cache - initialization of URL Manager took about 60% of execution time.
2. Disabling cache makes it slightly slower.
3. Using APCu doesn't change anything - it seems that APCu needs to serialize and deserialize object anyway. :/
4. I was able to avoid the whole deserialization overhead by replacing standard cache driver by custom logic based on var_export() and __set_state(). Over 50% performance boost.

So this is not an easy pick, but there is a room for improvement here. Cache driver based on var_export() may be interesting concept worth implementing in core, but it may be more tricky in practice (not all object can be serialized in this way).

Some reading:
https://medium.com/@dylanwenzlau/500x-faster-caching-than-redis-memcache-apc-in-php-hhvm-dcd26e8447ad
https://gist.github.com/Ocramius/2028520

[samdark]

That is definitely something to consider for Yii 3 cache.

[My6UoT9]

Looking through code of Symfony test case and ours I must admit I can not find anything that could help us, we are beaten fair and square. Symfony is also taking the 500 routes (which is no surprise since the test conditions say so) so we really can not call the argument of "misusing the framework" here because other frameworks are tested the same way. Symfony route matching is simply superior to ours and the benchmark is unfortunately created to test the route matching speed. This should rather open up the discussion how to improve our route matcher like they did.
So to summarize - the way it's tested is wrong but it's wrong for all testing subjects. Let's suck it up and focus on Yii 3.

Symfony does not use the 500 rules directly. It compiles out of the rules a php class, I don't see how this is a fair comparision. While Yii cannot automatically compile a class out of it.. it still seems to me no fair comparision.

https://symfony.com/blog/new-in-symfony-4-1-fastest-php-router

In order to speed up the application, during the compilation phase Symfony generates a PHP class called "matcher" which contains all the route definitions optimized to match incoming URLs.

[kamarton]

The rules can be found here:
https://github.com/phpbenchmarks/symfony-common/blob/symfony_4_rest-api/Resources/config/routing.yml

https://github.com/phpbenchmarks/symfony/blob/symfony_4.3_rest-api/config/routes.yaml

The 500 URL rule is a problematic by default, because a developer doesn't use that much rules. Organizes it into modules and controllers.

If want a good result, you should modify the UrlManager component to interpret the rules on-the-fly. Since the test is always run on the first rule, a good result can be achieved.

[markch123]

Hi Guys,

Do you think this would speed things up? The main idea is not to build all the rules and then cache, but to store them in a class (I guess similar concept in Symphony) and then build rules as required. Havn't tested this.

1. add new rulesClass property in urlManager
   'urlManager' => [ 'rulesClass' => 'config\Rules', 'enablePrettyUrl' => true, 'rules' => [ 'rest/objects/\w+/\w+' => 'rest/objects', ], ],

2. Add RulesClass.php.
   ````
   

namespace frontend\config;

use Yii;
use yii\base\InvalidConfigException;
use yii\web\UrlRule;
use yii\web\UrlRuleInterface;

class Rules {

private $verbs = 'GET|HEAD|POST|PUT|PATCH|DELETE|OPTIONS';

public function getRules() {
    return [
        ['rest/objects/\w+/\w+' => 'rest/objects'],
        ['rest/objects/\w+/\w+' => 'rest/objects']
    ];
}

// copied from UrlManager buildRule()
public function createUrlRule($ruleConfig, $ruleArray){
$key=$ruleArray[0];
$rule=$ruleArray[1];
if (is_string($rule)) {
$rule = ['route' => $rule];
if (preg_match("/^((?:($this->verbs),)($this->verbs))\s+(.)$/", $key, $matches)) {
$rule['verb'] = explode(',', $matches[1]);
// rules that are not applicable for GET requests should not be used to create URLs
if (!in_array('GET', $rule['verb'], true)) {
$rule['mode'] = UrlRule::PARSING_ONLY;
}
$key = $matches[4];
}
$rule['pattern'] = $key;
}
if (is_array($rule)) {
$rule = Yii::createObject(array_merge($ruleConfig, $rule));
}
if (!$rule instanceof UrlRuleInterface) {
throw new InvalidConfigException('URL rule class must implement UrlRuleInterface.');
}
return $rule;
}
}
`` Step 3 - AddrulesClass`` field to UrlManager and create class on init , if present.

Step 4 - Update UrlManager buildRules

protected function buildRules($ruleDeclarations)
{
   // if not set then ignore and work as per normal
    if (isset($this->rulesClass)){
        return $this->rulesClass->getRules();
    }

Step 5 - Update UrlManager parseRequest

     // At this point $this->rules is an array of urlRules or strings. Once a rule is matched
        // then drop out of loop. 
         foreach ($this->rules as $rule) {
             /* @var $rule UrlRule */
            $rule = isset($this->rulesClass) ? $this->rulesClass->createUrlRule($this->ruleConfig, $rule) : $rule;
            $result = $rule->parseRequest($this, $request);

If rules are arranged in most accessed order then you could potentially get away with one create command :)

[batyrmastyr]

The 500 URL rule is a problematic by default, because a developer doesn't use that much rules. Organizes it into modules and controllers.

After meeting with SEO guys, your routing will never look like
'<controller>/<action>’ => ’<controller>/<action>’, but more like 'gorenie/oops' => 'burning/unexpected', 'ohlazhdenie' => 'cooling/index'.
Well, yii2 has GroupUrlRule that allowed me to recombine our 170 rules into 90 ones. I don’t think our application is huge.

Well, even 100 rules is a lot when you have to generate dozens or hundreds of urls, so optimizations are appreciated not only for benchmark purposes.

[markch123]

I've made a first cut and unit test results are good. Next step is to check performance running a load test comparing "cached" versus "non cached" URLRule usage in URLManager. I'm pretty sure we will have some good results. Do we have something similar to phpbenchmarks?

[samdark]

No, we don't have a test like that. phpbench could be used to achieve good enough measurements like it was done for Yii 3 DI package: https://github.com/yiisoft/di/blob/master/phpbench.json

[lubosdz]

Presented benchmarks are academic, not practical, serving to present Nikic's fast route implementation - a very specific scenario. I don't remember using more than 10-15 URL rules during last few years in any application. It's a great PR for anyone who implements it, but unfortunately also misleading the community. If the benchmark scenario would be more practical (e.g. crawling randomly 100 - 500 pages, where not every page is redirected), there would be no such a difference. That, of course, does not imply, that there is no room for optimization :-) Most likely it can only be done in the next major version - Yii.3 - due to BC breaks.

[samdark]

Yii3 currently runs on top of Nikic fastroute as default routing driver btw.

[lubosdz]

Yii3 currently runs on top of Nikic fastroute as default routing driver btw.

Oh really? Are there any bench results against either scenario above or against routing in Yii2?

[samdark]

None yet. Busy with other things.

[lubosdz]

OK, found it.
Honestly, I am not really fond of too-complex implementations.
Code legibility & maintainability are also important aspects, along with the performance.
Yii2's routing is very simple and yet flexible & performant, in spite of bench tests above.