Laravel-swoole: Request times increases in time

[sweebee]

Since today i'm running an API on swoole in production.

After swoole is started, one of my endpoints takes around 80-90ms to load.

15 minutes later the same request takes around 130-170ms

30 minutes later 200-250ms

hour later 400-500ms

If i restart swoole again, its 80-70ms again.

I'm running it on a DO server with Ubuntu 18, php7.3, redis (cache) and nginx.

[sweebee]

Also tried a route with a blade template. After start its around 120ms. but after some hours its 1.2s to 1.5s for a request.

[sweebee]

for comparison, nginx with php-fpm the api is around 200ms and the route with blade around 300ms.

[albertcht]

This sounds interesting. Can you try to monitor the memory usage change for swoole workers? And does this happen to all your endpoints?

[sweebee]

I have a simple default api endpoint which only returns:

return response()->json([
            "status" => "ok",
            "time" => date('Y-m-d H:i:s'),
            "timezone" => date_default_timezone_get()
        ]);

This endpoint is always fast. But endpoints with more logic gets slower.

If I look in htop I see 14% (of 1gb) memory usage by swoole_http_server.

Live in action (ignore invalid ssl):

https://167.71.79.88/api/ (always fast)
https://167.71.79.88/api/shop/sgh/0eded121-3a8c-4ef4-893a-8d02863aaac6 (gets slower)

[sweebee]

I just restarted and the memory % of swoole_http_server is 4.5%.

this endpoint https://167.71.79.88/api/shop/sgh/0eded121-3a8c-4ef4-893a-8d02863aaac6 is around 50 - 70ms now.

[sweebee]

Funny thing is, now it doesnt get slow (its not used in production) because there aren't any visitors.

so endpoints which are used a lot gets slower.

[albertcht]

I'm not sure what exactly caused this issue now. Can you try to modify max_request from 3000 to 500 or a lower value to see if this issue can be improved. I believe there's somewhere causing this issue in your code.

[sweebee]

I'm running it in production again with max_requests set to 100. Lets see if its still 80-90ms in in hour.

[sweebee]

I fired 1000 requests at it with apache AB test and its 400-500ms. even under no load. I have no clue what causes this.

[sweebee]

did some more tests, with nginx its slower, but if i directly open it on port 1215 its always fast.

@albertcht do you think nginx can be the cause?

This is my nginx config:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;
        return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
    ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
    root /home/http/mijnetickets.nl/current/public;
    index index.php;

    location = /index.php {
        try_files /not_exists @swoole;
    }
    location / {
        try_files $uri $uri/ @swoole;
    }

    location @swoole {
        set $suffix "";

        if ($uri = /index.php) {
            set $suffix ?$query_string;
        }

        proxy_set_header Host $http_host;
        proxy_set_header Scheme $scheme;
        proxy_set_header SERVER_PORT $server_port;
        proxy_set_header REMOTE_ADDR $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_pass http://127.0.0.1:1215$suffix;
    }
}

[albertcht]

A slightly slower response time in Nginx is normal (proxy). But I can't find any problems from your Nginx config.

BTW, what's your test in production after setting max_request to 100?

[pingcheng]

there is definitely an issue somewhere, as you already did some tests, here are some conclusions you had

1. A simple API is always fast
2. A complex API's response time increase with the usage
3. Restart swoole:http server can reset the response time
4. If request directly to the swoole server, it always fast

So, I will suggest some debugging routes for you. Since the simple API would not get slower, so I suggest you compare the difference between these two APIs, especially if you used other services like database, Redis connection, check your Laravel middleware as well, maybe one of them did some hidden operations without your awareness.

Then, I suggest you can track which line of code costs the time, you can do any way you familiar with, the easiest method may be print out many timestamps (with milliseconds) before suspect lines. Once you got which line cost more time, then you will know where the issue is.