Spreed: Nextcloud Talk is really slow when using Nginx as WebServer

[warnerbryce]

Steps to reproduce

1. Install Nextcloud 18.0.3 on a server with Apache2
2. Test the Talk app with users and have a fast discussion.
3. Verify that every time you press enter, the user get the text quickly
4. Disable Apache2 and turn on Nginx
5. Do the same test on Talk app, you will see that sometimes your text will be slow to appear and the person who press enter will have to wait 5 to 10 seconds before the release of the text.

Expected behaviour

Normally it should be instantaneous.

Actual behaviour

Slow as hell but sometimes (a few times) it works great

Browser

Firefox / Chrome / Safari

Talk app

Talk app version: But present from a long time

Custom TURN server configured: yes but it doesn't mater

Custom STUN server configured: yes but it doesn't mater

Server configuration

Operating system: Ubuntu 18.04

Web server: Nginx

Database: Tested on MariaDB 10.3 and PostgreSQL

PHP version: Tested on PHP 7.2 7.3 and 7.4 used with fpm or without same result

Nextcloud Version: from 16 to 18.0.3

Nothing in log show bottlenec

I Know that NC encourage the use of Apache2 but a lot of user prefer to use Nginx, i hope someone could catch what is wrong.

[nickvergessen]

There seems to be an issue with nginx and http2.

2211

[warnerbryce]

You closed it fast 💨
I tested with http2 and without, same behaviour.

[warnerbryce]

There seems to be an issue with nginx and http2.

2211

Please re-open it, i tested without HTTP2 on Nginx and the behaviour is the same, same issue

[warnerbryce]

https://github.com/nextcloud/spreed/issues/2211#issuecomment-590726874
@nickvergessen I open this ISSUE because of that.

[nickvergessen]

Then you need to be more specific.
Which nginx version are you using, which php packages are installed. Did you install nginx like recommended in the Nextcloud Server documentation or did you follow another guide?

[warnerbryce]

I have follow with How To : https://riegers.in/nextcloud-installation-guide-ubuntu-18-04/
but i get this bug before form another How To from an Nextcloud well informed guy.
I will make a test now with this : https://docs.nextcloud.com/server/18/admin_manual/installation/nginx.html#nextcloud-in-the-webroot-of-nginx

I have nginx v.1.17.9 but again since it's a long issue the version of Nginx doesn't mater a lot.
Php 7.4 with zip xsl xmlrpc xml tidy sybase sqlite3 soap snmp readline pspell phpdbg opcache odbc mysql mbstring ldap json intl interbase imap gmp gd fpm enchant dev dba curl common cli cgi bz2 bcmath

[dtissen]

@warnerbryce I have exakt this setting... still slow.

[warnerbryce]

@nickvergessen
i have for the moment no Hangs with the Nextcloud Nginx config file. I had to made some little modifications but for the moment it works.
I will test that on another client that have the same problem.
If it works i will answer here shortly.
@dtissen I use the nextcloud Nginx config file you can get it here : https://docs.nextcloud.com/server/18/admin_manual/installation/nginx.html#nextcloud-in-the-webroot-of-nginx
and i made this mods :
line 1 : upstream php-nextcloud #because i have already another upstream with the name php-handler
line 2 : #server 127.0.0.1:9000; #because i use a .sock file
line 3 : server unix:/var/run/php/php7.4-fpm.sock; #because i use php7.4-fpm

Obviously i change every cloud.exemple.com to my FDQN of my cloud
HTTP2 is enabled too but i don't have this problem, maybe it's fixed. If i have hangs again i will remove http2 in the 443 part of the conf file

Line 22 and 23 i mod where my certs are
Line 47 i mod where my nextcloud folder is
Line 72 and 73 moded with my usage
line 110 : fastcgi_params php-nextcloud #needed to fit what i wrote in line 1

Edit : some news
I have still the same issue for one instance but not on the other i will write down the difference between them
Don’t work :
Nc 17.0.3
Php7.3-fpm
HTTP2 on or off gave the same result

Work :
Nc 18.0.3
php7.4-fpm
HTTP2 enabled

I’m gonna update my php7.3 to 7.4 and see what it does.

[warnerbryce]

"@nickvergessen
That are my final testing :
Even after passing my instance on Nc 18.0.3 with the latest Talk app and PHP7.4-fpm i have this error but Disabling HTTP2 now make the Talk App responsive whereas on 17.0.5 no ...
The only thing that different between my two instances is POWER.
One is a dedicated server Core i5 6th Gen Ubuntu 18.04
The other is inside a VM on a macmini Core i5 3th Gen Ubuntu 18.04
They both have the same software installed and the same conf files.

So my statement is :
If you have Nginx and having slow downs on Nc Talk, go on the last version of php-fpm and Nextcloud, if it's still the same disable HTTP2.
I'm clueless why HTTP2 enabled works well on one instance and not on the other, except performances, but looking with HTOP gave no bottleneck while having slow downs on Talk..."

EDIT : Drop everything i wrote on this post to the trashbin.
My testing operation was wrong. I was using my iPhone without Wifi for making a conversation with my self on a computer attached to a wireless network. It was really slow...
In fact because of the Covid-19 i have a terrible bandwidth with my 4G Network provider 5Mbits/s down and 0,01Mbits/s up... that why i get bad result on one of the instance.
Knowing that :
With Nextcloud 18.0.3 - php7.4-fpm - Nginx and HTTP2 enabled i have good performances for Nextcloud Talk !
I can't get back to the past to test it with Nextcloud 17.0.5 but now my instances works.
Last word : using the nginx template from the How To is getting bad results with HTTP2 enabled, using the nginx template from Nextcloud Doc give good performances with HTTP2 enabled.

Thank you.

[nickvergessen]

Thanks for testing and glad it works now.

Leaves me a bit puzzled as to what exactly is the problem, but as long as updating everything works out, that shall be the solution.

[Pushergene]

I can confirm it. I am using Docker with .examples using with-nginx-proxy because it has a good way to generate and mount SSL keys. When the call works, then the quality is extremly good. But it is so slow if u want to accept, especially when other device is using iOS. Is it maybe because of the STUN? I am using the one of t-online.de
Can you fix it please? I am using newest Versions.

[nickvergessen]

Did you try not using a server from t-online?
Because the issue here always is about everything being slow. If only calling/starting a call is slow, it is definetly a different issue.

[nickvergessen]

So, after some checking you have to use:

• php + mpm_prefork or
• php_fpm + mpm_event

Any other mix is not working well.

[JeWe37]

@nickvergessen I am already using nginx as the proxy for other services. Would the slightly awkward alternative of simply putting a full docker apache install behind a regular nginx proxy that takes care of things like SSL and the subdomain work? Has the underlying issue been identified as of yet?

[warnerbryce]

@JeWe37 in fact you have to use the nginx conf file that you can find in the nextcloud documentation. It works for me.
I was using the conf file from a HOW to from c-reigner and it don’t work for Talk.

[JeWe37]

Could you make a gist of your conf file or so? I used a minimally modified version of the conf from here: https://docs.nextcloud.com/server/18/admin_manual/installation/nginx.html
There must be some setting that is preventing it from working for some and not others. I have already tried adding a TURN server and disabling HTTP 2.0 with little success...

[warnerbryce]

@JeWe37 this is exactly the one i use, and since i use it no more lags in discussions even with HTTP/2.
What kind of problems do you have ?

[JeWe37]

At times it works great, others messages take 20 seconds to receive and joining calls is virtually impossible. I have not been able to identify a clear pattern as to when it happens. There seems to be some trouble perhaps with multiple devices using the same account(i.e. say my Linux desktop and iPad), but even that is not a definitive indicator of when it is going to fail. Here is my config: https://gist.github.com/JeWe37/3afcbb81e67e41cda146a9cfdbf6a649
There is some redirection stuff for HTTP in other files still but that's it. The fastcgi path was only edited cause my nextcloud root is inside the html folder.
I'm also not sure if it perhaps has something to do with my TURN server configuration. But prior to configuring that(I'm not sure if it is working currently) it did not work either.

[bpl34567]

I had huge performance issues with Nextcloud Talk, sometimes the whole Nextcloud was terrible slow. I updated php to php7.4 fpm, updated my settings to use fast_cgi. I didn't not really help but I could solve the performance issues by setting some php config values as described here:
https://stackoverflow.com/questions/10003978/php-fpm-and-nginx-502-bad-gateway

Maybe it will help someone else to solve performance issues.

[Selaron]

I was also facing these issues as soon as more than 2 users invoked Talk web interface:

• Slow chat messages sometimes nearly 30s delay
• Video conferencing very slow for 2 participants, impossible for 3 (some users saw circle wait indicators only)
• Very slow overall nextcloud interface

The solution for me was to increase pm.max_children in /etc/php/7.4/fpm/pool.d/www.conf to 16 which by default is set to value 5 on debian. Each talk user blocks 2 php-fpm processes via long polling, thus 3 users is to much for 5 processes.

Consequently none of the hints above helped (enable/disable http2, upgrade php7.3 to 7.4, using nginx config from NC documentations, ...)

I found the hint here: https://github.com/nextcloud/spreed/issues/2010#issuecomment-601755020