Spreed: Nextcloud Talk is really slow when using HTTP2

[sL5gqj6Q]

Steps to reproduce

1. Install Nextcloud Talk App from the appstore in the browser
2. User A starts a chat with User B, both registered on the same Nextcloud server
3. Messages take up to 20 seconds to be delivered
4. Also replies from User B to User A need up to 20 seconds to be delivered

Expected behaviour

I expect messages to be delivered nearly instantaneuously

Actual behaviour

Nextcloud Talk feels slow and laggy. User A opens a chat with User B in his Browser. There is a constant loading circle around User A, as if he cannot join the session. User B is invited. Chat messages from both users take up to 20 seconds to send/receive.

Browser

Microphone available: yes

Camera available: yes

Operating system: User A: Linux Mint 19.2 , User B: Android 9

Browser name: User A: Firefox 69 , User B: Nextcloud Talk Android App v6.0.0.beta3

Browser version: see above

Talk app

Talk app version: 5.0.4

Custom TURN server configured: no

Custom STUN server configured:no

Server configuration

Operating system: Raspbian 8.0 (jessie)

Web server: Apache

Database: MariaDB

PHP version: 7.0.27

Nextcloud Version: 15.0.11

List of activated apps:

Enabled:

• accessibility: 1.1.0
• activity: 2.8.2
• announcementcenter: 3.4.1
• bruteforcesettings: 1.4.0
• calendar: 1.6.6
• checksum: 0.4.3
• cloud_federation_api: 0.1.0
• comments: 1.5.0
• contacts: 3.1.3
• dav: 1.8.2
• federatedfilesharing: 1.5.0
• federation: 1.5.0
• files: 1.10.0
• files_pdfviewer: 1.4.0
• files_sharing: 1.7.0
• files_texteditor: 2.7.0
• files_trashbin: 1.5.0
• files_versions: 1.8.0
• files_videoplayer: 1.4.0
• firstrunwizard: 2.4.0
• gallery: 18.2.0
• logreader: 2.0.0
• lookup_server_connector: 1.3.0
• music: 0.10.1
• nextcloud_announcements: 1.4.0
• notes: 3.0.3
• notifications: 2.3.0
• oauth2: 1.3.0
• password_policy: 1.5.0
• provisioning_api: 1.5.0
• quota_warning: 1.4.0
• serverinfo: 1.5.0
• sharebymail: 1.5.0
• spreed: 5.0.4
• support: 1.0.0
• survey_client: 1.3.0
• systemtags: 1.5.0
• tasks: 0.11.3
• theming: 1.6.0
• twofactor_backupcodes: 1.4.1
• twofactor_totp: 2.1.2
• updatenotification: 1.5.0
• workflowengine: 1.5.0
  Disabled:
• admin_audit
• encryption
• files_external
• user_ldap

Nextcloud configuration:

{
"system": {
"instanceid": "REMOVED SENSITIVE VALUE",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"localhost",
"REMOVED URL",
"REMOVED URL"
],
"datadirectory": "REMOVED SENSITIVE VALUE",
"overwrite.cli.url": "http:\/\/localhost\/nextcloud",
"dbtype": "mysql",
"version": "15.0.11.1",
"dbname": "REMOVED SENSITIVE VALUE",
"dbhost": "REMOVED SENSITIVE VALUE",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"logtimezone": "UTC",
"installed": true,
"updater.release.channel": "stable",
"maintenance": false,
"theme": "",
"loglevel": 2,
"mail_smtpmode": "smtp",
"mail_smtpsecure": "tls",
"mail_from_address": "REMOVED SENSITIVE VALUE",
"mail_domain": "REMOVED SENSITIVE VALUE",
"mail_smtpauthtype": "LOGIN",
"mail_smtpauth": 1,
"mail_smtphost": "REMOVED SENSITIVE VALUE",
"mail_smtpport": "587",
"mail_smtpname": "REMOVED SENSITIVE VALUE",
"mail_smtppassword": "REMOVED SENSITIVE VALUE",
"memcache.local": "\OC\Memcache\APCu"
}
}

Server log (data/nextcloud.log)

not found

[tiotrom]

I can confirm it is the same for me. We use a VPS with 4GB of RAM and 8 core for nextcloud. Tested with Firefox and Android app. It is always very slow. I was thinking it might be our setup. We also have a TURN server in place.

[nickvergessen]

Are you using HTTP2?

[sL5gqj6Q]

Hi @nickvergessen ,
yes, my Nextcloud server supports HTTP2 .

[nickvergessen]

Http2 Support is currently a problem with chatting and webrtc

[tiotrom]

In my case I don't use HTTP2 and the chat is quite slow as well. But not 20 seconds. More about 2 seconds. Ok, I don't want to interrupt the conversation but I thought it may be useful. Talk is a bit slow - not "instant".

[nickvergessen]

@sL5gqj6Q so try to disable HTTP2 for now, until we found a useful working solution to the problem. Sorry about your inconvenience.

@tiotrom 2 seconds is not related to this problem here. But you still do not need to create a new issue for it. We wait by default 1 second to reduce the load for the server. Otherwise every user would bombard the server with requests all the time.

[sL5gqj6Q]

After disabling http2, messaging is possible nearly in real time. Thanks for the hint @nickvergessen . However, I hope that http2 support is on the agenda as one of the features to add to Nextcloud Talk.

[nickvergessen]

So after a discussion with a college today, I tried it again and for me it worked quite well.
I followed the following manual to set up http2: https://techwombat.com/enable-http2-apache-ubuntu-16-04/

maybe you forgot one of the changes on your system?

[cjhille]

I'm running Nextcloud behind a nginx reverse proxy and even manually disabling http2 (and reloading the proxy config) didn't solve the described issues for me. Is there any other way to make sure http2 is not used? Apparently http2 can't be passend with nginx reverse proxies properly :(

The reverse proxy setup is officially supported/maintained in the nextcloud docs (see examples folder) so I'd suggest to add the info, that spreed won't be working properly behind reverse proxies.

[warnerbryce]

Even without HTTP2 disabled, i have a real slow down on Talk with Nc 17.0.2 5-6 seconds before the text is send.

[jaabrao]

I was dealing with the issue of Talk being really slow, and stumbled upon this page after some searching. I see some people pointing at HTTP2 being enabled as the culprit.

Alas, when I checked, it was never on to begin with on my current server, nor on the old one. And both were dogs.

So I decided to enable it. And following all the instructions still left me with http1.1. The following link shared by nickvergessen nailed it:

I followed the following manual to set up http2: https://techwombat.com/enable-http2-apache-ubuntu-16-04/

After doing that - and making sure it tests positive for HTTP2 operation - it's fast as it should be. Not only Talk, but Nextcloud as a whole.

There are a few caveats to these instructions:

• I use Debian Stretch, so the whole PPA thing (Step 1) is pointless. Check if you actually need that.
• Current PHP in Debian as of this writing is 7.3, so it would be php7.3-fpm for example.
• I could skip Step 4 as well, as enabling http2 using "a2enmod http2" already creates a global directive for the Protocols. Check /etc/apache2/mods-available/http2.conf
• I restarted apache2 and php7.3-fpm. Because oh well.
• I also updated the php.ini from /etc/php/7.3/fpm/ to make sure all the settings match the optimizations I performed on the old php.ini file located in the apache2/ directory.

[Bellusterra]

it seems to have become more of an issue after the most recent update of Nextcloud Talk 8.0.2. but that could just be me.

[cjhille]

I checked the logs because my entire NC instance slowed down when switching back and forth between channels quickly. I noticed this in my container logs (which are based on the official nextcloud docker-compose setup):

*35272 upstream server temporarily disabled while reading response header from upstream, client: 109.104.41.102, server: cloud.REDACTED.de, request: "DELETE /ocs/v2.php/apps/spreed/api/v1/room/u3i7skpi/participants/active HTTP/2.0", upstream: "http://172.19.0.7:80/ocs/v2.php/apps/spreed/api/v1/room/u3i7skpi/participants/active", host: "cloud.REDACTED.de"

*35272 no live upstreams while connecting to upstream, client: 109.104.41.102, server: cloud.REDACTED.de, request: "DELETE /ocs/v2.php/apps/spreed/api/v1/room/u3i7skpi/participants/active HTTP/2.0", upstream: "http://cloud.REDACTED.de/ocs/v2.php/apps/spreed/api/v1/room/u3i7skpi/participants/active", host: "cloud.REDACTED.de"

This explains why the entire NC instance freezes. NGINX temporarily disables the upstream.

@nickvergessen Could it be that:

• without HTTP2 some responses take too long (some sort of long polling?) -> connection gets terminated with error -> nginx temporarily disables entire upstream?
• without HTTP2 talk makes too many concurrent requests and some capacity limit is reached?

[Bellusterra]

@cjhille you are running nextcloud docker behind nginx reverse proxy?

[cjhille]

@Bellusterra Yes, basically I'm using this official docker config with an additional reverse proxy (jwilder/nginx-proxy) in front for certificate automation.
However there are actually two (sequential) upstreams involved:

1. from jwilder/nginx to nextcloud_nginx
2. from nextcloud_nginx to php-fpm

The previous error log was from the first upstream, however the latter seems to be causing the issues for the former.

Error from the second (nextcloud_nginx to php-fpm) upstream (NOTE: fastcgi timed out):

[error] 6#6: *44159 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 109.104.41.102, server: , request: "GET /ocs/v2.php/apps/spreed/api/v1/room/gg6g7z3t/participants HTTP/1.1", upstream: "fastcgi://172.18.0.9:9000", host: "cloud.REDACTED.de"

[Bellusterra]

@cjhille I have a similar setup, nginx reverse proxy to another nginx in front of the docker nextcloud apache version. I am not getting the same errors you are however. Both nginx are run directly on the host machine however and not in docker, if that helps with troubleshooting at all.

[Bellusterra]

This may help you as well. I don't remember where I got these settings, but you can always give them a try and see if they fix your error and if not remove and restart your nginx.

http {

# INCREASE NGINX PROXY TIME-OUT

proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

[Dennis1993]

My nextcloud instance is slowly too. The messages will be delivered after 30 seconds and more.

Fresh installation:
PHP 7.4
MySQL 5.7
NextCloud 18.0.1
HTTP/2 activated

On a second server (upgraded from ownCloud for a few years) the same problem (PHP 7.3, MySQL 5.7, HTTP/1.1). A video call needs ober 1 minute to establish a connection to the partner.
Nothing shown in the protocols. The fresh installation is on a shared hosting for a test. The other installation is on an own root server and optimized by your "NC server tuning guide".

[nickvergessen]

You forgot to post your webserver version + configuration.
That is the most troublesome point in this discussion.

[nickvergessen]

So I checked this again because of all the comments. and it works very fine here:

$ apache2 -v
Server version: Apache/2.4.41 (Ubuntu)
Server built:   2019-08-21T20:43:38

$ php --version
PHP 7.3.15-1+ubuntu19.10.1+deb.sury.org+1 (cli) (built: Feb 20 2020 12:24:05) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.15, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.3.15-1+ubuntu19.10.1+deb.sury.org+1, Copyright (c) 1999-2018, by Zend Technologies
    with Xdebug v2.9.2, Copyright (c) 2002-2020, by Derick Rethans

To enable it I did:

sudo a2dismod php7.3
sudo a2enconf php7.3-fpm
sudo a2dismod mpm_prefork
sudo a2enmod mpm_event
sudo a2enconf php7.3-fpm 

[cjhille]

@nickvergessen I'm glad that see that you have a specific configuration (custom apache+php-fpm) that works flawlessly in your testing. However I would like to point out, that these days many people prefer just to use the official docker images, which includes a nginx+php-fpm/alpine variant (as they are demonstrated in the official docs). Have you tested running talk behind nginx? I'm guessing the underlying problem with nginx+php-fpm setups might be a configuration issue with Talks long running requests (as noted above).

Out of curiosity: If talk requires some more dependencies (i.e. the php modules you mentioned), how are you making sure, that theses deps are contained (or can be auto installed) in official images? Shouldn't Talk be trying to support the official images?

[warnerbryce]

I would say exactly the same.
The conf have to work on Apache and Nginx, both are hugely used in Nextcloud.

[nickvergessen]

Hope this doesn't sound rude, at least it's not meant to be!

No, I have not tested nginx. Sorry to say, but it's not officially supported by Nextcloud GmbH:
https://docs.nextcloud.com/server/18/admin_manual/installation/nginx.html

These configurations examples were originally provided by \@josh4trunks and are community-maintained.

Also the vm, snap, … are not officially supported:
https://docs.nextcloud.com/server/18/admin_manual/installation/source_installation.html

Please note that those two options are not officially supported by Nextcloud GmbH.

It's simply impossible for us with our teamsize to do everything.
Nextcloud Talk works with HTTP2 on Apache2, so it is not a software related problem in the code of Nextcloud Talk, but a configuration issue with your nginx server.
You can either ask the Docker people to have a look (if they know what is causing it), or you try to find the josh4trunks user and ask him if he has an idea why nginx is not capable to manage more then 1 connection per window with HTTP2

Out of curiosity: If talk requires some more dependencies (i.e. the php modules you mentioned), how are you making sure, that theses deps are contained (or can be auto installed) in official images? Shouldn't Talk be trying to support the official images?

There are no such dependencies for talk, that is what you need on apache2 for http2

[Kovo]

Same issue here on a fresh install of NC using nginx. Did not use the docker image. But installed it manually. Two people on a call works fine for the most part. Until a third joins, and then everything bogs down to a near halt. Even after ending the call, its still slow until it eventually fixes itself.

I can also confirm that there is no difference between http2/http1.1

[nickvergessen]

I can also confirm that there is no difference between http2/http1.1

Then you are having a very different issue

[Kovo]

I can also confirm that there is no difference between http2/http1.1

Then you are having a very different issue

How so? People in this thread have stated issues both with and without http2. Symptoms are also almost identical.

[JohnArcher]

We are running Nextcloud 18.0.3 and Talk 8.0.5/8.0.7 (tested both version). Our instance runs on the shared hoster all-inkl.com (Package "all-inkl Business") which is relatively popular here in Germany. I don't expect speed wonders, but also suffer from this 20 to 30 seconds lag for every message and permanent loading circles next to participant names.

We run on a Apache webserver (don't see the version number, but I assume it is quite the latest) and this infos from phpinfo:

System: Linux ####### 4.15.0-91-generic #92~16.04.1-Ubuntu SMP Fri Feb 28 14:57:22 UTC 2020 x86_64

Server API: FPM/FastCGI

PHP Version: 7.3.11-nmm1

This program makes use of the Zend Scripting Language Engine:
Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies
with the ionCube PHP Loader + ionCube24 v10.3.9, Copyright (c) 2002-2019, by ionCube Ltd.
with Zend OPcache v7.3.11-nmm1, Copyright (c) 1999-2018, by Zend Technologies

$_SERVER['HTTP2']: on

$_SERVER['SERVER_PROTOCOL']: HTTP/2.0

Is there anything else I could provide from phpinfo or do you have an idea or experience with all-inkl.com what to do in order to increase the speed? Thanks in advance!

[alve89]

I'm also on a shared host of all-inkl.com (all-inkl Premium) without having HTTP2 activated (yet). At all-inkl.com it's very easy to switch to a machine with HTTP2, so if the problem of [@JohnArcher is solved I could move there.

Or why is it even on HTTP 1.1 this slow?

Chat messages: 20+ sec
Calls: Nearly impossible (I didn't manage to run a single successful call yet)

[nickvergessen]

Can you share a public conversation link with me or a test account on your instance?

[alve89]

What would you prefer?

[alve89]

I just set a conversation as private one (activating the lobby). When I promote a user as moderator (and downgrade it again) both actions takte action in no time and I got notified about it on all clients. So that worked very quickly.
But the messages are still delayed.

[nickvergessen]

What would you prefer?

you can do both at the same time <my github name>@nextcloud.com

[alve89]

You should have mail.

[JohnArcher]

What would you prefer?

you can do both at the same time <my github name>@nextcloud.com

I have sent a mail to you.

[nickvergessen]

So, after some checking you have to use:

• php + mpm_prefork or
• php_fpm + mpm_event

All-Inkl uses:

• php_fpm + mpm_prefork

This limits the number of request handlers and causes the effects your are seeing ( https://tecadmin.net/apache-mpm-prefork-and-worker-and-event/ might help to understand).
A mail to support gave the following answer:

Das wird leider nichts im Webhosting Bereich. Denn die Erweiterung benötigt so viele Ressourcen, dass dies nur auf einen eigenen Server richtig funktionieren wird.

---

Google translate:
Unfortunately, this will not work in the web hosting area. Because the expansion needs so many resources that this only work properly on its own server will.

So yeah, Talk won't work on a Webhosting package, but you need a server for it, which totally makes sense.

[JohnArcher]

Thanks again for your help yesterday, @nickvergessen !

[jcastang]

Hello,

I was facing the same issues:

• Talk slow
• Messages takes up to 20sec to be displayed/sent
• Avatar were loading slow.

I tried with and without HTTP2 -> No differences.
I am on Ubuntu 18.04 and Nextcloud 18.0.3 / Talk 8.0.6

I was running with mpm_event and php-fpm.
After going back to mod_php and mpm_prefork all issues were gone.

Please test this configuration and feel free to report :)

Cheers,

[HenryDunham]

I can confirm: When using NextCloudPi with Nextcloud 18.0.4, Talk 8.0.8, Apache 2.4.38 and PHP 7.3.14, changing to mod_php and mpm_prefork leads to good performance of Talk.

Unfortunately I was not able to get reasonable performance for Talk with http2, php-fpm and mpm_event. I guess it’s an issue related to configuration.

_(Details on what I’ve tried so far: https://help.nextcloud.com/t/nextcloudpi-and-talk-configuration-incompatible/79469)_