Server: Make Nextcloud compatible with HTTP/2

[tessus]

Steps to reproduce

1. Activate mod_http2 in Apache using the following directives:

ProtocolsHonorOrder On
Protocols h2 http/1.1

Expected behaviour

It should work as before.

Actual behaviour

• Theming does not work (the login image was the original, stock image)
• After login: Connection to server lost error messages
• No files/directories are shown
• No avatar is shown
• When I was finally able to click the admin page:
  
  
  
  • Your web server is not yet set up properly to allow file synchronisation because the WebDAV interface seems to be broken.
  
  
  • Error occurred whilst checking server setup
  
  
  • Error occurred whilst checking server setup
  
  
  • Error occurred whilst checking server setup
  
  

Update: My web site, my gallery software, all other pages on my server worked without issues or errors with http2

Server configuration

Operating system: Linux

Web server: Apache/2.4.25

Database: mysql 5.6.25

PHP version: 7.1.4

Nextcloud version: 11.0.3 (stable) - 11.0.3.2

Updated from an older Nextcloud/ownCloud or fresh install: updated from 11.0.2

Where did you install Nextcloud from: nextcloud.com

Signing status:

List of activated apps:

Enabled:
 - activity: 2.4.1
 - activitylog: 0.0.1
 - admin_audit: 1.1.0
 - announcementcenter: 3.0.0
 - apporder: 0.3.3
 - bookmarks: 0.9.1
 - calendar: 1.5.2
 - comments: 1.1.0
 - contacts: 1.5.3
 - dav: 1.1.1
 - direct_menu: 0.10.1
 - external: 1.2
 - federatedfilesharing: 1.1.1
 - federation: 1.1.1
 - files: 1.6.1
 - files_external: 1.1.2
 - files_markdown: 1.0.1
 - files_pdfviewer: 1.0.1
 - files_sharing: 1.1.1
 - files_texteditor: 2.2
 - files_trashbin: 1.1.0
 - files_versions: 1.4.0
 - files_videoplayer: 1.0.0
 - firstrunwizard: 2.0
 - gallery: 16.0.0
 - issuetemplate: 0.2.1
 - lookup_server_connector: 1.0.0
 - nextcloud_announcements: 1.0
 - notes: 2.2.0
 - notifications: 1.0.1
 - password_policy: 1.1.0
 - provisioning_api: 1.1.0
 - serverinfo: 1.1.1
 - sharebymail: 1.0.1
 - systemtags: 1.1.3
 - tasks: 0.9.5
 - templateeditor: 0.2
 - theming: 1.1.1
 - twofactor_backupcodes: 1.0.0
 - updatenotification: 1.1.1
 - user_external: 0.4
 - workflowengine: 1.1.1

Disabled:
 - encryption
 - files_accesscontrol
 - files_automatedtagging
 - files_retention
 - logreader
 - survey_client
 - user_ldap
 - user_saml

The content of config/config.php:

{
    "instanceid": "oc2352d53521",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "datadirectory": "\/data\/nextcloud\/data",
    "dbtype": "mysql",
    "version": "11.0.3.2",
    "installed": true,
    "forcessl": true,
    "loglevel": 2,
    "maintenance": false,
    "trusted_domains": [
        "cloud.evermeet.cx"
    ],
    "share_folder": "\/Shared",
    "dbname": "nextcloud",
    "dbhost": "localhost",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "logdateformat": "Y-m-d H:i:s O",
    "logtimezone": "Europe\/Vienna",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpmode": "php",
    "mail_from_address": "evercloud",
    "mail_domain": "evermeet.cx",
    "mail_smtpsecure": "tls",
    "memcache.local": "\\OC\\Memcache\\Redis",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "redis": {
        "host": "\/data\/redis\/redis.sock",
        "port": 0,
        "timeout": 0
    },
    "appstore.experimental.enabled": true,
    "trashbin_retention_obligation": "auto",
    "updater.release.channel": "stable",
    "htaccess.RewriteBase": "\/",
    "overwrite.cli.url": "https:\/\/cloud.evermeet.cx",
    "auth.bruteforce.protection.enabled": false
}

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36 , Firefox 53.0

Operating system:

Logs

Web server error log

no errors (empty)

Nextcloud log (data/nextcloud.log)

no errors (empty)

Browser log

js error log console:

    Content Security Policy: Directive ‘frame-src’ has been deprecated. Please use directive ‘child-src’ instead.  (unknown)
JQMIGRATE: Migrate is installed, version 1.4.0  jquery-migrate.min.js:2:542
Shutting down notifications: [0] error  app.js:273:5
    _onFetchError https://cloud.evermeet.cx/apps/notifications/js/app.js:273:5
    j https://cloud.evermeet.cx/core/vendor/jquery/dist/jquery.min.js:2:26920
    fireWith https://cloud.evermeet.cx/core/vendor/jquery/dist/jquery.min.js:2:27738
    x https://cloud.evermeet.cx/core/vendor/jquery/dist/jquery.min.js:4:11276
    b/< https://cloud.evermeet.cx/core/vendor/jquery/dist/jquery.min.js:4:14740

[MorrisJobke]

Generally speaking this should be not a problem with Nextcloud itself, because HTTP2 is compatible with HTTP1.1 web applications. So I would say that this is a problem of the setup rather than nextcloud itself. See also #2024

As of https://help.nextcloud.com/t/http-2-yes-or-no/4026/22 it seems to work for many people.

If you still have problems check the web server config and ask in the forums for help.

[tessus]

I've already read these links before I posted this bug report.

I can assure you that I configured my server correctly. It worked with owncloud 8.x without issues. Then after a while a bug with a later version of h2 was introduced, which forced me to deactivate h2 again.
The current h2 module is stable and all my other web sites work just fine. Why is nextcloud the only thing that is not working?

I've also read about nextcloud dependency issues that certain resources have to be loaded after other ones. h2 might load them in parallel and/or in a different order.

[tessus]

Also changing the issue title is not ok. nextcloud is the only web application which is NOT working on my server.

[MorrisJobke]

Also changing the issue title is not ok. nextcloud is the only web application which is NOT working on my server.

We never claimed, that it works with HTTP2 ... so I rephrased to make it so. Nevertheless a "does not work" is really vaguely. Could you describe this on a per request basis? Maybe do an upload via curl and check if this works? Is this only the web UI that does not work? Do the mobile apps still work? Does the desktop client still works?

[MorrisJobke]

cc @oparoz because you seem to have tested this already.

[tessus]

Thank you for opening it again.

Agreed, my statement was vague, but the entire Web UI was basically not working.

The clients are still working, because they are using http/1.1. So yes, the Web UI was the only thing that was not working. As you can imagine I had to deactivate h2 as quickly as possible again.

If you want me to test something, please give me a detailed list of commands I should run or what kind of GUI tests I should perform.
(I will have to switch to h2, run the tests, switch back to http/1.1)

I can also create a user for you on my nextcloud instance. What I cannot do is leave the setup on h2.
People are using this nextcloud instance, which means I can't afford having a broken UI.

So we have to coordinate these tests somehow.

The js error I mentioned above does not occur currently (it's only there when h2 is activated):

Shutting down notifications: [0] error  
_onFetchError
j 
fireWith
x 
b/<

[oparoz]

Apache 2.4.25 has lots of issues with HTTP/2, but I don't see why Nextcloud would fail and not other websites. The browser debug console should give you more information on what's going on when the GUI makes requests.

[tessus]

Apache 2.4.25 has lots of issues with HTTP/2

You are correct, which is why I'm not using the mod_httpd2 module that comes with 2.4.25.
I'm using version 1.10.4 of mod_httpd2, which is less than a week old.

The browser debug console should give you more information

I will collect this information during my next test window. Any other information I should collect while I'm at it?

[timreeves]

I run several NC installations on Plesk-managed servers using nginx with HTTP/2 - no problems. Perhaps you might consider shifting to nginx?
nginx-plesk-phpfpm-nextcloud-plesk-onyx.conf.txt
Have uploaded my tried-and-tested nginx config (just change the socket address if not using Plesk PHP), to make the migration more tempting.

[tessus]

@timreeves No, I have not. Thank you for the config file, but asking someone to switch web server is almost the same thing as asking to switch operating systems, or replace vi with emacs or the other way around. I think you get where I'm going with this.

[MorrisJobke]

Okay - then close this as "apache http2 support is not good enough" because we can't do much here

[oparoz]

Just to be clear, there is no problem running Nextcloud on Apache using HTTP/2. 

This is just a configuration issue with a specific setup.

[tessus]

I disagree.

If I have 3 domains with the same Apache virtual host setup, the same SSL config, the same certificate, and only one of them is not working, what does this tell you? What does logic dictate?

The only specific setup is nextcloud. And it cannot be a config issue, since the same config works for all oher domains.

Maybe you remember this ticket, when an Enterprise customer is complaining about it.

[MorrisJobke]

The only specific setup is nextcloud. And it cannot be a config issue, since the same config works for all oher domains.

Keep always in mind, that you cannot always invert logic. It could also be, that one specific configuration flag is missing or a feature is misimplemented there, that is not needed by the other sides but by nextcloud.

As in the forums there are other setups that work fine here, I would first of all try to get this solved in the forums and ask the others for their configuration before opening a bug, because it seems that there is no fundamental problem with Nextcloud here, but how it is set up.

And please also keep in mind: I'm currently sitting here on a Sunday and replying to you in my spare time. Getting complains and then still spend some of my time is not the stuff I want to do all the time. I do this, because I also like the project and do my best to help as many people as possible. I could also immediately stop doing this.

[tessus]

It could also be, that one specific configuration flag is missing or a feature is misimplemented there, that is not needed by the other sides but by nextcloud.

Yes this could be the case, but most likely isn't. Unless the forum entries are wrong or incomplete.

I would first of all try to get this solved in the forums and ask the others for their configuration before opening a bug, because it seems that there is no fundamental problem with Nextcloud here, but how it is set up.

That's the thing. I did check the forums, before I opened the bug, and there are only 2 directives discussed when it comes to h2: ProtocolsHonorOrder and Protocols. Although I believe that the ProtocolsHonorOrder is unnecessary I still set it so that I have a setup proven to be working.

I also ran debugging sessions on the httpd server process and the mod_http2 module. Furthermore I checked the h2 frames and everything seems to be in order.

I'm currently sitting here on a Sunday...

I really appreciate that you answer my questions on a Sunday, but I can also wait. I never said I needed an answer right away. I know how busy a project can be and that finding solutions is not always trivial. I didn't complain that it is not working, I merely stated that I didn't think closing the bug was the right choice.

I hoped that a developer ran into a similar problem and could help me out. Just because it is not in the forums doesn't mean that a similar thing hasn't happened before and wasn't solved by a developer without adding the solution to the documentation or forum. That's all.

[Obihoernchen]

I have similar issues with HTTP/2!

My setup:

• openSUSE 42.2
• Apache 2.4.26 with event MPM
• php7-fpm (7.0.7)
• HTTP/2 (Protocols h2 http/1.1)
• Nextcloud 12.0.0

I'm using this setup with ~10 different sites without a problem, but Nextcloud has issues with HTTP/2.

If I use Protocols h2 http/1.1 big downloads (>512MB) don't work. Nextcloud suddenly stops to send data after a few seconds and the downloaded file is incomplete. Unfortunately I can't find any valuable log entries (no real error etc.).
If I change my Nextcloud vhost configuration to: Protocols http/1.1 it works 100% of time!

I confirmed this several times (10+) with curl, Firefox, Chrome and 3 different internet connections. Forcing the client to use HTTP/1.1 works as well.

Anything I can do to help debugging this issue?

[j-ed]

I can confirm that problem on my system with Apache2 v2.4.26, mod_http v1.10.6 and libnghttp2 v1.23.1 with Nextcloud 12.0.0. It might be of interest that I can see AH00052: child pid 3822 exit signal Segmentation fault (11) messages in the error_log file as soon as you're going to login Nextcloud using http2 protocol. As soon as I disable http2 everything works like a charm again.
My assumption is that Nextcloud somehow causes mod_http2 to crash. Due to the fact that mod_http2 should be able to handle any kind of requests in a proper way, my bet is that an undiscovered problem exist either in mod_http2 or libnghttp2.

[icing]

Thanks @j-ed! Someone is talking to the developer! Yay! The probability of solving this has gone way up.

If someone else besides @j-ed is able to reproduce and can give an extract from the error log or even a full threaddump of a core, that would be most helpful.

[icing]

Just as an FYI for all interested parties here:

The crash happens in libphp5 because enabling h2 in Apache httpd drags php5 into a multi-threaded execution model it is not prepared to handle. This is a known issue. The advice by Apache has been for a long time to switch php processing to much better and more performance models as documented in https://wiki.apache.org/httpd/php

So, nextcloud is fine, mod_http2 is fine. It is the legacy of 20 years of successful, productive use that keeps biting us. So be it.

Thanks to everyone who helped resolve this.

[Obihoernchen]

Mhh I think this ^ is another issue.
I don't use php5. I use Apache mod_proxy_fcgi and php-fpm (PHP7 version), so Apache shouldn't load any php libs.

[icing]

The issue where j-ed provided logs and a stacktrace definitely was php5.

If you could provide the same over at mod_h2 github, I'll look into it.

Am 24.06.2017 um 00:18 schrieb Obihörnchen [e-mail]:

Mhh I think this ^ is another issue.
I don't use php5. I use Apache mod_proxy_fcgi and php-fpm (PHP7 version), so Apache shouldn't load any php libs.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[j-ed]

@Obihoernchen Please check, if you can find segmentation fault entries in the web servers log files.
If yes, you should make sure that a core file is created by following the steps described on this website: http://mattharris.org/2016/02/09/http-2-4-ah00051-child-pid-22471-exit-signal-segmentation-fault-11/