Packages: Nginx 1.16.0 http2 is not available

[zdz2019]

CONFIG_NGINX_HTTP_V2 is enabled,
nginx -V ,Already contains "--with-http_v2_module"
After modifying the configuration file and enabling http2, the page will not open.
The same operation is no problem on centos7.
I think this error is related to libopenssl, but since I didn't get the error log, I am not sure what the problem is.
Software used: nginx 1.16.0 openssl 1.1.1b

[dengqf6]

Works for me.
What kind of error does your browser report?
Is Nginx running? Have you tried nginx -t?

[zdz2019]

I am using x86_64,
Nginx is working properly,
nginx -t is successful,
Only http2 is not working properly,
Browser prompts are not accessible,Just like offline.
Without using http2, everything is back to normal,
I did not add other patches, they are the default state.

[dengqf6]

Browser prompts are not accessible,Just like offline.

I need the error code. What browser are you using? e.g. for Chrome/Chromium there is a list of error codes on chrome://network-errors/
Does nginx report any errors at runtime? Have you checked system log?

[zdz2019]

Thank God, I finally found the problem.
Modify the "ssl_protocols" parameter in the configuration file,
I have been using "ssl_protocols TLSv1.2 TLSv1.3;"This is no problem on CentOS.
But openwrt can't, change it to "ssl_protocols TLSv1.3", restart nginx, then http2 will be available.
Thank you for your reply and gave me inspiration.

[zdz2019]

This fault is hard to find because there is no error log generated, no error code, and it may be necessary to fix the nginx or OpenSSL package on OpenWrt?

[dengqf6]

I don't have to disable TLSv1.2 to make http2 work.
Here is my config

listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers "TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-GCM-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256:EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:DHE+AESGCM:DHE:!RSA!aNULL:!eNULL:!LOW:!RC4:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!CAMELLIA:!SEED";
ssl_session_tickets off;

[zdz2019]

The domain name was found to be the same as centos, causing the website to resolve on vps.
I still can't enable http2 on openwrt, I recompile the test.
It seems that this problem is difficult to check.

[zdz2019]

What changes have you made to the openssl compilation option? Or just the default?

[dengqf6]

CONFIG_PACKAGE_libopenssl=y
CONFIG_OPENSSL_WITH_ASM=y
CONFIG_OPENSSL_WITH_DEPRECATED=y
CONFIG_OPENSSL_WITH_ERROR_MESSAGES=y
CONFIG_OPENSSL_WITH_TLS13=y
CONFIG_OPENSSL_WITH_SRP=y
CONFIG_OPENSSL_WITH_CMS=y
CONFIG_OPENSSL_WITH_EC=y
CONFIG_OPENSSL_WITH_CHACHA_POLY1305=y
CONFIG_OPENSSL_PREFER_CHACHA_OVER_GCM=y
CONFIG_OPENSSL_WITH_PSK=y
CONFIG_OPENSSL_ENGINE=y
CONFIG_PACKAGE_libopenssl-conf=y
CONFIG_PACKAGE_libopenssl-devcrypto=y

nginx -V: (Note that it's mvebu ARM but it reports x86_64 i.e. host architecture)

nginx version: nginx/1.16.0 (x86_64-pc-linux-gnu)
built with OpenSSL 1.1.1b  26 Feb 2019
TLS SNI support enabled
configure arguments: --target=arm-openwrt-linux --host=arm-openwrt-linux --build=x86_64-pc-linux-gnu --program-prefix= --program-suffix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/man --infodir=/usr/info --disable-nls --crossbuild=Linux::arm --prefix=/usr --conf-path=/etc/nginx/nginx.conf --with-http_ssl_module --add-module=/data/maintainer/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/nginx-ssl/nginx-1.16.0/nginx-naxsi/naxsi_src --with-ipv6 --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --add-module=/data/maintainer/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/nginx-ssl/nginx-1.16.0/nginx-headers-more --add-module=/data/maintainer/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/nginx-ssl/nginx-1.16.0/nginx-brotli --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-cc=ccache_cc --with-cc-opt='-I/data/maintainer/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/include -I/data/maintainer/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/include -I/data/maintainer/openwrt/staging_dir/toolchain-arm_cortex-a9+neon_gcc-7.4.0_musl_eabi/usr/include -I/data/maintainer/openwrt/staging_dir/toolchain-arm_cortex-a9+neon_gcc-7.4.0_musl_eabi/include/fortify -I/data/maintainer/openwrt/staging_dir/toolchain-arm_cortex-a9+neon_gcc-7.4.0_musl_eabi/include -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=hard -iremap/data/maintainer/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/nginx-ssl/nginx-1.16.0:nginx-1.16.0 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -fvisibility=hidden -ffunction-sections -fdata-sections -DNGX_LUA_NO_BY_LUA_BLOCK' --with-ld-opt='-L/data/maintainer/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib -L/data/maintainer/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/lib -L/data/maintainer/openwrt/staging_dir/toolchain-arm_cortex-a9+neon_gcc-7.4.0_musl_eabi/usr/lib -L/data/maintainer/openwrt/staging_dir/toolchain-arm_cortex-a9+neon_gcc-7.4.0_musl_eabi/lib -znow -zrelro -Wl,--gc-sections' --without-http_upstream_zone_module

[zdz2019]

Recompile the test and still can't use http2,
This time I opened the access log for nginx.
When http2 is enabled, the access log is empty.
This means that the browser can't access the site at all, but I don't know why.
Do other friends who also use the x86_64 platform have this problem?

[zdz2019]

I tried it again and again, and there was such an access log, and the website still could not be opened.
[18/May/2019:23:27:51 +0800] "GET / HTTP/2.0" 200 3598 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
[18/May/2019:23:27:52 +0800] "GET / HTTP/2.0" 200 3608 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
[18/May/2019:23:27:52 +0800] "GET / HTTP/2.0" 200 3602 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"

[dengqf6]

You have not told me what kind of error code your browser shows yet.

[zdz2019]

The browser has no error code, just can't access it.
Always displayed: ERR_SPDY_COMPRESSION_ERROR
No need to change other parameters, just delete http2 to resume access.

[dengqf6]

ERR_SPDY_COMPRESSION_ERROR is the said error code

[zdz2019]

Have tested that openssl supports the ALPN protocol, why is there a SDPY error?

[zdz2019]

Nginx compiler parameters are no problem, I did not add other changes, can use SSL and TLS1.3, can not use http2, can not find the reason.

[dengqf6]

@Ansuel

[Ansuel]

can this be tested with other device/browser?

[zdz2019]

The same is true for other browsers (IE 11/Edge on Windows 10). I can't solve this problem myself, so I am here to ask for help.

[zdz2019]

The latest test, http2 can be used normally on the bcm53xx platform, the device is phicomm-k3, using the default compilation parameters, the x86_64 platform still can not use http2

[dengqf6]

Have you tried different GCC or binutils version?

[zdz2019]

Use the default version, GCC_VERSION="7.4.0", BUNITILS_VERSION="2.31.1"

[ysc3839]

I have the same issue. OS is OpenWrt x86-64 snapshot, nginx is nginx-all-module_1.16.0-1_x86_64.ipk.
I replaced my domain name with myhost.example.com below.

/etc/nginx/nginx.conf

user nobody nogroup;
worker_processes  1;

pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    client_body_buffer_size 10K;
    client_header_buffer_size 1k;
    client_max_body_size 1G;
    large_client_header_buffers 2 1k;

    gzip on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 1;
    gzip_proxied any;

    brotli on;
    brotli_comp_level 6;
    brotli_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript image/svg+xml;

    ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_dhparam /etc/nginx/ssl/dhparam.pem;
    ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
    ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
    ssl_session_timeout 10m;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off; # Requires nginx >= 1.5.9
    ssl_stapling on; # Requires nginx >= 1.3.7
    ssl_stapling_verify on; # Requires nginx => 1.3.7

    include /etc/nginx/conf.d/*;
}

/etc/nginx/conf.d/h2test.conf

server {
    listen 23233 ssl http2;
    listen [::]:23233 ssl http2;
    server_name myhost.example.com;

    ssl_certificate /etc/nginx/ssl/myhost.example.com.cert.pem;
    ssl_certificate_key /etc/nginx/ssl/myhost.example.com.key.pem;

    root /www;
}

curl test output

$ curl -vv --http2 https://myhost.example.com:23233/
*   Trying <IP Address>...
* TCP_NODELAY set
* Connected to myhost.example.com (<IP Address>) port 23233 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=myhost.example.com
*  start date: Jun 17 12:58:40 2019 GMT
*  expire date: Sep 15 12:58:40 2019 GMT
*  subjectAltName: host "myhost.example.com" matched cert's "myhost.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e6f890b900)
> GET / HTTP/2
> Host: myhost.example.com:23233
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* Connection #0 to host myhost.example.com left intact
curl: (16) Error in the HTTP2 framing layer

And Chrome gives an ERR_SPDY_COMPRESSION_ERROR error.

[dengqf6]

May be a compiler optimization bug. You may try GCC 9.1

[ysc3839]

@dengqf6 Tested with gcc 9.1, same issue.

Advanced configuration options (for developers)
Toolchain Options
Binutils Version (Binutils 2.31.1)
GCC compiler Version (gcc 9.x)

[dengqf6]

Please report the issue upstream

[zuosc]

same problem +1

[D-Bood]

I have openwrt running on Phicomm N1 and the same problem.

p.s. I have solved the problem by compling from source code with Entware.

[dengqf6]

Please test if #9732 fixes this issue.

[val-kulkov]

Here is the relevant part of the log output from chrome://net-export that explains, from Chrome browser's standpoint, what is going wrong:

t=28541465 [st=18]    HTTP2_SESSION_RECV_SETTINGS
t=28541465 [st=18]    HTTP2_SESSION_SEND_SETTINGS_ACK
t=28541465 [st=18]    HTTP2_SESSION_RECV_SETTING
                      --> id = "3 (SETTINGS_MAX_CONCURRENT_STREAMS)"
                      --> value = 128
t=28541465 [st=18]    HTTP2_SESSION_UPDATE_STREAMS_SEND_WINDOW_SIZE
                      --> delta_window_size = 1
t=28541465 [st=18]    HTTP2_SESSION_RECV_SETTING
                      --> id = "4 (SETTINGS_INITIAL_WINDOW_SIZE)"
                      --> value = 65536
t=28541465 [st=18]    HTTP2_SESSION_RECV_SETTING
                      --> id = "5 (SETTINGS_MAX_FRAME_SIZE)"
                      --> value = 16777215
t=28541465 [st=18]    HTTP2_SESSION_RECV_WINDOW_UPDATE
                      --> delta = 2147418112
                      --> stream_id = 0
t=28541465 [st=18]    HTTP2_SESSION_UPDATE_SEND_WINDOW
                      --> delta = 2147418112
                      --> window_size = 2147483647
t=28541465 [st=18]    HTTP2_SESSION_RECV_SETTINGS_ACK
t=28541465 [st=18]    HTTP2_SESSION_CLOSE
                      --> description = "Framer error: 6 (DECOMPRESS_FAILURE)."
                      --> net_error = -363 (ERR_SPDY_COMPRESSION_ERROR)
t=28541465 [st=18]    HTTP2_SESSION_POOL_REMOVE_SESSION
t=28541465 [st=18] -HTTP2_SESSION

So far I have been unable to identify the root cause of this issue. I will see if can find some hints from nginx configuration and compilation logs.

[val-kulkov]

@dengqf6 : unfortunately no, #9732 (upgrading to 1.16.1) does not fix this issue.

[zuosc]

@ysc3839 have you solved it?

[ysc3839]

@zuosc No.

[qinkangdeid]

same problem +1

[avengermsoft]

same problem +1

[avengermsoft]

Does it work for anyone?

P/S/ Version = nginx/1.17.7-3 (x86_64-pc-linux-gnu)

[val-kulkov]

A ticket has been submitted upstream at https://trac.nginx.org/nginx/ticket/1928

[val-kulkov]

@heil @Ansuel : I would much appreciate if you could look into the Nginx ticket 1928 as referenced above. One of patches to compile Nginx on OpenWrt may be causing this present issue with http2 on x86_64. I am unable to see however how these patches may be causing the problem because they appear correct to me.

On the other hand, Nginx does not appear to have this problem with http2 when compiled natively. Therefore there must be something here in OpenWrt that triggers the problem, whether it is the patches or possibly some hardcoded constants in OpenWrt. I declare myself stumped at this present moment.

[val-kulkov]

After messing around a little bit more, I found it. Endianness is the culprit. Once Nginx's auto/endianness is brute-forced to set endianness to the "little endian", http2 starts working normally on x86_64 as it should.

@heil @Ansuel : do you have any suggestions how we should determine the endianness of the target platform while cross-compiling Nginx? The current test used by Nginx configure (in auto/endianness) relies on the execution of a compiled code, which predictably fails and therefore the fallback option of "big endian" is taken.

It is possible to make an OpenWrt-specific patch to detect x86_64 and set the endianness to "little endian", but this is inelegant and very x86_64-specific. Is there a more general approach we can use?

[peter-stadler]

I made a draft PR. It is working for x86_64 now and I suppose for all little endian systems, do you know a big endian system for testing?

[peter-stadler]

(Tested it for malta-be, looks good)

[avengermsoft]

Version = nginx/1.17.7-5 (x86_64-pc-linux-gnu)

Works. Many thanks!!!

[peter-stadler]

I am happy that it helps :-)
Edit: Kudos to @val-kulkov who did the main work, i made only the small patch after he tracked the issue down.

[Ansuel]

@hnyman @peter-stadler think this can be closed?