Ingress-nginx: gRPC servers cannot use dynamic HPACK

[stevewolter]

NGINX Ingress controller version: 0.13.0
Kubernetes version (use kubectl version): v1.8.8-gke.0
Environment: GKE

What happened:

I ran a golang gRPC server behind an nginx ingress. Many (but not all) RPCs fail with 502. nginx logs show "upstream sent invalid http2 table index: 64 while reading response header from upstream".

What you expected to happen:

No error.

How to reproduce it (as minimally and precisely as possible):

(since this is an upstream bug, I'm not including full repro)

Anything else we need to know:

This is a bug in nginx (https://trac.nginx.org/nginx/ticket/1538#ticket), I'm posting a bug here for reference.

[aledbf]

@stevewolter thank you for the report

[aledbf]

Closing. Please check response in nginx ticket

[mumoshu]

@stevewolter @aledbf Hi. You concluded this to be a bug in nginx. JFYI but it seems like one in grpc-go.

nginx explicitly announces that it does not support dynamic header compression by sending the SETTINGS_HEADER_TABLE_SIZE value set to 0, see ​here. Any attempt of an upstream server to use indexes from the dynamic range is a bug in the upstream server
https://trac.nginx.org/nginx/ticket/1538#ticket

We already have an issue in https://github.com/grpc/grpc-go/issues/1928, and the PR is in-p at https://github.com/grpc/grpc-go/pull/2045

That being said, using a grpc-go built from the PR head would solve this issue.

Anyway, this is a bug outside of ingress-nginx. I just wanted to leave a note 😉

[stevewolter]

Thanks for the pointers, much appreciated.

[aledbf]

There is a workaround that fixes this issues https://trac.nginx.org/nginx/ticket/1397#comment:13
(I hope NGINX 1.15.3 will include the fix)

[keatz55]

Any updates on this? I am trying to use ArgoCD behind some ingress rules and this is preventing me from using the ArgoCD CLI which depends on GRPC.

[aledbf]

@keatz55 please use 0.19.0. That version already uses NGINX 1.15.3

[keatz55]

@aledbf I see that the change made it into the 1.15.3 release and I am using 0.19.0, however, I am still getting the following error while trying to use GRPC:

[error] 1787#1787: *403612 upstream sent invalid http2 table index: 64 while reading response header from upstream