Orientdb: OrientDB no longer works behind NGINX reverse proxy.

[mmacfadden]

OrientDB Version, operating system, or hardware.

• v2.2.8

  Operating System

• [x] Linux

• [ ] MacOSX
• [ ] Windows
• [ ] Other Unix
• [ ] Other, name?
  

  Expected behavior and actual behavior

The issue I am having is that somewhere after 2.2.0, trying to use nginx as reverse proxy for OrientDB no longer works. When I try to log into the database I get the following error in the browsers:

https://localhost/orientdb/command/-9199765851322639105/sql/-/-1?format=rid,type,version,class,graph net::ERR_INCOMPLETE_CHUNKED_ENCODING

In the NGINX console i get:

2016/09/02 06:16:34 [error] 8#8: *2 upstream prematurely closed connection while reading upstream, client: 172.18.0.1, server: localhost, request: "POST /orientdb/command/-9199765851322639105/sql/-/20?format=rid,type,version,class,graph HTTP/1.1", upstream: "http://172.18.0.5:2480/command/-9199765851322639105/sql/-/20?format=rid,type,version,class,graph", host: "localhost", referrer: "https://localhost/orientdb/studio/index.html"

And in OrientDB console I get:

2016-09-02 06:45:53:254 SEVER Internal server error:
java.net.SocketException: Socket closed [ONetworkProtocolHttpDb]

This worked in version 2.2.0, but when I upgrade to 2.2.5 it stopped working. It is still not working in 2.2.8.

I have done a bunch of reading and it seems that this is likely to do some HTTP streaming / chunked encoding issues. I noticed that proxies are very sensitive to this, and I also see that in recent version orient has updated how it is doing response streaming.

Steps to reproduce the problem

I am going to try to create a simple set of docker containers that reproduce the problem, and also narrow down which 2.2.x version cause the problem.

[mmacfadden]

@tglman So far I have found that the reverse proxy setup works in 2.2.3 but is broken in 2.2.4. I have a set of docker-compose files which demonstrates both the working and not working versions. So the bug was introduced in 2.2.4. I am going to post them somewhere.

[mmacfadden]

It also looks like between 2.2.3 and 2.2.4 is where the streaming functionality was introduced. For example commit 4a9fec067b03ba576a6a023d60577421c94b60e1.

So I am pretty sure my guess around the issue is correct.

[mmacfadden]

I can confirm that setting network.http.streaming=false resolves the issue, which is what I have done for now.

[mmacfadden]

I have posted a set of docker containers the show the code working in 2.2.3, not working in 2.2.4 and working in 2.2.4 with streaming disabled.

You can check out the project here:

https://github.com/mmacfadden/orientdb-reverse-proxy

[lvca]

Good. So I'm closing this issue, ok?

[mmacfadden]

@lvca I don't know that I would close the issue. I think there still may be a bug in the new streaming feature. I have been reading up and using the chunked transfer encoding for HTTP is pretty tricky in a lot of system. There were a lot of systems that didn't work with streaming HTTP when behind a proxy. In some cases the proxy was to blame, but in most cases the HTTP Service (e.g. OrientDB in this case) was not 100% implementing the HTTP 1.1 spec properly.

I assume that you all have implemented the streaming feature because you feel it is valuable. At this point I have had to disable it. I think we should leave it open until we determine if the issue is with OrientDB or NGINX. Since NGINX is a VERY common system and has supported streaming encoding for a long time, I think it is more likely that there is something not quite right with OrientDBs implementation of the streaming functionality. Like I said I have seen a lot of systems who implemented streaming for the first time and had an issue with proxying.

I could be wrong though, but until we invetigate a little more, I wouldn't close the issue, since there could actually be an issue with OrientDB worth investigating. I do have a workaround for now but I am willing to do some more digging if you are willing to keep the bug open ;-)

[smolinari]

If I could put in my 2 cents. Since the root cause hasn't been clearly determined, the issue should stay open, until it is.

Scott

[lvca]

Sorry I thought everything worked by using a flag, but you're right this is still a bug.

[nagarajasr]

@mmacfadden hi tried your docker setup https://github.com/mmacfadden/orientdb-reverse-proxy
and it seems to work fine for me. the "not working" link does in fact work, and this is what i see in the logs:

proxy_1          | 2016/09/16 11:59:00 [warn] 5#5: *5 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/00/0000000001 while reading upstream, client: 192.168.242.1, server: localhost, request: "GET /orientdb_2_2_4/studio/scripts/scripts.js HTTP/1.1", upstream: "http://172.18.0.3:2480/studio/scripts/scripts.js", host: "192.168.242.100:8080", referrer: "http://192.168.242.100:8080/orientdb_2_2_4/studio/index.html"

(note: i'm using docker-machine and docker-compose on windows 7 host)

[tglman]

Hi All,

We did some changes on the support of streaming, and some fixes on the http protocol as well, this should not be a problem anymore from 2.1.10.

do you confirm ?

[voondo]

Unfortunately, it is still an issue with 2.2.12-SNAPSHOT (commit 8373f0b261d196ee8af2c742043a815a88685244) , nginx 1.11.3 and _network.http.streaming=true_ (works fine if _false_)

nginx log:

[error] 404#0: *57 upstream prematurely closed connection while reading upstream, client: [MYIP], server: [MYSERVER], request: "POST /api/command/devaprint-production/sql/-/20?format=rid,type,version,class,graph HTTP/2.0", upstream: "http://127.0.0.1:10480/command/devaprint-production/sql/-/20?format=rid,type,version,class,graph", host: "[MYSERVER]", referrer: "https://[MYSERVER]/"

orientdb log:

Remote client connected from: OClientConnection [id=20, source=?, since=1476193675217]
Internal server error:
                                                            java.net.SocketException: Socket closed

[mmacfadden]

@voondo Just so we can start homing in on this. What OS / Browser are you using.

I was on a Mac primarily using Chrome. Another of our developers was on Windows with Chrome. I don't think it has to do with the browser, but just wondering.

[voondo]

@mmacfadden Chrome / Linux

[tglman]

hi @mmacfadden, @voondo,

Can you try the 2.2.14, we fixed some wrong value in flags, the issue should be gone now.

Regards

[mmacfadden]

I will test this out shortly.

[tglman]

hi @mmacfadden,

Any new on this ?

Bye

[robfrank]

tested with 2.2.20: all work fine