Exist: REST interface put method failing

Created on 3 Oct 2018  路  17Comments  路  Source: eXist-db/exist

What is the problem

We have code that copies files from one server to the other using the standard REST interface functionality eXist provides. Here is a dressed down code example:

let $full-path-source := '/db/TEST/somefile.xml'
let $full-path-dest := $full-path-source

let $server := 'server-name-or-ip' 

let $request :=
  <hc:request href="http://{$server}:8080/exist/rest{$full-path-dest}" method="put" auth-method="basic" 
      username="admin" password="adminpassword" send-authorization="true">
    <hc:body media-type="{xmldb:get-mime-type(xs:anyURI($full-path-source))}"/>
  </hc:request>

return
  hc:send-request($request, (), doc($full-path-source))

The environment is all ok as far as I can see. The source file exists, is well-formed, the destination collection exists and is writable.

Since using 4.4.0 there are problems with this:

  • When the file to send is bigger than some threshold (an XML file of 84Kb fails, a very small one succeeds, usually...) an error pops up
  • The error is, rather puzzling: Error serializing the body content (on the hc:send-request()).
  • But there is nothing wrong with the file and why wouldn't it be able to serialize? It seems like a wrong message for some other problem occurring.
  • Very rarely the bug does not happen and the file transfers...
  • The source is a WIndows machine, the destination a Unix machine, all running 4.4.0
  • To make it even more strange: When I do the same with a Windows machine as destination, my whole source system hangs up
  • There is nothing in the exist.log about this
  • We run the script from within oXygen but also using a rest call to it
  • We had no problems with 4.3.1

What did you expect

File copied to remote system

Describe how to reproduce or add a test

Have two separate servers running eXist, preferably the source on Windows and the destination on Unix
Adapt the script above with the right stuff (address, password, etc).
Try it to run on the source machine

Context information

Please always add the following information

  • eXist version: 4.4.0 (Build 201809211803)
  • Java version 1.8.0_181
  • Operating system W10 (source) and Unix (dest)
  • 64 bit
  • JAR installed
  • Any custom changes in e.g. conf.xml?

    • Data and Journal directory moved to outside eXist installation directory

    • Global cache module enables

    • RestXQ and expath auto deploy trigger disabled

bug
Source
picture xatapult

Most helpful comment

@eriksiegel Okay I sent fixes for the Preemptive Authentication stuff. It looks to me like that can't have ever worked in the EXPath HTTP Client. So now that Preemptive works, you should not see the problem, and that workaround will work for you. Unfortunately the underlying issue is still there when not using preemptive auth.

Fixing the other stuff might be much more difficult. If it involves fixes to the HTTP Client itself, I would be inclined to leave that until we deliver and implement HTTP Client 2.0.

picture adamretter on 11 Oct 2018
❤1 👍1

All 17 comments

Hi Eric, just to make sure: are these machines behind a reverse-proxy, could that limit the max size allowed for file transfers (i know nginx does that by default)?

duncdrum picture duncdrum on 3 Oct 2018

@duncdrum Maybe, I'll check. But it worked before, so... sounds unlikely. And anyway, the error message doesn't make sense if that's the cause.

Another problem could be in the http client module I'm using. The native hc module is no longer recommended I think?

The standard conf.xml is a bit schizophrenic about it: It flags it as deprecated and the entry is commented out, but it also has a entry that enables it... Maybe that's something to improve in the standard distribution?

Anyway, I'm going to try the expath http client also.

xatapult picture xatapult on 3 Oct 2018

Using the expath module makes no difference

there is no NGINX or other proxy in the way

xatapult picture xatapult on 3 Oct 2018

I'm no longer sure about the "since 4.4.0" statement I made: it was used recently only on a 4.4.0 installation, so it might be in older versions as well.

xatapult picture xatapult on 8 Oct 2018

@eriksiegel So to be clear. Do you have exactly the same problem with the legacy HTTP module and the EXPath HTTP module?

Could you share your simple example that uses the EXPath HTTP module?

adamretter picture adamretter on 8 Oct 2018

@adamretter Ah, good you ask. I thought I used the legacy http client but it turned out to be the EXPath client after all, right from the beginning. Confusion on my side and sorry I didn't amend this issue once I realized it.

What about the example in the first comment? That's code that pretty consistently runs into errors on my machine(s). It uses the EXPath client.

xatapult picture xatapult on 9 Oct 2018

@eriksiegel, okay as I suspected. Can you make it super simple for me to reproduce so I can send my time on fixing the issue instead of setting up for it. Would you be able to send me a zip file with the collection structure (and just a very few XML files!) and the XQuery script in it. Before sending, can you just test that zip file against a clean eXist-db to make sure it reproduces?

If you can get me that, I can probably fix it quickly...

adamretter picture adamretter on 9 Oct 2018

@adamretter I'll try to do this somewhere this week. You need at least two machines I suspect.

We've also encountered something similar when using eXist's ANT extensions to store data on a remote machine. Scripts worked before, now lots of errors. It might be linked. I'll investigate.

xatapult picture xatapult on 9 Oct 2018

Ok, I've set up a test situation, here is what I've done:

  • I have to W10 machines, all updates installed, latest Java 1.8.0_181
  • Both are on my own LAN, no reverse proxies, no Apache, no NGINX
  • I installed eXist 4.4.0 (using the 4.4.0 install .jar file) on both machines without changing defaults, except the directory where things are installed (to prevent overwriting my "real" installation)
  • Started both up, not as a service
  • Created a collection /db/TRANSFER on both (as admin)
  • On machine A populated the /db/TRANSFER collection with files in the attached zip: TRANSFER.zip
  • Made the test-sync-error.xql file executable for all
  • Ran the XQuery script on machine A. I tried both from the REST interface (in browser http://localhost:8080/exist/rest/db/TRANSFER/test-sync-error.xql) and from oXygen. Same results.

    • If you want to do this on your own system be sure to update the destination server's ip and password in the script. Its all in variables at the top.

Results:

For small files (transferring smallfile.xml), success. But always two suspicious messages in the exist.log file: 

2018-10-09 14:11:50,282 [qtp1268841662-53] INFO  (NativeBroker.java [removeXMLResource]:2681) - Removing document smallfile.xml (922) ... 
2018-10-09 14:11:53,132 [qtp1268841662-76] WARN  (TransactionManager.java [close]:186) - Transaction was not committed or aborted, auto aborting!

For transferring bigger files (transferring ~550Kb mediumfile.xml): The source system hangs up completely (I haven't noticed a timeout but I'm noit very patient :) ) and this message in the exist.log file:

2018-10-09 14:06:31,509 [qtp1268841662-70] WARN  (TransactionManager.java [close]:186) - Transaction was not committed or aborted, auto aborting!

@adamretter Is this enough information?

xatapult picture xatapult on 9 Oct 2018

@eriksiegel looks good to me. I will try and take a look tomorrow...

adamretter picture adamretter on 9 Oct 2018

@eriksiegel smallfile.xml is missing from your zip. It only contains mediumfile.xml (566KB) and bigfile.xml (2.1MB).

When running with mediumfile.xml I get the error Broken pipe (Write failed) with the stacktrace:

org.xmldb.api.base.XMLDBException: exerr:ERROR Error serializing the body content [at line 18, column 5]
    at org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:197)
    at org.exist.xmldb.LocalXPathQueryService.lambda$execute$1(LocalXPathQueryService.java:161)
    at org.exist.xmldb.function.LocalXmldbFunction.apply(LocalXmldbFunction.java:46)
    at org.exist.xmldb.AbstractLocal.withDb(AbstractLocal.java:196)
    at org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:160)
    at org.exist.client.QueryDialog$QueryThread.run(QueryDialog.java:571)
Caused by: org.exist.xquery.XPathException: exerr:ERROR Error serializing the body content [at line 18, column 5]
    at org.expath.exist.SendRequestFunction.sendRequest(SendRequestFunction.java:158)
    at org.expath.exist.SendRequestFunction.eval(SendRequestFunction.java:134)
    at org.exist.xquery.BasicFunction.eval(BasicFunction.java:74)
    at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:41)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276)
    at org.exist.xquery.EnclosedExpr.eval(EnclosedExpr.java:85)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276)
    at org.exist.xquery.ElementConstructor.eval(ElementConstructor.java:329)
    at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.XQuery.execute(XQuery.java:261)
    at org.exist.xquery.XQuery.execute(XQuery.java:185)
    at org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:194)
    ... 5 more
Caused by: org.expath.httpclient.HttpClientException: Error serializing the body content
    at org.expath.httpclient.impl.ApacheHttpConnection.connect(ApacheHttpConnection.java:124)
    at org.expath.httpclient.impl.HttpRequestImpl.send(HttpRequestImpl.java:56)
    at org.expath.exist.SendRequestFunction.sendOnce(SendRequestFunction.java:173)
    at org.expath.exist.SendRequestFunction.sendRequest(SendRequestFunction.java:152)
    ... 26 more
Caused by: java.io.IOException: Error serializing the body content
    at org.expath.httpclient.impl.ApacheHttpConnection$RequestBodyProducer.writeTo(ApacheHttpConnection.java:625)
    at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:73)
    at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
    at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.expath.httpclient.impl.ApacheHttpConnection.connect(ApacheHttpConnection.java:109)
    ... 29 more
Caused by: org.expath.httpclient.HttpClientException: Error serializing the result
    at org.expath.httpclient.impl.SinglePartRequestBody.serialize(SinglePartRequestBody.java:198)
    at org.expath.httpclient.impl.ApacheHttpConnection$RequestBodyProducer.writeTo(ApacheHttpConnection.java:622)
    ... 41 more
Caused by: org.expath.tools.ToolsException: A problem occurred while serializing the node set: Broken pipe (Write failed)
    at org.expath.tools.model.exist.EXistSequence.serialize(EXistSequence.java:89)
    at org.expath.httpclient.impl.SinglePartRequestBody.serialize(SinglePartRequestBody.java:195)
    ... 42 more
Caused by: org.xml.sax.SAXException: Broken pipe (Write failed)
javax.xml.transform.TransformerException: Broken pipe (Write failed)
    at org.exist.util.serializer.SAXSerializer.characters(SAXSerializer.java:354)
    at org.exist.util.serializer.ReceiverToSAX.characters(ReceiverToSAX.java:120)
    at org.exist.storage.serializers.XIncludeFilter.characters(XIncludeFilter.java:148)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:306)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:130)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:90)
    at org.exist.storage.serializers.Serializer.serializeToReceiver(Serializer.java:1065)
    at org.exist.storage.serializers.Serializer.itemToSAX(Serializer.java:1028)
    at org.exist.storage.serializers.Serializer.toSAX(Serializer.java:921)
    at org.exist.util.serializer.XQuerySerializer.serializeXML(XQuerySerializer.java:70)
    at org.exist.util.serializer.XQuerySerializer.serialize(XQuerySerializer.java:50)
    at org.exist.util.serializer.XQuerySerializer.serialize(XQuerySerializer.java:36)
    at org.expath.tools.model.exist.EXistSequence.serialize(EXistSequence.java:87)
    ... 43 more
Caused by: javax.xml.transform.TransformerException: Broken pipe (Write failed)
    at org.exist.util.serializer.XMLWriter.characters(XMLWriter.java:325)
    at org.exist.util.serializer.IndentingXMLWriter.characters(IndentingXMLWriter.java:128)
    at org.exist.util.serializer.XMLWriter.characters(XMLWriter.java:334)
    at org.exist.util.serializer.SAXSerializer.characters(SAXSerializer.java:352)
    ... 63 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
    at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
    at org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
    at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
    at org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:122)
    at org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:179)
    at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:113)
    at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)
    at org.exist.util.serializer.XMLWriter.writeCharSeq(XMLWriter.java:572)
    at org.exist.util.serializer.XMLWriter.writeChars(XMLWriter.java:529)
    at org.exist.util.serializer.XMLWriter.characters(XMLWriter.java:323)
    ... 66 more
Caused by: org.exist.xquery.XPathException: exerr:ERROR Error serializing the body content [at line 18, column 5]
    at org.expath.exist.SendRequestFunction.sendRequest(SendRequestFunction.java:158)
    at org.expath.exist.SendRequestFunction.eval(SendRequestFunction.java:134)
    at org.exist.xquery.BasicFunction.eval(BasicFunction.java:74)
    at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:41)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276)
    at org.exist.xquery.EnclosedExpr.eval(EnclosedExpr.java:85)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276)
    at org.exist.xquery.ElementConstructor.eval(ElementConstructor.java:329)
    at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.LetExpr.eval(LetExpr.java:111)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.PathExpr.eval(PathExpr.java:276)
    at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
    at org.exist.xquery.XQuery.execute(XQuery.java:261)
    at org.exist.xquery.XQuery.execute(XQuery.java:185)
    at org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:194)
    at org.exist.xmldb.LocalXPathQueryService.lambda$execute$1(LocalXPathQueryService.java:161)
    at org.exist.xmldb.function.LocalXmldbFunction.apply(LocalXmldbFunction.java:46)
    at org.exist.xmldb.AbstractLocal.withDb(AbstractLocal.java:196)
    at org.exist.xmldb.LocalXPathQueryService.execute(LocalXPathQueryService.java:160)
    at org.exist.client.QueryDialog$QueryThread.run(QueryDialog.java:571)
Caused by: org.expath.httpclient.HttpClientException: Error serializing the body content
    at org.expath.httpclient.impl.ApacheHttpConnection.connect(ApacheHttpConnection.java:124)
    at org.expath.httpclient.impl.HttpRequestImpl.send(HttpRequestImpl.java:56)
    at org.expath.exist.SendRequestFunction.sendOnce(SendRequestFunction.java:173)
    at org.expath.exist.SendRequestFunction.sendRequest(SendRequestFunction.java:152)
    ... 26 more
Caused by: java.io.IOException: Error serializing the body content
    at org.expath.httpclient.impl.ApacheHttpConnection$RequestBodyProducer.writeTo(ApacheHttpConnection.java:625)
    at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:73)
    at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
    at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.expath.httpclient.impl.ApacheHttpConnection.connect(ApacheHttpConnection.java:109)
    ... 29 more
Caused by: org.expath.httpclient.HttpClientException: Error serializing the result
    at org.expath.httpclient.impl.SinglePartRequestBody.serialize(SinglePartRequestBody.java:198)
    at org.expath.httpclient.impl.ApacheHttpConnection$RequestBodyProducer.writeTo(ApacheHttpConnection.java:622)
    ... 41 more
Caused by: org.expath.tools.ToolsException: A problem occurred while serializing the node set: Broken pipe (Write failed)
    at org.expath.tools.model.exist.EXistSequence.serialize(EXistSequence.java:89)
    at org.expath.httpclient.impl.SinglePartRequestBody.serialize(SinglePartRequestBody.java:195)
    ... 42 more
Caused by: org.xml.sax.SAXException: Broken pipe (Write failed)
javax.xml.transform.TransformerException: Broken pipe (Write failed)
    at org.exist.util.serializer.SAXSerializer.characters(SAXSerializer.java:354)
    at org.exist.util.serializer.ReceiverToSAX.characters(ReceiverToSAX.java:120)
    at org.exist.storage.serializers.XIncludeFilter.characters(XIncludeFilter.java:148)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:306)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:273)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:130)
    at org.exist.storage.serializers.NativeSerializer.serializeToReceiver(NativeSerializer.java:90)
    at org.exist.storage.serializers.Serializer.serializeToReceiver(Serializer.java:1065)
    at org.exist.storage.serializers.Serializer.itemToSAX(Serializer.java:1028)
    at org.exist.storage.serializers.Serializer.toSAX(Serializer.java:921)
    at org.exist.util.serializer.XQuerySerializer.serializeXML(XQuerySerializer.java:70)
    at org.exist.util.serializer.XQuerySerializer.serialize(XQuerySerializer.java:50)
    at org.exist.util.serializer.XQuerySerializer.serialize(XQuerySerializer.java:36)
    at org.expath.tools.model.exist.EXistSequence.serialize(EXistSequence.java:87)
    ... 43 more
Caused by: javax.xml.transform.TransformerException: Broken pipe (Write failed)
    at org.exist.util.serializer.XMLWriter.characters(XMLWriter.java:325)
    at org.exist.util.serializer.IndentingXMLWriter.characters(IndentingXMLWriter.java:128)
    at org.exist.util.serializer.XMLWriter.characters(XMLWriter.java:334)
    at org.exist.util.serializer.SAXSerializer.characters(SAXSerializer.java:352)
    ... 63 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
    at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
    at org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
    at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
    at org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:122)
    at org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:179)
    at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:113)
    at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)
    at org.exist.util.serializer.XMLWriter.writeCharSeq(XMLWriter.java:572)
    at org.exist.util.serializer.XMLWriter.writeChars(XMLWriter.java:529)
    at org.exist.util.serializer.XMLWriter.characters(XMLWriter.java:323)
    ... 66 more
adamretter picture adamretter on 10 Oct 2018

So my working theory so far is this -

  1. The Client opens a HTTP connection to the server
  2. The Client starts sending data to the server
  3. The Server starts receiving the data from the client, reads the HTTP request headers and sees that client is not authorized.
  4. The Server responds HTTP 401 Unauthorized to the client
  5. The server closes the connection.
  6. The client, which is still writing data to the server, gets a Broken pipe (Write failed) error from the OS and throws a SocketException.

The reason this works for a small file is that the client completes sending the data by time (4) happens, and so (6) never occurs. For larger files the client can't send all of the data before the server closes the connection.

The question is, who is at fault, the client or the server?

adamretter picture adamretter on 10 Oct 2018

Okay I have been able to isolate the HTTP Client code from eXist-db and reproduce the issue simply with the following Java code acting as the Client:

public class Issue2186 {

    private static final Path file = Paths.get("/tmp/mediumfile.xml");

    private static HttpEntity getRequestBody() {
        final ContentProducer producer = os -> {
//            copyFast(file, os);

            // NOTE: shows the broken pipe error
            copySlow(file, os);
        };
        final EntityTemplate entityTemplate = new EntityTemplate(producer);
        entityTemplate.setContentType("application/xml");
        entityTemplate.setChunked(true);
        return entityTemplate;
    }

    private static void copyFast(final Path file, final OutputStream os) throws IOException {
        Files.copy(file, os);
    }

    private static void copySlow(final Path file, final OutputStream os) throws IOException {
        try(final LineNumberReader reader = new LineNumberReader(Files.newBufferedReader(file));
                final OutputStreamWriter writer = new OutputStreamWriter(new CloseShieldOutputStream(os), UTF_8)) {

            String line;
            while((line = reader.readLine()) != null) {
                writer.write(line + "\n");
            }
        }
    }

    public static void main(final String args[]) throws IOException {
        final CloseableHttpClient httpClient = HttpClients.createDefault();
        try {
            final HttpPut httpPut = new HttpPut("http://localhost:8080/exist/rest/db/mediumfile.xml");
            httpPut.setEntity(getRequestBody());

            final CloseableHttpResponse httpResponse = httpClient.execute(httpPut);
            try {
                System.out.println(httpResponse.getStatusLine());
            } finally {
                httpResponse.close();
            }
        } finally {
            httpClient.close();
        }
    }
}

Running the above code results in the output and then errors:

Oct 10, 2018 10:07:15 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:07:15 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Oct 10, 2018 10:07:15 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:07:15 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Oct 10, 2018 10:07:15 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:07:15 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Exception in thread "main" java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
    at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
    at org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
    at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
    at org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:122)
    at org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:179)
    at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
    at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
    at java.io.Writer.write(Writer.java:157)
    at Issue2186.copySlow(Issue2186.java:44)
    at Issue2186.lambda$0(Issue2186.java:26)
    at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:73)
    at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
    at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at Issue2186.main(Issue2186.java:55)

So whilst I can reproduce it, I am still not sure who is at fault here. The client or the server?

I think this will require me getting some clarification from the Apache HTTP Component project , and/or possibly the Jetty project?

adamretter picture adamretter on 10 Oct 2018

@adamretter Thanks for investigating so quickly. And: Oops. Sorry for omitting smallfile.xml. It was just a tiny file with a single (empty) XML element.

The Server starts receiving the data from the client, reads the HTTP request headers and sees that client is not authorized.

But the testscript sends authorization... (at least its specified). So where does it gets missing then?

And the EXPath http client worked fine in the past. So what's changed then in the newest release(s) of eXist that broke it?

xatapult picture xatapult on 11 Oct 2018

And the EXPath http client worked fine in the past. So what's changed then in the newest release(s) of eXist that broke it?

You mentioned 4.3.1 and 4.4.0. I can see no changes made to either the EXPath HTTP Client or the Jetty Server during that time. So I really can't see that anything could have broken it...

Interestingly I can see that the authentication is not being send pre-emptively, which I think it should be as you set send-authorization="true". The headers I see in reality are:

PUT /exist/rest/db/mediumfile.xml HTTP/1.1
Content-Type: application/xml; charset=UTF-8
Transfer-Encoding: chunked
Host: localhost:8080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_162)
Accept-Encoding: gzip,deflate

So we might have two different issues here:

  1. Why is the auth header not being pre-emptively sent when we set that?
  2. Why is it failing even with reactive authentication?

Regards (2), I did get a reply from my enquiry to the Apache HTTP Client mainly list about how to fix this -

https://lists.apache.org/thread.html/2b70b2c6a1f565d2c35cbf53d520d51337da2caeccd917dda366531d@%3Chttpclient-users.hc.apache.org%3E

adamretter picture adamretter on 11 Oct 2018

@eriksiegel Okay I sent fixes for the Preemptive Authentication stuff. It looks to me like that can't have ever worked in the EXPath HTTP Client. So now that Preemptive works, you should not see the problem, and that workaround will work for you. Unfortunately the underlying issue is still there when not using preemptive auth.

Fixing the other stuff might be much more difficult. If it involves fixes to the HTTP Client itself, I would be inclined to leave that until we deliver and implement HTTP Client 2.0.

adamretter picture adamretter on 11 Oct 2018
❤1 👍1

Okay so fixing the EXPath HTTP Client 1.0 implementation to correctly support the issue we have would be a lot of work. Instead, I think we will say that we won't fix it.

Instead my plan is to implement the EXPath HTTP Client 2.0, and when I do that, I will make sure to use the Asynchronous mode of the Apache HTTP Client.

adamretter picture adamretter on 11 Oct 2018
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Bpolitycki picture Bpolitycki  路  4Comments
dizzzz picture dizzzz  路  4Comments
opax picture opax  路  3Comments
cmil picture cmil  路  3Comments
wolfgangmm picture wolfgangmm  路  3Comments