Exist: Packagemanager can't read repo

[ahenket]

eXist-db 4.3.1 - Dashboard 1.1.0 - any platform
repo 0.6.0

When I try to get to packages from our own repository (http://decor.nictiz.nl/apps/public-repo-dev) I get a 404 in the package manager. Looking a little deeper the problem is that when it tries to load the apps.xml list, it interprets that list to be text/xml instead of application/xml.

This happens in function packages:public-repo-contents in the line

let $data := http:send-request($request)

You can replay that in Postman/curl and see what comes back (application/xml) using a simple http GET on:

http://decor.nictiz.nl/apps/public-repo-dev/public/apps.xml?version=4.3.1&source=

But you when http:send-request() executes, you'll get:

<hc:response xmlns:hc="http://expath.org/ns/http-client" status="200" message="OK" spent-millis="136">
    <hc:header name="server" value="nginx/1.10.2"/>
    <hc:header name="date" value="Mon, 30 Jul 2018 01:55:16 GMT"/>
    <hc:header name="content-type" value="text/plain; charset=utf-8"/>
    <hc:header name="transfer-encoding" value="chunked"/>
    <hc:header name="connection" value="keep-alive"/>
    <hc:header name="x-xquery-cached" value="true"/>
    <hc:body media-type="text/plain"/>
</hc:response>&lt;apps version="4.3.1"&gt; ... &lt;/apps&gt;

To work around this problem I've replaced the call http:send-request($request) with httpclient:get(xs:anyURI($url),false(), $request) because the older HTTP client module will accept the body as XML.

Reproduction code:

xquery version "3.0";

declare namespace http = "http://expath.org/ns/http-client";

let $base   := 'http://decor.nictiz.nl/apps/public-repo-dev'

let $url    := $base || "/public/apps.xml?version=4.3.1&source=" || util:system-property("product-source")

let $request :=
    <http:request method="get" href="{$url}" timeout="10">
        <http:header name="Cache-Control" value="no-cache"/>
    </http:request>

let $expath-response    := httpclient:get(xs:anyURI($url),false(), $request)
let $exist-response     := http:send-request($request)

return
    <x url="{$url}">
        <!-- response by expath http client -->
        {$expath-response}
        <!-- response by exist-db http client -->
        {$exist-response}
    </x>

[joewiz]

Pardon if I missed it, but what version of the public-repo app are you running on the server?

[ahenket]

I didn't mention that part: updated the issue. We based it off 0.6.0

[adamretter]

So cURL shows the following response:

$ curl -v "http://decor.nictiz.nl/apps/public-repo-dev/public/apps.xml?version=4.3.1&source="
*   Trying 37.61.206.75...
* TCP_NODELAY set
* Connected to decor.nictiz.nl (37.61.206.75) port 80 (#0)
> GET /apps/public-repo-dev/public/apps.xml?version=4.3.1&source= HTTP/1.1
> Host: decor.nictiz.nl
> User-Agent: curl/7.50.3
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.10.2
< Date: Mon, 30 Jul 2018 15:57:08 GMT
< Content-Type: text/plain; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: XSESSIONID=1i50ndtraz4wt4tgnah8xknnl;Path=/
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-XQuery-Cached: true
< 
<apps version="4.3.1">
...

It seems that the public-repo on your server is returning the data as text/plain. This is almost certainly why the EXPath HTTP Client isn't giving you the results that you expect.

You said above:

it interprets that list to be text/xml instead of application/xml.

However, can I assume that based on the cURL output, that you meant to say "text/plain" and not "text/xml"?

So I think... you're receiving text but you want XML? You have two options here:

1. Fix the public-repo app so that it returns XML and not text.
2. Coerce the EXPath HTTP Client to parse the result as XML even though the server is telling it it is text (or use fn:parse on the result).

IMHO you should go for (1), as (2) is just putting a band-aid over the problem.

[adamretter]

Looks like I discovered something similar before: https://github.com/eXist-db/public-xar-repo/issues/17

@ahenket what happens if you access the eXist-db server URL directly for "http://decor.nictiz.nl/apps/public-repo-dev/public/apps.xml?version=4.3.1&source=" i.e. from the server itself, likely something like http://localhost:8080/exist/apps/public-repo-dev/public/apps.xml?version=4.3.1&source=

[ahenket]

I checked using postman and found the return type to be application/xml. I checked the server and list.xql has method xml and explicitly sets Content-Type application/xml

I’m not sure what more I can do server side

[adamretter]

@ahenket Okay but something must be confusing us here. As you seem to have different results to me, what do you get when you run:

curl -v "http://decor.nictiz.nl/apps/public-repo-dev/public/apps.xml?version=4.3.1&source="

Also what do you get when you run that on the server itself as:

curl -v "http://localhost:8080/exist/apps/public-repo-dev/public/apps.xml?version=4.3.1&source="

[ahenket]

< HTTP/1.1 200 OK
< Date: Tue, 31 Jul 2018 04:10:00 GMT
< Set-Cookie: XSESSIONID=16012xe6vul9q9cni1ifexnvr;Path=/
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Type: text/plain
< X-XQuery-Cached: false
< Transfer-Encoding: chunked
< Server: Jetty(8.1.9.v20130131)
< 
{ [data not shown]
<apps version="4.3.1" test="">

I've added that test="" thing to see if I was poking the right module. The module is called list.xql and starts with:

declare namespace list="http://exist-db.org/apps/public-repo/list";
declare namespace output="http://www.w3.org/2010/xslt-xquery-serialization";

import module namespace config="http://exist-db.org/xquery/apps/config" at "config.xqm";

declare option output:method "xml";
declare option output:media-type "application/xml";

If I call apps/public-repo-dev/modules/list.xql directly I get application/xml. So apparently when I call it through the main controller.xql that does a forward it falls back to text/plain?

else if ($exist:path = "/public/apps.xml") then
    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
        <forward url="{$exist:controller}/modules/list.xql"/>
    </dispatch>

[ahenket]

Ok so chewing some more on that last part I've updated the controller.xql to say:

else if ($exist:path = "/public/apps.xml") then (
    response:set-header('Content-Type', 'application/xml'),

    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
        <forward url="{$exist:controller}/modules/list.xql"/>
    </dispatch>
)

That does the trick and I don't need a client patch anymore. That is very good news, but leaves on the table why the controller.xql would override the Content-Type set by list.xql.

[adamretter]

@ahenket I don't know why the controller.xql is overriding the media-type. That would be a question for @wolfgangmm.

Would you be willing send a PR to fix your issue upstream in the public-xar-repo app? See this issue report - https://github.com/eXist-db/public-xar-repo/issues/17

[ahenket]

Not quite sure, but I think I've done what you asked and added the last tidbit to that issue under public-xar-repo app

[adamretter]

@ahenket thanks. Would you be able to open a PR (Pull Request) against the public-xar-repo to fix the problem and close that issue?