Conan: Run server behind reverse proxy with SSL offloading?

[SteffenL]

How would I run the Conan server behind a reverse proxy with SSL offloading, e.g. using IIS?

server.conf:

ssl_enabled: False
port: 9300
# Public port where files will be served. If empty will be used "port"
public_port: 80
host_name: my.domain

Remotes in conan.conf:

[remotes]
conan.io: https://server.conan.io
local: http://localhost:9300
sl_dev: https://my.domain

URL rewrites in web.config (IIS):

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="ReverseProxyInboundRule1" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="http://127.0.0.1:9300/{R:1}" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

This only works partially e.g. during uploads, because the first requests from the client are over HTTPS while the next ones are over HTTP. Changing public_port to 443 makes the server return URLs like http://my.domain:443/[snip].

I tried to fix this myself (see e6de2666400409e452ee36a4d31937fd3c4969f6), but I seem to be running into more issues.

[memsharded]

It is possible that the current configuration does not support this setup yet, up to my knowledge, conan has already been used with forward proxies only. On the other hand, a reverse proxy is supposed to be transparent for the client, so apparently, there is no reason why it shouldn't work.

I know you tried to change ssl_enabled=True before, but just in case, have you tried again since you added the host_name? I know nothing about IIS, but probably it could be reproduced with other reverse proxies too, it has to be investigated, thanks for reporting, cheers!

[SteffenL]

Thanks for your reply! Was hoping nobody had seen my initial experiments. :)

I have now made sure that it works fine when not using the proxy, and only using HTTP.

As soon as I use the proxy and still only HTTP, I get errors like this one on the server during upload:

127.0.0.1 - - [02/Mar/2016 14:55:26] "PUT /v1/files/pugg/1.0.0/demo/testing/export/conanmanifest.txt?signature=[snip] HTTP/1.1
" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 61412)
Traceback (most recent call last):
  File "c:\python27\lib\SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "c:\python27\lib\SocketServer.py", line 657, in __init__
    self.finish()
  File "c:\python27\lib\SocketServer.py", line 716, in finish
    self.wfile.close()
  File "c:\python27\lib\socket.py", line 283, in close
    self.flush()
  File "c:\python27\lib\socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10054] An existing connection was forcibly closed by the remote host
----------------------------------------

I am a bit confused about this since I thought it did work at some point initially. I disabled the software firewall on the server just in case.

I will experiment a little bit more with ssl_enabled and other things.

[memsharded]

Yes, I saw them! :) In fact, even if you made some mistake that you managed to solve yourself, it is better not to delete the comments, leave them as reference! Nothing to be afraid of ashamed of :)

I still cannot figure out what could be happening, so next steps will be trying to reproduce the issue. I have been having a look to nginx reverse proxy, it shouldn't be very difficult. The only problem is we are quite busy now trying to release 0.8 and advance some other core features, but we will come back to this issue when possible. Hope that in the meantime a forward proxy, or no proxy could be enough for you. BTW, do you have any special requirement for such a setup? Any constraint from a corporate environment? Just curious, willing to learn about use cases and usage in production environments, thanks!

[SteffenL]

I did a small test. To keep things simple, I used this config on the server:

ssl_enabled: True
port: 9300
# Public port where files will be served. If empty will be used "port"
public_port: 
host_name: my.domain

Then I tried to simply access https://127.0.0.1:9300/ on the server itself in a browser (fails while http works). SSL is not enabled on that port, at least.. Actually, I do not know which port it is listening to now because IIS is already listening on 443.. Is it possible to make the server listen on a different port for HTTPS?

[memsharded]

No, I am afraid that this cannot be configured right now that way. Will check if possible to add it, thanks for telling.

[SteffenL]

Regarding the deleted comments: Thanks! Will keep that in mind next time! :)

I did not think I would need/use SSL support in the Conan server, since I planned to use a reverse proxy anyway.. Native support for SSL in the Conan server or changing the port is not really a requirement assuming I can just get the requests proxied properly in this case. I do not have this problem with other apps proxied though IIS, so I am not quite sure what the problem can be.

I am sure a lot of people will still expect to be able to change the HTTPS port Conan is listening on. In my case, I would like to serve (as far as the client sees it) only via HTTPS in this specific case, either via the reverse proxy on port 443, or without it on any other available port.

The reason why I want to run Conan behind a reverse proxy (hide it from the client) is both because it could help reduce some risks, and because of the limitations faced on Windows at the moment without production-ready native support for containers. I prefer not to forward more ports than necessary to the machine and tweak the local software firewall, which is more security stuff to manage.

I currently run the show by myself here, but there may still be a few virtual constraints:

• Cost of running a separate server just for Conan which will be idle most of the time (at least for the time being).
• I depend on IIS to run some ASP.NET apps, and IIS itself is managed by WebsitePanel which I do not want to mess with.

I think it would be sensible to run Conan on a separate server, but it would be wasteful in my case. :)

[SteffenL]

Might add that my idea is indeed to use Conan in "production" as well as for hobby-projects, but as you know, I run the show alone at the moment.

I might use a couple Conan server instances: a more busy one to aid in-house development (CI server uploads the latest development packages continuously), and another I can share more freely with e.g. developers creating third-party plugins for my "products".

In this case, I would probably want the first server to require authentication for read/write access, while the other permits anonymous reads and authenticated writes.

Have not experimented enough with this to make a good decision yet, but I do not really want to make things more complicated than necessary. Maybe Conan already supports this, and I will look more at it later.

[lasote]

Hi @SteffenL,

About the SSL configuration, to use https with conan you will need a proxy that handles the ssl connections and redirect the traffic to conan server. Python servers like gunicorn doesn't provide support for ssl connection, that's because it's a common (and best) practice to have a proxy working as a load balancer or firewall etc (commonly nginx in linux systems).
The correct configuration should be:

ssl_enabled: True # It just change the generated file upload/download urls to have the "https://" instead "http://"

port: 9300 # The port where you want to run conan server (not the exposed to internet)

# Public port where files will be served. If empty will be used "port"

public_port: 443 # 443 should be fine

host_name: yourdomain.com # The exposed external domain (only used for generate the file upload/download urls)

Then you can use gunicorn to launch your local conan server, explained here: http://docs.conan.io/en/latest/server.html

Using nginx (I don't know too much about ISS), the configuration to use SSL is:

server { 
    listen 443;
    server_name myserver.com;
    location / {
       proxy_pass http://localhost:9300/;
    }
    ssl on;
    ssl_certificate /path_to/ssl/server.crt;
    ssl_certificate_key /path_to/ssl/server.key;
}

About the two servers, one with authentication and another without it, yes, its currently possible setting the [write_permissions] and [read_permissions] in the server conf file for each one.
We are currently working in a better remote management, Now there are some problems with authentication, it requires the credentials each time the remote change. So we hope it works better soon.

Thanks!

[lasote]

I close the issue, please if it didn't work write us.
Thanks!

[SteffenL]

Sorry for the delay, and thank you for your reply! Something came up and I have not yet been able to investigate this further. Hopefully I will get the time soon.

[lasote]

Great! report when you have time. I reopen the issue.

[SteffenL]

I got some more time to work on my things, which forced me to try to solve this issue. There has been no progress since I already rely heavily on IIS and its integration with other parts of the system.

• The server hangs when handling the "PUT /v1/files/..." requests during package upload, whenever I use IIS as a reverse proxy.
• Setting ssl_enabled to True does not let me access the server over HTTPS using the port I want to use (443 is already in use by IIS).

It seems like my option is to use unencrypted HTTP over port 9300 or something (80 is already in use by IIS), and without the reverse proxy.

I tried gunicorn just for the sake of doing it, but it does not seem to support Windows.

It is a bit risky for me to change the web server configuration at the moment, so I think I am going to give up for now.

[onekiloparsec]

I've tried with a nginx configuration as described by @lasote on March 3, but to no avail, despite having the server accessible correctly through https (and also redirected by http).

But the conan command search produces always the same error:

ERROR: Unable to connect to <name of your remote>

To me, something fishy is going with connections. I wish I could say I'll look into it, but I prefer not giving wrong expectations...

[memsharded]

Yes, we haven't had the chance to solve this issue, because up to our knowledge it is not in the conan codebase. The conan proxy configuration is just a small layer that passes it to the underlying python requests library, and we haven't had enough time to deep dive into it to address this issue.

As requested by @osechet, we have added in https://github.com/conan-io/conan/pull/685 the error message of the exception to that one, in case it can be of help for these cases. It will be released in next conan 0.16.

I guess that in your trial, the server didn't output any trace at all, so it was just the client not reaching the server, is that true? Thanks for reporting!

[onekiloparsec]

Thanks. With an improved error message, it will certainly be easier to investigate. I'll post results in due time, after 0.16 release.

[onekiloparsec]

I;ve been able to put breakpoint on code, and this is precisely the error I get (in remote_manager.py line 151):

{ProxyError}HTTPSConnectionPool(host='conan.motion-recall.local', port=443): Max retries exceeded with url: /v1/conans/search (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 400 Bad Request',)))

[onekiloparsec]

I managed to understand that my search request is calling the URL https://<server_host_name>/v1/conans/search (note the absence of ending /).

For that call, I get a very strange result in my browser: a blank page with Connect to https written on upper left. That's it.

Knowing a bit how REST APIs work, I appended a / to the URL. And for that call, I get a 404 Not Found: with a message:

Not found: '/conans/search/'

Note the absence of v1/... I don't know if that helps, but to me, there is something weird in the conan server. Maybe I'm wrong.

[memsharded]

I have been testing conan_server and with the browser, and the URLs of the API are without the final slash, that should be an issue. The browser messages for the search and other api calls are fine.

So I guess the issue might be more related to httpS than coming from the paths, and thus from the proxy?

[lasote]

Hi, I've solved some issues using SSL with a proxy like nginx. I will submit a PR and will update the docs with a real example of how to use conan_server with nginx and ssl. Early next week will release 0.16 with those changes.
Thanks for your patience.

[memsharded]

This has been fixed and released in 0.16.
Check the docs too: http://docs.conan.io/en/latest/server.html#server-parameters

Looking forward your feedback to ensure this issue can be closed. Thanks very much!

[onekiloparsec]

Thanks a lot. Will check asap.

[onekiloparsec]

No success here. Conan==0.16 both locally and on the remote server. Message:

$ conan search -r=<remote_name>
ERROR: HTTPSConnectionPool(host='<remote_hostname>', port=443): Max retries exceeded with url: /v1/conans/search (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 400 Bad Request',)))

Unable to connect to <remote_name>=<remote_hostname>

I am investigating. My .conan_server/server.conf file:

[server]
# WARNING! Change default variable of jwt_secret. You should change it periodically
# It only affects to current authentication tokens, you can change safetely anytime
# When it changes user are just forced to log in again
jwt_secret: <key>
jwt_expire_minutes: 120

ssl_enabled: False
port: 9301
# Public port where files will be served. If empty will be used "port"
public_port: 443
host_name: <remote_hostname> 

# Choose file adapter, "disk" for disk storage
# Authorize timeout are seconds the client has to upload/download files until authorization expires
store_adapter: disk
authorize_timeout: 1800

# Just for disk storage adapter
# updown_secret is the key used to generate the upload/download authorization token
disk_storage_path: ~/.conan_server/data
disk_authorize_timeout: 1800
updown_secret: APuTYrbOARsSEtABcMzygfxN


[write_permissions]

[read_permissions]
*/*@*/*: *

[users]
#default_user: defaultpass
demo: demo

My nginx server file:

server {
    listen 80;
    server_name <remote_hostname>;
    return 301 https://$server_name:9300$request_uri;
}

server { 
    listen 443;
    server_name <remote_hostname>;

    access_log /home/git/logs/nginx.conan.access.log;
    error_log /home/git/logs/nginx.conan.error.log;

    location / {
       proxy_pass http://0.0.0.0:9301/;
    }

    ssl on;
    ssl_certificate /etc/ssl/nginx/<remote_hostname>.crt;
    ssl_certificate_key /etc/ssl/nginx/<remote_hostname>.key;
}

[onekiloparsec]

One more thing: we use self-signed certificates.

[lasote]

@onekiloparsec Ok, I'm opening a new issue to optionally deactivate the requests certificate check.
Can we close this?

[onekiloparsec]

Sorry for late replay. Sounds reasonable to close this one, although I can't confirm...

[memsharded]

Yes, most likely is the certificates issue, so we can close this one. If #720 doesn't finally fix it, please re-open. Thanks for your feedback!

[memsharded]

The functionality for self generated certs has been implemented by #745, and landed in develop. To use it basically: $ conan remote add myremote https://myurl False. It will be released in next 0.17, but feedback from running from develop branch is very welcome. Thanks!

[onekiloparsec]

Hi. Thanks for working on this! Unfortunately, the search command still fails, as far as I can tell (I'm on develop on both side (server + client):

ERROR: HTTPSConnectionPool...

with :

$ conan remote list
<myremote>: https://conan.<hostname.local> [Verify SSL: False]

[memsharded]

You are welcome! Is the error still the same ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 400 Bad Request',?

[onekiloparsec]

Yes

[memsharded]

Another approach that could be tested, just in case, would be to append the server .crt file contents to ~/.conan/cacert.pem file in the client. In that case, you could try to use the normal remote, with VerifySSL=True

[lasote]

Try to set the public_port to 80 but ssl_enabled=True, this is my server.conf:

ssl_enabled: True
port: 9300
# Public port where files will be served. If empty will be used "port"
public_port: 80
host_name: myservername.mydomain.com

And this is my nginx conf:

server {
    listen 443;
    server_name myservername.mydomain.com;

    location / {
      proxy_pass http://localhost:9300/;
    }
    ssl on;
    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;
}

[SteffenL]

I am returning to this issue after quite a while, and can finally conclude it as resolved.

Here is what I did:

• Use the latest version of Conan client and server.
• Comment out the following lines in the Conan server config: ssl_enabled, public_port, host_name.
• Add a reverse proxy inbound rule in IIS.
• Change the application pool pipeline mode of the IIS site from Integrated to Classic, otherwise conan upload may time out consistently on read.
• Increase the max content length from the default 30 MB to avoid HTTP status 413 (Payload Too Large).

Example web.config file that works for my use case:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="ReverseProxyInboundRule1" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="http://localhost:9300/{R:1}" />
                </rule>
            </rules>
        </rewrite>
        <security>
            <requestFiltering>
                <requestLimits maxAllowedContentLength="500000000" />
            </requestFiltering>
        </security>
    </system.webServer>
</configuration>

Edit: There may be a problem left with caching because there were "missing" binaries in a remote that I had just uploaded to. Some time later, they were present as expected.

[lasote]

@SteffenL Thanks for your time to share how you solved the issue, we really appreciate it.