Awx: ssl_certificate defined, but not used

Created on 31 May 2019 · 11Comments · Source: ansible/awx

ISSUE TYPE

Bug Report

SUMMARY

The ssl_certificate value is defined and uncommented in inventory, and it gets used correctly in the docker-compose.yml output file, but the awxweb container's /etc/nginx/nginx.conf file has no SSL declaration, resulting in HTTP access only.

ENVIRONMENT

AWX version: 4.0.0
AWX install method: docker on linux
Ansible version: 2.7.10
Operating System: CentOS Linux release 7.6.1810 (Core)
Web Browser: Firefox Quantum 67 on OSX

STEPS TO REPRODUCE

I have edited inventory, removing the comment on ssl_certificate (line 57), and adding the path to the file containing my private key and certificate chain concatenated together.

When I run sudo ansible-playbook -i inventory install.yml, the installation process succeeds, and I get a functional AWX instance available at port 80. Trying to access the site over HTTPS results in an immediate failure:

curl -v https://127.0.0.1/
* About to connect() to 127.0.0.1 port 443 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file

EXPECTED RESULTS

I expect the awxweb container to get my SSL certificate configuration and serve the site over HTTPS.

ACTUAL RESULTS

The awxweb container's /etc/nginx/nginx.conf has no SSL declaration. Here is the nginx.conf file inside the awxweb container:

[root@awxweb awx]# cat /etc/nginx/nginx.conf
#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

    server {
                listen 8052 default_server;

        # If you have a domain name, this is where to add it
        server_name _;
        keepalive_timeout 65;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;            proxy_set_header X-Forwarded-Port 443;
        }
    }
}

ADDITIONAL INFORMATION

The comments and changes in #3604 suggest that this should "just work", and it almost does. The awxcompose/docker-compose.yml file correctly defines the certificate file to mount into the container, as well as maps tcp/443 from the host to tcp/8053 in the container:

services:

  web:
    image: ansible/awx_web:4.0.0
    container_name: awx_web
    depends_on:
      - rabbitmq
      - memcached
      - postgres
    ports:
      - "443:8053"
      - "80:8052"
    hostname: awxweb
    user: root
    restart: unless-stopped
    volumes:
      - "/opt/awx-data/awxcompose/SECRET_KEY:/etc/tower/SECRET_KEY"
      - "/opt/awx-data/awxcompose/environment.sh:/etc/tower/conf.d/environment.sh"
      - "/opt/awx-data/awxcompose/credentials.py:/etc/tower/conf.d/credentials.py"
      - "/opt/awx-data/projects:/var/lib/awx/projects:rw"
      - "/opt/awx-data/ssl/star.domain.key.pem:/etc/nginx/awxweb.pem:ro"
    environment:
      http_proxy:
      https_proxy:
      no_proxy:

The certificate does get mounted inside the running awxweb container:

[root@awxweb awx]# ls -lh /etc/nginx/awxweb.pem
-rw-rw-r--. 1 327117572 1111 12K May 31 17:27 /etc/nginx/awxweb.pem
[root@awxweb awx]# md5sum /etc/nginx/awxweb.pem
4eb2817755b1b0358db9793621993d20  /etc/nginx/awxweb.pem

The MD5 sum of the certificate as seen inside the container matches the MD5 sum from the Linux host:

md5sum /opt/awx-data/ssl/star.domain.key.pem
4eb2817755b1b0358db9793621993d20  /opt/awx-data/ssl/star.domain.key.pem

bug

Source

skpy

All 11 comments

I'm experiencing the exact same problem.
ssl_certificate is set and uncommented in the inventory file but the nginx.conf file in the aws-web container does not include SSL configuration.

Workaround for now is to enter the container and manually using docker exec -it aws-web bash and edit /etc/nginx/nginx.conf with the SSL configuration defined in the template installer/roles/image_build/templates/nginx.conf.j2

Then reload nginx with /sbin/nginx -s reload and exit the container.

mjeco on 6 Jun 2019

@mjeco @skpy are you building your own images, or using the official ones (see: )https://github.com/ansible/awx/blob/devel/INSTALL.md#official-vs-building-images? @jeffbyrnes' contribution in https://github.com/ansible/awx/pull/3604/ _should_ cover this, but if you're still using the official awx 4.0.0 image from dockerhub, you're not going to get the changes here: https://github.com/ansible/awx/pull/3604/files#diff-faeb74d21fb8b558d8ddf62c5401df2c

(because they haven't been released in a new official version of awx yet)

ryanpetrello on 6 Jun 2019

👍1

@ryanpetrello yes I'm using the official awx images. Thanks for the explanation.

mjeco on 6 Jun 2019

If you all are still encountering issues after building your own images (or in the next version of awx), let me know and I'll reopen this so we can investigate.

ryanpetrello on 6 Jun 2019

Thanks @ryanpetrello ! I'm building the images now to validate this.

I tried to work around this problem by putting a Traefik proxy in front of AWX, but that rejected all logins due to CSRF failures. I couldn't find anything obvious in the AWX issue history regarding this. I tried following the directions here but that didn't seem to be enough.

For the sake of completeness, is there a document properly describing how to put an HTTPS proxy in front of an HTTP-listening AWX that deals with CORS / CSRF / whatever else?

skpy on 6 Jun 2019

Hey @skpy,

For the sake of completeness, is there a document properly describing how to put an HTTPS proxy in front of an HTTP-listening AWX that deals with CORS / CSRF / whatever else?

We currently don't have great documentation for this in awx. If you figure out what's up, please open a pull request detailing what you discovered 😄.

Django's CSRF requires strict matching on the referer protocol; I'm betting there's a mismatch regarding what your proxy is forwarding (https vs http).

I'm betting you're getting a 403 from code somewhere in this area of Django's CSRF middleware:

https://github.com/django/django/blob/1.11.20/django/middleware/csrf.py#L259

ryanpetrello on 6 Jun 2019

👍1

@ragingpastry If ssl is supported is decided at building time. Therefore with the current solution it will never work with upstream containers. You have to build them by yourself.
At least this is my guess why it's not working (nginx SSL-configuration is done in the image_build role).

Tested with awx-web 6.1.0.