Stf: Setup HTTPS for STF Local

[thuanpv]

Hello,
I am trying to setup HTTPS for STF local using NGINX. I want to do it because STF is embedded inside my PHP code (which already be setup to work with HTTPS). Note that my STF setup works totally fine with normal HTTP URLs. I don't try the Docker setup as explained in the DEPLOYMENT document because I just need some simple setup like stf local. First, I prepare the following simple nginx configuration file and test it with a very simple node app running on port 7100, and it works -- an HTTPS URL directs me to the app.

erver {
        listen 80; 
        listen [::]:80;
        listen       443 ssl;
        ssl_certificate example.crt;
        ssl_certificate_key example.key;
        root /var/www/html;
        # Add index.php to the list if you are using PHP 
        index index.php index.html index.htm index.nginx-debian.html;

        server_name example.com;
        location / { 
                proxy_pass http://localhost:7100;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering off;
        }   
}

After that, to run STF Local with HTTPS I change all URLs starting with http to https in lib/cli/local/index.js and start OpenSTF by running stf local --public-ip=my public ip. However, when I type my URL (*.com) I got the following socket error in OpenSTF

ERR/poorxy 19402 [*] Proxy had an error Error: socket hang up
    at TLSSocket.onHangUp (_tls_wrap.js:1117:19)
    at Object.onceWrapper (events.js:291:19)
    at emitNone (events.js:91:20)
    at TLSSocket.emit (events.js:186:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Since the NGINX conf. works well with the simple NodeJS app, I think the error comes from some incorrect setting/initialization for OpenSTF.

Anyone has successfully set it up and know how to fix this issue? Your help will be greatly appreciated!
Thanks

[guadaran]

Hi @thuanpv,

I wonder how you solved this problem. Do you have any tips to share for implement HTTPS on STF?

Thank you.

[thuanpv]

Hi guadaran,
Yes, I managed to setup HTTPS on STF Local. It took me lot of time to read the DEPLOYMENT.md and dig into the code to understand how STL Local works. I will try to write a full document when I have some free time. Following are the key points

1. The starting code for STF local is in this file lib/cli/local/index.js. Reading the file is good for us to understand the whole workflow of STF in a local setup.
2. In STF Local, poorxy works as a HTTP server to "route" the user requests to corresponding services and when we setup HTTPS we should use Nginx to do so ==> remove the poorxy service from lib/cli/local/index.js
3. In lib/cli/local/index.js, I change the settings from IP:port to IP or IP/service (e.g., IP/auth/mock) and http to https/ws to wss so that Nginx can route the requests and the HTTPS certificate works. It seems to me that URIs like domain:port_number is not flagged as secure address suppose that you have a SSL certificate for your domain.
4. The nginx.conf is similar to the one in DEPLOYMENT.md but you need to change the ports. The port numbers for all services can be retrieved from lib/cli/local/index.js
   Hope it helps.

Thuan

[guadaran]

Hi @thuanpv,

Great! Thank you for sharing these tips.

Regards.

[maykel535]

Hi @thuanpv,

I run stf in lxc ubuntu and works well, but Can you tell me how to resolve this?? I like run stf with https over nginx reverse proxy.

I follow this step:

https://github.com/thuanpv/stf/blob/master/doc/DEPLOYMENT.md#nginx-configuration

But not works... I get redirect to http:IP/auth/mock

Thanks in advanced.

[stevelixin]

@maykel535 , not sure if you have got the answer, I met the same problem as yours, did you have the solution?

[maykel535]

Finally, I have installed the openstf by docker and all ok works very nice

El vie., 18 may. 2018 7:03, stevelibuzz [e-mail] escribió:

@maykel535 https://github.com/maykel535 , not sure if you have got the
answer, I met the same problem as yours, did you have the solution?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openstf/stf/issues/767#issuecomment-390095291, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADiAWOVEsKdJqc66uV182wujTuUdJl_Dks5tzlYogaJpZM4QyEXF
.

[entomber]

3\. In lib/cli/local/index.js, I change the settings from IP:port to IP or IP/service (e.g., IP/auth/mock) and http to https/ws to wss so that Nginx can route the requests and the HTTPS certificate works.

I was able to set up HTTPS on STF local following the steps outlined by @thuanpv, however I don't understand where or how to change "http to https/ws to wss". I tried changing http to wss but the site did not load at all, so I just changed them all to https. Device screens are all gray when selected, is that due to not changing to https/ws or wss? If so, how can I change it?

[bharanikodukula]

@entomber were you able to fix it.

[entomber]

@bharanikodukula Yes I fixed it. I haven't used the local https setup in months as I'm now using a docker deployment for production and http for local. However, below should be all the changes I made:

1. Follow instructions in @thuanpv 's post, but instead of his step 3 do the following:
   In lib/cli/local/index.js, change http to https and remove the ports, except for --websocket-url leave the port in like https://%s:%d/
2. In lib/cli/device/index.js and lib/cli/provider/index.js, replace ws with wss
3. In lib/units/websocket/index.js within the module.exports, create an https server with SSL cert and key instead of using default http server, like so:

  var opt = {
    key: fs.readFileSync('/etc/nginx/SSL/server.key'),
    cert: fs.readFileSync('/etc/nginx/SSL/ssl-bundle.crt')
  };

  var server = https.createServer(opt)
  var io = socketio.listen(server, {
        serveClient: false
      , transports: ['websocket']
      })

1. In lib/units/device/plugins/screen/stream.js find function createServer() and the line below:

var wss = new WebSocket.Server({
        port: screenOptions.publicPort
      , perMessageDeflate: false
      })

replace with:

var credentials = {
        key: fs.readFileSync('/etc/nginx/SSL/server.key'),
        cert: fs.readFileSync('/etc/nginx/SSL/ssl-bundle.crt')
      };

      var httpsServer = https.createServer(credentials)
      httpsServer.listen(screenOptions.publicPort)

      var wss = new WebSocket.Server({
        server: httpsServer
      , perMessageDeflate: false
      })

I hope that helps.

[bharanikodukula]

@entomber Thanks for the help .However iam getting the below error after making the configuration changes

019-08-02T13:59:36.141Z INF/processor 21079 [proc001] Device dealer connected to "tcp://127.0.0.1:7115"
../../cli websocket

Options:
-h, --help Show help. [boolean]
-V, --version Show version. [boolean]
--connect-push, -c App-side ZeroMQ PULL endpoint to connect to.
[array] [required]
--connect-sub, -u App-side ZeroMQ PUB endpoint to connect to.
[array] [required]
--port, -p The port to bind to. [number] [default: 7110]
--secret, -s The secret to use for auth JSON Web Tokens. Anyone who
knows this token can freely enter the system if they want,
so keep it safe. [string] [required]
--ssid, -i The name of the session ID cookie.
[string] [default: "ssid"]
--storage-url, -r URL to the storage unit. [string] [required]

Each option can be be overwritten with an environment variable by converting the
option to uppercase, replacing dashes with underscores and prefixing it with
STF_WEBSOCKET_ (e.g. STF_WEBSOCKET_STORAGE_URL).

Unexpected identifier
2019-08-02T13:59:36.158Z FTL/cli:local 21063 [*] Child process had an error ExitError: Exit code "1"
at ChildProcess. (/usr/lib/node_modules/stf/lib/util/procutil.js:49:23)

My websocket index.js files looks like this

module.exports = function(options) {
var opt = {
key: fs.readFileSync('/etc/ssl/certs/internal_xyd.key')
cert: fs.readFileSync('/etc/ssl/certs/internal_xydcrt')
};

var server = https.createServer(opt)
var io = socketio.listen(server, {
serveClient: false
, transports: ['websocket']
})
var log = logger.createLogger('websocket')

[entomber]

You're missing a period in your cert filename: internal_xydcrt

Did you rebuild STF after your code changes?

1. run gulp clean
2. run sudo npm link

[bharanikodukula]

@entomber no i did not rebuild .I will rebuild the code and give a try

[entomber]

@bharanikodukula did rebuilding fix the issue for you?

[bharanikodukula]

no i did not help .I built using npm install stf

[man4567890]

Hi @entomber, I'm getting this, I'm not running nginx, but httpd, I've generated self signed certficates, but as soon as I start stf, in the logs I see
`2020-03-09T12:14:48.550Z INF/provider 13663 [] Tracking devices
2020-03-09T12:14:48.632Z INF/storage:plugins:apk 13712 [] Listening on port 7104
2020-03-09T12:14:48.766Z INF/storage:temp 13696 [] Listening on port 7102
2020-03-09T12:14:48.905Z INF/auth-mock 13669 [] Listening on port 7120
2020-03-09T12:14:48.911Z INF/db 13669 [] Connecting to 127.0.0.1:28015
2020-03-09T12:14:49.016Z INF/app 13670 [] Using webpack
usr/lib/node_modules/stf/lib/cli websocket

Options:
-h, --help Show help. [boolean]
-V, --version Show version. [boolean]
--connect-push, -c App-side ZeroMQ PULL endpoint to connect to.
[array] [required]
--connect-sub, -u App-side ZeroMQ PUB endpoint to connect to.
[array] [required]
--port, -p The port to bind to. [number] [default: 7110]
--secret, -s The secret to use for auth JSON Web Tokens. Anyone who
knows this token can freely enter the system if they want,
so keep it safe. [string] [required]
--ssid, -i The name of the session ID cookie.
[string] [default: "ssid"]
--storage-url, -r URL to the storage unit. [string] [required]

Each option can be be overwritten with an environment variable by converting the
option to uppercase, replacing dashes with underscores and prefixing it with
STF_WEBSOCKET_ (e.g. STF_WEBSOCKET_STORAGE_URL).

fs is not defined
2020-03-09T12:14:49.122Z FTL/cli:local 13622 [] Child process had an error ExitError: Exit code "1"
at ChildProcess. (/usr/lib/node_modules/stf/lib/util/procutil.js:49:23)
at emitTwo (events.js:126:13)
at ChildProcess.emit (events.js:214:7)
at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
2020-03-09T12:14:49.122Z INF/cli:local 13622 [] Shutting down all child processes
2020-03-09T12:14:49.129Z INF/util:lifecycle 13637 [app001] Winding down for graceful exit
2020-03-09T12:14:49.129Z INF/util:lifecycle 13645 [proc001] Winding down for graceful exit
2020-03-09T12:14:49.129Z INF/util:lifecycle 13639 [dev001] Winding down for graceful exit
2020-03-09T12:14:49.132Z INF/util:lifecycle 13650 [proc002] Winding down for graceful exit
2020-03-09T12:14:49.132Z INF/util:lifecycle 13655 [reaper001] Winding down for graceful exit
`

Any idea on what the problem could be, or does anyone have a complete write up on how to do this please?

[entomber]

Your log says 'fs is not defined'. Check that you loaded fs in the modified files that use it, e.g. var fs = require('fs')

[man4567890]

Hi @entomber

168: $ grep -nr "var fs" *
units/app/index.js:3:var fs = require('fs')
units/app/middleware/webpack.js:22: var fs = compiler.outputFileSystem = new MemoryFileSystem()
units/auth/saml2.js:1:var fs = require('fs')
units/device/resources/minicap.js:1:var fs = require('fs')
units/device/resources/minirev.js:1:var fs = require('fs')
units/device/resources/minitouch.js:2:var fs = require('fs')
units/storage/s3.js:4:var fs = require('fs')
util/download.js:1:var fs = require('fs')
util/pathutil.js:2:var fs = require('fs')
util/refresh.js:1:var fs = require('fs')
util/storage.js:3:var fs = require('fs')
diginetiot@digitatanetworks168:lib $ pwd
/usr/lib/node_modules/stf/lib
168:lib $
I don't think that I changed anything concerning var fs, I think my problem is with the proxy settings, do you perhaps have a printout of the NGINX configuration for me please, so I can see how this is done

[entomber]

@man4567890 if you added the SSL key/cert snippets in my post to lib/units/websocket/index.js or lib/units/device/plugins/screen/stream.js, you would need to add fs as it's used to read in the files. As for nginx, I don't have the config anymore but it was very similar to the one in DEPLOYMENT.md.

[man4567890]

Hi @entomber , thanks for the information, but where do I define the fs?

[man4567890]

Hi, Ok the fs problem is solved by adding var fs = require('fs') , NGINX is up and running with generated certificates, but I still cannot open the GUI, nothing in NGINX logs, when you say

In lib/cli/local/index.js, change http to https and remove the ports, except for --websocket-url leave the port in like https://%s:%d/ <

For example

    // provider
    , procutil.fork(path.resolve(__dirname, '..'), [
          'provider'
        , '--name', argv.provider
        , '--min-port', argv.providerMinPort
        , '--max-port', argv.providerMaxPort
        , '--connect-sub', argv.bindDevPub
        , '--connect-push', argv.bindDevPull
        , '--group-timeout', argv.groupTimeout
        , '--public-ip', argv.publicIp
        , '--storage-url'
        , util.format('https://localhost:%d/', argv.port)
        , '--adb-host', argv.adbHost
        , '--adb-port', argv.adbPort
        , '--vnc-initial-size', argv.vncInitialSize.join('x')
        , '--mute-master', argv.muteMaster
        ]
        .concat(argv.allowRemote ? ['--allow-remote'] : [])
        .concat(argv.lockRotation ? ['--lock-rotation'] : [])
        .concat(!argv.cleanup ? ['--no-cleanup'] : [])
        .concat(!argv.screenReset ? ['--no-screen-reset'] : [])
        .concat(argv.serial))

      // auth
    , procutil.fork(path.resolve(__dirname, '..'), [
          util.format('auth-%s', argv.authType)
        , '--port', argv.authPort
        , '--secret', argv.authSecret
        , '--app-url', util.format(
            'https://%s:%d/'
          , argv.publicIp
          , argv.port
          )
        ].concat(JSON.parse(argv.authOptions)))

How do I remove the ports, do I remove the argv.port from the lines that contain util.format('https://localhost:%d/', argv.port) so that it looks like this, util.format('https://localhost:%d/') or do I remove the %d so that it looks like this util.format('https://localhost/', argv.port) or do I just remove the %d in the lines https://%s:%d/ so that it looks like this https://%s/

I get no errors in stf logs, selinux logs or NGINX logs, please could you clarify what is meant by
In lib/cli/local/index.js, change http to https and remove the ports, except for --websocket-url leave the port in like https://%s:%d/

[entomber]

How do I remove the ports, do I remove the argv.port from the lines that contain util.format('https://localhost:%d/', argv.port) so that it looks like this, util.format('https://localhost:%d/') or do I remove the %d so that it looks like this util.format('https://localhost/', argv.port) or do I just remove the %d in the lines https://%s:%d/ so that it looks like this https://%s/

It should look like util.format('https://localhost/') for everything except the websocket parts which would should be something like:

'--websocket-url', util.format(
            'https://%s:%d/'
          , argv.publicIp
          , argv.websocketPort
          )

and util.format('https://localhost/', argv.port)

[tshah16]

@thuanpv and @guadaran

for second steps mentioned by you
In STF Local, poorxy works as a HTTP server to "route" the user requests to corresponding services and when we setup HTTPS we should use Nginx to do so ==> remove the poorxy service from lib/cli/local/index.js

which lines needs to be removed to remove the poorxy?

poorxy [*] Proxy had an error Error: socket hang up
at TLSSocket.onHangUp (_tls_wrap.js:1148:19)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:111:20)
at TLSSocket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickCallback (internal/process/next_tick.js:181:9)

[man4567890]

hi @shahtapan , this is what I did in /usr/lib/node_modules/stf/lib/cli/local/index.js , but I still cannot get it to work in https on a local machine, not sure if this is the way to remove the poorxy. I think I have problem with the way I set up NGINX

     // apk processor
    , procutil.fork(path.resolve(__dirname, '..'), [
          'storage-plugin-apk'
        , '--port', argv.storagePluginApkPort
        , '--storage-url'
        //, util.format('https://localhost/', argv.port)
        , util.format('https://localhost/')
        ])

      // poorxy
      //, procutil.fork(path.resolve(__dirname, '..'), [
      //    'poorxy'
      //  , '--port', argv.port
      //  , '--app-url'
      //  , util.format('https://localhost/', argv.appPort)
      //  , '--auth-url'
      //  , util.format('https://localhost/', argv.authPort)
      //  , '--api-url'
      //  , util.format('https://localhost:%d/', argv.apiPort)
      //  , '--websocket-url'
      //  , util.format('https://localhost/', argv.websocketPort)
      //  , '--storage-url'
      //  , util.format('https://localhost/', argv.storagePort)
      //  , '--storage-plugin-image-url'
      //  , util.format('https://localhost/', argv.storagePluginImagePort)
      //  , '--storage-plugin-apk-url'
      //  , util.format('https://localhost/', argv.storagePluginApkPort)
      //  ])
    ]

And my NGINX config, the local machine IP is 172.28.200.90

#daemon off;
user  nginx;
worker_processes 4;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
  worker_connections 1024;
}

http {
log_format custom '$remote_addr - $remote_user [$time_local] '
                                 '"$request" $status $body_bytes_sent '
                                 '"$http_referer" "$http_user_agent" '
                                 '"$http_x_forwarded_for" $request_id ';
access_log  /var/log/nginx/access.log custom;

  upstream stf_app {
    server 172.28.200.90:7120 max_fails=0;
  }

  upstream stf_auth {
    server 172.28.200.90:7100 max_fails=0;
  }

  upstream stf_storage_apk {
    server 172.28.200.90:7104 max_fails=0;
  }

  upstream stf_storage_image {
    server 172.28.200.90:7103 max_fails=0;
  }

  upstream stf_storage {
    server 172.28.200.90:7102 max_fails=0;
  }

  upstream stf_websocket {
    server 172.28.200.90:7110 max_fails=0;
  }

  upstream stf_api {
    server 172.28.200.90:7106 max_fails=0;
  }

  types {
    application/javascript  js;
    image/gif               gif;
    image/jpeg              jpg;
    text/css                css;
    text/html               html;
  }

  map $http_upgrade $connection_upgrade {
    default  upgrade;
    ''       close;
  }

  server {
    listen 80;
    server_name 172.28.200.90;
    return 301 https://$server_name$request_uri;
  }

  server {
    listen 443 ssl;
    server_name 172.28.200.90;
    keepalive_timeout 70;
    root /dev/null;

    # https://mozilla.github.io/server-side-tls/ssl-config-generator/
    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/cert.key;
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:10m;
    ssl_dhparam /etc/nginx/ssl/dhparam.pem;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;

    #add_header Strict-Transport-Security max-age=15768000;

    #ssl_stapling on;
    #ssl_stapling_verify on;
    #ssl_trusted_certificate /etc/nginx/ssl/cert.pem;

    resolver 8.8.4.4 8.8.8.8 valid=300s;
    resolver_timeout 10s;

    # Handle [email protected]
    #location ~ "^/d/floor4/([^/]+)/(?<port>[0-9]{5})/$" {
    #  proxy_pass http://172.28.200.90:$port/;
    #  proxy_http_version 1.1;
    #  proxy_set_header Upgrade $http_upgrade;
    #  proxy_set_header Connection $connection_upgrade;
    #  proxy_set_header X-Forwarded-For $remote_addr;
    #  proxy_set_header X-Real-IP $remote_addr;
    #}

    # Handle [email protected]
    #location ~ "^/d/floor8/([^/]+)/(?<port>[0-9]{5})/$" {
    #  proxy_pass http://172.28.200.90:$port/;
    #  proxy_http_version 1.1;
    #  proxy_set_header Upgrade $http_upgrade;
    #  proxy_set_header Connection $connection_upgrade;
    #  proxy_set_header X-Forwarded-For $remote_addr;
    #  proxy_set_header X-Real-IP $remote_addr;
    #}

    location /auth/ {
      proxy_pass http://stf_auth/auth/;
    }

    location /api/ {
      proxy_pass http://stf_api/api/;
    }

    location /s/image/ {
      proxy_pass http://stf_storage_image;
    }

    location /s/apk/ {
      proxy_pass http://stf_storage_apk;
    }

    location /s/ {
      client_max_body_size 1024m;
      client_body_buffer_size 128k;
      proxy_pass https://stf_storage;
    }

    location /socket.io/ {
      proxy_pass http://stf_websocket;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Real-IP $http_x_real_ip;
    }

    location / {
      proxy_pass http://stf_app;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Real-IP $http_x_real_ip;
    }
  }
}

but I would not trust my way of doing this, like I said, even with the way I have it now, https still does not work