Libretime: Libretime behind Nginx Reverse Proxy w/ LetsEncrypt URL issues
I have been having a hard time getting Libretime to run correctly behind my reverse proxy. I have added the CORS as mentioned in https://github.com/LibreTime/libretime/issues/822#issuecomment-546609291
The reverse proxy is working with other sites, but I am missing something I think either in the /etc/airtime/airtime.conf file, or the apache2 airtime.conf file. Can someone point me in the right direction?
I am able to login and change/view settings. But the playout page, previewing music, etc. is not displaying. It appears to be forcing http://mysite, even though the reverse proxy is set to always re-direct to https.
Example log:
an 28 11:42:34 TerryRadio airtime-playout[9038]: 2020-01-28 11:42:34,846 [pypoliqqueue] [INFO ] waiting indefinitely for schedule
Jan 28 11:42:40 TerryRadio airtime-playout[9038]: 2020-01-28 11:42:40,771 [api_client] [ERROR] HTTP Error 500: Internal Server Error
Jan 28 11:42:40 TerryRadio airtime-playout[9038]: 2020-01-28 11:42:40,846 [api_client] [ERROR] HTTP Error 500: Internal Server Error
Jan 28 11:42:40 TerryRadio airtime-playout[9038]: 2020-01-28 11:42:40,912 [listenerstat] [ERROR] Exception: HTTP Error 500: Internal Server Error
Jan 28 11:44:41 TerryRadio airtime-playout[9038]: 2020-01-28 11:44:41,501 [api_client] [ERROR] HTTP Error 500: Internal Server Error
Jan 28 11:44:41 TerryRadio airtime-playout[9038]: 2020-01-28 11:44:41,580 [api_client] [ERROR] HTTP Error 500: Internal Server Error
Jan 28 11:44:41 TerryRadio airtime-playout[9038]: 2020-01-28 11:44:41,640 [listenerstat] [ERROR] Exception: HTTP Error 500: Internal Server Error
Jan 28 11:45:01 TerryRadio CRON[11095]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jan 28 11:46:56 TerryRadio airtime-playout[9038]: 2020-01-28 11:46:56,750 [listenerstat] [ERROR] Exception: <urlopen error [Errno -3] Temporary failure in name resolution>
A default apache conf:
terry@TerryRadio:/etc/apache2/sites-available$ cat airtime.conf
<VirtualHost *:80>
ServerAdmin [email protected]
DocumentRoot /usr/share/airtime/php/airtime_mvc/public/
php_admin_value upload_tmp_dir /tmp
php_value post_max_size 500M
php_value upload_max_filesize 500M
php_value request_order "GPC"
php_value session.gc_probability 0
php_value session.auto_start 0
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/json
<Directory /usr/share/airtime/php/airtime_mvc/public/>
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
RewriteRule ^.*$ index.php [NC,L]
DirectoryIndex index.php
AllowOverride all
Order allow,deny
Allow from all
Require all granted
</Directory>
</VirtualHost>
My airtime.conf, relevant section :
[general]
api_key = (removed)
web_server_user = www-data
base_url = airtime.mysite.mysite.org
base_port = 80
base_dir = /
cache_ahead_hours = 1
airtime_dir =
station_id =
auth = local
Finally, the reverse proxy conf:
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name airtime.*;
include /config/nginx/ssl.conf;
client_max_body_size 0;
location / {
include /config/nginx/proxy.conf;
resolver 127.0.0.11 valid=30s;
proxy_pass http://192.168.1.205;
proxy_set_header Range $http_range;
proxy_set_header If-Range $http_if_range;
}
}
Also, a screenshot of what I mean from developer mode in browser:
guythnick
All 30 comments
Also, I see in the zendphp.log, these entries when I believe a file attempts to broadcast:
2020-01-28T14:45:24-06:00 ERR (3): airtime.terryradio.duckdns.org [ErrorController.php:54 - errorAction()] - An internal application error has occurred.: ErrorException: Invalid argument supplied for foreach() in /usr/share/airtime/php/airtime_mvc/application/models/ListenerStat.php:160
Stack trace:
#0 /usr/share/airtime/php/airtime_mvc/application/models/ListenerStat.php(160): exception_error_handler(2, 'Invalid argumen...', '/usr/share/airt...', 160, Array)
#1 /usr/share/airtime/php/airtime_mvc/application/controllers/ApiController.php(1409): Application_Model_ListenerStat::insertDataPoints(NULL)
#2 /usr/share/airtime/php/vendor/zf1s/zend-controller/library/Zend/Controller/Action.php(516): ApiController->pushStreamStatsAction()
#3 /usr/share/airtime/php/vendor/zf1s/zend-controller/library/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('pushStreamStats...')
#4 /usr/share/airtime/php/vendor/zf1s/zend-controller/library/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))
#5 /usr/share/airtime/php/vendor/zf1s/zend-application/library/Zend/Application/Bootstrap/Bootstrap.php(105): Zend_Controller_Front->dispatch()
#6 /usr/share/airtime/php/vendor/zf1s/zend-application/library/Zend/Application.php(384): Zend_Application_Bootstrap_Bootstrap->run()
#7 /usr/share/airtime/php/airtime_mvc/application/airtime-boot.php(84): Zend_Application->run()
#8 /usr/share/airtime/php/airtime_mvc/public/index.php(68): require_once('/usr/share/airt...')
#9 {main}
While I'm not sure why it isn't working, this is the proxy config I have used before without issue: https://gitlab.com/paddatrapper/ansible/blob/master/roles/libretime/files/proxy.conf.j2
I think I set the CORS URLs to the following:
http://example.org
https://example.org
http://localhost
https://localhost
paddatrapper
on 29 Jan 2020
I basically have the same config, but using a letsencrypt docker instance that automatically handles the certs and redirects to https. The main radio splash page (before login) still is showing the same errors regardless.
I have found that when I have to add 'listen 80;' for the uploads / playout to work correctly. Here is a snippet of my analyzer log, the first instance is without it listening on port 80, and the second I added it back in. I would like to be able to use the app completely on https.
You can see the callback URL is the same (http) on both instances, but when the reverse proxy is not listening on port 80, it fails with a 400 error.
2020-01-29 17:39:24,898 [message_listener] [INFO ] - Received '{"tmp_file_path":"\/media\/sf_Music\/organize\/08 - Robyn - Call Your Girlfriend Stretched 15min.mp3","storage_backend":"file","import_directory":"\/media\/sf_Music\/\/imported\/1","original_filename":"08 - Robyn - Call Your Girlfriend Stretched 15min.mp3","callback_url":"http:\/\/airtime.site.site.org\/\/rest\/media\/2715","api_key":"sdsdsdsdsd","file_prefix":""}' on routing_key ''
2020-01-29 17:39:26,007 [cuepoint_analyzer] [WARNI] ['silan', '-b', '-F', '0.99', '-f', 'JSON', '-t', '1.0', u'/media/sf_Music/organize/08 - Robyn - Call Your Girlfriend Stretched 15min.mp3'] -6
2020-01-29 17:39:33,917 [status_reporter] [ERROR] HTTP request failed. Exception was: 400 Client Error: Bad Request for url: https://airtime.site.site.org//rest/media/2715
Working after adding 'listen 80;' :
2020-01-29 17:44:49,414 [message_listener] [INFO ] - Received '{"tmp_file_path":"\/media\/sf_Music\/organize\/08 - Robyn - Call Your Girlfriend Stretched 15min.mp3","storage_backend":"file","import_directory":"\/media\/sf_Music\/\/imported\/1","original_filename":"08 - Robyn - Call Your Girlfriend Stretched 15min.mp3","callback_url":"http:\/\/airtime.site.site.org\/\/rest\/media\/2716","api_key":"sdsdsdsdsd","file_prefix":""}' on routing_key ''
2020-01-29 17:44:49,780 [cuepoint_analyzer] [WARNI] ['silan', '-b', '-F', '0.99', '-f', 'JSON', '-t', '1.0', u'/media/sf_Music/organize/08 - Robyn - Call Your Girlfriend Stretched 15min.mp3'] -6
2020-01-29 17:44:55,851 [status_reporter] [INFO ] HTTP request sent successfully.
That is probably a bug in the api_client python application. That is what pypo and friends uses to communicate with the PHP and as far as I can remember, silan is run by one of the python apps
paddatrapper
on 30 Jan 2020
This is not a proper fix by any means, but I was able to fix the main splash page by changing the below HTTPHelper.php file. It now basically is forcing the https in the URLs internally, and not using the port to determine that. I also commented out the port in the URL since it would consider that a mismatch.
I still don't have this working without listening to port 80 on the proxy, so I think there is still something wrong with the api_client part of it. I tried doing something simiar to below in api_client.py and re-compiling, but it did not work.
to:
$scheme = "https";
if ($secured && !empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] !== 'off') {
$scheme = "https";
$basePort = "443"; //Airtime Pro compatibility hack
}
$portStr = "";
if (!(($scheme == "http" && $basePort == "80")
|| ($scheme == "https" && $basePort == "443"))) {
#$portStr = ":${basePort}";
}
The PHP should read the X-Forwarded-* header fields and use those to determine the protocol to use in order to solve this properly. It is more tricky for the Python stuff... Unless they, by default hit the reverse proxy and let it communicate with the PHP which is hacky...
paddatrapper
on 4 Feb 2020
@guythnick what are the contents of /config/nginx/proxy.conf?
paddatrapper
on 4 Feb 2020
Hey this looks like something I'm keen on using - just making a comment here so I get notified. I'll see if I can try and help in some way (e.g. testing).
kmahelona
on 21 Feb 2020
Im running into the same with HAproxy.
Aphris-Karu
on 30 Apr 2020
Ok I'm hitting this again on Debian Buster, using 3.0~alpha.8. I've added the work around using listen 80 for the moment and will try investigate why this is occurring
paddatrapper
on 7 May 2020
It seems that airtime_analyzer is trying to contact http://example.com/, is redirected through the proxy and can't handle that. The solution is to either use localhost, and not example.com (as they will always be on the same machine) or use https explicitly in airtime_analyzer
paddatrapper
on 7 May 2020
Using localhost would require replacing this:
https://github.com/LibreTime/libretime/blob/386918da2bf567f32889a0e90c4ecd35e72e5ebc/airtime_mvc/application/models/airtime/CcFiles.php#L165
The limitation is that forcing http://localhost here results in the same issue if someone changes the apache config to redirect to HTTPS only. It will work for http and http behind a https reverse proxy. The long-term solution would be to integrate airtime_analyzer into the PHP. This is because airtime_analyzer doesn't actually need to be separate. It was designed to be separate to support multi-tenancy for Airtime's SaaS offering. That is not really the case any more. Alternatively include a config flag selecting http or https for backend services
paddatrapper
on 7 May 2020
As it stands right now, this is one of two remaining release blockers yet to be claimed. https://github.com/LibreTime/libretime/projects/2
Word on the street is this issue is a bit complex, but it seems like there are some proposed solutions in the comments here.
Wondering if there's anyone interested/able to take this on.
gusaus
on 7 May 2020
I'm not going to have much time to deal with fixing this properly. I think we need to investigate why the redirect isn't being handled correctly. It could be an issue with Nginx or how the redirect is handled in airtime_analyzer. airtime_analyzer uses requests to do the HTTP stuff.
paddatrapper
on 7 May 2020
Relevant part of airtime_analyzer: https://github.com/LibreTime/libretime/blob/8498029cd983536a6e3016780d0b1693985a8e01/python_apps/airtime_analyzer/airtime_analyzer/status_reporter.py#L99-L134
paddatrapper
on 7 May 2020
Also causes issues with airtime-celery:
paddatrapper
on 7 May 2020
The more I think about it, the more I think a config flag for backend callback URL base would be the way to go. Least invasive solution
paddatrapper
on 7 May 2020
Essentially this will cause an issue wherever the python code has a callback URL to return data to the rest framework. These are defined in the PHP calls to Celery, RabbitMQ, etc.
paddatrapper
on 7 May 2020
More testing - this affects api_client and pypo too
paddatrapper
on 7 May 2020
So yeah I agree that the solution is probably to setup a back-end callback URL. Would it need to be set in the /etc/airtime/airtime.conf file because that is what Python can easily access vs. requiring adding postgres integration to be able to set it via the web.
Robbt
on 8 May 2020
I think /etc/airtime/airtime.conf is the best place to do it. It should default to localhost, port 80 anyway. An admin changing it must really have good reason to.
Another thing I encountered, that we should probably document is that I needed to set the Icecast hostname (in /etc/icecast2/icecast.xml), otherwise the Stream Settings page was unable to connect to the stream (403 Forbidden) and master and show mounts would connect and the web UI would show that they were connected, but no audio would be heard and scheduled play would continue as normal
paddatrapper
on 8 May 2020
A fix for this needs to update https://github.com/LibreTime/libretime/blob/master/docs/manual/reverse-proxy/index.md to remove reference to this issue
paddatrapper
on 13 May 2020
1074 addresses the PHP part of this. I'll have a look how to evaluate this new flag in the pathon part too. Stay tuned,
caveman99
on 20 Jul 2020
Install guide says:
"There are known bugs when using LibreTime behind a reverse proxy (#957 tracks the issue and contains a temporary workaround)."
What is exactly the concrete (or best) workaround in this thread?
(I use Apache as reverse proxy on a frontend server; website works fine proxying HTTP but not when proxying HTTPS)
Install guide talks also about "SSL redirection" but I'm not redirecting anything but proxying to LibreTime backend server.
narcisgarcia
on 23 Oct 2020
@guythnick I've had problems with this too. I think the setting that needs to be changed is in the airtime.conf file:
[general]
api_key = xxxxxxxxxxxxxxxxxxxx
web_server_user = www-data
base_url = 192.168.200.10
base_port = 80
base_dir = /
cache_ahead_hours = 1
airtime_dir =
station_id =
auth = local
Try changing base_url to the URL of your proxy and see what happens.
zklosko
on 23 Oct 2020
Thank you zklosko you put me on the path for Apache scenario (AirTime host):
/etc/apache2/sites-enabled/airtime.conf
- Change
by - Add SSLEngine, SSLCertificateChainFile, SSLCertificateFile, SSLCertificateKeyFile directives to use x.509 certificate
- (if needed) Add CSP, SOP and other additional security directives.
/etc/airtime/airtime.conf
- Sure base_url points to public FQDN, such as myradio.example.net or www.example.net
- base_port = 443
- Sure base_dir points to public subdirectory, such as "/" for myradio.example.net or "/myradio" for www.example.net/myradio
- force_ssl = true
/etc/icecast2/icecast.xml
- Replace "
localhost " by public FQDN such as "myradio.example.net " or "www.example.net "
(I suppose this should be done for ALL Internet scenarios)
I don't understand the need to configure any CORS thing if I have everything at same public schema://name:port
narcisgarcia
on 24 Oct 2020
(GitHub ghosts? Apologies for such rows.)
Here to say that with Border Radio we had enough time to troubleshoot a bug on LibreTime's widgets, not working because of an environment similar to your one. With our changeset now LibreTime does not adopt absolute URLs and this means widgets work as expected whatever is the hostname. Let's see when it will be merged. Happy hacking!
1119
valerio-bozzolan
on 28 Nov 2020
I've been looking into this a little bit and I'm feeling stumped. I understand NGINX is a reverse proxy of choice but I've heard some arguments against reverse proxies recently (looking for the tab I closed; will edit once I have a link). Would it be bad to assign the SSL certificate with Apache directly, which will handle the redirecting from 80 to 443?
Icecast can also serve on 443 with an SSL cert but only if it's on a server that isn't already using 443 (like for the Libretime interface). Supposedly, web browsers will also autoredirect HTTP requests to HTTPS when using port 8443, so could we could serve Libretime on 443 (redirected from 80) and Icecast on 8443 (8000)?
zklosko
on 22 Dec 2020
I think the topic here is to support whatever reverse proxy, not just nginx. I had the same issue reported here with a simple Apache reverse proxy as frontend webserver.
valerio-bozzolan
on 23 Dec 2020
@zklosko you can just run SSL on Apache with no reverse proxy in front. That just doesn't work for all use cases. In fact SSL directly in apache is already well supported and documented - https://libretime.org/docs/ssl
paddatrapper
on 23 Dec 2020
Related issues
frecuencialibre
路
9Comments
zklosko
路
11Comments
techbp
路
13Comments
hairmare
路
5Comments
hairmare
路
14Comments