Core: no mjpeg stream through apache2 proxy using ssl

[vquiering]

Home Assistant release with the issue:
0.67.1

Last working Home Assistant release (if known):
Tried with 0.66 before, same issue

Operating environment (Hass.io/Docker/Windows/etc.):
Raspbian 9.1 - virtualenv

Component/platform:
https://www.home-assistant.io/components/camera.mjpeg/
https://www.home-assistant.io/docs/ecosystem/apache/

Description of problem:
When serving through apache2 proxy with ssl enabled vhost, there is no camera stream available.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

camera:
  - platform: mjpeg
    name: 'camera_id'
    still_image_url: http://192.168.0.2/image/jpeg.cgi
    mjpeg_url: http://192.168.0.2/video/mjpg.cgi
    username: !secret mjpeg_user
    password: !secret mjpeg_password

Traceback (if applicable):
hass log:

Apr 19 10:33:44 myhass hass[17214]: 2018-04-19 10:33:44 INFO (MainThread) [homeassistant.components.http.view] Serving /api/camera_proxy/camera.camera_id to 127.0.0.1 (auth: False)
Apr 19 10:33:45 myhass hass[17214]: 2018-04-19 10:33:45 INFO (MainThread) [homeassistant.components.http.view] Serving /api/camera_proxy_stream/camera.camera_id to 127.0.0.1 (auth: False)
Apr 19 10:33:47 myhass hass[17214]: 2018-04-19 10:33:47 INFO (MainThread) [homeassistant.components.http.view] Serving /states to 127.0.0.1 (auth: False)
Apr 19 10:33:47 myhass hass[17214]: 2018-04-19 10:33:47 INFO (MainThread) [homeassistant.components.http.view] Serving /api/themes to 127.0.0.1 (auth: True)
Apr 19 10:33:47 myhass hass[17214]: 2018-04-19 10:33:47 INFO (MainThread) [homeassistant.components.http.view] Serving /api/translations/en to 127.0.0.1 (auth: True)
Apr 19 10:33:47 myhass hass[17214]: 2018-04-19 10:33:47 INFO (MainThread) [homeassistant.components.http.view] Serving /api/websocket to 127.0.0.1 (auth: False)
Apr 19 10:33:48 myhass hass[17214]: 2018-04-19 10:33:48 INFO (MainThread) [homeassistant.components.http.view] Serving /api/camera_proxy/camera.camera_id to 127.0.0.1 (auth: False)

apache2 log:

192.168.0.1 - - [19/Apr/2018:10:26:29 +0200] "GET /api/websocket?latest HTTP/1.1" 200 104595 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15"
192.168.0.1 - - [19/Apr/2018:10:33:23 +0200] "GET /states HTTP/1.1" 200 1786 "https://myurl/service_worker.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15"
192.168.0.1 - - [19/Apr/2018:10:33:23 +0200] "GET /service_worker.js HTTP/1.1" 304 322 "https://myurl/service_worker.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15"
192.168.0.1 - - [19/Apr/2018:10:33:23 +0200] "GET /service_worker.js HTTP/1.1" 304 322 "https://myurl/service_worker.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15"
192.168.0.1 - - [19/Apr/2018:10:33:24 +0200] "GET /api/themes HTTP/1.1" 200 286 "https://myurl/states" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15"
192.168.0.1 - - [19/Apr/2018:10:33:24 +0200] "GET /api/translations/en HTTP/1.1" 200 786 "https://myurl/states" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15"
192.168.0.1 - - [19/Apr/2018:10:33:24 +0200] "GET /api/camera_proxy/camera.camera_id?token=c58164a69089deaa6703779cd1bebc8585ce027cced95718ead1a57e6b9d7461&time=1524126804471 HTTP/1.1" 200 23771 "https://myurl/states" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15"
192.168.0.1 - - [19/Apr/2018:10:33:26 +0200] "GET /api/config/entity_registry/camera.camera_id HTTP/1.1" 404 292 "https://myurl/states" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15"
192.168.0.1 - - [19/Apr/2018:10:33:26 +0200] "GET /api/camera_proxy_stream/camera.camera_id?token=c58164a69089deaa6703779cd1bebc8585ce027cced95718ead1a57e6b9d7461 HTTP/1.1" 200 24353 "https://myurl/states" "Mozilla/5

Additional information:
Browser reports:

Failed to load resource: the server responded with a status of 404 (Not Found)
https://myurl/api/config/entity_registry/camera.camera_id

[vquiering]

Tried it with different browsers.
Does NOT work on:

• macOS Safari
• iOS Safari
• iOS Home-Assistant App

Does work on:

• macOS Firefox

[jamieshaw]

I'm seeing this issue as well, it appears to be a caching issue with Apple-based browsers, including the SafariControllerView.

I'm finding that if I reload the page with an empty cache (Shift-Click the reload icon in macOS Safari), or force quit iOS Safari or the Home Assistant app, the camera stream will load. Then refreshing the page without emptying the cache causes the issue to resurface.

To add specifics; I'm serving HA behind a reverse-proxied instance of macOS Server's apache2 server with SSL.

[vquiering]

@jamieshaw
Thank you very much. Looks like it is working in the iOS app as well. Don't know how I tested it that it did not work.

[jamieshaw]

Not sure if it's worth keeping this issue open as it may be an issue with how Home Assistant is rendered in Safari; and not necessarily a greater bug in Apple's browsers.

Until further investigation to rule out a bug in Safari's behaviour, I'd recommend leaving open.

[johnclawson]

I've got the same problem with HA 0.68.1 running behind an nginx proxy. No mjpeg stream to Safari on iOS or MacOS.

[carlostico]

Same problem here works with Chrome and Firefox not Safari ..

[moralmunky]

MacOS
Home Assistant 0.69.1
Apache Reverse Proxy (homebrew)
One FFMPEG camera playing an MP4 file
One FFMPED camera playing an RSTP camera stream
4 Static image JPG cameras
Chrome, Firefox, Safari, iOS app

Noticed the behavior after upgrading from .67 to .69.1. I'm seeing 200 success messages when starting the stream in the server logs and no errors in homeassistant.components.http.view or haffmpeg.core debug logs. The video stream is alive and will display through homekit. It displays a broken image icon, no errors in the web console other than the 4040 about the entity registry which another bug report says is normal.

Video Thumbnails: Display in all browsers
MP4 Stream: Displays only in Chrome
RTSP Stream: Doesn't display in any browser
Static JPG: Display full-size window in chrome and Firefox

[christ]

Found two possible causes. Depending on the type of camera, you might be hitting the max allowed connections. The Foscams I have are limited to 4 streams. Which brings me to the semi-workaround, I've disabled KeepAlive on my httpd proxy. The call api/camera_proxy_stream/camera.whatever sends a KeepAlive in the header, if you hit HA directly, the python webserver doesn't respond with KeepAlive while Apache httpd does. There are still times when the image doesn't load (still limited to 4 streams), but it's far less frequent.

[vquiering]

Just upgraded to home-assistant 0.73.
The stream ist working now through safari. Can anyone else confirm?

[flounderscore]

I have a similar problem with uvc (UniFi Camera NVR) and haproxy (SSL). The live stream shows just fine when I directly to home-assistant's local IP and port 8123, but not when going through haproxy.

Edit: Sorry, forgot to mention: works with chrome, doesn't work with safari.
Edit2: Shift-Click reload temporarily fixes the issue on Safari.

[nicx]

I am affected too when using Safari with HA 0.79.3, with Chrome it works fine.
When using Safari directly (http://192.168.0.10:8123) without proxy it works even with Safari.

[christ]

Found something on https://www.home-assistant.io/docs/ecosystem/apache/ that seems to have helped out a bit. Specifically disablereuse=on being set for the proxy. I just enabled that and so far it's been way more reliable loading the stream. I'm still limited by my cameras ability to only have 4 streams at once, but this is far better than before.

[nicx]

any news to this? who could help us? whats needed to help us?

[balloobbot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

[nicx]

nope still the same problem with latest release :(

[tkislan]

I have the same issue. Fails in Safari, but works in Chrome.
There is an error in the console, but no stacktrace nor reference to anything actually.

[Error] The operation couldn’t be completed. ( error 0.)
[Error] Cannot load .

[shmuelzon]

For what it's worth, here's what I found so far:
Just as @tkislan, I saw the same errors in safari. Using Wireshark, I can see that Safari closes the connection with a TLS/SSL alert "Level: Warning, Description: Close Notify".
The HTTP request is pretty much the same between an SSL connection (proxied by nginx) and a direct HTTP request to HA:

GET /api/camera_proxy_stream/camera.generic_camera?token=<token> HTTP/1.1
Host: raspberrypi.local:8123
Connection: keep-alive
Accept: image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15
Accept-Language: en-us
Referer: http://raspberrypi.local:8123/lovelace/default_view
Accept-Encoding: gzip, deflate

The server response is a bit different as the server name is different and some cache-control headers are included.
HTTP:

HTTP/1.1 200 OK
Content-Type: multipart/x-mixed-replace; boundary=--frameboundary
Transfer-Encoding: chunked
Date: Tue, 09 Apr 2019 14:31:19 GMT
Server: Python/3.5 aiohttp/3.5.4

HTTPS:

HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Tue, 09 Apr 2019 14:59:54 GMT
Content-Type: multipart/x-mixed-replace; boundary=--frameboundary
Transfer-Encoding: chunked
Connection: keep-alive
Strict-Transport-Security: max-age=31536000; includeSubdomains
Cache-Control: private
Cache-Control: no-cache
Cache-Control: no-store

Another thing that might be worth mentioning is that the initial chunk when using SSL is smaller and does not include the entire MJPEG frame:

ffa
--frameboundary
Content-Type: image/jpeg
Content-Length: 14893

As opposed to the HTTP variant:

48c6
--frameboundary
Content-Type: image/jpeg
Content-Length: 18560

The latter include the entire frame boundary and the frame itself (as expected).
Maybe Safari can't handle the MJPEG frame being split to multiple HTTP chunks?

Anyways, the requests and responses are the same for both Safari and Chrome so it's "just" a matter of Safari not handling it correctly.

Edit: I also compared the network dump on the proxy server and can confirm nginx is manipulating the HTTP chunks and seems to be limiting them to 0x1000 (4096) bytes.

Edit 2: it seems the MJPEG streams don’t really need to be chunk-encoded. I’ll try to see if I can remove that and if it makes any difference.

[shmuelzon]

After trying out a few things, I don't think it's directly related to the chunked encoding. Using mjpg-streamer as the server with the same nginx proxy works as expected.
I did notice that HA sends the boundary incorrectly in the header:

Content-Type: multipart/x-mixed-replace; boundary=--frameboundary

When it should be without the two dashes, i.e.:

Content-Type: multipart/x-mixed-replace; boundary=frameboundary

Fixing this does cause Safari to accept the first JPEG frame, but when the second one starts it resets the connection again. Not sure yet if it's something to do with HTTP chunks or the second multipart, but it's a start...

[nicx]

this is really annoying, still having that issue :(

[shmuelzon]

Sadly, I was never able to get to the root cause of the issue. It’s not directly related to SSL nor chunked encoding as I have another working example. I tried producing the same responses from mjpg-streamer in HA (same headers, same chunk sizes, etc.) but that didn’t work. The only difference remaining was the JPEG frame itself.
I was able to reach the point that Safari read the first complete frame but then it rejected it for some reason. Webkit’s code is quite complex and I couldn’t fine the actual location where the decision that the HTTP content isn’t an image was made, only caught it somewhere down the line...

[jamieshaw]

So I've just had another look into this, as it's one of the bugs still in my config.

I'm wondering if it's just Safari's poor service worker support in version 12? The latest technology preview doesn't appear to have an issue…

[nicx]

Hi @jamieshaw, I just tested it with the latest Tech Preview too... but I have exact the same issue. It is still only working with Chrome or Firefox, not with Safari. :(

[shmuelzon]

It’s been a while but, IIRC, after fixing the HTTP boundary, Webkit didn’t recognize the first frame as an image and closed the connection.
The only lead I have right now is the fact that the working example is an EXIF image while the non-working one is a JFIF. I don’t know if it’s related but I wanted to try and re-encode the frame in HA to see if that makes any difference but I never got around to trying.
@jamieshaw, @nicx, could you please check the JPEG format of your cameras and tell me which JPEG type it is? Looking at the first few bytes of the image should be enough. Maybe that could tell us if it’s related or not...

[nicx]

@shmuelzon am using Hikvision cams, they are using JFIF type.

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[vandalon]

I'm seeing this exact same issue with home assistant 0.104 and https enabled. Only in safari, chrome, firefox Fully, IOS app and Kiosk browser working fine.