Core: Camera streams not working

[DubhAd]

The problem

Camera streams are broken for me in 0.115.0b2. The snapshots in the cards update, but no stream shows, not even static images. At best I get errors in the log file (below)

Environment

• Home Assistant Core release with the issue: HA 0.115.0b2
• Last working Home Assistant Core release (if known): HA 0.114.4
• Operating environment (OS/Container/Supervised/Core): Debian/venv
• Integration causing this issue: Generic RTSP camera, MJPEG camera, local file
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/generic_ip_camera/ https://www.home-assistant.io/integrations/mjpeg/ https://www.home-assistant.io/integrations/local_file/

Problem-relevant configuration.yaml

camera:
- platform: local_file
  name: "Front camera from Doods"
  file_path: "/data/homeassistant/cameras/front_camera_latest.jpg"
- platform: generic
  name: 'Front camera'
  still_image_url: !secret front_camera_image
  stream_source: !secret front_camera_stream
  username: !secret front_camera_user
  password: !secret front_camera_pass
  authentication: basic
- platform: mjpeg
  name: 'Front camera motion'
  still_image_url: !secret front_motion_still
  mjpeg_url: !secret front_motion_mjpeg

Traceback/Error logs

2020-09-09 12:04:12 ERROR (stream_worker) [libav.rtsp] Undefined type (31)
2020-09-09 12:04:12 ERROR (stream_worker) [libav.h264] FMO not supported
2020-09-09 12:04:12 ERROR (stream_worker) [libav.h264] non-existing PPS 2 referenced
2020-09-09 12:04:12 ERROR (stream_worker) [libav.Eval] Undefined constant or missing '(' in 'skip_sidx'
2020-09-09 12:04:12 ERROR (stream_worker) [libav.mp4 muxer] Unable to parse option value "skip_sidx+frag_discont"
2020-09-09 12:04:12 ERROR (stream_worker) [libav.mp4 muxer] Error setting option movflags to value frag_custom+empty_moov+default_base_moof+skip_sidx+frag_discont.
2020-09-09 12:04:12 ERROR (stream_worker) [homeassistant.components.stream.worker] Stream connection failed: rtsp://192.168.1.42/ISAPI/Streaming/channels/101/
Traceback (most recent call last):
  File "/srv/ha_venv/3.8.5/lib/python3.8/site-packages/homeassistant/components/stream/worker.py", line 45, in stream_worker
    _stream_worker_internal(hass, stream, quit_event)
  File "/srv/ha_venv/3.8.5/lib/python3.8/site-packages/homeassistant/components/stream/worker.py", line 243, in _stream_worker_internal
    mux_video_packet(packet)  # mutates packet timestamps
  File "/srv/ha_venv/3.8.5/lib/python3.8/site-packages/homeassistant/components/stream/worker.py", line 173, in mux_video_packet
    buffer.output.mux(packet)
  File "av/container/output.pyx", line 204, in av.container.output.OutputContainer.mux
  File "av/container/output.pyx", line 210, in av.container.output.OutputContainer.mux_one
  File "av/container/output.pyx", line 180, in av.container.output.OutputContainer.start_encoding
  File "av/container/core.pyx", line 257, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.ValueError: [Errno 22] Invalid argument: '<none>'; last error log: [mp4 muxer] Error setting option movflags to value frag_custom+empty_moov+default_base_moof+skip_sidx+frag_discont.
2020-09-09 12:04:47 ERROR (stream_worker) [libav.Eval] Undefined constant or missing '(' in 'skip_sidx'
2020-09-09 12:04:47 ERROR (stream_worker) [libav.mp4 muxer] Unable to parse option value "skip_sidx+frag_discont"
2020-09-09 12:04:47 ERROR (stream_worker) [libav.mp4 muxer] Error setting option movflags to value frag_custom+empty_moov+default_base_moof+skip_sidx+frag_discont.
2020-09-09 12:04:47 ERROR (stream_worker) [homeassistant.components.stream.worker] Stream connection failed: rtsp://192.168.1.42/ISAPI/Streaming/channels/101/
Traceback (most recent call last):
  File "/srv/ha_venv/3.8.5/lib/python3.8/site-packages/homeassistant/components/stream/worker.py", line 45, in stream_worker
    _stream_worker_internal(hass, stream, quit_event)
  File "/srv/ha_venv/3.8.5/lib/python3.8/site-packages/homeassistant/components/stream/worker.py", line 243, in _stream_worker_internal
    mux_video_packet(packet)  # mutates packet timestamps
  File "/srv/ha_venv/3.8.5/lib/python3.8/site-packages/homeassistant/components/stream/worker.py", line 173, in mux_video_packet
    buffer.output.mux(packet)
  File "av/container/output.pyx", line 204, in av.container.output.OutputContainer.mux
  File "av/container/output.pyx", line 210, in av.container.output.OutputContainer.mux_one
  File "av/container/output.pyx", line 180, in av.container.output.OutputContainer.start_encoding
  File "av/container/core.pyx", line 257, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.ValueError: [Errno 22] Invalid argument: '<none>'; last error log: [mp4 muxer] Error setting option movflags to value frag_custom+empty_moov+default_base_moof+skip_sidx+frag_discont.

Additional information

ffmpeg version 4.2.2

[probot-home-assistant[bot]]

generic documentation
generic source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

local_file documentation
local_file source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

stream documentation
stream source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @hunterjm, @uvjustin, mind taking a look at this issue as its been labeled with an integration (stream) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[hunterjm]

What make/model camera is the generic camera source?

Are all 3 cameras broken (those with stream and the MJPEG/local file camera)?

[uvjustin]

Seems like it might be a problem with the libav libraries. Is everything the same from when you had it working with HA 0.114.4?

[DubhAd]

Yeah, 0.114 worked. I could quite understand if it's a libav issue, from the errors, but I'm not sure where to start other than _upgrade_.

All three no longer show the stream, even the one that's just a static image.

The generic camera is a Hikvison IPC-T140.

[uvjustin]

What version do you have in /usr/local/include/libavcodec/version.h ?

[hunterjm]

I think this may also be two separate issues. There was a frontend issue where MJPEG streams weren't showing correctly that should be fixed in an upcoming beta, but the one where the camera does support stream (generic with stream_source set) is what's throwing the other errors.

[pdobrien3]

I am also getting the same error on 0.115.0b4 with Hikvision DS-2CD2142FWD-IS.

Also these:

• Unable to parse option value "skip_sidx+frag_discont"
• Error setting option movflags to value frag_custom+empty_moov+default_base_moof+skip_sidx+frag_discont.
• Undefined constant or missing '(' in 'skip_sidx'

[uvjustin]

@pdobrien3 @DubhAd What architectures (eg armhf, x86_64) are your HA servers running on?

[pdobrien3]

Raspberry Pi 3B Home Assistant Core in a VEnv

[hunterjm]

What OS and OS version @pdobrien3? The additional errors are definitely going to be helpful in tracking this down.

[uvjustin]

@hunterjm If @DubhAd is also using an ARM architecture, I'd wager it's coming from the ffmpeg libraries, since the PyAV binary wheels are not available for ARM.

[hunterjm]

I don’t have a Pi, but I do have a Jetson Nano laying around that I can use to try and reproduce this on in a venv. Will also try with the Home Assistant ARM container since there is a pre-built AV wheel available for alpine/3.8.

[vandalon]

I'm seeing the same issue with the generic camera integration with a simple still_image_url.
This is on a raspberry pi4 running ha 0.115.b4 and the stream component disabled.
There are no errors in my logs nor in the web console but the images just don't show up when using the glance card with camera_view set to live. When set to auto the preview image does work though.

[pdobrien3]

Raspbian 10.4

[hunterjm]

I'm seeing the same issue with the generic camera integration with a simple still_image_url.
This is on a raspberry pi4 running ha 0.115.b4 and the stream component disabled.
There are no errors in my logs nor in the web console but the images just don't show up when using the glance card with camera_view set to live. When set to auto the preview image does work though.

Can we create a separate issue for that? Let's use this ticket for tracking the av errors which are related to stream.

[uvjustin]

For those with the stream error, can you share what you get for "av.library_versions" in your HA python interpreter?

[uvjustin]

@vandalon Your error is with the still_image_url and not the stream component, no?

[vandalon]

@vandalon Your error is with the still_image_url and not the stream component, no?

Yes, I was too quick :)

I think it might be frontend related even. but I don't have much time to dive into it right now

[hunterjm]

The MJPEG/non-stream related portion of this ticket is resolved in 0.115.0b5

[hunterjm]

On another note, my Jetson Nano SD Card reader is broken, so I've got no ARM devices to test on... Can try to run an emulated docker container on my host, but that might take some time to setup.

[Quentame]

Reproducing the exact same error on current dev (commit 5117a1684177e95697400ada52ecad7bcbc9c8bd) with the Synology DSM integration : #39937

It includes commit "Updated frontend to 20200909.0 (#39869)" from 0.115.0b5

Environnement: venv with macOS

[uvjustin]

Reproducing the exact same error on current dev (commit 5117a16) with the Synology DSM integration : #39937

It includes commit "Updated frontend to 20200909.0 (#39869)" from 0.115.0b5

Environnement: venv with macOS

This doesn't seem like the exact same error...

[hunterjm]

@DubhAd - Can you provide us with more info about your installation architecture/device/etc?

I spun up a debian 10.4-slim armv7 image and installed Home Assistant on it last night. Took forever with QEMU. Tonight I can try adding a camera to the install and installing AV, but that's going to take a while too I think. Want to make sure it's worth it.

[uvjustin]

@DubhAd - Can you provide us with more info about your installation architecture/device/etc?

I spun up a debian 10.4-slim armv7 image and installed Home Assistant on it last night. Took forever with QEMU. Tonight I can try adding a camera to the install and installing AV, but that's going to take a while too I think. Want to make sure it's worth it.

All signs point to PyAV not working because of a problem with the backend ffmpeg libraries. If this is the case spinning one up yourself won't be of much help if you get the proper libraries installed.

[hunterjm]

Ah yes, we are still waiting on av.library_versions from both @DubhAd and @pdobrien3 as well.

[pdobrien3]

I would be happy to help but I am not familiar with how to provide the information you need.

[hunterjm]

Here's the output for my dev environment. You can run the one-liner inside your venv.

$ echo -e "import av\nprint(av.library_versions)" | python3
{'libavutil': (56, 31, 100), 'libavcodec': (58, 54, 100), 'libavformat': (58, 29, 100), 'libavdevice': (58, 8, 100), 'libavfilter': (7, 57, 100), 'libswscale': (5, 5, 100), 'libswresample': (3, 5, 100)}

[pdobrien3]

{'libavutil': (56, 22, 100), 'libavcodec': (58, 35, 100), 'libavformat': (58, 20, 100), 'libavdevice': (58, 5, 100), 'libavfilter': (7, 40, 101), 'libswscale': (5, 3, 100), 'libswresample': (3, 3, 100)}

[hunterjm]

@pdobrien3 - So it looks like av was built against FFmpeg 4.1.6 on your install, and 4.2.4 on mine. Can you uninstall av from your venv, upgrade FFmpeg to at least 4.2.4, and try re-installing av in your venv and see if that solves the error?

[pdobrien3]

Pip install av?

[snaiperr]

Dear @hunterjm , the official raspbian repository does not have ffmpeg version 4.2.4.
Could you please advise how to update ffmpeg without compiling ?

[uvjustin]

The skip_sidx option was not added until Dec 2018 (https://patchwork.ffmpeg.org/project/ffmpeg/patch/[email protected]/) so it's probably not in ffmpeg 4.1.x. Given that the pi users don't have a binary wheel and the buster stable repository is still on 4.1.x, maybe we should take this option out and work around it if necessary. I'll make a PR in a bit

[uvjustin]

@DubhAd @pdobrien3 @snaiperr Can you check if this PR resolves the issue?

[pdobrien3]

I was running stream on HAss version 0.114. This only started showing up in the beta

[uvjustin]

Yes, from 0.114 to 0.115 we changed the container format for the HLS streams from TS to fMP4 in order to support H.265, and this issue seems to be related to the change. We've been testing it in dev for the past 3 or 4 weeks, but we didn't come across this problem until now (probably because not many people run the dev branch on RPis).

[pdobrien3]

Ok, I have only been playing with beta for the last couple releases. I am researching how to test a pr. If you could point me somewhere I would happily test. Maybe it is to big of a lift to teach though?

[uvjustin]

@pdobrien3 I think @hunterjm will end up merging this so it will get into the next beta/docker build.
But it's such a small change, if you're feeling adventurous you could just edit the file /usr/src/homeassistant/homeassistant/components/stream/hls.py and change line 151 from
"movflags": "frag_custom+empty_moov+default_base_moof+skip_sidx+frag_discont",
to
"movflags": "frag_custom+empty_moov+default_base_moof+frag_discont",
and reboot HA.

[snaiperr]

Dear @uvjustin , thank you! Now it works well!

[pdobrien3]

Yes, thank you. Working on my end also.

[DubhAd]

@DubhAd Can you check if this PR resolves the issue?

Will do. I'll also upgrade FFMPEG from 4.2.2 to 4.2.4 or later

[DubhAd]

I made the one line change and the streams load 👍

[hunterjm]

Good to hear. I merged the PR so should be resolved globally in beta 8.