Core: Stream - Malformed AAC bitstream detected

[cgtobi]

The problem

When trying to access a stream from a Netatmo camera it loads forever. This happens on a RaspberryPi 3 with Ubuntu 20.04 inside a venv. Unfortunately I can't reproduce this in my dev environment, Ubuntu 18.04.5 LTS with ffmpeg 7:4.3-2~18.04.york0.

Environment

• Home Assistant Core release with the issue: Home Assistant 0.115.0b0
• Last working Home Assistant Core release (if known): Home Assistant 0.114.0
• Operating environment (OS/Container/Supervised/Core): Ubuntu 20.04.1 LTS, core (venv)
• Integration causing this issue: stream with Netatmo
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/stream/ & https://www.home-assistant.io/integrations/netatmo/

Problem-relevant configuration.yaml

Traceback/Error logs

Sep 09 10:59:11 hapi hass[1991]: 2020-09-09 10:59:11 ERROR (stream_worker) [libav.mp4] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
Sep 09 10:59:12 hapi hass[1991]: Exception ignored in: 'av.container.output.OutputContainer.__dealloc__'
Sep 09 10:59:12 hapi hass[1991]: Traceback (most recent call last):
Sep 09 10:59:12 hapi hass[1991]:   File "av/container/output.pyx", line 23, in av.container.output.close_output
Sep 09 10:59:12 hapi hass[1991]:   File "av/container/core.pyx", line 257, in av.container.core.Container.err_check
Sep 09 10:59:12 hapi hass[1991]:   File "av/error.pyx", line 323, in av.error.err_check
Sep 09 10:59:12 hapi hass[1991]:   File "av/container/pyio.pyx", line 39, in av.container.pyio.pyio_write_gil
Sep 09 10:59:12 hapi hass[1991]: TypeError: 'NoneType' object is not callable

Additional information

Installed ffmpeg version: 7:4.2.2-1ubuntu1

[probot-home-assistant[bot]]

netatmo documentation
netatmo source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

stream documentation
stream source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @hunterjm, @uvjustin, mind taking a look at this issue as its been labeled with an integration (stream) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[uvjustin]

@hunterjm I saw you opened an issue on PyAV regarding bitstream filters back in Feb 2019: https://github.com/PyAV-Org/PyAV/issues/489. Any luck with getting them to work?
Otherwise we might have to try to disable audio for these feeds either by detection or by a configuration.yaml setting.

[uvjustin]

@cgtobi When you say you can't reproduce this in your dev environment, do you mean that the dev environment doesn't have this error or that you were unable to test it there?

[cgtobi]

@uvjustin dev environment (x86_64) doesn't show the error.

[uvjustin]

There are a lot of problems which stem from missing/incompatible dependencies for PyAV and they helped work around that by releasing binary wheels. Unfortunately I don't see any PyAV binary wheels for the RPi. Can you make sure ffmpeg-dev is installed through apt and then reinstall pyav 8.0.2 in pip?

[hunterjm]

@hunterjm I saw you opened an issue on PyAV regarding bitstream filters back in Feb 2019: PyAV-Org/PyAV#489. Any luck with getting them to work?

@uvjustin - I was able to work around that some other way. FFMpeg errors aren't always the most helpful. The Netatmo integration provides the stream as HLS already, and a quick search of documentation shows this:

This filter is required for example when copying an AAC stream from a raw ADTS AAC or an MPEG-TS container to MP4A-LATM, to an FLV file, or to MOV/MP4 files and related formats such as 3GP or M4A.

Since Netatmo HLS is MPEG-TS I'm assuming that's why we are getting the error. Not sure how to auto-check and enable that though... Maybe his dev box is also automatically adding that and prod is not?

[cgtobi]

I am reinstalling av right now. Might take a while on the Pi.

For reference: pip install av --no-binary av

[cgtobi]

This didn't change anything unfortunately.

[uvjustin]

Did you also reinstall the ffmpeg-dev libraries? Is the dev environment that works running 0.115.0b0 too?

[cgtobi]

Those dependencies are installed and updated to the latest available version:

libavformat-dev/focal-updates,focal-security,now 7:4.2.4-1ubuntu0.1 armhf 
libavcodec-dev/focal-updates,focal-security,now 7:4.2.4-1ubuntu0.1 armhf 
libavdevice-dev/focal-updates,focal-security,now 7:4.2.4-1ubuntu0.1 armhf
libavutil-dev/focal-updates,focal-security,now 7:4.2.4-1ubuntu0.1 armhf
libavfilter-dev/focal-updates,focal-security,now 7:4.2.4-1ubuntu0.1 armhf
libavformat-dev/focal-updates,focal-security,now 7:4.2.4-1ubuntu0.1 armhf
libswscale-dev/focal-updates,focal-security,now 7:4.2.4-1ubuntu0.1 armhf
libswresample-dev/focal-updates,focal-security,now 7:4.2.4-1ubuntu0.1 armhf

There is no ffmpeg-dev on Ubuntu.

The dev environment runs the latest dev, so at least equivalent to 0.115.0b0.

[uvjustin]

Since the netatmo stream source is hls, is is probably better to just proxy the hls playlist and segments directly instead of processing the packets like we do for rtsp streams.
However, I'm not sure why this is not working and it's curious that it works on one platform and not another. The differences seem to be 1) ffmpeg version and probably 2) architecture.
I had a quick look at the ffmpeg commits and I didn't see anything between 4.2 and 4.3 that would cause/fix this error.
@hunterjm The bitstream filter should be automatically applied here https://ffmpeg.org/doxygen/trunk/movenc_8c_source.html#l07017 . For some reason it's not happening on the one system but it is on the other.

[uvjustin]

@cgtobi Just to check, what do you get for av.library_versions in your venv python interpreter?

[cgtobi]

On prod:

Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import av
>>> av.library_versions
{'libavutil': (56, 31, 100), 'libavcodec': (58, 54, 100), 'libavformat': (58, 29, 100), 'libavdevice': (58, 8, 100), 'libavfilter': (7, 57, 100), 'libswscale': (5, 5, 100), 'libswresample': (3, 5, 100)}

On dev:

Python 3.8.5 (default, Jul 20 2020, 19:48:14)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import av
>>> av.library_versions
{'libavutil': (56, 31, 100), 'libavcodec': (58, 54, 100), 'libavformat': (58, 29, 100), 'libavdevice': (58, 8, 100), 'libavfilter': (7, 57, 100), 'libswscale': (5, 5, 100), 'libswresample': (3, 5, 100)}

[uvjustin]

I see, nothing to investigate there :-(. Those library versions are identical and correspond to ffmpeg 4.2.x which makes sense as your RPi is using 4.2.x and the binary wheel for 8.0.2 which I assume your dev box is using also uses 4.2.x.

[cgtobi]

Insteresting. I messed too much with my dev setup last night. Because I can now reproduce the error with the binary wheel.

And after building with --no-binary on dev I now get:

 {'libavutil': (56, 51, 100), 'libavcodec': (58, 91, 100), 'libavformat': (58, 45, 100), 'libavdevice': (58, 10, 100), 'libavfilter': (7, 85, 100), 'libswscale': (5, 7, 100), 'libswresample': (3, 7, 100)}

But the error stays. So at least now it's broken consistently.

Now I don't why exactly it worked in the past days and why it is now broken after reinstalling all dependencies etc.

[uvjustin]

Since the netatmo stream source is hls, is is probably better to just proxy the hls playlist and segments directly instead of processing the packets like we do for rtsp streams.
However, I'm not sure why this is not working and it's curious that it works on one platform and not another. The differences seem to be 1) ffmpeg version and probably 2) architecture.
I had a quick look at the ffmpeg commits and I didn't see anything between 4.2 and 4.3 that would cause/fix this error.
@hunterjm The bitstream filter should be automatically applied here https://ffmpeg.org/doxygen/trunk/movenc_8c_source.html#l07017 . For some reason it's not happening on the one system but it is on the other.

@hunterjm Actually the autobsf flag gets unset when we use empty_moov : https://ffmpeg.org/doxygen/trunk/movenc_8c_source.html#l06330 . This would explain the error when remuxing from a mpeg-ts container.
Any ideas how to apply the bitstream filter manually? In the meantime we can just disable audio for HLS streams.

[uvjustin]

Insteresting. I messed too much with my dev setup last night. Because I can now reproduce the error with the binary wheel.

And after building with --no-binary on dev I now get:

 {'libavutil': (56, 51, 100), 'libavcodec': (58, 91, 100), 'libavformat': (58, 45, 100), 'libavdevice': (58, 10, 100), 'libavfilter': (7, 85, 100), 'libswscale': (5, 7, 100), 'libswresample': (3, 7, 100)}

But the error stays. So at least now it's broken consistently.

Now I don't why exactly it worked in the past days and why it is now broken after reinstalling all dependencies etc.

Hmm, not sure if this is better or worse :-)

[cgtobi]

According to feedback from another user this happens on supervised on latest beta as well.

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 45, in stream_worker
    _stream_worker_internal(hass, stream, quit_event)
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 243, in _stream_worker_internal
    mux_video_packet(packet)  # mutates packet timestamps
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 173, in mux_video_packet
    buffer.output.mux(packet)
  File "av/container/output.pyx", line 204, in av.container.output.OutputContainer.mux
  File "av/container/output.pyx", line 227, in av.container.output.OutputContainer.mux_one
  File "av/container/core.pyx", line 257, in av.container.core.Container.err_check
  File "av/error.pyx", line 336, in av.error.err_check
av.error.PermissionError: [Errno 1] Operation not permitted: '<none>'; last error log: [mp4] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)

Santé du système
arch    x86_64
chassis    desktop
dev    false
docker    true
docker_version    19.03.12
hassio    true
host_os    Ubuntu 20.04.1 LTS
installation_type    Home Assistant Supervised
os_name    Linux
os_version    5.4.0-47-generic
python_version    3.8.4
supervisor    242
timezone    Europe/Paris
version    0.115.0b4
virtualenv    false

[uvjustin]

Thanks. I think we have identified the issue, just need to make a workaround.

[hunterjm]

Thanks. I think we have identified the issue, just need to make a workaround.

https://pyav.org/docs/develop/api/container.html#flags has a way of setting it? I can't find any examples on how to do that though...

[hunterjm]

Nevermind, that gets set then removed again on the moov_init... We either need pyAV to implement BSF, implement the filter ourselves, or add a check on the source container to disable audio for MPEG-TS/HLS sources.

Long term, proxying browser supported formats (HLS, Progressive MP4) directly would definitely be preferable, but that won't be a 0.115 fix.

[hunterjm]

FWIW, my vote is to remove audio for now since these users currently don't have it in Home Assistant anyway.

[uvjustin]

I'll disable audio for HLS sources for now. Post a PR in a bit. BTW, I'm working on top of that other overflow/non-monotonic timestamp PR I made..want me to take that out?

[hunterjm]

I'll disable audio for HLS sources for now. Post a PR in a bit. BTW, I'm working on top of that other overflow/non-monotonic timestamp PR I made..want me to take that out?

Yeah, I think it's best to do it as a new branch off of dev based on the standing policy not to include anything other than bugfixes after a beta has been cut.

[cgtobi]

Thank you, guys, for the quick fix.