Olive: [EXPORT] Huge memory usage on exporting a sequence, followed by a crash

[brmather]

Commit Hash ac4be090

Platform Linux

Summary

Huge memory usage, followed by a crash on exporting a sequence to mp4.

This looks similar to #1242 except that for me it applies to any video I try to export that is over approximately 1.5 minutes duration. My machine has 64 GB of RAM and it is filling all of it and more when I try to export a 2 minute 1080p mp4 clip using the AppImage provided. Initially I tried to compile Olive but I ran into the FFMPEG 4.3.1 error (#1176) and I'm not clever enough to work out how to compile Olive against a different version of FFMPEG.

Any help would be greatly appreciated! Here are some logs when Olive crashes:

Additional Information / Output

[swscaler @ 0x7f9c02fbe080] Warning: data is not aligned! This can lead to a speed loss
[libx264 @ 0x7f9a88006b40] using SAR=1/1
[libx264 @ 0x7f9a88006b40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 0x7f9a88006b40] profile High, level 4.0, 4:2:0, 8-bit
[libx264 @ 0x7f9a88006b40] 264 - core 161 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=18 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/home/ben/Dropbox/Shared/Karas_30th/Sequence 1.mp4':
    Stream #0:0: Video: h264, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31
    Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
[DEBUG] Using graphics driver: NVIDIA Corporation ((null):0)

(process:84149): GLib-CRITICAL **: 17:19:07.613: g_hash_table_contains: assertion 'hash_table != NULL' failed

(process:84149): GLib-CRITICAL **: 17:19:07.613: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed
[83545:83545:20201007,171907.616220:ERROR elf_image_reader.cc:116] no preferred header address
[83545:83545:20201007,171907.616299:ERROR process_reader_linux.cc:520] no module mappings 0x7f9c346fa000
[83545:83545:20201007,171907.616367:ERROR elf_image_reader.cc:116] no preferred header address
[83545:83545:20201007,171907.616376:ERROR process_reader_linux.cc:520] no module mappings 0x7f9c345e2000
[83545:83545:20201007,171907.616556:ERROR elf_image_reader.cc:116] no preferred header address
[83545:83545:20201007,171907.616564:ERROR process_reader_linux.cc:520] no module mappings 0x7f9c34524000

[zevero]

After I had no success to export my film with the latest windows nightlies, I gave it a try on a different machine on linux debian 10.6 by relinking all the files. First with appimage, then I compiled it from source.qt 5.12, gcc 8.3.0, ffmpeg 4.1.6.
In both cases there seems to be a memory leak being filled at a rate of 2GB per minute. With 16GB RAM I can export sequences up to 15s taking around 5m. Once the app is out of memory it justs displays [Killed] in the console and exits.
My source is a gopro8 Video H264 with 2704x2028 29.7.
I export to MEPG-4 H264 1440x1080 29.7. But other formats and resolutions behave same (even quicktime prores). Thread count does not have any influence on speed or threads used.
The exported video is scrambled UNLESS it has a title!?! Only sequences with a title have nice video output. Audio is fine.

What a pitty. It was so much fun editing my video.

Btw. on linux I can't save new projects to an .ove file, since there is no box for the filename.
I also tried with two other mpeg-4 videos. The first one keeps crashing when dragging to the timeline. The other one gives a [Segmentation fault] immediately on export.

How do you guys compile, what machines do you have, that you don't experience any export issues?
It is really amazing that you do this project ... and very daunting :)

[Simran-B]

The export problem sounds very much like the known issue Matt talked about in Discord:

The way it's set up, the renderer asynchronously sends frames to the encoder and I think it's just a matter of the renderer renders frames faster than the encoder encodes them, so they just build up in memory until the encoder gets through them all

You might be running OOM because the encoder lags behind. You could try to find an export setting that is very fast and see if the renderer fills up memory slower. That would speak for the theory.

[fmpwizard]

I'm running into the same issue, I'm working with a 2 minute 4k clip, when I export, memory goes all the way to 19GB and then olive crashes/exits. I'm using the latest prebuilt appImage Olive-de48eba7-Linux-x86_64.AppImage
Is there any details I can provide to help with debugging this? I can share the clip I have 587MB mp4 file

[itsmattkc]

render-singleton contains a fix for this. Should be merged into master within the next few days.

[itsmattkc]

render-singleton is merged. Memory usage should be within much more reasonable levels now.