Handbrake: x264-10bit encoding Memory Leak

[Schreina]

Hello,
(no native speaker, sorry :))

I am using the nightly build HandBrake-20160828-429fce1_x86_64-Win_GUI
and when encoding any file with the libx264_main10.dll (EDIT libx264_main10.7.dll was a typo) (from 11.August 2016) my PCs RAM fills up continually and permanently until i close Handbrake or Handbrake crashes.

There are no problems with 8bit-Encoding whatsoever. No problems with 10bit-encoding with the original x264.exe from videolan.org and command-line-encoding.

Problem happens in all presents (tested from fast to veryslow)
(See attached screenshots, after about 5 minutes of encoding 5 Gigabye of Ram are used, and it continues to use up gradually more, 10 Gig after 10 minutes, time with present veryslow),

Attached are the activity log from such an encode short enough to not crash the PC by filling up all available RAM. If you need anything else please tell me so.

activity_log6572.txt

[sr55]

Confirmed in Master. Confirmed with HandBrakeCLI.

While 10bit shows the most notable increase in memory, even using x264 appears to be leaking, but at a slower rate.

What I also noticed was that after stopping an x264 encode, the memory is not being freed.

[jstebbins]

When I first saw this report, I ran some valgrind tests against a queue of x264 encodes. I saw and fixed a couple of small memory leaks 058fc01 and 0dbad99. But neither of these should be big enough for what is reported here. valgrind reported no other leaks.

[bradleysepos]

@Schreina Is the dll named libx264_main10.7.dll or libx264_main10.dll?

[Schreina]

@bradleysepos it's the libx264_main10.dll i got from here https://forum.handbrake.fr/viewtopic.php?f=11&t=34165
Problem persists with newest nighly build as of 16. September

i really dont know why there is a 7 in the filename in my first post.

[bradleysepos]

Okay, no problem, I was just curious.

[bradleysepos]

Can you provide us with a small sample .mov that is problematic?

[Schreina]

You mean videofile to encode from? I happens with every file and codec-container-combination (mkv-mpeg2-SD-resolution and mov-h264-FullHD tested so far)
65MB-Testclip https://www.dropbox.com/s/7w09vn6a0c4y8fh/D71_0547.MOV?dl=1

[bradleysepos]

Thanks.

[jstebbins]

Fixed c18bbea
I only ran 8 bit encodes earlier since I didn't have the 10bit encoder handy. I forgot that there is actually an extra buffer involved in the 10bit encodes and it was not getting freed.

[sr55]

Nice, I'll confirm the fix tomorrow. Thanks

[sr55]

Confirmed Fixed.