Pencil: Sound export audio samples synchronization issues

[Jose-Moreno]

Issue Summary

After performing troubleshooting over the forum on this thread the problem encountered was the following:

Upon exporting the file shared by the user, the resulting video prompts synchronization issues where the sound starts playing BEFORE the first frame changes to the next one.

In my tests, at first there's an approximate difference of 10 frames between video and audio, however after inspecting the resulting soundtrack, each sound sample seems to be shifted progressively so simply shifting the track for x frames is impossible as each sample is shifted individually in time.

It almost seems as if the sound has not been mapped to the frame measurement during export, however when performing playback with the Pencil2D editor it works as expected.

Note: The overlapping sounds in the sound layer track are not causing the synchronization issues. I already tested removing them and the same problem persists. I actually think it's because it's a really long soundtrack however we still can't rule out it's because of the resulting track length in FFMPEG or that it's because each sound should have it's own separate track (which would be very difficult to do even in a dedicated video editor due to the amount of sound samples)

Expected Results

Exported video sound should have synchronized video and audio

Steps to reproduce

1. Download the file provided by the user
2. Playback the file a few times (for cache purposes as the sound lags a bit if it's not cached and if your cache number is too low)
3. Then try a full playback to notice sound cues for reference.
4. Export the video (In test MP4 was used)
5. Notice there is a mismatch between audio and video which was not shown in the editor.

System Information

• Pencil2D Version:
  Original User: Wind 64 - October 12th Nightly Build

Tests:
Version: 0.6.2
commit: da6259877363bddad49414f58847d8823125f34c
date: 2018-12-20_12:07:36
Development build
Operating System: Windows 7 SP 1 (6.1)
CPU Architecture: x86_64

[scribblemaniac]

The issue should be fixed in the latest nightly build now. The source of the issue was with the sampling rate of the audio files being different from what was expected.

[Jose-Moreno]

@scribblemaniac I just tested this by exporting from the latest build. Indeed it's much better. Awesome job!

Related to this issue I'm a bit conflicted with the following problem: as an example you can notice in the linked user file in frame 281 where there's a scene change to a title card and you see the sound starts there, however in the exported video the sound starts playing at frame 275, 6 frames before it actually should start. Now there seems that a slight general delay still persists between the exported movie and how it plays in the editor when using Quicktime Player (and inconsistently Windows Media Player).

However viewing the same video in VLC or using KDEnlive editor the sound is correctly aligned with the video and no delay is perceived. I don't know If I should note that down for future release notes, since it seems a problem with Quicktime player in general but I'd like to hear your opinion on that as well before calling this solved.

[Jose-Moreno]

Actually now that I'm looking at the video now in KDE using a project at the same FPS rate (12fps) and there is indeed a 2 frame delay between where the sounds are placed in the Pencil2D editor and where the sounds are in the final video as you can see in the following images:

Of course this is not nearly as bad as it was before since 2 frames are not as noticeable (unless you are quicktime player apparently), but it still is a problem that needs to be solved.

Edit: Alright, now it's down to a 1 frame difference mainly because I noticed Qt and KDenlive start at 0 whereas Pencil2D exports from frame 1, so I moved the movie in the timeline 1 frame forward and now it's one less delay frame to worry about :stuck_out_tongue_closed_eyes:

[scribblemaniac]

@Jose-Moreno Yeah there's probably an off-by-one error somewhere. In fact I think I know where it is. I'll look into it later. It's not like Pencil2D is well suited for audio syncing even after this is fixed. As far as I know there are still multiple issues with the preview audio synchronization.