Openshot-qt: Audio Desynced on a Synced Video

[RoastedFeznt]

Describe the bug:
I am working with an mp4 clip of a video game. The mp4 clip is synced outside of OpenShot but becomes desynced when imported. The clip is 30 FPS and my profile is also set to 30 FPS.

Steps to reproduce the behavior:

1. Import video file
2. Preview or export video file

Expected behavior:
.mp4 and its audio play synced, as it does outside of the program.

System Details:

• OpenShot Version: 2.4.4
• Operating System: Windows 10

[SuslikV]

1. Can you post the file example?
2. Can you test the file from this post: https://github.com/OpenShot/openshot-qt/issues/1290#issuecomment-489569399 ?

[RoastedFeznt]

I'll do you one better. I have two versions of the footage. One is the file before it was added into openshot. The other is AFTER being added to openshot.

Version Pre-OpenShot
https://www.youtube.com/watch?v=kLzcCKmEiSw&feature=youtu.be

Version Run Through OpenShot
https://www.youtube.com/watch?v=0fEy1OMBKYE&feature=youtu.be

I am working with multiple MP4s, none of which have the same issue.

Edit: Attempted to upload both files here in a zip, but both are greater than 10 mb in size.

[RoastedFeznt]

@SuslikV I also loaded in the sync test video. It synced fine with no setup changes.

[SuslikV]

Can you post MediaInfo (.txt format) of the source file?

[RoastedFeznt]

Footage.txt

Got it

[SuslikV]

Can you add this file sample to google drive (for example) and post link here to share it? I want to test it on my end. MediaInfo report itself doesn't show any oddity that may cause described behavior. It's strange.

[RoastedFeznt]

https://drive.google.com/open?id=1WaArINk3QXC72Fty6Hhuh_zxoJ7eb9Xg

Done and done. Should also mention I did not change any OpenShot settings from their defaults on a new Windows 10 install.

[SuslikV]

I only found that OpenShot shows extremely high video_pts_offset=-29184.0000 in the log with this file.

[RoastedFeznt]

I don't know what that means or how to make it not offset. Or why it would offset.

[RoastedFeznt]

I manually fixed the file's audio desync by separating out its audio and then trimming. There is still no reason for it to have desynced by 2 seconds on import.

[SuslikV]

Surely, something wrong with OpenShot's import here. The file imports into Audacity quite well - sound has actual timing (and it uses FFmpeg as OpenShot for the import).

[SuslikV]

I found a way to crash OpenShot with posted sample file. Steps to reproduce the application crash:
1) Import Game Footage Fine.mp4 into OpenShot (you may drag-n-drop it)
2) Add file from p.1 to Timeline (at the beginning of any track)
3) Click Timeline cursor at about 30 seconds position
4) Click Timeline cursor at about 3 seconds position
5) Application will crash inside libopenshot.dll

@ferdnyc Can you confirm the crash?

[SuslikV]

Small update. It turns out that indeed the crash happens inside the FFmpegReader::ReadStream because of seek optimization that closes (then reopens) file and thus resets the packet from the stream to null (audio packet in our case). All following references to invalid structure fails. I have solution to workaround this, but it doesn't solves original desync issue. So, this probably will split into number of commits.

More over, there is other issue that appears to be present in openshot from the begging. The video frames may be placed into cache with different order (you'll see the frame in Preview that is not part of the sequence in the middle of some section). This requires more time to investigate.

[SuslikV]

After some digging into OpenShot I found next (not much of course):
1) OpenShot based on FFmpeg v3.4.2 fails to sync mentioned file.
2) OpenShot based on FFmpeg v4.2 playbacks file without issues.
3) Media has open GOP structure of the video, 7 B-frames (almost like Intel's encoder).
4) pts/dts of the first frames returns similar values in FFmpeg's probe (for both mentioned versions) but not in OpenShot...

The "bad sync" log (FFmpeg v3.4.2): 1 video sample (first video sample with pts=0, dts=-512 is missing) and 1 audio sample:

Click to expand

...
logger_libopenshot:INFO FFmpegReader::GetFrame (requested_frame=1.0000, last_frame=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (requested_frame=1.0000, OPEN_MP_NUM_PROCESSORS=2.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (GetNextPacket) (requested_frame=1.0000, processing_video_frames_size=0.0000, processing_audio_frames_size=0.0000, minimum_packets=2.0000, packets_processed=0.0000, is_seeking=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Video) (num_packets_since_video_frame=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Video) (packet->pts=3584.0000)
logger_libopenshot:INFO FFmpegReader::CheckMissingFrame (requested_frame=1.0000, has_missing_frames=0.0000, missing_video_frames.size()=0.0000, checked_count=1.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (GetNextPacket) (requested_frame=1.0000, processing_video_frames_size=0.0000, processing_audio_frames_size=0.0000, minimum_packets=2.0000, packets_processed=1.0000, is_seeking=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Audio) (num_packets_since_video_frame=1.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Audio) (packet->pts=0.0000)
...

The "good sync" log (FFmpeg v4.2), 2 video samples and 1 audio sample:

Click to expand

...
logger_libopenshot:INFO FFmpegReader::GetFrame (requested_frame=1.0000, last_frame=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (requested_frame=1.0000, OPEN_MP_NUM_PROCESSORS=2.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (GetNextPacket) (requested_frame=1.0000, processing_video_frames_size=0.0000, processing_audio_frames_size=0.0000, minimum_packets=2.0000, packets_processed=0.0000, is_seeking=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Video) (num_packets_since_video_frame=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Video) (packet->pts=0.0000)
logger_libopenshot:INFO FFmpegReader::CheckMissingFrame (requested_frame=1.0000, has_missing_frames=0.0000, missing_video_frames.size()=0.0000, checked_count=1.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (GetNextPacket) (requested_frame=1.0000, processing_video_frames_size=0.0000, processing_audio_frames_size=0.0000, minimum_packets=2.0000, packets_processed=1.0000, is_seeking=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Video) (num_packets_since_video_frame=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Video) (packet->pts=4096.0000)
logger_libopenshot:INFO FFmpegReader::CheckMissingFrame (requested_frame=1.0000, has_missing_frames=0.0000, missing_video_frames.size()=0.0000, checked_count=2.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (GetNextPacket) (requested_frame=1.0000, processing_video_frames_size=0.0000, processing_audio_frames_size=0.0000, minimum_packets=2.0000, packets_processed=2.0000, is_seeking=0.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Audio) (num_packets_since_video_frame=1.0000)
logger_libopenshot:INFO FFmpegReader::ReadStream (Audio) (packet->pts=0.0000)
...

Both logs are from the custom (more verbose) version of the OpenShot.

Conclusion
While it works well (synced) only for never versions of FFmpeg, the probe util of the FFmpeg itself shows no difference (identical pts/dts). Also the file re-encoded with the FFmpeg (both versions) works OK in OpenShot (synced). Finding the root of the bug may take even more time.

Workaround: use latest available daily build of the OpenShot (it based on FFmpeg v4.x right now).
https://www.openshot.org/download/

Edit: my ffprobe tests was based on ffmpeg-20180324-db2a7c9-win64-shared that works well, but exactly FFmpeg v3.4.2 (2018-02-25) ffprobe fails same way as OpenShot (wrong pts, dts of the frame and missing first packet).

[SuslikV]

OK. Now, I know a bit more about the FFmpeg seeking inside files. It seems that FFmpeg libs (that OpenShot uses) was changed 2018-03-09 that affects the mentioned sync issue. OpenShot uses default parser (no custom code) of the shared lib, so any later build of FFmpeg should work well (some early builds of it works well too).

It is more like third party lib issue (that was fixed later and thus was fixed in most recent test build of the OpenShot too).

[ferdnyc]

FFmpeg libs (that OpenShot uses) was changed 2018-03-09 that affects the mentioned sync issue. OpenShot uses default parser (no custom code) of the shared lib, so any later build of FFmpeg should work well (some early builds of it works well too).

The important date then, from an OpenShot perspective, would be 2019-07-20. Checking the build server logs, that's the date (just over a month ago) when Daily Builds of OpenShot on Windows started being packaged with FFmpeg 4.0.

[stale[bot]]

Thank you so much for submitting an issue to help improve OpenShot Video Editor. We are sorry about this, but this particular issue has gone unnoticed for quite some time. To help keep the OpenShot GitHub Issue Tracker organized and focused, we must ensure that every issue is correctly labelled and triaged, to get the proper attention.
This issue will be closed, as it meets the following criteria: - No activity in the past 180 days - No one is assigned to this issue
We'd like to ask you to help us out and determine whether this issue should be reopened. - If this issue is reporting a bug, please can you attempt to reproduce on the latest daily build to help us to understand whether the bug still needs our attention. - If this issue is proposing a new feature, please can you verify whether the feature proposal is still relevant.
Thanks again for your help!