C-toxcore: Video corruption: Don't drop video keyframes

Created on 5 Nov 2016 · 14Comments · Source: TokTok/c-toxcore

Originally posted by @tux3 in https://github.com/irungentoo/toxcore/issues/1479:

Currently as I understand it toxcore shoves all the video data over UDP and hopes it reaches the destination.

This creates horrible video corruption when a key frame is lost. On desktop streaming, intra frames are super rare so we can easily go 10s at a time with completely garbled video output. It's a common thing when calling EchoBot for example, with both uTox and qTox.

Toxcore should try on the sending end to retransmit iframes until it either gets an ACK or the next iframe is ready, and on the receiving end it shouldn't give Tox clients inter frames that depend on missing intra frames.

For more information: https://github.com/qTox/qTox/issues/2529

P1 bug network toxav

Source

ovalseven8

All 14 comments

@mannol Please help

alexeysvrv on 6 Nov 2016

For anyone watching, I'm not sure keyframes are that important anymore.

And either way, there's a lot more work than just this one issue.

GrayHatter on 6 Nov 2016

👎3

@GrayHatter Why keyframes are not important? Video and so will be broken?

alexeysvrv on 6 Nov 2016

Isotoxin currently uses lossless video transferring (only Isotoxin-Isotoxin calls, obviously), so it usable to share huge resolution desktop with good quality.
Lossy video transfering is horrible. A fortiori, lossy way of toxcore. Do you think toxcore lossy is just sending without confirmation of sending was successful? No way! Toxcore lossy is _DON'T SEND PACKET WHEN SENDING QUEUE IS FULL_! What a terrible solution...

isotoxin on 6 Nov 2016

@alexeysvrv sending JUST keyframes isn't the issue. The problem is toxav doesn't try to recover from a high data loss event, or temporary network instability. It's currently a 'dumb' protocol, that tries it's best, and never adapts.

Fixing the sending of just keyframes will solve a single issue, one that's not often reported. That is to say, if we magically never drop a keyframe, video quality will still be problematic.

GrayHatter on 6 Nov 2016

@GrayHatter What is the solution?

alexeysvrv on 6 Nov 2016

@alexeysvrv make ToxAV smarter. I was more verbose in #232

GrayHatter on 6 Nov 2016

@GrayHatter Are you doing this? You had a branch with the progress of the fix video

alexeysvrv on 7 Nov 2016

I had a branch with some experimental changes, but I abandoned it because I didn't like that method once I started.

I didn't do anything remotely like my current suggestion, and the newest toxav branch I have tries to switch from VPX to a newer codec, (that doesn't work)

GrayHatter on 7 Nov 2016

@GrayHatter You try to fix it? There is time to do this?

alexeysvrv on 7 Nov 2016

I'll have time eventually, but it's not likely I'll have the time within the next few months.

GrayHatter on 7 Nov 2016

@isotoxin i am trying to tweak c-toxcore video. can you help me?
are there windows binaries for isotoxin?
contact me please

zoff99 on 21 Nov 2017

👍1

@zoff99 careful with isotoxin. It's adds in a lot of extra magic that's not cross compatible with toxcore. Granted most of it probably should be in toxcore, but...

/me shrugs.

I'd be happy to help out with ToxAV if you need help. #utox on freenode with questions.

GrayHatter on 25 Nov 2017

I think this was fixed by zoff before.