Node-ytdl-core: error 500 when trying to download a specific video

[nikooo777]

Hello there,

I am trying to move away from youtube-dl for a specific issue I found and also due to the author not actively maintaining the lib anymore.

I see your project is getting updated so i wanted to test it out.

Unfortunately I found out the same issue on your project which I invite you to try.

Try downloading http://www.youtube.com/watch?v=fxlbIS8CoW8 using best quality (by default)

const fs = require('fs');
const ytdl = require('ytdl-core');

ytdl('http://www.youtube.com/watch?v=fxlbIS8CoW8')
  .pipe(fs.createWriteStream('video.flv'));

This should suffice to break the library. for some unknown reasons this specific video and several others will not work.

I am downloading thousand of vidoes for a project and while this might happen rarely, it happens quite a lot for me.

Let me know if you need further details to reproduce the bug.

Hopefully you'll find the reason, i wasn't able to.

Best regards,
Nikooo777

[tzarebczan]

@fent any idea when this could be fixed?

[fent]

Don't know what is causing this so can't say.

[nikooo777]

I believe there is something wrong with the audio channel, i'm not 100% sure of that tho.

I managed to catch the error thus avoiding a crash by doing this:

const fs = require('fs');
const ytdl = require('ytdl-core');

var video = ytdl('http://www.youtube.com/watch?v=fxlbIS8CoW8');
video.pipe(fs.createWriteStream('video.flv'));

video.on('error', (e) => {
    console.error("error " + e);
});
video.on('end', () => {
    console.log('[YoutubeDownload] : Download finished for video ');
});

[fent]

That just catches the error, avoiding the node process from crashing. But there still is an error..

[cacheflow]

Found the issue and patched it. Some of the highest quality video URLs are sometimes not accessible for whatever reason on Youtube. It even happened with my own parser that I tested. Made a PR that checks the request and downloads the baseline video if the highest quality fails to download.

https://github.com/fent/node-ytdl-core/pull/164

[nikooo777]

i think i managed to break it further. I applied your patch and got stuck at this one:

http://www.youtube.com/watch?v=NEn1ayFOD28

Exception has occurred: Error
RangeError: Invalid string length
    at IncomingMessage.<anonymous> (/home/niko/repositories/youtube-poc/node_modules/miniget/lib/index.js:71:32)
    at emitOne (events.js:96:13)
    at IncomingMessage.emit (events.js:191:7)
    at readableAddChunk (_stream_readable.js:178:18)
    at IncomingMessage.Readable.push (_stream_readable.js:136:10)
    at HTTPParser.parserOnBody (_http_common.js:123:22)
    at TLSSocket.socketOnData (_http_client.js:411:20)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:191:7)
    at readableAddChunk (_stream_readable.js:178:18)

Is this related?

[cacheflow]

I'll have to look into it. @fent is there any reason why you're using miniget over request? Is it because the latter is too large?

[fent]

Yep, that's the main reason, it's almost 3 MB last I checked. Much of its features aren't used by ytdl-core either.

[fent]

Hmm, it doesn't look like it's erroring anymore. Can you try yourself again?

[fent]

Nevermind, bad test.

[cacheflow]

Tested it again and same error. :(. What I can't figure out is why this only occurs for this video or a very small subset of them.

[unknownguy2002]

Processing the url seems fine, but it is the hosting url that gets the HTTP error 500

[d4rckh]

Did you tried to change the format of the file from .flv to .mp4 ?

[maple3142]

This problem not only happened to ytdl-core.
I tested it on youtube-dl (a popular youtube download cli) and ytdl (my youtube download implementation).
Both of them failed to download https://www.youtube.com/watch?v=fxlbIS8CoW8 either, maybe it is youtube server problem.

youtube-dl test command: youtube-dl -v -f 22 http://www.youtube.com/watch?v=fxlbIS8CoW8 (error only happened on specific format)

[fent]

I've been testing this, itag 22 is the only format returning a 500 error. All others work, at least for this video. https://github.com/fent/node-ytdl-core/issues/212 is similar, that itag 22 errors, but the error is different.

a possible workaround could be having a format fallback option that would try other formats if the first choice failed. it wouldn't fix the issue that itag 22 is having, but would circumvent it.

I'm thinking making the syntax like

formatFallback: false | 'all' | filtered

• false - turned off, would not try other formats
• 'filtered' - would try other formats that have been filtered. that is, if the user used filter: 'audioonly' for example, it would try the 2nd choice of audioonly formats if the 1st choice fails.
• 'all' - would try the next best choice regardless of filter

it might also be worth it to have a formatFallbackMax: 3 for maximum formats to fallback on

[fent]

kind of a weird "fix". basically, default sorting of formats are re-ordered so that those with no contentLength, such as itag 22 here, are placed last. additionally, m3u8 playlists are preferred over dash playlists due to https://github.com/fent/node-ytdl-core/issues/732.

if you think this sorting needs more tweaking or disagree, let me know or submit a PR.