Youtubeexplode: HTTP 404 error when getting video using GetVideoAsync()

[Kriste00]

Enter Youtube video ID or URL: https://www.youtube.com/watch?v=cGbP2y7Obas
Loading...

Unhandled Exception: System.Net.Http.HttpRequestException: Answer status codes do not indicate that the request was successful: 404 (Not Found). by System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode () by YoutubeExplode.Services.Extensions.d__0.MoveNext () --- End of track tracking from previous location where exception was taken --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () by System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (Task task) by System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult () at YoutubeExplode.YoutubeClient.d__34.MoveNext () --- End of track tracking from previous location where exception was taken --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () by System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (Task task) by System.Runtime.CompilerServices.TaskAwaiter1.GetResult () by DemoConsole.Program.d__2.MoveNext () --- End of track tracking from previous location where exception was taken --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () by System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (Task task) by System.Runtime.CompilerServices.TaskAwaiter.GetResult () by DemoConsole.Program.Main (String [] args)

[darkdoctor]

Just monitored the issue.

It looks like youtube just has stopped this service.
/get_video_metadata?video_id=videoid

That is why, so now we have to wait for developer to see another way to complete this service, or to remove from project.

[Kriste00]

Is /get_video_info?video_id= the same as /get_video_metadata?video_id=?

[Tyrrrz]

I just woke up. Will look into it asap. Looks like @darkdoctor is right, get_video_metadata doesn't work anymore.

[asdf1234anon]

I'd suggest moving that get_video_metadata stuff into a new method - since getting the video/audio stream does not require that stuff.

Btw, how do I support your project? great work

[Tyrrrz]

Yeah that seems like a good idea. Right now, I can't find any drop-in replacement for get_video_metadata anyway, but it'd be unfortunate if I had to remove a big part of metadata from models because we can't obtain it anymore.

You can recommend the library to a friend or help with development. Currently there's no other way to support it.

[Tyrrrz]

It looks like most of the missing metadata can be extracted from JSON on the embed page, except for video likes/dislikes and description. The only place I found it so far is on the watch page, which is not accessible for some videos.

[acaladolopes]

I'm also having the same issue..
I'm mostly interested in obtaining the subtitles and not the video or the audio. Have you seen subtitles information in that json you mention?

Thanks

[Tyrrrz]

Subtitles are not affected. get_video_metadata was used for description, likes, dislikes, extended author information (name, url, avatar, banner, etc). Most of the author information can be obtained elsewhere, but description/likes/dislikes don't seem to be available anywhere.

[Tyrrrz]

I'll investigate more into it tomorrow, my brains are fried after staring at Chrome Developer Console for so long. 😆
As a temporary measure, I can publish a pre-release package that ignores this additional metadata (corresponding properties will be empty). Would anyone benefit from this?

[Kriste00]

Yes, but is there no way to get the description?

[Kriste00]

And views

[ndt0126]

Please do release a work-around, I only need the urls.

[Tyrrrz]

@Kriste00 can't find it anywhere except for the watch page. Maybe someone else can help? Views are fine tho.
@ndt0126 ok, I will do that then. I imagine you're not the only one.

[sibeliuz]

@Tyrrrz yes, i'm interested too, but will this work-around allow to download video from subject URL (https://www.youtube.com/watch?v=cGbP2y7Obas) or similar videos ?

[BrunoZell]

I'd appreciate a prerelease too

[Tyrrrz]

@sibeliuz I'm not seeing anything special about that video, so I think so. As of right now, GetVideoAsync doesn't work on anything.

[ComputerMaster1st]

Hi There,

Would reading the page source be of any use instead of using the "get_video_metadata"/"get_video_info"?

[HiradNikoo]

If you need to read the page for info CSQuery is for that!

[Tyrrrz]

@ComputerMaster1st, yes for some things, but not for all.

[BrunoZell]

Alternatively HtmlAgilityPack

[Kriste00]

GetPlaylist returns videos with likes and dislikes without getting the 404.

[Tyrrrz]

Unsurprisingly, since it uses a different API :)

[ComputerMaster1st]

I've been playing with the source a bit and I've found out that

"var request = $"{YoutubeHost}/get_video_info?video_id={videoId}&sts={context.Sts}&el=info&ps=default&hl=en";"

gives 404 while

"var request = $"{YoutubeHost}/get_video_info?video_id={videoId}&el=info&ps=default&hl=en";"

gives 403. Probably useless though.

[Tyrrrz]

@ComputerMaster1st indeed, this just makes it fail earlier. Sts is important.

[ComputerMaster1st]

What I'm struggling to understand is, why can us users can see it and yet the library doesn't...

URL to prove the url structure in code should work (if the sts is correct): https://www.youtube.com/get_video_info?video_id=TbdArWwdNlU&sts=17492&el=info&ps=default&hl=en

[Tyrrrz]

Here's a package version that doesn't query get_video_metadata for those that need a solution ASAP. Properly fixing this may take some time.

https://www.nuget.org/packages/YoutubeExplode/3.4.0-workaround404

[Tyrrrz]

@ComputerMaster1st the library normally uses reverse-engineered AJAX requests because they tend to be more reliable and easier to parse. What happened is one such endpoint got removed. Scrapping page source is definitely possible, although undesirable. The major problem about it -- you won't be able to access video's watch page if it has some restrictions (e.g. age), which wasn't a problem with these requests.

[ComputerMaster1st]

Fix works perfectly for me so far. I'm using this project of yours for my discord bot btw. I'll keep an eye on for now just incase something unexpected pops up. ;)

[Tyrrrz]

Please do note that it's just a workaround and not a fix.

[ComputerMaster1st]

I can wait for a more permanent fix. Hence why I'm watching atm.

[vinaykapadia]

Good luck with the fix, Tyrrrz. I'll see if I can look into some solutions as well. Using the workaround for now.

[darkdoctor]

Greetings.

Tried the last update, workaround for 404, but still It gives me the same error.
Please just assign empty strings to meta datas and return them as empty or null.

[Tyrrrz]

Please just assign empty strings to meta datas and return them as empty or null.

That's exactly what it does

[darkdoctor]

You can get the channel name from this
https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v=oF80BveZt38&format=json

[Tyrrrz]

Yeah, I'm aware.
Here's what I've found so far:

get_video_info/?el=info
+ easily accessible closed caption track manifest
+ works on videos that don't allow embedding
~ has some minor additional info
- doesn't work on age-restricted videos
- doesn't have author channel information
- doesn't have like/dislikes/description

get_video_info/?el=adunit
+ has author channel information
+ works on videos that don't allow embedding
~ closed caption tracks accessible through JSON
- doesn't work on age-restricted videos
- doesn't have like/dislikes/description

get_video_info/?el=embedded
+ works on age-restricted videos
~ closed caption tracks accessible through JSON
- doesn't work on videos that don't allow embedding
- doesn't have author channel information
- doesn't have like/dislikes/description

get_video_info/?el=detailpage
= same as el=adunit

get_video_info/?el=
= same as el=info

get_video_info/
= same as el=embedded

watch page
+ has all information
+ works on videos that don't allow embedding
- very annoying to parse because everything is either in heavily nested JSON or HTML
- doesn't work on age-restricted videos

embed page
+ works on videos that don't allow embedding (ironically)
+ works on age-restricted videos
+ we download it already anyway
+ has author channel information
- no media stream information (will still have to use other methods)
- doesn't have like/dislikes/description

[Tyrrrz]

My plan to proceed is to split GetVideoAsync into multiple workflows that would either get metadata, get media streams or get closed caption tracks, etc. That way the library can utilize different queries that provide different data. That also means if one API endpoint breaks, the others are more likely to remain intact. The only downside is more GET requests, but considering get_video_info content length is really small, I don't think it's going to be an issue.

[AnonymusRaccoon]

Hey, I'm using your library on a xamarin android project. Since the youtube change, using
var videoInfo = await client.GetVideoAsync(videoID); on a function unable this one to be called (when you call the function, the app crash but the function isn't executed). The workaround don't solve the isue. Great project btw.

[Tyrrrz]

What error are you getting with workaround?

[AnonymusRaccoon]

This is the code i run:
private void foo() { Play(videoID); }
And this the function Play:
async void Play(string videoID) { YoutubeClient client = new YoutubeClient(); var videoInfo = await client.GetVideoAsync(videoID);

I have this exeption when the function foo is executed:

Unhandled Exception:

System.TypeLoadException: Could not load type of field 'MusicApp.Resources.Portable_Class.YoutubeEngine+d__13:5__2' (5) due to: Could not resolve type with token 01000068 (from typeref, class/assembly YoutubeExplode.Models.Video, YoutubeExplode, Version=3.4.0.0, Culture=neutral, PublicKeyToken=null) assembly:YoutubeExplode, Version=3.4.0.0, Culture=neutral, PublicKeyToken=null type:YoutubeExplode.Models.Video member:

[BrunoZell]

It could be that you have referenced a library that is using an older version of YoutubeExcplode, but the compiled project is using a newer one. There were some model changes inbetween and since the newer dll will be used the code in the library will crash

[AnonymusRaccoon]

I'm using this references for youtube explode:

using YoutubeExplode; using YoutubeExplode.Models; using YoutubeExplode.Models.MediaStreams;

I don't think they are form the old library

[BrunoZell]

Are you actually using multiple libraries in you project? If so, check the package version in the csproj:

<PackageReference Include="YoutubeExplode" Version="3.4.0-workaround404" />

I think the huge rework was in version 3.4.0, so any version prior to that can't mix with newer ones.

Btw @Tyrrrz do you know Semantic Versioning? As i know NuGet is using that, so that could have avoided such conflicts.

[AnonymusRaccoon]

I've only the last version on the project.

[BrunoZell]

Okay, then I can't help you. I just had the issue of version incompatability recently.

[Tyrrrz]

I have no idea what that error implies, can you reproduce this on a new project?

What kind of incompatibility issues were you having? Can you message me on gitter?

[AnonymusRaccoon]

Ok, i just reproduce the same thing on a blank project but it's working. I don't know what i've made wrong on my project but it was working before youtube change. Sorry to bother you.

[darkdoctor]

Hi dear Tyrrrz.

Any idea when you can publish the fix for this issue , I really need channel titles because I assign each channel name to each song.

Thanks for your hard work.

[Tyrrrz]

Not sure yet, but hopefully early next week.

[lesleyxyz]

Not trying to be impatient, but will the fix still list all the possible video/audio/mixed streams? Or does the workaround already have this possibility? Or will I have to change my code completely?

Thanks :)

[vinaykapadia]

As far as I can tell, the workaround will still list all the possible streams. It's just metadata that may be missing.

[Tyrrrz]

Indeed the workaround already works for that and so will the fix. When the fix is released, you will, however, need to change some code because the streams are going to be extracted separately from metadata.

[Tyrrrz]

Still haven't found a good way to extract description and likes/dislikes. Parsing it off watch page involves really weird pattern matching and a lot of parsing.

[Tyrrrz]

Let's move rework discussion to #70

[Tyrrrz]

I just pushed a new pre-release package with the fix -- https://www.nuget.org/packages/YoutubeExplode/4.0.0-alpha. Please test it if you can, if everything goes well I'll merge it into master on 15th of December.

[ComputerMaster1st]

All seems well so far.

[Tyrrrz]

v4.0.0 released!