Spotify-downloader: MetadataSearch.on_youtube_and_spotify fails with NoYouTubeVideoFoundError

[billsioros]

The issue occurs on the spotdl.metadata.providers.youtube module on the YouTubeSearch.search method

_Note: I have left out parts of the code for brievity. Please let me know if you need any further clarifications_

search_url = self.generate_search_url(query)
logger.debug('Fetching YouTube results for "{}" at "{}".'.format(query, search_url))
html = self._fetch_response_html(search_url)
videos = self._fetch_search_results(html, limit=limit) # An empty list is being returned

The search_url parameter, supplied to the MetadataSearch._fetch_response_html method, has a value of https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q=SAINt%20JHN%20-%2094%20Bentley%20lyrics, which is a valid URL.

I can track the error back to the YouTubeSearch._fetch_search_results method

videos_html = html.find_all(
    "div", {"class": "yt-lockup-dismissable yt-uix-tile"}
) # This returns no results, but the response from YouTube is indeed in HTML
if videos_html:
    return self._fetch_search_results_from_html(videos_html, limit=limit)

The exact call stack is the following:

Click to expand !

_fetch_search_results (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\spotdl\metadata\providers\youtube.py:189)
search (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\spotdl\metadata\providers\youtube.py:234)
search (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\spotdl\metadata\providers\youtube.py:241)
search (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\spotdl\metadata\providers\youtube.py:443)
_best_on_youtube_search_for_type_query (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\spotdl\metadata_search.py:254)
_on_youtube_and_spotify_for_type_spotify (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\spotdl\metadata_search.py:286)
on_youtube_and_spotify (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\spotdl\metadata_search.py:159)
download_track (\mnt\c\Users\bills\Desktop\Dotify\backend\api\provider\spotify.py:150)
track (\mnt\c\Users\bills\Desktop\Dotify\backend\api\views\track.py:21)
dispatch_request (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\flask\app.py:1936)
full_dispatch_request (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\flask\app.py:1950)
wsgi_app (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\flask\app.py:2447)
__call__ (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\flask\app.py:2464)
__call__ (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\flask\cli.py:337)
execute (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\werkzeug\serving.py:312)
run_wsgi (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\werkzeug\serving.py:323)
handle_one_request (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\werkzeug\serving.py:379)
handle (\usr\lib\python3.8\http\server.py:427)
handle (\mnt\c\Users\bills\Desktop\Dotify\backend\.env\lib\python3.8\site-packages\werkzeug\serving.py:345)
__init__ (\usr\lib\python3.8\socketserver.py:720)

As far as I understand, the resulting HTML seems to be faulty, as it does not contain any items with a class of "yt-lockup-dismissable yt-uix-tile".

I have simulated the call outside of spotdl i.e. i used urllib.request and BeautifulSoup in order to save the resulting HTML as a .html file and I have confirmed that there indeed are no instances of elements with the aforementioned classes.

P.S. I have attempted to download a song both via python code and via the command line script and both result in the same error.

Thanks in advance for any input !

Cheers

[MikhailZex]

I believe your using v2. Support has been dropped. And the devs who worked on v2 have left so it's hard to provide you with support. You can try pinging @ritiek . Preferably upgrade to v3.

[billsioros]

Hello mate, thank you for your prompt response. I even pip install spotdl again. Maybe it didn't install the new version ? Should I maybe git clone the repo and pip install .

P.S. I had implemented a workflow of mine and to be sincere the complete support drop messed me up big time but it's understandable. I'm currently trying to migrate my workflow to spotdl v3 and I'm facing such issues. The issue occurs even when I'm using the command line utility as I mentioned before, so I'm kinda lost

Thanks again ! Cheers !

[MikhailZex]

run pip-autoremove spotdl first, then pip install spotdl.

The code to get a youtube link:

from spotdl.search.spotifyClient import initialize
from spotdl.search.utils import search_for_song
from spotdl.search.SongObj import songObj

# everything will throw an error without this
initialize (
     clientID='xyz',
     clientSecret='abc'
)

# method 1
q = search_for_song("song name goes here")
# method 2
w = songObj.from_url("spotify url/uri goes here")

# songObj objects contain all the info about a song you would like to ever know
link = q.get_youtube_link()
link = w.get_youtube_link()

# JSON formatted details of song, save this to file to skip the whole internet lookup the next
# time you need the details of this song
dataDump = q.get_data_dump()

# do this to parse a data dump
e = songObj.from_dump(dataDump)

# there is a lot more to a songObj, lookup the source code. If I'm correct that's all you'll
# need from the whole package.

[MikhailZex]

v3 should be easier to use than v2 (as a library). the spotdl.download sub-packages is buggy. For reliability sake, I suggest that you write up a custom downloader. If you need any help, drop a comment here.

[billsioros]

Thank you very much. I'll give it a try and let you know.

Cheers !

[billsioros]

Hello yet again :) I think I have figured out the general approach that I need to follow. Thanks for the much needed help !

[billsioros]

run pip-autoremove spotdl first, then pip install spotdl.

The code to get a youtube link:

from spotdl.search.spotifyClient import initialize
from spotdl.search.utils import search_for_song
from spotdl.search.SongObj import songObj

# everything will throw an error without this
initialize (
     clientID='xyz',
     clientSecret='abc'
)

# method 1
q = search_for_song("song name goes here")
# method 2
w = songObj.from_url("spotify url/uri goes here")

# songObj objects contain all the info about a song you would like to ever know
link = q.get_youtube_link()
link = w.get_youtube_link()

# JSON formatted details of song, save this to file to skip the whole internet lookup the next
# time you need the details of this song
dataDump = q.get_data_dump()

# do this to parse a data dump
e = songObj.from_dump(dataDump)

# there is a lot more to a songObj, lookup the source code. If I'm correct that's all you'll
# need from the whole package.

For anyone else wondering, this helped a lot !

[MikhailZex]

Glad to help.