Spotify-downloader: HTTP Error 429: Too Many Requests

[SaeedSheikhi]

What is the purpose of your issue?

• [ ] Bug
• [ ] Feature Request
• [x] Question
• [ ] Other

Description

Rate limit will occure while fetching lyrics(shown in the log), while youtube-dl is working with same IP.
Is there any solution except changing IP address, e.g, make lyrics optional and skippable

Log

spotdl -s https://open.spotify.com/track/25AHHw85uL4TI6ZDKkmG15?si=p1nFwd32TFaPpmsUwuhXOg -f . --log-level=DEBUG
DEBUG: Input song is a Spotify URL
DEBUG: Fetching metadata for given track URL
DEBUG: Fetching lyrics
DEBUG: {'album': {'album_type': 'album',
               'artists': [{'external_urls': {'spotify': 'https://open.spotify.com/artist/2w9zwq3AktTeYYMuhMjju8'},
                            'href': 'https://api.spotify.com/v1/artists/2w9zwq3AktTeYYMuhMjju8',
                            'id': '2w9zwq3AktTeYYMuhMjju8',
                            'name': 'INNA',
                            'type': 'artist',
                            'uri': 'spotify:artist:2w9zwq3AktTeYYMuhMjju8'}],
               'external_urls': {'spotify': 'https://open.spotify.com/album/1zOyANSJozlJ21CPIwIXzq'},
               'href': 'https://api.spotify.com/v1/albums/1zOyANSJozlJ21CPIwIXzq',
               'id': '1zOyANSJozlJ21CPIwIXzq',
               'images': [{'height': 640,
                           'url': 'https://i.scdn.co/image/8b7de8621a8c8667a838cebdf0af7aad10279aff',
                           'width': 640},
                          {'height': 300,
                           'url': 'https://i.scdn.co/image/c662b63f4585410a33ab35aa7d1db8e1fae1a099',
                           'width': 300},
                          {'height': 64,
                           'url': 'https://i.scdn.co/image/8de9278fec0bf1b62452206bffd33a9b4da4a816',
                           'width': 64}],
               'name': 'Nirvana',
               'release_date': '2017-12-11',
               'release_date_precision': 'day',
               'total_tracks': 12,
               'type': 'album',
               'uri': 'spotify:album:1zOyANSJozlJ21CPIwIXzq'},
     'artists': [{'external_urls': {'spotify': 'https://open.spotify.com/artist/2w9zwq3AktTeYYMuhMjju8'},
                  'href': 'https://api.spotify.com/v1/artists/2w9zwq3AktTeYYMuhMjju8',
                  'id': '2w9zwq3AktTeYYMuhMjju8',
                  'name': 'INNA',
                  'type': 'artist',
                  'uri': 'spotify:artist:2w9zwq3AktTeYYMuhMjju8'}],
     'copyright': 'Global Records',
     'disc_number': 1,
     'duration': 194.886,
     'explicit': False,
     'external_ids': {'isrc': 'FRX201795585'},
     'external_urls': {'spotify': 'https://open.spotify.com/track/25AHHw85uL4TI6ZDKkmG15'},
     'genre': 'Dance Pop',
     'href': 'https://api.spotify.com/v1/tracks/25AHHw85uL4TI6ZDKkmG15',
     'id': '25AHHw85uL4TI6ZDKkmG15',
     'is_local': False,
     'lyrics': None,
     'name': 'Tropical',
     'popularity': 38,
     'preview_url': 'https://p.scdn.co/mp3-preview/b1b4996e3691a8bf076ae8cfb4a60633cd692f3b?cid=4fe3fecfe5334023a1472516cc99d805',
     'publisher': 'Global Records',
     'release_date': '2017-12-11',
     'spotify_metadata': True,
     'total_tracks': 12,
     'track_number': 4,
     'type': 'track',
     'uri': 'spotify:track:25AHHw85uL4TI6ZDKkmG15',
     'year': '2017'}
DEBUG: Opening URL: https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q=INNA%20-%20Tropical%20lyrics
Traceback (most recent call last):
  File "/usr/local/bin/spotdl", line 11, in <module>
    load_entry_point('spotdl', 'console_scripts', 'spotdl')()
  File "/opt/spotify-downloader/spotdl/spotdl.py", line 63, in main
    match_args()
  File "/opt/spotify-downloader/spotdl/spotdl.py", line 27, in match_args
    track_dl = downloader.Downloader(raw_song=track)
  File "/opt/spotify-downloader/spotdl/downloader.py", line 98, in __init__
    self.content, self.meta_tags = youtube_tools.match_video_and_metadata(raw_song)
  File "/opt/spotify-downloader/spotdl/youtube_tools.py", line 80, in match_video_and_metadata
    content = go_pafy(track, meta_tags)
  File "/opt/spotify-downloader/spotdl/youtube_tools.py", line 41, in go_pafy
    track_url = generate_youtube_url(raw_song, meta_tags)
  File "/opt/spotify-downloader/spotdl/youtube_tools.py", line 217, in generate_youtube_url
    url = url_fetch.scrape()
  File "/opt/spotify-downloader/spotdl/youtube_tools.py", line 311, in scrape
    item = self._fetch_response(search_url).read()
  File "/opt/spotify-downloader/spotdl/youtube_tools.py", line 397, in _fetch_response
    return urllib.request.urlopen(url)
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 504, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 696, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 429: Too Many Requests

[ritiek]

Rate limit will occure while fetching lyrics(shown in the log), while youtube-dl is working with same IP.
Is there any solution except changing IP address, e.g, make lyrics optional and skippable

The cause of the error is after it is done with checking for lyrics. The culprit seems to be YouTube as the tool errors out when the tool is searching for the track on YouTube. By the way, we already have a PR #524 for disabling lyrics although it's high time since I review it.

FWIW have you been downloading a lot of songs lately? Since:

HTTP Error 429: Too Many Requests

probably means that.

Anyway, you could try generating your own YouTube API key and set it in your config.yml (see wiki for more info). Hopefully, that should fix this issue.

[SaeedSheikhi]

thank you @ritiek for your fast response.
yes I think downloading more than 1k tracks per day was the reason of the problem, waiting some hours, and problem were gone. I faced this issue multiple time in current week and i knew that it's not an issue relevant to the Spotify-Downloader, exceeding quota or something similar or third-party service issue.
but anyway the repo is working perfectly. adding optional lyrics can make repo more stable and reliable.

Thanks again, cheers 🍻

[linusg]

downloading more than 1k tracks per day

[SaeedSheikhi]

@linusg and also with 15 Failure Attempts 😂 really why.

queue
      .create("download", {
        link: req.body.link,
        requestId: req.body.requestId,
        type: "track"
      })
      .priority("high")
      .attempts(15)
      .backoff({ type: "exponential" })
      .save(err => {
        if (!err) res.json(job);
      });

honestly i'm trying to clone Spotify db by daily crawling, and at the end ( if something bad doesn't happen :| ) make an open source music-api to resolve spotify links faster than youtube fetching, converting, e.t.c. :D you know. something to help community and tools to be more robust :D

[cyberboysumanjay]

It seems YouTube changed something which is causing this error. The only solution is to get a list of proxies and rotate them one by one.

Also, as @ritiek mentioned this has nothing to do with this library but this is forced by YouTube recently.
You can find more discussion about this here

I'm using spotdl as a library in my python project and I would like to use proxy programmatically. In case anyone can throw some light it would be helpful.

[Aeris1One]

Does using YouTube API by creating an API Key and setting it inside config.json solves this bug ?

[ritiek]

@Aeris1One It should. You could be the first to let us know!

[Aeris1One]

It didn't, just tried -_-
Surely because the IP is banned, so even using the API is prohibited...
But maybe we cannot be banned using the API, I was banned before setting API key.
Maybe we can download more than 1k/day using API without havine this error, but maybe not... I'll will not try personally, don't want to be banned another time if it doesn't work ^^

[ritiek]

Oh well. That's bad. It could be possible though. If your ISP allots a
dynamic IP address, rebooting your modem should workaround this.

[Aeris1One]

I use a VPS hosting so no way to change IP 'cause I've some domains pointed to my server and I don't want to have to modify settings, I'll wait some days, maybe YouTube will unban me...

[FransM]

I got the same issue. Not sure why this suddenly happened. I didn't donwload much. Youtube in browser still works.

pytube has a ticket for the same issue: https://github.com/nficano/pytube/issues/601

Didn't get the time yet to try it, will do tomorrow or so unless someone else picks this up

[MikhailZex]

No lyrics support in v3, we aren't fixing v2 issues. Please look at PR's - there is one adding lyrics support.

[MikhailZex]

I know it's late, just clearing out old issues.

[Aeris1One]

No lyrics support in v3, we aren't fixing v2 issues. Please look at PR's - there is one adding lyrics support.

This issue is not about lyrics, but about the "429 - Too Many Request" error which will surely happen in v3 if downloading too many tracks.

[MikhailZex]

Ok. Well in the new implementation, we aren't using geniuses API but re and urllib instead (PR yet to be merged) . You shouldn't have the problem now.

[quyleanh]

How can I set YouTube API key on config.yml file? @ritiek

[MikhailZex]

Ritiek is a bit busy, he might take a while to reply (just giving you the heads up)