Spotify-downloader: Unmerged pytube PR causes KeyError: 'adaptiveFormats'

Created on 10 Aug 2020 · 13Comments · Source: spotDL/spotify-downloader

What is the purpose of your issue?

[x] Bug
[ ] Feature Request
[ ] Question
[ ] Other

Description

There's a pytube error that says keyError: 'adaptiveFormats', which seems to be fixed in this pull request

Log

spotdl -s "https://open.spotify.com/track/0Hxrvs9UxUhzLfgcUy4hNJ" -m -sf '{artist} - {track-name}' -ll DEBUG
```python DEBUG:spotdl.command_line.__main__:24:
spotdl 2.2.2

DEBUG:spotdl.authorize.services.spotify:71:
Setting master Spotify credentials.

DEBUG:spotdl.command_line.core:130:
Received arguments:
{'manual': True, 'no_metadata': False, 'no_encode': False, 'overwrite': 'skip', 'quality': 'best', 'input_ext': 'automatic', 'output_ext': 'mp3', 'write_to': None, 'trim_silence': False, 'search_format': '{artist} - {track-name}', 'dry_run': False, 'no_spaces': False, 'output_file': '{track-name} - {artist}.{output-ext}', 'skip_file': 'old.txt', 'write_successful_file': 'old.txt', 'spotify_client_id': '4fe3fecfe5334023a1472516cc99d805', 'spotify_client_secret': '0f02b7c483c04257984695007a4a8d5c', 'log_level': 'DEBUG', 'song': ['https://open.spotify.com/track/0Hxrvs9UxUhzLfgcUy4hNJ'], 'list': None, 'playlist': None, 'album': None, 'all_albums': None, 'username': None, 'write_m3u': False, 'processor': 'synchronous', 'remove_config': False}

DEBUG:spotdl.authorize.services.spotify:66:
Reading cached master Spotify credentials.

INFO:spotdl.command_line.core:140:
Downloading "https://open.spotify.com/track/0Hxrvs9UxUhzLfgcUy4hNJ"

DEBUG:spotdl.authorize.services.spotify:66:
Reading cached master Spotify credentials.

DEBUG:spotdl.metadata_search:283:
Extracting YouTube and Spotify metadata for input Spotify URI.

DEBUG:spotdl.metadata_search:346:
Extracting Spotify metadata for input Spotify URI.

DEBUG:spotdl.metadata.providers.spotify:77:
Fetching Spotify metadata for "https://open.spotify.com/track/0Hxrvs9UxUhzLfgcUy4hNJ".

DEBUG:spotdl.metadata.providers.youtube:232:
Fetching YouTube results for "Victor Ruiz - Brasilia" at "https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q=Victor%20Ruiz%20-%20Brasilia".

DEBUG:spotdl.metadata.providers.youtube:189:
YouTube returned malformed HTML. Attempting to parse possible JSON data.

Skip downloading this track
Victor Ruiz - Brasilia (Original Mix)
https://www.youtube.com/watch?v=bLy3wR7l30s [7:37]
Victor Ruiz AV Any Mello @ Só Track Boa. Curitiba
https://www.youtube.com/watch?v=27ZaE5_5eGU [3:04]
Victor Ruiz AV Any Mello @ 5uinto 300
https://www.youtube.com/watch?v=t6odFejmQjg [10:56]
Victor Ruiz - Brasilia - (vocal- RAD6) ZUPPO.
https://www.youtube.com/watch?v=Jd0Jgh5KyOw [7:34]
VICTOR RUIZ AV. ANY MELLO @ LOW CLUB BRASÍLIA
https://www.youtube.com/watch?v=sGGYRHr3Lrg [4:23]
Vintage Culture live at @ So Track Boa Festival - São Paulo 2019 FULL SHOW
https://www.youtube.com/watch?v=EPYsq5InQyQ [2:38:22]
AS MELHORES DOS ANOS 2000 SUMMER ELETROHITS - SET NOSTALGIA VINTAGE CULTURE, CAT DEALERS & KVSH
https://www.youtube.com/watch?v=2XjSGK59jLY [2:18:36]
AS MELHORES DE 2018 - SET VINTAGE CULTURE, CHEMICAL SURF, KVSH & DUBDOGZ #SóTrackBoa
https://www.youtube.com/watch?v=13HgCJL0IZQ [2:06:32]
D-Nox & Beckers @ 5uinto 368
https://www.youtube.com/watch?v=b2KvI3ESFNc [1:00:29]
1. Luccas Carlos - Sem Ninguém part. Gee Rocha (Ao Vivo)
  
  https://www.youtube.com/watch?v=sNvrzssZAOQ [3:58]

INFO:spotdl.util:81:
Enter a number:

1
DEBUG:spotdl.metadata.providers.youtube:419:
Fetching YouTube metadata for "https://www.youtube.com/watch?v=bLy3wR7l30s".

DEBUG:spotdl.metadata.providers.youtube:427:
YouTube returned an unexpected response for pytube.YouTube(https://www.youtube.com/watch?v=bLy3wR7l30s). Retries left: 4

DEBUG:spotdl.metadata.providers.youtube:419:
Fetching YouTube metadata for "https://www.youtube.com/watch?v=bLy3wR7l30s".

DEBUG:spotdl.metadata.providers.youtube:427:
YouTube returned an unexpected response for pytube.YouTube(https://www.youtube.com/watch?v=bLy3wR7l30s). Retries left: 3

DEBUG:spotdl.metadata.providers.youtube:419:
Fetching YouTube metadata for "https://www.youtube.com/watch?v=bLy3wR7l30s".

DEBUG:spotdl.metadata.providers.youtube:427:
YouTube returned an unexpected response for pytube.YouTube(https://www.youtube.com/watch?v=bLy3wR7l30s). Retries left: 2

DEBUG:spotdl.metadata.providers.youtube:419:
Fetching YouTube metadata for "https://www.youtube.com/watch?v=bLy3wR7l30s".

DEBUG:spotdl.metadata.providers.youtube:427:
YouTube returned an unexpected response for pytube.YouTube(https://www.youtube.com/watch?v=bLy3wR7l30s). Retries left: 1

DEBUG:spotdl.metadata.providers.youtube:419:
Fetching YouTube metadata for "https://www.youtube.com/watch?v=bLy3wR7l30s".

DEBUG:spotdl.metadata.providers.youtube:427:
YouTube returned an unexpected response for pytube.YouTube(https://www.youtube.com/watch?v=bLy3wR7l30s). Retries left: 0

DEBUG:spotdl.metadata.providers.youtube:419:
Fetching YouTube metadata for "https://www.youtube.com/watch?v=bLy3wR7l30s".

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/spotdl/metadata/providers/youtube.py", line 421, in from_url
content = pytube.YouTube(url)
File "/usr/local/lib/python3.8/site-packages/pytube/__main__.py", line 92, in __init__
self.descramble()
File "/usr/local/lib/python3.8/site-packages/spotdl/patch/youtube.py", line 164, in descramble
apply_descrambler(self.player_config_args, fmt)
File "/usr/local/lib/python3.8/site-packages/spotdl/patch/youtube.py", line 78, in apply_descrambler
json.loads(stream_data["player_response"])["streamingData"][
KeyError: 'adaptiveFormats'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/spotdl/metadata/providers/youtube.py", line 421, in from_url
content = pytube.YouTube(url)
File "/usr/local/lib/python3.8/site-packages/pytube/__main__.py", line 92, in __init__
self.descramble()
File "/usr/local/lib/python3.8/site-packages/spotdl/patch/youtube.py", line 164, in descramble
apply_descrambler(self.player_config_args, fmt)
File "/usr/local/lib/python3.8/site-packages/spotdl/patch/youtube.py", line 78, in apply_descrambler
json.loads(stream_data["player_response"])["streamingData"][
KeyError: 'adaptiveFormats'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.8/site-packages/spotdl/util.py", line 37, in run
self._return = self._target(
File "/usr/local/lib/python3.8/site-packages/spotdl/command_line/core.py", line 349, in threaded_metadata
metadata = search_metadata.on_youtube_and_spotify()
File "/usr/local/lib/python3.8/site-packages/spotdl/metadata_search.py", line 159, in on_youtube_and_spotify
metadata = caller(self.track)
File "/usr/local/lib/python3.8/site-packages/spotdl/metadata_search.py", line 287, in _on_youtube_and_spotify_for_type_spotify
youtube_metadata = self.providers["youtube"].from_url(youtube_video["url"])
File "/usr/local/lib/python3.8/site-packages/spotdl/metadata/providers/youtube.py", line 433, in from_url
return self.from_url(url, retries=retries)
File "/usr/local/lib/python3.8/site-packages/spotdl/metadata/providers/youtube.py", line 433, in from_url
return self.from_url(url, retries=retries)
File "/usr/local/lib/python3.8/site-packages/spotdl/metadata/providers/youtube.py", line 433, in from_url
return self.from_url(url, retries=retries)
[Previous line repeated 2 more times]
File "/usr/local/lib/python3.8/site-packages/spotdl/metadata/providers/youtube.py", line 421, in from_url
content = pytube.YouTube(url)
File "/usr/local/lib/python3.8/site-packages/pytube/__main__.py", line 92, in __init__
self.descramble()
File "/usr/local/lib/python3.8/site-packages/spotdl/patch/youtube.py", line 164, in descramble
apply_descrambler(self.player_config_args, fmt)
File "/usr/local/lib/python3.8/site-packages/spotdl/patch/youtube.py", line 78, in apply_descrambler
json.loads(stream_data["player_response"])["streamingData"][
KeyError: 'adaptiveFormats'
```

bug

Source

gordominossi

All 13 comments

Thanks for the report! Would you be interested in making a PR to fix this in spotdl?

ritiek on 10 Aug 2020

Yes. But I need some guidance. Could you point me to other PRs that fixed issues like this one?

gordominossi on 10 Aug 2020

Check out #798 for an example. You basically need to override the affected method in pytube with its correct counterpart. However, in our case we already have the new apply_descrambler method defined, so you could simply update this method as per https://github.com/nficano/pytube/pull/607.

ritiek on 10 Aug 2020

Thanks :)

gordominossi on 10 Aug 2020

So yeah... I got it fixed... Kinda
When I try to download from videos that don't have adaptiveFormats the query results in video only streams and after a slight modification I get

ERROR:spotdl.command_line.core:429:
No matching streams found for given input format: "automatic".

I logged the results from

content = pytube.YouTube(url)

and got

[<Stream: itag="18" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.42001E" acodec="mp4a.40.2" progressive="True" type="video">]

gordominossi on 10 Aug 2020

How do I fix the query?

gordominossi on 10 Aug 2020

Can you make a PR with whatever you have at the moment? Otherwise it's hard to know why this should happen.

ritiek on 16 Aug 2020

👍1

Any way to get audio from those videos instead of returning an error, though?

gordominossi on 14 Sep 2020

@gordominossi I guess pytube issues are irrigating out in v3.

It would help if you could replace pytube3 with pytubeX (a maintained fork).

If your up to the work, open an issue and a Draft PR, I'll look through your code during dev and help you keep it in line with our design principles.

MikhailZex on 7 Oct 2020

👍1

Hey, will you be working on PytubeX usage? I can get someone to work on that if your busy.

MikhailZex on 26 Oct 2020

I am sorry, I forgot about this issue. Have been very busy with work. I will take some other feature/improvement that has a lower priority. Don't count on me

gordominossi on 26 Oct 2020

No issues. This is actually low priority.

Checkout #899 #894 #682 #599, all low priority.

MikhailZex on 26 Oct 2020

👍1

Shifted to pytube

MikhailZex on 31 Oct 2020

Was this page helpful?

0 / 5 - 0 ratings