Pytube: KeyError: 'cipher'

[twwwy]

from pytube import YouTube
yt = YouTube('https://www.youtube.com/watch?v=3AtDnEC4zak')

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/.conda/envs/Karaoke/lib/python3.7/site-packages/pytube/extract.py in apply_descrambler(stream_data, key)
    296                 }
--> 297                 for format_item in formats
    298             ]

~/.conda/envs/Karaoke/lib/python3.7/site-packages/pytube/extract.py in <listcomp>(.0)
    296                 }
--> 297                 for format_item in formats
    298             ]

KeyError: 'url'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-3-bc6543d387ce> in <module>
----> 1 yt = YouTube('https://www.youtube.com/watch?v=3AtDnEC4zak')

~/.conda/envs/Karaoke/lib/python3.7/site-packages/pytube/__main__.py in __init__(self, url, defer_prefetch_init, on_progress_callback, on_complete_callback, proxies)
     90         if not defer_prefetch_init:
     91             self.prefetch()
---> 92             self.descramble()
     93 
     94     def descramble(self) -> None:

~/.conda/envs/Karaoke/lib/python3.7/site-packages/pytube/__main__.py in descramble(self)
    130             if not self.age_restricted and fmt in self.vid_info:
    131                 apply_descrambler(self.vid_info, fmt)
--> 132             apply_descrambler(self.player_config_args, fmt)
    133 
    134             if not self.js:

~/.conda/envs/Karaoke/lib/python3.7/site-packages/pytube/extract.py in apply_descrambler(stream_data, key)
    299         except KeyError:
    300             cipher_url = [
--> 301                 parse_qs(formats[i]["cipher"]) for i, data in enumerate(formats)
    302             ]
    303             stream_data[key] = [

~/.conda/envs/Karaoke/lib/python3.7/site-packages/pytube/extract.py in <listcomp>(.0)
    299         except KeyError:
    300             cipher_url = [
--> 301                 parse_qs(formats[i]["cipher"]) for i, data in enumerate(formats)
    302             ]
    303             stream_data[key] = [

KeyError: 'cipher'

[MatheoDumont]

Same

[chris-maier]

same here.

[mickael-btc]

same :(

[gorkemt]

extract.py
line 295

stream_data[key] = [
                {
                    "url": format_item["url"]), <----- No more url  replaced with signatureCipher
                    "type": format_item["mimeType"],
                    "quality": format_item["quality"],
                    "itag": format_item["itag"],
                    "bitrate": format_item.get("bitrate"),
                    "is_otf": (format_item.get("type") == otf_type),
                }
                for format_item in formats
            ]

[MrspiLLnyK]

same

[ritiek]

Does #643 fix this for you guys?

[kelvin84hk]

Does #643 fix this for you guys?

Yes, I have the same error and fixed it as suggested:

except KeyError:
cipher_url = [
parse_qs(formats[i]["signatureCipher"]) for i, data in enumerat$
]

[famabland]

Does #643 fix this for you guys?
yes, this fix worked.

[katsar0v]

When can we expect a release with the fix?

[shampeter]

I fixed the code with a little more flexibility. Hopefully whatever the keyword would be in the future can be just an adjustment in some settings.

cipher_url = [
### parse_qs(formats[i]["cipher"]) for i, data in enumerate(formats)
(lambda i,data: [ parse_qs(data[k]) for k in ("cipher","signatureCipher") if k in data ])(i, data) for i, data in enumerate(formats)
]

Above list comprehension would result in list within list. Below expression extract the sublist to crate a one-dimension list.

cipher_url = [item for sublist in cipher_url for item in sublist]

[seanvelir]

643 worked for me

[pranshukharkwal]

A bit naive query here, can I do something on my code (which is using pytube) to fix the problem, instead of making changes to the library's files?
@ritiek @seanvelir @shampeter @famabland @kelvin84hk

[ritiek]

@pranshukharkwal Yes, it's possible. Here's how you would do it:

# The below imports are required by the patch
import json
from urllib.parse import parse_qs, unquote

# This function is based off on the changes made in
# https://github.com/nficano/pytube/pull/643

def apply_descrambler(stream_data, key):
    """Apply various in-place transforms to YouTube's media stream data.
    Creates a ``list`` of dictionaries by string splitting on commas, then
    taking each list item, parsing it as a query string, converting it to a
    ``dict`` and unquoting the value.
    :param dict stream_data:
        Dictionary containing query string encoded values.
    :param str key:
        Name of the key in dictionary.
    **Example**:
    >>> d = {'foo': 'bar=1&var=test,em=5&t=url%20encoded'}
    >>> apply_descrambler(d, 'foo')
    >>> print(d)
    {'foo': [{'bar': '1', 'var': 'test'}, {'em': '5', 't': 'url encoded'}]}
    """
    otf_type = "FORMAT_STREAM_TYPE_OTF"

    if key == "url_encoded_fmt_stream_map" and not stream_data.get(
        "url_encoded_fmt_stream_map"
    ):
        formats = json.loads(stream_data["player_response"])["streamingData"]["formats"]
        formats.extend(
            json.loads(stream_data["player_response"])["streamingData"][
                "adaptiveFormats"
            ]
        )
        try:
            stream_data[key] = [
                {
                    "url": format_item["url"],
                    "type": format_item["mimeType"],
                    "quality": format_item["quality"],
                    "itag": format_item["itag"],
                    "bitrate": format_item.get("bitrate"),
                    "is_otf": (format_item.get("type") == otf_type),
                }
                for format_item in formats
            ]
        except KeyError:
            cipher_url = []
            for data in formats:
                cipher = data.get("cipher") or data["signatureCipher"]
                cipher_url.append(parse_qs(cipher))
            stream_data[key] = [
                {
                    "url": cipher_url[i]["url"][0],
                    "s": cipher_url[i]["s"][0],
                    "type": format_item["mimeType"],
                    "quality": format_item["quality"],
                    "itag": format_item["itag"],
                    "bitrate": format_item.get("bitrate"),
                    "is_otf": (format_item.get("type") == otf_type),
                }
                for i, format_item in enumerate(formats)
            ]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qsl(i)}
            for i in stream_data[key].split(",")
        ]


import pytube
pytube.__main__.apply_descrambler = apply_descrambler
video = pytube.YouTube("https://www.youtube.com/watch?v=w2FbOH9uexQ")

Make sure to retain pytube's license along with this code if you intend to use it in 3rd party scripts.

[kbuilds]

While waiting for the maintainer to address these PRs, feel free to use our public version with #643 merged in:

https://gitlab.com/obuilds/public/pytube

I use GitLab because I am a savage

You can install the tagged version like so:

pip install git+https://gitlab.com/obuilds/public/pytube@ob-v1

or add the following to your requirements.txt

git+https://gitlab.com/obuilds/public/pytube@ob-v1

Hopefully our boy sees these MRs soon and gets a new release out.

[ritiek]

@kbuilds Very nice but while I'm not sure what the situation is now but it used to be necessary to have git installed to use git+https://... URLs in dependencies, so while most Linux users have git installed but non-power Windows users are going to have trouble with this.

[kbuilds]

@ritiek You would need to have git installed in order for this to work. I just wasn't motivated to create a new pypi entry.

[pranshukharkwal]

@ritiek GitBash works for windows users.
Anyways @kbuilds , I have installed the library from gitlab,, how do I import it in my code now? Like earlier it was :
from pytube import YouTube
now what? This is giving the same error as before

[PiotrWieczorek98]

@kbuilds Thank you, works great. Used pip to install

[kbuilds]

No problem, @PiotrWieczorek98

@pranshukharkwal That is the correct import. You didn't mention what kind of error you were getting.

[pranshukharkwal]

@kbuilds Since I am not using a virtual environment, and already have the earlier version of pytube installed already, even after installing from your gitlab repository, I am getting that same of "Cipher" keyerror, as if it is using the original pytube library, and not the patched one.

[janLuisAntoc]

@kbuilds Good day! I already used your own version of pytube but I am receiving another error. The last lines of the error I received is shown below:

File "C:\Users\Admin\PycharmProjects\Youtube-Downloader\venv\lib\site-packages\pytube\extract.py", line 319, in
for i, format_item in enumerate(formats)
KeyError: 'url'

I'll patiently wait for your response. Thank you very much.

[kbuilds]

@pranshukharkwal @janLuisAntoc Unfortunately, it's going to be really difficult to know why you are running into issues unless we know a lot more about your setups.

Also, I would highly recommend using a virtual environment to help eliminate a lot of these kinds of issues.

[janLuisAntoc]

Hello @kbuilds. Thank you for your swift response. I'm currently doing a project regarding YouTube downloader in PyCharm IDE.

[Suji04]

I'm trying this on google colab. Same issue.

[yogesh-12]

trying this on google colab.
Any suggestions to fix this issue?

[pranshukharkwal]

@Suji04 @yogesh-12
Install the package from the GitLab repository shared by @kbuilds in Google Colab. Check the code below

!pip install git+https://gitlab.com/obuilds/public/pytube@ob-v1
from pytube import YouTube
yt = YouTube('https://www.youtube.com/watch?v=-tJYN-eG1zk')
`

[yogesh-12]

@pranshukharkwal
Thanks for the prompt response man!!!
But I get the following error when I run the code given above

When I try to use pytube3 to fix it, i again get back the same issue

[yogesh-12]

@pranshukharkwal @Suji04
I upgraded and it worked!!!

The below code is working :

!pip install git+https://gitlab.com/obuilds/public/pytube@ob-v1 --upgrade
from pytube import YouTube
yt = YouTube('https://www.youtube.com/watch?v=-tJYN-eG1zk')

[Suji04]

thanks @pranshukharkwal
It worked!

[kbuilds]

Looks like someone is a fan of Queen

[janLuisAntoc]

@Suji04 @yogesh-12
Install the package from the GitLab repository shared by @kbuilds in Google Colab. Check the code below

!pip install git+https://gitlab.com/obuilds/public/pytube@ob-v1
from pytube import YouTube
yt = YouTube('https://www.youtube.com/watch?v=-tJYN-eG1zk')
`

Good day! How could I uninstall this if I already installed it? I would like to test another solution for this problem so I would like to ask how could I uninstall this package. Thank you!

[janLuisAntoc]

@pranshukharkwal @Suji04
I upgraded and it worked!!!

The below code is working :

!pip install git+https://gitlab.com/obuilds/public/pytube@ob-v1 --upgrade
from pytube import YouTube
yt = YouTube('https://www.youtube.com/watch?v=-tJYN-eG1zk')

Thank you so much! This worked for a video that I was not able to download last week.

[johnzan0743]

Does #643 fix this for you guys?

Yes, I have the same error and fixed it as suggested:

except KeyError:
cipher_url = [
parse_qs(formats[i]["signatureCipher"]) for i, data in enumerat$
]

Thank you so much! This one really works for me now !

[irahorecka]

same

[MukulKirtiVerma]

Just go to the pytube\extract.py (in pytube library)file: the path of file will be : "C:\ProgramData\Anaconda3\lib\site-packages\pytube\extract.py", (in windows)

Open extract,py file and search for line: parse_qs(formats[i]["cipher"]) for i, data in enumerate(formats)

now replace 'cipher' with 'signatureCipher'.

save it.

now run your code again

[RONNCC]

@MukulKirtiVerma @janLuisAntoc would love a PR to fix this issue :)

[Remus432]

Hey guys, I'm having an issue over here trying to use a variable for the link

import pytube

new_clip = "https://www.youtube.com/watch?v=VVuC5B7UyQY"

yt = pytube.YouTube(new_clip)

streamyt = yt.streams.all()

print(streamyt)

I get this error:

Do you guys know why this is happening?

[avigupta10]

@Remus432 same please help

[Crashkurs]

Same here

[avigupta10]

please help i want to submit this project asap in my college.
but this error smh!!!

[MukulKirtiVerma]

Update file pytube/cipher.py line .no 31
By removing _

On Tue, 28 Jul, 2020, 6:35 PM Avihwr, [e-mail] wrote:

please help i want to submit this project asap in my college.
but this error smh!!!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nficano/pytube/issues/641#issuecomment-665027219, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AI3FU3W7CLTHXL6UVN2BBFLR53EIRANCNFSM4NM2BXIQ
.

[Aaditay]

@MukulKirtiVerma Well that doesn't work after removing the _ from line 31;

This leads to further error
File "C:\Users\aadit\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pytube\extract.py", line 301, in <listcomp> parse_qs(formats[i]["cipher"]) for i, data in enumerate(formats) KeyError: 'cipher'

[konnovdev]

@Remus432 see this https://github.com/nficano/pytube/issues/695

[irahorecka]

I solved the 'cipher' error by making a file, say pytube_patch.py, and binding the apply_descrambler function to the pytube library in a .py file using pytube, for example, main.py.

pytube_patch.py

# The below imports are required by the patch
import json
from urllib.parse import parse_qs, unquote

# This function is based off on the changes made in
# https://github.com/nficano/pytube/pull/643


def apply_descrambler(stream_data, key):
    """Apply various in-place transforms to YouTube's media stream data.
    Creates a ``list`` of dictionaries by string splitting on commas, then
    taking each list item, parsing it as a query string, converting it to a
    ``dict`` and unquoting the value.
    :param dict stream_data:
        Dictionary containing query string encoded values.
    :param str key:
        Name of the key in dictionary.
    **Example**:
    >>> d = {'foo': 'bar=1&var=test,em=5&t=url%20encoded'}
    >>> apply_descrambler(d, 'foo')
    >>> print(d)
    {'foo': [{'bar': '1', 'var': 'test'}, {'em': '5', 't': 'url encoded'}]}
    """
    if key == "url_encoded_fmt_stream_map" and not stream_data.get(
        "url_encoded_fmt_stream_map"
    ):
        formats = json.loads(stream_data["player_response"])["streamingData"]["formats"]
        formats.extend(
            json.loads(stream_data["player_response"])["streamingData"][
                "adaptiveFormats"
            ]
        )
        otf_type = "FORMAT_STREAM_TYPE_OTF"

        try:
            stream_data[key] = [
                {
                    "url": format_item["url"],
                    "type": format_item["mimeType"],
                    "quality": format_item["quality"],
                    "itag": format_item["itag"],
                    "bitrate": format_item.get("bitrate"),
                    "is_otf": (format_item.get("type") == otf_type),
                }
                for format_item in formats
            ]
        except KeyError:
            cipher_url = [
                parse_qs(formats[i]["signatureCipher"])
                for i, data in enumerate(formats)
            ]
            stream_data[key] = [
                {
                    "url": cipher_url[i]["url"][0],
                    "s": cipher_url[i]["s"][0],
                    "type": format_item["mimeType"],
                    "quality": format_item["quality"],
                    "itag": format_item["itag"],
                    "bitrate": format_item.get("bitrate"),
                    "is_otf": (format_item.get("type") == otf_type),
                }
                for i, format_item in enumerate(formats)
            ]
    else:
        stream_data[key] = [
            {k: unquote(v) for k, v in parse_qs(i)} for i in stream_data[key].split(",")
        ]

main.py (located in same directory as pytube_patch.py)

from .pytube_patch import apply_descrambler

# Set apply_descrambler method in pytube module to pytube_patch.apply_descrambler
pytube.__main__.apply_descrambler = apply_descrambler

[MukulKirtiVerma]

now see my first comment .
their i show how to solve this error

On Tue, Jul 28, 2020 at 11:11 PM Aaditay [e-mail] wrote:

@MukulKirtiVerma https://github.com/MukulKirtiVerma Well that doesn't
work after removing the _ from line 31;

This leads to further error
File
"C:\Users\aadit\AppData\Local\Programs\Python\Python38-32\lib\site-packagespytube\extract.py",
line 301, in parse_qs(formats[i]["cipher"]) for i, data in
enumerate(formats) KeyError: 'cipher'

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nficano/pytube/issues/641#issuecomment-665179211, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AI3FU3X3LLQK5C3I6OJARG3R54EVLANCNFSM4NM2BXIQ
.

[avigupta10]

Guys i figured it out:

For ' Too Many values to unpack error:

https://github.com/nficano/pytube/pull/701/commits/773866382c3412e01f97f242e753cf32f52aaefa#diff-e5b8b42cf1afd9ee025a9e6d48697ca0

Go to this link and Replace all the data inside your cipher file with the given inside the link.It'll work.

[GrandmaStitch]

Does #643 fix this for you guys?

Yes, #643 helps me to fix this problem.

[anonymous770]

How can i fix it

[irahorecka]

Why don't you try the maintained fork of pytube, pytubex

[Joedmin580]

Same for me

[tfdahlin]

In order to fix this issue, please install the latest version of pytube from this repository. To do this:

1. Uninstall any other versions of pytube that you have installed with python -m pip uninstall pytube pytube3 pytubex
2. Install from this repo with python -m pip install git+https://github.com/nficano/pytube.

[RONNCC]

Believe this is fixed in most recent version of Pytube @irahorecka @Joedmin580 @anonymous770

Let me know if this still doesnt work -- thanks

[Joedmin580]

@RONNCC Everything's working thank you so much!

[RONNCC]

@Joedmin580 it's due to work of @tfdahlin 😄