Udemy-dl: Exception: 504 Gateway Time-out

[romrayster]

Describe the bug
Cannot download course, get the following error:

python ./udemy-dl.py https://www.udemy.com/course/nodejs-the-complete-guide/ -k cookies3.txt 
              __                               ____  
   __  ______/ /__  ____ ___  __  __      ____/ / /  
  / / / / __  / _ \/ __ `__ \/ / / /_____/ __  / /   
 / /_/ / /_/ /  __/ / / / / / /_/ /_____/ /_/ / /    
 \__,_/\__,_/\___/_/ /_/ /_/\__, /      \__,_/_/     
                           /____/
                                 Version : 1.0
                                 Author  : Nasir Khan (r0ot h3x49)
                                 Github  : https://github.com/r0oth3x49

[i] : Trying to login using session cookie
[i] : Logged in successfully.
Traceback (most recent call last):on ..
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/extract.py", line 339, in _extract_course_json
    resp = self._extract_large_course_content(url=url)
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/extract.py", line 311, in _extract_large_course_content
    data = self._session._get(url).json()
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/session.py", line 47, in _get
    raise Exception(f"{session.status_code} {session.reason}")
Exception: 504 Gateway Time-out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./udemy-dl.py", line 611, in <module>
    main()
  File "./udemy-dl.py", line 581, in main
    udemy_obj.course_download(
  File "./udemy-dl.py", line 283, in course_download
    course = udemy.course(
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/udemy.py", line 40, in course
    return Udemy(url, username, password, cookies, basic, skip_hls_stream, callback)
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/internal.py", line 65, in __init__
    super(InternUdemyCourse, self).__init__(*args, **kwargs)
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/shared.py", line 300, in __init__
    self._fetch_course()
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/internal.py", line 78, in _fetch_course
    self._info = self._real_extract(self._url, skip_hls_stream=self._skip_hls_stream)
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/extract.py", line 608, in _real_extract
    course_json = self._extract_course_json(url, course_id, portal_name)
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/extract.py", line 347, in _extract_course_json
    resp = self._extract_large_course_content(url=url)
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/extract.py", line 320, in _extract_large_course_content
    resp = self._session._get(_next).json()
  File "/home/roman/Cloud/DriveTrust Drive/Udemy/udemy-dl/udemy/session.py", line 47, in _get
    raise Exception(f"{session.status_code} {session.reason}")
Exception: 504 Gateway Time-out

To Reproduce

Steps to reproduce the behavior:

1. udemy course url: [e.g https://www.udemy.com/course/nodejs-the-complete-guide/] '...'
2. Run the following command to reproduce the error: python ./udemy-dl.py https://www.udemy.com/course/nodejs-the-complete-guide/ -k cookies3.txt
3. See error

Expected behavior
Course should download

Python Environment (please complete the following information):

• Python 3.8.3
• Ubuntu 18.04

[jcrawford]

I am also seeing this issue as well as the one below when the gateway does not timeout

Traceback (most recent call last):on ..
  File "udemy-dl.py", line 611, in <module>
    main()
  File "udemy-dl.py", line 581, in main
    udemy_obj.course_download(
  File "udemy-dl.py", line 283, in course_download
    course = udemy.course(
  File "/Users/jcrawford/udemy-dl/udemy/udemy.py", line 40, in course
    return Udemy(url, username, password, cookies, basic, skip_hls_stream, callback)
  File "/Users/jcrawford/udemy-dl/udemy/internal.py", line 65, in __init__
    super(InternUdemyCourse, self).__init__(*args, **kwargs)
  File "/Users/jcrawford/udemy-dl/udemy/shared.py", line 300, in __init__
    self._fetch_course()
  File "/Users/jcrawford/udemy-dl/udemy/internal.py", line 78, in _fetch_course
    self._info = self._real_extract(self._url, skip_hls_stream=self._skip_hls_stream)
  File "/Users/jcrawford/udemy-dl/udemy/extract.py", line 608, in _real_extract
    course_json = self._extract_course_json(url, course_id, portal_name)
  File "/Users/jcrawford/udemy-dl/udemy/extract.py", line 347, in _extract_course_json
    resp = self._extract_large_course_content(url=url)
  File "/Users/jcrawford/udemy-dl/udemy/extract.py", line 311, in _extract_large_course_content
    data = self._session._get(url).json()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[PaulNegoescu]

I can also back this up on a completely different course:

              __                               ____
   __  ______/ /__  ____ ___  __  __      ____/ / /
  / / / / __  / _ \/ __ `__ \/ / / /_____/ __  / /
 / /_/ / /_/ /  __/ / / / / / /_/ /_____/ /_/ / /
 \__,_/\__,_/\___/_/ /_/ /_/\__, /      \__,_/_/
                           /____/
                                 Version : 1.0
                                 Author  : Nasir Khan (r0ot h3x49)
                                 Github  : https://github.com/r0oth3x49

[i] : Trying to login using session cookie
[-] : Udemy Says: 403 Forbidden session cookie seems to be expired...
[i] : Updating session cookie..
[i] : Logged in successfully.
Traceback (most recent call last):on ..
  File "udemy-dl.py", line 611, in <module>
    main()
  File "udemy-dl.py", line 581, in main
    udemy_obj.course_download(
  File "udemy-dl.py", line 283, in course_download
    course = udemy.course(
  File "C:\work\udemy-dl\udemy\udemy.py", line 40, in course
    return Udemy(url, username, password, cookies, basic, skip_hls_stream, callback)
  File "C:\work\udemy-dl\udemy\internal.py", line 65, in __init__
    super(InternUdemyCourse, self).__init__(*args, **kwargs)
  File "C:\work\udemy-dl\udemy\shared.py", line 300, in __init__
    self._fetch_course()
  File "C:\work\udemy-dl\udemy\internal.py", line 78, in _fetch_course
    self._info = self._real_extract(self._url, skip_hls_stream=self._skip_hls_stream)
  File "C:\work\udemy-dl\udemy\extract.py", line 608, in _real_extract
    course_json = self._extract_course_json(url, course_id, portal_name)
  File "C:\work\udemy-dl\udemy\extract.py", line 347, in _extract_course_json
    resp = self._extract_large_course_content(url=url)
  File "C:\work\udemy-dl\udemy\extract.py", line 311, in _extract_large_course_content
    data = self._session._get(url).json()
  File "C:\Python38\lib\site-packages\requests\models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Also the cookie is most definitely not expired, still, I logged in with email and password and received the above error. It worked previously but I do think it's a Udemy change because I tried with a previous version of udemy-dl and it does not fix the issue.

[r0oth3x49]

@PaulNegoescu if it says 403 it is getting blocked by udemy server while using your cookie which is why shows that msg to you. did you provided user/pass as well? because i didn't added any functionality as of yet to update session cookie by asking for a new token on the go.

[PaulNegoescu]

@PaulNegoescu if it says 403 it is getting blocked by udemy server while using your cookie which is why shows that msg to you. did you provided user/pass as well? because i didn't added any functionality as of yet to update session cookie by asking for a new token on the go.

I did provide user and password and I'm getting the message logged in successfully as you can see in the dump above, afterwards it just seems to time out. Thing is, I checked the cookie a lot of times and it is valid and currently set in my browser.

[dragonhelios]

getting similar issue than romrayster and jcrawford on version 1.0. Worth noting that the error does not occur on 0.5 for me.

          __                               ____

__ ______/ /__ ____ ___ __ __ ____/ / /
/ / / / __ / _ \/ __ `__ \/ / / /_____/ __ / /
/ /_/ / /_/ / __/ / / / / / /_/ /_____/ /_/ / /
__,_/__,_/___/_/ /_/ /_/__, / __,_/_/
/____/
Version : 1.0
Author : Nasir Khan (r0ot h3x49)
Github : https://github.com/r0oth3x49

[i] : Trying to login using session cookie
[i] : Logged in successfully.
Traceback (most recent call last):on ..
File "./udemy-dl.py", line 611, in
main()
File "./udemy-dl.py", line 598, in main
udemy_obj.course_listdown(
File "./udemy-dl.py", line 148, in course_listdown
course = udemy.course(
File "/home/nlynzaad/git/udemy-dl/udemy/udemy.py", line 40, in course
return Udemy(url, username, password, cookies, basic, skip_hls_stream, callback)
File "/home/nlynzaad/git/udemy-dl/udemy/internal.py", line 65, in __init__
super(InternUdemyCourse, self).__init__(args, *kwargs)
File "/home/nlynzaad/git/udemy-dl/udemy/shared.py", line 300, in __init__
self._fetch_course()
File "/home/nlynzaad/git/udemy-dl/udemy/internal.py", line 78, in _fetch_course
self._info = self._real_extract(self._url, skip_hls_stream=self._skip_hls_stream)
File "/home/nlynzaad/git/udemy-dl/udemy/extract.py", line 608, in _real_extract
course_json = self._extract_course_json(url, course_id, portal_name)
File "/home/nlynzaad/git/udemy-dl/udemy/extract.py", line 347, in _extract_course_json
resp = self._extract_large_course_content(url=url)
File "/home/nlynzaad/git/udemy-dl/udemy/extract.py", line 311, in _extract_large_course_content
data = self._session._get(url).json()
File "/usr/lib/python3.8/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[PaulNegoescu]

Just throwing in a possible cause. It might be that cloudflare bot detection has something to do with it. It seems to catch more scripts lately than it used to.

[r0oth3x49]

Just throwing in a possible cause. It might be that cloudflare bot detection has something to do with it. It seems to catch more scripts lately than it used to.

@PaulNegoescu i don't think so cloudflare is involved in API level, it only restricts from bots to login using user/pass but cookies should work fine in all cases. The error is more related to broken json contents, while 504 timeout is because of big course i guess which i don't have any in my account and requested others to either share the cookie or creds on my email so that i can reproduce and fix.

[dragonhelios]

@r0oth3x49 let me know if you still require creds to test

[r0oth3x49]

@dragonhelios yes i still need an account to reproduce the issue or a cookie file with course link which throws the above error, both 504 Timeout and json error.
email: [email protected]

[dragonhelios]

sent

[r0oth3x49]

sent

pushing a fix now.

[r0oth3x49]

You all should use --skip-hls when downloading large course contents that way udemy-dl will avoid requesting hls based streams and the fetching will be faster otherwise it will take time.

[iamgovindthakur]

You all should use --skip-hls when downloading large course contents that way udemy-dl will avoid requesting hls based streams and the fetching will be faster otherwise it will take time.

will it work.? or I'm ready to provide my cookie access token details

[r0oth3x49]

You all should use --skip-hls when downloading large course contents that way udemy-dl will avoid requesting hls based streams and the fetching will be faster otherwise it will take time.

will it work.? or I'm ready to provide my cookie access token details

it should work now try testing and let me know if you still face the issue, --skip-hls is to done things faster.

[xsirvent]

will it work.? or I'm ready to provide my cookie access token details

New code is working for me.
--skip-hls is really faster!

Thanks

[iamgovindthakur]

You all should use --skip-hls when downloading large course contents that way udemy-dl will avoid requesting hls based streams and the fetching will be faster otherwise it will take time.

will it work.? or I'm ready to provide my cookie access token details

it should work now try testing and let me know if you still face the issue, --skip-hls is to done things faster.

--skip-hls throwing error bro :(

will i have to clone the latest repo

[xsirvent]

Use last version, which is 1.0 and not 0.5!

[r0oth3x49]

will it work.? or I'm ready to provide my cookie access token details

New code is working for me.
--skip-hls is really faster!

Thanks

remember --skip-hls will skip streams such as 1080p.
usually most of the courses provide 1080p in HLS based stream and rest till 720p are normal mp4 videos.
so if you want to download the video in 1080p (if exists) then you don't need to use that switch. i will later on add a warning msg if -q 1080 and --skip-hls both arguments are used togather.

you should either use -q 1080 or --skip-hls which by default downloads the highest availble resolution while in most of the courses normal highest available mp4 videos are 720p.

[iamgovindthakur]

Use last version, which is 1.0 and not 0.5!

Thanks sir cloning will work na..?

[r0oth3x49]

You all should use --skip-hls when downloading large course contents that way udemy-dl will avoid requesting hls based streams and the fetching will be faster otherwise it will take time.

will it work.? or I'm ready to provide my cookie access token details

it should work now try testing and let me know if you still face the issue, --skip-hls is to done things faster.

--skip-hls throwing error bro :(

will i have to clone the latest repo

just run a git pull inside of udemy-dl if you have cloned it otherwise download the master zip.

[xsirvent]

Thanks sir cloning will work na..?

git push will update the code to the last version

[r0oth3x49]

Thanks sir cloning will work na..?

git push will update the code to the last version

should be git pull not push

[xsirvent]

should be git pull not push
Oops :-)

[iamgovindthakur]

Thanks sir cloning will work na..?

git push will update the code to the last version

should be git pull not push

yeah its work but as i told you in previous issue i have already downloaded 70% of course but after the --skip-hls added its downloading the course from first chapter it should resume but it's not.My udemy course consist of 36 chapter and i have downloaded till 25 chapter but it's downloading from start and replacing the previous downloaded file one by one please help.

[r0oth3x49]

Thanks sir cloning will work na..?

git push will update the code to the last version

should be git pull not push

yeah its work but as i told you in previous issue i have already downloaded 70% of course but after the --skip-hls added its downloading the course from first chapter it should resume but it's not.My udemy course consist of 36 chapter and i have downloaded till 25 chapter but it's downloading from start and replacing the previous downloaded file one by one please help.

this is strange until unless the name of lecture and videos not getting changed udemy-dl should resume downloading instead. --skip-hls is used to avoid requesting hls based streams that's it. btw hls based streams will always be downloaded from scratch if you are also using --quality with 1080 then it's normal behavior to download the video again because there is no way we can resume HLS stream so far in my knowledge i don't know if we can resume downloading HLS streams.

but yes it has a support to resume a course or a mp4 video.

[iamgovindthakur]

Thanks sir cloning will work na..?

git push will update the code to the last version

should be git pull not push

yeah its work but as i told you in previous issue i have already downloaded 70% of course but after the --skip-hls added its downloading the course from first chapter it should resume but it's not.My udemy course consist of 36 chapter and i have downloaded till 25 chapter but it's downloading from start and replacing the previous downloaded file one by one please help.

this is strange until unless the name of lecture and videos not getting changed udemy-dl should resume downloading instead. --skip-hls is used to avoid requesting hls based streams that's it. btw hls based streams will always be downloaded from scratch if you are also using --quality with 1080 then it's normal behavior to download the video again because there is no way we can resume HLS stream so far in my knowledge i don't know if we can resume downloading HLS streams.

but yes it has a support to resume a course or a mp4 video.

thanx i use my brain and some of your syntax and i used downloading video from a specific range of chapter and i downloaded the only remaining chapter by --chapter-start 25 --chapter-end 36 command. And if you think it's strange i will capture the video and i will show exactly what's happening behind the scene to you, actually for some video it was skipping by showing the (already downloaded) but for most of the video it was downloading from beginning as part file and after finished downloading it was replacing the previously downloaded file, anyway --chapter-start 25 --chapter-end 36 this take me out. Thanks once again Love from Bihar,India.

[r0oth3x49]

Thanks sir cloning will work na..?

git push will update the code to the last version

should be git pull not push

yeah its work but as i told you in previous issue i have already downloaded 70% of course but after the --skip-hls added its downloading the course from first chapter it should resume but it's not.My udemy course consist of 36 chapter and i have downloaded till 25 chapter but it's downloading from start and replacing the previous downloaded file one by one please help.

this is strange until unless the name of lecture and videos not getting changed udemy-dl should resume downloading instead. --skip-hls is used to avoid requesting hls based streams that's it. btw hls based streams will always be downloaded from scratch if you are also using --quality with 1080 then it's normal behavior to download the video again because there is no way we can resume HLS stream so far in my knowledge i don't know if we can resume downloading HLS streams.
but yes it has a support to resume a course or a mp4 video.

thanx i use my brain and some of your syntax and i used downloading video from a specific range of chapter and i downloaded the only remaining chapter by --chapter-start 25 --chapter-end 36 command. And if you think it's strange i will capture the video and i will show exactly what's happening behind the scene to you, actually for some video it was skipping by showing the (already downloaded) but for most of the video it was downloading from beginning as part file and after finished downloading it was replacing the previously downloaded file, anyway --chapter-start 25 --chapter-end 36 this take me out. Thanks once again Love from Bihar,India.

thanks man glad to help you, have fun.

[dragonhelios]

@r0oth3x49 Thanks for the assist.