Gallery-dl: Error downloading user videos from Twitter while providing profile link

[KaMyKaSii]

The download is successful with individual video links (but saved in an "i" folder rather than one with the correct username). Apparently it's a problem of youtube-dl itself, I'm not sure if it will be possible for you to find a solution

$ gallery-dl -v twitter.com/bb_puta
[gallery-dl][debug] Version 1.11.0-dev
[gallery-dl][debug] Python 3.7.4 - Linux-4.4.141-perf+-aarch64-with-libc
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.3
[gallery-dl][debug] Starting DownloadJob for 'twitter.com/bb_puta'
[gallery-dl][debug] Updating urllib3 ciphers
[twitter][debug] Using TwitterTimelineExtractor for 'twitter.com/bb_puta'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/profiles/show/bb_puta/ti
meline/tweets?include_available_features=1&include_entities=1&reset_error_state=false&la
ng=en HTTP/1.1" 200 19339
/sdcard/gallery-dl/twitter/bb_puta/1116823538307039232_1.jpg
[downloader.ytdl][debug] [twitter] 1184559664303878150: Downloading webpage
[downloader.ytdl][error] ERROR: Unable to extract OpenGraph title; please report this is
sue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://
yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag a
nd include its complete output.
[download][error] Failed to download 1116823538307039232_1.jpg
[downloader.ytdl][debug] [twitter] 1162026165206487040: Downloading webpage
[downloader.ytdl][error] ERROR: Unable to extract OpenGraph title; please report this is
sue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://
yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag a
nd include its complete output.
[download][error] Failed to download 1116823538307039232_1.jpg
[downloader.ytdl][debug] [twitter] 1161866872616366081: Downloading webpage
[downloader.ytdl][error] ERROR: Unable to extract OpenGraph title; please report this is
sue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://
yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag a
nd include its complete output.
[download][error] Failed to download 1116823538307039232_1.jpg
/sdcard/gallery-dl/twitter/bb_puta/1161866431119679488_1.jpg
/sdcard/gallery-dl/twitter/bb_puta/1160620638358384641_1.jpg
/sdcard/gallery-dl/twitter/bb_puta/1160620638358384641_2.jpg
/sdcard/gallery-dl/twitter/bb_puta/1160620638358384641_3.jpg
/sdcard/gallery-dl/twitter/bb_puta/1160620638358384641_4.jpg
/sdcard/gallery-dl/twitter/bb_puta/1156267861675008006_1.jpg
[downloader.ytdl][debug] [twitter] 1156069994196852736: Downloading webpage
[downloader.ytdl][error] ERROR: Unable to extract OpenGraph title; please report this is
sue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://
yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag a
nd include its complete output.
[download][error] Failed to download 1156267861675008006_1.jpg
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/profiles/show/bb_puta/ti
meline/tweets?include_available_features=1&include_entities=1&max_position=1155506042488
442880&reset_error_state=false&lang=en HTTP/1.1" 200 19141
[downloader.ytdl][debug] [twitter] 1154461534921396230: Downloading webpage
^C
KeyboardInterrupt
$

[mikf]

See #352. Downloading Twitter videos with youtube-dl while being logged in doesn't work at the moment, but you can disable forwading cookies to youtube-dl and it will work again, although I'm not sure if you can download private videos like that.

edit: you should put console logs in triple backticks to make them more readable:

```
\ ```

[KaMyKaSii]

See #352. Downloading Twitter videos with youtube-dl while being logged in doesn't work at the moment, but you can disable forwading cookies to youtube-dl and it will work again, although I'm not sure if you can download private videos like that.

edit: you should put console logs in triple backticks to make them more readable:

<your log here>

Thanks, your tip has fixed downloading public videos but not private videos, as you suspected. I did a bit of research and found that private videos don't need authentication to download, it just needs to get the url. The log below is an attempt to download a private video, apparently gallery-dl is unable to extract the url. And if it is not gallery-dl's job to extract direct video links to youtube-dl, could you consider an option to extract them and use ffmpeg instead?

$ gallery-dl -v https://mobile.twitter.com/ecchikawaiigirl/status/1184539361641140226
[gallery-dl][debug] Version 1.11.0-dev
[gallery-dl][debug] Python 3.7.4 - Linux-4.4.141-perf+-aarch64-with-libc
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.3
[gallery-dl][debug] Starting DownloadJob for 'https://mobile.twitter.com/ecchikawaiigirl
/status/1184539361641140226'
[gallery-dl][debug] Updating urllib3 ciphers
[twitter][debug] Using TwitterTweetExtractor for 'https://mobile.twitter.com/ecchikawaii
girl/status/1184539361641140226'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/web/status/1184539361641
140226 HTTP/1.1" 302 103
[urllib3.connectionpool][debug] https://twitter.com:443 "GET / HTTP/1.1" 200 50449
[twitter][error] An unexpected error occurred: ValueError - substring not found. Please 
run gallery-dl again with the --verbose flag, copy its output and report this issue on h
ttps://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/gallery_dl/job.py", 
line 47, in run
    for msg in self.extractor:
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/gallery_dl/extractor
/twitter.py", line 42, in items
    for tweet in self.tweets():
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/gallery_dl/extractor
/twitter.py", line 264, in tweets
    end = page.index('class="js-tweet-stats-container')
ValueError: substring not found
$

[mikf]

Extracting and downloading video streams can be rather complicated, hence youtube-dl gets used for it here. Maybe I'll look into it, but I'd rather wait for youtube-dl to fix their code.

... And I just realized that gallery-dl suffers from the same problem as youtube-dl when it comes to individual Tweets: Requesting a Tweet page while being logged in would get it in the new layout, so gallery-dl clears its cookies (and authentication data) before accessing that Tweet, since it currently only supports the old Twitter interface.

[alice945]

Requesting a Tweet page while being logged in would get it in the _new_ layout, so gallery-dl clears its cookies (and authentication data) before accessing that Tweet, since it currently only supports the old Twitter interface.

Changing your user agent to "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" forces the old layout, it's what I've been using ever since twitter changed it. Maybe you could make it use this user agent anytime you use authentication?

[mikf]

Thank you @alice945. Setting a user agent alone wasn't enough to convince Twitter to revert to its old layout, but it got me looking in the "right" direction. (See https://github.com/mikf/gallery-dl/commit/2eb38810c5ffb7c675581543ca83091e4adbda92)

The "only" problem now is getting youtube-dl to do the same when extracting a video stream from a Twitter page, but as far as I can tell that's impossible. The only way appears to be updating youtube-dl's code and opening a pull request there, but I'd rather do what @KaMyKaSii suggested and look into getting the video information without youtube-dl.

[mikf]

OK, I've managed to implement video extraction (https://github.com/mikf/gallery-dl/commit/49a6b1b6c03944c715b1d9cc11177f2396a82cc2). It worked for me with and without being logged in, but I don't have access to any protected account so I couldn't test that.

[KaMyKaSii]

OK, I've managed to implement video extraction (49a6b1b). It worked for me with and without being logged in, but I don't have access to any protected account so I couldn't test that.

It worked here, thank you very much!

[github-userx]

Mike your work with gallery-dl is just impressive! And very appreciated!