Gallery-dl: Issue with -g option for some website albums

[net00-1]

There's this issue in which gallery-dl with the -g option to print direct image URLs will only print the URL of an intermediary page. This happens on both Instagram profiles and imgur.com/gallery/ URLs.

For example, if I send this command I am supposed to get the first 10 image URLs from Microsoft's Instagram page:

gallery-dl -g "https://www.instagram.com/microsoft/" --range 1-10

What I am seeing returned by the program instead is this single URL:

https://www.instagram.com/microsoft/posts/

This does not happen if I tell gallery-dl to download normally without -g, the 10 images are downloaded correctly:

gallery-dl "https://www.instagram.com/microsoft/" --range 1-10 (without -g)

The same thing happens with imgur.com/gallery/ URLs:

gallery-dl -g "https://www.imgur.com/gallery/SOMEID" --range 1-10 will just return "https://www.imgur.com/a/SOMEID" instead of the 10 images, and does not happen without -g.

If I take the single URL that is returned and run it through gallery-dl again I'll actually get the images, as if it was an intermediary webpage or redirect.

Here's the output with -v both with and without -g:

gallery-dl -vg "https://www.instagram.com/microsoft" --range '1-10'

> [gallery-dl][debug] Version 1.16.1
> [gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.19041
> [gallery-dl][debug] requests 2.24.0 - urllib3 1.25.11
> [gallery-dl][debug] Starting UrlJob for 'https://www.instagram.com/microsoft'
> [instagram][debug] Using InstagramUserExtractor for 'https://www.instagram.com/microsoft'
> https://www.instagram.com/microsoft/posts/

gallery-dl -v "https://www.instagram.com/microsoft" --range '1-10'

> [gallery-dl][debug] Version 1.16.1
> [gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.19041
> [gallery-dl][debug] requests 2.24.0 - urllib3 1.25.11
> [gallery-dl][debug] Starting DownloadJob for 'https://www.instagram.com/microsoft'
> [instagram][debug] Using InstagramUserExtractor for 'https://www.instagram.com/microsoft'
> [instagram][debug] Using InstagramPostsExtractor for 'https://www.instagram.com/microsoft/posts/'
> [urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.instagram.com:443
> [urllib3.connectionpool][debug] https://www.instagram.com:443 "GET /microsoft/ HTTP/1.1" 200 35496
> [urllib3.connectionpool][debug] Starting new HTTPS connection (1): instagram.fntr3-1.fna.fbcdn.net:443
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/134712486_315395023097602_2596498174021177357_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=1&_nc_ohc=6jXA7f5ZqAIAX9WxvNg&tp=1&oh=7b4239a6d2fffdfe32982c38f6dc9957&oe=601CFC77 HTTP/1.1" 200 9789
> * .\gallery-dl\instagram\microsoft\2478169519687051701_2478169514788105565.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/134676610_403020647480791_4075185667738756515_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=1&_nc_ohc=OViQW1Kx5KgAX-UrsUj&tp=1&oh=1ecf7896f078f3dade908cb137b685b8&oe=6019ECF6 HTTP/1.1" 200 14537
> * .\gallery-dl\instagram\microsoft\2478169519687051701_2478169514813317862.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/135141279_1885610984923531_4423342326733026117_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=1&_nc_ohc=vCW7LjQbGiYAX92GwoU&tp=1&oh=1b21a1595642ae60955ee15c0ccbab2b&oe=601AC29A HTTP/1.1" 200 17304
> * .\gallery-dl\instagram\microsoft\2478169519687051701_2478169514846972522.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/s1080x1080/133948392_159035775973817_6201687638323115971_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=105&_nc_ohc=_NR-cpbASS0AX93aVwc&tp=1&oh=43fea55ae318d6155c8cf526d7516456&oe=601AE4BC HTTP/1.1" 200 22458
> * .\gallery-dl\instagram\microsoft\2476438956114672625.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/133827649_666209854073162_8859153311993474395_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=102&_nc_ohc=qeOkozl1UjgAX_gFIoA&tp=1&oh=a56878452848e88b14f5d2d5a4456986&oe=6019D5FB HTTP/1.1" 200 67096
> * .\gallery-dl\instagram\microsoft\2475720117093559273.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/133624722_3265933510175750_6695938396438787354_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=102&_nc_ohc=VDcbdRvej2QAX94WLDu&tp=1&oh=7b0f39eb8a7667870d43292d571608db&oe=601AEC06 HTTP/1.1" 200 37793
> * .\gallery-dl\instagram\microsoft\2475024646117232960.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/s1080x1080/132408611_130490712124765_966690171624069092_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=101&_nc_ohc=rQ2LYVXuvbAAX_-drUl&tp=1&oh=fc8058844a0f560089aa388c56a7e81d&oe=601C3D74 HTTP/1.1" 200 145546
> * .\gallery-dl\instagram\microsoft\2472811881348496164_2472811876944668917.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/s1080x1080/133694410_713814119279177_4440836622433032742_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=105&_nc_ohc=Pl2x1ob_tW0AX9imTpD&tp=1&oh=c6b8dd887688a70a37302cde45e9c653&oe=601D2BAC HTTP/1.1" 200 93288
> * .\gallery-dl\instagram\microsoft\2472811881348496164_2472811876927848922.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/s1080x1080/132427162_1133429763759199_9112933347941456883_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=107&_nc_ohc=js37k5qQc60AX8FGEVX&tp=1&oh=44d1f3b2f2f32331a5d7484a2ab2946e&oe=601A6F9C HTTP/1.1" 200 29653
> * .\gallery-dl\instagram\microsoft\2471494693975077882.jpg
> [urllib3.connectionpool][debug] https://instagram.fntr3-1.fna.fbcdn.net:443 "GET /v/t51.2885-15/e35/s1080x1080/132163756_866341834143296_341798184892026154_n.jpg?_nc_ht=instagram.fntr3-1.fna.fbcdn.net&_nc_cat=108&_nc_ohc=T9LJKyNwRlIAX8bobi6&tp=1&oh=1cbc129d5b6649073cae391ce5520a2d&oe=601A9B1B HTTP/1.1" 200 58576
> * .\gallery-dl\instagram\microsoft\2470707171942474543.jpg

[mikf]

You can use -g multiple times, or now with https://github.com/mikf/gallery-dl/commit/1c9435e0dfbcf39679fc76c3d6076bc55ed1e0e4 -G once, to tell it to resolve those intermediate URLs (each small -g tells it to go one level deeper).

$ gallery-dl -ggg --range 1 https://www.instagram.com/microsoft
https://instagram.ftxl3-1.fna.fbcdn.net/v/t51.2885-15/e35/136072666_3796612113759767_5262551286627151533_n.jpg?_nc_ht=instagram.ftxl3-1.fna.fbcdn.net&_nc_cat=111&_nc_ohc=M6IlRChYmcsAX_LToR-&tp=1&oh=594f89ed75eec3253e1a0a41c949b25e&oe=60229D9D

[net00-1]

Thanks for looking into this! -G is the best solution.