Gallery-dl: Questions, Feedback and Suggestions #2

Created on 23 Jan 2018  ·  97Comments  ·  Source: mikf/gallery-dl

Continuation of the old issue as a central place for any sort of question or suggestion not deserving their own separate issue.

11 had gotten too big, took several seconds to load, and was closed as a result.

There is also https://gitter.im/gallery-dl/main if that seems more appropriate.

Meta Questions
Source
picture mikf

Most helpful comment

@AKluge
You are missing quotation marks around the string you want to test against.
It should be "tags[-1] != '\uXXXX\uXYYY...'" , i.e. single quotes around the \uXXXX\uXYYY part.

If you want to test if a string is inside a list, you can use the in operator: "'foo' in tags"

--filter "is_muted == False" ignored the muted files, but in the config-file the "image-filter":"is_muted == False" statement is still not working

The command-line option does exactly the same as the image-filter config option (*), so maybe you did something wrong here, but what you posted looks alright.

I've been using this configuration to test and it works just fine:

{ "extractor": {
    "pixiv": {
        "image-filter": "is_muted == False",
        "username": "foo",
        "password": "bar"
    }
} }

I would also recommend "not is_muted" to test if something is false. This would also work if is_muted is None or an empty string/list.

(*) it effectively sets a global value for image-filter and therefore overwrites all image-filter entries from a config file

@taiyu-len

also pixiv sending me an email everytime i use this is a bit annoying,

Shouldn't happen anymore: https://github.com/mikf/gallery-dl/commit/8faf03ed847ae1cf2bb19d04f678204ed49c2e24
I've never noticed this myself, since all email addresses for my Pixiv accounts are (now) invalid.

Concerning directory metadata, you can put a yield Message.Directory, work before any yield Message.Url, url, work to "fix" it, like so:

diff --git a/gallery_dl/extractor/pixiv.py b/gallery_dl/extractor/pixiv.py
index 115b1fb..8716a2d 100644
--- a/gallery_dl/extractor/pixiv.py
+++ b/gallery_dl/extractor/pixiv.py
@@ -31,7 +31,6 @@ class PixivExtractor(Extractor):
         metadata = self.get_metadata()

         yield Message.Version, 1
-        yield Message.Directory, metadata

@@ -55,11 +54,13 @@ class PixivExtractor(Extractor):
                     "_ugoira600x600", "_ugoira1920x1080")
                 work["frames"] = ugoira["frames"]
                 work["extension"] = "zip"
+                yield Message.Directory, work
                 yield Message.Url, url, work

             elif work["page_count"] == 1:
                 url = meta_single_page["original_image_url"]
                 work["extension"] = url.rpartition(".")[2]
+                yield Message.Directory, work
                 yield Message.Url, url, work

@@ -67,6 +68,7 @@ class PixivExtractor(Extractor):
                     url = img["image_urls"]["original"]
                     work["num"] = "_p{:02}".format(num)
                     work["extension"] = url.rpartition(".")[2]
+                    yield Message.Directory, work
                     yield Message.Url, url, work

But keep in mind that, as for right now, gallery-dl isn't optimized for this kind of thing. Usually general metadata for directories and specialized metadata for filenames is more than enough.

@wankio
You can now set deviantart.original to "images" to only download original image files and fall back to the preview version otherwise (https://github.com/mikf/gallery-dl/commit/d8492df51b40ff96f3dae51fa42b8cc89a4135e8)

picture mikf on 16 Oct 2018
👍2

All 97 comments

A bit of feedback:
I've been downloading some stuff from Tumblr now, and everything seems to be working like a charm so far. Kudos to you.
Including --write-unsupported FILE, detecting and saving embedded links from Vine, Instagram and links to other external sites. Safe to assume it'll work just the same with embeds from YouTube and Vimeo, which I thought to be pretty common on Tumblr, but it's entirely possible that the processed blogs so far actually don't had a single one. I try that later on my own test blog, just to be sure.

This also led me to another idea/suggestion:
How about a --write-log FILE feature?
I'm aware that there is something like tee, and even something similar for PowerShell, but I think the purpose for the logging feature (at least at default setting) would not be to replicate the full output printed to the console, no, only any relevant errors or warnings, i.e. like 404, 403 and also if there occurs some timeout or something, which can be caused easily by connectivity issues, not by gallery-dl itself.

Not sure, just an idea...

Hrxn picture Hrxn on 25 Jan 2018

Sounded like a useful feature, so I tried to put something together: https://github.com/mikf/gallery-dl/commit/97f4f15ec0f15b6bca10bf33bf83f3fcd50c1e12. It should more or less behave like one would expect, but there are at least two things that might be better handled otherwise:

Edit: Never mind the points below, I thought about it and decided to change its behavior. It is no longer persistent and stores exactly the same log messages as shown on screen (https://github.com/mikf/gallery-dl/commit/c9a9664a65b4160857834355d234ffe46107559d).

  • Log files are currently persistent across gallery-dl invocations and can grow indefinitely. Other options would be deleting its old content each time or even log rotations (Python has built-in support for that as well).
  • Debug log messages will never be written to a log file, even when using '--verbose'. It would be easier to copy the --verbose output from a text file instead of a console window, but writing all of the debug output to an otherwise concise log file didn't seem like a good idea.
mikf picture mikf on 26 Jan 2018

One small addition to logging behaviour:

Not entirely sure without a test case at hand now, but how's the current output for images on Tumblr relying on the fallback mechanism? I don't know if I remember that correctly, but it appeared that trying to download some specific image (e.g. some old images, old URL scheme etc.) with s3.amazonaws which then resulted in an error (403) prints the corresponding messages to the terminal, but the next successful download had the same Post ID in the name, so that would be the fallback URL used here.
I hope you know what I mean, if not I'll try some random blogs again and wait for that error to appear and copy the messages here.
Because the question is, should the error be printed to the output/logged to file in this case here? Because even if the "raw" URL does not work and we run into an actual 403, the fallback URL/mechanism still works and downloads that image instead successfully, so it's a bit debatable if that would really qualify as a real "error" 😄

_Edit:_

Had the opportunity to think about this again, and I'm not sure if it's actually worth to bother. Sure, it may be less than ideal (Think of your average end-user©® and the reaction: _"OMG It says there's some error"_), but this can be solved by simply, uh, explaining the stuff. And if this really warrants to deal with errors that are actually not so much of an error, by having to handle different error types, error classes, error codes and whatever, just for the sake of what, actually, consistency (?), I'm not really sure anymore.

Hrxn picture Hrxn on 28 Jan 2018

I think it is actually worth to bother. With the way things were, it was impossible to tell if all files have been downloaded successfully by just looking at a log file, and error messages from an image with fallback URL were kind of misleading as well, since the image download in question did succeed in the end.

I added two more logging messages to hopefully remedy this (https://github.com/mikf/gallery-dl/commit/db7f04dd975af53e120cfe3151b88f0bc57371cf):

  1. Failed to download <filename> when an image could not be downloaded, even when using fallback URLs.
  2. Trying fallback URL #<number> to indicate that the last error message is not fatal.

Maybe it would be better to categorize all HTTP errors as warnings and only show the Failed to download … message as definite error?

mikf picture mikf on 28 Jan 2018

Maybe it would be better to categorize all HTTP errors as warnings and only show the Failed to download … message as definite error?

Yeah, sounds good.
👍

Hrxn picture Hrxn on 28 Jan 2018

@mikf,

  • which extractor class from common module should i use to make CustomExtractor? is there any requirement for each extractor class?
  • can you explain the extractor class' attribute? (e.g. from BooruExtractor have 'basecategory', 'filename_fmt', 'api_url', 'per_page', 'page_start', 'page_limit', etc)
  • is there any rule on how metadata attribute on extractor class should be built, or is it up to each extractor?
rachmadaniHaryono picture rachmadaniHaryono on 30 Jan 2018

  1. Generally you should use the basic Extractor class, but, as always, it depends. There are some general extractor sub-classes (BooruExtractor, FoolslideExtractor, FoolfuukaExtractor, ...) and it might also be helpful to just copy an existing extractor module and adjust it to your needs.
    As for requirements: set the category, subcategory, filename_fmt, directory_fmt and pattern class attributes to some reasonable values (see, for example, slideshare.py).

  2. category and subcategory are essentially an extractor's name and are used for config-lookup.
    directory_fmt and filename_fmt are default values for the directory and filename options.
    pattern is a list of regex-strings. An extractor is used if one of them matches the given URL. The resulting match-object is the second parameter to an extractor's __init__() method.
    basecategoryhas to do with shared config values, just ignore it.

    The other attributes you listed are BooruExtractor-specific:

    • api_url: URL to send API requests to
    • per_page: number of post-entries per page
    • page_start: the first page (0 or 1 depending on site)
    • page_limit: largest valid page number

  3. You kind of asked the same thing before.
    It is up to each extractor, but similar ones should use the same key-names. For image-metadata, you should always provide the filename extension as extension or at least set it to None.

mikf picture mikf on 30 Jan 2018

is there any rule on how metadata attribute on extractor class should be built, or is it up to each extractor?

You kind of asked the same thing before.
It is up to each extractor, but similar ones should use the same key-names. For image-metadata, you should always provide the filename extension as extension or at least set it to None.

actually i got the wrong impression from chan.py. i thought that to use the match object later on items method it should be stored on class' attribute metadata. but after i look at slideshare.py, any name can be used (e.g. user and presentation)

here is what i can come up with https://gist.github.com/rachmadaniHaryono/e7d40fcc5b9cd6ecc1f9151c4f0f5d84

full code https://github.com/rachmadaniHaryono/RedditImageGrab/blob/master/redditdownload/api.py

this module will not download a file, but it will only extract from url

rachmadaniHaryono picture rachmadaniHaryono on 31 Jan 2018

@mikf can you give example for https://github.com/mikf/gallery-dl/commit/6a07e3836603407ee7fc17305f0ae7165b76f83c ?

rachmadaniHaryono picture rachmadaniHaryono on 2 Feb 2018 
from my_project import module_with_extractors
class SomeExtractor(Extractor):
    ...

from gallery_dl import extractor
extractor.add(SomeExtractor)
extractor.add_module(module_with_extractors)

You should use these functions instead of manually manipulating extractor._cache and relying on implementation details.

mikf picture mikf on 2 Feb 2018

I'm doing something wrong? And i try option in config file, don't working.

2018-02-25_000317

ChiChi32 picture ChiChi32 on 24 Feb 2018

Which version of gallery-dl is that? Can you run gallery-dl -v please?

Hrxn picture Hrxn on 25 Feb 2018

Can we have percent-encoding conversions for saved files? I.e. replacing %20 in filename with whitespace, %22 with ", etc.

Bfgeshka picture Bfgeshka on 25 Feb 2018

@Hrxn , I'm a bit embarrassed ... I found a strange thing. I have 2 folders, gallery_dl and gallery_dln. The first is the old version 1.1.2, the second is 1.2.1. Both are in the same directory. When I run any command using the bat file from the folder with the new version, the modules are taken from the old one. When I run -version from the 1.2.1 folder, 1.1.2 is displayed. I do not think that this is a problem program, rather Windous or Python. I apologize for the disturbance.

ChiChi32 picture ChiChi32 on 25 Feb 2018

@ChiChi32 the __main__.py file expects to sit inside a directory named gallery_dl.
In your specific case it adds F:\Python to its PYTHONPATH environment and then imports the gallery_dl package, which is the older 1.1.2 version.
If you want to use multiple versions at the same time, you could try a directory-structure like

|- gallery-dl-1.1.2
|  \- gallery_dl
|     |- __main__.py
|     |- ...
\- gallery-dl-1.2.1
   \- gallery_dl
      |- __main__.py
      |- ...

@Bfgeshka Sure, I think I'll add another conversion option for format strings to let users unquote the "offending" parts of a filename.
These percent-encoding conversions (and similar) for each metadata-field are usually already handled as necessary. Where did you find something that hasn't been properly converted?

mikf picture mikf on 25 Feb 2018

@mikf I encountered it in direct link download.

Bfgeshka picture Bfgeshka on 26 Feb 2018

Some small thing I've noticed. Not a real issue deserving of a ticket, I presume.
But still curious what it means, or what is the cause behind it.

PS E:\> gallery-dl.exe -v 'https://gfycat.com/distortedmemorableibizanhound'
[gallery-dl][debug] Version 1.3.1
[gallery-dl][debug] Python 3.4.4 - Windows-10-10.0.16299
[gallery-dl][debug] requests 2.18.4 - urllib3 1.22
[gallery-dl][debug] Starting DownloadJob for 'https://gfycat.com/distortedmemorableibizanhound'
[gfycat][debug] Using GfycatImageExtractor for 'https://gfycat.com/distortedmemorableibizanhound'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): gfycat.com
[urllib3.connectionpool][debug] https://gfycat.com:443 "GET /cajax/get/distortedmemorableibizanhound HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): giant.gfycat.com
[urllib3.connectionpool][debug] https://giant.gfycat.com:443 "GET /DistortedMemorableIbizanhound.webm HTTP/1.1" 200 18240849
* E:\Transfer\INPUT\GLDL\Anims\\fluid in an invisible box750-720p DistortedMemorableIbizanhound.webm
PS E:\>

The download seems to work, just as it's apparent above. But the output is a bit different, not what I'm used to see, just observe this output path: E:\Transfer\INPUT\GLDL\Anims\\fluid in an invisible box750- [...]

The extra backslash, as if some directory is missing in between.

I post the configuration used here:

  • The general part (keywords, and keywords-default)
{
    "base-directory": "E:\\Transfer\\INPUT\\GLDL",
    "netrc": false,

    "downloader":
    {
        "part": true,
        "part-directory": null,
        "http":
        {
            "rate": null,
            "retries": 5,
            "timeout": 30,
            "verify": true
        }
    },
    "extractor":
    {
        "keywords": {"bkey": "", "ckey": "", "tkey": "", "skey": "", "mkey": ""},
        "keywords-default": "",
        "archive": "E:\\Transfer\\INPUT\\GLDL\\_Archives\\gldl-archive-global.db",
        "skip": true,
        "sleep": 0,
[...]
  • Gfycat
        "gfycat":
        {
            "directory": ["Anims", "{bkey}", "{ckey}", "{tkey}", "{skey}", "{mkey}"],
            "filename": "{title:?/ /}{gfyName}.{extension}",
            "format": "webm"
        },

__But__ it does not happen here, for example:

  • Imgur
        "imgur":
        {
            "image":
            {
                "directory": ["{bkey}", "{ckey}", "{tkey}", "{skey}", "{mkey}", "Images"],
                "filename": "{title:?/ /}{hash}.{extension}"
            },
            "album":
            {
                "directory": ["{bkey}", "{ckey}", "{tkey}", "{skey}", "{mkey}", "Albums", "{album[title]:?/ /}{album[hash]}"],
                "filename": "{album[hash]}_{num:>03}_{hash}.{extension}"
            },
            "archive": "E:\\Transfer\\INPUT\\GLDL\\_Archives\\gldl-archive-imgur.db",
            "mp4": true
        },

Single Image:

PS E:\> gallery-dl.exe -v 'https://imgur.com/5m4CFZS'
[gallery-dl][debug] Version 1.3.1
[gallery-dl][debug] Python 3.4.4 - Windows-10-10.0.16299
[gallery-dl][debug] requests 2.18.4 - urllib3 1.22
[gallery-dl][debug] Starting DownloadJob for 'https://imgur.com/5m4CFZS'
[imgur][debug] Using ImgurImageExtractor for 'https://imgur.com/5m4CFZS'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): imgur.com
[urllib3.connectionpool][debug] https://imgur.com:443 "GET /5m4CFZS HTTP/1.1" 200 49800
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): i.imgur.com
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /5m4CFZS.png HTTP/1.1" 200 1940189
* E:\Transfer\INPUT\GLDL\Images\Caley Rae Pavillard 5m4CFZS.png
PS E:\>

Album:

PS E:\> gallery-dl.exe -v 'https://imgur.com/a/jQxtc'
[gallery-dl][debug] Version 1.3.1
[gallery-dl][debug] Python 3.4.4 - Windows-10-10.0.16299
[gallery-dl][debug] requests 2.18.4 - urllib3 1.22
[gallery-dl][debug] Starting DownloadJob for 'https://imgur.com/a/jQxtc'
[imgur][debug] Using ImgurAlbumExtractor for 'https://imgur.com/a/jQxtc'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): imgur.com
[urllib3.connectionpool][debug] https://imgur.com:443 "GET /a/jQxtc/all HTTP/1.1" 200 62847
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): i.imgur.com
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /t9CD48N.jpg HTTP/1.1" 200 126079
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_001_t9CD48N.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /VoGBS4N.jpg HTTP/1.1" 200 148669
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_002_VoGBS4N.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /svbJXyy.jpg HTTP/1.1" 200 146013
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_003_svbJXyy.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /kDjvkrD.jpg HTTP/1.1" 200 130492
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_004_kDjvkrD.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /GxPVJSw.jpg HTTP/1.1" 200 154477
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_005_GxPVJSw.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /tUIUbSL.jpg HTTP/1.1" 200 194268
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_006_tUIUbSL.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /vcvv1r0.jpg HTTP/1.1" 200 193132
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_007_vcvv1r0.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /YBQddcB.jpg HTTP/1.1" 200 147301
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_008_YBQddcB.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /FkuxOXZ.jpg HTTP/1.1" 200 169420
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_009_FkuxOXZ.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /MB30wRC.jpg HTTP/1.1" 200 223108
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_010_MB30wRC.jpg
[urllib3.connectionpool][debug] https://i.imgur.com:443 "GET /TGnsGoh.jpg HTTP/1.1" 200 147744
* E:\Transfer\INPUT\GLDL\Albums\jQxtc\jQxtc_011_TGnsGoh.jpg
PS E:\>

Will add some more tests eventually, to see if I can get any different results with various input file options.

But so far, it seems to have something to do with
"directory": ["{bkey}", .... ..., output path beginning with my custom keyword
vs.
"directory": ["Anims", "{bkey}",... ... output path starts with a fixed directory.

Hrxn picture Hrxn on 22 Mar 2018

This happens because of os.path.join()'s behavior when using an empty string as the last argument:

>>> from os.path import join
>>> join("", "d1", "", "d2")
'd1/d2'
>>> join("", "d1", "", "d2", "")
'd1/d2/'

It adds a slash (or back-slash on Windows) to the end if the last argument is an empty string.

I've been using path = directory + separator + filename to build the final complete path with the assumption that all directories don't have a path-separator at the end, which, in your case, resulted in two of them ("...\Anims\" + "\" + "fluid in an invisible box...").

mikf picture mikf on 22 Mar 2018

Ah, thanks. Makes sense. This behaviour of os.path.join() is again something which makes me wonder if such behavior is intentional, or if it is just some quirk.
This time at least it's a quirk that affects all platforms in the same way, right? 😄

Edit:
Ha, my mistake. It's probably intentional, I see where this could be useful.

Hrxn picture Hrxn on 22 Mar 2018

BTW, everything works with the latest commit, example URL above is correct and did not encounter it anywhere else!

Hrxn picture Hrxn on 23 Mar 2018
👍1

Just something quick I noticed - gallery-dl appears to be unable to handle certain emoji appearing in captions on tumblr (and maybe elsewhere??).
(Warning - the post I was able to trigger this with this on is rather nsfw)
Running --list-keywords on an offending post with --verbose and piping the error output with 2> to a file gets me

[gallery-dl][debug] Version 1.3.2
[gallery-dl][debug] Python 3.4.4 - Windows-7-6.1.7601-SP1
[gallery-dl][debug] requests 2.18.4 - urllib3 1.22
[gallery-dl][debug] Starting KeywordJob for 'http://aurahack18.tumblr.com/post/172338300565'
[tumblr][debug] Using TumblrPostExtractor for 'http://aurahack18.tumblr.com/post/172338300565'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.tumblr.com
[urllib3.connectionpool][debug] https://api.tumblr.com:443 "GET /v2/blog/aurahack18.tumblr.com/info?api_key=O3hU2tMi5e4Qs5t3vezEi6L0qRORJ5y9oUpSGsrWu8iA3UCc3B HTTP/1.1" 200 371
[urllib3.connectionpool][debug] https://api.tumblr.com:443 "GET /v2/blog/aurahack18.tumblr.com/posts?reblog_info=true&id=172338300565&api_key=O3hU2tMi5e4Qs5t3vezEi6L0qRORJ5y9oUpSGsrWu8iA3UCc3B&offset=0&limit=50 HTTP/1.1" 200 1374
[tumblr][error] An unexpected error occurred: UnicodeEncodeError - 'cp932' codec can't encode character 'u2661' in position 3: illegal multibyte sequence. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[tumblr][debug] Traceback
Traceback (most recent call last):
File "E:gallery-dlgallery_dljob.py", line 64, in run
File "E:gallery-dlgallery_dljob.py", line 117, in dispatch
File "E:gallery-dlgallery_dljob.py", line 131, in handle_urllist
File "E:gallery-dlgallery_dljob.py", line 236, in handle_url
File "E:gallery-dlgallery_dljob.py", line 279, in print_keywords
UnicodeEncodeError: 'cp932' codec can't encode character 'u2661' in position 3: illegal multibyte sequence

This is using the executable download, btw.

reversebreak picture reversebreak on 4 Apr 2018

You are running this via CMD.exe I presume?

What happens if you do this first in CMD: chcp 65001
And then run gallery-dl?

Edit:

Or try to use Powershell. I've moved completely to Powershell by now as well..

Hrxn picture Hrxn on 4 Apr 2018

This is a more general problem with the interaction between Windows, the Python interpreter, Unicode, code pages and so on.

As @Hrxn mentioned, you should be able to work around this yourself by changing the default code page to UTF-8 via chcp 65001. Another way is to set the PYTHONIOENCODING environment variable to utf-8 before running gallery-dl:

E:\>set PYTHONIOENCODING=utf-8
E:\>py -3.4 -m gallery_dl -K http://aurahack18.tumblr.com/post/172338300565
...

Python 3.6 and above also doesn't have this problem (it implements PEP 528), so using this instead of the standalone exe might be another option.

I tried to implement a simple workaround in https://github.com/mikf/gallery-dl/commit/0381ae53184108d2c6985d7200244b402a5c3b55 by setting the default error handler for stdout and co. to replace all non-encodable characters with a question mark. Tested this on my Windows 7 VM with Python3.3 to 3.6 and it seems to work.

mikf picture mikf on 4 Apr 2018

Thanks. I got Python 3.6, installed it with PIP as per instructions and it doesn't crash anymore. I put Python on my PATH and it's the same user experience as using the EXE anyways.

Now, I might be missing something, but is there any way to extract "gallery order" for use in the filenames from Tumblr?
Right now, I can point gallery-dl at a tumblr post, and it gets all the pictures fine - but sometimes that means for short two-to-ten page comic posts the files are downloaded out of order.
After doing some testing and poking around it seems that the order that the files display in a photo post isn't necessarily the same as the order of their filenames (or the 'name' parameter).

You can test this yourself by creating a photo post, uploading say four photos one at a time, then saving the post.
Go back and edit the post, and drag photo 2 before photo 1, and save it again.
If you use CTRL+RIGHTCLICK+"View Image" on each in turn your tabs should go to filenames based off of _o2, _o1, _o3, _o4.
If you point gallery-dl at it it'll download it just fine, but the end result will sort as _o1, _o2, _o3, _o4.
This will show the gallery in the wrong order, which is terrible for comic posts.
Unusually, gallery-dl seems to download them in _o2, _o1, _o3, _o4 order, (according to the on-screen status), but I can't see it exposing that order to the user in parameters anywhere.

In addition to this, I don't see a way to extract the 'core name' of a file for use in the extractor.*.filename parameter.
Tumblr filenames when downloaded without a filename parameter come out something like
tumblr_fakeusername_12345678912o1.jpg
However, using the filename parameter to add extra stuff to the filename means you can't get that clean end anymore.
The closest parameter is 'name', which comes out something like
tumblr_12345678912o1_r1_1280
or similar, when all you really want is the 12345678912o1 that only gallery-dl's default naming scheme seems to get access to.

reversebreak picture reversebreak on 7 Apr 2018

Right now, I can point gallery-dl at a tumblr post, and it gets all the pictures fine - but sometimes that means for short two-to-ten page comic posts the files are downloaded out of order.
After doing some testing and poking around it seems that the order that the files display in a photo post isn't necessarily the same as the order of their filenames (or the 'name' parameter).

Yes, I know what you mean. This was never a problem for me so far, because I've only downloaded picture sets that are just, well, a set of pictures, apparently, so the order was actually not relevant. But I agree, it's entirely different for something like a comic strip.

The filenames in a set post do not reflect the displayed order of the elements, as you already said. You also stated the reason for this, if you make a picture post and upload some files, they get generated names in this order. But this can be rearranged now, changing the order of the displayed items. What happens is that the structure you see in the end (in HTML) has the rearranged order as done by the creator of the post, but the filenames keep being the same as they were at the upload.

If you point gallery-dl at it it'll download it just fine, but the end result will sort as _o1, _o2, _o3, _o4.
This will show the gallery in the wrong order, which is terrible for comic posts.

I assume what you mean with _end result_ here is the order of the actual downloaded files. Yes, that is how they are sorted by the filesystem, in "natural" order.

Unusually, gallery-dl seems to download them in _o2, _o1, _o3, _o4 order, (according to the on-screen status), but I can't see it exposing that order to the user in parameters anywhere.

Now this bit is really interesting. Because gallery-dl just takes what it gets from the API, and this seems to indicate that the API returns the single elements in the correct order, i.e. as rendered in a browser.
This would be good, because I think this is something which could be fixed without jumping through any hoops. So gallery-dl can avoid to fetch the HTML for a post entry and extract the order from there..

In addition to this, I don't see a way to extract the 'core name' of a file for use in the extractor.*.filename parameter.

Not exactly sure what you mean here.
The filename (standard) is this:
https://github.com/mikf/gallery-dl/blob/ffc0c67701ee0902c86d27f334d51a132cc475e2/gallery_dl/extractor/tumblr.py#L54
Compared to your example at the end, the number in the filename is {id}, and that oX part is {offset}.

Small side note: This is a bit confusing within the source code, because "offset" is also used as the name for the parameter to retrieve the posts from the API.

Hrxn picture Hrxn on 7 Apr 2018

Ah, yes I did mean the final order as sorted by the filesystem - since there's no way right now to get 'gallery order' from gallery-dl, then the only ordering the filesystem has to go off of is the filename with the oX at the end (the "offset", as you said).

Just as a note, I haven't done thorough testing on all cases of the reordered gallery, so I haven't proven the ordering comes out like that in all cases.
Assuming it does, then I expect it'd be pretty easy to put a counter on the loop that downloads photos in posts.

Ah, sorry, I was using several different posts for testing and got confused about the outputs. I didn't notice the ID parameter was being used for the default - I thought I was getting a short form of the name parameter at the end of the default filename.

reversebreak picture reversebreak on 7 Apr 2018

Another assumption that turned out to be false ... I assumed the _o1, _o2, ... parts of image URLs were assigned by the order in which they are returned by Tumblr's API. Instead these "offsets" are determined by upload order and the API returns images as the blog owner has arranged them, which can be completely arbitrary.

Here is a test-post if you want a simple example for that: https://mikf123.tumblr.com/post/172687798174/photo-order-test . Images where uploaded 1 to 4 and I re-arranged them afterwards. gallery-dl will use "o1" for image number 4, which also has a o4 in its URL, and so on.

The offset value from gallery-dl is just a simple counter that goes up by 1 for every item returned, which would work if images always stayed in their original order and it also functions as a generic solution for audio and video files which don't have an oX part.

https://github.com/mikf/gallery-dl/commit/4a26ae32df4cadee82959ae390a815dd7685c07e adds an option which, if enabled, sorts photos of a photoset by the oX parts in their URL. This should assign the correct offset value for each image (except if one is deleted ...) and have them in the right order. (The default value is false for backward compatibility's sake)

Another thing that might be useful is the "middle-part" of a Tumblr filename, which is now available for format strings as hash (https://github.com/mikf/gallery-dl/commit/6b72be8ee6d66259c28367dc6d963ac6c9988fec).
The post ID (and not something like this middle-part) is used in the default filename because of the order when viewing these files in a file manager or image viewer. Post IDs are nice increasing numbers while these hash values are basically random.

(... and if any of you have better names for sort or hash, please let me know. I'm terrible at naming things)

mikf picture mikf on 7 Apr 2018

(... and if any of you have better names for sort or hash, please let me know. I'm terrible at naming things)

No, I think the names are fine. My remark was just that _"offset"_ is used in two different contexts in tumblr.py, I wanted to just mention this as a possible caveat, to prevent misunderstandings. But both are used (and named) correctly, I'd say. The Tumblr API Doc explicitly mentions "offset" as the parameter name for the /posts endpoint. As for the other "offset" (from within a single post, not offset between multiple posts), I don't really know, this could just be named {object-..} [order|number|count], whatever, or just {order} 😄

And sorry for adding a bit of confusion myself here:
The {id} used in the filename is created by gallery-dl, and that is the ID of the post, as mikf correctly pointed out. And this is what you want to use, because the order of downloaded files in the filesystem is now the same as the order of the posts on the blog as displayed via browser.

The filename used/generated by Tumblr itself is something entirely different, and that number (before the oN) is, as mentioned above, simply some sort of hash value based on the file. So @reversebreak can now use {hash} for the output of the files handled by gallery-dl, yes. But, to be honest, I don't know why someone would use this. I can't think of any possible use case right now. Because it is just a random hash value. Useful for dumping many files into the same directory, sure, but gallery-dl could already handle this before, because post ID + "object-offset" is already unique. But if @reversebreak wants to use it, sure, why not.

https://github.com/mikf/gallery-dl/blob/4a26ae32df4cadee82959ae390a815dd7685c07e/gallery_dl/extractor/tumblr.py#L98-L101

Do I get this here right? The default value being false,
https://github.com/mikf/gallery-dl/blob/4a26ae32df4cadee82959ae390a815dd7685c07e/gallery_dl/extractor/tumblr.py#L66

unless some user explicitly sets this option, the sort function (together with _get_tumblr_offset() never gets used.

So, what this sort here actually does, is sorting the URLs to be processed in the order of the filename as generated by Tumblr ("filename-offset"), right?

So, in conclusion, without calling this function, gallery-dl already uses the order as returned by the API - unless I'm totally blind and miss some other change in that commit?

But that already is the order that we actually want, right? The description in configuration.rst:

+ Otherwise these photos will be returned in the order the blog owner
+ arranged them in.

I mean, yes, that is the preferred order we want. Again, I don't understand the use case? Why would you _not_ want the order the blog owner arranged them in, and instead the order of the upload? I doubt that anyone does this upload in a specific order. They just dump some files on Tumblr, and they generate the "filename-offset", just to keep the filenames unique. But there is not any planned order here, or am I totally confused now? The planned order is the one as arranged by the blog owner, why would anyone want to use another order?

But the arranged order is obviously relevant, and @reversebreak correctly noticed this and made a good request here. I simply never noticed this before, although I've probably downloaded thousands of photosets. But apparently not one comic (or anything else where I immediately would notice the order), so this is why I did not stumble upon this yet.

Small additional note here: I think that I remember having both a browser open with a certain Tumblr blog and an image viewer at the same time, but the image viewer was set to sort/display by last modification date, so this would explain why I did not observe this behaviour yet.
This would also be a quick workaround for @reversebreak , just display by modification date.
But don't pin me down on this, basically just a recollection out of hazy memory.

Hrxn picture Hrxn on 7 Apr 2018

I mean, yes, that is the preferred order we want. Again, I don't understand the use case? Why would you not want the order the blog owner arranged them in, and instead the order of the upload? ...

gallery-dl has always just returned images in the same order the API returned them in, which is also the same as the order the owner arranged them in, but apparently that is the wrong order for some comics:

This will show the gallery in the wrong order, which is terrible for comic posts.

The sort option (as well as the hash value) are basically an attempt to allow users to sort images in the "original" order, which could solve @reversebreak's problem.

Do I get this here right? The default value being false, unless some user explicitly sets this option, the sort function (together with _get_tumblr_offset() never gets used.

As I said: "backward compatibility". This will most likely change if this option actually sticks, but only on bigger version updates. It affects which image gets assigned which offset value and that is not something that should just be altered.

edit: That is the same reason I don't want to rename {offset} right now. It would break existing configs. I guess I could add a bit of redundancy and the same value twice: one as (deprecated) offset and another one as {num} or whatever

mikf picture mikf on 7 Apr 2018

Also, I kind of missed this before:

Unusually, gallery-dl seems to download them in _o2, _o1, _o3, _o4 order, (according to the on-screen status)

That doesn't happen, ever. The oX values for default filenames are basically a loop counter, as you have called it before, and only increase.

mikf picture mikf on 7 Apr 2018

That doesn't happen, ever. The oX values for default filenames are basically a loop counter, as you have called it before, and only increase.

Ah, okay. I guess that would explain this misunderstanding. It has no further significance that this item number being ultimately used by gallery-dl is just a loop counter, and that the filename produced here resembles (at least at this part) the filename format used by Tumblr is just coincidence, more or less. Or your choice to follow the naming convention at least partially..

edit: That is the same reason I don't want to rename {offset} right now. It would break existing configs. I guess I could add a bit of redundancy and the same value twice: one as (deprecated) offset and another one as {num} or whatever

Yes, that is a very good reason to keep it, obviously. I'm not even suggesting to change it at all, just a thought to mention this potential "pitfall", just in case. Preemptive concern... I'm starting to regret that I've even brought it up in the first place.. 😄

gallery-dl has always just returned images in the same order the API returned them in, which is also the same as the order the owner arranged them in, but apparently that is the wrong order for some comics

Glad to hear, because the API basically has always returned the correct order, and gallery-dl kept that.
I think I see the culprit know, that this is some sort of edge case @reversebreak discovered where this all breaks down. By the way, at this point an example link to an affected post would be a good idea. Otherwise this can't be tested.
I put the blame on Tumblr's theming capabilities. The display of the blog can be customized pretty freely with HTML, CSS, etc. so this might be the cause why the actual order of the displayed elements has been rearranged _again_ here..

Hrxn picture Hrxn on 7 Apr 2018

Apologies, all. I think I may have led us all on a wild goose chase here.

I think this all stems from my assumption that the oX tag at the end of the default filename that gallery-dl creates when downloading tumblr galleries was the same as the oX that appears at the end of the name parameter. So in my efforts to introduce words to the filenames and not just IDs, my filename parameter was initially something like
{id}_{slug}_(name}.{extension}
However, that means that the resulting files would come out reflecting the tumblr root filenames. As a result of _that_, it meant that the resulting on-disk galleries would be in the wrong order as they are naturally sorted by the filename.
This also explains what I was seeing in the output logs with stuff coming out in _o2, _o1, _o3, _o4 order, as that reflects the tumblr name parameter.

I turned off all filename parameters, and just went with the default tonight to test it, and it actually comes out in the right order on-disk, even using a rearranged gallery as the source. The default filename comes out with o1 o2 o3 in the order they appear in the gallery, even though the source filenames are _o2, _o1, _o3.

When looking through -K to build the filename string I saw offset in that list - but there wasn't any indication that that meant "this is the index number of a picture in the gallery in a particular post".
I looked at the tumblr API and saw 'offset' there as a parameter to a search/filter and thought it was something to do with that.

So I guess in summary:

  • Tumblr images are named with an increasing numerical tag in upload order, o1 o2 o3 etc
  • Tumblr image galleries can be in an arbitrary order, completely independent of filename order
  • The Tumblr API returns images in gallery order, not filename order
  • gallery-dl default filenames are tagged with a label on the end o1 o2 o3 etc. This number is completely independent of the o1 o2 o3 that may be on the tumblr filename, and is instead dependent on the order of the images returned by the API. gallery-dl is actually working perfectly fine by default.
  • It is very much not obvious what the offset keyword does when using -K, leading to the only obvious sources of numbering for someone constructing an altered filename the name or filename keywords. offset sharing a name with a real tumblr API parameter doesn't help

So in reality, the only thing that needs fixing out of all this is making it clear what offset does to a casual user of the program. Either rename it to something like gallery_index or decorate the output from -K to provide an explanation for it (and maybe other parameters). Make it clear that if you want gallery order in your filenames that that's the parameter to include.

reversebreak picture reversebreak on 9 Apr 2018
👍1

Thanks for investigating and reporting back; my apologies for the confusing offset name and default filename structure.

The intent was to somewhat mimic Tumblr's filenames by adding a oX at the end of every downloaded filename, which was supposed to correspond to the oX index assigned by Tumblr. Turns out this doesn't work, since images can be re-ordered, and just causes confusion. I will renameoffset and assign a better default filename format in 1.4.0 .

As for

decorate the output from -K to provide an explanation for it (and maybe other parameters)

It would be a major undertaking to add explanations for all keywords (if you start with one, you might as well explain them all). as their are just too many. These values are _usually_ named in a self-explanatory way or come straight from a site's API results, which would then be explained by their API docs.

mikf picture mikf on 9 Apr 2018
👍1

Thank you. :smiley:
I only put in the decoration suggestion as an option - if you're going to change the name of the keyword, then that solves the problem too.

Now, as for something different:
I'm trying to get the subset of posts from my mega list of URLs that are reblogs from another post.
I try calling
gallery-dl -i .\targets.txt -g --filter "reblogged==True"
(in PowerShell) and I do get a list of URLs being output. Yet when I copy a couple of URLs from the output list, they're not reblogs at all!
I'm not sure whether I'm using the wrong parameter, or whether I've done something wrong in the filter expression (having no experience with Python myself).
Can you see what I'm doing wrong here?

reversebreak picture reversebreak on 10 Apr 2018

Hmm, I'm not sure why this doesn't work. Your filter is fine and should only let images from reblogged posts through. (You could even shorten it to --filter "reblogged", but that is besides the point)

Information about a post being a reblog is, again, coming from Tumblr's API, although the reblogged parameter is generated by gallery-dl itself:
https://github.com/mikf/gallery-dl/blob/a1fa4b43b07f09a56feb916e02a923522bf0ae20/gallery_dl/extractor/tumblr.py#L83-L86
reblogged is only set to True if the reblogged_from_id key is available, which _should_ only happen for reblogs.

I tested this again on my few test posts and it works as it should:

# original post
$ gallery-dl -K https://mikf123.tumblr.com/post/167623548569/
...
reblogged
  False

# reblog of the above post
$ gallery-dl -K https://mikf123.tumblr.com/post/169341068404/
...
reblogged
  True
reblogged_from_id
  167623548569

Maybe you could post one of the posts/URLs which gets categorized as reblog even though it isn't?

mikf picture mikf on 10 Apr 2018

This one is quite weird.
Using
http://aruurara.tumblr.com/post/172577442134
as an example, if you call
gallery-dl http://aruurara.tumblr.com/post/172577442134 -K
the response list will list it as having reblogged as false, as is expected and correct.

Put that same URL in a text file as the only contents, named say single.txt, and call
gallery-dl -i .\single.txt -g --filter "reblogged==True"
and you get blank output, as you expect.

However, using a file in which that URL is only one of the many URLs in that file, it shows up in the output!
For example, I've created a minimal testcase file multi.txt which contains two URLs, both of which respond as false for reblogged when queried individually. Get it at https://pastebin.com/PbAAMZJr
Call
gallery-dl -i .\multi.txt -g --filter "reblogged==True"
and you get both URLs as output.

I'm not sure if there's a case where you have multiple input from a file and non-reblog posts don't show up - I haven't found it yet.

reversebreak picture reversebreak on 10 Apr 2018

Oh, you are confusing the progress indicator with the downloaded files. When there are 2 or more URLs as input, gallery-dl by default prints the input URL to show its progress in its input-list.
You can disable this feature in your config file or by using -o output.progress=false.

mikf picture mikf on 10 Apr 2018

And before another misunderstanding happens: the Tumblr extractor provides multiple URLs for each image and when you print them with -g you get something like this:

https://s3.amazonaws.com/data.tumblr.com/4085165f7ead78cb63de964cf48fb6cd/tumblr_p6rrvu8UHo1s3sz4ho1_raw.png
| https://s3.amazonaws.com/data.tumblr.com/4085165f7ead78cb63de964cf48fb6cd/tumblr_p6rrvu8UHo1s3sz4ho1_500.png
| https://78.media.tumblr.com/4085165f7ead78cb63de964cf48fb6cd/tumblr_p6rrvu8UHo1s3sz4ho1_1280.png

Only the first one (without a | in front) is the "main" URL and the other two are its fallback, in case the main URL doesn't work.

mikf picture mikf on 10 Apr 2018

Ah, I see.

I just saw that output whilst you wrote the second post.

Hmm, so the URLs it gives you aren't the URLs of the posts, it's the direct URLs of the images themselves... I guess I can't use -g for my purposes then.
I was trying to use the filters in combination with -g to get a list of the post URLs that are reblogs of something else.

Thanks for the info.

reversebreak picture reversebreak on 10 Apr 2018

Try --filter "reblogged and print(post_url)"

mikf picture mikf on 10 Apr 2018

Abusing side-effects for fun and profit!
gallery-dl -i .\targets.txt -g --filter "reblogged and print(post_url)" -o output.progress=false works! Also supresses the image URL output for some reason!?!?? (Which is exactly what I wanted to happen, just not what I expected to happen)

reversebreak picture reversebreak on 10 Apr 2018

Images URLs are suppressed because the filter expression always evaluates to False.
Either reblogged is false and the whole thing is considered false without touching the print() part
or reblogged is true and print() gets evaluated, which returns None which is also considered false.

mikf picture mikf on 10 Apr 2018

That makes sense.

I'm hitting some sort of API limit, which seems odd as with the filtering I should be pulling down less than my real mass-download. I downloaded a few thousand pictures a couple of days ago, but after a few hundred requests using
gallery-dl -i .\targets.txt -g --filter "reblogged and print(post_url)" -o output.progress=false
I get
[tumblr][error] {'meta': {'status': 429, 'msg': 'Limit Exceeded'}, 'response': [], 'errors': [{'title': 'Limit Exceeded', 'detail': 'It did not work'}]}
repeatedly
Does the gallery-dl API key have a global limit, or is it associated with my tumblr account? Should I apply for my own OAuth key? And how do I see how long I have to wait anyway?

Actually - now that I think about it, not having to download the image files in between requests may really increase the rate at which gallery-dl hits the server for information.
There's a downloader.rate setting, but that's number of bytes a second, not a limit on query thoroughput...

In addition to that, the same post is being printed multiple times in the output (sometimes). Not sure why that's happening either.

reversebreak picture reversebreak on 11 Apr 2018

Tumblr's API has a global, application-wide limit of 1000 API requests per hour / 5000 per day (There was already some discussion about this on another issue). Image downloads don't count towards the limit, only how often gallery-dl accesses the API endpoints to get its information. This happens before any filter can get applied, so filtering out some posts sadly doesn't accomplish anything in that regard.

Should I apply for my own OAuth key?

Yes, you should. Especially in your case, when you want to access a couple thousand single posts, each requiring 1 request. There are instructions here, if you need any.

And how do I see how long I have to wait anyway?

I just took another look at the response headers sent by Tumblr and there is actually some information there:

X-Ratelimit-Perday-Limit 5000
X-Ratelimit-Perday-Remaining 4997
X-Ratelimit-Perday-Reset 86256
X-Ratelimit-Perhour-Limit 1000
X-Ratelimit-Perhour-Remaining 997
X-Ratelimit-Perhour-Reset 3456

I guess I could add a sleep() call to wait for the hourly limit ...

In addition to that, the same post is being printed multiple times in the output (sometimes). Not sure why that's happening either.

Some posts have multiple images and the filter expression is evaluated for each of them, resulting in the same post-url printed once for each image. You could pipe the output through uniq, but that (or anything similar) is probably not available to you.

mikf picture mikf on 12 Apr 2018

Right, I didn't realise I was stepping on all other gallery-dl user's toes.

I've registered a key myself, but apparently I'm using up all of my own allocation on a single run of this too.
A sleep call when you're going to hit the limit would be great for anyone running a batch job.
Maybe if you're running off the default shared API keys, it should sleep in increasing increments each time, like how ethernet works?
I'm not sure how the API ratelimit 'refills', whether all at once after an hour or dripping a minute's worth at a time.
Some way of determining what your X-Ratelimit-BLAH current values are would be great too. Right now there's no way to tell.
On top of that, maybe add a generic message if an extractor is using the default key values that come with gallery-dl and the user hits a limit, reminding the user to get the appropriate API key for that extractor.

There's an equivalent pipeable command to uniq in PowerShell, so I'm OK fixing up the output afterwards. I might just use gVIM anyway.

reversebreak picture reversebreak on 13 Apr 2018

I see you've added a sleep call to the newest release - thank you.

The only thing I'd recommend is altering the message it produces when the limit is hit - put in a reference to the current time in the error log call. That way if someone walks away from their computer and sees
[tumblr][info] Hourly API rate limit exceeded; waiting 1000 seconds for rate limit reset
upon coming back they don't wonder "1000 seconds from when?".

reversebreak picture reversebreak on 25 Apr 2018

Done (8b79eaa). I've changed it so it shows the time when the limit will reset. Seems more useful than manually having to add 1000 seconds to the time the message was printed at.

mikf picture mikf on 25 Apr 2018

Is there a way to download by search terms instead of by user on deviantart? When I try "gallery-dl https://www.deviantart.com/popular-all-time/?q=alm+fire+emblem" I get "[gallery-dl][error] No suitable extractor found for 'https://deviantart.com/popular-all-time/?q=alm+fire+emblem'"

jeremiahfallin picture jeremiahfallin on 8 May 2018

Downloading by search terms is currently not supported.
DeviantArt provides an API endpoint for that, so implementing this should be quite easy.

Edit: done (ec15877)

mikf picture mikf on 8 May 2018
👍1

Would it be possible to implement downloading by search terms on pixiv as well?

jeremiahfallin picture jeremiahfallin on 12 May 2018

Asking here as I wouldn't know if opening up a whole new issue for this would be appropriate, but I'm having issue authenticating with Pixiv. I try my user ID and my email, a combination of different command lines listed, and I continue to get:

[pixiv][error] Authentication failed. Please provide a valid username/password pair.

I've even tried with multiple different accounts.

gallery-dl -u <username> -p <password> URL is the main command I've been trying, but I've tried all I could find, none work.

nikoplymouth picture nikoplymouth on 24 Jun 2018

Maybe your user-id or email or password contain some special characters and get somehow modified by your shell? Try putting these values in double quotes (or single quotes if you are not using Windows): gallery-dl -u "<username>" -p "<password>" URL

But, since you have tried with multiple accounts, I kind of doubt that this is all there is ...
I've also created a test account to see if everything works as it should (at least on my end), and it does:

$ gallery-dl -u [email protected] -p qweasd https://www.pixiv.net/member_illust.php?id=173530
[pixiv][info] Logging in as [email protected]
...

Try using this account and see if at least that one works for you.
(pixiv ID for that account is user_pmhg4438 and can be used instead of the email address)

mikf picture mikf on 24 Jun 2018

The other account you listed here did work correctly as is, but putting my user and password in quotes also fixed it for my accounts. Weird, but at least I was able to get it working. Thanks!

nikoplymouth picture nikoplymouth on 24 Jun 2018

One suggestion that I have is to add support for sites with a non-tumblr URL but hosted by Tumblr. Sites can use their own custom URL with Tumblr, but this can actually be detected by downloading the page HTML and looking at the header/meta tags etc. Tumblr includes a unique meta tag there, even for sites that don't have Tumblr in the URL. Then you can feed their data into the Tumblr API as usual.

reversebreak picture reversebreak on 8 Jul 2018

^ I think I can give my personal favorite answer here: Feature is already included!

I think you mean this, right? #71

The example URL mentioned in this issue: http://www.b-authentique.com/.

You need to tell gallery-dl to use the Tumblr extractor here by writing the URL argument like this: tumblr:http://www.b-authentique.com/.

Or try your own example URL like tumblr:www.your-url-here.net

Hrxn picture Hrxn on 8 Jul 2018

Ah, that's useful!
Does that work in a -a file input?

reversebreak picture reversebreak on 8 Jul 2018

If you mean file input, i.e. like -i --input-file, (does -a work as well?), then yes, it does work!

Hrxn picture Hrxn on 8 Jul 2018

Sorry, I got my downloaders confused. -a is file input for youtube-dl. Whoops.
Good to hear that though!

reversebreak picture reversebreak on 8 Jul 2018

in booru board (danboo,gel,sankaku,...) can it support "results" in folder name ?

e.g : [booruboard]/?tags=aaaaaaaa , found 355 results with this tags, it will write number of results found in folder name if we enable it.
it's like
"directory":["XXXX","{search_tags} {number of results found}"]

thank

wankio picture wankio on 12 Aug 2018

Something to watch out for - I'm seeing multiple reports on Tumblr about the URLs not working sometimes for raw images - not sure of how prevalent it is. I haven't done a mass download myself in a while.

The Tumblr downloader may need to be adjusted.

reversebreak picture reversebreak on 13 Aug 2018

@wankio with the way gallery-dl currently gets its data from booru-sites, that's not possible.Wouldn't it work to count the files in a directory after all downloads have finished and rename it afterwards (not manually obviously, but with a small script)?

@reversebreak Yeah, I've heard about that. It seems original image downloads from Tumblr have been disabled for the time being, maybe permanently. The Tumblr module does still "work", but the first few attempts to download an image will fail, until gallery-dl tries the last fallback URL.

... sometimes ... not sure of how prevalent it is

From what I've read so far, nothing seems to work anymore except the re-compressed 1280p versions hosted on Tumblr itself.

mikf picture mikf on 13 Aug 2018

oh so that, i must use another script to get all results count for each tags before i use gallery-dl to batch download it :D

can gallery-dl have option to download all "More like this" in each pin link ? https://www.pinterest.com/pin/657314508089801717/

thank

wankio picture wankio on 15 Aug 2018

You can now download related ("More like this") pins by adding a #related to a Pinterest URL (https://github.com/mikf/gallery-dl/commit/63fa0b2006bd2796e4056ba9056b24b0a945efa6): https://www.pinterest.com/pin/657314508089801717/#related

After 200 or so items the same pins seem to repeat over and over again, which causes some weird issues where it kind of gets stuck in an infinite loop. For right now I would recommend to limit the amount of related pins with --range 1-195 or something.

mikf picture mikf on 15 Aug 2018

thank you, i will test it 👍

update : related pins seem like limited at 700 pins, i just ran one related link and it downloaded 407 pins(re-run several times) and some pins have difference url

Dont have id after /pin/
https://www.pinterest.com/pin/AXDNpyEJui4avdYqocFJayKFJdVMAVk-vfi60ytXWn9GUBGVTa13nYI/

Same Image but difference url
https://www.pinterest.com/pin/477944579195865111/
https://www.pinterest.co.uk/pin/327988785347464385/

with this extension i must run serveral times to get nearly full related pins (before it get Forbidden AccessDenied ...so i think gallery-dl have loop problem once it reach Forbidden-accessdenied content

https://chrome.google.com/webstore/detail/pindown-free/hojoabneemdhaakgfkmiaeafjjlcaebp

wankio picture wankio on 16 Aug 2018

Hello, I was wondering if there was an option to download just the first x pages of a DeviantArt gallery? Some of the favourites sections can take hours to trawl fully when I only need the recently added submissions. The "abort-on-skip" isn't useful if I want to download the dozen images after the skipped one. Any suggestions?

AdmiralHaddock picture AdmiralHaddock on 18 Aug 2018

You can use --range 1-50 to stop after the first 50 images, for example.

mikf picture mikf on 18 Aug 2018

It works! Thank you.

AdmiralHaddock picture AdmiralHaddock on 18 Aug 2018

How about adding wallhaven support?

AnyByte picture AnyByte on 14 Sep 2018

Also I suggest to add -vf scale=trunc(iw/2)*2:trunc(ih/2)*2 in gallery-dl/gallery_dl/postprocessor/ugoira.py by default in order to avoid height/width not divisible by two error.

AnyByte picture AnyByte on 14 Sep 2018

I've never had a "not divisible by two" error happen, but that's probably because, according to one of the answers in your link, 4:4:4 chroma sub-sampling doesn't have this problem and that's what FFmpeg uses if no -pix_fmt is specified.

I don't think this scale-filter should be added by default, since this error only happens if a different -pix_fmt is set and it's the user's own responsibility to specify working command-line options for FFmpeg, but it would certainly be possible to add a "avoid-not-divisible-by-two" option.

mikf picture mikf on 14 Sep 2018

Try converting images with odd number in width or height and you will get this error without any flags, but yeah maybe you're right that this should remain optional as if user wanted to change output video scale this could result in conflict and would need additional handling.

For example with this command ffmpeg -r 1/5 -i %06d.jpg out.mp4 and image dimensions being 1000x843 im getting:

[libx264 @ 000002053424e380] height not divisible by 2 (1000x843)
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

And if using gallery-dl it just generates an empty mp4 file.

AnyByte picture AnyByte on 17 Sep 2018

I did, and works fine. Maybe it depends on the FFmpeg version?

$ gallery-dl --ignore-config --ugoira-conv https://danbooru.donmai.us/posts/3251265 
ffmpeg version n4.0.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 8.2.0 (GCC)
...
Output #0, webm, to './gallery-dl/danbooru/danbooru_3251265_61363f26c6265130594cbeb9a5d53c63.webm':
  Metadata:
    encoder         : Lavf58.12.100
    Stream #0:0: Video: vp9 (libvpx-vp9), yuv420p, 675x675 [SAR 1:1 DAR 1:1], q=-1--1, 200 kb/s, 8 fps, 1k tbn, 8 tbc
    Metadata:
      encoder         : Lavc58.18.100 libvpx-vp9
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
...
mikf picture mikf on 17 Sep 2018

yeah it works fine with --ugoira-conv but with this config:

"danbooru":
{
    "postprocessors":
    [{
        "name": "ugoira",
        "extension": "mp4",
        "keep-files": true,
        "ffmpeg-output": false
    }]
},

it doesnt

AnyByte picture AnyByte on 17 Sep 2018

can pixiv extractor support illust_id ?
www.pixiv.net/member_illust.php?mode=medium&illust_id=xxxxxxxx ty

wankio picture wankio on 19 Sep 2018

That has already been supported since 2a97296e699442843152a46d47bbdb0a5c6234af.
You need to put the URL between double quotation marks "..." because it contains an ampersand &. That's probably why it doesn't work for you.

mikf picture mikf on 19 Sep 2018

oh thank, i dont even know it already supproted

but if people not ask, how they know to use it "..." when trying to download single pixiv post ? did we missing guide sector? like filter to ignore something, use --range to download from specific pics,...

wankio picture wankio on 19 Sep 2018

Not sure what you mean?

Using quotation marks for URLs is standard practice for any shell, basically. One should always use it.

And what about a single post? gallery-dl always works in the same fashion, if you browse Pixiv, or any other site for that matter, you just click on that posting you want and the URL to use for gallery-dl is what is displayed in the browser's address bar (You can also right-click on that link element while surfing and copy the link from there, of course).

Hrxn picture Hrxn on 20 Sep 2018

Hello again... Maybe I'll ask a stupid question now, but does the blacklist in "extract.recursive.blacklist" work only for this extractor or can it be used for deviantart for example? And if it does not work, is it planned for a blacklist for keywords for extractors?

ChiChi32 picture ChiChi32 on 24 Sep 2018

This is not about blacklisting keywords, this is only to prevent recursive usage of extractors specified there..

Hrxn picture Hrxn on 24 Sep 2018

@ChiChi32
I believe what you are looking for is the --filter command-line option.

@wankio
--filter and --range are explained in the output of gallery-dl --help, although a few more examples in the README wouldn't hurt, I guess.

mikf picture mikf on 24 Sep 2018

@mikf I know about this option (although I admit I forgot about it), but ... 1. The command line is inconvenient if you need to filter out a few words. 2. All sites will be filtered, which is not very suitable.
And this is not criticism, but in some cases the documentation is very fragmented. About -filter and its options many probably do not even guess if you did not use --help.

ChiChi32 picture ChiChi32 on 24 Sep 2018

on deviantart, how i can disable download zip file when i'm enabled "original": true , thank :) ..i only saw filter with image

wankio picture wankio on 5 Oct 2018

Not really possible, I think.. If the actual linked "original" is indeed a ZIP archive file, this is what you're supposed to get.

I'm not surprised to see this, I've seen all kinds of different file formats uploaded on DA.

But if you could provide an example link to such an Deviation entry, it sure would help 😄

Hrxn picture Hrxn on 5 Oct 2018

https://www.deviantart.com/oofiloo/art/GF-PROMETHEUS-NO-ARMOR-ORIGINAL-BONES-472067552
if it found zip or other extensions, download both zip,ex... + image or only image

i need to run it twice with and w/o original to download full sample/image, it's not a problem but i think it should have a filter, if original is zip or other, download preview image instead

wankio picture wankio on 5 Oct 2018

Pretty nice program, though i have come across a few minor issues

In the json output from --dump-json it uses numbers for big integers (twitters tweet_id and such)
which is a bit of a problem because json implementations are not required to precisely handle such large numbers, the program jq for example would output different values
having it output big ints as strings would be preferable.

another issue ive had is extractor.pixiv.directory (and perhaps other sites)
does not have the same keywords available for use in the format string as filename does.
which makes organizing a bit more difficult, post processing could work but its probably not as reliable.

is there a way to save the metadata alongside the downloaded images, and if not perhaps a extractor.*.metadata which works like filename would be nice.
like before, this could probably be done manually with post processing i guess.

taiyu-len picture taiyu-len on 7 Oct 2018

@taiyu-len

having it output big ints as strings would be preferable.

output.num-to-str (https://github.com/mikf/gallery-dl/commit/48a8717a7c988f49f457be31557fa72d94d9efb4), but it converts all numeric values to strings. Would it be preferable to only convert integers > 2**52 and < -2**52, i.e. anything a double can't handle without losing information, or is it OK like this?

does not have the same keywords available for use in the format string as filename does.

That's by design, because in most cases the extra file-specific metadata is only its filename and extension from the download URL and maybe its width and height, so nothing you would want/need to create an extra directory for. Instead of re-evaluating the entire format string and calling makedirs() for each file, I wanted to have it only done once and then put every file into that one target directory.

is there a way to save the metadata alongside the downloaded images

Not yet, but I'm working on something.

mikf picture mikf on 9 Oct 2018

@mikf

is it OK like this?

should work well, can always convert strings to numbers if theres ever a need for doing operations on the values.

That's by design, because in most cases the extra file-specific metadata is only its filename and extension from the download URL and maybe its width and height, so nothing you would want/need to create an extra directory for.

in my case for pixiv files, i use the illust_id and title for subdirectories, which is useful for multi image galleries.
checking the source, the keywords for the directory come from this section, but only exposes the user section.

for the PixivWorkExtractor (havent looked at bookmark/favorite extractor yet)
this change would make the rest of the Illust object available for use in the directory naming,
though it does include some other junk like urls. but the original version does too with the profile picture url.
could filter it out, or just leave it up to the user whether they want to include page_views or whatever dumb things in the directory name.

diff --git a/gallery_dl/extractor/pixiv.py b/gallery_dl/extractor/pixiv.py
index 0005f92..8491bf7 100644
--- a/gallery_dl/extractor/pixiv.py
+++ b/gallery_dl/extractor/pixiv.py
@@ -214,7 +214,7 @@ class PixivWorkExtractor(PixivExtractor):

     def get_metadata(self, user=None):
         self.work = self.api.illust_detail(self.illust_id)
-        return PixivExtractor.get_metadata(self, self.work["user"])
+        return self.work;

also pixiv sending me an email everytime i use this is a bit annoying,
pixivs help says it should recognize the connection after the first time, but it doesnt seem to be doing that. maybe i need to use it more.

taiyu-len picture taiyu-len on 11 Oct 2018

Hi, I have a question to the config-file / filter. I want to exclude tags on pixiv and maybe other sides.

After running the gallery-dl -K option, I get the tags[] array with uXXXX char-codes and when i try to run e.g:
gallery-dl --filter "tags[-1] != uXXXXuXYYY..." URL (the ignored tag was the last in the array)
I get the warning:
_[gallery-dl][warning] unexpected character after line continuation character (image filter, line 1)_
and the ignored files are still downloaded

I tried other tags[] related queries but none worked. Can you please explain what I'm missing or is it a bug? Anyway thanks in advance

Edit: I added one tag to the Mute-list on pixiv and the is_muted attribute change to true and the --filter "is_muted == False" ignored the muted files, but in the config-file the "image-filter":"is_muted == False" statement is still not working

AKluge picture AKluge on 15 Oct 2018

@AKluge
You are missing quotation marks around the string you want to test against.
It should be "tags[-1] != '\uXXXX\uXYYY...'" , i.e. single quotes around the \uXXXX\uXYYY part.

If you want to test if a string is inside a list, you can use the in operator: "'foo' in tags"

--filter "is_muted == False" ignored the muted files, but in the config-file the "image-filter":"is_muted == False" statement is still not working

The command-line option does exactly the same as the image-filter config option (*), so maybe you did something wrong here, but what you posted looks alright.

I've been using this configuration to test and it works just fine:

{ "extractor": {
    "pixiv": {
        "image-filter": "is_muted == False",
        "username": "foo",
        "password": "bar"
    }
} }

I would also recommend "not is_muted" to test if something is false. This would also work if is_muted is None or an empty string/list.

(*) it effectively sets a global value for image-filter and therefore overwrites all image-filter entries from a config file

@taiyu-len

also pixiv sending me an email everytime i use this is a bit annoying,

Shouldn't happen anymore: https://github.com/mikf/gallery-dl/commit/8faf03ed847ae1cf2bb19d04f678204ed49c2e24
I've never noticed this myself, since all email addresses for my Pixiv accounts are (now) invalid.

Concerning directory metadata, you can put a yield Message.Directory, work before any yield Message.Url, url, work to "fix" it, like so:

diff --git a/gallery_dl/extractor/pixiv.py b/gallery_dl/extractor/pixiv.py
index 115b1fb..8716a2d 100644
--- a/gallery_dl/extractor/pixiv.py
+++ b/gallery_dl/extractor/pixiv.py
@@ -31,7 +31,6 @@ class PixivExtractor(Extractor):
         metadata = self.get_metadata()

         yield Message.Version, 1
-        yield Message.Directory, metadata

@@ -55,11 +54,13 @@ class PixivExtractor(Extractor):
                     "_ugoira600x600", "_ugoira1920x1080")
                 work["frames"] = ugoira["frames"]
                 work["extension"] = "zip"
+                yield Message.Directory, work
                 yield Message.Url, url, work

             elif work["page_count"] == 1:
                 url = meta_single_page["original_image_url"]
                 work["extension"] = url.rpartition(".")[2]
+                yield Message.Directory, work
                 yield Message.Url, url, work

@@ -67,6 +68,7 @@ class PixivExtractor(Extractor):
                     url = img["image_urls"]["original"]
                     work["num"] = "_p{:02}".format(num)
                     work["extension"] = url.rpartition(".")[2]
+                    yield Message.Directory, work
                     yield Message.Url, url, work

But keep in mind that, as for right now, gallery-dl isn't optimized for this kind of thing. Usually general metadata for directories and specialized metadata for filenames is more than enough.

@wankio
You can now set deviantart.original to "images" to only download original image files and fall back to the preview version otherwise (https://github.com/mikf/gallery-dl/commit/d8492df51b40ff96f3dae51fa42b8cc89a4135e8)

mikf picture mikf on 16 Oct 2018
👍2

sankaku: HTTP request failed: ('Connection broken: OSError("(10054, \'WSAECONNRESET\')")', OSError("(10054, 'WSAECONNRESET')"))

why this happen ? i thought it must re-download 10 times before timeout ?? ty

wankio picture wankio on 17 Nov 2018

Seems like Sankaku unexpectedly aborted your connection to its servers and the underlying requests- and urllib3-libraries reported it as a weird/unrecognized exception, so gallery-dl stopped its extraction.

It does retry a HTTP request (up to 10 times for Sankaku) if the error is reported as ConnectionError or Timeout, but this appears to be a Windows specific "connection reset" error I havn't seen before. In the meantime just retry the same URL and hope it doesn't happen again ...

mikf picture mikf on 18 Nov 2018

I have ffmpeg on my path, but the pixiv downloader still just downloads ugoira as ZIP and leaves them like that. I've looked through the options but can't figure out how to make them convert.
Could you assist?

reversebreak picture reversebreak on 29 Nov 2018

You need to enable the ugoira post-processor module.

One way, to see if it works, is to use --ugoira-conv:

$ gallery-dl --ugoira-conv "https://www.pixiv.net/member_illust.php?mode=medium&illust_id=71828129"

If everything works as it should, you may want to set a permanent option with better encoding settings in your config file. A working example looks something like this:

{
    "extractor":
    {
        "postprocessors": [{
            "name": "ugoira",
            "whitelist": ["pixiv", "danbooru"],
            "extension": "webm",
            "ffmpeg-twopass": true,
            "ffmpeg-output": true,
            "ffmpeg-args": ["-c:v", "libvpx-vp9", "-an", "-b:v", "0", "-crf", "30"]
        }]
    }
}

See also postprocessors and ugoira options
There are also post-processor examples in docs/gallery-dl-example.conf

mikf picture mikf on 29 Nov 2018
❤1

How i can download specific months in tumblr.com/archive ?

Because when i'm using gallery-dl on big tumblr (3-10years old) 100% it will reached api limit, if i change api i must rerun it from start or from --range and it will get limit too.

TY

wankio picture wankio on 14 Dec 2018

@wankio I don't think it is really possible to request posts from a specific month or time-frame using Tumblr's API (correct me if I'm wrong).
Aren't 5000 posts, or 25000 posts when using the total daily limit, enough, assuming you are using your own API key?
Maybe you could try only requesting one specific post type at a time, i.e. only photo posts on one day, text posts the next day, etc., to space out the total amount of API calls over a couple of days.

mikf picture mikf on 14 Dec 2018

yes, my own api key. now i'm using tumbex to get post link auto.

And gallery-dl can do same "--mirror" as wget ?
TY

wankio picture wankio on 15 Dec 2018

And gallery-dl can do same "--mirror" as wget ?

No, it can't. There is recursive:<URL>, as you probably know, but that does something quite different. Just use wget itself (with the appropriate filters), if you need that sort of functionality.

mikf picture mikf on 1 Jan 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

indrakaw picture indrakaw  ·  4Comments
CaptainJawZ picture CaptainJawZ  ·  5Comments
rawesomeawesome picture rawesomeawesome  ·  3Comments
colpocleisis picture colpocleisis  ·  3Comments
OvertPerson picture OvertPerson  ·  5Comments