Gallery-dl: Cannot find the file specified on Pixiv

[earfluffy]

So gallery-dl in general has been working fine for me, but it fails when I try to use it on Pixiv and I'm not sure why.

I entered the URL like you normally would:
gallery-dl "https://www.pixiv.net/member_illust.php?id=1226647"
and it gave me this:

C:\Users\earfl>gallery-dl --verbose "https://www.pixiv.net/member_illust.php?id=1226647"
gallery-dl: Version 1.6.1
gallery-dl: Python 3.7.1 - Windows-10-10.0.17134-SP0
gallery-dl: requests 2.21.0 - urllib3 1.24.1
gallery-dl: Starting DownloadJob for 'https://www.pixiv.net/member_illust.php?id=1226647'
pixiv: Using PixivUserExtractor for 'https://www.pixiv.net/member_illust.php?id=1226647'
urllib3.connectionpool: Starting new HTTPS connection (1): app-api.pixiv.net:443
urllib3.connectionpool: https://app-api.pixiv.net:443 "GET /v1/user/detail?user_id=1226647 HTTP/1.1" 200 664
pixiv: Active postprocessor modules: [<gallery_dl.postprocessor.classify.ClassifyPP object at 0x03FAA610>, <gallery_dl.postprocessor.zip.ZipPP object at 0x03FAA630>, <gallery_dl.postprocessor.exec.ExecPP object at 0x03FD7290>]
urllib3.connectionpool: https://app-api.pixiv.net:443 "GET /v1/user/illusts?user_id=1226647 HTTP/1.1" 200 10217
urllib3.connectionpool: Starting new HTTPS connection (1): i.pximg.net:443
urllib3.connectionpool: https://i.pximg.net:443 "GET /img-original/img/2018/08/13/12/00/00/70171595_p0.jpg HTTP/1.1" 416 206
pixiv: Unable to download data: [WinError 2] The system cannot find the file specified

Any help would be much appreciated.

[Hrxn]

Usually some kind of error related to the filesystem. Could be caused by incorrect settings used inside your config, for example for the output paths, or post-processors or something..

Can you try a run with gallery-dl --ignore-config and see what happens?

[earfluffy]

So, I tried several things. I tried an --ignore-config -s run (with username and password, of course), a run with just -s and finally, a run with just --ignore-config.

The first one worked fine, though of course I didn't actually get any files, and a normal one also worked fine if I had done a -s run first, but only if I had done that (which I thought was weird). The one that actually worked _was_ the --ignore-config run, as you suspected.

I've checked my config.json, though, and I'm not sure where the error is. Ideas?

EDIT: I should note that I'm using gallery-dl on a laptop with an OS drive (C:) and a storage drive (D:). By default, gallery-dl stores downloaded files in C:\Users\earfl\gallery-dl\, but in config.json I have it set to store them in D:\gallery-dl\, which seems like it might cause the issue, even if only indirectly.

[Hrxn]

What do you mean by default?
Taken from the example (default) config file here:

{
    "extractor":
    {
        "base-directory": "./gallery-dl/",
        "postprocessors": null,
        "archive": null,
        "cookies": null,
        "proxy": null,
        "skip": true,
        "sleep": 0,
        "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0",

        [extractor options here etc.]
        ...

So as default, the output gets always written to a base-directory called gallery-dl which in turn gets created in whatever current working directory gallery-dl is being run (note the . at the beginning of the base-dir value string)

[earfluffy]

Right. I've been running it from the command line (or perhaps that's the only way to run it; I know less about this sort of thing than I think/pretend I do, sometimes), so ./gallery-dl/ is in my user folder, in this case earfl. Since I want to store the downloaded files on the much larger HDD (D:) instead of the SSD (C:) Windows is installed on, my config file looks like this (also, I used this one as a template, which may be the problem):

{
    "extractor":
    {
        "base-directory": "D:/gallery-dl/",
        "archive": "D:/gallery-dl/archive.sqlite3",

    [extractor options here]
    ...

I hope that clarifies the situation.

[Hrxn]

Well yeah, you're right about running it from the command line, but I think that is not the point here.
It's the difference between a relative and an absolute path, i.e. ./my-dir (or ~/my-dir) vs /my-dir, or for Windows something like D:/my-dir..
Also, this example gallery-dl.conf you linked is not the problem, it does not make any difference.

Okay, since authentication is required for Pixiv, you already have your credentials in gallery-dl.conf and this config file is correctly set up and working. So far, so good.

Now you want to collect all your downloads in a location with enough storage capacity etc., and that is exactly the reason why base-directory exists. I mean, I'm doing exactly the same for obvious reasons.

And if you have set your base-directory to an absolute path, like you did with D:/gallery-dl/, it does not matter where you are running gallery-dl from. Completely irrelevant. Unlike with the default settings, where your currently active directory (CWD) is relevant.

So you want that location to be D:\gallery-dl\. So, you have to use one of the two variants in your conf:

        "base-directory": "D:/gallery-dl/",
        "base-directory": "D:\\gallery-dl\\",

Which you already have, everything right so far.
The actual output directory is then determined by the extractor setting. For Pixiv, this means something like this:

 "pixiv":
        {
            "archive": "D:/gallery-dl/archive-pixiv.sqlite3",
            [If you want to use an extractor-specific archive file to keep track of downloads]
            "filename": "{user[id]}_{id}{num}.{extension}",
            "directory": ["{category}", "{user[id]} {user[account]}"],
            "username": "XXXXXXXXXXXX",
            "password": "XXXXXXXXXXXX",
            [...]

These are basically the default settings, and get used when running gallery-dl on a Pixiv profile URL like in your first post here.

"directory": ["{category}", "{user[id]} {user[account]}"] specifies the output directory inside of your base directory.

Here it's pretty simple, {category} is always the name of the extractor itself, so here it's simply pixiv.

So in conclusion, your output directory where the data gets written to looks now like this:
D:\gallery-dl\pixiv\

And there, for every profile a directory in the format of "{user[id]} {user[account]}"

[earfluffy]

Okay, cool. It seems like it's a filename issue, maybe. The file it originally failed to find above (i.e., my first post) was 70171595_p0.jpg at the specified URL, but the file that's actually there, if you go check, is called 70171595_p0_master1200.jpg and https://i.pximg.net/img-original/img/2018/08/13/12/00/00/70171595_p0.jpg gets you a 403 Forbidden error while https://i.pximg.net/img-master/img/2018/08/13/12/00/00/70171595_p0_master1200.jpg gets you a girl holding a cat.

Do you know what might be causing this? It seems like it must be something in the config file, since ignoring it worked, but I don't know what. It's not the "filename" option, right? Since that just builds the client-side name for the downloaded title and doesn't determine where the extractor looks (unless it does something else)?

EDIT: That's not the issue (or at least not the one causing the [WinError 2] The system cannot find the file specified error). I looked at another failed run and it was pointing at the correct URL but failed anyway.

[Hrxn]

Yeah, a filename error, like character encoding or something, would be different..

Not sure, is there any other program running, in the background or as a service or similar, that might interfere with filesystem access?

The only other thing I can think of right now is the specific Python installation on your system, something might be wrong here.

[mikf]

tl;dr: The exec post processor can't find the executable it is trying to run. Delete it (and maybe the other post processor entries as well) and it should work.

I initially thought it might have something to do with the classify and zip post processors used at the same time, but this worked fine, even on Windows. So a file didn't get moved prematurely and paths weren't changed in a weird way, as I suspected.

Then I tried it with gallery-dl-example.conf as template and, again, it worked just fine, until I tried it on Windows and got the same error as you did. Maybe some Windows specific file-access-shenanigans ...

I eventually installed the latest dev version and ran it with -v:

Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\gallery_dl\job.py", line 52, in run
    self.dispatch(msg)
  File "C:\Program Files\Python36\lib\site-packages\gallery_dl\job.py", line 96, in dispatch
    self.handle_url(url, kwds)
  File "C:\Program Files\Python36\lib\site-packages\gallery_dl\job.py", line 222, in handle_url
    pp.run(self.pathfmt)
  File "C:\Program Files\Python36\lib\site-packages\gallery_dl\postprocessor\exec.py", line 26, in run
    for arg in self.args
  File "C:\Program Files\Python36\lib\site-packages\gallery_dl\postprocessor\exec.py", line 30, in _exec
    retcode = subprocess.Popen(args).wait()
  File "C:\Program Files\Python36\lib\subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "C:\Program Files\Python36\lib\subprocess.py", line 990, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

And there you have it. The subprocess.Popen() call fails because there is no echo on Windows.

[earfluffy]

Ahh. I was wondering if it might be the post processor. Thanks for all the help!