Gallery-dl: Deviantart Extractor - Image Resolution Issue

[biznizz]

Have just noticed a strange quirk using Deviantart extractor; while the ripper is great at getting full resolutions of older pictures, it seems to have some issues with newer ones.

Ex: https://www.deviantart.com/nivarra/art/COM-Carmelita-CanCan-798781565
https://www.deviantart.com/nivarra/art/COM-Kiss-Pounce-780320385

The full resolution of the first picture is 1200x1200, but the download res is 600x600. While using the ripper on the gallery, the ripped picture is the smaller 600x600.

The second picture is a few months older, with full res of 1500x995, and a download res of 800x531. The ripper getting this picture ripped it at the full res.

It looks like that after March 21, 2019 or so, the site was reconfigured in a way that ripping full res is not possible unless the artist sets it up so. Was just wondering if there was anything that could be done.

[wankio]

I think it can't if download is hidden by artist/uploader

[Hrxn]

So, basically like that "trick" with GIF files? This site is a mess...

[mikf]

I doubt there is a "trick" available this time. Until ID 790677560 it was possible to rewrite the URLs of non-downloadable deviations so they would yield a full resolution image (See 5ec55ec, 9587aea, and https://github.com/r888888888/danbooru/issues/4069), but that doesn't work for newer images and now we are stuck with re-encoded and small JPGs, unless someone finds a way ...

[Hrxn]

The full resolution of the first picture is 1200x1200, but the download res is 600x600. While using the ripper on the gallery, the ripped picture is the smaller 600x600.

But this implies that the 1200x1200 is accessible from the browser? Or not?

[biznizz]

But this implies that the 1200x1200 is accessible from the browser? Or not?

The way it works is that, the download button has two resolutions. One is the smaller resolution that you'd normally get by pressing the button (some times there isn't a button, but I have a browser script that forces the download button to show), and the other is the full resolution of the pic.

So the button would show "600x600 (1200x1200)".

For older pictures, the ripper would get the second, full resolution picture. But because DA changed their site configuration (again) a few months ago, the pictures posted after this would only get ripped at in the smaller resolution.

It was a pleasant surprise to see this ripper was capable of getting (older pictures') full resolution when that wasn't normally possible.

[biznizz]

Also, think I've noticed something. If a picture has a DA watermark on it and it's also shrunken to a smaller size, using the ripper will only scrape the small, watermarked pic. Don't know if this is always how it's been, but it's a strange thing I've noticed.

[Hrxn]

[..] some times there isn't a button, but I have a browser script that forces the download button to show [..]

What kind of browser script is this?

[biznizz]

What kind of browser script is this?

It was something I found on Greasyfork.

https://greasyfork.org/en/scripts/4679-ts-deviantart-download-link

[biznizz]

Well, just noticed another issue. This picture was uploaded a year ago, so it shouldn't have any issues unlike the new uploads this year: https://www.deviantart.com/transflame/art/I-want-to-be-HUMAN-again-765882538

Using gallery on it rips a picture that's 774x7740, but the normal download res is 1024x10240, and the full res is 1445x14550.

Is there a size limit to pics that gallery-dl can rip?

[Hrxn]

Well, the limit is definitely not with gallery-dl, or what do you mean?

[biznizz]

I mean, I'm not sure. I've used the ripper to get other very big pics off DA with no problem; and this is the first time that I've noticed this thing happening, where the ripper downloaded such a super large picture and it's been nearly reduced to half it's original resolution.

I've tried running the process on this pic several times, and it's the greatly reduced picture everytime. I've never touched the py file in the extractor folder, and there shouldn't be any issues with my config file that causes this.

"deviantart":
{
"refresh-token": "redacted",
"flat": true,
"folders": false,
"journals": "html",
"mature": true,
"metadata": false,
"original": true,
"quality": 100,
"extra": true,
"wait-min": 0

[mikf]

For older DeviantArt images that don't allow an original file download, gallery-dl rewrites their URL to get an "intermediary" version that, 99% of the time, is higher quality than before - except here. I'm guessing this has something to do with the image's aspect ratio and a height limit of the "intermediary" version, maybe?

To give a concrete example:
The image URL presented by DeviantArt is

https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/f7f803e5-c49e-4d9c-ba43-a8e7df90a07e/dcnziqy-dab1058a-fca5-4511-bd73-002789d58ccd.png/v1/fill/w_1024,h_10240,strp/i_want_to_be_human_again_by_transflame_dcnziqy-fullview.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTAyNDAiLCJwYXRoIjoiXC9mXC9mN2Y4MDNlNS1jNDllLTRkOWMtYmE0My1hOGU3ZGY5MGEwN2VcL2RjbnppcXktZGFiMTA1OGEtZmNhNS00NTExLWJkNzMtMDAyNzg5ZDU4Y2NkLnBuZyIsIndpZHRoIjoiPD0xMDI0In1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmltYWdlLm9wZXJhdGlvbnMiXX0.0a7GgC31lMDQmKusL9bsvz2JdjlpIxRs1N7J-STX1Mk

and gallery-dl transforms it to the following, hoping to get a better version of that image:

https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/intermediary/f/f7f803e5-c49e-4d9c-ba43-a8e7df90a07e/dcnziqy-dab1058a-fca5-4511-bd73-002789d58ccd.png

Is there a way to search by image resolution / aspect ratio on DeviantArt so I can figure out where the "cutoff" point is?

[ShapeShifter499]

I'm confused by
"quality": 100,

The man page suggests that setting means 100% compression which I think is a bad thing.

Does that setting mean 100 percent quality (as close as original as you can get) or 100 percent compression? I'd rather be downloading as close to original as possible.

[Hrxn]

What do you mean? This?

JPEG compression quality for newer images hosted on wixmp servers.

Yea, this is a quality setting, not compression level or something, so q=100 is the highest you can use for dA still returning an image.
This is all server-side, though, i.e. it is still somewhat compressed, obviously.

[ShapeShifter499]

@Hrxn I'm trying to download images I uploaded to my own account and I wanted to ensure I got the highest quality as I already sent. I'm only doing this because my computer had a hardware failure before I could backup and I wanted to make backups.

I have enabled oauth login

[Hrxn]

Ah, that is a good question. I am not sure if the original/raw files are publicly accessible via the site itself.
But somewhere in the profile settings should be an option to export the complete profile data.
I mean it's your data, after all. And if you still find a local file, you could compare checksums against a file downloaded from dA.

[ShapeShifter499]

@Hrxn I just said I had a hardware failure. If I could compare against a local file I would but I can't.

[Hrxn]

I'm aware.. but you maybe have some files saved to another data storage device (flash drive, disc, whatever), or send some files directly to a friend or something.

[ShapeShifter499]

@Hrxn only a small set of files. Most were one time downloads I managed to upload but lost the local copy for. Commissions or freebie art I got. I only was doing this for a couple of months. I got lazy on backup and organize. Then I went and spilled coffee all over my laptop killing the one location these files were on.

[mikf]

@ShapeShifter499 If you haven't disabled the Free Download option when uploading your images to DeviantArt, gallery-dl will download the original, unaltered files. quality doesn't apply in those cases. It only gets used for images where the original download has been disabled. (See #369)

[kattjevfel]

For what it's worth the second link (https://www.deviantart.com/nivarra/art/COM-Kiss-Pounce-780320385) now downloads the proper 1500x995 file, but the first link still downloads a 600x600 file.