Gallery-dl: Image corruption / distortion / artifacting

[jtara1]

edit: This could be considered an issue that the user is responsible for managing.

There's a possibility this issue is irrelevant to this module / program. After downloading a comic, I run a script that uses zipfile module to zip each image from each chapter into its own zip file separated by chapters.

The possible issue with gallery-dl is:

• I may send a KeyBoardInterupt or kill the process running the script to stop the downloading of a comic, then resume the same comic later.
• I think I may have turned on or off my VPN mid download which would be like disabling my internet connection mid download.

I've had similar issues w/ other image downloading scripts.

image corruption e.g.s:

edit: to confirm it's not the source that is corrupt (http://www.mangareader.net/feng-shen-ji/26/13)

[Hrxn]

How/when do you run your script exactly? Why is it possible to interrupt gallery-dl in the first place?

[Bfgeshka]

It happens due to gallery-dl sudden stop or losing connection. I guess it would be better if @mikf can add logic for deleting not completely downloaded file on exit.

[mikf]

I may send a KeyBoardInterupt or kill the process running the script to stop the downloading of a comic, then resume the same comic later.

A KeyboardInterrupt or any other exception raised during a download causes a partially downloaded file to be deleted, but killing the process obviously leaves the file in place. Installing some signal handlers could help against anything that isn't a SIGKILL, but then there is question of how this would work on Windows.

I think I may have turned on or off my VPN mid download which would be like disabling my internet connection mid download.

I don't know how Requests/urllib3 handles client-side network outages mid-download (exception? just silently closing/aborting the connection?), but a remote-server closing the connection preemptively gets silently ignored and could also lead to partially downloaded files, although that has never happened to me as far as I can tell.

I think there are three things that should be done here:

• Use .part files during the download and rename them upon completion
• Implement continuation of partial downloads
• Check if the file size matches the Content-Length header

[Bfgeshka]

By the way @mikf, add ability to set path for this .part file, so it can be stored in /tmp or in tmpfs mount or wherever user wants.

[mikf]

I've _finally_ managed to rewrite the downloader modules to at least implement those three features I listed above, which should hopefully solve any issues regarding incomplete downloads and image corruption.

I tested this quite thoroughly on my own machine and a Windows 7 VM, but it would be nice if someone could test this themselves and report any errors/problems and/or possible improvements.

[Bfgeshka]

This little thing was not critical, but bugging with routine of checking last file of interrupted download session.
Good one.