Wp-calypso: Media Library: Large export files are difficult to download.

[cecoates]

Steps to reproduce

1. Use a site that has more than 2GB of media.
2. Visit https://wordpress.com/settings/export/
3. Click the "Download" button for "Export media library"
4. Wait for download to hit around 1.5GB or 2GB

What I expected

For the download to continue until completion.

What happened instead

Despite repeated attempts, download always times out somewhere between 1GB and 2GB.

Browser / OS version

Chrome Version 70.0.3538.54 (Official Build) beta (64-bit)
Firefox 65.0a1 (2018-10-23) (64-bit)
Pop!_OS 18.10

Screenshot / Video

Context / Source

The new https://github.com/Automattic/wp-calypso/pull/27581 feature is great. However, media libraries on WP.com can be quite large since our plans offer anywhere from 6GB to unlimited storage for media.

For that reason, it would be helpful if there was an option to pause/resume a download, or for downloads to be split up into manageable chunks.

This user, for example:

7486851-hc / 1514114-zen

Is unable to download their file without timing out. Myself and another HE have the same issues, as well.

[pauljacobson]

We have other reports of this issue, here:

• 1663418-zen
• 1599506-zen
• 1589967-zen

[joweber123]

More reports of this issue:

• 1923417-zen

[druesome]

I helped a user that is having the same issue. I could not download the file from my browser as it cuts off half-way, but had better luck using Jdownloader (a download manager). It might help to suggest that the user use a download manager software. Just copy the link to the export file and let the program download it.

[lancewillett]

This shouldn't be marked high priority is it's a not a common error, and there are workarounds as druesome mentioned.

However, we should look at it as an enhancement in the future; CC @Automattic/delta-samus and @bisko for the Samus importers backlog.

[formosattic]

2151981-zen
2165134-zen
2563964-zen
2814548-zen

[andfinally]

I've added this as a Trello to the Samus backlog.

[BrookeDot]

I forget where this was originally found but wget can also be used which is standard on all unix and unix-like system. IE:
wget --tries=0 --continue https://example.wordpress.com/media-export.php?to=1836&ts=1541489&key=db4dzxc3dcvxzw2bcd13e42197b76

[BrookeDot]

Ah here is is :) p7DVsv-5AH-p2 #comment-16744

[dcoleonline]

Also in 2719468-hc, but I suggested a download manager program. Their file was 5.9GB.

I'm not sure where else to put this, but it would be nice if we could break it into smaller files for them and give them links to download each one.

[pablinos]

Yes, you need a download manager for larger files. The export process does support restarting, so should the download break partway through, if the client downloading it issues a restart command, then it'll pick it where it left off. As mentioned wget is pretty much built for this purpose, but I'm sure there will be a number of browser extensions that would do this.

Perhaps in future, we should implement a 'Save to Google Drive' feature or something.

[dcoleonline]

Suggested a download manager in 3380709-zen

[mriyazuddin]

Suggested a download manager in 3380709-zen

User has tried https://jdownloader.org/jdownloader2 and that seems to have worked well for them.

[pento]

This issue came to my attention recently, while I don't have the time to fix it now, I do have some thoughts on possible solutions. The primary issue is that the export script has a 5 minute time limit: if it hasn't finished downloading in 5 minutes, the download will need to be resumed. This does work well, but it's not super obvious that it's needed.

Small Changes

• Add some help text to the media export box, instructing users to use a download manager if their browser fails.

Medium Changes

• Increase the time limit of the export script, so more people can successfully finish it without running into timeouts.
• As the export script already supports from and to parameters, an advanced UI may be able to effectively split the download up into multiple files of approximately equal size.

Large Changes

• Switch to asynchronous exports. Some of the pieces for this already exist (see wp vip-export files, for example), though it would need additional work to notify the user when the export file has been generated, and to clean up export files after a set period.
• Export to third party services. Google Drive is an option, as @pablinos suggested. Dropbox may be another, though Keyring currently doesn't support it, that would need to be added, too.

[pablinos]

Yeah it'd be good to iterate on this and improve it. Allowing it to be split in the UI, would be a cool feature.

Increase the time limit of the export script, so more people can successfully finish it without running into timeouts.

We looked into this as we were developing the feature at the GM and Systems were very reluctant to increase that time limit. I can't remember the exact reasons, but they made perfect sense at the time.

Partly, it would involve a lot of re-configuration and exceptions made for that script, and it's not worth the effort if there are other solutions.

[bisko]

@pento what about adding a "download" manager to Calypso's Export section as some file sharing sites do? This can handle the download more gracefully and wouldn't require too much magic in the backend.

I was thinking about this issue a few months ago and there seem to be some libraries that might help out with this, e.g. https://github.com/jimmywarting/StreamSaver.js

[pablinos]

That looks like a good option @bisko. If we can add a lightweight download manager, that would probably solve this for the majority of people.