Fetch: Downloads Failing / Better error description

[cdn34]

Two issues, the first is that when something fails, most of the time we get a generic request_not_successful message, which is not very useful when when trying to understand exactly why it failed.

As for the second issue,

In my app (iOS / android), the urls that are sent to the downloader may or may not be restricted (may be redirected to a paywall page if restricted).

On iOS using NSURLSessionDownloadTask , if the url is indeed a paywall, instead of it failing, the html content is returned which I can then use to check for restricted signs and act accordingly.

On android using this library, it works similarly to a certain degree. On certain urls it returns the HTML content, on others it just fails with a request_not_successful message.

An example would be :

Works on both android / iOS (Returns HTML content): http://jandonline.org/article/S2212267214006005/pdf

Works on iOS, fails on android (request_not_successful): https://www.magonlinelibrary.com/doi/pdf/10.12968/bjon.2013.22.7.364?t=

Library version: 2.3.4
Fetch configuration:

CookieManager cookieManager = new CookieManager();
        cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
        OkHttpClient okHttpClient = new OkHttpClient
                .Builder()
                .followRedirects(true)
                .followSslRedirects(true)
                .cookieJar(new JavaNetCookieJar(cookieManager)).build();

        FetchConfiguration fetchConfiguration = new FetchConfiguration.Builder(context)
                .setDownloadConcurrentLimit(4)
                .setProgressReportingInterval(100)
                .setNamespace("PPDownloader")
                .setHttpDownloader(new OkHttpDownloader(okHttpClient))
                .build();

[tonyofrancis]

@cdn34

1. Fetch returns the request_not_successful when It is unable to determine the true cause of the error. It will only get better when the library matures and issues like this are reported. Fetch also returns the exceptions that are thrown when possible. download.getError().getThrowable(). Note: .getThrowable() may return null.
2. I was able to determine your error for the failed url. The server is returning a 403 forbidden.

I will keep this issue ticket open as an enhancement for the library. Working on a fix that will report these error codes for the next release

[jpvs0101]

I was able to determine your error for the failed url. The server is returning a 403 forbidden.

Why don't you just return the server error string?

[tonyofrancis]

@jpvs0101 because the error is not always a server error. It could be a library error, or invalid configuration on the developers part who knows. Returning the Error object with the attached throwable is the best option.

[tonyofrancis]

@cdn34 Fetch 2.3.5 has been released. You can now get the http response with the error code. Sample:

    @Override
    public void onError(@NotNull Download download, @NotNull Error error, @Nullable Throwable throwable) {
       final Throwable throwable1 = download.getError().getThrowable();
       final Downloader.Response response = download.getError().getHttpResponse();
       if (response != null) {
           if (response.getCode() == 403) {
               final String message = response.getErrorResponse();
               Log.d("HttpErrorMessage", message);
           }
       }
    }

Please let me know if this works well for you.

[cdn34]

@tonyofrancis Thanks, I'll try it tomorrow.

[cdn34]

@tonyofrancis Just tried it and it's working. Thanks for the help.

Edit:
Just tried on previously successful requests and now it seems that no matter which URL I pass in, it always ends up in onError. onCompleted doesn't seem to be ever called anymore.

[tonyofrancis]

@cdn34 what error are you getting?. Logs? Url?

[cdn34]

@tonyofrancis
PDF url: https://ac.els-cdn.com/S2589004218301354/1-s2.0-S2589004218301354-main.pdf?_tid=092f1cb4-9533-434f-81c9-3f3ab5563f0d&acdnat=1547221107_0348c3ab0f03bf426571a58b7b192cde
Response code: 206
Error: UNKNOWN_IO_ERROR
response.getErrorResponse(): Returns the content of the PDF

[tonyofrancis]

CookieManager cookieManager = new CookieManager();
        cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
        OkHttpClient okHttpClient = new OkHttpClient
                .Builder()
                .followRedirects(true)
                .followSslRedirects(true)
                .cookieJar(new JavaNetCookieJar(cookieManager)).build();

        FetchConfiguration fetchConfiguration = new FetchConfiguration.Builder(context)
                .setDownloadConcurrentLimit(4)
                .setProgressReportingInterval(100)
                .setNamespace("PPDownloader")
                .setHttpDownloader(new OkHttpDownloader(okHttpClient, Downloader.FileDownloaderType.SEQUENTIAL))
                .build();

Let me know if this works. If not I will have a second look

[cdn34]

@tonyofrancis Just tried it, the issue is still there.

[tonyofrancis]

@cdn34 The link your provided https://ac.els-cdn.com/S2589004218301354/1-s2.0-S2589004218301354-main.pdf?_tid=092f1cb4-9533-434f-81c9-3f3ab5563f0d&acdnat=1547221107_0348c3ab0f03bf426571a58b7b192cde leads to a webpage not a pdf. Is that the intended download?

[cdn34]

@tonyofrancis No, it's supposed to be a pdf, but the url seems to expire. If you go to https://www.sciencedirect.com/science/article/pii/S2589004218301354, and open the pdf (Download PDF) in a new tab you can get a similar url to the one I pasted here in the first place.

[tonyofrancis]

@cdn34 I am able to download the pdf with this url https://www.sciencedirect.com/science/article/pii/S2589004218301354/pdfft?md5=00aecd3728dbf715179f20b8dc0e69b5&pid=1-s2.0-S2589004218301354-main.pdf&isDTMRedir=true&download=true Cannot with any other url

[cdn34]

@tonyofrancis Ok, thanks for having a look. I'll try to give you other examples of when this issue occurs this weekend.

[cdn34]

@tonyofrancis

I've tried with a few other links and the result is always the same, the url doesn't seem to matter. But just so you can try with the same urls I did, here are some:

• https://bmcbioinformatics.biomedcentral.com/track/pdf/10.1186/1471-2105-9-248
• http://www.jtcvs.org/article/S0022522313015195/pdf

Also here are some of the logs. The library seems to encounter some issue when downloading. When going back to the previous version (2.3.4), downloads complete normally.

D/FetchLog: FileDownloader starting Download DownloadInfo(id=-750932295, namespace='PPDownloader', url='http://www.jtcvs.org/article/S0022522313015195/pdf', file='/storage/emulated/0/pp/fb284d86-ea0f-4aa4-813e-e32f6b07dd8d.pdf', group=0, priority=HIGH, headers={}, downloaded=0, total=-1, status=QUEUED, error=NONE, networkType=ALL, created=1547392107053, tag=null, enqueueAction=UPDATE_ACCORDINGLY, identifier=0, downloadOnEnqueue=true, extras={})

E/FetchLogError: FileDownloader download:DownloadInfo(id=-750932295, namespace='PPDownloader', url='http://www.jtcvs.org/article/S0022522313015195/pdf', file='/storage/emulated/0/pp/fb284d86-ea0f-4aa4-813e-e32f6b07dd8d.pdf', group=0, priority=HIGH, headers={}, downloaded=0, total=-1, status=DOWNLOADING, error=NONE, networkType=ALL, created=1547392107053, tag=null, enqueueAction=UPDATE_ACCORDINGLY, identifier=0, downloadOnEnqueue=true, extras={})
    java.io.IOException: closed
        at okio.RealBufferedSource$1.read(RealBufferedSource.java:435)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:347)
        at com.tonyodev.fetch2.downloader.SequentialFileDownloaderImpl.writeToOutput(SequentialFileDownloaderImpl.kt:221)
        at com.tonyodev.fetch2.downloader.SequentialFileDownloaderImpl.run(SequentialFileDownloaderImpl.kt:119)
        at com.tonyodev.fetch2.downloader.DownloadManagerImpl$start$$inlined$synchronized$lambda$1.run(DownloadManagerImpl.kt:101)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
        at java.lang.Thread.run(Thread.java:764)

D/FetchLog: Removed download DownloadInfo(id=-750932295, namespace='PPDownloader', url='http://www.jtcvs.org/article/S0022522313015195/pdf', file='/storage/emulated/0/pp/fb284d86-ea0f-4aa4-813e-e32f6b07dd8d.pdf', group=0, priority=HIGH, headers={}, downloaded=0, total=-1, status=REMOVED, error=UNKNOWN_IO_ERROR, networkType=ALL, created=1547392107053, tag=null, enqueueAction=UPDATE_ACCORDINGLY, identifier=0, downloadOnEnqueue=true, extras={})

Even though the downloads are failing, response.getErrorResponse() seems to return the content of the file being downloaded in onError.

[tonyofrancis]

@cdn34 I was able to get the pdfs to download successfully. The issue seems to be that your okhttp client is not configured for redirection. This is the default okhttp configuration for Fetch.

OkHttpClient.Builder()
            .readTimeout(20_000L, TimeUnit.MILLISECONDS)
            .connectTimeout(15_000L, TimeUnit.MILLISECONDS)
            .cache(null)
            .followRedirects(true)
            .followSslRedirects(true)
            .retryOnConnectionFailure(false)
            .cookieJar(getDefaultCookieJar())
            .build()

Using this configuration I was able to download the files provided at the links with no issue. Also Fetch version 2.3.6 has been released with bug fixes for a memory leak.

[cdn34]

@tonyofrancis Just updated and tried that configuration. It seems to be working, thanks for the help.