Fetch: Failed to download chunked type file!

[jpvs0101]

The downloader seems like NOT supporting Transfer-Encoding: chunked.
Downloaded some unrelated html instead of mp3 file.

Link: https://hindugodsongs.in/

final FetchConfiguration fetchConfiguration = new FetchConfiguration.Builder(this)
   .setDownloadConcurrentLimit(3)
   .enableAutoStart(false)
   .setProgressReportingInterval(1100)
   .setHttpDownloader(new HttpUrlConnectionDownloader(Downloader.FileDownloaderType.SEQUENTIAL)) 
   .build();
Fetch.Impl.setDefaultInstanceConfiguration(fetchConfiguration);

Logcat:
D/LibGlobalFetchLib: Enqueued download DownloadInfo(id=-1786947277, namespace='LibGlobalFetchLib', url='https://hindugodsongs.in/get/download/shani-gayatri-mantra-[hindugodsongs.in].mp3', file='content://com.android.providers.downloads.documents/document/73', group=0, priority=HIGH, headers={Cookie=__cfduid=d22766a6f223a771943be99db82f7db1d1543158628, User-Agent=Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36}, downloaded=0, total=-1, status=QUEUED, error=NONE, networkType=ALL, created=1543163829063, tag=null, enqueueAction=UPDATE_ACCORDINGLY, identifier=0, downloadOnEnqueue=true, extras={"KEY_DOWNLOAD_SUPPORTS_RESUME":"false","KEY_DOWNLOAD_FILE_NAME":"shani-gayatri-mantra-[hindugodsongs.in].mp3 (1)","KEY_DOWNLOAD_USER_AGENT":"Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36","KEY_DOWNLOAD_SAVE_PATH":"content://com.android.providers.downloads.documents/document/73"}) D/LibGlobalFetchLib: PriorityIterator backoffTime reset to 500 milliseconds D/LibGlobalFetchLib: PriorityIterator started D/LibGlobalFetchLib: PriorityIterator backoffTime reset to 500 milliseconds D/LibGlobalFetchLib: Added DownloadInfo(id=-1786947277, namespace='LibGlobalFetchLib', url='https://hindugodsongs.in/get/download/shani-gayatri-mantra-[hindugodsongs.in].mp3', file='content://com.android.providers.downloads.documents/document/73', group=0, priority=HIGH, headers={Cookie=__cfduid=d22766a6f223a771943be99db82f7db1d1543158628, User-Agent=Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36}, downloaded=0, total=-1, status=QUEUED, error=NONE, networkType=ALL, created=1543163829063, tag=null, enqueueAction=UPDATE_ACCORDINGLY, identifier=0, downloadOnEnqueue=true, extras={"KEY_DOWNLOAD_SUPPORTS_RESUME":"false","KEY_DOWNLOAD_FILE_NAME":"shani-gayatri-mantra-[hindugodsongs.in].mp3 (1)","KEY_DOWNLOAD_USER_AGENT":"Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36","KEY_DOWNLOAD_SAVE_PATH":"content://com.android.providers.downloads.documents/document/73"}) D/LibGlobalFetchLib: Queued DownloadInfo(id=-1786947277, namespace='LibGlobalFetchLib', url='https://hindugodsongs.in/get/download/shani-gayatri-mantra-[hindugodsongs.in].mp3', file='content://com.android.providers.downloads.documents/document/73', group=0, priority=HIGH, headers={Cookie=__cfduid=d22766a6f223a771943be99db82f7db1d1543158628, User-Agent=Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36}, downloaded=0, total=-1, status=QUEUED, error=NONE, networkType=ALL, created=1543163829063, tag=null, enqueueAction=UPDATE_ACCORDINGLY, identifier=0, downloadOnEnqueue=true, extras={"KEY_DOWNLOAD_SUPPORTS_RESUME":"false","KEY_DOWNLOAD_FILE_NAME":"shani-gayatri-mantra-[hindugodsongs.in].mp3 (1)","KEY_DOWNLOAD_USER_AGENT":"Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36","KEY_DOWNLOAD_SAVE_PATH":"content://com.android.providers.downloads.documents/document/73"}) for download D/LibGlobalFetchLib: DownloadManager starting download DownloadInfo(id=-1786947277, namespace='LibGlobalFetchLib', url='https://hindugodsongs.in/get/download/shani-gayatri-mantra-[hindugodsongs.in].mp3', file='content://com.android.providers.downloads.documents/document/73', group=0, priority=HIGH, headers={Cookie=__cfduid=d22766a6f223a771943be99db82f7db1d1543158628, User-Agent=Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36}, downloaded=0, total=-1, status=QUEUED, error=NONE, networkType=ALL, created=1543163829063, tag=null, enqueueAction=UPDATE_ACCORDINGLY, identifier=0, downloadOnEnqueue=true, extras={"KEY_DOWNLOAD_SUPPORTS_RESUME":"false","KEY_DOWNLOAD_FILE_NAME":"shani-gayatri-mantra-[hindugodsongs.in].mp3 (1)","KEY_DOWNLOAD_USER_AGENT":"Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36","KEY_DOWNLOAD_SAVE_PATH":"content://com.android.providers.downloads.documents/document/73"}) D/LibGlobalFetchLib: FileDownloader starting Download DownloadInfo(id=-1786947277, namespace='LibGlobalFetchLib', url='https://hindugodsongs.in/get/download/shani-gayatri-mantra-[hindugodsongs.in].mp3', file='content://com.android.providers.downloads.documents/document/73', group=0, priority=HIGH, headers={Cookie=__cfduid=d22766a6f223a771943be99db82f7db1d1543158628, User-Agent=Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36}, downloaded=0, total=-1, status=QUEUED, error=NONE, networkType=ALL, created=1543163829063, tag=null, enqueueAction=UPDATE_ACCORDINGLY, identifier=0, downloadOnEnqueue=true, extras={"KEY_DOWNLOAD_SUPPORTS_RESUME":"false","KEY_DOWNLOAD_FILE_NAME":"shani-gayatri-mantra-[hindugodsongs.in].mp3 (1)","KEY_DOWNLOAD_USER_AGENT":"Mozilla/5.0 (Linux; Android 8.1.0; Redmi 2 Build/OPM7.181005.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari/537.36","KEY_DOWNLOAD_SAVE_PATH":"content://com.android.providers.downloads.documents/document/73"}) W/OkHttpClient: A connection to https://hindugodsongs.in/ was leaked. Did you forget to close a response body? D/LibGlobalFetchLib: PriorityIterator backoffTime reset to 500 milliseconds

For your reference:
https://stackoverflow.com/questions/5737945/java-how-to-download-chunked-content-correctly

[tonyofrancis]

@jpvs0101 Just pasted the request link in a web browser and I was redirected to an html page not an mp3 download.???
https://hindugodsongs.in/get/download/shani-gayatri-mantra-[hindugodsongs.in].mp3

[jpvs0101]

Just go to https://hindugodsongs.in/.
Download any song (link).

[tonyofrancis]

@jpvs0101 Seems like the default HttpUrlConnectionDownloader is not redirecting to the download link correctly. What if you extend HttpUrlConnectionDownloader and override method onPreExecute and add this line client.setFollowRedirects = true. Let me know if this works and I can add it to the next version of Fetch.

final FetchConfiguration fetchConfiguration = new FetchConfiguration.Builder(this)
                .enableRetryOnNetworkGain(true)
                .setDownloadConcurrentLimit(3)
                .setHttpDownloader(new HttpsUrlConnectionDownloader(Downloader.FileDownloaderType.PARALLEL))
                .build();


  public class HttpsUrlConnectionDownloader extends HttpUrlConnectionDownloader {

        public HttpsUrlConnectionDownloader(@NotNull Downloader.FileDownloaderType fileDownloaderType) {
            super(fileDownloaderType);
        }

        @Override
        public void onPreClientExecute(@NotNull HttpURLConnection client, @NotNull Downloader.ServerRequest request) {
            super.onPreClientExecute(client, request);
            client.setFollowRedirects = true // add this line
            if (request.getUrl().startsWith("https")) {
                HttpsURLConnection httpsURLConnection = (HttpsURLConnection) client;
                httpsURLConnection.setSSLSocketFactory(getSSLSocketFactory());
            }
        }

[jpvs0101]

Compile error

error: onPreClientExecute(HttpURLConnection,ServerRequest) in HttpsUrlConnectionDownloader cannot implement onPreClientExecute(T,ServerRequest) in Downloader
    public void onPreClientExecute(@NotNull HttpURLConnection client, @NotNull Downloader.ServerRequest request) {
                ^
  return type void is not compatible with Unit
  where T,R are type-variables:
    T extends Object declared in interface Downloader
    R extends Object declared in interface Downloader
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 error
:app:compileDebugJavaWithJavac FAILED

Also tried below, but not downloading (same as before)

 HttpUrlConnectionDownloader.HttpUrlConnectionPreferences httpUrlConnectionPreferences = new HttpUrlConnectionDownloader.HttpUrlConnectionPreferences();
        httpUrlConnectionPreferences.setFollowsRedirect(true);

        final FetchConfiguration fetchConfiguration = new FetchConfiguration.Builder(this)
                .setDownloadConcurrentLimit(3)
                .enableAutoStart(false)
                .setProgressReportingInterval(1100)
                .setHttpDownloader(new HttpUrlConnectionDownloader(httpUrlConnectionPreferences, Downloader.FileDownloaderType.PARALLEL))//new HttpUrlConnectionDownloader(Downloader.FileDownloaderType.PARALLEL))
                .build();
        Fetch.Impl.setDefaultInstanceConfiguration(fetchConfiguration);

I guess that it is not about redirecting...

[tonyofrancis]

Looking into the method return type. What happens if you use the okhttp downloader???

[jpvs0101]

I will try...

Update 1:
Tried with okHttpDownloader but same issue
Update 2:
When try 2.3.3, Failed to resolve: com.tonyodev.fetch2:fetch2:2.3.3

[tonyofrancis]

Fetch 2.3.3 has not been released yet. Sorry I may have added release note prematurely.

[tonyofrancis]

In terms of the urls, I am unable to get them to work myself. I believe that the server may be doing some kid of strange redirecting. Are you able to get samples from that site/server Administrator or more information on how they allow downloading via http clients?. As I stated before. The urls I tested always redirected me back to the homepage. Tested with Fetch and on several desktop web browsers.

[tonyofrancis]

Update: Fetch 2.3.3 that fixes the return type issue is now available.

[jpvs0101]

Links are downloaded in all major browsers, even in mobile browsers,

Transfer-Encoding: chunked
refer issue #194

They actually provide a download, I guess, it's not about redirecting, it gives the file like streaming.... that's all I know so far, rest is in your hand!

Update:
Tried with all combinations, still not working! :(
For your reference:
https://stackoverflow.com/questions/5737945/java-how-to-download-chunked-content-correctly

[tonyofrancis]

@jpvs0101 After much testing I can assure you that Fetch handles Chunk downloading correctly and It is fully supported by both the Parallel and Sequential downloaders. Fetch does not only depend on the content Length to download a file. Issue 194 was resolved in releases after the versions mentioned in that issue. At that time it was only supported for the Sequential downloader.

Link tested: https://github.com/tonyofrancis/Fetch/archive/v2.3.zip (Transfer-Encoding: chunked)
MD5 hash matched both the desktop version and the version downloaded with Fetch. md5 - b761a43a10a61d2b8dab1919560a9ee2

With that said I implore you to verify that the urls you have will lead to the download file. Please contact the sites Administrator.

[jpvs0101]

I just give you a hint, but I don't know what is the actual issue, I know only the issue exists, the same link downloaded using firefox, ADM on Android 8.1. Please find the attachment, it has screen recording...
fetch_issue.mp4.zip

[tonyofrancis]

@jpvs0101 this site is down so I cannot continue further testing unless another link is provided

[jpvs0101]

https://hindugodsongs.in/

Site is working on my side, how can i find another site with same issue?

[tonyofrancis]

@jpvs0101 site seems to be back up. Still testing. Cannot seem to figure out what Fetch is missing to make this work. I have already looked high and low. Wish I knew what the other libs did to fix this.

[jpvs0101]

Thanks, I have alraedy sent you a link from stack overflow for your
reference. Also it is better idea to look into other libs. Like I said
before, I always want to fetch to handle all the inner stuff, so that the
developers can focus on other taks. Also please take a look into blob type
urls.

A few more steps to make fetch world's best downloading lib for android...

[tonyofrancis]

@jpvs0101 I was able to figure out the issue. Those links need a referrer header added to them. What I will do is add the default referrer which is the website itself if one was not set before making the request.

[tonyofrancis]

@jpvs0101 Fetch 2.3.4 has be released. Please test and let me know if it is working for you.

[jpvs0101]

Holy shit :1st_place_medal: ,
it is now working man...
It supports resume too...
Also it seems like this #288 bug is fixed!

V 2.3.4
final OkHttpClient okHttpClient = new OkHttpClient.Builder()
.readTimeout(20, TimeUnit.SECONDS) //increase read timeout as needed
.connectTimeout(20, TimeUnit.SECONDS) //increase connection timeout as needed
.followRedirects(true)
.followSslRedirects(true)
.build();
return new OkHttpDownloader(okHttpClient,
Downloader.FileDownloaderType.PARALLEL);

What have you done?

Update:
Also works fine using default HttpUrlConnectionDownloader(Downloader.FileDownloaderType.PARALLEL) even without okhttp!

[tonyofrancis]

The links need a Referer header attached.

request.addHeader("Referer", "https://hindugodsongs.in/")

If those are not added to the request, Fetch will insert a default one when the request is being made.