Retrofit: Multipart data uploading progress bar

[egor-n]

Is there a way to show an uploading progress bar while making a @Multipart request?

[JakeWharton]

We don't offer any kind of progress support because we do not require that the content length be known before the request execution. This means that as the request is hitting the network, you can be streaming the contents of the body from a source of unknown length. This is especially true of multi-part requests where there are multiple opportunities for streamed parts.

Now of course, you might be thinking that all your content is small and completely buffered so the length is known. The problem with this is that we've found it difficult to design an API that can handle both cases elegantly.

I might look into this further when development of version 2 begins in the next month or two. But I can't make any promises, and the current code is not configured well for something like this so I'll say no for now.

[Tremelune]

Even if streaming a large file from disk (or any other source), there are usually ways of knowing the size of the content before starting the transfer. Would it be difficult to add a provision for progress if the total content length was always provided (as opposed to inferred by Retrofit)?

With mobile, it's very possible for a network to be dog-slow and reliable, such that uploading a 1MB file can take minutes. If you want to do any media transfers with Retrofit, the ability to gauge progress is a must.

We implemented it with HttpClient by extending HttpEntityWrapper and making some very simple calculations on some of the write(byte) calls and passing the current progress to a passed-in listener periodically. Dunno how that translates to the Retrofit abstractions, but it would be a bummer to lose the feature.

[JakeWharton]

@Tremelune You could do the same thing with the TypedOutput request body instance.

In a more general case, I'll evaluate what support of this would look like for v2.

[Tremelune]

@JakeWharton Is there an example of how to do this somewhere? It seems that TypedByteArray will have writeTo(OutputStream out) called exactly once for a 100KB file. I'm not sure how to sneak the listener in to the single-byte write without wrapping Request.getBody() somewhere in the client code.

[Tremelune]

Sadly, my solution of wrapping connection.getOutputStream() during request preparation has proven to be unusably slow. It seems that when overriding OutputStream.write(b) to delegate to the underlying connection stream (RealBufferedSink.outputStream() in OkHttp 2.0) the transfer slows to a crawl. I haven't been able to figure out why yet.

I am as of yet unaware of a solution, even one that would involve mucking up the Retrofit source.

[davicdsalves]

Check http://stackoverflow.com/questions/23348812/android-retrofit-onprogressupdate-for-showing-progress-notification/24772058#24772058

[Tremelune]

This was a tremendous help, thank you. I was reluctant to implement the IO while loop myself, as I didn't want to impact performance, but it seems to be just as fast. I guess I just needed to see an implementation to nudge me in the right direction.

Also, it seems like cancelation could be done by checking a passed-in object that held a boolean and terminating inside the loop if it is ever flipped.

[davicdsalves]

No problem, as I said in the answer the writeTo() is the same as the one you find in TypedFile except for the listener, I didn't see any performance change when using it.

Indeed you could expand it and have something like listener.shouldCancelTransfer() in the loop and terminate it.

[droidchef]

Can this be a possible solution ?
http://stackoverflow.com/questions/23348812/android-retrofit-onprogressupdate-for-showing-progress-notification

[philipgiuliani]

Thanks for the solution on StackOverflow! Works fine.

[davicdsalves]

I'm glad it worked for you @philipgiuliani. Maybe it could be an official solution?

[DavidMihola]

I tried this and observed that while it took only a few seconds for writeTo() to go through all of a 4MB image file I only received a response from our server about a minute later:

06-11 21:42:50.005 17539-21710/com.our.app.dev D/Retrofit﹕ ---> END HTTP (4121330-byte body)
06-11 21:42:50.105 17539-21710/com.our.app.dev D/dalvikvm﹕ GC_FOR_ALLOC freed 15882K, 28% free 58773K/80988K, paused 42ms, total 42ms
06-11 21:43:39.635 17539-21710/com.our.app.dev D/Retrofit﹕ <--- HTTP 200 http://app.dev.our-server.com/webservice/api/Image/upload (49634ms)

The effect of course is that the progress bar fills very quickly but then it takes a long time for the actual response to arrive.

Is that because writeTo writes to some kind of buffered stream and therefore the time that writeTo takes does not really reflect the time it takes the data over the net? Or is that because our server itself took so long to process the image?

I am sorry I have to ask, but I don't know Retrofit's internals well enough to know what's going on here...

[davicdsalves]

@DavidMihola i saw it happening as well, it just takes more seconds to write the last bytes.
Maybe changing the order of the listener would fix it.
Instead of

            this.listener.transferred(total);
            out.write(buffer, 0, read);

Use

            out.write(buffer, 0, read);
            this.listener.transferred(total);

That should give you a better feedback when the upload finishes.

[DavidMihola]

@davicdsalves That makes sense, thank you!

I still have the feeling that the actual sending of the request takes somewhat longer than the feedback seems to indicate but I haven't had the time to actually watch it with Wireshark or something...

[tomblenz]

@DavidMihola Chances are you have the log level set to include the body. In order to log the body, it essentially goes through the whole rigamarole of writing the request, but to your device's memory rather than the network. You thus may be seeing the logging output rather than the actual transmission? Try using a lower log level

[androiddev-smartside]

@davicdsalves thanks for your solution, however when uploading image from a camera, I don't have a File object, so I use TypedByteArray( mimeString, bytesFromCamera) instead.
I peaked into the TypedBytedArray source but couldn't understand how I could integrate your solution.

[philipgiuliani]

@androiddev-smartside checkout this one!

ProgressiveTypedByteArray:
https://github.com/OneDrive/onedrive-explorer-android/blob/master/app/src/main/java/com/microsoft/onedrive/apiexplorer/ProgressiveTypedByteArray.java

ProgressListener:
https://github.com/OneDrive/onedrive-explorer-android/blob/master/app/src/main/java/com/microsoft/onedrive/apiexplorer/ProgressListener.java

You can easily create your own one. Just copy the ProgressiveTypedByteArray from the lib and adapt it to have a ProgressListener.

[androiddev-smartside]

@philipgiuliani wow, all of a sudden I love Microsoft OneDrive, who could tell! Thanks, that one seems to work fine so far.

[DavidMihola]

@duckinferno Yes, that was indeed the case - I set the log level to BASIC and now the updates align a lot better with my subjective feeling of the upload (I still haven't checked the actual transfer with Wireshark but now after the 100% it takes only one or two seconds for the response to arrive, so I think that should be OK).

[davicdsalves]

I'm amazed that this is still a hot topic and often I get an upvote on SO. I'm not following retrofit development anymore, but it doesn't have an official solution for this on the last versions?

[JakeWharton]

Retrofit 2 uses OkHttp and progress for OkHttp can be found @ https://github.com/square/okhttp/blob/master/samples/guide/src/main/java/com/squareup/okhttp/recipes/Progress.java

[DavidMihola]

@davicdsalves @JakeWharton Thank you for pointing out that Retrofit 2 can do this out of the box. I just thought that I should confirm @duckinferno's solution - better late than never.

[shahinfasihi]

hi, this link can be very helpful
https://gist.github.com/eduardb/dd2dc530afd37108e1ac
after all
i use this and try to explain it simply (if i am wrong fix me ,tnx ;) )
first get this class
https://gist.github.com/shahinfasihi/b9f33883aa0b506acfa1
and after that copy these Lines in your AsyncTask doInBackground Method
https://gist.github.com/shahinfasihi/67f9b1f44dddf57f64c5
i am using these hope to work fine for you

[priyochatterjee08]

@JakeWharton
https://github.com/square/okhttp/blob/master/samples/guide/src/main/java/com/squareup/okhttp/recipes/Progress.java..... This link does not exist.

[JakeWharton]

Progress.java under https://github.com/square/okhttp/blob/master/samples/guide

[alexnavarro]

@JakeWharton It's not an Android example. Can you have any Android example? Thanks

[davicdsalves]

@alexnavarro What you mean? This work on Android too:
https://github.com/square/okhttp/blob/master/samples/guide/src/main/java/okhttp3/recipes/Progress.java

[alexnavarro]

I saw this example and I got quite confused because I'm using retrofit and I need intrercept RequestBody. In the example the ResponseBody was interceped. Can you understand me?

[TakeoffAndroid]

In v2.0 you can show the progress bar by simply using the request body from the call object. Please find the code that I have used after a long struggle

RequestBody  requestBody = call.request().body();

            long fileLength = requestBody.contentLength();
            byte[] buffer = new byte[2048];
            FileInputStream in = new FileInputStream(filePath);
            long uploaded = 0;

            try {
                int read;
                Handler handler = new Handler(Looper.getMainLooper());
                while ((read = in.read(buffer)) != -1) {

                    // update progress on UI thread for 5 seconds
                    handler.postDelayed(new ProgressUpdater(uploaded, fileLength,callbacks),5000);

                    uploaded += read;
//                    sink.write(buffer, 0, read);
                }
            } finally {
                in.close();
            }

  private static class ProgressUpdater implements Runnable {
        private long mUploaded;
        private long mTotal;
        private RetrofitFileUploadCallbacks mCallbacks;

        public ProgressUpdater(long uploaded, long total, RetrofitFileUploadCallbacks callbacks) {
            mUploaded = uploaded;
            mTotal = total;
            mCallbacks = callbacks;
        }

        @Override
        public void run() {
            mCallbacks.onFileUploadProgress((int)(100 * mUploaded / mTotal));
        }
    }

[kulsjoshi]

@TakeoffAndroid Do you have any kind of demo with your code?

[luca992]

@TakeoffAndroid Pretty sure the only thing you are tracking is the progress of a while loop reading a file 2048 bytes at a time... not the data that is actually being uploaded

edit: I figured out when using HttpLoggingInterceptor it calls writeTo and passes data into a local buffer just for logging purposes. So the first time the file is read by the logger it goes very quickly and does not represent the file upload. (The second time the file is read is what should be tracked)... I posted a working solution on stackoverflow a while ago, it includes a workaround for HttpLoggingInterceptor

[shubham1g5]

Anyone have any idea about how to get Progress in case I have a multipart request that uploads a dynamic number of files to the server. I am migrating to Retrofit/OkHTTP from Apache HTTP client and was earlier achieving this by extending the MultipartEntity and tracking transferred bytes. Though I can't do the same with it's OkHTTP's equivalent MultipartBody as it's declared as a final class. Are there any workarounds I can implement for my case ? (Relevant SO Question - https://stackoverflow.com/q/45019438/3811963)

[nb312]

There are solutions for making the progress in this link 👍 https://gist.github.com/eduardb/dd2dc530afd37108e1ac

[nb312]

You can just take the first file and just use the data change for your own progress bar.

[joshuapinter]

For those landing here that are still stuck on Retrofit 1.x, here's an easy, working solution that we ended up using: https://stackoverflow.com/a/24772058/293280.

It involves creating a ProgressListener interface, a modified TypedFile and just using the listener when uploading the modified TypedFile.

It took a while to track this down so hopefully this helps others still running on version 1.x.