Retrofit: OutOfMemory while sending big file

[sportwtany]

When I send big file (about 50 Mb) by multipart, Retrofit catch OutOfMemoryError.
How I can send big files by Retrofit?

[mindshifter]

https://github.com/square/retrofit/blob/master/retrofit/src/main/java/retrofit/client/UrlConnectionClient.java#L62

Try and set the content length to -1 :)

[sportwtany]

mindshifter, server don't support chunk mode =(

[JakeWharton]

Can you provide more information about how you are calling Retrofit? What does your interface method look like? What values are you calling it with? Do you have logging enabled?

[JakeWharton]

Retrofit will stream Multipart part bodies directly to the network so the fact that you send a big body should not cause an OOM unless you are causing it to be buffered somehow.

[sportwtany]

My interface:

private interface FileUploader{
        @Multipart
        @POST("/upload")
        void upload(@EncodedQuery("action") String action,
                    @EncodedQuery("file_id") String file_id,
                    @EncodedQuery("user_id") String user_id,
                    @Part("recipient_id")TypedString visitorId,
                    @Part("transfer_id")TypedString transferId,
                    @Part("file") TypedFile file,
                    Callback<Response> callback);
    }

Calling request:

RestAdapter adapter = new RestAdapter.Builder()
                        .setEndpoint(endPoint)
                        .setLogLevel(RestAdapter.LogLevel.FULL)
                        .build();
FileUploader uploader = adapter.create(FileUploader.class);
uploader.upload(action, fileId, userId, visitor, transferId, file, new Callback<Response>() {
                    @Override
                    public void success(Response response, Response response2) {
                        UtilLog.logE(response.getStatus() + " SUCCESS");
                    }

                    @Override
                    public void failure(RetrofitError error) {
                        UtilLog.logE("ERROR ");
                        error.printStackTrace();
                    }
                });

And logs:
06-24 09:26:49.735 D/Retrofit﹕ ---> HTTP POST http://ENDPOINT/upload?action=file.upload&file_id=lgyjnmh3rhog8pvi&user_id=account%3A94528%3Amember%3A94529
06-24 09:26:49.735 D/Retrofit﹕ Content-Type: multipart/form-data; boundary=85f8492b-982a-44ca-83ec-3e5bae84d354
06-24 09:26:49.735 D/Retrofit﹕ Content-Length: 51873672
06-24 09:26:50.940 I/dalvikvm﹕ "Retrofit-/upload?action=file.upload&file_id=lgyjnmh3rhog8pvi&user_id=account%3A94528%3Amember%3A94529" prio=5 tid=27 RUNNABLE
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.mime.TypedFile.writeTo(TypedFile.java:78)
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.mime.MultipartTypedOutput$MimePart.writeTo(MultipartTypedOutput.java:50)
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.mime.MultipartTypedOutput.writeTo(MultipartTypedOutput.java:133)
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.Utils.readBodyToBytesIfNecessary(Utils.java:62)
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.RestAdapter.logAndReplaceRequest(RestAdapter.java:432)
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:312)
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.RestAdapter$RestHandler.access$100(RestAdapter.java:220)
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.RestAdapter$RestHandler$2.obtainResponse(RestAdapter.java:278)
06-24 09:26:50.950 I/dalvikvm﹕ at retrofit.CallbackRunnable.run(CallbackRunnable.java:42)
06-24 09:26:50.955 I/dalvikvm﹕ at retrofit.Platform$Android$2$1.run(Platform.java:142)
06-24 09:26:50.955 I/dalvikvm﹕ [ 06-24 09:26:50.955 8416:10100 D/Retrofit ]
---- ERROR http://ENDPOINT/upload?action=file.upload&file_id=lgyjnmh3rhog8pvi&user_id=account%3A94528%3Amember%3A94529
06-24 09:26:50.960 D/Retrofit﹕ java.lang.OutOfMemoryError
at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:91)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:201)
at retrofit.mime.TypedFile.writeTo(TypedFile.java:78)
at retrofit.mime.MultipartTypedOutput$MimePart.writeTo(MultipartTypedOutput.java:50)
at retrofit.mime.MultipartTypedOutput.writeTo(MultipartTypedOutput.java:133)
at retrofit.Utils.readBodyToBytesIfNecessary(Utils.java:62)
at retrofit.RestAdapter.logAndReplaceRequest(RestAdapter.java:432)
at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:312)
at retrofit.RestAdapter$RestHandler.access$100(RestAdapter.java:220)
at retrofit.RestAdapter$RestHandler$2.obtainResponse(RestAdapter.java:278)
at retrofit.CallbackRunnable.run(CallbackRunnable.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
at retrofit.Platform$Android$2$1.run(Platform.java:142)
at java.lang.Thread.run(Thread.java:841)
06-24 09:26:50.960 D/Retrofit﹕ ---- END ERROR

[JakeWharton]

By calling .setLogLevel(RestAdapter.LogLevel.FULL) you are forcing Retrofit to buffer the entire request body into memory so that it can log. That's what the call to readBodyToBytesIfNecessary in the stack trace is doing.

Enabling logging like this should only be done when debugging.

[sportwtany]

JakeWharton, disabling logging help me. Thank you! :)

[joshuapinter]

:thumbsup: Wow, this saved me. Thanks guys.

[schilakamarri]

Helped me too.Thank you very much!

[WOSHICAIXIANFENG]

@JakeWharton I also find this issue when I download big file from the backend server. I already disabled the log.

My inferface:
@Headers("Content-Type:application/octet-stream")
@GET("/companies/{companyId}/media/{timestamp}")
void getAllFilesOfCompany(@Path("companyId") long companyId, @Path("timestamp") long timestamp, Callback callback);

My RestAdapter:

Log:
09-24 16:34:22.818: E/dalvikvm-heap(21899): Out of memory on a 16773136-byte allocation.
09-24 16:34:22.818: I/dalvikvm(21899): "Retrofit-/companies/26/media/0" prio=5 tid=15 RUNNABLE
09-24 16:34:22.818: I/dalvikvm(21899): | group="main" sCount=0 dsCount=0 obj=0x4359f508 self=0x6414f7b8
09-24 16:34:22.818: I/dalvikvm(21899): | sysTid=22177 nice=10 sched=0/0 cgrp=apps/bg_non_interactive handle=1679096840
09-24 16:34:22.818: I/dalvikvm(21899): | state=R schedstat=( 5671990000 4517816000 17493 ) utm=447 stm=120 core=0
09-24 16:34:22.818: I/dalvikvm(21899): at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:~91)
09-24 16:34:22.818: I/dalvikvm(21899): at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:201)
09-24 16:34:22.818: I/dalvikvm(21899): at retrofit.Utils.streamToBytes(Utils.java:44)
09-24 16:34:22.818: I/dalvikvm(21899): at retrofit.Utils.readBodyToBytesIfNecessary(Utils.java:81)
09-24 16:34:22.818: I/dalvikvm(21899): at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:343)
09-24 16:34:22.818: I/dalvikvm(21899): at retrofit.RestAdapter$RestHandler.access$100(RestAdapter.java:220)
09-24 16:34:22.818: I/dalvikvm(21899): at retrofit.RestAdapter$RestHandler$2.obtainResponse(RestAdapter.java:278)
09-24 16:34:22.818: I/dalvikvm(21899): at retrofit.CallbackRunnable.run(CallbackRunnable.java:42)
09-24 16:34:22.818: I/dalvikvm(21899): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
09-24 16:34:22.818: I/dalvikvm(21899): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
09-24 16:34:22.818: I/dalvikvm(21899): at retrofit.Platform$Android$2$1.run(Platform.java:142)
09-24 16:34:22.818: I/dalvikvm(21899): at java.lang.Thread.run(Thread.java:856)

[RohitJPatil]

how do i handle that error in failure???

[manmohansoni]

Same issue I am also facing. In my case my app is feching huge json data from the server.

[swankjesse]

@manmohansoni please ask on StackOverflow. This tracker is for bug reports and feature requests only.

[manmohansoni]

@swankjesse if in a request, response string is large and retrofit gets crashed due to out of memory. I think it is a bug. Please let me know, any solution you have for it. The issue was not coming with small responses.

[artem-zinnatullin]

You're probably converting response to object in memory, if so — it's not a
bug of Retrofit, it does what you said it to do.

You can try to switch to reading InputStream from response and consume by
small chunks to avoid OOM.

On 31 Aug 2016 4:09 pm, "manmohansoni" [email protected] wrote:

@swankjesse https://github.com/swankjesse if in a request, response
request is large and retrofit gets crashed due to out of memory. I think it
is a bug. Please let me know, any solution you have for it. The issue was
not coming with small responses.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/square/retrofit/issues/540#issuecomment-243758309,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA7B3HIU5DdIcXuxZw2H-c6A7Hd9qhAfks5qlX0JgaJpZM4CGjP8
.

[manmohansoni]

@artem-zinnatullin thanks Artem. This is quit right reply, now i got the direction to move. Thanks a lot and let me try to another way to fix OOM.

[artem-zinnatullin]

Sure, np.

Another direction you can try to go into is switch to pagination requests
if possible, usually if backend can reply with some huge response backend
developers also add pagination to the API, like: &limit=10&lastId=value

On 31 Aug 2016 5:11 pm, "manmohansoni" [email protected] wrote:

@artem-zinnatullin https://github.com/artem-zinnatullin thanks Artem.
This is quit right reply, now i got the direction to move. Thanks a lot and
let me try to another way to fix OOM.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/square/retrofit/issues/540#issuecomment-243776486,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA7B3EzD8yH_KYN6I5t76P836S8UkV68ks5qlYuJgaJpZM4CGjP8
.

[ralphgabrielle]

This helps, kindly set your interceptor level to none
interceptor.setLevel(HttpLoggingInterceptor.Level.NONE);

[joshuapinter]

@ralphgabrielle That's what @JakeWharton was saying here: https://github.com/square/retrofit/issues/540#issuecomment-46934042.