Realm-java: Realm storage data is too large

[nthuat]

I've encountered a problem about storage. Please help me to resolve it.

Goal

Reduce data storage usage

Steps & Code to Reproduce

This is "default.realm" I got it through the instruction. Size is: 67,1MB

But, System Storage shows it used 203 MB. Too much!!!

Version of Realm and tooling

Realm version(s): 2.2.1

Realm sync feature enabled: no

Android Studio version: 2.2.2

Which Android version and device: Android 6.0.1, HTC Desire 630 dual sim.

[cmelchior]

@thuat26 Have you tried running Realm.compactRealm(config)? In some situation Realm will allocate more space than actually needed. See more here: https://realm.io/docs/java/latest/#large-realm-file-size

[nthuat]

I've known that method. But my data is encrypted and the document says: "Compacting an encrypted Realm file is not supported yet. "

        Realm.init(context);
        RealmConfiguration configuration = new RealmConfiguration.Builder()
                .encryptionKey(KEY)
                .schemaVersion(8)
                .migration(new Migration())
                .build();
        Realm.setDefaultConfiguration(configuration);

Is there another method for my case?

[Zhuinden]

Compacting encrypted realm is possible since 2.1.0

[Zhuinden]

But you probably just don't close a Realm instance on a background thread. That's where all of these "realm file is too big" questions come from.

[beeender]

@thuat26 https://github.com/realm/realm-java/issues/3860#issuecomment-263808334 What @Zhuinden said is probably your problem. Normally there is no need to call compactRealm() if all the Realm instances in the background threads are closed peacefully -- Realm will reuse those spaces.

[nthuat]

My background thread imports data to realm from big file. This is my code. The file has about 162k lines.

try {
            FileInputStream fileIn = new FileInputStream(compressedFile);
            GZIPInputStream gZIPInputStream = new GZIPInputStream(fileIn);
            BufferedReader br = new BufferedReader(new InputStreamReader(gZIPInputStream));
            String line;
            final DecodedPlaceDeserializer decodedPlaceDeserializer = new DecodedPlaceDeserializer();
            Place place = null;
            List<Place> places = new ArrayList<>();
            final Realm realm = Realm.getDefaultInstance();
            while ((line = br.readLine()) != null) {
                place = decodedPlaceDeserializer.decodeString(line);
                if (place != null) {
                    places.add(place);
                    if (places.size() > 1000) {
                        realm.executeTransaction(new Realm.Transaction() {
                            @Override
                            public void execute(Realm realm) {
                                realm.insertOrUpdate(places);
                                places.clear();
                            }
                        });
                    }
                }
            }
            br.close();
            gZIPInputStream.close();
            fileIn.close();
            Log.d("DownloadPackage", "The file was decompressed successfully!");
            deleteAllFiles();
        } catch (Exception ex) {
            Log.d("DownloadPackage", "Ex: " + ex);
        } finally {
            realm.close();
        }
    }

Are there any problems about above code?

[Zhuinden]

Yes, you are inserting every element in a new transaction instead of inserting all elements in one transaction

[nthuat]

I tried to insert all elements in one transaction. But, it's block another transaction I want run in main thread. Example, I want login to Facebook and save access token to realm while importing database. If creating one transaction, UI will be blocked. This is old my code:

try {
            FileInputStream fileIn = new FileInputStream(compressedFile);
            GZIPInputStream gZIPInputStream = new GZIPInputStream(fileIn);
            BufferedReader br = new BufferedReader(new InputStreamReader(gZIPInputStream));
            String line;
            final DecodedPlaceDeserializer decodedPlaceDeserializer = new DecodedPlaceDeserializer();
            Place place = null;
            final Realm realm = Realm.getDefaultInstance();
            realm.beginTransaction();
            while ((line = br.readLine()) != null) {
                place = decodedPlaceDeserializer.decodeString(line);
                if (place != null) {
                    realm.insertOrUpdate(place);
                }
            }
            realm.commitTransaction();
            br.close();
            gZIPInputStream.close();
            fileIn.close();
            Log.d("DownloadPackage", "The file was decompressed successfully!");
            deleteAllFiles();
        } catch (Exception ex) {
            Log.d("DownloadPackage", "Ex: " + ex);
        } finally {
            realm.close();
        }

[Zhuinden]

You should not write on the UI thread, you should use executeTransactionAsync() or move the write transaction to a background thread

[nthuat]

Of course, I know this. My app has a scenario: My background thread runs on a service. It will live until importing data successfully even closing app. And I want to save profile info to realm and show immediately when I reopen app, ex: user profile, access token, etc. If I use executeTransactionAsync() or move the write transaction to a background thread, it must wait until importing data finishes. Then I quit and reopen app, background task does not finish, how can I get my profile info?

[Zhuinden]

You could for example do the write in an IntentService, and listen for when the operation is complete using a RealmChangeListener

[beeender]

@thuat26 How big is your data to be imported? if it is 162k lines and 1k chars in one line, maybe 203MB is a reasonable Realm db file size?

[nthuat]

@Zhuinden Problem is time for importing data is about 15 minutes, so user can't wait.

[Zhuinden]

I'm pretty sure you can't just freeze the UI thread for 15 minutes.

[beeender]

@thuat26 is the service the only place you modify Realm data? I am surprised if it causes a unexpected Realm file size, since every transaction will shift the Realm version to the newest and very unlikely any spaces will be wasted.

[nthuat]

@Zhuinden , @beeender Importing data task (A) will not freeze UI. My problem is another tasks need to insert to realm must wait (A) finishes. How can I insert data to Realm from many sources at a time?

[kneth]

@thuat26 In order to be ACID write transactions are blocking. So only one write transaction can be "active" and the other will wait. Multiple reads can happen at the same time.

15 minutes sound like a very long write transaction. It might be worth profiling your code to see if the decoding or inserting is the bottleneck.

[nthuat]

@kneth I've tested my code. Decoding seem very fast and encoding will be slow when the process percentage is about 60%. I think I have a "new" question: How to import large data from file into Realm effectively?

[Zhuinden]

@thuat26 http://stackoverflow.com/a/39385985/2413303

[nthuat]

public void storeBookings(final List<Booking> bookings) {
    mRealm.executeTransaction(new Realm.Transaction() {
        @Override
        public void execute(Realm realm) {
            realm.insertOrUpdate(bookings);
        }
    });
}

@Zhuinden My data set is too big (>= 500k elements). Each element is a complex realm object (A). If I create only 1 transaction for importing data, the time for this process is about 15 minutes and another realm object (B) need to be inserted into realm must wait. It seems too long for waiting. How can I insert (B) into Realm immediately while process of (A) running?

[Zhuinden]

For 500000 elements, I would definitely consider stream-based decoding into a single element, instead of obtaining a whole list of them, or creating a new object for each element. Your GC probably goes haywire during the process.

[nthuat]

@Zhuinden Thanks for suggestion. You can give more details about your idea. An example makes me feel good :)

[Zhuinden]

well you can try using realm.createAllFromJson() which does this (I haven't used it before so I cannot tell how well it works), or similar to http://stackoverflow.com/a/35350064/2413303 this jackson streaming api

When i needed this kind of streaming, I had the parsers generated by LoganSquare, I copied them out and added a single instance that is reset between saved elements and then directly inserting it to the Realm.

        @Override
        public ScheduleForChannelXML parse(JsonParser jsonParser)
                throws IOException {
            instance.setAgeLimit(null); //reset
            instance.setCategory(null);
            instance.setCategoryId(0);
            instance.setDuration(0);
            instance.setEpisodeId(null);
            instance.setIptvProgramId(0);
            instance.setOttChannelId(0);
            instance.setProgramId(0);
            instance.setScheduleId(0);
            instance.setSetRec(null);
            instance.setStartTime(null);
            instance.setTitle(null);
            instance.setIptvChannelId(0);
            instance.setRealIptvProgramId(0);
            instance.setSanomaChannelId(0);

            if(jsonParser.getCurrentToken() == null) {
                jsonParser.nextToken();
            }
            if(jsonParser.getCurrentToken() != JsonToken.START_OBJECT) {
                jsonParser.skipChildren();
                return null;
            }
            while(jsonParser.nextToken() != JsonToken.END_OBJECT) {
                String fieldName = jsonParser.getCurrentName();
                jsonParser.nextToken();
                parseField(instance, fieldName, jsonParser);
                jsonParser.skipChildren();
            }
            scheduleRepository.insertOrUpdate(realm, defaultSchedule);

and

       @Override
        public void parseField(ScheduleForChannel instance, String fieldName, JsonParser jsonParser)
                throws IOException {
            if("agelimit".equals(fieldName)) {
                instance.setAgeLimit(jsonParser.getValueAsString(null));
            } else if("category".equals(fieldName)) {
                instance.setCategory(jsonParser.getValueAsString(null));
            } else if("categoryid".equals(fieldName)) {
                instance.setCategoryId(jsonParser.getValueAsLong());
            } else if("duration".equals(fieldName)) {
                instance.setDuration(jsonParser.getValueAsInt());
            } else if("episodeid".equals(fieldName)) {
                instance.setEpisodeId(jsonParser.getValueAsString(null));
            } else if("iptvprogramid".equals(fieldName)) {
                instance.setIptvProgramId(jsonParser.getValueAsLong());
            } else if("programid".equals(fieldName)) {
                instance.setProgramId(jsonParser.getValueAsLong());
            } else if("scheduleid".equals(fieldName)) {
                instance.setScheduleId(jsonParser.getValueAsLong());
            } else if("starttime".equals(fieldName)) {
                instance.setStartTime(jsonParser.getValueAsString(null));
            } else if("title".equals(fieldName)) {
                instance.setTitle(jsonParser.getValueAsString(null));
            }
        }

But you'll see, the point is that 1 object is used for N elements

[nthuat]

@Zhuinden It looks good but my app doesn't use json format. I've created new structure for my file. Data will be decoded before being inserted into realm.
A line sample is:

1fb7e8a1-f588-4293-a2f6-f3bce468d09eⓦQOcx7VdGSs0F0XbiN7Mm4A==ⓦĐường Bình Lợi 245/30F Phường 13 Quận Bình Thạnh Thành Phố Hồ Chí MinhⓦHồ Chí Minhⓦho-chi-minhⓦVietnamⓦeeBNzWbBr7qXAtJeDm516ygjkgxEgDfno/StbJdNluU=ⓦ106.7030440830688ⓦ05081972ⓦⓦ2014-03-15T15:38:54.639Zⓦ2014-10-11T16:24:43.547Zⓦ05081972Ⓕ2016-06-07T03:47:38.414ZⒻ2

You can see more details in my file.
https://drive.google.com/file/d/0B7Mr5lOZ50jDWlRGSmhVRWZkcFU/view?usp=sharing

[Zhuinden]

Welp. Then you'll need to build your DIY state machine for it

[nthuat]

Humm, I'll keep your suggestion about jsonParser in mine.

[kneth]

I'll close the issue. Don't hesitate to ask another time. But in general, StackOverflow is a much better place to ask :-)

[Zhuinden]

You can also consider creating the Realm file and pre-shipping it as assetFile().

[marks5]

@Zhuinden
How did you implemented your scheduleRepository.insertOrUpdate(realm, defaultSchedule); ?

I made a path exactly what you did, I mean, using Gson but my problem now is, if I execute non async the GC stay cool, 40mb~ and If I do executeAsync my GC goes wul 500MB easily. Didnt crashed yet but using 500mb of memory isnt cool. Thanks in adv.

[Zhuinden]

@marks5 same as here: https://stackoverflow.com/a/39385985/2413303

so I only create 1 Realm object and then set its fields and then insert from unmanaged. It works well for objects without links.

If you use Async, then try to keep the async transaction count low.