Realm-java: Exception: RealmMigrationNeededException: RealmMigration must be provided

[MiralDesai]

Goal

What do you want to achieve?

Run migrations and have them work correctly.

Expected Results

I have already provided a migration, yet I'm still getting exceptions.

Actual Results

Caused by io.realm.exceptions.RealmMigrationNeededException: The 'Auth' class is missing from the schema for this Realm. at io.realm.AuthRealmProxy.validateTable(SourceFile:171) at io.realm.DefaultRealmModuleMediator.validateTable(SourceFile:117) at io.realm.Realm.initializeRealm(SourceFile:349) at io.realm.Realm.createAndValidate(SourceFile:314) at io.realm.Realm.createInstance(SourceFile:265) at io.realm.RealmCache.createRealmOrGetFromCache(SourceFile:143) at io.realm.Realm.getDefaultInstance(SourceFile:210)

Steps & Code to Reproduce

In the latest version of my app, I add a new object called "Auth". I bumped the schema version to 6 and added some lines in my migration helper class to get that done too. Here is how the migration goes:

This is all inside the migrate(DynamicRealm realm, long oldVersion, long newVersion) method:

RealmSchema schema = realm.getSchema();
if (oldVersion == 5) {
            schema.create("Auth").addField("token", String.class);
            oldVersion++;
        }

Why is the migration failing? i'm also getting a exception for another object where I added a field, and it's expecting 9 but got 8 fields. That shouldn't be the case either because i fixed that in my migration. To me it seems like the migrations are simply failing.

I remember getting this exception myself when testing because I didn't add the migration, but before I released it, I added the migration in and testing it again.

Version of Realm and tooling

Realm version(s): 3.0.0

Realm sync feature enabled: no

Android Studio version: Latest stable release.

Which Android version and device: A range, Android 5, 7, 8. Samsung, Google, LG etc

[Zhuinden]

What was the previous schema version and what is the new schema version of the configuration?

[MiralDesai]

Previous schema version was 5, new one is 6.

I also have if blocks similar to those for all schema versions from 1 -3 that are just empty with oldVersion++, to ensure someone coming from an old version of the app, will still go through all migration steps.

[beeender]

If you set a breakpoint at schema.create("Auth").addField("token", String.class);, will it stop and create the RealmObjectSchema successfully?

[MiralDesai]

I did the above and the schema was updated to include Auth, including the field name.

oldVersion = 5;
newVersion = 6;
schema.get("Auth").getFieldNames() //returns Set of size 1 with token field

I always test the upgrade path before releasing to production, and at least on my side it has worked numerous times. However I'm still getting these crash reports.

[beeender]

where did you call Realm.setDefaultConfiguration()? When you call it, is the migration a part of the configuration?

Please notice setDefaultConfiguration() is not a thread safe call, so you need to make sure it is called before any other threads calling Realm.getDefaultInstance().

[MiralDesai]

I have a class for encrypting my realm, where i set the default configuration. I call it in a few places, including my Application class.

RealmConfiguration encryptedConfig = new RealmConfiguration.Builder()
                .name("name")
                .schemaVersion(6)
                .encryptionKey(hash)
                .migration(new RealmMigrationHelper())
                .build();
        Realm.removeDefaultConfiguration();
        Realm.setDefaultConfiguration(encryptedConfig);
        return encryptedConfig;

The migration is part of the configuration. I definitely call Realm.getDefaultInstances() before I set the above encryptedConfig, but as you can see I remove the previous default configuration first. I do it so that I can get the file name of the previous config, and check if it's encrypted or not. So then i know whether I need to do a migration from unencrypted to encrypted, or simply just set the encrypted config again.

Edit: Feels like I'm in a bit of a loop because I have to query my database before encrypting it.. which of course requires Realm.getDefaultInstance();

Technically, how does this negatively impact the migration?

[beeender]

without the code to show the entire logic, it is hard to tell what is wrong with that.

One possible case could be:

// Config without a migration block
RealmConfiguration config1 = new RealmConfiguration.Builder()
                                    .name("same_name")
                                    .schemaVersion(1)
                                    .build();

// Config with a migration block
RealmConfiguration config2 = new RealmConfiguration.Builder()
                                    .name("same_name")
                                    .schemaVersion(1)
                                    .migration(myMigration) // This config has a migration block
                                    .build();

// If the db file has been migrated to right schema, then this will succeed. But if it hasn't, exception will be thrown.
Realm.getInstance(config1);


// If the db file has been migrated to right schema, then this will succeed. If it hasn't, migration will be thrown.
Realm.getInstance(config2);

So some questions:

1. How many configurations have you created in total?
2. Which thread do you call setDefaultConfiguration() on?
3. The code piece shows that how do you convert the realm from unencrypted to encrypted?

[MiralDesai]

I only have two configurations. It's background thread, in it's own class. One that has a migration, and is encrypted, and another which is not encrypted, and had deleteRealmIfMigratationNeeded() applied to it.

This is essentially what I do on launch and in 2 other places:

RealmConfiguration unencryptedConfig = Realm.getDefaultInstance().getConfiguration();
        File file = new File(unencryptedConfig.getPath());
    if (!file.getName().equals("encrytedrealm")) { 
        //Set the default config that I mentioned above.
    } else {
        //encrypted realm doesn't exist yet, copy realm to it, and set the encrypted config set above
    }

So via Realm.getDefaultInstance() I'm checking if the realm is already encrypted, if it is, set the config, if it isn't migrate to it. The new config includes the database migration.

I think I can see one the issue here at least, since the Auth class is required straight away, before this code above might have been run, the schema not going to include the Auth class when I require it. I can find a way to fix that, by checking schema version or something.

I'm inclined to believe this is the problem.

To answer your last question, I migrate from unencrypted to encrypted by copying the realm with the writeEncryptedCopyTo() method provided by realm.

[beeender]

One more thing not related with the original issue, Realm.getDefaultInstance().getConfiguration(); will create a anonymous Realm instance which is not closed properly. That may cause leaks.

[MiralDesai]

It creates a default realm right? That's the experience I had when debugging. I also check for that and delete it.

Anyway thanks, noted it.

[MiralDesai]

My fix for this is to add the realm migration to the default config that gets created when I call Realm.getDefaultInstance().getConfiguration();. That default realm will obviously not do a migration right now, but if I had it in, it should run correctly, up until the point I create the encrypted config, where I can delete it.

Can someone clarify if that is the expected behavior should I take this route?

[Zhuinden]

@cmelchior I can't help but think that setting a default configuration in Realm.init() was a mistake.

[beeender]

If you never set default configuration through Realm.setDefaultConfiguration(), the default configuration will be: new RealmConfiguration.Builder(context).build() which doesn't have a migration block.

[MiralDesai]

That's what I mean though, I'll create a default configuration before calling Realm.getDefaultInstance().getConfiguration();that includes a migration block.

new RealmConfigurationBuilder(this).migration(new Migration()).build();

I also agree with @Zhuinden, Realm.init(), should ideally not set a default configuration, instead developers should be forced to add their own configuration. I don't see anything wrong with forcing developers to do such a thing, there's a good change they're already creating a configuration like I am.

[cmelchior]

The choice of setting a standard default configuration was to enable users to get started _really_ quickly. This is still a nice goal to have I think, but perhaps we could improve the error messages, e.g if an exception is thrown while the Realm provided default configuration is used we could mention that. Something like <OriginalError>. The Realm provided default configuration was used. Most likely you forgot to set your own using Realm.setDefaultConfiguration(RealmConfiguration)

[MiralDesai]

I guess I'm a bit bias since I've been using Realm for a while now and setting a configuration is painless to me, and something I will always do.

[Zhuinden]

@cmelchior I think the problem with setting default config in init() is when you need to set one either way once you need to add migrations, in which case it seems that if you hadn't set the default config properly, then you'll just run into a problem that you think you are setting the migration in the right place, but then you realize you aren't?

Also I'm not sure Realm.setDefaultConfiguration() is thread-safe, is it? I should check if it's volatile - it's something that seems like it should be considering Realm.getDefaultInstance() is used on multiple threads.

[cmelchior]

Realm.setDefaultConfiguration should be threadsafe, at least it is behind a lock from a quick glance.

My main problem with requiring people to use Realm.setDefaultConfiguration() is that it is yet another thing you have to remember to get started with Realm. In my perfect world, you would just do Realm.getDefaultInstance() and it would work out of the box.

There is a natural tension between making it easier for new people and "power" users though.
I think my main point is that, _if_ you forget to set the default configuration you would get an exception either way.

If it is a requirement you would get something like IllegalStateException = No default configuration set yet

If it is optional you would get something like RealmMigrationNeededException = <Some migration error>. Did you remember to set your own default configuration?.

In both cases, it would crash if you forget setting it. Bot having it optional would make it work in more cases. This comes at the expense of a slighly more conveluted error message, but I can live with that since that error would only hit users that have used Realm for a while.

One of the primary design drivers we have, is that new users should be able to pick up Realm as easy as possible

I hope that make sense?

[MiralDesai]

The annoying thing I've come across is that I now _always_ have to transfer from an unencrypted realm to an encrypted one. I can't check on launch if the current realm is encrypted because that requires calling Realm.getDefaultInstance().getConfiguration(); which in itself might create a default realm, without any migrations.

So on launch I have to create the default.realm with a migration, and then migrate to encrypted. There is a security flaw here where for a moment, I have an unencrypted realm on launch.

Also by creating a default.realm config everytime, surely that negates migrations? I mean a new default.realm file will be created, with the latest schema, which will then be copied to an encrypted version. All data within the realm will be deleted since we started with a brand new file right?

Ultimately I feel I'm stuck, and cannot fix this issue.

[cmelchior]

We added Realm.getDefaultConfiguration() in 3.4.0, which just returns the reference. It doesn't create any Realms.

[MiralDesai]

Thanks, I'll give it a try.

[MiralDesai]

It works when doing a fresh install, however when upgrading from the previous version of the app, the user gets logged out. This is because of Realm.init() creating a default realm, and clearing out everything in my database. I'm then left with an entirely empty default realm.

Surely this isn't the expected behaviour? Is this a related issue or more due to upgrading the version of realm I'm using from 3.0.0 to 3.4.0? Maybe the file format change..? Something I've never seen before.

[cmelchior]

Realm.init() doesn't create any Realms. It just creates the default Realm configuration using setDefaultConfiguration(new RealmConfiguration.Builder(context).build());

So if the Realm is deleted, it must be some other code doing it.

[MiralDesai]

Slight confused, surely when I call Realm.getDefaultInstance() in the very first class, it will then create that realm with the default config? I guess that's the second part to it as discussed above.

I'm saving the user's access token in the Auth class I mentioned above, and when I launch the app via an upgrade over the previous version, that class is now null. Not something I've seen before. Normally app upgrades go just fine.

The only realm related code is Realm.Init() and a query to get the access token from the database to check if the user is authenticated or not.

[cmelchior]

Yes, Realm.getDefaultInstance() will create the Realm using whatever config is configured as the default, but if something it deleted, it must be because a custom configuration is used. Realms default configuration will throw a RealmMigrationNeededException if anything is wrong with the Realm on disk

[MiralDesai]

OK I see, my previous realm file is _not_ deleted. It's there, however it's not being used, a default.realm is created and my app is now using that. There are now two realm files. I'm not really sure why.

Why is my configuration from the previous version of the app not used, but a brand new realm has now been created? Surely Realm.init() is not overwriting my realm configuration, or does it always create a configuration and set it as the default?

Edit: Sorry poor choice of words, updated

[MiralDesai]

Ok I've fixed the issue it seems, still debugging so I won't confirm just yet, will update if anything changes. So far the change I have made is to set a custom configuration in my application class.

[kneth]

@MiralDesai Thanks. I'll leave the issue open.

[MiralDesai]

I sent out an update with the fix (Adding the migration class to the default realm config) and I'm still getting the same crash reports. Two came through today.

One being the same as in my original comment: The 'Auth' class is missing from the schema for this Realm.

The other however was from a different object (User) and got me thinking, the error was Field count is less than expected - expected 6 but was 4

Now when I implemented Realm back in February I didn't have migrations, I had deleteRealmIfMigrationRequired();. The user class had 4 fields back then. Now it has 6. Bare in mind I cannot be sure that the crashes are coming from users who were on very old versions and upgrade to the latest, in fact I would bet against that being the case. So not 100% sure on my theory.

Anyway keeping with that point, is it possible that these crashes are because I need to include the full migration, from day one in my migration class, even though I didn't do them back then?

I'm still not sure why the Auth object might be missing though, because I definitely have a migration for that class.

Hopefully this is enough information.

[kneth]

@MiralDesai You will need to include a full migration. If a user has the initial version of the app installed, and she upgrades to the latest, the Realm on disk will have a model class with 4 fields. But the app's model class has 6 fields. The exception thrown is therefore absolutely correct.

[MiralDesai]

Been a while since I was off sick. I have now updated our app to have a full migration, yet I'm still getting the dreaded RealmMigrationNeededException. This time I was able to track how the user upgraded, and the state of the code.

The exception is as follows (It's the same one from my previous post about the user object:
Caused by io.realm.exceptions.RealmMigrationNeededException: Field count is less than expected - expected 6 but was 4

The user was on version 2.3.0 of the app and the Realm config looked like this:

Realm.removeDefaultConfiguration();
        RealmConfiguration configuration = new RealmConfiguration.Builder()
                .name("name")
                .schemaVersion(4)
                .deleteRealmIfMigrationNeeded()
                .build();
        Realm.setDefaultConfiguration(configuration);

The user updated to the latest version where the the realm config is the one posted above:

RealmConfiguration encryptedConfig = new RealmConfiguration.Builder()
                .name("name")
                .schemaVersion(6)
                .encryptionKey(hash)
                .migration(new RealmMigrationHelper())
                .build();
        Realm.removeDefaultConfiguration();
        Realm.setDefaultConfiguration(encryptedConfig);
        return encryptedConfig;

The main problem with all of this is that the user object is identical between then and now. It has 6 fields. We updated from 4 fields to 6 before version 2.3.0 and deleteRealmIfMigrationNeeded() should have dealt with that right?

Am I missing something? I'm really confused as to how the realm on disk could different to the model class since nothing changed between those versions.

Just a side note, I appreciate the help given here, I know it's dragging on a bit, but genuinely perplexed about this.

[dalinaum]

Hello @MiralDesai

Does your migration start with if (oldVersion == 5) {? If it starts with oldVersion == 5 and user's version is 4, it will not migrate to new version.

Otherwise, can you upload your full migration code here?

[bmeike]

Hey @MiralDesai ,
Did that fix it? Are you good to go?

-blake

[MiralDesai]

I started my migration from version 1 of the schema, all the way to 5. I think my main issue is that at some point I updated an object without bumping the schema version and just let deleteRealmIfMigrationNeeded() work away.

I am happy to close this because we have come up with another solution (deprecating old versions) that will prevent this in the future. Luckily we're at a point where not many users are affected.

It might be nice in the future, to be told exactly which fields are causing the issue in the exception that is thrown? That would make things a lot easier to debug.

[Zhuinden]

@MiralDesai luckily when https://github.com/realm/realm-java/pull/4887 is merged, that will happen

[MiralDesai]

That's great to see. Will close this and and look out for that.