Hive: Unhandled Exception: HiveError: This is an internal error.

[cyberpwnn]

Steps to Reproduce
No good way to reproduce, ends up just leaving the app in a dead-start state since a box cannot be opened. All i'm running is openBox on startup (which normally causes no problems)

E/flutter (16607): [ERROR:flutter/lib/ui/ui_dart_state.cc(148)] Unhandled Exception: HiveError: This is an internal error. Please open an issue on GitHub and if possible provide a way to reproduce this bug.
E/flutter (16607): #0      framesFromBytes 
package:hive/…/binary/frames_from_bytes.dart:22
E/flutter (16607): #1      FrameIoHelper.framesFromFile 
package:hive/…/io/frame_io_helper.dart:57
E/flutter (16607): <asynchronous suspension>
E/flutter (16607): #2      StorageBackendVm.initialize 
package:hive/…/backend/storage_backend_vm.dart:109
E/flutter (16607): <asynchronous suspension>
E/flutter (16607): #3      BoxBase.initialize 
package:hive/…/box/box_base.dart:80
E/flutter (16607): #4      HiveImpl.openBox 
package:hive/src/hive_impl.dart:76
E/flutter (16607): <asynchronous suspension>
E/flutter (16607): #5      MyGuideStorage.init 
package:app/util/myguide_storage.dart:43
E/flutter (16607): <asynchronous suspension>
E/flutter (16607): #6      main 
package:app/main.dart:21
E/flutter (16607): #7      _runMainZoned.<anonymous closure>.<anonymous closure>  (dart:ui/hooks.dart:239:25)
E/flutter (16607): #8      _rootRun  (dart:async/zone.dart:1124:13)
E/flutter (16607): #9      _CustomZone.run  (dart:async/zone.dart:1021:19)
E/flutter (16607): #10     _runZoned  (dart:async/zone.dart:1516:10)
E/flutter (16607): #11     runZoned  (dart:async/zone.dart:1500:12)
E/flutter (16607): #12     _runMainZoned.<anonymous closure>  (dart:ui/hooks.dart:231:5)
E/flutter (16607): #13     _startIsolate.<anonymous closure>  (dart:isolate-patch/isolate_patch.dart:305:19)
E/flutter (16607): #14     _RawReceivePortImpl._handleMessage  (dart:isolate-patch/isolate_patch.dart:172:12)

Version

• Platform: Android (Hardware Q) and IOS (simulator latest)
• Flutter version: v1.10.15-pre.151
• Hive version: 1.1.0-beta2

Note that the same issue happens when downgrading back to flutter stable

Finally, During this installation, I did not clear data (using box.clear) so i dont think it's related to #71

[leisim]

This seems to happen very rarely or only under certain conditions. Unfortunately I cannot do much if I'm unable to reproduce it. All the integration tests work fine...

It would be really helpful if you or someone else finds a way to reproduce this bug. I'll also try to find it. Until then you should switch to the latest stable version (1.1.0+1).

[cyberpwnn]

Is #71 resolved in 1.1.0+1?

[leisim]

Yes it is. But I don't think this issue is related.

[cyberpwnn]

Just to know when looking for a way to reproduce it, is it that the data is corrupted somehow and it cannot be read when the box is opened? or is it something else having to do with the box itself?

[leisim]

Is it that the data is corrupted somehow and it cannot be read when the box is opened.

Exactly.

There are two things that can cause this:

1. You use a custom TypeAdapter which doesn't work correctly
2. Hive has a bug which corrupts the box in cetain conditions.

[cyberpwnn]

Well im not using type adapters. Ill take a look and double check what im actually writing to boxes. Perhaps im writing something in such a way or some sort of type that is causing it. Thanks for the insight.

[leisim]

If you write an usupported type, it should fail immediately without actually persisting anything. Thanks for trying to reproduce the issue...

[dat7e]

I also encountered this on 1.1.0-beta2, after I run integration tests on simulator a few times.

[VERBOSE-2:ui_dart_state.cc(148)] Unhandled Exception: HiveError: This is an internal error. Please open an issue on GitHub and if possible provide a way to reproduce this bug.
#0      framesFromBytes (package:hive/src/binary/frames_from_bytes.dart:22:7)
#1      FrameIoHelper.framesFromFile (package:hive/src/io/frame_io_helper.dart:57:12)
<asynchronous suspension>
#2      StorageBackendVm.initialize (package:hive/src/backend/storage_backend_vm.dart:109:24)
<asynchronous suspension>
#3      BoxBase.initialize (package:hive/src/box/box_base.dart:80:20)
#4      HiveImpl.openBox (package:hive/src/hive_impl.dart:76:17)
<asynchronous suspension>
#5      _openBoxes (package:birdy/main_delegate.dart:76:10)
#6      _asyncThenWrapperHelper.<anonymous closure> (dart:async-patch/async_patch.dart:71:64)
#7      _rootRunUnary (dart:async/zone.dart:1132:38)
#8      _CustomZone.runUnary (dart:async/zone.dart:1029:19)
#9      _FutureListener.handleValue (dart:async/future_impl.dart:137:18)
#10     Future._propagateTo<…>

[leisim]

@dat7e Do you run Hive integration tests or your own? Would you be willing to share a test which fails for you?

[dat7e]

Sorry I haven't been clearer. I ran my own integration tests (e2e to be more exact) which I wrote for my app. The tests are very simple ones that fill out and submit a login form. When valid credentials are given, the user is redirected to a news feed which fetches data via http and caches the news items using Hive box.putAll. I know these tests should pass. However, after a few times they failed because the app booted to a blank screen with the above error message printed to console. Since then I have deleted the app on the simulator, now my tests run fine.

This is how I open the boxes:

Future<void> _openBoxes() async {
  final dir = await getApplicationDocumentsDirectory();
  Hive.init(dir.path);

  await Future.wait([
    Hive.openBox('settings'),
    Hive.openBox('stories'),
  ]);
}

Future<void> main() async {
  await _openBoxes();
  // other setup tasks
  runApp(MyApp());
}

[leisim]

Would it be possible to share some code that reproduces the problem? I have a hard time to reproduce the issue myself...

[dat7e]

Currently no, sorry. It's under NDA and all that jazz. However the feature I tested is small and I might be able to extract it to a small project, hopefully in a few days. Will also try to find out how to consistently reproduce the issue. Right now it looks to me a lot like #71, could these two be related in any way?

[leisim]

Yes I think they have the same root cause. Please upgrade to 1.1.1 to make sure the issue is not solved already.

Thank you very much for your help :)

[Gorniv]

I have the same error at 1.1.1 after crash at debugging. I don't know how to reproduce it.

[dat7e]

my colleague encountered this on iOS simulator today, hive version 1.1.1

[cyberpwnn]

I'm opening boxes on startup, One box specifically keeps failing. Im just throwing stuff out here but in my case it's the "state" box which is written to very frequently.

Could it be possible that data isnt being written correctly, or is being written quickly (asyncs) that is causing the box to become corrupted? I have a few other boxes loaded before the state box on startup and they never have issues.

[leisim]

I still have no idea what might cause this problem :/ All the unit tests write a huge amount of entries very quickly and they never corrupted a box.

[cyberpwnn]

Ill be working on something hotloading etc for about 30 minutes, then at some point, as usual something wont hotload, so i restart the app and it fails to launch (black screen) with the same error.

I have no way to reproduce this and i've never seen it happen outside of hotloading / development, but Im pretty much forced to make a wrapped storage interface in the event that this problem does happen on production.

Clearly something is being written or in a specific way that is corrupting it, but that is not enough information for a fix. I cannot reproduce it 100% it just happens randomly during development.

[leisim]

I have a really hard time finding this bug and no idea where to start.
I'm starting to believe that it actually has something to do with hot reloading. Maybe a write operation is interrupted and the bug is in the Hive corruption recovery.

[cyberpwnn]

That could be possible. With the very little testing i've actually done with production & whatnot i cant be sure.

I have made a temporary bandaid fix for now to basically wipe the hive file that gets corrupted and create a new box (the only box failing is the one i use for network caches, so re-creating it isnt so bad)

I will be doing some legit tests pretty soon with others, I'll let you know if anything comes up. But I agree, I think it has something to do with hotloading, I have only seen it after restarting the application (after rapid hotloading doesnt update the app).

[leisim]

Thanks for your help!

[SergeShkurko]

I have a similar problem when I develop on windows for Android. When I write on Mac OS for iOS there is no problem

[leisim]

I have a similar problem when I develop on windows for Android. When I write on Mac OS for iOS there is no problem

You mean in an emulator or with Flutter desktop?

[SergeShkurko]

android ios emulators

[leisim]

Okay, thanks! If find a way to reproduce it, I would be grateful for a hint or demo...

[markmooibroek]

I had one of these in my sentry logs today (using hive 1.1.1). I cleared the box file with clearing the app data, and it worked again. I am using 3 generated type adapters, but nothing fancy. If I can get this reproduced in a simple way I'll make sure to post it here. These are the harder nuts to crack.

frame_io_helper.dart in _KeyReader.readKeys at line 66
<asynchronous suspension>
frame_io_helper.dart in FrameIoHelper.keysFromFile at line 30
<asynchronous suspension>
storage_backend_vm.dart in StorageBackendVm.initialize at line 122
<asynchronous suspension>
box_base.dart in BoxBase.initialize at line 82
hive_impl.dart in HiveImpl.openBox at line 87
<asynchronous suspension>
games_repository.dart in GamesRepository.getSavedGame at line 10
<asynchronous suspension>
game_event.dart in SetActiveGame.handle at line 78
<asynchronous suspension>
base_model.dart in BaseModel.dispatch at line 17
game_state.dart in GameState.setGlobalState.<fn> at line 64
zone.dart in _rootRunUnary at line 1132
zone.dart in _CustomZone.runUnary at line 1029
zone.dart in _CustomZone.runUnaryGuarded at line 931
stream_impl.dart in _BufferingStreamSubscription._sendData at line 336
stream_impl.dart in _DelayedData.perform at line 591
stream_impl.dart in _StreamImplEvents.handleNext at line 707
stream_impl.dart in _PendingEvents.schedule.<fn> at line 667
zone.dart in _rootRun at line 1120
zone.dart in _CustomZone.run at line 1021
zone.dart in _CustomZone.runGuarded at line 923
zone.dart in _CustomZone.bindCallbackGuarded.<fn> at line 963
zone.dart in _rootRun at line 1124
zone.dart in _CustomZone.run at line 1021
zone.dart in _CustomZone.runGuarded at line 923
zone.dart in _CustomZone.bindCallbackGuarded.<fn> at line 963
schedule_microtask.dart in _microtaskLoop at line 41
schedule_microtask.dart in _startMicrotaskLoop at line 50

All relevant box operations:

```dart
Future getSavedGame(String uri) async {
var gameBox = await Hive.openBox("games", lazy: true) as LazyBox;
var savedGame = await gameBox.get(uri);

if (savedGame != null) {
  return savedGame as SavedGame;
}else {
  return null;
}

}

Future saveGame(LocalGame game, BoardState currentBoardState) async {
var gameBox = await Hive.openBox("games", lazy: true) as LazyBox;
var savedGame = SavedGame.from(game, currentBoardState);
await gameBox.put(game.uri, savedGame);
return;
}

Future deleteAllSaveGames() async {
var gameBox = await Hive.openBox("games", lazy: true) as LazyBox;
await gameBox.clear();
return;
}
```

[leisim]

@markmooibroek Thanks!

[trousev]

I have same issue, for some unknown reason one of my boxes is currently corrupted and I'm receiving this:

E/flutter (20821): [ERROR:flutter/lib/ui/ui_dart_state.cc(144)] Unhandled Exception: HiveError: This is an iternal error. Please open an issue on GitHub and provide steps to reproduce this problem if possible.

when I'm trying to open flutter box.

The real problem is that I can't delete box without attempt to open it, which will fail, so, I can't recover application into working state without removing database files directly (which names I'm not supposed to know).

Maybe it worth implementing something like

Hive.removeBoxWithoutOpeningIt('myBox');

?

[leisim]

@trousev I agree that there should be an option to delete a box without opening it. I'll implement this in the next version (which should probably be published next week)

But the real problem is that I'm still unable to reproduce the bug :(

[dahabit]

I got the same error Today Android/ios simulators, and the error disappear when I changed the box name
it's very important to us to be able to remove box without opening it

[marcelgarus]

I got the same error. It's not really deterministically reproducible but it happens quite often in our project – just by clicking around in different sections I'm able to reliably run into this error in less than a minute.

[leisim]

just by clicking around in different sections I'm able to reliably run into this error in less than a minute.

Thanks, I'll test that...

[leisim]

I have made some progress: Thanks to the schul-cloud project from @marcelgarus I believe the problem is related to the compaction of boxes.
Could someone maybe test if the error still happens with the following code:

Hive.openBox('yourBox', compactionStrategy: (a,b)=> false);

[marcelgarus]

I tried the workaround and it seems to fix the error. 👍

[wisecoders]

I am also facing the same error in ios as well android. I have added your workaround. but I have to uninstall the app and reinstall. I will let you know how it goes.

[SergeShkurko]

@leisim I have assumptions about this. I’ll try to explain with an example: I have an application that simultaneously accesses a file from several different sections of code, which periodically caused the content in file incorrect, because during the recording of the first request, the recording of the second one begins, later the contents of the file were incorrect

I think that here the problem can be repeated and to solve it, you can use a such solution: https://github.com/tekartik/synchronized.dart

[wisecoders]

Still having issues. I believe when you are using the same box across multiple screens, it causes issues.

Is it ok to use the Same box with watchedbox in two different screens?

[leisim]

@wisecoders

Still having issues.

Even when you disable compaction and clear the cache?
If yes, it would be very helpful if you could share some code...

Is it ok to use the Same box with watchedbox in two different screens?

Absolutely.

[wisecoders]

Yes...even after disable compaction, i am getting this issue.

It is proprietary code :-(. I can do a screenshare with you. Any email ids?

[wisecoders]

Is it possible to detect exception , recreate the box and open it again.

[wisecoders]

I have modified my code to detect exception and delete the box file (say my_box.hive) and lock file (my_box.lock) and trying to open the box fresh. I will keep you posted about how it handles.

@leisim Do you think if this is the right approach?

[wisecoders]

@leisim Any luck? I am stuck. my app build around hiveDB. :-(

[ueman]

Could it be, that Hive gets initialized before you call runApp()? Because it could be that path_provider doesn't already works correctly. So then you have to call WidgetsFlutterBinding.ensureInitialized(); as first in your main method.

Edit:
After looking through your code @marcelgarus it seems to be the case. cc @leisim

Edit 2:
After some more digging this could really be the reason.
As per https://github.com/flutter/flutter/issues/27642 Method Channels don't work before calling runApp() or WidgetsFlutterBinding.ensureInitialized();. So path_provider returns null as path. This results in Hive being initialized with a null path for which it doesn't have a check. And then after being used it can't do anything but crash.

So Hive should check against null as path. If you use Flutter you should call WidgetsFlutterBinding.ensureInitialized(); before initializing Hive with Hive.init(path).

[wisecoders]

@ueman I am initializing the Hive on Main and Opening 5 Boxes in the process. Out of 5, only one is always failing. all others are working as expected.

[alaindixon]

Hi,

I got the same issue :( on my IOS simulator. It's weird, it was working before.
I've tried the solutions above, it does not work for me.

Could it be a file corruption?
Because I got this error right after I did record new fields.

Alain

[leisim]

Good news. I found the problem and I'm working on a fix. The problem is that compaction doesn't happen when it is supposed to.

If disabling compaction (the workaround above) does not work for you, please open another issue.

@alaindixon This sounds more like a problem with the added fields. Did you follow the guide on adding fields in the docs?

[OliverWelther]

Hi,

I'm having the same problem and I've created a minimal sample to reproduce it reliably (at least on Android and iOS).

The following steps will throw an Exception on the latest version of the master branch (5605d1c):

1. Open a box
2. Put at least 64KB of data in the box
3. Delete any entry in the box
4. Compact and re-open the box.

Step 4 will throw the exception. It's working fine if you don't compact the box or don't delete any entry.

I've tried to find the root-cause, but I could not find the exact line of code. It is definitely related to the BufferedFileReader. If you replace it in the compact function with a simple RandomAccessFile that always moves to the correct offset and then reads the frame, then everything works.
It probably depends on when the BufferedFileReader loads more data (loadBytes) and there is still unread data in the buffer. That's why you have to write at least 64KB to the box to reproduce it. This corresponds to the defaultChunkSize of the reader.

Here is a minimal Flutter app for reproduction. I'm on the latest flutter master branch.

```import 'dart:io';
import 'package:flutter/material.dart';
import 'package:hive/hive.dart';
import 'package:path_provider/path_provider.dart';

void main() async {
WidgetsFlutterBinding.ensureInitialized();
var docs = await getApplicationDocumentsDirectory();
Hive.init(docs.path);
runApp(MyApp());
}

class MyApp extends StatelessWidget {
@override
Widget build(BuildContext context) {
return MaterialApp(
theme: ThemeData(primarySwatch: Colors.blue),
home: HomePage(),
);
}
}

class HomePage extends StatelessWidget {
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(
title: Text('Hive Test'),
),
body: Center(
child: RaisedButton(
child: Text('Run test'),
onPressed: () => _reproduceCrash(),
),
),
);
}

Future _reproduceCrash() async {
try {
var box = await Hive.openBox('box');
// replace the (x) => x with (x) => '1234' and you'll get a "Recovering corrupted box" hint instead
var oneKb = List.generate(128, ((x) => x));

  // write at least 64KB to the box
  // decrease this value to 61 or below and the test will run
  await box.putAll({for (int i = 0; i < 64; i++) i: oneKb});

  // now delete at least one entry. Doesn't matter which
  await box.delete(0);

  await box.compact();
  await box.close();

  // will throw an "This is an internal error" Exception
  box = await Hive.openBox('box');
  await box.close();
} catch (e, s) {
  print('$e\n$s');

  // Without deleting the box, we cannot re-run the test.
  // Not sure why, but Hive.deleteBoxFromDisk with also throw.
  var docs = await getApplicationDocumentsDirectory();
  await File('${docs.path}/box.hive').delete();
}

}
}
```

[leisim]

@OliverWelther I don't know how to thank you. I really works and I'll look for the problem right now.

[leisim]

Thanks to @OliverWelther I finally found the problem. I used BytesBuilder(copy: false) instead of BytesBuilder(copy: true) in the BufferedFileWriter. The data in the BytesBuilder was only a view of the data in BufferedFileReader. In some situations this would lead to wrong data in the BytesBuilder.

Edit: I will publish a test version of 1.2.0 which includes this fix shortly.

[MichealReed]

@leisim is there a branch containing the fix anywhere? Running into the same issue here.

[leisim]

@MichealReed Are you using version 1.2.0-dev2?

[MichealReed]

I did not notice the dev packages. Thank you!

[MichealReed]

Update: I can confirm that the issue persists on 1.2.0-dev2. I encounter it occasionally after a hot restart and have to wipe the app's cache.

[leisim]

@michaelreed, please use v1.2.0, it is impossible that you encounter that issue on the stable version.

[MichealReed]

Upgraded to v1.2.0, thanks.

[MichealReed]

Got the error again on v1.2.0 stable.

[MichealReed]

@leisim I encountered this again today and have had reports from others of the same happening. Only way to remedy is to delete the app's data.

[leisim]

@MichealReed you're the only one I know of who still has this problem :/ (the others just forgot to update to the latest version). It's very hard to diagnose this issue without additional information / code.

[MichealReed]

We are both definitely on 1.2.0. It is much less frequent, but still happens. Not sure what code to provide as I'm not sure what exactly triggers it, but I have only seen this on Android so far. I would suspect that the majority of people using hive have not upgraded and this issue will mainly surface on new implementations/projects.

[ReniDelonzek]

I had similar problems when trying to start Hive twice or when trying to open an already opened box. My solution was to create a singleton that guarantees the handling of that. I think it would be interesting to have that at the library level