Orientdb: Fail to restore database after forced shutdown

[matthiasbe]

OrientDB Version: 3.0.2

Java Version: jdk1.8.0_144

OS: Windows 10

Expected behavior

Be able to restart database system and retrieve database as it was.

Actual behavior

Database fails to restart with error

2018-07-27 09:30:35:874 SEVER {db=mydb} Magic number verification failed for page `0` of `myclass.cpm`.

This error displays multiple time for different classes.

Steps to reproduce

After orientdb server manually forced shutdown (closed terminal window or pc shutdown without closing orientdb process),

Extra information

When database is closed using <Ctrl-C>, some extra logs are displayed in terminal and then database can be re-opened without error.

This problem is present for 2 dev in our team (others shutdown there environment properly) with same config.

I can send starting logs by mail if needed.

[laa]

Hi @matthiasbe .

Hm... strange we have run crash tests periodically without problems.
Any way.

Could you check the same behavior on 3.0.4 version and if the issue persists, could you do the following.
Could you crash database, make a backup of the database, start server only after that and then send me logs of server start and content of the database backup.
Please notice that you have to make the backup after the crash but _before_ you start the server.
Looking forward to seeing results.

[HZooly]

Hey @laa ! 👋

I tried with OrientDB 3.0.4 version and got the same issue than with previous versions.
I've recorded a backup of my database, checking with my boss if I can transfer it to you.

More details about what I get in my Spring Tools Suite console :

Caused by: com.orientechnologies.orient.core.exception.OPageIsBrokenException: Following files and pages are detected to be broken ['account.cpm' :0;], storage is switched to 'read only' mode. Any modification operations are prohibited. To restore database and make it fully operational you may export and import database to and from JSON.

at checkReadOnlyConditions

[laa]

Hi @HZooly any updates?

[HZooly]

@laa Is it ok if I send you a corrupted Account.cpm?

[laa]

@HZooly nope, I need to check other things in the database too. OK if you can not to send me the whole DB could you send me just a server log? I probably will ask you to look on some pieces of the database after that. So you will provide me needed information without sending of the whole database.

[HZooly]

Ok sending you an email with Spring console logs and OrientDB console log.

[laa]

Hi @HZooly , if you do not perform truncate of classes but merely delete records from classes, do you still see this issue? Could you try it?

[matthiasbe]

Hi @laa
We indeed see some TRUNCATE operation at our application startup. After some investigation it appears to come from the following sql command : alter class 'myClass' abstract true (see this piece of code )

Any hint and what to use instead of type.setAbstract(true) ?
Trying to remove this and see if restoring works, and I'll get back to you.

By the way, thanks for your help !

PS : some TRUNCATE operations are also occure when importing a database

[matthiasbe]

Removing type.sebAbstract(true) didn't solve the issue

[matthiasbe]

Is it possible that it came from the following code :

Map<String, Object> props = new HashMap<>();
// Add things to the hashmap ...
OrientVertex vx = g.addVertex("class:Account", props);
g.commit();

Where :

• g is an OrientGraph comming from a orient graph factory with a connection poll
• props is a hashmap containing the vertex' properties

No issue when :

• reset the database
• run our application
• stop just before this part of the code
• shutdown orientdb server ungracefully
• restart OrientDB and run the program again
  => everything is restarted without the error.

Issue when :

• reset the database
• run our application
• stop just after this part of the code
• shutdown orientdb server ungracefully
• restart OrientDB and run the program again
  => got the Magic number verification failed.

[laa]

Hi @matthiasbe probably I got the problem, I will provide a new fix on Tuesday I suppose. No promises though depends on my load.

[laa]

Hi @matthiasbe could you try this build?

[laa]

https://drive.google.com/file/d/1xLBIckaTma387B9vrgTRjS_xknLHabX4/view?usp=sharing

[matthiasbe]

Hi @laa
I ran your build with the java API version 3.0.5.
Unfortunately, error Magic number verification failed for page '0' of 'account.cpm' still occurs after restarting ungracefully stopped orientdb server, when trying to access this Account class via Java API during startup of the application (database is switched to read-only when trying to reach the Account class)

Is there some additionnal debugging information I could provide ?

[matthiasbe]

Do I have to use the _Java API_ jars located in the lib/ folder of the archive you provided in my java project ? This is not really the way I've setup my java project, as I'm using Gradle.

[laa]

Hi @matthiasbe . A version in which I made changes is 3.3.6-SNAPSHOT. You could make a build using 3.0.x branch and check result. Just run mvn clean install -DskipTests=true

[matthiasbe]

I finally achieved to use your build in our project.

But the bug is still present 😳

At least now I know how to use a custom build

[laa]

@matthiasbe could you publish me build number just to double check?

[laa]

Build number should be printed at the servers start.

[matthiasbe]

Hi @laa

Build number : ec3353960ff332b2d2ac85fcc7bbd445f167f066

Here is a screen shot of the error (you can see the release version here)

[laa]

Hi @matthiasbe ,
This is incorrect build number, it should be 947645b013b0de08d6c61ff0635ec690aac35426 . I will create build for you today to try. To install it just run: mvn install:install-file -Dfile=orientdb-core-3.0.6-SNAPSHOT.jar -DgroupId=com.orientechnologies -DartifactId=orientdb-parent -Dversion=3.0.6-SNAPSHOT -Dpackaging=jar

[laa]

Hi @matthiasbe could you use this build? https://drive.google.com/file/d/1MlzGIEfnuqUsffc93uE_zUuZOmjAeC9W/view?usp=sharing .

[matthiasbe]

Hi @laa
Issue is still present using the orientdb-core-3.0.6-SNAPSHOT.jar file you provided.

I have used this file both for server side and Java API of my application

[laa]

@matthiasbe just to make it clear, you not only opened but also created a database and then crashed server using the same distribution. Did not you?

[matthiasbe]

With this setup :

• Start the server using the provided jar
• Create new db
• Run our application to the point of the addVertex() call on the Account class. I think it's the first update call after schema initialisation
• Stop orientdb ungracefully
• Stop application
• Restart orientdb
• Run application
  -> RuntimeException and database set to read-only when reaching addVertex() call again
  -> Magic number verification failed in orientdb server output

[laa]

@matthiasbe I suppose you do not see a message like WARN: File ... already exists and will be truncated, do not you?

[laa]

Hi @matthiasbe could you check this jar https://drive.google.com/file/d/1MlzGIEfnuqUsffc93uE_zUuZOmjAeC9W/view?usp=sharing

[matthiasbe]

To precise how I've build this :

• pull 3.0.x branch from github
• at the root dir : mvn clean install -DskipTests=true
• placed the jar you provided in the root dir and then : mvn install:install-file -Dfile=orientdb-core-3.0.6-SNAPSHOT.jar -DgroupId=com.orientechnologies -DartifactId=orientdb-parent -Dversion=3.0.6-SNAPSHOT -Dpackaging=jar
• replace the orientdb-core-3.0.6-SNAPSHOT.jar with the one you provided in my eclipse build path
• launch orientdb\distribution-tp2\target\orientdb-community-tp2-3.0.6-SNAPSHOT.dir\orientdb-community-tp2-3.0.6-SNAPSHOT\bin\server.bat

Issue is still present ...

Am I using the right build ? Build number at orientdb startup is 8d0427e2245184c7ed0ac6f9669b1be13af2c23f
There are other distribution folders :

• orientdb\distribution-tp2\target\orientdb-community-3.0.6-SNAPSHOT
• orientdb\distribution\target\orientdb-community-3.0.6-SNAPSHOT.dir\orientdb-community-3.0.6-SNAPSHOT\bin

Both also have the build number 8d0427e2245184c7ed0ac6f9669b1be13af2c23f
Does it make a difference ?

Looking forward to get to the bottom of this 😣

[laa]

Should be fixed in latest 3.0.x version, appreciate if you try to check that.

[matthiasbe]

Hi,
It's not possible for me to debug this anymore, as I have changed project since then.