Marten: Data mysteriously disappeared

[siberianguy]

Over the weekend I've migrated our database from RavenDb to Marven and the overall process was extremely smooth (great job!). But today there was a very frightening accident. So I've been adding data to the database using BulkInsert (a couple of thousand documents per second) and it everything was fine. But then I tried to read data from the same database and two things happened:

1. The program saving data using BulkInsert stoped working: every BulkInsert resulted in System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable It seems like it could be connected to pooling, so I disabled it and it seems to work, at least for now
2. All the data from the table I've been reading from, disappeared. Taking into account the previous error, it looks like for all BulkInserts there was a global transaction that was rolled back. I find it very unlikely, just wanted to share this hoping may be more experienced Marten users could share their insights.

[siberianguy]

I figured it out and, honestly, I'm shocked. So, a program that recorded data has a duplicated field definition:

using (var inputStore = DocumentStore
                .For(options =>
                {
                    options.Connection(inputConnection);

                    options.Schema.For<OrderBookSnapshot>().Duplicate(x => x.DateTime);
                }))

And a program that was reading data did not specify it as it did not care about that index. After enabling a log, I found out the reading side found a schema mismatch and just dropped a table. Wow! Gigabytes of data were lost on a read operation. Later I fixed it by adding options.AutoCreateSchemaObjects = AutoCreate.CreateOrUpdate. It's not clear what is a default option based on documentation but shouldn't it be a behaviour that doesn't involve killing all the data?

[jeremydmiller]

@siberianguy The documentation talks about this pretty clearly. We use the "All" setting as the default to make the getting started story faster for users. I'm sorry this happened to you though.

[jeremydmiller]

@siberianguy Hey, and probably most importantly, you need to use the exact same DocumentStore configuration any time you're connecting to a shared Marten-ized database, or you can expect more of these kinds of problems.

[Miggleness]

@jeremydmiller what do you think of changing default to AutoCreate.Create in the next major release (v3)? I feel this is a safer default and full schema management by application should be an opt-in option.

[siberianguy]

+1 for Create or CreateOrUpdate as a default option. I doubt I was the only person who was influenced by this. It reminds me MongoDb running in an unsafe mode in early version. One of the good ideas behind RavenDb was "safe by default" (though they went a little too far).

It also might be worth pointing it out in Getting Started as that's what people who are getting into Marten read. Now their data might get killed if they did not read through the whole documentation.

[jeremydmiller]

@Miggleness @siberianguy We can change the default in 3.0, but that's not happening this year. I'd happily take a PR changing the text in the docs if you have suggestions.

[jeremydmiller]

Closing this one, I think we've come to a conclusion.