Cockroach: core: provide an temporary linearalizability mode (for taking consistent backups)

[Bessonov]

Is your feature request related to a problem? Please describe.
This is a follow up from forum discussion.

Possibility to create data consistent backups is the most important feature of every database. Current implementation creates a consistent backup in regard to global state of CRDB. While the global state is OK from operational point of view, this is very bad for backups. There is no way to tell which data is not consistent.

High availability of CRDB doesn't help here, because HA isn't backup solution.

Describe the solution you'd like

As described in forum post, the main idea is to put CRDB temporary in linearizable mode. This can be achieved by temporal hold of writes (like google spanner) for probably 250ms or something global (like clock) to coordinate writes with reduced write throughput for same 250ms.

From my point of view, temporal delay of writes or reduced write throughput is totally OK, if there is a need for data consistent backups, even if it takes more than 250ms.

Describe alternatives you've considered
Alternatives are welcome!

Additional context
I don't know if there is an easy way to implement it in CRDB. But it is essential.

[Bessonov]

I have another idea that might be much easier to implement. What do you think about some sort of global transaction to retrieve a global time travel timestamp? Like (pseudo code):

BEGIN GLOBAL WRITE;

SELECT CURRENT SYSTEM TIME;

COMMIT;

Which lock all tables for writes, get consolidated timestamp and unlock all tables again. After that you can use returned value (ex. 2016-01-25T10:10:10.555555) to perform global consistent time travel:

cockroach dump <database> <flags> --as-of='2016-01-25T10:10:10.555555'

Beside of much easier implementation and consistent backups, you get two benefits:

1. most small locking window to get consistent state timestamp for the time travel, not only for backups
2. developers can use it to enforce linearizability if they put DML queries in the GLOBAL WRITE transaction

What do you think, guys?

[Bessonov]

Hey @rolandcrosby @awoods187 , any news on this topic?

[Bessonov]

A friendly reminder @awoods187 @rolandcrosby @roncrdb @justinj .

[rolandcrosby]

Hi @Bessonov, thank you for your patience, and sorry for not getting back on this earlier. We don't have any work planned on this for the 19.2 release this fall, but I've taken note of the problem you're trying to solve and am investigating possible solutions for future releases.

[Bessonov]

@rolandcrosby thank you very much for your response! I would be very happy if it can get into 20.1 release.

[Bessonov]

The syntax of postgres could be used to set the mode:

SET TRANSACTION ISOLATION LEVEL LINEARIZABLE;
SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL LINEARIZABLE;

@rolandcrosby any chance to get it into 20.1?

[Bessonov]

It would be nice to get it in 20.2.

[Bessonov]

Hey @rolandcrosby any news on this? How this issue is affected by the 20.2 release and deprecation of crdb dump? full backups doesn't resolve it, right?

[rolandcrosby]

Hi @Bessonov, I'm no longer involved with CockroachDB so can't say for sure. You can try asking on the forum or Slack if nobody else replies on this issue.

[ajwerner]

It feels like if you could get a ranged shared lock over all of the tables then you'd achieve the relevant goal. We don't have immediate plans to build a replicated version of such a lock but we do have an intention to build an unreplicated version in https://github.com/cockroachdb/cockroach/issues/55896 and https://github.com/cockroachdb/cockroach/issues/49684. My sense is that it doesn't even matter that these would be best effort, so long as they all succeeded at some consistent timestamp, which would mean refreshing them if there were a push, then that timestamp would be a temporal barrier.

Doing that, of course, may stall the database, however, if done with the right priorities might be a pretty good answer. Once #55896 is done, I think this becomes relatively straightforward to incorporate into the BACKUP command as an option and then #49684 would make it less impactful for reads. No commitments on when any of this will be put together but I do believe the prerequisites for this approach are slated to happen sooner rather than later.