Yugabyte-db: [YSQL] Table Creation Partially Fails

[hudclark]

YB Version: 2.2.0.0.

We are using a 18 node cluster. NOTE: 6 nodes are blacklisted from the others and have been 'dead' for at least 1 week.

All SQL commands were sent via ./bin/ysql

What happened

1. From the ysql shell, issued a command

CREATE TABLE table_name (
  <schema>
);

1. After about 30 seconds, received the error

ERROR:  Already present: [Already present (yb/consensus/retryable_requests.cc:241): Duplicate request]

1. Running \dt does not show the new table
2. The YugaByte dashboard does show the new table
3. Retried the query

CREATE TABLE table_name (
  <schema>
);

1. After 30 seconds, received the same error as before
2. \dt does not show the new table
3. The YugaByte dashboard shows two tables with the same name.

[nspiegelberg]

@hudclark : As a workaround, you should be able to issue DROP TABLE table_name and then re-create in the latest version of 2.2. See issue #4710, the fix was issued 2 weeks ago, so ensure that you're on the latest 2.2 release.

[ddorian]

@hudclark

In the meantime, can you try ./bin/yb-admin delete_table_by_id <table_id> and try recreating ?

[hudclark]

@ddorian / @nspiegelberg Thank you for the suggestions.

Since this is a database used in production, we're finalizing our disaster recovery playbook before running any more schema changes. I'm expecting to try out your suggestions and report back early next week.

[bmatican]

@hudclark If by any chance this happens again, or if you still have the setup where it happened initially, it might also help if you could share two types of logs with us, to better understand what might have happened:

• the postgres logs -- eg: yb-tserver/logs/postgres.* on the host you had connected to with your ysqlsh
• the master leader logs -- eg: yb-master/logs/yb-master.INFO, on the host that was the master leader at that time

If you're not comfortable sharing those on the issue, also happy to connect on Slack!