Yugabyte-db: table creation failure in v2.1.2.0

Created on 17 Mar 2020 · 9Comments · Source: yugabyte/yugabyte-db

I am running into an issue simply creating tables in v2.1.2.0. I have created a simple reproduction with the following steps running on OSX 10.15.3

Gregorys-MacBook-Pro:yb-bug ghaskins$ ~/yugabyte-2.1.2.0/bin/yb-ctl destroy
Destroying cluster.
Gregorys-MacBook-Pro:yb-bug ghaskins$ ~/yugabyte-2.1.2.0/bin/yb-ctl create
Creating cluster.
Waiting for cluster to be ready.
----------------------------------------------------------------------------------------------------
| Node Count: 1 | Replication Factor: 1                                                            |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.1:5433/postgres                                  |
| YSQL Shell          : ~/yugabyte-2.1.2.0/bin/ysqlsh                                              |
| YCQL Shell          : ~/yugabyte-2.1.2.0/bin/cqlsh                                               |
| YEDIS Shell         : ~/yugabyte-2.1.2.0/bin/redis-cli                                           |
| Web UI              : http://127.0.0.1:7000/                                                     |
| Cluster Data        : /Users/ghaskins/yugabyte-data                                              |
----------------------------------------------------------------------------------------------------

For more info, please use: yb-ctl status
Gregorys-MacBook-Pro:yb-bug ghaskins$ ~/yugabyte-2.1.2.0/bin/cqlsh
Connected to local cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> SOURCE 'bug.cql';
bug.cql:24:OperationTimedOut: errors={'127.0.0.1': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=127.0.0.1

where bug.cql contains

CREATE KEYSPACE test;
USE test;

CREATE TABLE table1 (
    foo varchar,
    bar varchar,
    baz varchar,
    PRIMARY KEY (foo, bar, baz)
);

CREATE TABLE table2 (
    foo varchar,
    bar varchar,
    baz varchar,
    PRIMARY KEY (bar, foo, baz)
);

CREATE TABLE table3 (
    foo varchar,
    bar varchar,
    baz varchar,
    PRIMARY KEY (baz, foo, bar)
);

aredocdb communitrequest

Source

ghaskins

Most helpful comment

Yep, that fixed it! Thank you!

ghaskins on 17 Mar 2020

👍2

All 9 comments

Here is the relevant log

I0317 08:43:31.020771 131129344 catalog_manager.cc:1816] CreateTable from 127.0.0.1:60168:
name: "table3"
schema {
  columns {
    name: "baz"
    type {
      main: STRING
    }
    is_key: true
    is_hash_key: true
    is_nullable: false
    is_static: false
    is_counter: false
    sorting_type: 0
    order: 2
  }
  columns {
    name: "foo"
    type {
      main: STRING
    }
    is_key: true
    is_nullable: false
    is_static: false
    is_counter: false
    sorting_type: 1
    order: 0
  }
  columns {
    name: "bar"
    type {
      main: STRING
    }
    is_key: true
    is_nullable: false
    is_static: false
    is_counter: false
    sorting_type: 1
    order: 1
  }
  table_properties {
    contain_counters: false
    is_transactional: false
    consistency_level: STRONG
    use_mangled_column_name: false
    num_tablets: 2
    is_ysql_catalog_table: false
    is_backfilling: false
  }
}
num_tablets: 2
partition_schema {
}
table_type: YQL_TABLE_TYPE
namespace {
  name: "test"
  database_type: YQL_DATABASE_CQL
}
colocated: true
I0317 08:43:31.020848 131129344 partition.cc:388] Creating partitions with num_tablets: 2
I0317 08:43:31.020859 131129344 catalog_manager.cc:2045] CreateTable with IndexInfo 
I0317 08:43:31.021392 131129344 catalog_manager.cc:2198] Successfully created table table3 [id=b6bc551d1e52447b87122d7022d717e2] per request from 127.0.0.1:60168
W0317 08:44:01.106143 135958528 catalog_manager.cc:5588] Tablet e13841b65ba8470fbb59f8b4bf643a3c (table table3 [id=b6bc551d1e52447b87122d7022d717e2]) was not created within the allowed timeout. Replacing with a new tablet 001f525e27744e1a961776aa1b839ef5
W0317 08:44:01.107224 135958528 catalog_manager.cc:5588] Tablet ef91e8775fb44625b1f61f093d00db6f (table table3 [id=b6bc551d1e52447b87122d7022d717e2]) was not created within the allowed timeout. Replacing with a new tablet b2b7e1a1e8f644c48d800bc62a18010b
I0317 08:44:01.107753 135958528 catalog_manager.cc:5370] Sending DeleteTablet for 0 replicas of tablet e13841b65ba8470fbb59f8b4bf643a3c
I0317 08:44:01.107779 135958528 catalog_manager.cc:5370] Sending DeleteTablet for 0 replicas of tablet ef91e8775fb44625b1f61f093d00db6f
I0317 08:44:23.494875 124153856 reactor.cc:450] Master_R010: Timing out connection Connection (0x0000000110748d98) server 127.0.0.1:60164 => 127.0.0.1:7100 - it has been idle for 65.0987s (delta: 65.0987, current time: 668205, last activity time: 668140)
W0317 08:44:31.173465 135958528 catalog_manager.cc:5588] Tablet b2b7e1a1e8f644c48d800bc62a18010b (table table3 [id=b6bc551d1e52447b87122d7022d717e2]) was not created within the allowed timeout. Replacing with a new tablet a3b36ad893524cff95581a3f37d52a2c
W0317 08:44:31.173550 135958528 catalog_manager.cc:5588] Tablet 001f525e27744e1a961776aa1b839ef5 (table table3 [id=b6bc551d1e52447b87122d7022d717e2]) was not created within the allowed timeout. Replacing with a new tablet b07b8a88ac8342589c19a2fa98168d0f
I0317 08:44:31.175060 135958528 catalog_manager.cc:5370] Sending DeleteTablet for 0 replicas of tablet b2b7e1a1e8f644c48d800bc62a18010b
I0317 08:44:31.175099 135958528 catalog_manager.cc:5370] Sending DeleteTablet for 0 replicas of tablet 001f525e27744e1a961776aa1b839ef5
I0317 08:44:31.175369 120934400 tcp_stream.cc:308] { local: 127.0.0.1:60205 remote: 127.0.0.1:9100 }:  Recv failed: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 61)
W0317 08:44:31.175491 139714560 async_rpc_tasks.cc:224] TS f30255ad1b904a4f8e71dab727ce91a5: Create Tablet RPC failed for tablet b07b8a88ac8342589c19a2fa98168d0f: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 61)

ghaskins on 17 Mar 2020

This also seems to reproduce the issue:

CREATE KEYSPACE test;
USE test;

CREATE TABLE mytable (
    foo varchar,
    bar varchar,
    baz varchar,
    PRIMARY KEY (foo)
) WITH transactions = { 'enabled' : true };

CREATE INDEX myindex ON mytable (bar);

ghaskins on 17 Mar 2020

hi @ghaskins :

W0317 08:44:31.175491 139714560 async_rpc_tasks.cc:224] TS f30255ad1b904a4f8e71dab727ce91a5: Create Tablet RPC failed for tablet b07b8a88ac8342589c19a2fa98168d0f: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 61)

This message above might be the relevant one to get to the bottom of.

a) Can you please share the yb-tserver.INFO logs also from around the same time?

b) For storage, does the Mac you are trying this on have a HDD or SSD?

kmuthukk on 17 Mar 2020

Im guessing this might be the smoking gun

$ cat ~/yugabyte-data/node-1/disk-1/yb-data/tserver/logs/yb-tserver.FATAL
Log file created at: 2020/03/17 13:02:24
Running on machine: Gregorys-MacBook-Pro.local
Application fingerprint: version 2.1.2.0 build PRE_RELEASE revision d16f8e36d68311b1d9130ea28cec9e338b9f870e build_type RELEASE built at 10 Mar 2020 12:25:50 PST
Running duration (h:mm:ss): 0:01:02
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0317 13:02:24.931187 302985216 replica_state.cc:478] Check failed: _s.ok() Bad status: IO error (yb/util/env_posix.cc:1502): Unable to write consensus meta file for tablet 08798bdd61f04dbbb399d04e211e3e37 to path /Users/ghaskins/yugabyte-data/node-1/disk-1/yb-data/tserver/consensus-meta/08798bdd61f04dbbb399d04e211e3e37: Call to mkstemp() failed on name template /Users/ghaskins/yugabyte-data/node-1/disk-1/yb-data/tserver/consensus-meta/08798bdd61f04dbbb399d04e211e3e37.tmp.XXXXXX: Too many open files (system error 24)

ghaskins on 17 Mar 2020

As to (b). this is a 2019 MacBookPro with NVME.

ghaskins on 17 Mar 2020

Aaah @ghaskins have you increased your ulimits?

See https://docs.yugabyte.com/latest/deploy/manual-deployment/system-config/#ulimits