Clickhouse: Alter on ReplicatedMergeTree table fails with error

Created on 2 Apr 2020 · 7Comments · Source: ClickHouse/ClickHouse

Describe the bug
Alter on ReplicatedMergeTree table fails with error:
<Error> executeQuery: Code: 450, e.displayText() = DB::Exception: No such volume 'hu, 02 Apr 2020 07:43:45 GMT\0%2Fclickhouse' for given storage policy. (version 20.1.2.4 (official build)) (from 0.0.0.0:0) (in query: /* ddl_entry=query-0000002740 */ ALTER TABLE loadtest.stat_rpl ADD COLUMN IF NOT EXISTS `stream_path` String )

How to reproduce

Version: 20.1.2.4 (official build)
CREATE TABLE statements for all tables involved

CREATE TABLE loadtest.stat_rpl (`mdate` Date) ENGINE = ReplicatedMergeTree('/clickhouse/tables/loadtest/{shard}/stat_rpl', '{replica}') PARTITION BY mdate ORDER BY mdate TTL mdate + toIntervalDay(3) SETTINGS index_granularity = 8192

Queries to run that lead to unexpected result

ALTER TABLE loadtest.stat_rpl ON CLUSTER production ADD COLUMN stream_path String;

Expected behavior
Alter query are expected to work

Error message and/or stacktrace

2020.04.02 10:44:39.807499 [ 69 ] {7e336140-cc63-4d35-97ee-43b3c379e98f} <Error> executeQuery: Code: 450, e.displayText() = DB::Exception: No such volume `hu, 02 Apr 2020 07:43:45 GMT\0%2Fclickhouse` for given storage policy. (version 20.1.2.4 (official build)) (from 0.0.0.0:0) (in query: /* ddl_entry=query-0000002740 */ ALTER TABLE loadtest.stat_rpl ADD COLUMN IF NOT EXISTS `stream_path` String ), Stack trace (when copying this message, always include the lines below):

0. 0xbc31d9c Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)  in /usr/bin/clickhouse
1. 0x4f6ccd9 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)  in /usr/bin/clickhouse
2. 0x8ffacce DB::MergeTreeData::setTTLExpressions(std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::shared_ptr<DB::IAST>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::shared_ptr<DB::IAST> > > > const&, std::__1::shared_ptr<DB::IAST> const&, bool)  in /usr/bin/clickhouse
3. 0x90092b8 DB::MergeTreeData::checkAlterIsPossible(DB::AlterCommands const&, DB::Settings const&)  in /usr/bin/clickhouse
4. 0x96c052d DB::InterpreterAlterQuery::execute()  in /usr/bin/clickhouse
5. 0x8dc07b8 ?  in /usr/bin/clickhouse
6. 0x8dc3bab DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)  in /usr/bin/clickhouse
7. 0x8ba1151 DB::DDLWorker::tryExecuteQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::DDLTask const&, DB::ExecutionStatus&)  in /usr/bin/clickhouse
8. 0x8ba4027 DB::DDLWorker::tryExecuteQueryOnLeaderReplica(DB::DDLTask&, std::__1::shared_ptr<DB::IStorage>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<zkutil::ZooKeeper> const&)  in /usr/bin/clickhouse
9. 0x8ba7b0f DB::DDLWorker::processTask(DB::DDLTask&, std::__1::shared_ptr<zkutil::ZooKeeper> const&)  in /usr/bin/clickhouse
10. 0x8ba8699 DB::DDLWorker::processTasks()  in /usr/bin/clickhouse
11. 0x8ba98f0 DB::DDLWorker::runMainThread()  in /usr/bin/clickhouse
12. 0x8babcf1 ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() const  in /usr/bin/clickhouse
13. 0x4fa3b77 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>)  in /usr/bin/clickhouse
14. 0x4fa212f ?  in /usr/bin/clickhouse
15. 0x76db start_thread  in /lib/x86_64-linux-gnu/libpthread-2.27.so
16. 0x12188f __clone  in /lib/x86_64-linux-gnu/libc-2.27.so

2020.04.02 10:44:39.939813 [ 69 ] {97dff4a3-f8da-4351-baf9-f63c9cdf3950} <Error> executeQuery: Code: 241, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 7.00 EiB (attempt to allocate chunk of 8070450532247926624 bytes), maximum: 9.31 GiB (version 20.1.2.4 (official build)) (from 0.0.0.0:0) (in query: /* ddl_entry=query-0000002740 */ ALTER TABLE loadtest.stat_rpl ADD COLUMN IF NOT EXISTS `stream_path` String ), Stack trace (when copying this message, always include the lines below):

0. 0xbc31d9c Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)  in /usr/bin/clickhouse
1. 0x4f6ccd9 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)  in /usr/bin/clickhouse
2. 0x4965780 ?  in /usr/bin/clickhouse
3. 0x4f68306 MemoryTracker::alloc(long)  in /usr/bin/clickhouse
4. 0x4f13048 operator new(unsigned long)  in /usr/bin/clickhouse
5. 0xc6a4522 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__grow_by_and_replace(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, char const*)  in /usr/bin/clickhouse
6. 0xc6a4793 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::assign(char const*, unsigned long)  in /usr/bin/clickhouse
7. 0x8ffa461 DB::MergeTreeData::setTTLExpressions(std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::shared_ptr<DB::IAST>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::shared_ptr<DB::IAST> > > > const&, std::__1::shared_ptr<DB::IAST> const&, bool)  in /usr/bin/clickhouse
8. 0x90092b8 DB::MergeTreeData::checkAlterIsPossible(DB::AlterCommands const&, DB::Settings const&)  in /usr/bin/clickhouse
9. 0x96c052d DB::InterpreterAlterQuery::execute()  in /usr/bin/clickhouse
10. 0x8dc07b8 ?  in /usr/bin/clickhouse
11. 0x8dc3bab DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)  in /usr/bin/clickhouse
12. 0x8ba1151 DB::DDLWorker::tryExecuteQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::DDLTask const&, DB::ExecutionStatus&)  in /usr/bin/clickhouse
13. 0x8ba4027 DB::DDLWorker::tryExecuteQueryOnLeaderReplica(DB::DDLTask&, std::__1::shared_ptr<DB::IStorage>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<zkutil::ZooKeeper> const&)  in /usr/bin/clickhouse
14. 0x8ba7b0f DB::DDLWorker::processTask(DB::DDLTask&, std::__1::shared_ptr<zkutil::ZooKeeper> const&)  in /usr/bin/clickhouse
15. 0x8ba8699 DB::DDLWorker::processTasks()  in /usr/bin/clickhouse
16. 0x8ba98f0 DB::DDLWorker::runMainThread()  in /usr/bin/clickhouse
17. 0x8babcf1 ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() const  in /usr/bin/clickhouse
18. 0x4fa3b77 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>)  in /usr/bin/clickhouse
19. 0x4fa212f ?  in /usr/bin/clickhouse
20. 0x76db start_thread  in /lib/x86_64-linux-gnu/libpthread-2.27.so
21. 0x12188f __clone  in /lib/x86_64-linux-gnu/libc-2.27.so

Full logs: https://gist.github.com/erste/d0f6c88b03e25c12f533afcd00cf09f2

Additional context
Clickhouse configs: https://gist.github.com/erste/0e18bfbb838a09a355bdfffeba1aae18

bug comp-dddl comp-multidisk comp-ttl duplicate st-need-repro v20.1

Source

erste

All 7 comments

Can you show your table schema (show create table)?

alesapin on 3 Apr 2020

Sorry, i cant due to security issues.
All i could provide is in the first message:
CREATE TABLE loadtest.stat_rpl (`mdate` Date) ENGINE = ReplicatedMergeTree('/clickhouse/tables/loadtest/{shard}/stat_rpl', '{replica}') PARTITION BY mdate ORDER BY mdate TTL mdate + toIntervalDay(3) SETTINGS index_granularity = 8192

erste on 3 Apr 2020

👍1

Sorry, I cannot reproduce the problem, both on your version and on current master. I recommend you to update to the latest 20.3 stable version.

alesapin on 6 Apr 2020

We purge all data, recreate all databases and tables and issue still persist.
And ith simple database as in first message alter table also not working.

Another strange behavior we noticed recently. Our cluster have one leader for all replicas:

SELECT
    database,
    table,
    is_leader,
    can_become_leader,
    is_readonly,
    replica_name
FROM system.replicas
WHERE (database = 'loadtest') AND (table = 'stat_rpl')

┌─database─┬─table────┬─is_leader─┬─can_become_leader─┬─is_readonly─┬─replica_name─────────────────┐
│ loadtest │ stat_rpl │         0 │                 1 │           0 │ clickhouse1                  │
└──────────┴──────────┴───────────┴───────────────────┴─────────────┴──────────────────────────────┘
┌─database─┬─table────┬─is_leader─┬─can_become_leader─┬─is_readonly─┬─replica_name─────────────────┐
│ loadtest │ stat_rpl │         0 │                 1 │           0 │ clickhouse2                  │
└──────────┴──────────┴───────────┴───────────────────┴─────────────┴──────────────────────────────┘
┌─database─┬─table────┬─is_leader─┬─can_become_leader─┬─is_readonly─┬─replica_name─────────────────┐
│ loadtest │ stat_rpl │         1 │                 1 │           0 │ clickhouse3                  │
└──────────┴──────────┴───────────┴───────────────────┴─────────────┴──────────────────────────────┘
┌─database─┬─table────┬─is_leader─┬─can_become_leader─┬─is_readonly─┬─replica_name─────────────────┐
│ loadtest │ stat_rpl │         0 │                 1 │           0 │ clickhouse4                  │
└──────────┴──────────┴───────────┴───────────────────┴─────────────┴──────────────────────────────┘

But on working cluster we have at least one leader per shard. Maybe this is root cause of this issue?

Zookeeper config:

tickTime=2000
initLimit=30000
syncLimit=10
dataDir=/var/lib/zookeeper
clientPort=2181
server.1=10.1.1.105:2888:3888
server.2=10.1.1.106:2888:3888
server.3=10.1.1.107:2888:3888
server.4=10.1.1.108:2888:3888
preAllocSize=131072
snapCount=3000000
leaderServes=yes
maxClientCnxns=2000
maxSessionTimeout=60000000
autopurge.snapRetainCount=10
autopurge.purgeInterval=1
standaloneEnabled=false

erste on 6 Apr 2020

@alesapin
Exception cames from this lines:
https://github.com/ClickHouse/ClickHouse/blob/3a1e375b8ab8a264b14bd646cc677149aa910a92/dbms/src/Storages/MergeTree/MergeTreeData.cpp#L689-L697

Maybe getStoragePolicy returns trash because global_context was not initialized (or badly / partially initialized) for DDLWorker?

/cc @excitoon