Clickhouse: DB::Exception: Unknown codec family code : 4:

Created on 21 Feb 2019 · 9Comments · Source: ClickHouse/ClickHouse

(you don't have to strictly follow this form)

Describe the bug
We have replicated table with ReplicatedMergeTree engine. Suddenly one of the replicas began to return an error for select query:

<hostname> :) select sum(cityHash64(request)) from db.table where event_date = '2019-02-20'                                                                                    

SELECT sum(cityHash64(request))
FROM db.table
WHERE event_date = '2019-02-20'

Received exception from server (version 19.1.6):
Code: 432. DB::Exception: Received from localhost:9000, 127.0.0.1. DB::Exception: Unknown codec family code : 4: (while reading column request): (while reading from part /data/clickhouse/data/db/table/20190220_20190220_37849_37849_0/ from mark 0 with max_rows_to_read = 8192). 

0 rows in set. Elapsed: 4.944 sec. Processed 72.35 million rows, 13.49 GB (14.63 million rows/s., 2.73 GB/s.)

clickhouse-server tried to check this part, but result was ok

2019.02.20 20:58:56.984566 [ 27 ] {} <Warning> db.table (ReplicatedMergeTreePartCheckThread): Checking part 20190220_20190220_37849_37849_0
2019.02.20 20:58:56.988305 [ 27 ] {} <Warning> db.table (ReplicatedMergeTreePartCheckThread): Checking data of part 20190220_20190220_37849_37849_0.
2019.02.20 20:58:57.076686 [ 27 ] {} <Information> db.table (ReplicatedMergeTreePartCheckThread): Part 20190220_20190220_37849_37849_0 looks good.

2019.02.20 20:58:57.141819 [ 1926 ] {eb495c00-a322-4003-9013-a532f55241ba} <Error> executeQuery: Code: 432, e.displayText() = DB::Exception: Unknown codec family code : 4: (while reading column request): (while reading from part /data/clickhouse/data/db/table/20190220_20190220_37849_37849_0/ from mark 0 with max_rows_to_read = 8192), e.what() = DB::Exception (from 10.3.0.195:44332) (in query: <query>))), Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x9fde646]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3317a42]
2. /usr/bin/clickhouse-server(DB::CompressionCodecFactory::get(unsigned char) const+0x14f) [0x76b290f]
3. /usr/bin/clickhouse-server(DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&)+0x995) [0x76ae4b5]
4. /usr/bin/clickhouse-server(DB::CompressedReadBufferFromFile::seek(unsigned long, unsigned long)+0xad) [0x76af95d]
5. /usr/bin/clickhouse-server(DB::MergeTreeReader::Stream::seekToMark(unsigned long)+0x85) [0x75b4735]
6. /usr/bin/clickhouse-server() [0x75b4d0f]
7. /usr/bin/clickhouse-server(DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, unsigned long, DB::IDataType::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::IDataType::DeserializeBinaryBulkState>&) const+0x28) [0x3482ad8]
8. /usr/bin/clickhouse-server(DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)+0x2b2) [0x75b0512]
9. /usr/bin/clickhouse-server(DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)+0x387) [0x75b0ce7]
10. /usr/bin/clickhouse-server(DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)+0x136) [0x75abdd6]
11. /usr/bin/clickhouse-server(DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)+0x46c) [0x75ac89c]
12. /usr/bin/clickhouse-server(DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)+0x10f) [0x75addbf]
13. /usr/bin/clickhouse-server(DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)+0x696) [0x75ae346]
14. /usr/bin/clickhouse-server(DB::MergeTreeBaseSelectBlockInputStream::readFromPart()+0x7e8) [0x75a6858]
15. /usr/bin/clickhouse-server(DB::MergeTreeBaseSelectBlockInputStream::readImpl()+0xbd) [0x75a7afd]
16. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
17. /usr/bin/clickhouse-server(DB::FilterBlockInputStream::readImpl()+0xbd) [0x73ae89d]
18. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
19. /usr/bin/clickhouse-server(DB::ExpressionBlockInputStream::readImpl()+0x2e) [0x73adb6e]
20. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
21. /usr/bin/clickhouse-server(DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)+0x12d) [0x6f7582d]
22. /usr/bin/clickhouse-server(DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)+0x200) [0x6f75e50]
23. /usr/bin/clickhouse-server(std::thread::_State_impl<std::thread::_Invoker<std::tuple<DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::process()::{lambda()#1}> > >::_M_run()+0x4f) [0x6f7633f]
24. /usr/bin/clickhouse-server() [0xaa3e73f]
25. /lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7efd17686064]
26. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7efd16cae62d]

So, this broken part won't be downloaded from replica, causing this error appear again and again.

I tried to run OPTIMIZE table FINAL command for partition with a broken part, but unsuccessfully.

data from system.merges table:

Row 1:
──────
database:               db
table:                  table
replica_name:           host
position:               0
node_name:              queue-0001297481
type:                   MERGE_PARTS
create_time:            2019-02-20 20:59:12
required_quorum:        0
source_replica:         host
new_part_name:          20190220_20190220_37825_37851_2
parts_to_merge:         ['20190220_20190220_37825_37830_1','20190220_20190220_37831_37836_1','20190220_20190220_37837_37842_1','20190220_20190220_37843_37848_1','20190220_20190220_37849_37849_0','20190220
_20190220_37850_37850_0','20190220_20190220_37851_37851_0']
is_detach:              0
is_currently_executing: 0
num_tries:              3273
last_exception:         Code: 432, e.displayText() = DB::Exception: Unknown codec family code : 4: (while reading column request): (while reading from part /data/clickhouse/data/db/table/20
190220_20190220_37849_37849_0/ from mark 0 with max_rows_to_read = 8192): Cannot fetch required block. Stream MergeTreeSequentialBlockInputStream, part 4, e.what() = DB::Exception
last_attempt_time:      2019-02-21 15:29:09
num_postponed:          43
postpone_reason:        Not executing log entry for part 20190220_20190220_37825_37851_2 because source parts size (283.90 MiB) is greater than the current maximum (19.80 MiB).
last_postpone_time:     2019-02-21 11:20:16

How to reproduce

Which ClickHouse server version to use
19.1.6
Which interface to use, if matters
clickhouse-client

Expected behavior

example query executed successfully
in case of broken part it should be downloaded from replica

So, there are 2 problems:

Unknown codec family code : 4: error
Unability to download broken part from second replica

bug

Source

PhantomPhreak

Most helpful comment

SYSTEM DROP MARK CACHE
It can really solve the problem . My ClickHouse server version 19.3.3 revision 54415

wsg96321 on 28 Feb 2019

👍2

All 9 comments

this is also an issue in 19.3.5

rpfennin on 24 Feb 2019

related #4415

4ertus2 on 25 Feb 2019

Do you use any custom codec (zstd, lz4hc)?

alesapin on 25 Feb 2019

We're using zstd compression

    <!-- Compression settings -->
    <compression>
        <case>
            <min_part_size>22548578304</min_part_size>
            <method>zstd</method>
        </case>
    </compression>

PhantomPhreak on 26 Feb 2019

We need to check one hypothesis. Does query SYSTEM DROP MARK CACHE solves the problem?

alesapin on 27 Feb 2019

SYSTEM DROP MARK CACHE
It can really solve the problem . My ClickHouse server version 19.3.3 revision 54415

wsg96321 on 28 Feb 2019

👍2

We need to check one hypothesis. Does query SYSTEM DROP MARK CACHE solves the problem?

I can't reproduce this issue now and can't check if it works or not.

Last time i solved the problem by deleting broken part and server restarting, ClickHouse server found missing part and fetched it from replica.

PhantomPhreak on 28 Feb 2019

Got the error today:

2019.03.05 13:39:16.077741 [ 31 ] {} <Warning> db.table (ReplicatedMergeTreePartCheckThread): Checking part 20190302_20190302_3092_3403_6
2019.03.05 13:39:16.078143 [ 31 ] {} <Warning> db.table (ReplicatedMergeTreePartCheckThread): Checking data of part 20190302_20190302_3092_3403_6.
2019.03.05 13:39:16.078717 [ 3 ] {} <Error> db.table (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code
: 87, e.displayText() = DB::ErrnoException: Cannot seek through file /data/clickhouse/data/db/tablel/20190302_20190302_3092_3403_6/user_id.bin, errno: 22, strerror: Invalid argument: (while
 reading column user_id): (while reading from part /data/clickhouse/data/db/table/20190302_20190302_3092_3403_6/ from mark 0 with max_rows_to_read = 8192), e.what() = DB::ErrnoException, S
tack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x9fde646]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3317a42]
2. /usr/bin/clickhouse-server(DB::throwFromErrno(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)+0x1d8) [0x9fc1c88]
3. /usr/bin/clickhouse-server(DB::ReadBufferFromFileDescriptor::doSeek(long, int)+0x24d) [0x9ffbaad]
4. /usr/bin/clickhouse-server(DB::CompressedReadBufferFromFile::seek(unsigned long, unsigned long)+0x70) [0x76af920]
5. /usr/bin/clickhouse-server(DB::MergeTreeReader::Stream::seekToMark(unsigned long)+0x85) [0x75b4735]
6. /usr/bin/clickhouse-server() [0x75b4d0f]
7. /usr/bin/clickhouse-server(DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, unsigned long, DB::IDataType::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::IDataType::Deserialize
BinaryBulkState>&) const+0x28) [0x3482ad8]
8. /usr/bin/clickhouse-server(DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool,
 unsigned long, bool)+0x2b2) [0x75b0512]
9. /usr/bin/clickhouse-server(DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)+0x387) [0x75b0ce7]
10. /usr/bin/clickhouse-server(DB::MergeTreeSequentialBlockInputStream::readImpl()+0x328) [0x7278118]
11. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
12. /usr/bin/clickhouse-server(DB::ExpressionBlockInputStream::readImpl()+0x2e) [0x73adb6e]
13. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
14. /usr/bin/clickhouse-server(DB::MaterializingBlockInputStream::readImpl()+0x3b) [0x73c220b]
15. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
16. /usr/bin/clickhouse-server(DB::MergingSortedBlockInputStream::init(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&)+0xce) [0x73daaae]
17. /usr/bin/clickhouse-server(DB::MergingSortedBlockInputStream::readImpl()+0x65) [0x73dc085]
18. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
19. /usr/bin/clickhouse-server(DB::MergeTreeDataMergerMutator::mergePartsToTemporaryPart(DB::MergeTreeDataMergerMutator::FuturePart const&, DB::MergeListEntry&, long, DB::DiskSpaceMonitor::Reservation*, bool)+0x289f) [0x722602f]
20. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::tryExecuteMerge(DB::ReplicatedMergeTreeLogEntry const&)+0x6ea) [0x718b9ba]
21. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&)+0x28b) [0x719422b]
22. /usr/bin/clickhouse-server() [0x719490f]
23. /usr/bin/clickhouse-server(DB::ReplicatedMergeTreeQueue::processEntry(std::function<std::shared_ptr<zkutil::ZooKeeper> ()>, std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&, std::function<bool (std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>)+0x55) [0x72b9665]
24. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::queueTask()+0x1ef) [0x716b19f]
25. /usr/bin/clickhouse-server(DB::BackgroundProcessingPool::threadFunction()+0x43e) [0x71d54ee]
26. /usr/bin/clickhouse-server() [0xaa3e73f]
27. /lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f16450c5064]
28. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f16446ed62d]

2019.03.05 13:39:16.288121 [ 31 ] {} <Information> db.table (ReplicatedMergeTreePartCheckThread): Part 20190302_20190302_3092_3403_6 looks good.

Tried to drop mark cache as suggested using command
SYSTEM DROP MARK CACHE
Server crashed with following records in the log

2019.03.05 13:41:50.892023 [ 2 ] {} <Error> BaseDaemon: ########################################
2019.03.05 13:41:50.892110 [ 2 ] {} <Error> BaseDaemon: (from thread 1130660) Received signal Segmentation fault (11).
2019.03.05 13:41:50.892125 [ 2 ] {} <Error> BaseDaemon: Address: NULL pointer.
2019.03.05 13:41:50.892136 [ 2 ] {} <Error> BaseDaemon: Access: read.
2019.03.05 13:41:50.892147 [ 2 ] {} <Error> BaseDaemon: Unknown si_code.
2019.03.05 13:41:50.929921 [ 2 ] {} <Error> BaseDaemon: 0. /usr/bin/clickhouse-server(DB::Context::dropMarkCache() const+0x80) [0x6f074d0]
2019.03.05 13:41:50.929977 [ 2 ] {} <Error> BaseDaemon: 1. /usr/bin/clickhouse-server(DB::InterpreterSystemQuery::execute()+0xd88) [0x6f823b8]
2019.03.05 13:41:50.929990 [ 2 ] {} <Error> BaseDaemon: 2. /usr/bin/clickhouse-server() [0x706681e]
2019.03.05 13:41:50.930020 [ 2 ] {} <Error> BaseDaemon: 3. /usr/bin/clickhouse-server(DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum)+0x66) [0x70686d6]
2019.03.05 13:41:50.930037 [ 2 ] {} <Error> BaseDaemon: 4. /usr/bin/clickhouse-server(DB::TCPHandler::runImpl()+0x48e) [0x33262ee]
2019.03.05 13:41:50.930051 [ 2 ] {} <Error> BaseDaemon: 5. /usr/bin/clickhouse-server(DB::TCPHandler::run()+0x2b) [0x33274db]
2019.03.05 13:41:50.930065 [ 2 ] {} <Error> BaseDaemon: 6. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0xa0886af]
2019.03.05 13:41:50.930078 [ 2 ] {} <Error> BaseDaemon: 7. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x16a) [0xa088a8a]
2019.03.05 13:41:50.930092 [ 2 ] {} <Error> BaseDaemon: 8. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x77) [0xa6d2037]
2019.03.05 13:41:50.930106 [ 2 ] {} <Error> BaseDaemon: 9. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0xa6cdef8]
2019.03.05 13:41:50.930118 [ 2 ] {} <Error> BaseDaemon: 10. /usr/bin/clickhouse-server() [0xaa3e73f]
2019.03.05 13:41:50.930131 [ 2 ] {} <Error> BaseDaemon: 11. /lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f16450c5064]
2019.03.05 13:42:33.307443 [ 1 ] {} <Information> : Starting ClickHouse 19.1.6 with revision 54413
2019.03.05 13:42:33.314961 [ 1 ] {} <Information> Application: starting up

PhantomPhreak on 5 Mar 2019