Clickhouse: DB::ErrnoException: Cannot seek through file

[0i]

(you don't have to strictly follow this form)

Describe the bug
A clear and concise description of what the bug is.
message: "Received from srv01:9000, 172.31.17.253. DB::ErrnoException. DB::ErrnoException: Cannot seek through file /data/store/clickhouse/data/xxx/logs/20190214_36275967_36282135_37/logs.bin, errno: 22, strerror: Invalid argument: (while reading column clicks): (while reading from part /data/store/clickhouse/data/logs/clicks_log/20190214_36275967_36282135_37/ from mark 0 with max_rows_to_read = 32768).↵IN:SELECT SUM(clicks) AS total_clicks FROM logs WHERE log_date BETWEEN '2019-02-10' AND '2019-02-17' AND log_hour BETWEEN '2019-02-10 16:00:00' AND '2019-02-17 15:59:59' AND nid = 1000 LIMIT 0,10 FORMAT JSON"

How to reproduce
It happens often, after restarting the clickhouse, it will be normal.

• Which ClickHouse server version to use
  19.1.6

• Which interface to use, if matters
  All

• Non-default settings, if any

• CREATE TABLE statements for all tables involved
• Sample data for all these tables, use clickhouse-obfuscator if necessary
• Queries to run that lead to unexpected result

Expected behavior
A clear and concise description of what you expected to happen.

Error message and/or stacktrace
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[dr1nf3rn0]

Reproduced on 19.1.6. Full stack:

2019.02.13 09:18:02.339898 [ 66 ] {} <Error> HTTPHandler: Code: 87, e.displayText() = DB::ErrnoException: Cannot seek through file /var/lib/clickhouse/data/DS_BMS_TEST/CC_STORE_DIM/f0478438e7458a6ce363150e36495c3d_0_0_0/BRANCH_NAME.bin, errno: 22, strerror: Invalid argument: (while reading column BRANCH_NAME): (while reading from part /var/lib/clickhouse/data/DS_BMS_TEST/CC_STORE_DIM/f0478438e7458a6ce363150e36495c3d_0_0_0/ from mark 0 with max_rows_to_read = 1), e.what() = DB::ErrnoException, Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x9fde646]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3317a42]
2. /usr/bin/clickhouse-server(DB::throwFromErrno(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)+0x1d8) [0x9fc1c88]
3. /usr/bin/clickhouse-server(DB::ReadBufferFromFileDescriptor::doSeek(long, int)+0x24d) [0x9ffbaad]
4. /usr/bin/clickhouse-server(DB::CompressedReadBufferFromFile::seek(unsigned long, unsigned long)+0x70) [0x76af920]
5. /usr/bin/clickhouse-server(DB::MergeTreeReader::Stream::seekToMark(unsigned long)+0x85) [0x75b4735]
6. /usr/bin/clickhouse-server() [0x75b4d0f]
7. /usr/bin/clickhouse-server(DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, unsigned long, DB::IDataType::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::IDataType::DeserializeBinaryBulkState>&) const+0x28) [0x3482ad8]
8. /usr/bin/clickhouse-server(DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)+0x2b2) [0x75b0512]
9. /usr/bin/clickhouse-server(DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)+0x387) [0x75b0ce7]
10. /usr/bin/clickhouse-server(DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)+0x136) [0x75abdd6]
11. /usr/bin/clickhouse-server(DB::MergeTreeRangeReader::continueReadingChain(DB::MergeTreeRangeReader::ReadResult&)+0x153) [0x75ad763]
12. /usr/bin/clickhouse-server(DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)+0x997) [0x75ae647]
13. /usr/bin/clickhouse-server(DB::MergeTreeBaseSelectBlockInputStream::readFromPart()+0x7e8) [0x75a6858]
14. /usr/bin/clickhouse-server(DB::MergeTreeBaseSelectBlockInputStream::readImpl()+0xbd) [0x75a7afd]
15. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
16. /usr/bin/clickhouse-server(DB::ExpressionBlockInputStream::readImpl()+0x2e) [0x73adb6e]
17. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
18. /usr/bin/clickhouse-server(DB::FilterBlockInputStream::readImpl()+0xbd) [0x73ae89d]
19. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
20. /usr/bin/clickhouse-server(DB::ExpressionBlockInputStream::readImpl()+0x2e) [0x73adb6e]
21. /usr/bin/clickhouse-server(DB::IProfilingBlockInputStream::read()+0x262) [0x6e21b62]
22. /usr/bin/clickhouse-server(DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)+0x12d) [0x6f7582d]
23. /usr/bin/clickhouse-server(DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)+0x200) [0x6f75e50]
24. /usr/bin/clickhouse-server(std::thread::_State_impl<std::thread::_Invoker<std::tuple<DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::process()::{lambda()#1}> > >::_M_run()+0x4f) [0x6f7633f]
25. /usr/bin/clickhouse-server() [0xaa3e73f]
26. /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4) [0x7fb181a3e4a4]
27. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb181070d0f]

```
BRANCH_NAME is of string datatype

[den-crane]

Do you use Docker ?
Do you use LowCardinality?

[0i]

Do you use Docker ?
Not use Docker

Do you use LowCardinality?
Not use LowCardinality

I upgraded ClickHouse to 19.3.4 and this error will occur, but 19.3.4 will automatically resume after a while. 19.1.6 I have to restart ClickHouse.

[M1ha-Shvn]

Same on 19.1.8. No docker:

Cannot seek through file /vol1/clickhouse/data/carrot/chevent/23139-201902_0_1342_205/user_id.bin, errno: 22, strerror: Invalid argument: (while reading column user_id): (while reading from part /vol1/clickhouse/data/carrot/chevent/23139-201902_0_1342_205/ from mark 0 with max_rows_to_read = 16384) (87)

[den-crane]

What filesystem are you using?
@0i through file /data/store/clickhouse
@M1hacka through file /vol1/clickhouse/data

It maybe related to DIRECT_IO.
[ 18.16.0 . The min_merge_bytes_to_use_direct_io option is set to 10 GiB by default. ]

https://github.com/yandex/ClickHouse/issues/4449

[M1ha-Shvn]

@den-crane I use XFS for /vol1

[0i]

@den-crane

I got this bug at today, And must restart ClickHouse. 😭
I use ext4 for /data/store.

root@srv:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvdf 1.5T 1.1T 375G 74% /data/store

[0i]

Just now, just restarted this bug again, how should I fix this bug? It seriously affects the availability of our system.

@den-crane

[dr1nf3rn0]

@0i
Rolling back to 18.6.1 version fixed this problem for us. Stop all clickhouse services and apt-get install specific version of clickhouse-server clickhouse-client clickhouse-common-static packages

[0i]

@alexey-milovidov
I think this bug is very serious. Can help us fix this bug?
thank you very much!