Clickhouse: segmentation fault on clickhouse-copier (v 19.4.1.3 with revision 54416)

Created on 3 Apr 2019 · 4Comments · Source: ClickHouse/ClickHouse

Try copy table from standalone node clickhouse (19.4.0) to 3 shards cluster in kubernetes (19.4.1.3)
Use for copy Job with image also like in a cluster.

i get error

root@kube-clickhouse-stage-copier-pkg9w:/# clickhouse-copier copier --config /app/zookeeper-servers.xml --task-path=/clickhouse/copytasks/analytics --log-level=debug
Include not found: zookeeper-servers
Include not found: clickhouse_distributed_ddl
Logging debug to /clickhouse-copier_20190403094407_60/log.log
Logging errors to /clickhouse-copier_20190403094407_60/log.err.log
Logging debug to console
2019.04.03 09:44:07.630320 [ 1 ] {} <Information> : Starting ClickHouse 19.4.1.3 with revision 54416
2019.04.03 09:44:07.630618 [ 1 ] {} <Information> Application: Starting clickhouse-copier (id 20190403094407_60, host_id kube%2Dclickhouse%2Dstage%2Dcopier%2Dpkg9w#20190403094407_60, path /clickhouse-copier_20190403094407_60, revision 54416)
2019.04.03 09:44:07.653418 [ 1 ] {} <Debug> ClusterCopier: Loading description, zxid=0
2019.04.03 09:44:07.655258 [ 4 ] {} <Error> BaseDaemon: ########################################
2019.04.03 09:44:07.655375 [ 4 ] {} <Error> BaseDaemon: (from thread 1) Received signal Segmentation fault (11).
2019.04.03 09:44:07.655441 [ 4 ] {} <Error> BaseDaemon: Address: NULL pointer.
2019.04.03 09:44:07.655479 [ 4 ] {} <Error> BaseDaemon: Access: read.
2019.04.03 09:44:07.655510 [ 4 ] {} <Error> BaseDaemon: Address not mapped to object.
2019.04.03 09:44:07.692719 [ 4 ] {} <Error> BaseDaemon: 0. clickhouse-copier(DB::ClusterCopier::init()+0xea2) [0x3822642]
2019.04.03 09:44:07.692829 [ 4 ] {} <Error> BaseDaemon: 1. clickhouse-copier(DB::ClusterCopierApp::mainImpl()+0x539) [0x380da49]
2019.04.03 09:44:07.692903 [ 4 ] {} <Error> BaseDaemon: 2. clickhouse-copier(DB::ClusterCopierApp::main(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x42) [0x36715b2]
2019.04.03 09:44:07.692950 [ 4 ] {} <Error> BaseDaemon: 3. clickhouse-copier(Poco::Util::Application::run()+0x35) [0x7086135]
2019.04.03 09:44:07.692985 [ 4 ] {} <Error> BaseDaemon: 4. clickhouse-copier(Poco::Util::ServerApplication::run(int, char**)+0xf0) [0x70a01f0]
2019.04.03 09:44:07.693019 [ 4 ] {} <Error> BaseDaemon: 5. clickhouse-copier(mainEntryClickHouseClusterCopier(int, char**)+0x144) [0x380c3f4]
2019.04.03 09:44:07.693052 [ 4 ] {} <Error> BaseDaemon: 6. clickhouse-copier(main+0x1e0) [0x3669f00]
2019.04.03 09:44:07.693295 [ 4 ] {} <Error> BaseDaemon: 7. /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f3fe9ec3b97]
Segmentation fault (core dumped)
root@kube-clickhouse-stage-copier-pkg9w:/#

With config

<yandex>
    <remote_servers>   
       <source_cluster>
          <shard>
              <weight>1</weight>
              <replica>
                  <host>192.168.100.69</host>
                  <port>9000</port>
              </replica>
          </shard>
      </source_cluster>     
      <default_cluster>

        <shard>
          <weight>1</weight>
          <replica>
              <host>kube-clickhouse-stage-0</host>
              <port>9000</port>
          </replica>
        </shard>

        <shard>
          <weight>1</weight>
          <replica>
              <host>kube-clickhouse-stage-1</host>
              <port>9000</port>
          </replica>
        </shard>

        <shard>
          <weight>1</weight>
          <replica>
              <host>kube-clickhouse-stage-2</host>
              <port>9000</port>
          </replica>
        </shard>

      </default_cluster>     
    </remote_servers>
    <max_workers>1</max_workers>

    <tables>
      <table_crm_fin_account>
        <cluster_pull>source_cluster</cluster_pull>
        <database_pull>analytics</database_pull>
        <table_pull>crm_fin_account</table_pull>

        <cluster_push>default_cluster</cluster_push>
        <database_push>analytics</database_push>
        <table_push>crm_fin_account</table_push>
        <engine>ENGINE = MergeTree PARTITION BY date ORDER BY date</engine>
        <sharding_key>rand()</sharding_key>
      </table_crm_fin_account>
    </tables>
</yandex>

Also i try to create table on cluster by hands - not helped.

What is problem?

bug comp-copier crash

Source

mobilesfinks

All 4 comments

Already fixed https://github.com/yandex/ClickHouse/pull/4835

proller on 3 Apr 2019

Update to the specified version 19.4.3.11
Same problem

root@kube-clickhouse-stage-copier-m9wr5:/app# ./copy_start.sh 
Include not found: zookeeper-servers
Include not found: clickhouse_distributed_ddl
Logging debug to /app/clickhouse-copier_20190403111609_48/log.log
Logging errors to /app/clickhouse-copier_20190403111609_48/log.err.log
Logging debug to console
2019.04.03 11:16:09.870627 [ 1 ] {} <Information> : Starting ClickHouse 19.4.3.11 with revision 54416
2019.04.03 11:16:09.870926 [ 1 ] {} <Information> Application: Starting clickhouse-copier (id 20190403111609_48, host_id kube%2Dclickhouse%2Dstage%2Dcopier%2Dm9wr5#20190403111609_48, path /app/clickhouse-copier_20190403111609_48, revision 54416)
2019.04.03 11:16:09.914187 [ 1 ] {} <Debug> ClusterCopier: Loading description, zxid=0
2019.04.03 11:16:09.915881 [ 4 ] {} <Error> BaseDaemon: ########################################
2019.04.03 11:16:09.916000 [ 4 ] {} <Error> BaseDaemon: (from thread 1) Received signal Segmentation fault (11).
2019.04.03 11:16:09.916116 [ 4 ] {} <Error> BaseDaemon: Address: NULL pointer.
2019.04.03 11:16:09.916154 [ 4 ] {} <Error> BaseDaemon: Access: read.
2019.04.03 11:16:09.916196 [ 4 ] {} <Error> BaseDaemon: Address not mapped to object.
2019.04.03 11:16:09.943806 [ 4 ] {} <Error> BaseDaemon: 0. clickhouse-copier(DB::ClusterCopier::init()+0xea2) [0x38227b2]
2019.04.03 11:16:09.943906 [ 4 ] {} <Error> BaseDaemon: 1. clickhouse-copier(DB::ClusterCopierApp::mainImpl()+0x539) [0x380dbb9]
2019.04.03 11:16:09.944000 [ 4 ] {} <Error> BaseDaemon: 2. clickhouse-copier(DB::ClusterCopierApp::main(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x42) [0x3671692]
2019.04.03 11:16:09.944061 [ 4 ] {} <Error> BaseDaemon: 3. clickhouse-copier(Poco::Util::Application::run()+0x35) [0x7086985]
2019.04.03 11:16:09.944108 [ 4 ] {} <Error> BaseDaemon: 4. clickhouse-copier(Poco::Util::ServerApplication::run(int, char**)+0xf0) [0x70a0a40]
2019.04.03 11:16:09.944152 [ 4 ] {} <Error> BaseDaemon: 5. clickhouse-copier(mainEntryClickHouseClusterCopier(int, char**)+0x144) [0x380c564]
2019.04.03 11:16:09.944185 [ 4 ] {} <Error> BaseDaemon: 6. clickhouse-copier(main+0x1e0) [0x3669fe0]
2019.04.03 11:16:09.944216 [ 4 ] {} <Error> BaseDaemon: 7. /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fa61064cb97]
./copy_start.sh: line 3:    48 Segmentation fault      (core dumped) clickhouse-copier copier --config /app/zookeeper-servers.xml --task-path=/clickhouse/copytasks/analytics --log-level=debug

mobilesfinks on 3 Apr 2019

The fix is not released yet.