Onpremise: Install Error with Sentry 10.0.0

[cedricyaal]

I've been installing sentry 10.0.0 from scratch a lot times this week without issue (by setting SENTRY_IMAGE="getsentry/sentry:10.0.0")

Since Friday 7 Feb 2020 afternoon, I am getting this error:

+ exec gosu snuba snuba bootstrap --force
2020-02-07 20:17:05,382 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-02-07 20:17:07,387 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-02-07 20:17:09,391 Connection to Kafka failed (attempt 2)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-02-07 20:17:10,703 Topic outcomes created
2020-02-07 20:17:10,703 Topic cdc created
2020-02-07 20:17:10,703 Topic events created
2020-02-07 20:17:10,704 Topic event-replacements created
2020-02-07 20:17:10,704 Topic snuba-commit-log created
2020-02-07 20:17:10,743 Tables for dataset outcomes created.
2020-02-07 20:17:10,745 Tables for dataset outcomes_raw created.
2020-02-07 20:17:10,748 Tables for dataset groupassignee created.
2020-02-07 20:17:10,749 Tables for dataset discover created.
Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 11, in <module>
    load_entry_point('snuba', 'console_scripts', 'snuba')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 138, in bootstrap
    conn.execute(statement.statement)
  File "/usr/src/snuba/snuba/clickhouse/native.py", line 60, in execute
    result = conn.execute(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/clickhouse_driver/client.py", line 214, in execute
    columnar=columnar
  File "/usr/local/lib/python3.7/site-packages/clickhouse_driver/client.py", line 334, in process_ordinary_query
    columnar=columnar)
  File "/usr/local/lib/python3.7/site-packages/clickhouse_driver/client.py", line 83, in receive_result
    return result.get_result()
  File "/usr/local/lib/python3.7/site-packages/clickhouse_driver/result.py", line 48, in get_result
    for packet in self.packet_generator:
  File "/usr/local/lib/python3.7/site-packages/clickhouse_driver/client.py", line 95, in packet_generator
    packet = self.receive_packet()
  File "/usr/local/lib/python3.7/site-packages/clickhouse_driver/client.py", line 112, in receive_packet
    raise packet.exception
clickhouse_driver.errors.ServerException: Code: 62.
DB::Exception: Syntax error: failed at position 2122 (line 7, col 13): TTL timestamp + toIntervalDay(retention_days)
            SETTINGS index_granularity=8192;. Expected one of: NOT, SETTINGS, FORMAT, LIKE, AND, OR, IN, BETWEEN, token, IS, NOT LIKE, NOT IN, GLOBAL IN, GLOBAL NOT IN, Comma, Dot, Arrow, QuestionMark, PRIMARY KEY, INTO OUTFILE. Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6f8abf6]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x372c822]
2. /usr/bin/clickhouse-server(DB::parseQuery(DB::IParser&, char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)+0xac) [0x6c1e36c]
3. /usr/bin/clickhouse-server() [0x6574305]
4. /usr/bin/clickhouse-server(DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool)+0x81) [0x6576141]
5. /usr/bin/clickhouse-server(DB::TCPHandler::runImpl()+0x752) [0x3739f82]
6. /usr/bin/clickhouse-server(DB::TCPHandler::run()+0x2b) [0x373a5cb]
7. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x708e5af]
8. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0xe9) [0x708ece9]
9. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x81) [0x7141f81]
10. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x713e148]
11. /usr/bin/clickhouse-server() [0xae06d1f]
12. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f714c57c6db]
13. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f714bafb88f]

This seems very similar to https://forum.sentry.io/t/new-sentry-10-docker-and-git-versions-unable-to-get-past-error/8047/21 that you fixed with https://github.com/getsentry/onpremise/pull/220/commits/bb9e4ea7ffe88231775176fab4500e43a9174fe4

For the sake of getting it work, I change my local ./install.sh with that

-$dcr snuba-api bootstrap --force
-$dcr snuba-api migrate

+
+until $($dcr clickhouse clickhouse-client -h clickhouse --query="SHOW TABLES;" | grep -q sentry_local); do
+  # `bootstrap` is for fresh installs, and `migrate` is for existing installs
+  # Running them both for both cases is harmless so we blindly run them
+  $dcr snuba-api bootstrap --force || true;
+  $dcr snuba-api migrate || true;
+done;

I have no idea of the implication of that.

The included install.sh script is supposed to be idempotent, which he is not :(.

By looking at the docker-compose.yml file, I am guessing the culprit are those two images which don't have a fixed version:
getsentry/symbolicator:latest
getsentry/snuba:latest'

Ideally, It would be nice to have a "10.0.0" tag on getsentry/onpremise depot, which would have the right dependencies setup.

Tks

[BYK]

This has nothing to do with symbolicator but it is related to Snuba, yes. I'll be investigating this.

[BYK]

I think this is resolved by using a more recent version of Clickhouse, which is already in the repo as 19.11?

[ebdavison]

I am encountering the same error here using a clone of the repo downloaded today which has 19.11 already in the docker-compose.yml file.

[BYK]

@ebdavison is this a fresh install or an upgrade?

[l0nax]

We are facing the same issue.
And I also tried to set up a fresh instance but this makes no difference.

[zetaab]

for me upgrading clickhouse from 19.4 -> 19.11 solved this issue

[macnibblet]

I'm running 19.17 and have this issue :/

[maxp1256]

Are there any updates? I've the same problem while upgrading from v10 -> v10.latest
How can I upgrade clickhouse?

[pnghai]

+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ set gosu snuba snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
2020-05-14 10:44:56,041 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}

Got this one on latest commit too, what should I do :( ?

[BYK]

I think the original issue here is solved. For anyone having Kafka related issues, please follow #472 instead.