Onpremise: Stats don't show rate limited and filtered events
I set a rate limit of 100 in the admin area. But when sentry rejects events with http status code 429
the stats area does not show any rate limited events:
This extends to events, that are filtered by Inbound Data Filters. Events are filtered but the gui show
Bug
marbon87
All 25 comments
Are you on the most recent version of the on-premise repo? We had some issues around stat counters which we fixed afterwards.
BYK
on 19 May 2020
Yes, I tested it with the latest version and a complete fresh setup this morning.
marbon87
on 19 May 2020
Can you share the output of the following commands please:
docker-compose logs snuba-outcomes-consumer
docker-compose logs ingest-consumer
/cc @untitaker
BYK
on 19 May 2020
Hey @marbon87 can you post your sentry.conf.py?
untitaker
on 19 May 2020
docker-compose logs snuba-outcomes-consumer
Attaching to sentry_onpremise_snuba-outcomes-consumer_1
[36msnuba-outcomes-consumer_1 |[0m + '[' c = - ']'
[36msnuba-outcomes-consumer_1 |[0m + snuba consumer --help
[36msnuba-outcomes-consumer_1 |[0m + set -- snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
[36msnuba-outcomes-consumer_1 |[0m + set gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
[36msnuba-outcomes-consumer_1 |[0m + exec gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:37:00,816 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 0}
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:38:00,802 Partitions revoked: [Partition(topic=Topic(name='outcomes'), index=0)]
[36msnuba-outcomes-consumer_1 |[0m + '[' c = - ']'
[36msnuba-outcomes-consumer_1 |[0m + snuba consumer --help
[36msnuba-outcomes-consumer_1 |[0m + set -- snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
[36msnuba-outcomes-consumer_1 |[0m + set gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
[36msnuba-outcomes-consumer_1 |[0m + exec gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:38:48,006 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 0}
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:38,379 Flushing 11 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=0, hi=10)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:38,541 Worker flush took 162ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:39,318 Flushing 83 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=11, hi=93)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:39,332 Worker flush took 13ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:40,103 Flushing 139 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=94, hi=232)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:40,125 Worker flush took 21ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:40,881 Flushing 79 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=233, hi=311)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:40,893 Worker flush took 12ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:41,720 Flushing 19 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=312, hi=330)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:41,725 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:42,501 Flushing 86 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=331, hi=416)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:42,515 Worker flush took 13ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:43,268 Flushing 91 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=417, hi=507)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:43,280 Worker flush took 11ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:44,048 Flushing 104 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=508, hi=611)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:44,063 Worker flush took 14ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:45,443 Flushing 60 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=612, hi=671)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:42:45,452 Worker flush took 8ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:11,950 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=672, hi=672)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:11,955 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:12,967 Flushing 58 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=673, hi=730)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:12,977 Worker flush took 9ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:13,728 Flushing 77 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=731, hi=807)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:13,742 Worker flush took 13ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:14,500 Flushing 85 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=808, hi=892)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:14,513 Worker flush took 12ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:15,266 Flushing 92 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=893, hi=984)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:15,278 Worker flush took 11ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:16,034 Flushing 82 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=985, hi=1066)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:16,045 Worker flush took 10ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:16,802 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1067, hi=1165)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:16,816 Worker flush took 13ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:17,576 Flushing 100 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1166, hi=1265)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:17,590 Worker flush took 13ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:18,346 Flushing 86 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1266, hi=1351)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:18,358 Worker flush took 11ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:19,594 Flushing 32 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1352, hi=1383)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:19,621 Worker flush took 26ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:45,446 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1384, hi=1384)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:45,450 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:46,479 Flushing 86 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1385, hi=1470)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:46,493 Worker flush took 13ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:47,283 Flushing 103 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1471, hi=1573)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:47,294 Worker flush took 11ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:48,050 Flushing 87 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1574, hi=1660)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:48,064 Worker flush took 14ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:48,817 Flushing 74 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1661, hi=1734)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:48,826 Worker flush took 8ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:49,579 Flushing 82 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1735, hi=1816)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:49,589 Worker flush took 9ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:50,342 Flushing 90 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1817, hi=1906)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:50,352 Worker flush took 9ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:51,107 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=1907, hi=2005)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:51,125 Worker flush took 17ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:52,739 Flushing 79 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2006, hi=2084)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:43:52,770 Worker flush took 30ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:33,253 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2085, hi=2085)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:33,258 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:34,198 Flushing 87 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2086, hi=2172)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:34,211 Worker flush took 12ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:34,967 Flushing 114 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2173, hi=2286)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:34,984 Worker flush took 17ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:35,765 Flushing 95 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2287, hi=2381)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:35,775 Worker flush took 9ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:36,533 Flushing 96 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2382, hi=2477)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:36,549 Worker flush took 15ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:37,303 Flushing 77 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2478, hi=2554)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:37,316 Worker flush took 12ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:38,070 Flushing 92 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2555, hi=2646)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:38,084 Worker flush took 13ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:38,840 Flushing 106 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2647, hi=2752)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:38,853 Worker flush took 12ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:40,243 Flushing 58 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2753, hi=2810)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:44:40,271 Worker flush took 26ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:48:16,961 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2811, hi=2811)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:48:16,973 Worker flush took 11ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:48:18,259 Flushing 25 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2812, hi=2836)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:48:18,266 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:48:19,800 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2837, hi=2935)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 04:48:19,810 Worker flush took 9ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:07,034 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2936, hi=2936)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:07,039 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:08,312 Flushing 8 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2937, hi=2944)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:08,316 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:09,241 Flushing 91 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=2945, hi=3035)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:09,249 Worker flush took 7ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:10,452 Flushing 8 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3036, hi=3043)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:10,467 Worker flush took 14ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:42,386 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3044, hi=3044)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:04:42,392 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:05:00,361 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3045, hi=3045)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:05:00,368 Worker flush took 7ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:05:02,932 Flushing 6 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3046, hi=3051)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:05:02,935 Worker flush took 2ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:05:51,257 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3052, hi=3052)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:05:51,263 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:06,366 Flushing 2 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3053, hi=3054)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:06,371 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:07,373 Flushing 11 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3055, hi=3065)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:07,377 Worker flush took 2ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:09,047 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3066, hi=3164)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:09,056 Worker flush took 9ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:24,112 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3165, hi=3165)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:24,118 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:26,339 Flushing 4 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3166, hi=3169)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:06:26,342 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:08:18,596 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3170, hi=3170)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:08:18,602 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:08:19,810 Flushing 8 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3171, hi=3178)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:08:19,813 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:08:21,643 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3179, hi=3277)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:08:21,674 Worker flush took 30ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:08:30,558 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3278, hi=3278)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:08:30,564 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:12,571 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3279, hi=3279)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:12,577 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:13,937 Flushing 6 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3280, hi=3285)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:13,940 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:14,870 Flushing 98 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3286, hi=3383)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:14,884 Worker flush took 13ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:15,885 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3384, hi=3384)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:15,891 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:30,910 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3385, hi=3385)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:12:30,917 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:14:21,898 Flushing 2 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3386, hi=3387)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:14:21,904 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:14:22,905 Flushing 4 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3388, hi=3391)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:14:22,912 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:14:24,609 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3392, hi=3490)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:14:24,649 Worker flush took 40ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:14:39,065 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3491, hi=3491)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:14:39,072 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:18:42,707 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3492, hi=3492)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:18:42,714 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:18:44,188 Flushing 593 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=3493, hi=4085)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:18:44,217 Worker flush took 29ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:20:31,635 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=4086, hi=4086)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:20:31,639 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:20:33,013 Flushing 712 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=4087, hi=4798)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:20:33,081 Worker flush took 67ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:21:01,592 Flushing 2 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=4799, hi=4800)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:21:01,597 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:21:02,354 Flushing 91 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=4801, hi=4891)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:21:02,367 Worker flush took 12ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:21:03,238 Flushing 12 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=4892, hi=4903)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:21:03,242 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:21:04,277 Flushing 5 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=4904, hi=4908)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:21:04,283 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:51:15,131 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=4909, hi=4909)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:51:15,136 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:51:16,584 Flushing 751 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=4910, hi=5660)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:51:16,617 Worker flush took 32ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:57:27,503 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=5661, hi=5661)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:57:27,510 Worker flush took 7ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:57:29,328 Flushing 559 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=5662, hi=6220)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:57:29,354 Worker flush took 25ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:57:40,724 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=6221, hi=6221)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:57:40,731 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:57:42,831 Flushing 728 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=6222, hi=6949)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:57:42,868 Worker flush took 36ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:58:20,967 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=6950, hi=6950)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:58:20,971 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:58:22,796 Flushing 732 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=6951, hi=7682)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:58:22,833 Worker flush took 36ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:59:04,808 Flushing 2 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7683, hi=7684)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:59:04,813 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:59:05,815 Flushing 7 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7685, hi=7691)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:59:05,820 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:59:07,605 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7692, hi=7790)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 05:59:07,637 Worker flush took 31ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:02:49,009 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7791, hi=7791)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:02:49,015 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:02:50,458 Flushing 15 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7792, hi=7806)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:02:50,461 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:02:52,339 Flushing 100 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7807, hi=7906)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:02:52,351 Worker flush took 11ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:04:19,976 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7907, hi=7907)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:04:19,980 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:04:21,555 Flushing 24 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7908, hi=7931)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:04:21,574 Worker flush took 17ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:04:22,776 Flushing 100 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=7932, hi=8031)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:04:22,797 Worker flush took 20ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:04:24,303 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=8032, hi=8032)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:04:24,308 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:08:14,784 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=8033, hi=8033)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:08:14,794 Worker flush took 9ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:08:16,352 Flushing 13 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=8034, hi=8046)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:08:16,358 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:08:17,298 Flushing 83 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=8047, hi=8129)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:08:17,309 Worker flush took 10ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:08:18,407 Flushing 16 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=8130, hi=8145)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:08:18,413 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:02,136 Flushing 129 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=8146, hi=8274)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:02,148 Worker flush took 11ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:54,385 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=8275, hi=8275)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:54,389 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:55,398 Flushing 1879 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=8276, hi=10154)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:55,576 Worker flush took 176ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:56,327 Flushing 3144 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=10155, hi=13298)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:56,621 Worker flush took 293ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:57,375 Flushing 3358 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=13299, hi=16656)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:57,675 Worker flush took 299ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:58,427 Flushing 4473 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=16657, hi=21129)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:58,827 Worker flush took 399ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:59,581 Flushing 4408 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=21130, hi=25537)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:10:59,959 Worker flush took 377ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:00,711 Flushing 4237 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=25538, hi=29774)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:01,082 Worker flush took 370ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:01,838 Flushing 4251 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=29775, hi=34025)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:02,205 Worker flush took 366ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:02,958 Flushing 4395 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=34026, hi=38420)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:03,362 Worker flush took 404ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:04,116 Flushing 4274 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=38421, hi=42694)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:04,507 Worker flush took 390ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:05,262 Flushing 4226 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=42695, hi=46920)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:05,647 Worker flush took 385ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:06,400 Flushing 4367 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=46921, hi=51287)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:06,823 Worker flush took 422ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:07,575 Flushing 4292 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=51288, hi=55579)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:07,949 Worker flush took 373ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:08,703 Flushing 4144 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=55580, hi=59723)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:09,073 Worker flush took 370ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:09,826 Flushing 4235 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=59724, hi=63958)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:10,231 Worker flush took 404ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:10,983 Flushing 4358 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=63959, hi=68316)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:11,377 Worker flush took 393ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:12,130 Flushing 4125 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=68317, hi=72441)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:12,498 Worker flush took 367ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:13,251 Flushing 4555 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=72442, hi=76996)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:13,672 Worker flush took 420ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:14,425 Flushing 4578 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=76997, hi=81574)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:14,695 Worker flush took 269ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:15,447 Flushing 7076 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=81575, hi=88650)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:15,820 Worker flush took 373ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:16,573 Flushing 7120 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=88651, hi=95770)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:16,889 Worker flush took 315ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:17,642 Flushing 7490 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=95771, hi=103260)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:18,055 Worker flush took 412ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:18,808 Flushing 6943 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=103261, hi=110203)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:19,163 Worker flush took 354ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:20,169 Flushing 33 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110204, hi=110236)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:11:20,175 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:22:35,207 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110237, hi=110237)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:22:35,212 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:22:36,278 Flushing 22 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110238, hi=110259)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:22:36,284 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:22:38,096 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110260, hi=110358)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:22:38,103 Worker flush took 6ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:23:14,295 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110359, hi=110359)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:23:14,313 Worker flush took 17ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:23:15,472 Flushing 10 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110360, hi=110369)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:23:15,477 Worker flush took 4ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:23:17,163 Flushing 99 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110370, hi=110468)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:23:17,174 Worker flush took 10ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:16,576 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110469, hi=110469)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:16,582 Worker flush took 5ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:18,199 Flushing 7 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110470, hi=110476)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:18,203 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:19,127 Flushing 96 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110477, hi=110572)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:19,139 Worker flush took 11ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:20,140 Flushing 3 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110573, hi=110575)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:20,144 Worker flush took 3ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:35,483 Flushing 1 items (from {Partition(topic=Topic(name='outcomes'), index=0): Offsets(lo=110576, hi=110576)}): forced:False size:False time:True
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:24:35,500 Worker flush took 17ms
[36msnuba-outcomes-consumer_1 |[0m 2020-05-19 06:41:43,678 Partitions revoked: [Partition(topic=Topic(name='outcomes'), index=0)]
docker-compose logs ingest-consumer
Attaching to sentry_onpremise_ingest-consumer_1
[36mingest-consumer_1 |[0m 04:38:43 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
[36mingest-consumer_1 |[0m 04:38:46 [INFO] sentry.plugins.github: apps-not-configured
[36mingest-consumer_1 |[0m 04:38:51 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]
[36mingest-consumer_1 |[0m 04:42:36 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [0L, 0L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:42:36 [INFO] batching-kafka-consumer: Worker flush took 26ms
[36mingest-consumer_1 |[0m 04:42:37 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1L, 100L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:42:38 [INFO] batching-kafka-consumer: Worker flush took 791ms
[36mingest-consumer_1 |[0m 04:42:38 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [101L, 200L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:42:38 [INFO] batching-kafka-consumer: Worker flush took 627ms
[36mingest-consumer_1 |[0m 04:42:39 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [201L, 300L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:42:39 [INFO] batching-kafka-consumer: Worker flush took 792ms
[36mingest-consumer_1 |[0m 04:42:39 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [301L, 400L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:42:40 [INFO] batching-kafka-consumer: Worker flush took 648ms
[36mingest-consumer_1 |[0m 04:42:40 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [401L, 500L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:42:41 [INFO] batching-kafka-consumer: Worker flush took 490ms
[36mingest-consumer_1 |[0m 04:42:41 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [501L, 600L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:42:41 [INFO] batching-kafka-consumer: Worker flush took 396ms
[36mingest-consumer_1 |[0m 04:42:42 [INFO] batching-kafka-consumer: Flushing 71 items (from {(u'ingest-events', 0): [601L, 671L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:42:42 [INFO] batching-kafka-consumer: Worker flush took 431ms
[36mingest-consumer_1 |[0m 04:43:10 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [672L, 672L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:43:10 [INFO] batching-kafka-consumer: Worker flush took 5ms
[36mingest-consumer_1 |[0m 04:43:12 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [673L, 772L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:12 [INFO] batching-kafka-consumer: Worker flush took 719ms
[36mingest-consumer_1 |[0m 04:43:12 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [773L, 872L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:13 [INFO] batching-kafka-consumer: Worker flush took 595ms
[36mingest-consumer_1 |[0m 04:43:13 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [873L, 972L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:14 [INFO] batching-kafka-consumer: Worker flush took 670ms
[36mingest-consumer_1 |[0m 04:43:14 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [973L, 1072L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:14 [INFO] batching-kafka-consumer: Worker flush took 528ms
[36mingest-consumer_1 |[0m 04:43:14 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1073L, 1172L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:15 [INFO] batching-kafka-consumer: Worker flush took 600ms
[36mingest-consumer_1 |[0m 04:43:15 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1173L, 1272L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:16 [INFO] batching-kafka-consumer: Worker flush took 602ms
[36mingest-consumer_1 |[0m 04:43:16 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1273L, 1372L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:16 [INFO] batching-kafka-consumer: Worker flush took 557ms
[36mingest-consumer_1 |[0m 04:43:17 [INFO] batching-kafka-consumer: Flushing 11 items (from {(u'ingest-events', 0): [1373L, 1383L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:43:17 [INFO] batching-kafka-consumer: Worker flush took 69ms
[36mingest-consumer_1 |[0m 04:43:44 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [1384L, 1384L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:43:44 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 04:43:45 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1385L, 1484L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:46 [INFO] batching-kafka-consumer: Worker flush took 687ms
[36mingest-consumer_1 |[0m 04:43:46 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1485L, 1584L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:47 [INFO] batching-kafka-consumer: Worker flush took 593ms
[36mingest-consumer_1 |[0m 04:43:47 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1585L, 1684L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:47 [INFO] batching-kafka-consumer: Worker flush took 634ms
[36mingest-consumer_1 |[0m 04:43:47 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1685L, 1784L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:48 [INFO] batching-kafka-consumer: Worker flush took 577ms
[36mingest-consumer_1 |[0m 04:43:48 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1785L, 1884L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:48 [INFO] batching-kafka-consumer: Worker flush took 630ms
[36mingest-consumer_1 |[0m 04:43:49 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1885L, 1984L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:49 [INFO] batching-kafka-consumer: Worker flush took 606ms
[36mingest-consumer_1 |[0m 04:43:49 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [1985L, 2084L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:43:50 [INFO] batching-kafka-consumer: Worker flush took 556ms
[36mingest-consumer_1 |[0m 04:44:32 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [2085L, 2085L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:44:32 [INFO] batching-kafka-consumer: Worker flush took 5ms
[36mingest-consumer_1 |[0m 04:44:33 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [2086L, 2185L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:44:34 [INFO] batching-kafka-consumer: Worker flush took 695ms
[36mingest-consumer_1 |[0m 04:44:34 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [2186L, 2285L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:44:34 [INFO] batching-kafka-consumer: Worker flush took 560ms
[36mingest-consumer_1 |[0m 04:44:34 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [2286L, 2385L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:44:35 [INFO] batching-kafka-consumer: Worker flush took 605ms
[36mingest-consumer_1 |[0m 04:44:35 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [2386L, 2485L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:44:36 [INFO] batching-kafka-consumer: Worker flush took 557ms
[36mingest-consumer_1 |[0m 04:44:36 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [2486L, 2585L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:44:36 [INFO] batching-kafka-consumer: Worker flush took 587ms
[36mingest-consumer_1 |[0m 04:44:36 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [2586L, 2685L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:44:37 [INFO] batching-kafka-consumer: Worker flush took 636ms
[36mingest-consumer_1 |[0m 04:44:37 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [2686L, 2785L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 04:44:37 [INFO] batching-kafka-consumer: Worker flush took 575ms
[36mingest-consumer_1 |[0m 04:44:39 [INFO] batching-kafka-consumer: Flushing 25 items (from {(u'ingest-events', 0): [2786L, 2810L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:44:39 [INFO] batching-kafka-consumer: Worker flush took 138ms
[36mingest-consumer_1 |[0m 04:48:15 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [2811L, 2811L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:48:15 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 04:48:18 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [2812L, 2910L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 04:48:18 [INFO] batching-kafka-consumer: Worker flush took 528ms
[36mingest-consumer_1 |[0m 05:04:05 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [2911L, 2911L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:04:05 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 05:04:08 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [2912L, 3010L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:04:08 [INFO] batching-kafka-consumer: Worker flush took 493ms
[36mingest-consumer_1 |[0m 05:06:05 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3011L, 3011L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:06:05 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 05:06:07 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [3012L, 3110L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:06:07 [INFO] batching-kafka-consumer: Worker flush took 509ms
[36mingest-consumer_1 |[0m 05:08:17 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3111L, 3111L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:08:17 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 05:08:19 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [3112L, 3210L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:08:20 [INFO] batching-kafka-consumer: Worker flush took 531ms
[36mingest-consumer_1 |[0m 05:12:11 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3211L, 3211L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:12:11 [INFO] batching-kafka-consumer: Worker flush took 10ms
[36mingest-consumer_1 |[0m 05:12:14 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [3212L, 3310L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:12:14 [INFO] batching-kafka-consumer: Worker flush took 605ms
[36mingest-consumer_1 |[0m 05:14:20 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3311L, 3311L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:14:20 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 05:14:22 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [3312L, 3410L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:14:23 [INFO] batching-kafka-consumer: Worker flush took 543ms
[36mingest-consumer_1 |[0m 05:18:41 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3411L, 3411L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:18:41 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 05:20:30 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3412L, 3412L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:20:30 [INFO] batching-kafka-consumer: Worker flush took 7ms
[36mingest-consumer_1 |[0m 05:21:00 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3413L, 3413L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:21:00 [INFO] batching-kafka-consumer: Worker flush took 5ms
[36mingest-consumer_1 |[0m 05:21:01 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [3414L, 3513L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 05:21:02 [INFO] batching-kafka-consumer: Worker flush took 675ms
[36mingest-consumer_1 |[0m 05:51:14 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3514L, 3514L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:51:14 [INFO] batching-kafka-consumer: Worker flush took 5ms
[36mingest-consumer_1 |[0m 05:57:26 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3515L, 3515L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:57:26 [INFO] batching-kafka-consumer: Worker flush took 7ms
[36mingest-consumer_1 |[0m 05:57:39 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3516L, 3516L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:57:39 [INFO] batching-kafka-consumer: Worker flush took 5ms
[36mingest-consumer_1 |[0m 05:58:19 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3517L, 3517L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:58:19 [INFO] batching-kafka-consumer: Worker flush took 5ms
[36mingest-consumer_1 |[0m 05:59:03 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3518L, 3518L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:59:03 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 05:59:05 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [3519L, 3617L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 05:59:06 [INFO] batching-kafka-consumer: Worker flush took 564ms
[36mingest-consumer_1 |[0m 06:02:47 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3618L, 3618L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:02:47 [INFO] batching-kafka-consumer: Worker flush took 5ms
[36mingest-consumer_1 |[0m 06:02:50 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [3619L, 3717L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:02:51 [INFO] batching-kafka-consumer: Worker flush took 543ms
[36mingest-consumer_1 |[0m 06:04:18 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3718L, 3718L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:04:18 [INFO] batching-kafka-consumer: Worker flush took 7ms
[36mingest-consumer_1 |[0m 06:04:21 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [3719L, 3817L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:04:22 [INFO] batching-kafka-consumer: Worker flush took 567ms
[36mingest-consumer_1 |[0m 06:08:13 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [3818L, 3818L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:08:13 [INFO] batching-kafka-consumer: Worker flush took 7ms
[36mingest-consumer_1 |[0m 06:08:16 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [3819L, 3917L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:08:17 [INFO] batching-kafka-consumer: Worker flush took 677ms
[36mingest-consumer_1 |[0m 06:10:00 [INFO] batching-kafka-consumer: Flushing 100 items (from {(u'ingest-events', 0): [3918L, 4017L]}): forced:False size:True time:False
[36mingest-consumer_1 |[0m 06:10:00 [INFO] batching-kafka-consumer: Worker flush took 571ms
[36mingest-consumer_1 |[0m 06:10:53 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [4018L, 4018L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:10:53 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 06:22:34 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [4019L, 4019L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:22:34 [INFO] batching-kafka-consumer: Worker flush took 6ms
[36mingest-consumer_1 |[0m 06:22:36 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [4020L, 4118L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:22:36 [INFO] batching-kafka-consumer: Worker flush took 576ms
[36mingest-consumer_1 |[0m 06:23:13 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [4119L, 4119L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:23:13 [INFO] batching-kafka-consumer: Worker flush took 5ms
[36mingest-consumer_1 |[0m 06:23:15 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [4120L, 4218L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:23:16 [INFO] batching-kafka-consumer: Worker flush took 538ms
[36mingest-consumer_1 |[0m 06:24:15 [INFO] batching-kafka-consumer: Flushing 1 items (from {(u'ingest-events', 0): [4219L, 4219L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:24:15 [INFO] batching-kafka-consumer: Worker flush took 7ms
[36mingest-consumer_1 |[0m 06:24:18 [INFO] batching-kafka-consumer: Flushing 99 items (from {(u'ingest-events', 0): [4220L, 4318L]}): forced:False size:False time:True
[36mingest-consumer_1 |[0m 06:24:18 [INFO] batching-kafka-consumer: Worker flush took 588ms
[36mingest-consumer_1 |[0m 06:41:32 [INFO] batching-kafka-consumer: Partitions revoked: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]
sentry.conf.py
# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.
from sentry.conf.server import * # NOQA
DATABASES = {
"default": {
"ENGINE": "sentry.db.postgres",
"NAME": "postgres",
"USER": "postgres",
"PASSWORD": "",
"HOST": "postgres",
"PORT": "",
}
}
# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True
# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings
###########
# General #
###########
# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True
SENTRY_OPTIONS["system.event-retention-days"] = int(env('SENTRY_EVENT_RETENTION_DAYS', '90'))
#########
# Redis #
#########
# Generic Redis configuration used as defaults for various things including:
# Buffers, Quotas, TSDB
SENTRY_OPTIONS["redis.clusters"] = {
"default": {
"hosts": {0: {"host": "redis", "password": "", "port": "6379", "db": "0"}}
}
}
#########
# Queue #
#########
# See https://docs.getsentry.com/on-premise/server/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.
rabbitmq_host = None
if rabbitmq_host:
BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
username="guest", password="guest", host=rabbitmq_host, vhost="/"
)
else:
BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
**SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
)
#########
# Cache #
#########
# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
"LOCATION": ["memcached:11211"],
"TIMEOUT": 3600,
}
}
# A primary cache is required for things such as processing events
SENTRY_CACHE = "sentry.cache.redis.RedisCache"
DEFAULT_KAFKA_OPTIONS = {
"bootstrap.servers": "kafka:9092",
"message.max.bytes": 50000000,
"socket.timeout.ms": 1000,
}
SENTRY_EVENTSTREAM = "sentry.eventstream.kafka.KafkaEventStream"
SENTRY_EVENTSTREAM_OPTIONS = {"producer_configuration": DEFAULT_KAFKA_OPTIONS}
KAFKA_CLUSTERS["default"] = DEFAULT_KAFKA_OPTIONS
###############
# Rate Limits #
###############
# Rate limits apply to notification handlers and are enforced per-project
# automatically.
SENTRY_RATELIMITER = "sentry.ratelimits.redis.RedisRateLimiter"
##################
# Update Buffers #
##################
# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)
SENTRY_BUFFER = "sentry.buffer.redis.RedisBuffer"
##########
# Quotas #
##########
# Quotas allow you to rate limit individual projects or the Sentry install as
# a whole.
SENTRY_QUOTAS = "sentry.quotas.redis.RedisQuota"
########
# TSDB #
########
# The TSDB is used for building charts as well as making things like per-rate
# alerts possible.
SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"
#########
# SNUBA #
#########
SENTRY_SEARCH = "sentry.search.snuba.EventsDatasetSnubaSearchBackend"
SENTRY_SEARCH_OPTIONS = {}
SENTRY_TAGSTORE_OPTIONS = {}
###########
# Digests #
###########
# The digest backend powers notification summaries.
SENTRY_DIGESTS = "sentry.digests.backends.redis.RedisBackend"
##############
# Web Server #
##############
SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
"memory-report": False,
# 'workers': 3, # the number of web workers
}
###########
# SSL/TLS #
###########
# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below
# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# SESSION_COOKIE_SECURE = True
# CSRF_COOKIE_SECURE = True
# SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
# End of SSL/TLS settings
############
# Features #
############
SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
{
feature: True
for feature in (
"organizations:discover",
"organizations:events",
"organizations:discover-basic",
"organizations:discover-query",
"organizations:events-v2",
"organizations:global-views",
"organizations:integrations-issue-basic",
"organizations:integrations-issue-sync",
"organizations:invite-members",
"organizations:sso-basic",
"organizations:sso-rippling",
"organizations:sso-saml2",
"projects:custom-inbound-filters",
"projects:data-forwarding",
"projects:discard-groups",
"projects:plugins",
"projects:rate-limits",
"projects:servicehooks",
)
}
)
######################
# GitHub Integration #
######################
GITHUB_EXTENDED_PERMISSIONS = ['repo']
#########################
# Bitbucket Integration #
########################
# BITBUCKET_CONSUMER_KEY = 'YOUR_BITBUCKET_CONSUMER_KEY'
# BITBUCKET_CONSUMER_SECRET = 'YOUR_BITBUCKET_CONSUMER_SECRET'
SENTRY_RELAY_WHITELIST_PK = (SENTRY_RELAY_WHITELIST_PK or []) + (["lgeHXYHpVoJ8_gOyummd193ov8xGvQtbt80DzIMd0SE"])
Everything looks fine to me except for this line:
锟絒36mingest-consumer_1 |锟絒0m 06:41:32 [INFO] batching-kafka-consumer: Partitions revoked: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]
@untitaker any ideas?
BYK
on 19 May 2020
A similar line appears on snuba-outcomes-consumer so I am not sure if this is indicative of an error or perhaps just the container being shut down.
When those stats do not appear I would check if the outcomes kafka topic actually gets new messages when an event has been rate limited or filtered. You should see one JSON message for every increment-by-one in your events accepted/filtered/ratelimited breakdown. I'm not sure what tool is best here but if you are getting any "flushed" messages at all from your snuba-outcomes-consumer when receiving nothing but rate limited or filtered events then it should be fine.
One last thing I would check is the output of docker-compose logs relay, if that doesn't yield anything I can try to reproduce this in a fresh Ubuntu VM but I was somewhat sure this worked for me already at some point.
untitaker
on 19 May 2020
docker-compose logs relay
Attaching to sentry_onpremise_relay_1
relay_1 | 2020-05-20T06:00:15Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.fritz.box. type: AAAA class: IN
relay_1 | caused by: Failed resolving hostname: no record found for name: web.fritz.box. type: AAAA class: IN
relay_1 | caused by: Failed resolving hostname: no record found for name: web.fritz.box. type: AAAA class: IN
relay_1 | 2020-05-20T06:00:16Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Timeout while waiting for response
relay_1 | 2020-05-20T06:00:17Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | 2020-05-20T06:00:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | 2020-05-20T06:00:23Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Server disconnected
relay_1 | caused by: Server disconnected
relay_1 | 2020-05-20T06:00:27Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-20T06:00:32Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
I have no idea why it want's to connect to web.fritz.box (.fritz.box is my local subnet)
marbon87
on 20 May 2020
I created a new environment from scratch with install.sh but the same error is logged for relay.
Here is the log from the install.sh:
禄 ./install.sh
set -e
dc="docker-compose --no-ansi"
dcr="$dc run --rm"
# Thanks to https://unix.stackexchange.com/a/145654/108960
log_file="sentry_install_log-`date +'%Y-%m-%d_%H-%M-%S'`.txt"
date +'%Y-%m-%d_%H-%M-%S'
exec &> >(tee -a "$log_file")
tee -a "$log_file"
MIN_DOCKER_VERSION='17.05.0'
MIN_COMPOSE_VERSION='1.23.0'
MIN_RAM=2400 # MB
SENTRY_CONFIG_PY='sentry/sentry.conf.py'
SENTRY_CONFIG_YML='sentry/config.yml'
RELAY_CONFIG_YML='relay/config.yml'
RELAY_CREDENTIALS_JSON='relay/credentials.json'
SENTRY_EXTRA_REQUIREMENTS='sentry/requirements.txt'
DID_CLEAN_UP=0
# the cleanup function will be the exit point
cleanup () {
if [ "$DID_CLEAN_UP" -eq 1 ]; then
return 0;
fi
echo "Cleaning up..."
$dc stop &> /dev/null
DID_CLEAN_UP=1
}
trap cleanup ERR INT TERM
echo "Checking minimum requirements..."
Checking minimum requirements...
DOCKER_VERSION=$(docker version --format '{{.Server.Version}}')
docker version --format '{{.Server.Version}}'
COMPOSE_VERSION=$($dc --version | sed 's/docker-compose version \(.\{1,\}\),.*/\1/')
$dc --version | sed 's/docker-compose version \(.\{1,\}\),.*/\1/'
RAM_AVAILABLE_IN_DOCKER=$(docker run --rm busybox free -m 2>/dev/null | awk '/Mem/ {print $2}');
docker run --rm busybox free -m 2>/dev/null | awk '/Mem/ {print $2}'
# Compare dot-separated strings - function below is inspired by https://stackoverflow.com/a/37939589/808368
function ver () { echo "$@" | awk -F. '{ printf("%d%03d%03d", $1,$2,$3); }'; }
# Thanks to https://stackoverflow.com/a/25123013/90297 for the quick `sed` pattern
function ensure_file_from_example {
if [ -f "$1" ]; then
echo "$1 already exists, skipped creation."
else
echo "Creating $1..."
cp -n $(echo "$1" | sed 's/\.[^.]*$/.example&/') "$1"
fi
}
if [ $(ver $DOCKER_VERSION) -lt $(ver $MIN_DOCKER_VERSION) ]; then
echo "FAIL: Expected minimum Docker version to be $MIN_DOCKER_VERSION but found $DOCKER_VERSION"
exit 1
fi
ver $DOCKER_VERSION
ver $MIN_DOCKER_VERSION
if [ $(ver $COMPOSE_VERSION) -lt $(ver $MIN_COMPOSE_VERSION) ]; then
echo "FAIL: Expected minimum docker-compose version to be $MIN_COMPOSE_VERSION but found $COMPOSE_VERSION"
exit 1
fi
ver $COMPOSE_VERSION
ver $MIN_COMPOSE_VERSION
if [ "$RAM_AVAILABLE_IN_DOCKER" -lt "$MIN_RAM" ]; then
echo "FAIL: Expected minimum RAM available to Docker to be $MIN_RAM MB but found $RAM_AVAILABLE_IN_DOCKER MB"
exit 1
fi
#SSE4.2 required by Clickhouse (https://clickhouse.yandex/docs/en/operations/requirements/)
# On KVM, cpuinfo could falsely not report SSE 4.2 support, so skip the check. https://github.com/ClickHouse/ClickHouse/issues/20#issuecomment-226849297
IS_KVM=$(docker run --rm busybox grep -c 'Common KVM processor' /proc/cpuinfo || :)
docker run --rm busybox grep -c 'Common KVM processor' /proc/cpuinfo || :
if (($IS_KVM == 0)); then
SUPPORTS_SSE42=$(docker run --rm busybox grep -c sse4_2 /proc/cpuinfo || :)
if (($SUPPORTS_SSE42 == 0)); then
echo "FAIL: The CPU your machine is running on does not support the SSE 4.2 instruction set, which is required for one of the services Sentry uses (Clickhouse). See https://git.io/JvLDt for more info."
exit 1
fi
fi
docker run --rm busybox grep -c sse4_2 /proc/cpuinfo || :
# Clean up old stuff and ensure nothing is working while we install/update
# This is for older versions of on-premise:
$dc -p onpremise down --rmi local --remove-orphans
Removing network onpremise_default
Network onpremise_default not found.
# This is for newer versions
$dc down --rmi local --remove-orphans
Removing network sentry_onpremise_default
Network sentry_onpremise_default not found.
echo ""
echo "Creating volumes for persistent storage..."
Creating volumes for persistent storage...
echo "Created $(docker volume create --name=sentry-data)."
docker volume create --name=sentry-data
Created sentry-data.
echo "Created $(docker volume create --name=sentry-postgres)."
docker volume create --name=sentry-postgres
Created sentry-postgres.
echo "Created $(docker volume create --name=sentry-redis)."
docker volume create --name=sentry-redis
Created sentry-redis.
echo "Created $(docker volume create --name=sentry-zookeeper)."
docker volume create --name=sentry-zookeeper
Created sentry-zookeeper.
echo "Created $(docker volume create --name=sentry-kafka)."
docker volume create --name=sentry-kafka
Created sentry-kafka.
echo "Created $(docker volume create --name=sentry-clickhouse)."
docker volume create --name=sentry-clickhouse
Created sentry-clickhouse.
echo "Created $(docker volume create --name=sentry-symbolicator)."
docker volume create --name=sentry-symbolicator
Created sentry-symbolicator.
echo ""
ensure_file_from_example $SENTRY_CONFIG_PY
sentry/sentry.conf.py already exists, skipped creation.
ensure_file_from_example $SENTRY_CONFIG_YML
sentry/config.yml already exists, skipped creation.
ensure_file_from_example $SENTRY_EXTRA_REQUIREMENTS
sentry/requirements.txt already exists, skipped creation.
if grep -xq "system.secret-key: '!!changeme!!'" $SENTRY_CONFIG_YML ; then
echo ""
echo "Generating secret key..."
# This is to escape the secret key to be used in sed below
# Note the need to set LC_ALL=C due to BSD tr and sed always trying to decode
# whatever is passed to them. Kudos to https://stackoverflow.com/a/23584470/90297
SECRET_KEY=$(export LC_ALL=C; head /dev/urandom | tr -dc "a-z0-9@#%^&*(-_=+)" | head -c 50 | sed -e 's/[\/&]/\\&/g')
sed -i -e 's/^system.secret-key:.*$/system.secret-key: '"'$SECRET_KEY'"'/' $SENTRY_CONFIG_YML
echo "Secret key written to $SENTRY_CONFIG_YML"
fi
replace_tsdb() {
if (
[ -f "$SENTRY_CONFIG_PY" ] &&
! grep -xq 'SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"' "$SENTRY_CONFIG_PY"
); then
tsdb_settings="SENTRY_TSDB = \"sentry.tsdb.redissnuba.RedisSnubaTSDB\"
# Automatic switchover 90 days after $(date). Can be removed afterwards.
SENTRY_TSDB_OPTIONS = {\"switchover_timestamp\": $(date +%s) + (90 * 24 * 3600)}"
if grep -q 'SENTRY_TSDB_OPTIONS = ' "$SENTRY_CONFIG_PY"; then
echo "Not attempting automatic TSDB migration due to presence of SENTRY_TSDB_OPTIONS"
else
echo "Attempting to automatically migrate to new TSDB"
# Escape newlines for sed
tsdb_settings="${tsdb_settings//$'\n'/\\n}"
cp "$SENTRY_CONFIG_PY" "$SENTRY_CONFIG_PY.bak"
sed -i -e "s/^SENTRY_TSDB = .*$/${tsdb_settings}/g" "$SENTRY_CONFIG_PY" || true
if grep -xq 'SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"' "$SENTRY_CONFIG_PY"; then
echo "Migrated TSDB to Snuba. Old configuration file backed up to $SENTRY_CONFIG_PY.bak"
return
fi
echo "Failed to automatically migrate TSDB. Reverting..."
mv "$SENTRY_CONFIG_PY.bak" "$SENTRY_CONFIG_PY"
echo "$SENTRY_CONFIG_PY restored from backup."
fi
echo "WARN: Your Sentry configuration uses a legacy data store for time-series data. Remove the options SENTRY_TSDB and SENTRY_TSDB_OPTIONS from $SENTRY_CONFIG_PY and add:"
echo ""
echo "$tsdb_settings"
echo ""
echo "For more information please refer to https://github.com/getsentry/onpremise/pull/430"
fi
}
replace_tsdb
echo ""
echo "Fetching and updating Docker images..."
Fetching and updating Docker images...
echo ""
# We tag locally built images with an '-onpremise-local' suffix. docker-compose pull tries to pull these too and
# shows a 404 error on the console which is confusing and unnecessary. To overcome this, we add the stderr>stdout
# redirection below and pass it through grep, ignoring all lines having this '-onpremise-local' suffix.
$dc pull -q --ignore-pull-failures 2>&1 | grep -v -- -onpremise-local || true
docker pull ${SENTRY_IMAGE:-getsentry/sentry:latest}
latest: Pulling from getsentry/sentry
Digest: sha256:ea27538efa3858da8e31a397dde47ac857f7eea5f76f475825a9723600aefec4
Status: Image is up to date for getsentry/sentry:latest
docker pull getsentry/snuba:latest
latest: Pulling from getsentry/snuba
Digest: sha256:6356cad084aedd0e1ebc5e39d0b7c5c6373c0046802f9ef776e2df9e5c5e0189
Status: Image is up to date for getsentry/snuba:latest
docker pull getsentry/symbolicator:latest
latest: Pulling from getsentry/symbolicator
Digest: sha256:d772af8fb4961ae8ca4377f3abd4f7048b900198ee8c4096fd9b9e120da6d054
Status: Image is up to date for getsentry/symbolicator:latest
docker pull getsentry/relay:latest
latest: Pulling from getsentry/relay
Digest: sha256:bded84963699f5bc5d5900d2d92369c30935ba71e19fb365d267808f42c9c40c
Status: Image is up to date for getsentry/relay:latest
echo ""
echo "Building and tagging Docker images..."
Building and tagging Docker images...
echo ""
# Build the sentry onpremise image first as it is needed for the cron image
$dc build --force-rm web
Building web
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
---> 32d3a26af6a7
Step 3/4 : COPY . /usr/src/sentry
---> Using cache
---> 39880702631f
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 49dd55316b22
Successfully built 49dd55316b22
Successfully tagged sentry-onpremise-local:latest
$dc build --force-rm
smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-replacer uses an image, skipping
symbolicator uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building snuba-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
---> 5a628e930b58
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*
---> Using cache
---> ddc83b7b0a02
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
---> Using cache
---> caac9ef71027
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
---> Using cache
---> 713799588bc4
Successfully built 713799588bc4
Successfully tagged snuba-cleanup-onpremise-local:latest
Building symbolicator-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
---> 515e555c0b18
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*
---> Using cache
---> 9e95508b94cf
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
---> Using cache
---> fcb0252a2878
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
---> Using cache
---> a56948a8b402
Successfully built a56948a8b402
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building web
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
---> 32d3a26af6a7
Step 3/4 : COPY . /usr/src/sentry
---> Using cache
---> 39880702631f
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 49dd55316b22
Successfully built 49dd55316b22
Successfully tagged sentry-onpremise-local:latest
Building cron
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
---> 32d3a26af6a7
Step 3/4 : COPY . /usr/src/sentry
---> Using cache
---> 39880702631f
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 49dd55316b22
Successfully built 49dd55316b22
Successfully tagged sentry-onpremise-local:latest
Building worker
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
---> 32d3a26af6a7
Step 3/4 : COPY . /usr/src/sentry
---> Using cache
---> 39880702631f
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 49dd55316b22
Successfully built 49dd55316b22
Successfully tagged sentry-onpremise-local:latest
Building ingest-consumer
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
---> 32d3a26af6a7
Step 3/4 : COPY . /usr/src/sentry
---> Using cache
---> 39880702631f
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 49dd55316b22
Successfully built 49dd55316b22
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
---> 32d3a26af6a7
Step 3/4 : COPY . /usr/src/sentry
---> Using cache
---> 39880702631f
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 49dd55316b22
Successfully built 49dd55316b22
Successfully tagged sentry-onpremise-local:latest
Building sentry-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
---> 49dd55316b22
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*
---> Using cache
---> 6f7c6d6be0a7
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
---> Using cache
---> 19fe5cc083ce
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
---> Using cache
---> 8fd338829e51
Successfully built 8fd338829e51
Successfully tagged sentry-cleanup-onpremise-local:latest
echo ""
echo "Docker images built."
Docker images built.
echo "Bootstrapping and migrating Snuba..."
Bootstrapping and migrating Snuba...
$dcr snuba-api bootstrap --force
Creating network "sentry_onpremise_default" with the default driver
Creating volume "sentry_onpremise_sentry-secrets" with default driver
Creating volume "sentry_onpremise_sentry-smtp" with default driver
Creating volume "sentry_onpremise_sentry-zookeeper-log" with default driver
Creating volume "sentry_onpremise_sentry-kafka-log" with default driver
Creating volume "sentry_onpremise_sentry-smtp-log" with default driver
Creating volume "sentry_onpremise_sentry-clickhouse-log" with default driver
Creating sentry_onpremise_redis_1 ...
Creating sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_zookeeper_1 ...
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_zookeeper_1 ... done
Creating sentry_onpremise_kafka_1 ...
Creating sentry_onpremise_redis_1 ... done
Creating sentry_onpremise_kafka_1 ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ set gosu snuba snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
2020-05-20 06:26:44,240 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-05-20 06:26:45,505 Topic outcomes created
2020-05-20 06:26:45,505 Topic ingest-sessions created
2020-05-20 06:26:45,505 Topic cdc created
2020-05-20 06:26:45,505 Topic events created
2020-05-20 06:26:45,505 Topic event-replacements created
2020-05-20 06:26:45,505 Topic snuba-commit-log created
2020-05-20 06:26:45,506 Topic errors-replacements created
2020-05-20 06:26:45,516 Creating tables for storage events
2020-05-20 06:26:45,524 Migrating storage events
2020-05-20 06:26:45,552 Creating tables for storage errors
2020-05-20 06:26:45,559 Migrating storage errors
2020-05-20 06:26:45,579 Creating tables for storage groupedmessages
2020-05-20 06:26:45,583 Migrating storage groupedmessages
2020-05-20 06:26:45,588 Creating tables for storage groupassignees
2020-05-20 06:26:45,592 Migrating storage groupassignees
2020-05-20 06:26:45,596 Creating tables for storage outcomes_raw
2020-05-20 06:26:45,600 Migrating storage outcomes_raw
2020-05-20 06:26:45,604 Creating tables for storage outcomes_hourly
2020-05-20 06:26:45,616 Migrating storage outcomes_hourly
2020-05-20 06:26:45,625 Creating tables for storage sessions_raw
2020-05-20 06:26:45,634 Migrating storage sessions_raw
2020-05-20 06:26:45,655 Creating tables for storage sessions_hourly
2020-05-20 06:26:45,680 Migrating storage sessions_hourly
2020-05-20 06:26:45,705 Creating tables for storage transactions
2020-05-20 06:26:45,723 Migrating storage transactions
echo ""
# Very naively check whether there's an existing sentry-postgres volume and the PG version in it
if [[ $(docker volume ls -q --filter name=sentry-postgres) && $(docker run --rm -v sentry-postgres:/db:Z busybox cat /db/PG_VERSION 2>/dev/null) == "9.5" ]]; then
docker volume rm sentry-postgres-new || true
# If this is Postgres 9.5 data, start upgrading it to 9.6 in a new volume
docker run --rm \
-v sentry-postgres:/var/lib/postgresql/9.5/data:Z \
-v sentry-postgres-new:/var/lib/postgresql/9.6/data:Z \
tianon/postgres-upgrade:9.5-to-9.6
# Get rid of the old volume as we'll rename the new one to that
docker volume rm sentry-postgres
docker volume create --name sentry-postgres
# There's no rename volume in Docker so copy the contents from old to new name
# Also append the `host all all all trust` line as `tianon/postgres-upgrade:9.5-to-9.6`
# doesn't do that automatically.
docker run --rm -v sentry-postgres-new:/from:Z -v sentry-postgres:/to:Z alpine ash -c \
"cd /from ; cp -av . /to ; echo 'host all all all trust' >> /to/pg_hba.conf"
# Finally, remove the new old volume as we are all in sentry-postgres now
docker volume rm sentry-postgres-new
fi
docker volume ls -q --filter name=sentry-postgres
docker run --rm -v sentry-postgres:/db:Z busybox cat /db/PG_VERSION 2>/dev/null
echo ""
echo "Setting up database..."
Setting up database...
if [ $CI ]; then
$dcr web upgrade --noinput
echo ""
echo "Did not prompt for user creation due to non-interactive shell."
echo "Run the following command to create one yourself (recommended):"
echo ""
echo " docker-compose run --rm web createuser"
echo ""
else
$dcr web upgrade
fi
Starting sentry_onpremise_zookeeper_1 ...
Starting sentry_onpremise_zookeeper_1 ... done
Creating sentry_onpremise_memcached_1 ...
Starting sentry_onpremise_redis_1 ...
Starting sentry_onpremise_redis_1 ... done
Creating sentry_onpremise_smtp_1 ...
Starting sentry_onpremise_clickhouse_1 ...
Starting sentry_onpremise_clickhouse_1 ... done
Starting sentry_onpremise_kafka_1 ...
Creating sentry_onpremise_symbolicator_1 ...
Starting sentry_onpremise_kafka_1 ... done
Creating sentry_onpremise_postgres_1 ...
Creating sentry_onpremise_snuba-api_1 ...
Creating sentry_onpremise_snuba-replacer_1 ...
Creating sentry_onpremise_snuba-outcomes-consumer_1 ...
Creating sentry_onpremise_snuba-consumer_1 ...
Creating sentry_onpremise_memcached_1 ... done
Creating sentry_onpremise_smtp_1 ... done
Creating sentry_onpremise_snuba-consumer_1 ... done
Creating sentry_onpremise_postgres_1 ... done
Creating sentry_onpremise_symbolicator_1 ... done
Creating sentry_onpremise_snuba-replacer_1 ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1 ... done
Creating sentry_onpremise_snuba-api_1 ... done
06:26:58 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
06:27:00 [INFO] sentry.plugins.github: apps-not-configured
Operations to perform:
Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying sentry.0001_initial... OK
Applying contenttypes.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying jira_ac.0001_initial... OK
Applying nodestore.0001_initial... OK
Applying sentry.0002_912_to_recent... OK
Applying sentry.0003_auto_20191022_0122... OK
Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau... OK
Applying sentry.0005_fix_content_types... OK
Applying sentry.0006_sentryapp_date_published... OK
Applying sentry.0007_auto_20191029_0131... OK
Applying sentry.0008_auto_20191030_0016... OK
Applying sentry.0009_auto_20191101_1608... OK
Applying sentry.0010_auto_20191104_1641... OK
Applying sentry.0011_remove_pagerdutyservice_service_id_from_state... OK
Applying sentry.0012_remove_pagerdutyservice_service_id... OK
Applying sentry.0013_auto_20191111_1829... OK
Applying sentry.0014_delete_sentryappwebhookerror... OK
Applying sentry.0015_delete_sentryappwebhookerror_db... OK
Applying sentry.0016_delete_alert_rule_deprecated_fields... OK
Applying sentry.0017_incident_aggregation... OK
Applying sentry.0018_discoversavedquery_version... OK
Applying sentry.0019_auto_20191114_2040... OK
Applying sentry.0020_auto_20191125_1420... OK
Applying sentry.0021_auto_20191203_1803... OK
Applying sentry.0021_auto_20191202_1716... OK
Applying sentry.0022_merge... OK
Applying sentry.0023_hide_environment_none_20191126... OK
Applying sentry.0024_auto_20191230_2052...Nothing to do, skipping migration.
OK
Applying sentry.0025_organizationaccessrequest_requester... OK
Applying sentry.0026_delete_event... OK
Applying sentry.0027_exporteddata... OK
Applying sentry.0028_user_reports... OK
Applying sentry.0029_discover_query_upgrade... OK
Applying sentry.0030_auto_20200201_0039... OK
Applying sentry.0031_delete_alert_rules_and_incidents... OK
Applying sentry.0032_delete_alert_email... OK
Applying sentry.0033_auto_20200210_2137... OK
Applying sentry.0034_auto_20200210_2311... OK
Applying sentry.0035_auto_20200127_1711... OK
Applying sentry.0036_auto_20200213_0106... OK
Applying sentry.0037_auto_20200213_0140... OK
Applying sentry.0038_auto_20200213_1904... OK
Applying sentry.0039_delete_incidentsuspectcommit... OK
Applying sentry.0040_remove_incidentsuspectcommittable... OK
Applying sentry.0041_incidenttrigger_date_modified... OK
Applying sentry.0042_auto_20200214_1607... OK
Applying sentry.0043_auto_20200218_1903... OK
Applying sentry.0044_auto_20200219_0018... OK
Applying sentry.0045_remove_incidentactivity_event_stats_snapshot... OK
Applying sentry.0046_auto_20200221_1735... OK
Applying sentry.0047_auto_20200224_2319... OK
Applying sentry.0048_auto_20200302_1825... OK
Applying sentry.0049_auto_20200304_0254... OK
Applying sentry.0050_auto_20200306_2346... OK
Applying sentry.0051_fix_auditlog_pickled_data... OK
Applying sentry.0052_organizationonboardingtask_completion_seen... OK
Applying sentry.0053_migrate_alert_task_onboarding... OK
Applying sentry.0054_create_key_transaction... OK
Applying sentry.0055_query_subscription_status... OK
Applying sentry.0056_remove_old_functions... OK
Applying sentry.0057_remove_unused_project_flag... OK
Applying sentry.0058_project_issue_alerts_targeting... OK
Applying sentry.0059_add_new_sentry_app_features... OK
Applying sentry.0060_add_file_eventattachment_index... OK
Applying sentry.0061_alertrule_partial_index... OK
Applying sentry.0062_key_transactions_unique_with_owner... OK
Applying sentry.0063_drop_alertrule_constraint... OK
Applying sentry.0064_project_has_transactions... OK
Applying sentry.0065_add_incident_status_method... OK
Applying sentry.0066_alertrule_manager... OK
Applying sentry.0067_migrate_rules_alert_targeting... OK
Applying sentry.0068_project_default_flags... OK
Applying sentry.0069_remove_tracked_superusers... OK
Applying sentry.0070_incident_snapshot_support... OK
Applying sentry.0071_add_default_fields_model_subclass... OK
Applying sentry.0072_alert_rules_query_changes... OK
Applying sentry.0073_migrate_alert_query_model... OK
Applying sentry.0074_add_metric_alert_feature... OK
Applying sentry.0075_metric_alerts_fix_releases... OK
Applying sentry.0076_alert_rules_disable_constraints... OK
Applying sentry.0077_alert_query_col_drop_state... OK
Applying sessions.0001_initial... OK
Applying sites.0001_initial... OK
Applying sites.0002_alter_domain_unique... OK
Applying social_auth.0001_initial... OK
06:27:46 [WARNING] sentry: Cannot initiate onboarding for organization (1) due to missing owners
Created internal Sentry project (slug=internal, id=1)
Would you like to create a user account now? [Y/n]: y
Email: [email protected]
Password:
Repeat for confirmation:
User created: [email protected]
Added to organization: sentry
Creating missing DSNs
Correcting Group.num_comments counter
SENTRY_DATA_NEEDS_MIGRATION=$(docker run --rm -v sentry-data:/data:Z alpine ash -c "[ ! -d '/data/files' ] && ls -A1x /data | wc -l || true")
docker run --rm -v sentry-data:/data:Z alpine ash -c "[ ! -d '/data/files' ] && ls -A1x /data | wc -l || true"
if [ "$SENTRY_DATA_NEEDS_MIGRATION" ]; then
echo "Migrating file storage..."
# Use the web (Sentry) image so the file owners are kept as sentry:sentry
# The `\"` escape pattern is to make this compatible w/ Git Bash on Windows. See #329.
$dcr --entrypoint \"/bin/bash\" web -c \
"mkdir -p /tmp/files; mv /data/* /tmp/files/; mv /tmp/files /data/files; chown -R sentry:sentry /data"
fi
if [ ! -f "$RELAY_CREDENTIALS_JSON" ]; then
echo ""
echo "Generating Relay credentials..."
# We need the ugly hack below as `relay generate credentials` tries to read the config and the credentials
# even with the `--stdout` and `--overwrite` flags and then errors out when the credentials file exists but
# not valid JSON. We hit this case as we redirect output to the same config folder, creating an empty
# credentials file before relay runs.
$dcr --no-deps -v $(pwd)/$RELAY_CONFIG_YML:/tmp/config.yml relay --config /tmp credentials generate --stdout > "$RELAY_CREDENTIALS_JSON"
echo "Relay credentials written to $RELAY_CREDENTIALS_JSON"
fi
Generating Relay credentials...
pwd
Relay credentials written to relay/credentials.json
RELAY_CREDENTIALS=$(sed -n 's/^.*"public_key"[[:space:]]*:[[:space:]]*"\([a-zA-Z0-9_-]\{1,\}\)".*$/\1/p' "$RELAY_CREDENTIALS_JSON")
sed -n 's/^.*"public_key"[[:space:]]*:[[:space:]]*"\([a-zA-Z0-9_-]\{1,\}\)".*$/\1/p' "$RELAY_CREDENTIALS_JSON"
if [ -z "$RELAY_CREDENTIALS" ]; then
>&2 echo "FAIL: Cannot read credentials back from $RELAY_CREDENTIALS_JSON."
>&2 echo " Please ensure this file is readable and contains valid credentials."
>&2 echo ""
exit 1
fi
if ! grep -q "\"$RELAY_CREDENTIALS\"" "$SENTRY_CONFIG_PY"; then
echo "SENTRY_RELAY_WHITELIST_PK = (SENTRY_RELAY_WHITELIST_PK or []) + ([\"$RELAY_CREDENTIALS\"])" >> "$SENTRY_CONFIG_PY"
echo "Relay public key written to $SENTRY_CONFIG_PY"
echo ""
fi
Relay public key written to sentry/sentry.conf.py
cleanup
Cleaning up...
echo ""
echo "----------------"
----------------
echo "You're all done! Run the following command to get Sentry running:"
You're all done! Run the following command to get Sentry running:
echo ""
echo " docker-compose up -d"
docker-compose up -d
echo ""
and here the docker-compose-log after starting:
Attaching to sentry_onpremise_nginx_1, sentry_onpremise_post-process-forwarder_1, sentry_onpremise_sentry-cleanup_1, sentry_onpremise_worker_1, sentry_onpremise_web_1, sentry_onpremise_ingest-consumer_1, sentry_onpremise_cron_1, sentry_onpremise_snuba-cleanup_1, sentry_onpremise_relay_1, sentry_onpremise_symbolicator-cleanup_1, sentry_onpremise_snuba-consumer_1, sentry_onpremise_snuba-api_1, sentry_onpremise_snuba-outcomes-consumer_1, sentry_onpremise_snuba-replacer_1, sentry_onpremise_symbolicator_1, sentry_onpremise_postgres_1, sentry_onpremise_smtp_1, sentry_onpremise_memcached_1, sentry_onpremise_kafka_1, sentry_onpremise_zookeeper_1, sentry_onpremise_redis_1, sentry_onpremise_clickhouse_1
post-process-forwarder_1 | 06:28:42 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
post-process-forwarder_1 | 06:28:45 [INFO] sentry.plugins.github: apps-not-configured
sentry-cleanup_1 | SHELL=/bin/bash
sentry-cleanup_1 | BASH_ENV=/container.env
sentry-cleanup_1 | 0 0 * * * gosu sentry sentry cleanup --days 90 > /proc/1/fd/1 2>/proc/1/fd/2
worker_1 | 06:28:41 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
worker_1 | 06:28:44 [INFO] sentry.plugins.github: apps-not-configured
worker_1 | 06:28:44 [INFO] sentry.bgtasks: bgtask.spawn (task_name=u'sentry.bgtasks.clean_dsymcache:clean_dsymcache')
worker_1 | 06:28:44 [INFO] sentry.bgtasks: bgtask.spawn (task_name=u'sentry.bgtasks.clean_releasefilecache:clean_releasefilecache')
worker_1 |
worker_1 | -------------- celery@1069117c1a7c v3.1.18 (Cipater)
worker_1 | ---- **** -----
worker_1 | --- * *** * -- Linux-4.15.0-1067-oem-x86_64-with-debian-10.1
worker_1 | -- * - **** ---
worker_1 | - ** ---------- [config]
worker_1 | - ** ---------- .> app: sentry:0x7f2e5f6d0650
worker_1 | - ** ---------- .> transport: redis://redis:6379/0
worker_1 | - ** ---------- .> results: disabled
worker_1 | - *** --- * --- .> concurrency: 16 (prefork)
worker_1 | -- ******* ----
worker_1 | --- ***** ----- [queues]
worker_1 | -------------- .> activity.notify exchange=default(direct) key=activity.notify
worker_1 | .> alerts exchange=default(direct) key=alerts
worker_1 | .> app_platform exchange=default(direct) key=app_platform
worker_1 | .> assemble exchange=default(direct) key=assemble
worker_1 | .> auth exchange=default(direct) key=auth
worker_1 | .> buffers.process_pending exchange=default(direct) key=buffers.process_pending
worker_1 | .> cleanup exchange=default(direct) key=cleanup
worker_1 | .> commits exchange=default(direct) key=commits
worker_1 | .> counters-0 exchange=counters(direct) key=
worker_1 | .> data_export exchange=default(direct) key=data_export
worker_1 | .> default exchange=default(direct) key=default
worker_1 | .> digests.delivery exchange=default(direct) key=digests.delivery
worker_1 | .> digests.scheduling exchange=default(direct) key=digests.scheduling
worker_1 | .> email exchange=default(direct) key=email
worker_1 | .> events.preprocess_event exchange=default(direct) key=events.preprocess_event
worker_1 | .> events.process_event exchange=default(direct) key=events.process_event
worker_1 | .> events.reprocess_events exchange=default(direct) key=events.reprocess_events
worker_1 | .> events.reprocessing.preprocess_event exchange=default(direct) key=events.reprocessing.preprocess_event
worker_1 | .> events.reprocessing.process_event exchange=default(direct) key=events.reprocessing.process_event
worker_1 | .> events.reprocessing.symbolicate_event exchange=default(direct) key=events.reprocessing.symbolicate_event
worker_1 | .> events.save_event exchange=default(direct) key=events.save_event
worker_1 | .> events.symbolicate_event exchange=default(direct) key=events.symbolicate_event
worker_1 | .> files.delete exchange=default(direct) key=files.delete
worker_1 | .> incidents exchange=default(direct) key=incidents
worker_1 | .> integrations exchange=default(direct) key=integrations
worker_1 | .> merge exchange=default(direct) key=merge
worker_1 | .> options exchange=default(direct) key=options
worker_1 | .> relay_config exchange=default(direct) key=relay_config
worker_1 | .> reports.deliver exchange=default(direct) key=reports.deliver
worker_1 | .> reports.prepare exchange=default(direct) key=reports.prepare
worker_1 | .> search exchange=default(direct) key=search
worker_1 | .> sleep exchange=default(direct) key=sleep
worker_1 | .> stats exchange=default(direct) key=stats
worker_1 | .> subscriptions exchange=default(direct) key=subscriptions
worker_1 | .> triggers-0 exchange=triggers(direct) key=
worker_1 | .> unmerge exchange=default(direct) key=unmerge
worker_1 | .> update exchange=default(direct) key=update
worker_1 |
web_1 | 06:28:40 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 06:28:43 [INFO] sentry.plugins.github: apps-not-configured
web_1 | *** Starting uWSGI 2.0.18 (64bit) on [Wed May 20 06:28:43 2020] ***
web_1 | compiled with version: 8.3.0 on 20 May 2020 01:15:35
web_1 | os: Linux-4.15.0-1067-oem #77lvm1 SMP Wed Feb 12 12:16:29 CET 2020
web_1 | nodename: bba10ac20726
web_1 | machine: x86_64
web_1 | clock source: unix
web_1 | detected number of CPU cores: 16
web_1 | current working directory: /
web_1 | detected binary path: /usr/local/bin/uwsgi
web_1 | !!! no internal routing support, rebuild with pcre support !!!
web_1 | your memory page size is 4096 bytes
web_1 | detected max file descriptor number: 1048576
web_1 | lock engine: pthread robust mutexes
web_1 | thunder lock: enabled
web_1 | uwsgi socket 0 bound to TCP address 0.0.0.0:9000 fd 3
web_1 | Python version: 2.7.16 (default, Oct 17 2019, 07:39:30) [GCC 8.3.0]
web_1 | Set PythonHome to /usr/local
web_1 | Python main interpreter initialized at 0x55b4ce8f7dc0
web_1 | python threads support enabled
web_1 | your server socket listen backlog is limited to 100 connections
web_1 | your mercy for graceful operations on workers is 60 seconds
web_1 | setting request body buffering size to 65536 bytes
web_1 | mapped 543520 bytes (530 KB) for 2 cores
web_1 | *** Operational MODE: threaded ***
web_1 | spawned uWSGI master process (pid: 18)
web_1 | spawned uWSGI worker 1 (pid: 22, cores: 2)
web_1 | 06:28:44 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 06:28:46 [INFO] sentry.plugins.github: apps-not-configured
web_1 | WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55b4ce8f7dc0 pid: 22 (default app)
web_1 | 172.21.0.15 - - [20/May/2020:06:28:46 +0000] "POST /api/0/relays/register/challenge/ HTTP/1.1" 200 663 "-" "actix-web/0.7.19"
web_1 | 172.21.0.15 - - [20/May/2020:06:28:50 +0000] "POST /api/0/relays/register/challenge/ HTTP/1.1" 200 663 "-" "actix-web/0.7.19"
web_1 | 172.21.0.15 - - [20/May/2020:06:28:55 +0000] "POST /api/0/relays/register/challenge/ HTTP/1.1" 200 663 "-" "actix-web/0.7.19"
web_1 | 172.21.0.15 - - [20/May/2020:06:29:02 +0000] "POST /api/0/relays/register/challenge/ HTTP/1.1" 200 663 "-" "actix-web/0.7.19"
web_1 | 172.21.0.15 - - [20/May/2020:06:29:02 +0000] "POST /api/0/relays/register/response/ HTTP/1.1" 200 565 "-" "actix-web/0.7.19"
snuba-cleanup_1 | SHELL=/bin/bash
snuba-cleanup_1 | BASH_ENV=/container.env
snuba-cleanup_1 | */5 * * * * gosu snuba snuba cleanup --dry-run False > /proc/1/fd/1 2>/proc/1/fd/2
snuba-cleanup_1 | 2020-05-20 06:30:02,786 Dropped 0 partitions on None
snuba-api_1 | + '[' a = - ']'
snuba-api_1 | + snuba api --help
snuba-api_1 | + set -- snuba api
snuba-api_1 | + set gosu snuba snuba api
snuba-api_1 | + exec gosu snuba snuba api
snuba-api_1 | *** Starting uWSGI 2.0.18 (64bit) on [Wed May 20 06:26:58 2020] ***
snuba-api_1 | compiled with version: 8.3.0 on 19 May 2020 19:06:59
snuba-api_1 | os: Linux-4.15.0-1067-oem #77lvm1 SMP Wed Feb 12 12:16:29 CET 2020
snuba-api_1 | nodename: 55448b885ba2
snuba-api_1 | machine: x86_64
snuba-api_1 | clock source: unix
snuba-api_1 | pcre jit disabled
snuba-api_1 | detected number of CPU cores: 16
snuba-api_1 | current working directory: /usr/src/snuba
snuba-api_1 | detected binary path: /usr/local/bin/uwsgi
snuba-api_1 | your memory page size is 4096 bytes
snuba-api_1 | detected max file descriptor number: 1048576
snuba-api_1 | lock engine: pthread robust mutexes
snuba-api_1 | thunder lock: enabled
snuba-api_1 | uwsgi socket 0 bound to TCP address 0.0.0.0:1218 fd 3
snuba-api_1 | Python version: 3.7.7 (default, May 15 2020, 11:37:57) [GCC 8.3.0]
snuba-api_1 | Set PythonHome to /usr/local
snuba-api_1 | Python main interpreter initialized at 0x562bafe437a0
snuba-api_1 | python threads support enabled
snuba-api_1 | your server socket listen backlog is limited to 100 connections
snuba-api_1 | your mercy for graceful operations on workers is 60 seconds
snuba-api_1 | mapped 145808 bytes (142 KB) for 1 cores
snuba-api_1 | *** Operational MODE: single process ***
snuba-api_1 | initialized 38 metrics
snuba-api_1 | spawned uWSGI master process (pid: 1)
snuba-api_1 | spawned uWSGI worker 1 (pid: 17, cores: 1)
snuba-api_1 | metrics collector thread started
snuba-api_1 | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x562bafe437a0 pid: 17 (default app)
snuba-api_1 | SIGINT/SIGQUIT received...killing workers...
snuba-api_1 | worker 1 buried after 1 seconds
snuba-api_1 | goodbye to uWSGI.
snuba-api_1 | + '[' a = - ']'
snuba-api_1 | + snuba api --help
snuba-api_1 | + set -- snuba api
snuba-api_1 | + set gosu snuba snuba api
snuba-api_1 | + exec gosu snuba snuba api
snuba-api_1 | *** Starting uWSGI 2.0.18 (64bit) on [Wed May 20 06:28:37 2020] ***
snuba-api_1 | compiled with version: 8.3.0 on 19 May 2020 19:06:59
snuba-api_1 | os: Linux-4.15.0-1067-oem #77lvm1 SMP Wed Feb 12 12:16:29 CET 2020
snuba-api_1 | nodename: 55448b885ba2
snuba-api_1 | machine: x86_64
snuba-api_1 | clock source: unix
snuba-api_1 | pcre jit disabled
snuba-api_1 | detected number of CPU cores: 16
snuba-api_1 | current working directory: /usr/src/snuba
snuba-api_1 | detected binary path: /usr/local/bin/uwsgi
snuba-api_1 | your memory page size is 4096 bytes
snuba-api_1 | detected max file descriptor number: 1048576
snuba-api_1 | lock engine: pthread robust mutexes
snuba-api_1 | thunder lock: enabled
snuba-api_1 | uwsgi socket 0 bound to TCP address 0.0.0.0:1218 fd 3
snuba-api_1 | Python version: 3.7.7 (default, May 15 2020, 11:37:57) [GCC 8.3.0]
snuba-api_1 | Set PythonHome to /usr/local
snuba-api_1 | Python main interpreter initialized at 0x55f002cb07a0
snuba-api_1 | python threads support enabled
snuba-api_1 | your server socket listen backlog is limited to 100 connections
snuba-api_1 | your mercy for graceful operations on workers is 60 seconds
snuba-api_1 | mapped 145808 bytes (142 KB) for 1 cores
snuba-api_1 | *** Operational MODE: single process ***
snuba-api_1 | initialized 38 metrics
snuba-api_1 | spawned uWSGI master process (pid: 1)
snuba-api_1 | spawned uWSGI worker 1 (pid: 17, cores: 1)
snuba-api_1 | metrics collector thread started
snuba-api_1 | WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x55f002cb07a0 pid: 17 (default app)
cron_1 | 06:28:38 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
cron_1 | 06:28:41 [INFO] sentry.plugins.github: apps-not-configured
relay_1 | 2020-05-20T06:28:37Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Timeout while waiting for response
relay_1 | 2020-05-20T06:28:38Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Timeout while waiting for response
relay_1 | 2020-05-20T06:28:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Failed to connect to host: No route to host (os error 113)
relay_1 | caused by: No route to host (os error 113)
relay_1 | caused by: No route to host (os error 113)
relay_1 | 2020-05-20T06:28:41Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | 2020-05-20T06:28:46Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-20T06:28:50Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-20T06:28:55Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
ingest-consumer_1 | 06:28:39 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
ingest-consumer_1 | 06:28:42 [INFO] sentry.plugins.github: apps-not-configured
ingest-consumer_1 | 06:28:47 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]
snuba-consumer_1 | + '[' c = - ']'
snuba-consumer_1 | + snuba consumer --help
snuba-consumer_1 | + set -- snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | + set gosu snuba snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | + exec gosu snuba snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | 2020-05-20 06:27:00,606 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 0}
snuba-consumer_1 | 2020-05-20 06:27:03,607 Partitions revoked: [Partition(topic=Topic(name='events'), index=0)]
snuba-consumer_1 | 2020-05-20 06:27:03,678 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 0}
snuba-consumer_1 | 2020-05-20 06:28:09,534 Partitions revoked: [Partition(topic=Topic(name='events'), index=0)]
snuba-consumer_1 | + '[' c = - ']'
snuba-consumer_1 | + snuba consumer --help
snuba-consumer_1 | + set -- snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | + set gosu snuba snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | + exec gosu snuba snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | 2020-05-20 06:28:42,042 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 0}
snuba-outcomes-consumer_1 | + '[' c = - ']'
snuba-outcomes-consumer_1 | + snuba consumer --help
snuba-outcomes-consumer_1 | + set -- snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | + set gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | + exec gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | 2020-05-20 06:27:03,619 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 0}
snuba-outcomes-consumer_1 | 2020-05-20 06:28:09,626 Partitions revoked: [Partition(topic=Topic(name='outcomes'), index=0)]
snuba-outcomes-consumer_1 | 2020-05-20 06:28:09,690 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 0}
snuba-outcomes-consumer_1 | 2020-05-20 06:28:09,750 Partitions revoked: [Partition(topic=Topic(name='outcomes'), index=0)]
snuba-outcomes-consumer_1 | + '[' c = - ']'
snuba-outcomes-consumer_1 | + snuba consumer --help
snuba-outcomes-consumer_1 | + set -- snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | + set gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | + exec gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | 2020-05-20 06:28:42,042 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 0}
snuba-replacer_1 | + '[' r = - ']'
snuba-replacer_1 | + snuba replacer --help
snuba-replacer_1 | + set -- snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + set gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + exec gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | 2020-05-20 06:27:02,574 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}
snuba-replacer_1 | 2020-05-20 06:28:09,575 Partitions revoked: [Partition(topic=Topic(name='event-replacements'), index=0)]
snuba-replacer_1 | + '[' r = - ']'
snuba-replacer_1 | + snuba replacer --help
snuba-replacer_1 | + set -- snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + set gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + exec gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | 2020-05-20 06:28:41,358 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}
symbolicator-cleanup_1 | SHELL=/bin/bash
symbolicator-cleanup_1 | BASH_ENV=/container.env
symbolicator-cleanup_1 | 55 23 * * * gosu symbolicator symbolicator cleanup > /proc/1/fd/1 2>/proc/1/fd/2
memcached_1 | Signal handled: Terminated.
smtp_1 | + sed -ri '
smtp_1 | s/^#?(dc_local_interfaces)=.*/\1='\''0.0.0.0 ; ::0'\''/;
smtp_1 | s/^#?(dc_other_hostnames)=.*/\1='\'''\''/;
smtp_1 | s/^#?(dc_relay_nets)=.*/\1='\''0.0.0.0\/0'\''/;
smtp_1 | s/^#?(dc_eximconfig_configtype)=.*/\1='\''internet'\''/;
smtp_1 | ' /etc/exim4/update-exim4.conf.conf
smtp_1 | + update-exim4.conf -v
smtp_1 | using non-split configuration scheme from /etc/exim4/exim4.conf.template
smtp_1 | undocumented line IGNORE_SMTP_LINE_LENGTH_LIMIT='true' found in
smtp_1 | /etc/exim4/update-exim4.conf.conf, generating exim macro
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 IPv6 socket creation failed: Address family not supported by protocol
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 Failed to create IPv6 socket for wildcard listening (No such file or directory): will use IPv4
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 exim 4.92 daemon started: pid=287, no queue runs, listening for SMTP on port 25 (IPv4)
smtp_1 | + sed -ri '
smtp_1 | s/^#?(dc_local_interfaces)=.*/\1='\''0.0.0.0 ; ::0'\''/;
smtp_1 | s/^#?(dc_other_hostnames)=.*/\1='\'''\''/;
smtp_1 | s/^#?(dc_relay_nets)=.*/\1='\''0.0.0.0\/0'\''/;
smtp_1 | s/^#?(dc_eximconfig_configtype)=.*/\1='\''internet'\''/;
smtp_1 | ' /etc/exim4/update-exim4.conf.conf
smtp_1 | + update-exim4.conf -v
smtp_1 | using non-split configuration scheme from /etc/exim4/exim4.conf.template
smtp_1 | undocumented line IGNORE_SMTP_LINE_LENGTH_LIMIT='true' found in
smtp_1 | /etc/exim4/update-exim4.conf.conf, generating exim macro
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 IPv6 socket creation failed: Address family not supported by protocol
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 Failed to create IPv6 socket for wildcard listening (Address family not supported by protocol): will use IPv4
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 exim 4.92 daemon started: pid=287, no queue runs, listening for SMTP on port 25 (IPv4)
symbolicator_1 | 2020-05-20T06:26:53Z [symbolicator::server] INFO: Starting http server: 0.0.0.0:3021
symbolicator_1 | 2020-05-20T06:26:53Z [actix_net::server::server] INFO: Starting 16 workers
symbolicator_1 | 2020-05-20T06:26:53Z [actix_net::server::server] INFO: Starting server on 0.0.0.0:3021
symbolicator_1 | 2020-05-20T06:26:53Z [symbolicator::server] INFO: Starting system
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::server] INFO: SIGTERM received, stopping
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-20T06:28:09Z [symbolicator::server] INFO: System shutdown complete
symbolicator_1 | 2020-05-20T06:28:33Z [symbolicator::server] INFO: Starting http server: 0.0.0.0:3021
symbolicator_1 | 2020-05-20T06:28:33Z [actix_net::server::server] INFO: Starting 16 workers
symbolicator_1 | 2020-05-20T06:28:33Z [actix_net::server::server] INFO: Starting server on 0.0.0.0:3021
symbolicator_1 | 2020-05-20T06:28:33Z [symbolicator::server] INFO: Starting system
postgres_1 | ********************************************************************************
postgres_1 | WARNING: POSTGRES_HOST_AUTH_METHOD has been set to "trust". This will allow
postgres_1 | anyone with access to the Postgres port to access your database without
postgres_1 | a password, even if POSTGRES_PASSWORD is set. See PostgreSQL
postgres_1 | documentation about "trust":
postgres_1 | https://www.postgresql.org/docs/current/auth-trust.html
postgres_1 | In Docker's default configuration, this is effectively any other
postgres_1 | container on the same system.
postgres_1 |
postgres_1 | It is not recommended to use POSTGRES_HOST_AUTH_METHOD=trust. Replace
postgres_1 | it with "-e POSTGRES_PASSWORD=password" instead to set a password in
postgres_1 | "docker run".
postgres_1 | ********************************************************************************
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default timezone ... Etc/UTC
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | creating configuration files ... ok
postgres_1 | running bootstrap script ... ok
postgres_1 | performing post-bootstrap initialization ... ok
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....LOG: database system was shut down at 2020-05-20 06:26:52 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: autovacuum launcher started
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | waiting for server to shut down....LOG: received fast shutdown request
postgres_1 | LOG: aborting any active transactions
postgres_1 | LOG: autovacuum launcher shutting down
postgres_1 | LOG: shutting down
postgres_1 | LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | LOG: could not create IPv6 socket: Address family not supported by protocol
postgres_1 | LOG: database system was shut down at 2020-05-20 06:26:53 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | LOG: autovacuum launcher started
postgres_1 | ERROR: relation "sentry_option" does not exist at character 114
postgres_1 | STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."key" = 'system.url-prefix'
postgres_1 | ERROR: relation "sentry_option" does not exist at character 114
postgres_1 | STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."key" = 'github.apps-install-url'
postgres_1 | ERROR: relation "sentry_option" does not exist at character 114
postgres_1 | STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."key" = 'system.url-prefix'
postgres_1 | ERROR: relation "sentry_option" does not exist at character 114
postgres_1 | STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."key" = 'system.url-prefix'
postgres_1 | ERROR: relation "sentry_projectkey" does not exist at character 371
postgres_1 | STATEMENT: SELECT "sentry_projectkey"."id", "sentry_projectkey"."project_id", "sentry_projectkey"."label", "sentry_projectkey"."public_key", "sentry_projectkey"."secret_key", "sentry_projectkey"."roles", "sentry_projectkey"."status", "sentry_projectkey"."date_added", "sentry_projectkey"."rate_limit_count", "sentry_projectkey"."rate_limit_window", "sentry_projectkey"."data" FROM "sentry_projectkey" WHERE ("sentry_projectkey"."project_id" = 1 AND "sentry_projectkey"."status" = 0 AND "sentry_projectkey"."roles" = (("sentry_projectkey"."roles" | 1))) ORDER BY "sentry_projectkey"."id" ASC LIMIT 1
postgres_1 | ERROR: relation "south_migrationhistory" does not exist at character 15
postgres_1 | STATEMENT: SELECT 1 FROM south_migrationhistory LIMIT 1
postgres_1 | LOG: received smart shutdown request
postgres_1 | LOG: autovacuum launcher shutting down
postgres_1 | LOG: shutting down
postgres_1 | LOG: database system is shut down
postgres_1 |
postgres_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1 |
postgres_1 | LOG: could not create IPv6 socket: Address family not supported by protocol
postgres_1 | LOG: database system was shut down at 2020-05-20 06:28:09 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | LOG: autovacuum launcher started
kafka_1 | ===> ENV Variables ...
kafka_1 | ALLOW_UNSIGNED=false
kafka_1 | COMPONENT=kafka
kafka_1 | CONFLUENT_DEB_VERSION=1
kafka_1 | CONFLUENT_PLATFORM_LABEL=
kafka_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
kafka_1 | CONFLUENT_VERSION=5.5.0
kafka_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
kafka_1 | HOME=/root
kafka_1 | HOSTNAME=1bc54168c0dc
kafka_1 | KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
kafka_1 | KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
kafka_1 | KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
kafka_1 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
kafka_1 | KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
kafka_1 | KAFKA_VERSION=
kafka_1 | KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
kafka_1 | LANG=C.UTF-8
kafka_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kafka_1 | PWD=/
kafka_1 | PYTHON_PIP_VERSION=8.1.2
kafka_1 | PYTHON_VERSION=2.7.9-1
kafka_1 | SCALA_VERSION=2.12
kafka_1 | SHLVL=1
kafka_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
kafka_1 | _=/usr/bin/env
kafka_1 | ===> User
kafka_1 | uid=0(root) gid=0(root) groups=0(root)
kafka_1 | ===> Configuring ...
kafka_1 | ===> Running preflight checks ...
kafka_1 | ===> Check if /var/lib/kafka/data is writable ...
kafka_1 | ===> Check if Zookeeper is healthy ...
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=1bc54168c0dc
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1067-oem
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=469MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=7077MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=477MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1 | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.21.0.4:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.21.0.5:37868, server: zookeeper/172.21.0.4:2181
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.21.0.4:2181, sessionid = 0x100007974b40000, negotiated timeout = 40000
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x100007974b40000 closed
kafka_1 | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100007974b40000
kafka_1 | ===> Launching ...
kafka_1 | ===> Launching kafka ...
kafka_1 | [2020-05-20 06:26:43,864] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka_1 | [2020-05-20 06:26:44,246] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
kafka_1 | [2020-05-20 06:26:44,247] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
kafka_1 | [2020-05-20 06:26:44,546] WARN No meta.properties file under dir /var/lib/kafka/data/meta.properties (kafka.server.BrokerMetadataCheckpoint)
kafka_1 | [2020-05-20 06:26:44,662] INFO Starting the log cleaner (kafka.log.LogCleaner)
kafka_1 | [2020-05-20 06:26:44,705] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
kafka_1 | [2020-05-20 06:26:44,971] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
kafka_1 | [2020-05-20 06:26:44,995] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:26:44,996] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:26:45,035] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-20 06:26:45,051] INFO Stat of the created znode at /brokers/ids/1001 is: 27,27,1589956005043,1589956005043,1,0,0,72058115686924289,180,0,27
kafka_1 | (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-20 06:26:45,051] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 27 (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-20 06:26:45,104] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-20 06:26:45,178] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-20 06:26:45,187] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:26:45,289] INFO Creating topic events with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:26:45,313] INFO Creating topic ingest-sessions with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:26:45,326] INFO Creating topic outcomes with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:26:45,352] INFO Creating topic snuba-commit-log with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:26:45,364] INFO Creating topic errors-replacements with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:26:45,375] INFO Creating topic cdc with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:26:45,383] INFO Creating topic event-replacements with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:26:54,537] INFO Creating topic __consumer_offsets with configuration {segment.bytes=104857600, compression.type=producer, cleanup.policy=compact} and initial partition assignment Map(23 -> ArrayBuffer(1001), 32 -> ArrayBuffer(1001), 41 -> ArrayBuffer(1001), 17 -> ArrayBuffer(1001), 8 -> ArrayBuffer(1001), 35 -> ArrayBuffer(1001), 44 -> ArrayBuffer(1001), 26 -> ArrayBuffer(1001), 11 -> ArrayBuffer(1001), 29 -> ArrayBuffer(1001), 38 -> ArrayBuffer(1001), 47 -> ArrayBuffer(1001), 20 -> ArrayBuffer(1001), 2 -> ArrayBuffer(1001), 5 -> ArrayBuffer(1001), 14 -> ArrayBuffer(1001), 46 -> ArrayBuffer(1001), 49 -> ArrayBuffer(1001), 40 -> ArrayBuffer(1001), 13 -> ArrayBuffer(1001), 4 -> ArrayBuffer(1001), 22 -> ArrayBuffer(1001), 31 -> ArrayBuffer(1001), 16 -> ArrayBuffer(1001), 7 -> ArrayBuffer(1001), 43 -> ArrayBuffer(1001), 25 -> ArrayBuffer(1001), 34 -> ArrayBuffer(1001), 10 -> ArrayBuffer(1001), 37 -> ArrayBuffer(1001), 1 -> ArrayBuffer(1001), 19 -> ArrayBuffer(1001), 28 -> ArrayBuffer(1001), 45 -> ArrayBuffer(1001), 27 -> ArrayBuffer(1001), 36 -> ArrayBuffer(1001), 18 -> ArrayBuffer(1001), 9 -> ArrayBuffer(1001), 21 -> ArrayBuffer(1001), 48 -> ArrayBuffer(1001), 3 -> ArrayBuffer(1001), 12 -> ArrayBuffer(1001), 30 -> ArrayBuffer(1001), 39 -> ArrayBuffer(1001), 15 -> ArrayBuffer(1001), 42 -> ArrayBuffer(1001), 24 -> ArrayBuffer(1001), 6 -> ArrayBuffer(1001), 33 -> ArrayBuffer(1001), 0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:28:11,257] INFO [/config/changes-event-process-thread]: Shutting down (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-20 06:28:11,257] INFO [/config/changes-event-process-thread]: Shutdown completed (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-20 06:28:11,257] INFO [/config/changes-event-process-thread]: Stopped (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-20 06:28:11,258] INFO [SocketServer brokerId=1001] Stopping socket server request processors (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:28:11,263] INFO [SocketServer brokerId=1001] Stopped socket server request processors (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:28:12,271] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)
kafka_1 | [2020-05-20 06:28:12,272] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)
kafka_1 | [2020-05-20 06:28:12,273] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)
kafka_1 | [2020-05-20 06:28:12,273] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)
kafka_1 | [2020-05-20 06:28:13,630] INFO [SocketServer brokerId=1001] Shutting down socket server (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:28:13,644] INFO [SocketServer brokerId=1001] Shutdown completed (kafka.network.SocketServer)
kafka_1 | ===> ENV Variables ...
kafka_1 | ALLOW_UNSIGNED=false
kafka_1 | COMPONENT=kafka
kafka_1 | CONFLUENT_DEB_VERSION=1
kafka_1 | CONFLUENT_PLATFORM_LABEL=
kafka_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
kafka_1 | CONFLUENT_VERSION=5.5.0
kafka_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
kafka_1 | HOME=/root
kafka_1 | HOSTNAME=1bc54168c0dc
kafka_1 | KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
kafka_1 | KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
kafka_1 | KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
kafka_1 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
kafka_1 | KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
kafka_1 | KAFKA_VERSION=
kafka_1 | KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
kafka_1 | LANG=C.UTF-8
kafka_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kafka_1 | PWD=/
kafka_1 | PYTHON_PIP_VERSION=8.1.2
kafka_1 | PYTHON_VERSION=2.7.9-1
kafka_1 | SCALA_VERSION=2.12
kafka_1 | SHLVL=1
kafka_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
kafka_1 | _=/usr/bin/env
kafka_1 | ===> User
kafka_1 | uid=0(root) gid=0(root) groups=0(root)
kafka_1 | ===> Configuring ...
kafka_1 | ===> Running preflight checks ...
kafka_1 | ===> Check if /var/lib/kafka/data is writable ...
kafka_1 | ===> Check if Zookeeper is healthy ...
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=1bc54168c0dc
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1067-oem
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=467MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=7077MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=477MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1 | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.21.0.4:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.21.0.10:48864, server: zookeeper/172.21.0.4:2181
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.21.0.4:2181, sessionid = 0x100007b22640000, negotiated timeout = 40000
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x100007b22640000 closed
kafka_1 | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100007b22640000
kafka_1 | ===> Launching ...
kafka_1 | ===> Launching kafka ...
kafka_1 | [2020-05-20 06:28:32,862] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka_1 | [2020-05-20 06:28:33,314] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
kafka_1 | [2020-05-20 06:28:33,315] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
kafka_1 | [2020-05-20 06:28:34,062] INFO Starting the log cleaner (kafka.log.LogCleaner)
kafka_1 | [2020-05-20 06:28:34,101] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
kafka_1 | [2020-05-20 06:28:34,296] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
kafka_1 | [2020-05-20 06:28:34,322] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:28:34,323] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:28:34,391] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-20 06:28:34,414] INFO Stat of the created znode at /brokers/ids/1001 is: 195,195,1589956114406,1589956114406,1,0,0,72058122895884289,180,0,195
kafka_1 | (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-20 06:28:34,415] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 195 (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-20 06:28:34,625] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-20 06:28:34,658] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
kafka_1 | [2020-05-20 06:28:42,292] INFO Creating topic ingest-transactions with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:28:42,308] INFO Creating topic ingest-events with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-20 06:28:42,318] INFO Creating topic ingest-attachments with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
redis_1 | 1:C 20 May 2020 06:26:38.690 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 20 May 2020 06:26:38.690 # Redis version=5.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 20 May 2020 06:26:38.690 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 20 May 2020 06:26:38.694 # Not listening to IPv6: unsupproted
redis_1 | 1:M 20 May 2020 06:26:38.695 * Running mode=standalone, port=6379.
redis_1 | 1:M 20 May 2020 06:26:38.695 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 20 May 2020 06:26:38.695 # Server initialized
redis_1 | 1:M 20 May 2020 06:26:38.695 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 20 May 2020 06:26:38.695 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 20 May 2020 06:26:38.695 * Ready to accept connections
redis_1 | 1:signal-handler (1589956091) Received SIGTERM scheduling shutdown...
redis_1 | 1:M 20 May 2020 06:28:11.291 # User requested shutdown...
redis_1 | 1:M 20 May 2020 06:28:11.291 * Saving the final RDB snapshot before exiting.
redis_1 | 1:M 20 May 2020 06:28:11.297 * DB saved on disk
redis_1 | 1:M 20 May 2020 06:28:11.297 # Redis is now ready to exit, bye bye...
redis_1 | 1:C 20 May 2020 06:28:31.114 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 20 May 2020 06:28:31.114 # Redis version=5.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 20 May 2020 06:28:31.114 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 20 May 2020 06:28:31.118 # Not listening to IPv6: unsupproted
redis_1 | 1:M 20 May 2020 06:28:31.118 * Running mode=standalone, port=6379.
redis_1 | 1:M 20 May 2020 06:28:31.118 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 20 May 2020 06:28:31.118 # Server initialized
redis_1 | 1:M 20 May 2020 06:28:31.118 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 20 May 2020 06:28:31.119 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 20 May 2020 06:28:31.119 * DB loaded from disk: 0.000 seconds
redis_1 | 1:M 20 May 2020 06:28:31.119 * Ready to accept connections
zookeeper_1 | ===> ENV Variables ...
zookeeper_1 | ALLOW_UNSIGNED=false
zookeeper_1 | COMPONENT=zookeeper
zookeeper_1 | CONFLUENT_DEB_VERSION=1
zookeeper_1 | CONFLUENT_PLATFORM_LABEL=
zookeeper_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1 | CONFLUENT_VERSION=5.5.0
zookeeper_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1 | HOME=/root
zookeeper_1 | HOSTNAME=34b5c8370ddb
zookeeper_1 | KAFKA_VERSION=
zookeeper_1 | LANG=C.UTF-8
zookeeper_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1 | PWD=/
zookeeper_1 | PYTHON_PIP_VERSION=8.1.2
zookeeper_1 | PYTHON_VERSION=2.7.9-1
zookeeper_1 | SCALA_VERSION=2.12
zookeeper_1 | SHLVL=1
zookeeper_1 | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1 | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1 | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1 | _=/usr/bin/env
zookeeper_1 | ===> User
zookeeper_1 | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1 | ===> Configuring ...
zookeeper_1 | ===> Running preflight checks ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1 | ===> Launching ...
zookeeper_1 | ===> Launching zookeeper ...
zookeeper_1 | [2020-05-20 06:26:40,629] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1 | [2020-05-20 06:26:40,739] WARN o.e.j.s.ServletContextHandler@167fdd33{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1 | [2020-05-20 06:26:40,739] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1 | ===> ENV Variables ...
zookeeper_1 | ALLOW_UNSIGNED=false
zookeeper_1 | COMPONENT=zookeeper
zookeeper_1 | CONFLUENT_DEB_VERSION=1
zookeeper_1 | CONFLUENT_PLATFORM_LABEL=
zookeeper_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1 | CONFLUENT_VERSION=5.5.0
zookeeper_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1 | HOME=/root
zookeeper_1 | HOSTNAME=34b5c8370ddb
zookeeper_1 | KAFKA_VERSION=
zookeeper_1 | LANG=C.UTF-8
zookeeper_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1 | PWD=/
zookeeper_1 | PYTHON_PIP_VERSION=8.1.2
zookeeper_1 | PYTHON_VERSION=2.7.9-1
zookeeper_1 | SCALA_VERSION=2.12
zookeeper_1 | SHLVL=1
zookeeper_1 | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1 | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1 | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1 | _=/usr/bin/env
zookeeper_1 | ===> User
zookeeper_1 | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1 | ===> Configuring ...
zookeeper_1 | ===> Running preflight checks ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1 | ===> Launching ...
zookeeper_1 | ===> Launching zookeeper ...
zookeeper_1 | [2020-05-20 06:28:30,623] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1 | [2020-05-20 06:28:30,726] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1 | [2020-05-20 06:28:30,726] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
clickhouse_1 | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse_1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse_1 | Include not found: networks
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
clickhouse_1 | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse_1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse_1 | Include not found: networks
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
@marbon87 #499 should resolve the relay issue. Can you try with that and see if it also resolves the stats issue?
BYK
on 20 May 2020
499 does not solve the relay issue for me.
marbon87
on 20 May 2020
@marbon87
499 does not solve the relay issue for me.
Sorry, I should have been more specific: you need to copy the setting changes in the sentry.conf.example.py into your sentry.conf.py manually and then restart the services (docker-compose restart) to see this in effect.
BYK
on 20 May 2020
Thanks for the hint, i should have seen that.
But unfortunately it does not help either, i get the same errros.
I merged the branch byk/ref/uwsgi and copied the changes from sentry.conf.example.py. My sentry.conf.py is the same as sentry.conf.example.py now.
I also created a completely new environment with install.sh (deleted containers, volumes, etc. before) but same error.
marbon87
on 20 May 2020
@marbon87 - if you are still seeing
web_1 | *** Operational MODE: threaded ***
web_1 | spawned uWSGI master process (pid: 18)
web_1 | spawned uWSGI worker 1 (pid: 22, cores: 2)
That implies that your config changes are not applied for some reason as in the patch, I increased the workers to 3. Also, it is normal to see _some_ errors at the beginning from relay as relay starts way faster than any Sentry service so until Sentry gets up and running, relay will keep retrying.
BYK
on 22 May 2020
The new config is configured, as the following output shows:
禄 docker-compose exec web cat /etc/sentry/sentry.conf.py
# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.
from sentry.conf.server import * # NOQA
DATABASES = {
"default": {
"ENGINE": "sentry.db.postgres",
"NAME": "postgres",
"USER": "postgres",
"PASSWORD": "",
"HOST": "postgres",
"PORT": "",
}
}
# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True
# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings
###########
# General #
###########
# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True
SENTRY_OPTIONS["system.event-retention-days"] = int(
env('SENTRY_EVENT_RETENTION_DAYS', '90')
)
#########
# Redis #
#########
# Generic Redis configuration used as defaults for various things including:
# Buffers, Quotas, TSDB
SENTRY_OPTIONS["redis.clusters"] = {
"default": {
"hosts": {0: {"host": "redis", "password": "", "port": "6379", "db": "0"}}
}
}
#########
# Queue #
#########
# See https://docs.getsentry.com/on-premise/server/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.
rabbitmq_host = None
if rabbitmq_host:
BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
username="guest", password="guest", host=rabbitmq_host, vhost="/"
)
else:
BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
**SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
)
#########
# Cache #
#########
# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
"LOCATION": ["memcached:11211"],
"TIMEOUT": 3600,
}
}
# A primary cache is required for things such as processing events
SENTRY_CACHE = "sentry.cache.redis.RedisCache"
DEFAULT_KAFKA_OPTIONS = {
"bootstrap.servers": "kafka:9092",
"message.max.bytes": 50000000,
"socket.timeout.ms": 1000,
}
SENTRY_EVENTSTREAM = "sentry.eventstream.kafka.KafkaEventStream"
SENTRY_EVENTSTREAM_OPTIONS = {"producer_configuration": DEFAULT_KAFKA_OPTIONS}
KAFKA_CLUSTERS["default"] = DEFAULT_KAFKA_OPTIONS
###############
# Rate Limits #
###############
# Rate limits apply to notification handlers and are enforced per-project
# automatically.
SENTRY_RATELIMITER = "sentry.ratelimits.redis.RedisRateLimiter"
##################
# Update Buffers #
##################
# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)
SENTRY_BUFFER = "sentry.buffer.redis.RedisBuffer"
##########
# Quotas #
##########
# Quotas allow you to rate limit individual projects or the Sentry install as
# a whole.
SENTRY_QUOTAS = "sentry.quotas.redis.RedisQuota"
########
# TSDB #
########
# The TSDB is used for building charts as well as making things like per-rate
# alerts possible.
SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"
#########
# SNUBA #
#########
SENTRY_SEARCH = "sentry.search.snuba.EventsDatasetSnubaSearchBackend"
SENTRY_SEARCH_OPTIONS = {}
SENTRY_TAGSTORE_OPTIONS = {}
###########
# Digests #
###########
# The digest backend powers notification summaries.
SENTRY_DIGESTS = "sentry.digests.backends.redis.RedisBackend"
##############
# Web Server #
##############
SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
# These ase for proper HTTP/1.1 support from uWSGI
# Without these it doesn't do keep-alives causing
# issues with Relay's direct requests.
"http-keepalive": True,
"http-chunked-input": True,
# the number of web workers
'workers': 3,
# Turn off memory reporting
"memory-report": False,
# Some stuff so uwsgi will cycle workers sensibly
'max-requests': 100000,
'max-requests-delta': 500,
'max-worker-lifetime': 86400,
# Duplicate options from sentry default just so we don't get
# bit by sentry changing a default value that we depend on.
'thunder-lock': True,
'log-x-forwarded-for': False,
'buffer-size': 32768,
'limit-post': 209715200,
'disable-logging': True,
'reload-on-rss': 600,
'ignore-sigpipe': True,
'ignore-write-errors': True,
'disable-write-exception': True,
}
###########
# SSL/TLS #
###########
# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below
# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# SESSION_COOKIE_SECURE = True
# CSRF_COOKIE_SECURE = True
# SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
# End of SSL/TLS settings
############
# Features #
############
SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
{
feature: True
for feature in (
"organizations:discover",
"organizations:events",
"organizations:discover-basic",
"organizations:discover-query",
"organizations:events-v2",
"organizations:global-views",
"organizations:integrations-issue-basic",
"organizations:integrations-issue-sync",
"organizations:invite-members",
"organizations:sso-basic",
"organizations:sso-rippling",
"organizations:sso-saml2",
"projects:custom-inbound-filters",
"projects:data-forwarding",
"projects:discard-groups",
"projects:plugins",
"projects:rate-limits",
"projects:servicehooks",
)
}
)
######################
# GitHub Integration #
######################
GITHUB_EXTENDED_PERMISSIONS = ['repo']
#########################
# Bitbucket Integration #
########################
# BITBUCKET_CONSUMER_KEY = 'YOUR_BITBUCKET_CONSUMER_KEY'
# BITBUCKET_CONSUMER_SECRET = 'YOUR_BITBUCKET_CONSUMER_SECRET'
SENTRY_RELAY_WHITELIST_PK = (SENTRY_RELAY_WHITELIST_PK or []) + (["aAlN2bYW-EdPWnJfM2yqd1-lD3zOIieERQTa2pJzoxg"])
@marbon87 then can the logs you shared may be old or could these be some other overrides as with these settings you should be seeing 3 uWSGI workers whereas the logs you suggest you only have 1. That doesn't make any sense.
BYK
on 25 May 2020
I created a new environment and now 3 workers are launched, but the error is still the same.
install.sh-log
MIN_DOCKER_VERSION='17.05.0'
MIN_COMPOSE_VERSION='1.23.0'
MIN_RAM=2400 # MB
SENTRY_CONFIG_PY='sentry/sentry.conf.py'
SENTRY_CONFIG_YML='sentry/config.yml'
RELAY_CONFIG_YML='relay/config.yml'
RELAY_CREDENTIALS_JSON='relay/credentials.json'
SENTRY_EXTRA_REQUIREMENTS='sentry/requirements.txt'
DID_CLEAN_UP=0
# the cleanup function will be the exit point
cleanup () {
if [ "$DID_CLEAN_UP" -eq 1 ]; then
return 0;
fi
echo "Cleaning up..."
$dc stop &> /dev/null
DID_CLEAN_UP=1
}
trap cleanup ERR INT TERM
echo "Checking minimum requirements..."
Checking minimum requirements...
DOCKER_VERSION=$(docker version --format '{{.Server.Version}}')
docker version --format '{{.Server.Version}}'
COMPOSE_VERSION=$($dc --version | sed 's/docker-compose version \(.\{1,\}\),.*/\1/')
$dc --version | sed 's/docker-compose version \(.\{1,\}\),.*/\1/'
RAM_AVAILABLE_IN_DOCKER=$(docker run --rm busybox free -m 2>/dev/null | awk '/Mem/ {print $2}');
docker run --rm busybox free -m 2>/dev/null | awk '/Mem/ {print $2}'
# Compare dot-separated strings - function below is inspired by https://stackoverflow.com/a/37939589/808368
function ver () { echo "$@" | awk -F. '{ printf("%d%03d%03d", $1,$2,$3); }'; }
# Thanks to https://stackoverflow.com/a/25123013/90297 for the quick `sed` pattern
function ensure_file_from_example {
if [ -f "$1" ]; then
echo "$1 already exists, skipped creation."
else
echo "Creating $1..."
cp -n $(echo "$1" | sed 's/\.[^.]*$/.example&/') "$1"
fi
}
if [ $(ver $DOCKER_VERSION) -lt $(ver $MIN_DOCKER_VERSION) ]; then
echo "FAIL: Expected minimum Docker version to be $MIN_DOCKER_VERSION but found $DOCKER_VERSION"
exit 1
fi
ver $DOCKER_VERSION
ver $MIN_DOCKER_VERSION
if [ $(ver $COMPOSE_VERSION) -lt $(ver $MIN_COMPOSE_VERSION) ]; then
echo "FAIL: Expected minimum docker-compose version to be $MIN_COMPOSE_VERSION but found $COMPOSE_VERSION"
exit 1
fi
ver $COMPOSE_VERSION
ver $MIN_COMPOSE_VERSION
if [ "$RAM_AVAILABLE_IN_DOCKER" -lt "$MIN_RAM" ]; then
echo "FAIL: Expected minimum RAM available to Docker to be $MIN_RAM MB but found $RAM_AVAILABLE_IN_DOCKER MB"
exit 1
fi
#SSE4.2 required by Clickhouse (https://clickhouse.yandex/docs/en/operations/requirements/)
# On KVM, cpuinfo could falsely not report SSE 4.2 support, so skip the check. https://github.com/ClickHouse/ClickHouse/issues/20#issuecomment-226849297
IS_KVM=$(docker run --rm busybox grep -c 'Common KVM processor' /proc/cpuinfo || :)
docker run --rm busybox grep -c 'Common KVM processor' /proc/cpuinfo || :
if (($IS_KVM == 0)); then
SUPPORTS_SSE42=$(docker run --rm busybox grep -c sse4_2 /proc/cpuinfo || :)
if (($SUPPORTS_SSE42 == 0)); then
echo "FAIL: The CPU your machine is running on does not support the SSE 4.2 instruction set, which is required for one of the services Sentry uses (Clickhouse). See https://git.io/JvLDt for more info."
exit 1
fi
fi
docker run --rm busybox grep -c sse4_2 /proc/cpuinfo || :
# Clean up old stuff and ensure nothing is working while we install/update
# This is for older versions of on-premise:
$dc -p onpremise down --rmi local --remove-orphans
Removing network onpremise_default
Network onpremise_default not found.
# This is for newer versions
$dc down --rmi local --remove-orphans
Removing network sentry_onpremise_default
echo ""
echo "Creating volumes for persistent storage..."
Creating volumes for persistent storage...
echo "Created $(docker volume create --name=sentry-data)."
docker volume create --name=sentry-data
Created sentry-data.
echo "Created $(docker volume create --name=sentry-postgres)."
docker volume create --name=sentry-postgres
Created sentry-postgres.
echo "Created $(docker volume create --name=sentry-redis)."
docker volume create --name=sentry-redis
Created sentry-redis.
echo "Created $(docker volume create --name=sentry-zookeeper)."
docker volume create --name=sentry-zookeeper
Created sentry-zookeeper.
echo "Created $(docker volume create --name=sentry-kafka)."
docker volume create --name=sentry-kafka
Created sentry-kafka.
echo "Created $(docker volume create --name=sentry-clickhouse)."
docker volume create --name=sentry-clickhouse
Created sentry-clickhouse.
echo "Created $(docker volume create --name=sentry-symbolicator)."
docker volume create --name=sentry-symbolicator
Created sentry-symbolicator.
echo ""
ensure_file_from_example $SENTRY_CONFIG_PY
sentry/sentry.conf.py already exists, skipped creation.
ensure_file_from_example $SENTRY_CONFIG_YML
sentry/config.yml already exists, skipped creation.
ensure_file_from_example $SENTRY_EXTRA_REQUIREMENTS
sentry/requirements.txt already exists, skipped creation.
if grep -xq "system.secret-key: '!!changeme!!'" $SENTRY_CONFIG_YML ; then
echo ""
echo "Generating secret key..."
# This is to escape the secret key to be used in sed below
# Note the need to set LC_ALL=C due to BSD tr and sed always trying to decode
# whatever is passed to them. Kudos to https://stackoverflow.com/a/23584470/90297
SECRET_KEY=$(export LC_ALL=C; head /dev/urandom | tr -dc "a-z0-9@#%^&*(-_=+)" | head -c 50 | sed -e 's/[\/&]/\\&/g')
sed -i -e 's/^system.secret-key:.*$/system.secret-key: '"'$SECRET_KEY'"'/' $SENTRY_CONFIG_YML
echo "Secret key written to $SENTRY_CONFIG_YML"
fi
replace_tsdb() {
if (
[ -f "$SENTRY_CONFIG_PY" ] &&
! grep -xq 'SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"' "$SENTRY_CONFIG_PY"
); then
tsdb_settings="SENTRY_TSDB = \"sentry.tsdb.redissnuba.RedisSnubaTSDB\"
# Automatic switchover 90 days after $(date). Can be removed afterwards.
SENTRY_TSDB_OPTIONS = {\"switchover_timestamp\": $(date +%s) + (90 * 24 * 3600)}"
if grep -q 'SENTRY_TSDB_OPTIONS = ' "$SENTRY_CONFIG_PY"; then
echo "Not attempting automatic TSDB migration due to presence of SENTRY_TSDB_OPTIONS"
else
echo "Attempting to automatically migrate to new TSDB"
# Escape newlines for sed
tsdb_settings="${tsdb_settings//$'\n'/\\n}"
cp "$SENTRY_CONFIG_PY" "$SENTRY_CONFIG_PY.bak"
sed -i -e "s/^SENTRY_TSDB = .*$/${tsdb_settings}/g" "$SENTRY_CONFIG_PY" || true
if grep -xq 'SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"' "$SENTRY_CONFIG_PY"; then
echo "Migrated TSDB to Snuba. Old configuration file backed up to $SENTRY_CONFIG_PY.bak"
return
fi
echo "Failed to automatically migrate TSDB. Reverting..."
mv "$SENTRY_CONFIG_PY.bak" "$SENTRY_CONFIG_PY"
echo "$SENTRY_CONFIG_PY restored from backup."
fi
echo "WARN: Your Sentry configuration uses a legacy data store for time-series data. Remove the options SENTRY_TSDB and SENTRY_TSDB_OPTIONS from $SENTRY_CONFIG_PY and add:"
echo ""
echo "$tsdb_settings"
echo ""
echo "For more information please refer to https://github.com/getsentry/onpremise/pull/430"
fi
}
replace_tsdb
echo ""
echo "Fetching and updating Docker images..."
Fetching and updating Docker images...
echo ""
# We tag locally built images with an '-onpremise-local' suffix. docker-compose pull tries to pull these too and
# shows a 404 error on the console which is confusing and unnecessary. To overcome this, we add the stderr>stdout
# redirection below and pass it through grep, ignoring all lines having this '-onpremise-local' suffix.
$dc pull -q --ignore-pull-failures 2>&1 | grep -v -- -onpremise-local || true
if [ -z "$SENTRY_IMAGE" ]; then
docker pull getsentry/sentry:${SENTRY_VERSION:-latest}
else
# We may not have the set image on the repo (local images) so allow fails
docker pull $SENTRY_IMAGE || true;
fi
latest: Pulling from getsentry/sentry
Digest: sha256:90481a1c09ab8309a088f0302973c4daf5f9be527b4ba3a08f9a6fd0f07fd071
Status: Image is up to date for getsentry/sentry:latest
echo ""
echo "Building and tagging Docker images..."
Building and tagging Docker images...
echo ""
# Build the sentry onpremise image first as it is needed for the cron image
$dc build --force-rm web
Building web
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
---> 3c2b24cc09c0
Step 4/5 : COPY . /usr/src/sentry
---> Using cache
---> 1172ea707c37
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 78f3b9f7311f
Successfully built 78f3b9f7311f
Successfully tagged sentry-onpremise-local:latest
$dc build --force-rm --parallel
smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-replacer uses an image, skipping
symbolicator uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building snuba-cleanup ...
Building symbolicator-cleanup ...
Building web ...
Building cron ...
Building worker ...
Building ingest-consumer ...
Building post-process-forwarder ...
Building sentry-cleanup ...
Building web
Building worker
Building ingest-consumer
Building cron
Building symbolicator-cleanup
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
---> 3c2b24cc09c0
Step 4/5 : COPY . /usr/src/sentry
---> Using cache
---> 1172ea707c37
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 78f3b9f7311f
Successfully built 78f3b9f7311f
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Successfully tagged sentry-onpremise-local:latest
---> 3c2b24cc09c0
---> 3c2b24cc09c0
Step 4/5 : COPY . /usr/src/sentryBuilding web ... done
Building snuba-cleanup
Step 4/5 : COPY . /usr/src/sentry ---> Using cache
---> Using cache
---> 1172ea707c37
---> 1172ea707c37
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
Step 1/5 : ARG SENTRY_VERSION=latestStep 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi ---> Using cache
---> 78f3b9f7311f
---> Using cache
---> 78f3b9f7311f
Step 1/5 : ARG BASE_IMAGESuccessfully built 78f3b9f7311f
Step 2/5 : ARG SENTRY_IMAGESuccessfully tagged sentry-onpremise-local:latest
Successfully built 78f3b9f7311f
Building ingest-consumer ... done
Building sentry-cleanup
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder
Step 2/5 : FROM ${BASE_IMAGE}Building worker ... done
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION} ---> 515e555c0b18
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*
---> 3c2b24cc09c0
---> Using cache
Step 4/5 : COPY . /usr/src/sentry
---> 9e95508b94cf
---> Using cache
Step 4/5 : COPY entrypoint.sh /entrypoint.sh ---> 1172ea707c37
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi ---> Using cache
---> Using cache
---> fcb0252a2878
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"] ---> 78f3b9f7311f
---> Using cache
Successfully built 78f3b9f7311f
Successfully tagged sentry-onpremise-local:latest
Building cron ... done
---> a56948a8b402
Successfully built a56948a8b402
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building symbolicator-cleanup ... done
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
---> 31f4b7fa2a08
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*
---> Using cache
---> 1f085e4ced7b
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
---> Using cache
---> 4239092e9ddc
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
---> Using cache
---> ac585a9bc1bc
Successfully built ac585a9bc1bc
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
Successfully tagged snuba-cleanup-onpremise-local:latest
---> 78f3b9f7311f
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*
Building snuba-cleanup ... done
---> Using cache
---> afaf2fada71f
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
---> Using cache
---> a6749156924c
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
---> Using cache
---> 9e189eb250ba
Successfully built 9e189eb250ba
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
---> 3c2b24cc09c0
Step 4/5 : COPY . /usr/src/sentry
---> Using cache
---> 1172ea707c37
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
---> Using cache
---> 78f3b9f7311f
Successfully built 78f3b9f7311f
Successfully tagged sentry-cleanup-onpremise-local:latest
Building sentry-cleanup ... done
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder ... done
echo ""
echo "Docker images built."
Docker images built.
ZOOKEEPER_LOG_FILE_COUNT=$($dcr zookeeper bash -c 'ls 2>/dev/null -Ubad1 -- /var/lib/zookeeper/log/version-2/* | wc -l | tr -d '[:space:]'')
$dcr zookeeper bash -c 'ls 2>/dev/null -Ubad1 -- /var/lib/zookeeper/log/version-2/* | wc -l | tr -d '[:space:]''
Creating network "sentry_onpremise_default" with the default driver
Creating volume "sentry_onpremise_sentry-secrets" with default driver
Creating volume "sentry_onpremise_sentry-smtp" with default driver
Creating volume "sentry_onpremise_sentry-zookeeper-log" with default driver
Creating volume "sentry_onpremise_sentry-kafka-log" with default driver
Creating volume "sentry_onpremise_sentry-smtp-log" with default driver
Creating volume "sentry_onpremise_sentry-clickhouse-log" with default driver
ZOOKEEPER_SNAPSHOT_FILE_COUNT=$($dcr zookeeper bash -c 'ls 2>/dev/null -Ubad1 -- /var/lib/zookeeper/data/version-2/* | wc -l | tr -d '[:space:]'')
$dcr zookeeper bash -c 'ls 2>/dev/null -Ubad1 -- /var/lib/zookeeper/data/version-2/* | wc -l | tr -d '[:space:]''
# This is a workaround for a ZK upgrade bug: https://issues.apache.org/jira/browse/ZOOKEEPER-3056
if [ "$ZOOKEEPER_LOG_FILE_COUNT" -gt "0" ] && [ "$ZOOKEEPER_SNAPSHOT_FILE_COUNT" -eq "0" ]; then
$dcr -v $(pwd)/zookeeper:/temp zookeeper bash -c 'cp /temp/snapshot.0 /var/lib/zookeeper/data/version-2/snapshot.0'
$dc run -e ZOOKEEPER_SNAPSHOT_TRUST_EMPTY=true zookeeper
fi
echo "Bootstrapping and migrating Snuba..."
Bootstrapping and migrating Snuba...
$dcr snuba-api bootstrap --force
Creating sentry_onpremise_zookeeper_1 ...
Creating sentry_onpremise_redis_1 ...
Creating sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_redis_1 ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_zookeeper_1 ... done
Creating sentry_onpremise_kafka_1 ...
Creating sentry_onpremise_kafka_1 ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ set gosu snuba snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
2020-05-25 13:16:44,645 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-05-25 13:16:45,900 Topic ingest-sessions created
2020-05-25 13:16:45,900 Topic events created
2020-05-25 13:16:45,900 Topic event-replacements created
2020-05-25 13:16:45,900 Topic snuba-commit-log created
2020-05-25 13:16:45,900 Topic errors-replacements created
2020-05-25 13:16:45,900 Topic outcomes created
2020-05-25 13:16:45,900 Topic cdc created
2020-05-25 13:16:45,914 Creating tables for storage events
2020-05-25 13:16:45,923 Migrating storage events
2020-05-25 13:16:45,959 Creating tables for storage errors
2020-05-25 13:16:45,967 Migrating storage errors
2020-05-25 13:16:45,990 Creating tables for storage groupedmessages
2020-05-25 13:16:45,995 Migrating storage groupedmessages
2020-05-25 13:16:46,000 Creating tables for storage groupassignees
2020-05-25 13:16:46,004 Migrating storage groupassignees
2020-05-25 13:16:46,014 Creating tables for storage outcomes_raw
2020-05-25 13:16:46,019 Migrating storage outcomes_raw
2020-05-25 13:16:46,027 Creating tables for storage outcomes_hourly
2020-05-25 13:16:46,036 Migrating storage outcomes_hourly
2020-05-25 13:16:46,039 Creating tables for storage sessions_raw
2020-05-25 13:16:46,044 Migrating storage sessions_raw
2020-05-25 13:16:46,050 Creating tables for storage sessions_hourly
2020-05-25 13:16:46,062 Migrating storage sessions_hourly
2020-05-25 13:16:46,068 Creating tables for storage transactions
2020-05-25 13:16:46,075 Migrating storage transactions
echo ""
# Very naively check whether there's an existing sentry-postgres volume and the PG version in it
if [[ $(docker volume ls -q --filter name=sentry-postgres) && $(docker run --rm -v sentry-postgres:/db busybox cat /db/PG_VERSION 2>/dev/null) == "9.5" ]]; then
docker volume rm sentry-postgres-new || true
# If this is Postgres 9.5 data, start upgrading it to 9.6 in a new volume
docker run --rm \
-v sentry-postgres:/var/lib/postgresql/9.5/data \
-v sentry-postgres-new:/var/lib/postgresql/9.6/data \
tianon/postgres-upgrade:9.5-to-9.6
# Get rid of the old volume as we'll rename the new one to that
docker volume rm sentry-postgres
docker volume create --name sentry-postgres
# There's no rename volume in Docker so copy the contents from old to new name
# Also append the `host all all all trust` line as `tianon/postgres-upgrade:9.5-to-9.6`
# doesn't do that automatically.
docker run --rm -v sentry-postgres-new:/from -v sentry-postgres:/to alpine ash -c \
"cd /from ; cp -av . /to ; echo 'host all all all trust' >> /to/pg_hba.conf"
# Finally, remove the new old volume as we are all in sentry-postgres now
docker volume rm sentry-postgres-new
fi
docker volume ls -q --filter name=sentry-postgres
docker run --rm -v sentry-postgres:/db busybox cat /db/PG_VERSION 2>/dev/null
echo ""
echo "Setting up database..."
Setting up database...
if [ $CI ]; then
$dcr web upgrade --noinput
echo ""
echo "Did not prompt for user creation due to non-interactive shell."
echo "Run the following command to create one yourself (recommended):"
echo ""
echo " docker-compose run --rm web createuser"
echo ""
else
$dcr web upgrade
fi
Creating sentry_onpremise_symbolicator_1 ...
Starting sentry_onpremise_zookeeper_1 ...
Starting sentry_onpremise_zookeeper_1 ... done
Starting sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_memcached_1 ...
Starting sentry_onpremise_clickhouse_1 ... done
Starting sentry_onpremise_redis_1 ...
Creating sentry_onpremise_postgres_1 ...
Starting sentry_onpremise_redis_1 ... done
Starting sentry_onpremise_kafka_1 ...
Starting sentry_onpremise_kafka_1 ... done
Creating sentry_onpremise_smtp_1 ...
Creating sentry_onpremise_snuba-api_1 ...
Creating sentry_onpremise_snuba-outcomes-consumer_1 ...
Creating sentry_onpremise_snuba-consumer_1 ...
Creating sentry_onpremise_snuba-replacer_1 ...
Creating sentry_onpremise_symbolicator_1 ... done
Creating sentry_onpremise_memcached_1 ... done
Creating sentry_onpremise_postgres_1 ... done
Creating sentry_onpremise_snuba-api_1 ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1 ... done
Creating sentry_onpremise_smtp_1 ... done
Creating sentry_onpremise_snuba-consumer_1 ... done
Creating sentry_onpremise_snuba-replacer_1 ... done
13:16:59 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
13:17:01 [INFO] sentry.plugins.github: apps-not-configured
[36;1mOperations to perform:[0m
[1m Apply all migrations: [0madmin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
[36;1mRunning migrations:[0m
Applying sentry.0001_initial...[32;1m OK[0m
Applying contenttypes.0001_initial...[32;1m OK[0m
Applying admin.0001_initial...[32;1m OK[0m
Applying admin.0002_logentry_remove_auto_add...[32;1m OK[0m
Applying contenttypes.0002_remove_content_type_name...[32;1m OK[0m
Applying auth.0001_initial...[32;1m OK[0m
Applying auth.0002_alter_permission_name_max_length...[32;1m OK[0m
Applying auth.0003_alter_user_email_max_length...[32;1m OK[0m
Applying auth.0004_alter_user_username_opts...[32;1m OK[0m
Applying auth.0005_alter_user_last_login_null...[32;1m OK[0m
Applying auth.0006_require_contenttypes_0002...[32;1m OK[0m
Applying auth.0007_alter_validators_add_error_messages...[32;1m OK[0m
Applying auth.0008_alter_user_username_max_length...[32;1m OK[0m
Applying jira_ac.0001_initial...[32;1m OK[0m
Applying nodestore.0001_initial...[32;1m OK[0m
Applying sentry.0002_912_to_recent...[32;1m OK[0m
Applying sentry.0003_auto_20191022_0122...[32;1m OK[0m
Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau...[32;1m OK[0m
Applying sentry.0005_fix_content_types...[32;1m OK[0m
Applying sentry.0006_sentryapp_date_published...[32;1m OK[0m
Applying sentry.0007_auto_20191029_0131...[32;1m OK[0m
Applying sentry.0008_auto_20191030_0016...[32;1m OK[0m
Applying sentry.0009_auto_20191101_1608...[32;1m OK[0m
Applying sentry.0010_auto_20191104_1641...[32;1m OK[0m
Applying sentry.0011_remove_pagerdutyservice_service_id_from_state...[32;1m OK[0m
Applying sentry.0012_remove_pagerdutyservice_service_id...[32;1m OK[0m
Applying sentry.0013_auto_20191111_1829...[32;1m OK[0m
Applying sentry.0014_delete_sentryappwebhookerror...[32;1m OK[0m
Applying sentry.0015_delete_sentryappwebhookerror_db...[32;1m OK[0m
Applying sentry.0016_delete_alert_rule_deprecated_fields...[32;1m OK[0m
Applying sentry.0017_incident_aggregation...[32;1m OK[0m
Applying sentry.0018_discoversavedquery_version...[32;1m OK[0m
Applying sentry.0019_auto_20191114_2040...[32;1m OK[0m
Applying sentry.0020_auto_20191125_1420...[32;1m OK[0m
Applying sentry.0021_auto_20191203_1803...[32;1m OK[0m
Applying sentry.0021_auto_20191202_1716...[32;1m OK[0m
Applying sentry.0022_merge...[32;1m OK[0m
Applying sentry.0023_hide_environment_none_20191126...[32;1m OK[0m
Applying sentry.0024_auto_20191230_2052...Nothing to do, skipping migration.
[32;1m OK[0m
Applying sentry.0025_organizationaccessrequest_requester...[32;1m OK[0m
Applying sentry.0026_delete_event...[32;1m OK[0m
Applying sentry.0027_exporteddata...[32;1m OK[0m
Applying sentry.0028_user_reports...[32;1m OK[0m
Applying sentry.0029_discover_query_upgrade...[32;1m OK[0m
Applying sentry.0030_auto_20200201_0039...[32;1m OK[0m
Applying sentry.0031_delete_alert_rules_and_incidents...[32;1m OK[0m
Applying sentry.0032_delete_alert_email...[32;1m OK[0m
Applying sentry.0033_auto_20200210_2137...[32;1m OK[0m
Applying sentry.0034_auto_20200210_2311...[32;1m OK[0m
Applying sentry.0035_auto_20200127_1711...[32;1m OK[0m
Applying sentry.0036_auto_20200213_0106...[32;1m OK[0m
Applying sentry.0037_auto_20200213_0140...[32;1m OK[0m
Applying sentry.0038_auto_20200213_1904...[32;1m OK[0m
Applying sentry.0039_delete_incidentsuspectcommit...[32;1m OK[0m
Applying sentry.0040_remove_incidentsuspectcommittable...[32;1m OK[0m
Applying sentry.0041_incidenttrigger_date_modified...[32;1m OK[0m
Applying sentry.0042_auto_20200214_1607...[32;1m OK[0m
Applying sentry.0043_auto_20200218_1903...[32;1m OK[0m
Applying sentry.0044_auto_20200219_0018...[32;1m OK[0m
Applying sentry.0045_remove_incidentactivity_event_stats_snapshot...[32;1m OK[0m
Applying sentry.0046_auto_20200221_1735...[32;1m OK[0m
Applying sentry.0047_auto_20200224_2319...[32;1m OK[0m
Applying sentry.0048_auto_20200302_1825...[32;1m OK[0m
Applying sentry.0049_auto_20200304_0254...[32;1m OK[0m
Applying sentry.0050_auto_20200306_2346...[32;1m OK[0m
Applying sentry.0051_fix_auditlog_pickled_data...[32;1m OK[0m
Applying sentry.0052_organizationonboardingtask_completion_seen...[32;1m OK[0m
Applying sentry.0053_migrate_alert_task_onboarding...[32;1m OK[0m
Applying sentry.0054_create_key_transaction...[32;1m OK[0m
Applying sentry.0055_query_subscription_status...[32;1m OK[0m
Applying sentry.0056_remove_old_functions...[32;1m OK[0m
Applying sentry.0057_remove_unused_project_flag...[32;1m OK[0m
Applying sentry.0058_project_issue_alerts_targeting...[32;1m OK[0m
Applying sentry.0059_add_new_sentry_app_features...[32;1m OK[0m
Applying sentry.0060_add_file_eventattachment_index...[32;1m OK[0m
Applying sentry.0061_alertrule_partial_index...[32;1m OK[0m
Applying sentry.0062_key_transactions_unique_with_owner...[32;1m OK[0m
Applying sentry.0063_drop_alertrule_constraint...[32;1m OK[0m
Applying sentry.0064_project_has_transactions...[32;1m OK[0m
Applying sentry.0065_add_incident_status_method...[32;1m OK[0m
Applying sentry.0066_alertrule_manager...[32;1m OK[0m
Applying sentry.0067_migrate_rules_alert_targeting...[32;1m OK[0m
Applying sentry.0068_project_default_flags...[32;1m OK[0m
Applying sentry.0069_remove_tracked_superusers...[32;1m OK[0m
Applying sentry.0070_incident_snapshot_support...[32;1m OK[0m
Applying sentry.0071_add_default_fields_model_subclass...[32;1m OK[0m
Applying sentry.0072_alert_rules_query_changes...[32;1m OK[0m
Applying sentry.0073_migrate_alert_query_model...[32;1m OK[0m
Applying sentry.0074_add_metric_alert_feature...[32;1m OK[0m
Applying sentry.0075_metric_alerts_fix_releases...[32;1m OK[0m
Applying sentry.0076_alert_rules_disable_constraints...[32;1m OK[0m
Applying sentry.0077_alert_query_col_drop_state...[32;1m OK[0m
Applying sentry.0078_incident_field_updates...[32;1m OK[0m
Applying sentry.0079_incidents_remove_query_field_state...[32;1m OK[0m
Applying sessions.0001_initial...[32;1m OK[0m
Applying sites.0001_initial...[32;1m OK[0m
Applying sites.0002_alter_domain_unique...[32;1m OK[0m
Applying social_auth.0001_initial...[32;1m OK[0m
13:17:52 [WARNING] sentry: Cannot initiate onboarding for organization (1) due to missing owners
Created internal Sentry project (slug=internal, id=1)
Would you like to create a user account now? [Y/n]: y
Email: [email protected]
Password:
Repeat for confirmation:
User created: [email protected]
Added to organization: sentry
Creating missing DSNs
Correcting Group.num_comments counter
SENTRY_DATA_NEEDS_MIGRATION=$(docker run --rm -v sentry-data:/data alpine ash -c "[ ! -d '/data/files' ] && ls -A1x /data | wc -l || true")
docker run --rm -v sentry-data:/data alpine ash -c "[ ! -d '/data/files' ] && ls -A1x /data | wc -l || true"
if [ "$SENTRY_DATA_NEEDS_MIGRATION" ]; then
echo "Migrating file storage..."
# Use the web (Sentry) image so the file owners are kept as sentry:sentry
# The `\"` escape pattern is to make this compatible w/ Git Bash on Windows. See #329.
$dcr --entrypoint \"/bin/bash\" web -c \
"mkdir -p /tmp/files; mv /data/* /tmp/files/; mv /tmp/files /data/files; chown -R sentry:sentry /data"
fi
if [ ! -f "$RELAY_CREDENTIALS_JSON" ]; then
echo ""
echo "Generating Relay credentials..."
# We need the ugly hack below as `relay generate credentials` tries to read the config and the credentials
# even with the `--stdout` and `--overwrite` flags and then errors out when the credentials file exists but
# not valid JSON. We hit this case as we redirect output to the same config folder, creating an empty
# credentials file before relay runs.
$dcr --no-deps -v $(pwd)/$RELAY_CONFIG_YML:/tmp/config.yml relay --config /tmp credentials generate --stdout > "$RELAY_CREDENTIALS_JSON"
echo "Relay credentials written to $RELAY_CREDENTIALS_JSON"
fi
Generating Relay credentials...
pwd
Relay credentials written to relay/credentials.json
RELAY_CREDENTIALS=$(sed -n 's/^.*"public_key"[[:space:]]*:[[:space:]]*"\([a-zA-Z0-9_-]\{1,\}\)".*$/\1/p' "$RELAY_CREDENTIALS_JSON")
sed -n 's/^.*"public_key"[[:space:]]*:[[:space:]]*"\([a-zA-Z0-9_-]\{1,\}\)".*$/\1/p' "$RELAY_CREDENTIALS_JSON"
if [ -z "$RELAY_CREDENTIALS" ]; then
>&2 echo "FAIL: Cannot read credentials back from $RELAY_CREDENTIALS_JSON."
>&2 echo " Please ensure this file is readable and contains valid credentials."
>&2 echo ""
exit 1
fi
if ! grep -q "\"$RELAY_CREDENTIALS\"" "$SENTRY_CONFIG_PY"; then
echo "SENTRY_RELAY_WHITELIST_PK = (SENTRY_RELAY_WHITELIST_PK or []) + ([\"$RELAY_CREDENTIALS\"])" >> "$SENTRY_CONFIG_PY"
echo "Relay public key written to $SENTRY_CONFIG_PY"
echo ""
fi
Relay public key written to sentry/sentry.conf.py
cleanup
Cleaning up...
echo ""
echo "----------------"
----------------
echo "You're all done! Run the following command to get Sentry running:"
You're all done! Run the following command to get Sentry running:
echo ""
echo " docker-compose up -d"
docker-compose up -d
echo ""
docker-compose exec web cat /etc/sentry/sentry.conf.py
# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.
from sentry.conf.server import * # NOQA
DATABASES = {
"default": {
"ENGINE": "sentry.db.postgres",
"NAME": "postgres",
"USER": "postgres",
"PASSWORD": "",
"HOST": "postgres",
"PORT": "",
}
}
# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True
# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings
###########
# General #
###########
# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True
SENTRY_OPTIONS["system.event-retention-days"] = int(
env('SENTRY_EVENT_RETENTION_DAYS', '90')
)
#########
# Redis #
#########
# Generic Redis configuration used as defaults for various things including:
# Buffers, Quotas, TSDB
SENTRY_OPTIONS["redis.clusters"] = {
"default": {
"hosts": {0: {"host": "redis", "password": "", "port": "6379", "db": "0"}}
}
}
#########
# Queue #
#########
# See https://docs.getsentry.com/on-premise/server/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.
rabbitmq_host = None
if rabbitmq_host:
BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
username="guest", password="guest", host=rabbitmq_host, vhost="/"
)
else:
BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
**SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
)
#########
# Cache #
#########
# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
"LOCATION": ["memcached:11211"],
"TIMEOUT": 3600,
}
}
# A primary cache is required for things such as processing events
SENTRY_CACHE = "sentry.cache.redis.RedisCache"
DEFAULT_KAFKA_OPTIONS = {
"bootstrap.servers": "kafka:9092",
"message.max.bytes": 50000000,
"socket.timeout.ms": 1000,
}
SENTRY_EVENTSTREAM = "sentry.eventstream.kafka.KafkaEventStream"
SENTRY_EVENTSTREAM_OPTIONS = {"producer_configuration": DEFAULT_KAFKA_OPTIONS}
KAFKA_CLUSTERS["default"] = DEFAULT_KAFKA_OPTIONS
###############
# Rate Limits #
###############
# Rate limits apply to notification handlers and are enforced per-project
# automatically.
SENTRY_RATELIMITER = "sentry.ratelimits.redis.RedisRateLimiter"
##################
# Update Buffers #
##################
# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)
SENTRY_BUFFER = "sentry.buffer.redis.RedisBuffer"
##########
# Quotas #
##########
# Quotas allow you to rate limit individual projects or the Sentry install as
# a whole.
SENTRY_QUOTAS = "sentry.quotas.redis.RedisQuota"
########
# TSDB #
########
# The TSDB is used for building charts as well as making things like per-rate
# alerts possible.
SENTRY_TSDB = "sentry.tsdb.redissnuba.RedisSnubaTSDB"
#########
# SNUBA #
#########
SENTRY_SEARCH = "sentry.search.snuba.EventsDatasetSnubaSearchBackend"
SENTRY_SEARCH_OPTIONS = {}
SENTRY_TAGSTORE_OPTIONS = {}
###########
# Digests #
###########
# The digest backend powers notification summaries.
SENTRY_DIGESTS = "sentry.digests.backends.redis.RedisBackend"
##############
# Web Server #
##############
SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
# These ase for proper HTTP/1.1 support from uWSGI
# Without these it doesn't do keep-alives causing
# issues with Relay's direct requests.
"http-keepalive": True,
"http-chunked-input": True,
# the number of web workers
'workers': 3,
# Turn off memory reporting
"memory-report": False,
# Some stuff so uwsgi will cycle workers sensibly
'max-requests': 100000,
'max-requests-delta': 500,
'max-worker-lifetime': 86400,
# Duplicate options from sentry default just so we don't get
# bit by sentry changing a default value that we depend on.
'thunder-lock': True,
'log-x-forwarded-for': False,
'buffer-size': 32768,
'limit-post': 209715200,
'disable-logging': True,
'reload-on-rss': 600,
'ignore-sigpipe': True,
'ignore-write-errors': True,
'disable-write-exception': True,
}
###########
# SSL/TLS #
###########
# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below
# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# SESSION_COOKIE_SECURE = True
# CSRF_COOKIE_SECURE = True
# SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
# End of SSL/TLS settings
############
# Features #
############
SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
{
feature: True
for feature in (
"organizations:discover",
"organizations:events",
"organizations:discover-basic",
"organizations:discover-query",
"organizations:events-v2",
"organizations:global-views",
"organizations:integrations-issue-basic",
"organizations:integrations-issue-sync",
"organizations:invite-members",
"organizations:sso-basic",
"organizations:sso-rippling",
"organizations:sso-saml2",
"projects:custom-inbound-filters",
"projects:data-forwarding",
"projects:discard-groups",
"projects:plugins",
"projects:rate-limits",
"projects:servicehooks",
)
}
)
######################
# GitHub Integration #
######################
GITHUB_EXTENDED_PERMISSIONS = ['repo']
#########################
# Bitbucket Integration #
########################
# BITBUCKET_CONSUMER_KEY = 'YOUR_BITBUCKET_CONSUMER_KEY'
# BITBUCKET_CONSUMER_SECRET = 'YOUR_BITBUCKET_CONSUMER_SECRET'
SENTRY_RELAY_WHITELIST_PK = (SENTRY_RELAY_WHITELIST_PK or []) + (["E_QPIZr7Lv5EC4ZbPbe9MPI0xEEnDaHyNefmIhaiUWI"])
docker-compose logs
Attaching to sentry_onpremise_nginx_1, sentry_onpremise_sentry-cleanup_1, sentry_onpremise_worker_1, sentry_onpremise_web_1, sentry_onpremise_cron_1, sentry_onpremise_post-process-forwarder_1, sentry_onpremise_ingest-consumer_1, sentry_onpremise_snuba-cleanup_1, sentry_onpremise_relay_1, sentry_onpremise_symbolicator-cleanup_1, sentry_onpremise_snuba-replacer_1, sentry_onpremise_snuba-consumer_1, sentry_onpremise_smtp_1, sentry_onpremise_snuba-outcomes-consumer_1, sentry_onpremise_postgres_1, sentry_onpremise_snuba-api_1, sentry_onpremise_memcached_1, sentry_onpremise_symbolicator_1, sentry_onpremise_kafka_1, sentry_onpremise_clickhouse_1, sentry_onpremise_zookeeper_1, sentry_onpremise_redis_1
sentry-cleanup_1 | SHELL=/bin/bash
sentry-cleanup_1 | BASH_ENV=/container.env
sentry-cleanup_1 | 0 0 * * * gosu sentry sentry cleanup --days 90 > /proc/1/fd/1 2>/proc/1/fd/2
worker_1 | 13:19:29 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
worker_1 | 13:19:31 [INFO] sentry.plugins.github: apps-not-configured
worker_1 | 13:19:31 [INFO] sentry.bgtasks: bgtask.spawn (task_name=u'sentry.bgtasks.clean_dsymcache:clean_dsymcache')
worker_1 | 13:19:31 [INFO] sentry.bgtasks: bgtask.spawn (task_name=u'sentry.bgtasks.clean_releasefilecache:clean_releasefilecache')
worker_1 |
worker_1 | -------------- celery@9dfacc3b2dcd v3.1.18 (Cipater)
worker_1 | ---- **** -----
worker_1 | --- * *** * -- Linux-4.15.0-1067-oem-x86_64-with-debian-10.1
worker_1 | -- * - **** ---
worker_1 | - ** ---------- [config]
worker_1 | - ** ---------- .> app: sentry:0x7fdc99be9d50
worker_1 | - ** ---------- .> transport: redis://redis:6379/0
worker_1 | - ** ---------- .> results: disabled
worker_1 | - *** --- * --- .> concurrency: 16 (prefork)
worker_1 | -- ******* ----
worker_1 | --- ***** ----- [queues]
worker_1 | -------------- .> activity.notify exchange=default(direct) key=activity.notify
worker_1 | .> alerts exchange=default(direct) key=alerts
worker_1 | .> app_platform exchange=default(direct) key=app_platform
worker_1 | .> assemble exchange=default(direct) key=assemble
worker_1 | .> auth exchange=default(direct) key=auth
worker_1 | .> buffers.process_pending exchange=default(direct) key=buffers.process_pending
worker_1 | .> cleanup exchange=default(direct) key=cleanup
worker_1 | .> commits exchange=default(direct) key=commits
worker_1 | .> counters-0 exchange=counters(direct) key=
worker_1 | .> data_export exchange=default(direct) key=data_export
worker_1 | .> default exchange=default(direct) key=default
worker_1 | .> digests.delivery exchange=default(direct) key=digests.delivery
worker_1 | .> digests.scheduling exchange=default(direct) key=digests.scheduling
worker_1 | .> email exchange=default(direct) key=email
worker_1 | .> events.preprocess_event exchange=default(direct) key=events.preprocess_event
worker_1 | .> events.process_event exchange=default(direct) key=events.process_event
worker_1 | .> events.reprocess_events exchange=default(direct) key=events.reprocess_events
worker_1 | .> events.reprocessing.preprocess_event exchange=default(direct) key=events.reprocessing.preprocess_event
worker_1 | .> events.reprocessing.process_event exchange=default(direct) key=events.reprocessing.process_event
worker_1 | .> events.reprocessing.symbolicate_event exchange=default(direct) key=events.reprocessing.symbolicate_event
worker_1 | .> events.save_event exchange=default(direct) key=events.save_event
worker_1 | .> events.symbolicate_event exchange=default(direct) key=events.symbolicate_event
worker_1 | .> files.delete exchange=default(direct) key=files.delete
worker_1 | .> incidents exchange=default(direct) key=incidents
worker_1 | .> integrations exchange=default(direct) key=integrations
worker_1 | .> merge exchange=default(direct) key=merge
worker_1 | .> options exchange=default(direct) key=options
worker_1 | .> relay_config exchange=default(direct) key=relay_config
worker_1 | .> reports.deliver exchange=default(direct) key=reports.deliver
worker_1 | .> reports.prepare exchange=default(direct) key=reports.prepare
worker_1 | .> search exchange=default(direct) key=search
worker_1 | .> sleep exchange=default(direct) key=sleep
worker_1 | .> stats exchange=default(direct) key=stats
worker_1 | .> subscriptions exchange=default(direct) key=subscriptions
worker_1 | .> triggers-0 exchange=triggers(direct) key=
worker_1 | .> unmerge exchange=default(direct) key=unmerge
worker_1 | .> update exchange=default(direct) key=update
worker_1 |
post-process-forwarder_1 | 13:19:31 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
post-process-forwarder_1 | 13:19:34 [INFO] sentry.plugins.github: apps-not-configured
cron_1 | 13:19:30 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
cron_1 | 13:19:33 [INFO] sentry.plugins.github: apps-not-configured
web_1 | 13:19:33 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 13:19:35 [INFO] sentry.plugins.github: apps-not-configured
web_1 | *** Starting uWSGI 2.0.18 (64bit) on [Mon May 25 13:19:35 2020] ***
web_1 | compiled with version: 8.3.0 on 24 May 2020 13:05:28
web_1 | os: Linux-4.15.0-1067-oem #77lvm1 SMP Wed Feb 12 12:16:29 CET 2020
web_1 | nodename: 0329e85267ed
web_1 | machine: x86_64
web_1 | clock source: unix
web_1 | detected number of CPU cores: 16
web_1 | current working directory: /
web_1 | detected binary path: /usr/local/bin/uwsgi
web_1 | !!! no internal routing support, rebuild with pcre support !!!
web_1 | your memory page size is 4096 bytes
web_1 | detected max file descriptor number: 1048576
web_1 | lock engine: pthread robust mutexes
web_1 | thunder lock: enabled
web_1 | uwsgi socket 0 bound to TCP address 0.0.0.0:9000 fd 3
web_1 | Python version: 2.7.16 (default, Oct 17 2019, 07:39:30) [GCC 8.3.0]
web_1 | Set PythonHome to /usr/local
web_1 | Python main interpreter initialized at 0x562ea4ea8100
web_1 | python threads support enabled
web_1 | your server socket listen backlog is limited to 100 connections
web_1 | your mercy for graceful operations on workers is 60 seconds
web_1 | setting request body buffering size to 65536 bytes
web_1 | mapped 1087040 bytes (1061 KB) for 6 cores
web_1 | *** Operational MODE: preforking+threaded ***
web_1 | spawned uWSGI master process (pid: 17)
web_1 | spawned uWSGI worker 1 (pid: 21, cores: 2)
web_1 | spawned uWSGI worker 2 (pid: 22, cores: 2)
web_1 | spawned uWSGI worker 3 (pid: 23, cores: 2)
web_1 | 13:19:36 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 13:19:36 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 13:19:36 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
web_1 | 13:19:39 [INFO] sentry.plugins.github: apps-not-configured
web_1 | 13:19:39 [INFO] sentry.plugins.github: apps-not-configured
web_1 | 13:19:39 [INFO] sentry.plugins.github: apps-not-configured
web_1 | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x562ea4ea8100 pid: 23 (default app)
web_1 | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x562ea4ea8100 pid: 22 (default app)
web_1 | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x562ea4ea8100 pid: 21 (default app)
snuba-cleanup_1 | SHELL=/bin/bash
snuba-cleanup_1 | BASH_ENV=/container.env
snuba-cleanup_1 | */5 * * * * gosu snuba snuba cleanup --dry-run False > /proc/1/fd/1 2>/proc/1/fd/2
snuba-cleanup_1 | 2020-05-25 13:20:02,154 Dropped 0 partitions on None
ingest-consumer_1 | 13:19:29 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
ingest-consumer_1 | 13:19:32 [INFO] sentry.plugins.github: apps-not-configured
ingest-consumer_1 | 13:19:37 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]
relay_1 | 2020-05-25T13:19:30Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Timeout while waiting for response
relay_1 | 2020-05-25T13:19:31Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Timeout while waiting for response
relay_1 | 2020-05-25T13:19:33Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Failed to connect to host: No route to host (os error 113)
relay_1 | caused by: No route to host (os error 113)
relay_1 | caused by: No route to host (os error 113)
relay_1 | 2020-05-25T13:19:34Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | 2020-05-25T13:19:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:19:43Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:19:48Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:19:56Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:20:08Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:20:25Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:20:50Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:21:29Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:22:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
relay_1 | 2020-05-25T13:23:27Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: IO error: Connection reset by peer (os error 104)
relay_1 | caused by: Connection reset by peer (os error 104)
symbolicator-cleanup_1 | SHELL=/bin/bash
symbolicator-cleanup_1 | BASH_ENV=/container.env
symbolicator-cleanup_1 | 55 23 * * * gosu symbolicator symbolicator cleanup > /proc/1/fd/1 2>/proc/1/fd/2
smtp_1 | + sed -ri '
smtp_1 | s/^#?(dc_local_interfaces)=.*/\1='\''0.0.0.0 ; ::0'\''/;
smtp_1 | s/^#?(dc_other_hostnames)=.*/\1='\'''\''/;
smtp_1 | s/^#?(dc_relay_nets)=.*/\1='\''0.0.0.0\/0'\''/;
smtp_1 | s/^#?(dc_eximconfig_configtype)=.*/\1='\''internet'\''/;
smtp_1 | ' /etc/exim4/update-exim4.conf.conf
smtp_1 | + update-exim4.conf -v
smtp_1 | using non-split configuration scheme from /etc/exim4/exim4.conf.template
smtp_1 | undocumented line IGNORE_SMTP_LINE_LENGTH_LIMIT='true' found in
smtp_1 | /etc/exim4/update-exim4.conf.conf, generating exim macro
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 IPv6 socket creation failed: Address family not supported by protocol
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 Failed to create IPv6 socket for wildcard listening (No such file or directory): will use IPv4
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 exim 4.92 daemon started: pid=287, no queue runs, listening for SMTP on port 25 (IPv4)
smtp_1 | + sed -ri '
smtp_1 | s/^#?(dc_local_interfaces)=.*/\1='\''0.0.0.0 ; ::0'\''/;
smtp_1 | s/^#?(dc_other_hostnames)=.*/\1='\'''\''/;
smtp_1 | s/^#?(dc_relay_nets)=.*/\1='\''0.0.0.0\/0'\''/;
smtp_1 | s/^#?(dc_eximconfig_configtype)=.*/\1='\''internet'\''/;
smtp_1 | ' /etc/exim4/update-exim4.conf.conf
smtp_1 | + update-exim4.conf -v
smtp_1 | using non-split configuration scheme from /etc/exim4/exim4.conf.template
smtp_1 | undocumented line IGNORE_SMTP_LINE_LENGTH_LIMIT='true' found in
smtp_1 | /etc/exim4/update-exim4.conf.conf, generating exim macro
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 IPv6 socket creation failed: Address family not supported by protocol
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 Failed to create IPv6 socket for wildcard listening (Address family not supported by protocol): will use IPv4
smtp_1 | 287 LOG: MAIN
smtp_1 | 287 exim 4.92 daemon started: pid=287, no queue runs, listening for SMTP on port 25 (IPv4)
snuba-replacer_1 | + '[' r = - ']'
snuba-replacer_1 | + snuba replacer --help
snuba-replacer_1 | + set -- snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + set gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + exec gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | 2020-05-25 13:17:05,994 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}
snuba-replacer_1 | 2020-05-25 13:19:02,942 Partitions revoked: [Partition(topic=Topic(name='event-replacements'), index=0)]
snuba-replacer_1 | + '[' r = - ']'
snuba-replacer_1 | + snuba replacer --help
snuba-replacer_1 | + set -- snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + set gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + exec gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | 2020-05-25 13:19:35,579 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}
snuba-outcomes-consumer_1 | + '[' c = - ']'
snuba-outcomes-consumer_1 | + snuba consumer --help
snuba-outcomes-consumer_1 | + set -- snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | + set gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | + exec gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | 2020-05-25 13:17:06,652 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 0}
snuba-outcomes-consumer_1 | 2020-05-25 13:19:02,643 Partitions revoked: [Partition(topic=Topic(name='outcomes'), index=0)]
snuba-outcomes-consumer_1 | + '[' c = - ']'
snuba-outcomes-consumer_1 | + snuba consumer --help
snuba-outcomes-consumer_1 | + set -- snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | + set gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | + exec gosu snuba snuba consumer --storage outcomes_raw --auto-offset-reset=earliest --max-batch-time-ms 750
snuba-outcomes-consumer_1 | 2020-05-25 13:19:33,461 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 0}
snuba-consumer_1 | + '[' c = - ']'
snuba-consumer_1 | + snuba consumer --help
snuba-consumer_1 | + set -- snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | + set gosu snuba snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | + exec gosu snuba snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | 2020-05-25 13:17:06,652 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 0}
snuba-consumer_1 | 2020-05-25 13:19:03,027 Partitions revoked: [Partition(topic=Topic(name='events'), index=0)]
snuba-consumer_1 | + '[' c = - ']'
snuba-consumer_1 | + snuba consumer --help
snuba-consumer_1 | + set -- snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | + set gosu snuba snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | + exec gosu snuba snuba consumer --storage events --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1 | 2020-05-25 13:19:33,462 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 0}
memcached_1 | Signal handled: Terminated.
postgres_1 | ********************************************************************************
postgres_1 | WARNING: POSTGRES_HOST_AUTH_METHOD has been set to "trust". This will allow
postgres_1 | anyone with access to the Postgres port to access your database without
postgres_1 | a password, even if POSTGRES_PASSWORD is set. See PostgreSQL
postgres_1 | documentation about "trust":
postgres_1 | https://www.postgresql.org/docs/current/auth-trust.html
postgres_1 | In Docker's default configuration, this is effectively any other
postgres_1 | container on the same system.
postgres_1 |
postgres_1 | It is not recommended to use POSTGRES_HOST_AUTH_METHOD=trust. Replace
postgres_1 | it with "-e POSTGRES_PASSWORD=password" instead to set a password in
postgres_1 | "docker run".
postgres_1 | ********************************************************************************
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default timezone ... Etc/UTC
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | creating configuration files ... ok
postgres_1 | running bootstrap script ... ok
postgres_1 | performing post-bootstrap initialization ... ok
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....LOG: database system was shut down at 2020-05-25 13:16:53 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: autovacuum launcher started
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | waiting for server to shut down...LOG: received fast shutdown request
postgres_1 | LOG: aborting any active transactions
postgres_1 | .LOG: autovacuum launcher shutting down
postgres_1 | LOG: shutting down
postgres_1 | LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | LOG: could not create IPv6 socket: Address family not supported by protocol
postgres_1 | LOG: database system was shut down at 2020-05-25 13:16:54 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: autovacuum launcher started
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | ERROR: relation "sentry_option" does not exist at character 114
postgres_1 | STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."key" = 'system.url-prefix'
postgres_1 | ERROR: relation "sentry_option" does not exist at character 114
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
clickhouse_1 | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse_1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse_1 | Include not found: networks
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
clickhouse_1 | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse_1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse_1 | Include not found: networks
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
kafka_1 | ===> ENV Variables ...
kafka_1 | ALLOW_UNSIGNED=false
kafka_1 | COMPONENT=kafka
kafka_1 | CONFLUENT_DEB_VERSION=1
kafka_1 | CONFLUENT_PLATFORM_LABEL=
kafka_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
kafka_1 | CONFLUENT_VERSION=5.5.0
kafka_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
kafka_1 | HOME=/root
kafka_1 | HOSTNAME=9afb2726589e
kafka_1 | KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
kafka_1 | KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
kafka_1 | KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
kafka_1 | KAFKA_MAX_REQUEST_SIZE=50000000
kafka_1 | KAFKA_MESSAGE_MAX_BYTES=50000000
kafka_1 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
kafka_1 | KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
kafka_1 | KAFKA_VERSION=
kafka_1 | KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
kafka_1 | LANG=C.UTF-8
kafka_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kafka_1 | PWD=/
kafka_1 | PYTHON_PIP_VERSION=8.1.2
kafka_1 | PYTHON_VERSION=2.7.9-1
kafka_1 | SCALA_VERSION=2.12
kafka_1 | SHLVL=1
kafka_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
kafka_1 | _=/usr/bin/env
kafka_1 | ===> User
kafka_1 | uid=0(root) gid=0(root) groups=0(root)
kafka_1 | ===> Configuring ...
kafka_1 | ===> Running preflight checks ...
kafka_1 | ===> Check if /var/lib/kafka/data is writable ...
kafka_1 | ===> Check if Zookeeper is healthy ...
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=9afb2726589e
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1067-oem
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=469MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=7077MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=477MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1 | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.22.0.4:2181. Will not attempt to authenticate using SASL (unknown error)
postgres_1 | STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."key" = 'github.apps-install-url'
postgres_1 | ERROR: relation "sentry_option" does not exist at character 114
postgres_1 | STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."key" = 'system.url-prefix'
postgres_1 | ERROR: relation "sentry_option" does not exist at character 114
postgres_1 | STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."key" = 'system.url-prefix'
postgres_1 | ERROR: relation "sentry_projectkey" does not exist at character 371
postgres_1 | STATEMENT: SELECT "sentry_projectkey"."id", "sentry_projectkey"."project_id", "sentry_projectkey"."label", "sentry_projectkey"."public_key", "sentry_projectkey"."secret_key", "sentry_projectkey"."roles", "sentry_projectkey"."status", "sentry_projectkey"."date_added", "sentry_projectkey"."rate_limit_count", "sentry_projectkey"."rate_limit_window", "sentry_projectkey"."data" FROM "sentry_projectkey" WHERE ("sentry_projectkey"."project_id" = 1 AND "sentry_projectkey"."status" = 0 AND "sentry_projectkey"."roles" = (("sentry_projectkey"."roles" | 1))) ORDER BY "sentry_projectkey"."id" ASC LIMIT 1
postgres_1 | ERROR: relation "south_migrationhistory" does not exist at character 15
postgres_1 | STATEMENT: SELECT 1 FROM south_migrationhistory LIMIT 1
postgres_1 | LOG: received smart shutdown request
postgres_1 | LOG: autovacuum launcher shutting down
postgres_1 | LOG: shutting down
postgres_1 | LOG: database system is shut down
postgres_1 |
postgres_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1 |
postgres_1 | LOG: could not create IPv6 socket: Address family not supported by protocol
postgres_1 | LOG: database system was shut down at 2020-05-25 13:19:02 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | LOG: autovacuum launcher started
symbolicator_1 | 2020-05-25T13:16:50Z [symbolicator::server] INFO: Starting http server: 0.0.0.0:3021
symbolicator_1 | 2020-05-25T13:16:50Z [actix_net::server::server] INFO: Starting 16 workers
symbolicator_1 | 2020-05-25T13:16:50Z [actix_net::server::server] INFO: Starting server on 0.0.0.0:3021
symbolicator_1 | 2020-05-25T13:16:50Z [symbolicator::server] INFO: Starting system
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::server] INFO: SIGTERM received, stopping
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [actix_net::server::worker] INFO: Shutting down worker, 0 connections
symbolicator_1 | 2020-05-25T13:19:02Z [symbolicator::server] INFO: System shutdown complete
symbolicator_1 | 2020-05-25T13:19:17Z [symbolicator::server] INFO: Starting http server: 0.0.0.0:3021
symbolicator_1 | 2020-05-25T13:19:17Z [actix_net::server::server] INFO: Starting 16 workers
symbolicator_1 | 2020-05-25T13:19:17Z [actix_net::server::server] INFO: Starting server on 0.0.0.0:3021
symbolicator_1 | 2020-05-25T13:19:17Z [symbolicator::server] INFO: Starting system
snuba-api_1 | + '[' a = - ']'
snuba-api_1 | + snuba api --help
snuba-api_1 | + set -- snuba api
snuba-api_1 | + set gosu snuba snuba api
snuba-api_1 | + exec gosu snuba snuba api
snuba-api_1 | *** Starting uWSGI 2.0.18 (64bit) on [Mon May 25 13:16:55 2020] ***
snuba-api_1 | compiled with version: 8.3.0 on 22 May 2020 15:59:27
snuba-api_1 | os: Linux-4.15.0-1067-oem #77lvm1 SMP Wed Feb 12 12:16:29 CET 2020
snuba-api_1 | nodename: bd83e61c6522
snuba-api_1 | machine: x86_64
snuba-api_1 | clock source: unix
snuba-api_1 | pcre jit disabled
snuba-api_1 | detected number of CPU cores: 16
snuba-api_1 | current working directory: /usr/src/snuba
snuba-api_1 | detected binary path: /usr/local/bin/uwsgi
snuba-api_1 | your memory page size is 4096 bytes
snuba-api_1 | detected max file descriptor number: 1048576
snuba-api_1 | lock engine: pthread robust mutexes
snuba-api_1 | thunder lock: enabled
snuba-api_1 | uwsgi socket 0 bound to TCP address 0.0.0.0:1218 fd 3
snuba-api_1 | Python version: 3.7.7 (default, May 20 2020, 21:10:21) [GCC 8.3.0]
snuba-api_1 | Set PythonHome to /usr/local
snuba-api_1 | Python main interpreter initialized at 0x55b4d9aad7b0
snuba-api_1 | python threads support enabled
snuba-api_1 | your server socket listen backlog is limited to 100 connections
snuba-api_1 | your mercy for graceful operations on workers is 60 seconds
snuba-api_1 | mapped 145808 bytes (142 KB) for 1 cores
snuba-api_1 | *** Operational MODE: single process ***
snuba-api_1 | initialized 38 metrics
snuba-api_1 | spawned uWSGI master process (pid: 1)
snuba-api_1 | spawned uWSGI worker 1 (pid: 17, cores: 1)
snuba-api_1 | metrics collector thread started
snuba-api_1 | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x55b4d9aad7b0 pid: 17 (default app)
snuba-api_1 | SIGINT/SIGQUIT received...killing workers...
snuba-api_1 | worker 1 buried after 1 seconds
snuba-api_1 | goodbye to uWSGI.
snuba-api_1 | + '[' a = - ']'
snuba-api_1 | + snuba api --help
snuba-api_1 | + set -- snuba api
snuba-api_1 | + set gosu snuba snuba api
snuba-api_1 | + exec gosu snuba snuba api
snuba-api_1 | *** Starting uWSGI 2.0.18 (64bit) on [Mon May 25 13:19:26 2020] ***
snuba-api_1 | compiled with version: 8.3.0 on 22 May 2020 15:59:27
snuba-api_1 | os: Linux-4.15.0-1067-oem #77lvm1 SMP Wed Feb 12 12:16:29 CET 2020
snuba-api_1 | nodename: bd83e61c6522
snuba-api_1 | machine: x86_64
snuba-api_1 | clock source: unix
snuba-api_1 | pcre jit disabled
snuba-api_1 | detected number of CPU cores: 16
snuba-api_1 | current working directory: /usr/src/snuba
snuba-api_1 | detected binary path: /usr/local/bin/uwsgi
snuba-api_1 | your memory page size is 4096 bytes
snuba-api_1 | detected max file descriptor number: 1048576
snuba-api_1 | lock engine: pthread robust mutexes
snuba-api_1 | thunder lock: enabled
snuba-api_1 | uwsgi socket 0 bound to TCP address 0.0.0.0:1218 fd 3
snuba-api_1 | Python version: 3.7.7 (default, May 20 2020, 21:10:21) [GCC 8.3.0]
snuba-api_1 | Set PythonHome to /usr/local
snuba-api_1 | Python main interpreter initialized at 0x561da222e7b0
snuba-api_1 | python threads support enabled
snuba-api_1 | your server socket listen backlog is limited to 100 connections
snuba-api_1 | your mercy for graceful operations on workers is 60 seconds
snuba-api_1 | mapped 145808 bytes (142 KB) for 1 cores
snuba-api_1 | *** Operational MODE: single process ***
snuba-api_1 | initialized 38 metrics
snuba-api_1 | spawned uWSGI master process (pid: 1)
snuba-api_1 | spawned uWSGI worker 1 (pid: 17, cores: 1)
snuba-api_1 | metrics collector thread started
snuba-api_1 | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x561da222e7b0 pid: 17 (default app)
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.22.0.5:58958, server: zookeeper/172.22.0.4:2181
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.22.0.4:2181, sessionid = 0x10000db70690000, negotiated timeout = 40000
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000db70690000 closed
kafka_1 | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000db70690000
kafka_1 | ===> Launching ...
kafka_1 | ===> Launching kafka ...
kafka_1 | [2020-05-25 13:16:44,341] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka_1 | [2020-05-25 13:16:44,708] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
kafka_1 | [2020-05-25 13:16:44,708] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
kafka_1 | [2020-05-25 13:16:44,988] WARN No meta.properties file under dir /var/lib/kafka/data/meta.properties (kafka.server.BrokerMetadataCheckpoint)
kafka_1 | [2020-05-25 13:16:45,096] INFO Starting the log cleaner (kafka.log.LogCleaner)
kafka_1 | [2020-05-25 13:16:45,135] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
kafka_1 | [2020-05-25 13:16:45,369] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
kafka_1 | [2020-05-25 13:16:45,391] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:16:45,392] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:16:45,431] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-25 13:16:45,446] INFO Stat of the created znode at /brokers/ids/1001 is: 27,27,1590412605438,1590412605438,1,0,0,72058536521695233,180,0,27
kafka_1 | (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-25 13:16:45,446] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 27 (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-25 13:16:45,496] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-25 13:16:45,576] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-25 13:16:45,591] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:16:45,687] INFO Creating topic events with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:16:45,710] INFO Creating topic ingest-sessions with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:16:45,725] INFO Creating topic outcomes with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:16:45,752] INFO Creating topic snuba-commit-log with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:16:45,763] INFO Creating topic cdc with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:16:45,772] INFO Creating topic errors-replacements with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:16:45,781] INFO Creating topic event-replacements with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:16:57,634] INFO Creating topic __consumer_offsets with configuration {segment.bytes=104857600, compression.type=producer, cleanup.policy=compact} and initial partition assignment Map(23 -> ArrayBuffer(1001), 32 -> ArrayBuffer(1001), 41 -> ArrayBuffer(1001), 17 -> ArrayBuffer(1001), 8 -> ArrayBuffer(1001), 35 -> ArrayBuffer(1001), 44 -> ArrayBuffer(1001), 26 -> ArrayBuffer(1001), 11 -> ArrayBuffer(1001), 29 -> ArrayBuffer(1001), 38 -> ArrayBuffer(1001), 47 -> ArrayBuffer(1001), 20 -> ArrayBuffer(1001), 2 -> ArrayBuffer(1001), 5 -> ArrayBuffer(1001), 14 -> ArrayBuffer(1001), 46 -> ArrayBuffer(1001), 49 -> ArrayBuffer(1001), 40 -> ArrayBuffer(1001), 13 -> ArrayBuffer(1001), 4 -> ArrayBuffer(1001), 22 -> ArrayBuffer(1001), 31 -> ArrayBuffer(1001), 16 -> ArrayBuffer(1001), 7 -> ArrayBuffer(1001), 43 -> ArrayBuffer(1001), 25 -> ArrayBuffer(1001), 34 -> ArrayBuffer(1001), 10 -> ArrayBuffer(1001), 37 -> ArrayBuffer(1001), 1 -> ArrayBuffer(1001), 19 -> ArrayBuffer(1001), 28 -> ArrayBuffer(1001), 45 -> ArrayBuffer(1001), 27 -> ArrayBuffer(1001), 36 -> ArrayBuffer(1001), 18 -> ArrayBuffer(1001), 9 -> ArrayBuffer(1001), 21 -> ArrayBuffer(1001), 48 -> ArrayBuffer(1001), 3 -> ArrayBuffer(1001), 12 -> ArrayBuffer(1001), 30 -> ArrayBuffer(1001), 39 -> ArrayBuffer(1001), 15 -> ArrayBuffer(1001), 42 -> ArrayBuffer(1001), 24 -> ArrayBuffer(1001), 6 -> ArrayBuffer(1001), 33 -> ArrayBuffer(1001), 0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:19:04,469] INFO [/config/changes-event-process-thread]: Shutting down (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-25 13:19:04,469] INFO [/config/changes-event-process-thread]: Shutdown completed (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-25 13:19:04,469] INFO [/config/changes-event-process-thread]: Stopped (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-25 13:19:04,470] INFO [SocketServer brokerId=1001] Stopping socket server request processors (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:19:04,474] INFO [SocketServer brokerId=1001] Stopped socket server request processors (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:19:05,495] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)
kafka_1 | [2020-05-25 13:19:05,495] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)
kafka_1 | [2020-05-25 13:19:05,496] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)
kafka_1 | [2020-05-25 13:19:05,496] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)
kafka_1 | [2020-05-25 13:19:08,074] INFO [SocketServer brokerId=1001] Shutting down socket server (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:19:08,109] INFO [SocketServer brokerId=1001] Shutdown completed (kafka.network.SocketServer)
kafka_1 | ===> ENV Variables ...
kafka_1 | ALLOW_UNSIGNED=false
kafka_1 | COMPONENT=kafka
kafka_1 | CONFLUENT_DEB_VERSION=1
kafka_1 | CONFLUENT_PLATFORM_LABEL=
kafka_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
kafka_1 | CONFLUENT_VERSION=5.5.0
kafka_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
kafka_1 | HOME=/root
kafka_1 | HOSTNAME=9afb2726589e
kafka_1 | KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
kafka_1 | KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
kafka_1 | KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
kafka_1 | KAFKA_MAX_REQUEST_SIZE=50000000
kafka_1 | KAFKA_MESSAGE_MAX_BYTES=50000000
kafka_1 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
kafka_1 | KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
kafka_1 | KAFKA_VERSION=
kafka_1 | KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
kafka_1 | LANG=C.UTF-8
kafka_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kafka_1 | PWD=/
kafka_1 | PYTHON_PIP_VERSION=8.1.2
kafka_1 | PYTHON_VERSION=2.7.9-1
kafka_1 | SCALA_VERSION=2.12
kafka_1 | SHLVL=1
kafka_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
kafka_1 | _=/usr/bin/env
kafka_1 | ===> User
kafka_1 | uid=0(root) gid=0(root) groups=0(root)
kafka_1 | ===> Configuring ...
kafka_1 | ===> Running preflight checks ...
kafka_1 | ===> Check if /var/lib/kafka/data is writable ...
kafka_1 | ===> Check if Zookeeper is healthy ...
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=9afb2726589e
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-1067-oem
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=467MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=7077MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=477MB
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1 | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1 | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.22.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.22.0.10:50736, server: zookeeper/172.22.0.2:2181
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.22.0.2:2181, sessionid = 0x10000ddcd420000, negotiated timeout = 40000
kafka_1 | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000ddcd420000 closed
kafka_1 | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000ddcd420000
kafka_1 | ===> Launching ...
kafka_1 | ===> Launching kafka ...
kafka_1 | [2020-05-25 13:19:22,250] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
kafka_1 | [2020-05-25 13:19:22,724] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
kafka_1 | [2020-05-25 13:19:22,724] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
kafka_1 | [2020-05-25 13:19:23,597] INFO Starting the log cleaner (kafka.log.LogCleaner)
kafka_1 | [2020-05-25 13:19:23,642] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
kafka_1 | [2020-05-25 13:19:23,913] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
kafka_1 | [2020-05-25 13:19:23,940] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:19:23,941] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:19:24,005] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-25 13:19:24,025] INFO Stat of the created znode at /brokers/ids/1001 is: 195,195,1590412764019,1590412764019,1,0,0,72058546669355009,180,0,195
kafka_1 | (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-25 13:19:24,025] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 195 (kafka.zk.KafkaZkClient)
kafka_1 | [2020-05-25 13:19:24,196] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1 | [2020-05-25 13:19:24,240] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
kafka_1 | [2020-05-25 13:19:32,547] INFO Creating topic ingest-transactions with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:19:32,570] INFO Creating topic ingest-events with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
kafka_1 | [2020-05-25 13:19:32,589] INFO Creating topic ingest-attachments with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)
redis_1 | 1:C 25 May 2020 13:16:36.928 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 25 May 2020 13:16:36.928 # Redis version=5.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 25 May 2020 13:16:36.928 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 25 May 2020 13:16:36.932 # Not listening to IPv6: unsupproted
redis_1 | 1:M 25 May 2020 13:16:36.932 * Running mode=standalone, port=6379.
redis_1 | 1:M 25 May 2020 13:16:36.932 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 25 May 2020 13:16:36.932 # Server initialized
redis_1 | 1:M 25 May 2020 13:16:36.932 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 25 May 2020 13:16:36.932 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 25 May 2020 13:16:36.933 * Ready to accept connections
redis_1 | 1:signal-handler (1590412744) Received SIGTERM scheduling shutdown...
redis_1 | 1:M 25 May 2020 13:19:04.481 # User requested shutdown...
redis_1 | 1:M 25 May 2020 13:19:04.481 * Saving the final RDB snapshot before exiting.
redis_1 | 1:M 25 May 2020 13:19:04.486 * DB saved on disk
redis_1 | 1:M 25 May 2020 13:19:04.486 # Redis is now ready to exit, bye bye...
redis_1 | 1:C 25 May 2020 13:19:20.374 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 25 May 2020 13:19:20.374 # Redis version=5.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 25 May 2020 13:19:20.374 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 25 May 2020 13:19:20.378 # Not listening to IPv6: unsupproted
redis_1 | 1:M 25 May 2020 13:19:20.378 * Running mode=standalone, port=6379.
redis_1 | 1:M 25 May 2020 13:19:20.378 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 25 May 2020 13:19:20.378 # Server initialized
redis_1 | 1:M 25 May 2020 13:19:20.378 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 25 May 2020 13:19:20.378 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 25 May 2020 13:19:20.378 * DB loaded from disk: 0.000 seconds
redis_1 | 1:M 25 May 2020 13:19:20.378 * Ready to accept connections
zookeeper_1 | ===> ENV Variables ...
zookeeper_1 | ALLOW_UNSIGNED=false
zookeeper_1 | COMPONENT=zookeeper
zookeeper_1 | CONFLUENT_DEB_VERSION=1
zookeeper_1 | CONFLUENT_PLATFORM_LABEL=
zookeeper_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1 | CONFLUENT_VERSION=5.5.0
zookeeper_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1 | HOME=/root
zookeeper_1 | HOSTNAME=3f4f0ede6649
zookeeper_1 | KAFKA_VERSION=
zookeeper_1 | LANG=C.UTF-8
zookeeper_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1 | PWD=/
zookeeper_1 | PYTHON_PIP_VERSION=8.1.2
zookeeper_1 | PYTHON_VERSION=2.7.9-1
zookeeper_1 | SCALA_VERSION=2.12
zookeeper_1 | SHLVL=1
zookeeper_1 | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1 | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1 | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1 | _=/usr/bin/env
zookeeper_1 | ===> User
zookeeper_1 | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1 | ===> Configuring ...
zookeeper_1 | ===> Running preflight checks ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1 | ===> Launching ...
zookeeper_1 | ===> Launching zookeeper ...
zookeeper_1 | [2020-05-25 13:16:42,036] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1 | [2020-05-25 13:16:42,162] WARN o.e.j.s.ServletContextHandler@167fdd33{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1 | [2020-05-25 13:16:42,162] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1 | ===> ENV Variables ...
zookeeper_1 | ALLOW_UNSIGNED=false
zookeeper_1 | COMPONENT=zookeeper
zookeeper_1 | CONFLUENT_DEB_VERSION=1
zookeeper_1 | CONFLUENT_PLATFORM_LABEL=
zookeeper_1 | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1 | CONFLUENT_VERSION=5.5.0
zookeeper_1 | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1 | HOME=/root
zookeeper_1 | HOSTNAME=3f4f0ede6649
zookeeper_1 | KAFKA_VERSION=
zookeeper_1 | LANG=C.UTF-8
zookeeper_1 | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1 | PWD=/
zookeeper_1 | PYTHON_PIP_VERSION=8.1.2
zookeeper_1 | PYTHON_VERSION=2.7.9-1
zookeeper_1 | SCALA_VERSION=2.12
zookeeper_1 | SHLVL=1
zookeeper_1 | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1 | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1 | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1 | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1 | _=/usr/bin/env
zookeeper_1 | ===> User
zookeeper_1 | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1 | ===> Configuring ...
zookeeper_1 | ===> Running preflight checks ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1 | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1 | ===> Launching ...
zookeeper_1 | ===> Launching zookeeper ...
zookeeper_1 | [2020-05-25 13:19:16,912] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1 | [2020-05-25 13:19:17,006] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1 | [2020-05-25 13:19:17,006] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
same problem on a different laptop where i never worked with sentry before and changed nothing in the repository
marbon87
on 25 May 2020
@marbon87 you see this error but you _can_ connect to Sentry web interface, right?
Is this on Linux or MacOS?
BYK
on 25 May 2020
Yes, I can connect to the web interface but errors are not shown. It happens on Linux and macOS
marbon87
on 25 May 2020
Could you reproduce the error?
marbon87
on 4 Jun 2020
Filtered events are shown now. Seems to be workng after a few hours / day of running sentry, but rate limited events are still not counted.
Does it have something to do with the relay service because relay is responding to rate limited events now, isn't it?
marbon87
on 2 Jul 2020
Not really sure. @jan-auer @untitaker any ideas?
BYK
on 2 Jul 2020
Hey @marbon87 I think we have finally found the bug. PR here: https://github.com/getsentry/sentry/pull/19760
untitaker
on 8 Jul 2020
🎉1
👍1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
6qiongtao
路
4Comments
TheRatG
路
4Comments
nature1995
路
4Comments
adrielliu
路
3Comments
rmisyurev
路
4Comments
Most helpful comment
Hey @marbon87 I think we have finally found the bug. PR here: https://github.com/getsentry/sentry/pull/19760