Ksql: Bug: Internal topic replication factor clashes with the policy.

[yarncraft]

Describe the bug
The error seems related to https://github.com/confluentinc/ksql/issues/815.

To Reproduce
Steps to reproduce the behavior, include:

1. The version of KSQL.
   Docker image: confluentinc/ksqldb-server:0.6.0
2. Sample source data.
   Confluent Cloud

Expected behavior
The server should start up properly.
(I was following the quickstart https://ksqldb.io/quickstart.html)

Actual behaviour
The server shuts down due to a policy violation:

1. CLI output
   See logs below

2. Error messages
   Caused by: org.apache.kafka.common.errors.PolicyViolationException: Topic replication factor must be 3
   ksqldb-server exited with code 255

3. KSQL logs
   bash ksqldb-server | [2019-11-21 13:36:15,983] INFO KsqlConfig values: ksqldb-server | ksql.access.validator.enable = auto ksqldb-server | ksql.avro.maps.named = true ksqldb-server | ksql.connect.url = http://localhost:8083 ksqldb-server | ksql.connect.worker.config = ksqldb-server | ksql.extension.dir = ext ksqldb-server | ksql.functions.substring.legacy.args = false ksqldb-server | ksql.insert.into.values.enabled = true ksqldb-server | ksql.internal.topic.replicas = 1 ksqldb-server | ksql.metric.reporters = [] ksqldb-server | ksql.metrics.extension = null ksqldb-server | ksql.metrics.tags.custom = ksqldb-server | ksql.named.internal.topics = on ksqldb-server | ksql.output.topic.name.prefix = ksqldb-server | ksql.persistence.wrap.single.values = true ksqldb-server | ksql.persistent.prefix = query_ ksqldb-server | ksql.query.inject.legacy.map.values.node = false ksqldb-server | ksql.query.persistent.active.limit = 2147483647 ksqldb-server | ksql.query.pull.enable = true ksqldb-server | ksql.query.pull.routing.timeout.ms = 30000 ksqldb-server | ksql.query.pull.skip.access.validator = false ksqldb-server | ksql.query.pull.streamsstore.rebalancing.timeout.ms = 10000 ksqldb-server | ksql.query.stream.groupby.rowkey.repartition = false ksqldb-server | ksql.schema.registry.url = http://localhost:8081 ksqldb-server | ksql.security.extension.class = null ksqldb-server | ksql.service.id = default_ ksqldb-server | ksql.sink.partitions = null ksqldb-server | ksql.sink.replicas = null ksqldb-server | ksql.sink.window.change.log.additional.retention = 1000000 ksqldb-server | ksql.transient.prefix = transient_ ksqldb-server | ksql.udf.collect.metrics = false ksqldb-server | ksql.udf.enable.security.manager = true ksqldb-server | ksql.udfs.enabled = true ksqldb-server | ksql.windowed.session.key.legacy = false ksqldb-server | ssl.cipher.suites = null ksqldb-server | ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ksqldb-server | ssl.endpoint.identification.algorithm = https ksqldb-server | ssl.key.password = null ksqldb-server | ssl.keymanager.algorithm = SunX509 ksqldb-server | ssl.keystore.location = null ksqldb-server | ssl.keystore.password = null ksqldb-server | ssl.keystore.type = JKS ksqldb-server | ssl.protocol = TLS ksqldb-server | ssl.provider = null ksqldb-server | ssl.secure.random.implementation = null ksqldb-server | ssl.trustmanager.algorithm = PKIX ksqldb-server | ssl.truststore.location = null ksqldb-server | ssl.truststore.password = null ksqldb-server | ssl.truststore.type = JKS ksqldb-server | (io.confluent.ksql.util.KsqlConfig:347) ksqldb-server | [2019-11-21 13:36:16,196] INFO ProcessingLogConfig values: ksqldb-server | ksql.logging.processing.rows.include = false ksqldb-server | ksql.logging.processing.stream.auto.create = false ksqldb-server | ksql.logging.processing.stream.name = KSQL_PROCESSING_LOG ksqldb-server | ksql.logging.processing.topic.auto.create = false ksqldb-server | ksql.logging.processing.topic.name = ksqldb-server | ksql.logging.processing.topic.partitions = 1 ksqldb-server | ksql.logging.processing.topic.replication.factor = 1 ksqldb-server | (io.confluent.ksql.logging.processing.ProcessingLogConfig:347) ksqldb-server | [2019-11-21 13:36:16,384] INFO Creating topic 'default_ksql_processing_log' (io.confluent.ksql.services.KafkaTopicClient:108) ksqldb-server | [2019-11-21 13:36:16,673] INFO Server shutting down (io.confluent.ksql.rest.server.KsqlServerMain:79) ksqldb-server | [2019-11-21 13:36:31,751] INFO Stopped NetworkTrafficServerConnector@6b19b79{HTTP/1.1,[http/1.1]}{0.0.0.0:8088} (org.eclipse.jetty.server.AbstractConnector:343) ksqldb-server | [2019-11-21 13:36:31,756] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:158) ksqldb-server | [2019-11-21 13:36:31,762] INFO Stopped o.e.j.s.ServletContextHandler@28c0b664{/ws,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1007) ksqldb-server | [2019-11-21 13:36:31,784] INFO Stopped o.e.j.s.ServletContextHandler@2c444798{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1007) ksqldb-server | [2019-11-21 13:36:46,773] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:63) ksqldb-server | io.confluent.ksql.exception.KafkaResponseGetFailedException: Failed to guarantee existence of topic default_ksql_processing_log ksqldb-server | at io.confluent.ksql.services.KafkaTopicClientImpl.createTopic(KafkaTopicClientImpl.java:135) ksqldb-server | at io.confluent.ksql.services.KafkaTopicClient.createTopic(KafkaTopicClient.java:92) ksqldb-server | at io.confluent.ksql.services.KafkaTopicClient.createTopic(KafkaTopicClient.java:70) ksqldb-server | at io.confluent.ksql.rest.util.ProcessingLogServerUtils.maybeCreateProcessingLogTopic(ProcessingLogServerUtils.java:83) ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.initialize(KsqlRestApplication.java:270) ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.startKsql(KsqlRestApplication.java:227) ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.startAsync(KsqlRestApplication.java:214) ksqldb-server | at io.confluent.ksql.rest.server.ExecutableServer.startAsync(ExecutableServer.java:44) ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.tryStartApp(KsqlServerMain.java:75) ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:61) ksqldb-server | Caused by: org.apache.kafka.common.errors.PolicyViolationException: Topic replication factor must be 3 ksqldb-server exited with code 255

Additional context
Included environment variables:

• KSQL_BOOTSTRAP_SERVERS: "confluentcloud-endpoint"
• KSQL_SASL_JAAS_CONFIG: "confluentcloud-sasl-config"
• KSQL_SECURITY_PROTOCOL: "SASL_SSL"
• KSQL_SASL_MECHANISM: "PLAIN"

[agavra]

Hello @yarncraft - I don't think this is a bug with KSQL. If you are using Confluent Cloud, the broker is configured to prevent any topic with a replication factor of less than 3. Since most developers are deploying a broker locally using the quickstart, they will not have three brokers so the configuration is a good default for them but not for you. Instead, try deploying it with the following configuration for the ksqldb server in your docker compose:

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.6.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker:9092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_KSQL_INTERNAL_TOPIC_REPLICAS: 3

(note the addition of KSQL_KSQL_INTERNAL_TOPIC_REPLICAS at the bottom)

[yarncraft]

Even with the addition of the last rule I still get:

` ksqldb-server | [2019-11-21 16:13:23,732] INFO KafkaJsonDeserializerConfig values: ksqldb-server | json.fail.unknown.properties = true ksqldb-server | json.key.type = class io.confluent.ksql.rest.entity.CommandId ksqldb-server | json.value.type = class java.lang.Object ksqldb-server | (io.confluent.kafka.serializers.KafkaJsonDeserializerConfig:347) ksqldb-server | [2019-11-21 16:13:23,734] INFO KafkaJsonDeserializerConfig values: ksqldb-server | json.fail.unknown.properties = true ksqldb-server | json.key.type = class java.lang.Object ksqldb-server | json.value.type = class io.confluent.ksql.rest.server.computation.Command ksqldb-server | (io.confluent.kafka.serializers.KafkaJsonDeserializerConfig:347) ksqldb-server | [2019-11-21 16:13:25,611] INFO KafkaJsonSerializerConfig values: ksqldb-server | json.indent.output = false ksqldb-server | (io.confluent.kafka.serializers.KafkaJsonSerializerConfig:347) ksqldb-server | [2019-11-21 16:13:25,611] INFO KafkaJsonSerializerConfig values: ksqldb-server | json.indent.output = false ksqldb-server | (io.confluent.kafka.serializers.KafkaJsonSerializerConfig:347) ksqldb-server | [2019-11-21 16:13:30,931] INFO KsqlRestConfig values: ksqldb-server | access.control.allow.headers = ksqldb-server | access.control.allow.methods = ksqldb-server | access.control.allow.origin = ksqldb-server | authentication.method = NONE ksqldb-server | authentication.realm = ksqldb-server | authentication.roles = [*] ksqldb-server | authentication.skip.paths = [/v1/metadata, /v1/metadata/id, /healthcheck] ksqldb-server | compression.enable = true ksqldb-server | debug = false ksqldb-server | idle.timeout.ms = 30000 ksqldb-server | ksql.healthcheck.interval.ms = 5000 ksqldb-server | ksql.server.command.response.timeout.ms = 5000 ksqldb-server | ksql.server.exception.uncaught.handler.enable = false ksqldb-server | ksql.server.install.dir = /usr ksqldb-server | ksql.server.preconditions = [] ksqldb-server | ksql.server.websockets.num.threads = 5 ksqldb-server | listeners = [http://0.0.0.0:8088] ksqldb-server | metric.reporters = [] ksqldb-server | metrics.jmx.prefix = rest-utils ksqldb-server | metrics.num.samples = 2 ksqldb-server | metrics.sample.window.ms = 30000 ksqldb-server | metrics.tag.map = [] ksqldb-server | port = 8080 ksqldb-server | query.stream.disconnect.check = 1000 ksqldb-server | request.logger.name = io.confluent.rest-utils.requests ksqldb-server | resource.extension.classes = [] ksqldb-server | response.mediatype.default = application/json ksqldb-server | response.mediatype.preferred = [application/json] ksqldb-server | rest.servlet.initializor.classes = [] ksqldb-server | shutdown.graceful.ms = 1000 ksqldb-server | ssl.cipher.suites = [] ksqldb-server | ssl.client.auth = false ksqldb-server | ssl.client.authentication = NONE ksqldb-server | ssl.enabled.protocols = [] ksqldb-server | ssl.endpoint.identification.algorithm = null ksqldb-server | ssl.key.password = [hidden] ksqldb-server | ssl.keymanager.algorithm = ksqldb-server | ssl.keystore.location = ksqldb-server | ssl.keystore.password = [hidden] ksqldb-server | ssl.keystore.type = JKS ksqldb-server | ssl.protocol = TLS ksqldb-server | ssl.provider = ksqldb-server | ssl.trustmanager.algorithm = ksqldb-server | ssl.truststore.location = ksqldb-server | ssl.truststore.password = [hidden] ksqldb-server | ssl.truststore.type = JKS ksqldb-server | websocket.path.prefix = /ws ksqldb-server | websocket.servlet.initializor.classes = [] ksqldb-server | (io.confluent.ksql.rest.server.KsqlRestConfig:347) ksqldb-server | [2019-11-21 16:13:31,004] INFO Starting server (io.confluent.ksql.rest.server.KsqlServerMain:74) ksqldb-server | [2019-11-21 16:13:31,029] INFO jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_222-b10 (org.eclipse.jetty.server.Server:359) ksqldb-server | [2019-11-21 16:13:31,349] INFO Started NetworkTrafficServerConnector@6b19b79{HTTP/1.1,[http/1.1]}{0.0.0.0:8088} (org.eclipse.jetty.server.AbstractConnector:293) ksqldb-server | [2019-11-21 16:13:31,350] INFO Started @17303ms (org.eclipse.jetty.server.Server:399) ksqldb-server | [2019-11-21 16:13:34,483] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session:333) ksqldb-server | [2019-11-21 16:13:34,484] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session:338) ksqldb-server | [2019-11-21 16:13:34,509] INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session:140) ksqldb-server | Nov 21, 2019 4:13:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime ksqldb-server | WARNING: A provider io.confluent.ksql.rest.server.resources.StatusResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.ksql.rest.server.resources.StatusResource will be ignored. ksqldb-server | Nov 21, 2019 4:13:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime ksqldb-server | WARNING: A provider io.confluent.ksql.rest.server.resources.streaming.StreamedQueryResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.ksql.rest.server.resources.streaming.StreamedQueryResource will be ignored. ksqldb-server | Nov 21, 2019 4:13:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime ksqldb-server | WARNING: A provider io.confluent.ksql.rest.server.resources.ServerMetadataResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.ksql.rest.server.resources.ServerMetadataResource will be ignored. ksqldb-server | Nov 21, 2019 4:13:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime ksqldb-server | WARNING: A provider io.confluent.ksql.rest.server.resources.RootDocument registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.ksql.rest.server.resources.RootDocument will be ignored. ksqldb-server | Nov 21, 2019 4:13:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime ksqldb-server | WARNING: A provider io.confluent.ksql.rest.server.resources.ServerInfoResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.ksql.rest.server.resources.ServerInfoResource will be ignored. ksqldb-server | Nov 21, 2019 4:13:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime ksqldb-server | WARNING: A provider io.confluent.ksql.rest.server.resources.KsqlResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.ksql.rest.server.resources.KsqlResource will be ignored. ksqldb-server | Nov 21, 2019 4:13:36 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime ksqldb-server | WARNING: A provider io.confluent.ksql.rest.server.resources.HealthCheckResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.ksql.rest.server.resources.HealthCheckResource will be ignored. ksqldb-server | [2019-11-21 16:13:36,651] INFO HV000001: Hibernate Validator 6.0.11.Final (org.hibernate.validator.internal.util.Version:21) ksqldb-server | [2019-11-21 16:13:37,504] INFO JVM Runtime does not support Modules (org.eclipse.jetty.util.TypeUtil:201) ksqldb-server | [2019-11-21 16:13:37,509] INFO Started o.e.j.s.ServletContextHandler@2c444798{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:824) ksqldb-server | [2019-11-21 16:13:37,766] INFO Started o.e.j.s.ServletContextHandler@28c0b664{/ws,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:824) ksqldb-server | [2019-11-21 16:13:37,817] INFO KSQL RESTful API listening on http://localhost:8088 (io.confluent.ksql.rest.server.KsqlRestApplication:211) ksqldb-server | [2019-11-21 16:13:37,818] INFO Using first listener URL for intra-node communication: http://localhost:8088 (io.confluent.ksql.rest.server.KsqlRestApplication:664) ksqldb-server | [2019-11-21 16:13:37,819] INFO KsqlConfig values: ksqldb-server | ksql.access.validator.enable = auto ksqldb-server | ksql.avro.maps.named = true ksqldb-server | ksql.connect.url = http://localhost:8083 ksqldb-server | ksql.connect.worker.config = ksqldb-server | ksql.extension.dir = ext ksqldb-server | ksql.functions.substring.legacy.args = false ksqldb-server | ksql.insert.into.values.enabled = true ksqldb-server | ksql.internal.topic.replicas = 3 ksqldb-server | ksql.metric.reporters = [] ksqldb-server | ksql.metrics.extension = null ksqldb-server | ksql.metrics.tags.custom = ksqldb-server | ksql.named.internal.topics = on ksqldb-server | ksql.output.topic.name.prefix = ksqldb-server | ksql.persistence.wrap.single.values = true ksqldb-server | ksql.persistent.prefix = query_ ksqldb-server | ksql.query.inject.legacy.map.values.node = false ksqldb-server | ksql.query.persistent.active.limit = 2147483647 ksqldb-server | ksql.query.pull.enable = true ksqldb-server | ksql.query.pull.routing.timeout.ms = 30000 ksqldb-server | ksql.query.pull.skip.access.validator = false ksqldb-server | ksql.query.pull.streamsstore.rebalancing.timeout.ms = 10000 ksqldb-server | ksql.query.stream.groupby.rowkey.repartition = false ksqldb-server | ksql.schema.registry.url = http://localhost:8081 ksqldb-server | ksql.security.extension.class = null ksqldb-server | ksql.service.id = default_ ksqldb-server | ksql.sink.partitions = null ksqldb-server | ksql.sink.replicas = null ksqldb-server | ksql.sink.window.change.log.additional.retention = 1000000 ksqldb-server | ksql.transient.prefix = transient_ ksqldb-server | ksql.udf.collect.metrics = false ksqldb-server | ksql.udf.enable.security.manager = true ksqldb-server | ksql.udfs.enabled = true ksqldb-server | ksql.windowed.session.key.legacy = false ksqldb-server | ssl.cipher.suites = null ksqldb-server | ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ksqldb-server | ssl.endpoint.identification.algorithm = https ksqldb-server | ssl.key.password = null ksqldb-server | ssl.keymanager.algorithm = SunX509 ksqldb-server | ssl.keystore.location = null ksqldb-server | ssl.keystore.password = null ksqldb-server | ssl.keystore.type = JKS ksqldb-server | ssl.protocol = TLS ksqldb-server | ssl.provider = null ksqldb-server | ssl.secure.random.implementation = null ksqldb-server | ssl.trustmanager.algorithm = PKIX ksqldb-server | ssl.truststore.location = null ksqldb-server | ssl.truststore.password = null ksqldb-server | ssl.truststore.type = JKS ksqldb-server | (io.confluent.ksql.util.KsqlConfig:347) ksqldb-server | [2019-11-21 16:13:38,508] INFO Corrected retention.ms on ksql internal topic. topic:_confluent-ksql-default__command_topic, retention.ms:9223372036854775807 (io.confluent.ksql.rest.util.KsqlInternalTopicUtils:99) ksqldb-server | [2019-11-21 16:13:38,892] INFO KsqlConfig values: ksqldb-server | ksql.access.validator.enable = auto ksqldb-server | ksql.avro.maps.named = true ksqldb-server | ksql.connect.url = http://localhost:8083 ksqldb-server | ksql.connect.worker.config = ksqldb-server | ksql.extension.dir = ext ksqldb-server | ksql.functions.substring.legacy.args = false ksqldb-server | ksql.insert.into.values.enabled = true ksqldb-server | ksql.internal.topic.replicas = 3 ksqldb-server | ksql.metric.reporters = [] ksqldb-server | ksql.metrics.extension = null ksqldb-server | ksql.metrics.tags.custom = ksqldb-server | ksql.named.internal.topics = on ksqldb-server | ksql.output.topic.name.prefix = ksqldb-server | ksql.persistence.wrap.single.values = true ksqldb-server | ksql.persistent.prefix = query_ ksqldb-server | ksql.query.inject.legacy.map.values.node = false ksqldb-server | ksql.query.persistent.active.limit = 2147483647 ksqldb-server | ksql.query.pull.enable = true ksqldb-server | ksql.query.pull.routing.timeout.ms = 30000 ksqldb-server | ksql.query.pull.skip.access.validator = false ksqldb-server | ksql.query.pull.streamsstore.rebalancing.timeout.ms = 10000 ksqldb-server | ksql.query.stream.groupby.rowkey.repartition = false ksqldb-server | ksql.schema.registry.url = http://localhost:8081 ksqldb-server | ksql.security.extension.class = null ksqldb-server | ksql.service.id = default_ ksqldb-server | ksql.sink.partitions = null ksqldb-server | ksql.sink.replicas = null ksqldb-server | ksql.sink.window.change.log.additional.retention = 1000000 ksqldb-server | ksql.transient.prefix = transient_ ksqldb-server | ksql.udf.collect.metrics = false ksqldb-server | ksql.udf.enable.security.manager = true ksqldb-server | ksql.udfs.enabled = true ksqldb-server | ksql.windowed.session.key.legacy = false ksqldb-server | ssl.cipher.suites = null ksqldb-server | ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ksqldb-server | ssl.endpoint.identification.algorithm = https ksqldb-server | ssl.key.password = null ksqldb-server | ssl.keymanager.algorithm = SunX509 ksqldb-server | ssl.keystore.location = null ksqldb-server | ssl.keystore.password = null ksqldb-server | ssl.keystore.type = JKS ksqldb-server | ssl.protocol = TLS ksqldb-server | ssl.provider = null ksqldb-server | ssl.secure.random.implementation = null ksqldb-server | ssl.trustmanager.algorithm = PKIX ksqldb-server | ssl.truststore.location = null ksqldb-server | ssl.truststore.password = null ksqldb-server | ssl.truststore.type = JKS ksqldb-server | (io.confluent.ksql.util.KsqlConfig:347) ksqldb-server | [2019-11-21 16:13:39,190] INFO ProcessingLogConfig values: ksqldb-server | ksql.logging.processing.rows.include = false ksqldb-server | ksql.logging.processing.stream.auto.create = false ksqldb-server | ksql.logging.processing.stream.name = KSQL_PROCESSING_LOG ksqldb-server | ksql.logging.processing.topic.auto.create = false ksqldb-server | ksql.logging.processing.topic.name = ksqldb-server | ksql.logging.processing.topic.partitions = 1 ksqldb-server | ksql.logging.processing.topic.replication.factor = 1 ksqldb-server | (io.confluent.ksql.logging.processing.ProcessingLogConfig:347) ksqldb-server | [2019-11-21 16:13:39,529] INFO Creating topic 'default_ksql_processing_log' (io.confluent.ksql.services.KafkaTopicClient:108) ksqldb-server | [2019-11-21 16:13:39,886] INFO Server shutting down (io.confluent.ksql.rest.server.KsqlServerMain:79) ksqldb-server | [2019-11-21 16:13:55,115] INFO Stopped NetworkTrafficServerConnector@6b19b79{HTTP/1.1,[http/1.1]}{0.0.0.0:8088} (org.eclipse.jetty.server.AbstractConnector:343) ksqldb-server | [2019-11-21 16:13:55,119] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:158) ksqldb-server | [2019-11-21 16:13:55,378] INFO Stopped o.e.j.s.ServletContextHandler@28c0b664{/ws,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1007) ksqldb-server | [2019-11-21 16:13:55,429] INFO Stopped o.e.j.s.ServletContextHandler@2c444798{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1007) ksqldb-server | [2019-11-21 16:14:10,484] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:63) ksqldb-server | io.confluent.ksql.exception.KafkaResponseGetFailedException: Failed to guarantee existence of topic default_ksql_processing_log ksqldb-server | at io.confluent.ksql.services.KafkaTopicClientImpl.createTopic(KafkaTopicClientImpl.java:135) ksqldb-server | at io.confluent.ksql.services.KafkaTopicClient.createTopic(KafkaTopicClient.java:92) ksqldb-server | at io.confluent.ksql.services.KafkaTopicClient.createTopic(KafkaTopicClient.java:70) ksqldb-server | at io.confluent.ksql.rest.util.ProcessingLogServerUtils.maybeCreateProcessingLogTopic(ProcessingLogServerUtils.java:83) ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.initialize(KsqlRestApplication.java:270) ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.startKsql(KsqlRestApplication.java:227) ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.startAsync(KsqlRestApplication.java:214) ksqldb-server | at io.confluent.ksql.rest.server.ExecutableServer.startAsync(ExecutableServer.java:44) ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.tryStartApp(KsqlServerMain.java:75) ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:61) ksqldb-server | Caused by: org.apache.kafka.common.errors.PolicyViolationException: Topic replication factor must be 3 ksqldb-server exited with code 255

[agavra]

FYI here is some documentation about how to connect ksqlDB to Confluent Cloud: https://docs.confluent.io/current/cloud/connect/ksql-cloud-config.html

To configure anything through the docker compose file, you need to prepend the config with KSQL_ and convert all . to _ (following the pattern in the compose file).

For the issue with the processing log topic, you need the ksql.logging.processing.topic.replication.factor=3 config to be set (KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR)

[agavra]

Also we'd love to learn more about what you're trying to accomplish with ksqlDB - feel free to join the community (https://ksqldb.io/community.html has links to both slack and google groups)!

[yarncraft]

Ok, it seems to be up and running now.
Thanks for the swift response, I already signed up for the Slack!