Cp-docker-images: How to set connect to auto create topics?

[sunshineo]

Like it said here http://docs.confluent.io/3.1.1/cp-docker-images/docs/quickstart.html?highlight=autocreation
And here https://github.com/confluentinc/cp-docker-images/blob/f432909b3323ff033338c6f4bec36b555f8d3caa/docs/quickstart.rst

It is possible to allow connect to auto-create these topics by enabling the autocreation setting. However ...

How could this compose works https://github.com/confluentinc/cp-docker-images/blob/f432909b3323ff033338c6f4bec36b555f8d3caa/examples/cp-all-in-one/docker-compose.yml ?
It said nothing about auto creation of topics, so where will

      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status

come from?

[samjhecht]

Hi @sunshineo - the broker actually creates the topics the first time you make a metadata request for it if they don't already exist. This is controlled by the auto.create.topics.enable property on the broker. (more info on broker configs here: https://kafka.apache.org/documentation/#brokerconfigs). In summary, if auto.create.topics.enable is set to "true", you should be able to use this docker-compose example with auto creation of topics. However, this is not recommended for production because Connect has specific requirements for the number of partitions (in some cases it should only be 1).

[sunshineo]

Hi @samjhecht , I am a n00b. Could you help me clarify some basics? Is 'Kafka Connect' a framework on top of 'Kafka' or does it come with 'Kafka' but one may not use it at all? Or is it owned by Confluentic? If we are going to use 'Kafka Connect', for example I plan to use jdbc-connector, which part requires topics in Kafka exactly? Is it the 'Kafka Connect' framework or the actual connectors? And why when I start the jdbc-connector, it seems to have the offset saved in a file? Is it because I'm using the stand-alone mode but if I use distributed works they will need to use Kafka topics to save offsets?
Thank you in advance!

[OneCricketeer]

@sunshineo Probably later than wanted, but Connect is a framework much like Kafka Streams, with no affiliation to Confluent Inc. Out of the box, Apache Kafka download only provides FileSource and FileSink Connects.
https://kafka.apache.org/documentation/#connect

The kafka-connect-jdbc connect project, on the other hand is just one Connect plugin out of many that implement the SourceTask and SinkTask interfaces provided by Connect. You can think of these as abstractions over Producer and Consumer (and in fact, that's what they really are). That JDBC connector is maintained by Confluent, though, you can find Debezium project, not by Confluent, and is also a Kafka Connect plugin that supports various database types.

Confluent also hosts the Connector Hub where you can see various Connector plugins that others have developed.

You need "the framework" to run the "connectors", much like you need kafka-clients to import Producer and Consumer, and you deploy your connectors, ideally as part of a Kafka Connect cluster, which is ideally installed independently from the brokers (for similar reasons that you are probably not going to be running producers and consumers directly on the brokers).

Distributed mode stores offsets in a topic, as compared to a file, yes.

Hopefully that answers your questions!

[sunshineo]

Thank you for the reply. I have changed job twice since I asked the question but thank you still.