Materialize: Create Confluent Cloud Kafka source Documentation

[mathieuisabel]

I'm trying to create a Kafka source based on Confluent Cloud but I can't seem to find documentation as to how that would need to be setup. I suspect I would need to pass authentication information while setting up the source but I'm not sure how to do it.

Thank you!

[benesch]

Hi @mathieuisabel, sorry about the delay in response here! I believe Confluent Cloud uses SASL authentication, which non of our stable releases currently support. We're actively working on this feature, though—the big PR, https://github.com/MaterializeInc/materialize/pull/2586, just landed—and it will be part of the v0.3 release.

[rjnn]

Once we land SASL in 0.2.1 this should be a relatively easy product win, albeit one that requires some amount of manual testing and documentation work.

[rjnn]

@awang this is probably a useful documentation issue to address since SASL landed. Probably also useful usability testing of SASL that we can do in-house.

[dferber90]

Is there a summary somewhere I could use as a pointer while we excitedly await the official docs?

I'm trying to connect Materialize to Confluent Cloud with a Confluent Schema Registry hosted on Confluent Cloud as well.

This is the equivalent docker-compose.yml I'm using for ksqlDB in case that's of any help to clarify my setup

# docker-compose supports environment variable substitution with the ${VARIABLE-NAME} syntax.
# Environment variables can be sourced in a variety of ways.  One of those ways is through
# a well known '.env' file located in the same folder as the docker-compose.yml file.  See the Docker
# documentation for details: https://docs.docker.com/compose/environment-variables/#the-env-file
#
# This feature is being used to parameterize some values within this file.  In this directory is also
# a .env file, which is actually a symbolic link to <examples-root>/utils/config.env.  That file
# contains values which get substituted here when docker-compose parses this file.
#
# If you'd like to view the docker-compose.yml file rendered with its environment variable substituions
# you can execute the `docker-compose config` command.  Take note that some demos provide additional
# environment variable values by exporting them in a script prior to running `docker-compose up`.
---
version: "2"

services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.9.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: xxxxxx.us-west1.gcp.confluent.cloud:9092
      KSQL_SASL_JAAS_CONFIG: >
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="${CLUSTER_API_KEY}"
        password="${CLUSTER_SECRET}";
      KSQL_SECURITY_PROTOCOL: SASL_SSL
      KSQL_SASL_MECHANISM: PLAIN
      KSQL_KSQL_INTERNAL_TOPIC_REPLICAS: 3
      KSQL_KSQL_SINK_REPLICAS: 3
      KSQL_KSQL_STREAMS_REPLICATION_FACTOR: 3
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 3
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      # added by me
      # see https://docs.ksqldb.io/en/latest/developer-guide/serialization/#avro
      KSQL_KSQL_SCHEMA_REGISTRY_URL: https://yyyyyy.us-central1.gcp.confluent.cloud
      #  https://docs.ksqldb.io/en/latest/operate-and-deploy/installation/server-config/security/#configure-ksqldb-for-secured-confluent-schema-registry
      KSQL_KSQL_SCHEMA_REGISTRY_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
      KSQL_KSQL_SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO: ${SCHEMA_REGISTRY_API_KEY}:${SCHEMA_REGISTRY_SECRET}

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.9.0
    container_name: ksqldb-cli
    depends_on:
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

[rjnn]

@dferber90 looks like we need one more little thing before we support Confluent Cloud; I've filed ##3418 to track the work there. Thanks for prompting me to check!

[awang]

Hi @mathieuisabel thanks for your patience! I believe this should now work for you, as we just added SASL support!