Cp-docker-images: Auto-load connectors from directory

[OneCricketeer]

Related to confluentinc/cp-docker-images#460

Should add some directory in kafka-connect-base that loads .json or .properties files on start.

See MySQL container for inspiration

[OneCricketeer]

One alternative, as shown by @rmoff

In the compose, override the container command

  volumes:
    - $PWD/scripts:/scripts  # TODO: Create this folder ahead of time, on your host
  command: 
    - bash 
    - -c 
    - |
      /etc/confluent/docker/run & 
      echo "Waiting for Kafka Connect to start listening on kafka-connect ⏳"
      while [ $$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors) -eq 000 ] ; do 
        echo -e $$(date) " Kafka Connect listener HTTP state: " $$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors) " (waiting for 200)"
        sleep 5 
      done
      nc -vz kafka-connect 8083
      echo -e "\n--\n+> Creating Kafka Connector(s)"
      /scripts/create-connectors.sh  # Note: This script is stored externally from container
      sleep infinity

[nwinkler]

Thanks - this was helpful! I prefer a slightly optimized version:

 volumes:
    - $PWD/scripts:/scripts  # TODO: Create this folder ahead of time, on your host
  command: 
    - bash 
    - -c 
    - |
      /etc/confluent/docker/run & 
      echo "Waiting for Kafka Connect to start listening on kafka-connect ⏳"
      while : ; do
        curl_status=$$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors)
        echo -e $$(date) " Kafka Connect listener HTTP state: " $$curl_status " (waiting for 200)"
        if [ $$curl_status -eq 200 ] ; then
          break
        fi
        sleep 5 
      done
      echo -e "\n--\n+> Creating Kafka Connector(s)"
      /scripts/create-connectors.sh  # Note: This script is stored externally from container
      sleep infinity

Changes over the above version:

• Only run curl once per iteration - no need to call it a second time just for printing the status. Also only one place to change the hostname and port.
• Check for status 200 instead of a status different than 000. I've had Connect return a 404 status during startup (since the HTTP port was up, but the endpoint was not deployed yet), in which case the create-connectors.sh script failed. Waiting for 200 ensures that the connectors endpoint is available.
• Removed the netcat call - not sure what that was needed for. Looks like a leftover from a previous version of the script...

FWIW, I run this from a separate service in my Docker Compose file - the image I use is appropriate/curl:latest. That way, you don't have to start the run command in the background...

[rmoff]

@nwinkler nice tips, thanks for sharing!