Ksql: How to define sink or source when using embeded kafka connect in headless mode

[psporysz]

I've tried to use KSQL_KSQL_QUERIES_FILE to declare my source definition:

CREATE SOURCE CONNECTOR fake-company WITH (
  'connector.class' = 'io.mdrogalis.voluble.VolubleSourceConnector',

  'genkp.company.sometimes.with' = '#{Internet.uuid}',
  'genkp.company.sometimes.matching' = 'company.key',
  'genv.company.id.with' = '#{Internet.uuid}',
  'genv.company.name.with' = '#{Company.name}',

  'global.throttle.ms' = '500'
);

unfortunately this kind of query works only in interactive mode:

ksqldb-server      | io.confluent.ksql.util.KsqlStatementException: Unsupported statement. Only the following statements are supporting in standalone mode:

Headless mode also disables REST endpoint for embeded kafka connect thus there is no way to define it like for external kafka connect instance. My task is quite simple: source data from fake topic generator, transform and then sink to jdbc sink but it looks like headless mode forces me to provision external kafka connect cluster. Is that correct?

[agavra]

At the moment we haven't added support for connect in headless mode. We may reconsider sometime in the future, but at the moment i think the best workaround would be to spin up an independent connect instance and interact with that one, pulling in streams/tables as desired in your ksql instance.

[psporysz]

That's what i finally did. I think the KSQLDB documentation (especially tutorials) could warn the reader about the KSQL syntax which will not work in headless mode, it would save me some wasted time ;)