Ksql: Why retention.ms on command topic has to be infinite[Long.MAX_VALUE]

[ericqqqqq]

I am trying to use ksql to connect to a kafka cluster from a cloud provider and the cloud provider has a 30 day retention limit on the topic. I tried to set retention.ms to 1 hour then it fails. Turns out I found the following code might be the reason:

file[5.1.1-post]
file[5.2.x]

The retention.ms is forced to be set in Long.MAX_VALUE. Is there any specific reason for this?

[ericqqqqq]

would this also affect the retention policy for the commands_topic

[agavra]

Hello @ericqqqqq - you can think of the command topic as a list of commands that your KSQL application should run. If the retention on this was anything less than infinite, a crashed KSQL server would never be able to recover!

Imagine the commands below:

-- submitted day 0
1: CREATE STREAM foo WITH (kafka_topic='foo' ...);
-- submitted day 3
2: CREATE STREAM bar AS SELECT * FROM foo;

If you restarted your KSQL applications after 30 days, then only the second command would run (the first is out of retention) and the query would fail.

As an alternative, you can deploy your KSQL application in "headless" mode by specifying a .sql file for it to read from instead of the command topic (the way the CLI interacts with it). If you do this, you will unfortunately not be able to use the CLI. To specify the sql file you can start your KSQL server using

./bin/ksql-server-start [/path/to/properties/file] --queries-file [path/to/queries/file]

[ericqqqqq]

@agavra

Thank you for your response, it makes sense. I will try the alternative way that you provided.

At the same time, I would like to ask:
I believe the retention.ms is claimed as configurable parameter in the Topic Configurations Doc. However, based on my current understanding, it is not valid in the ksql-server.properties (because no matter a command_topic exists or not, it will try to set that number to infinite). Instead, you can only configure this in the ksql server(listener port).

I think it can be me who is missing some docs. If there is any docs existing and explaining this, could you please share a link, or attach that link under Topic Configurations Doc. I would like to expect a warning/note under the retention.ms parameter. (This is just my personal advice).

[agavra]

@ericqqqqq - I'll have to think about a good way to phrase this documentation. The documentation you point to is for all of kafka, not just KSQL, so KSQL specific information may not be best served there.

FYI @JimGalasyn.

[JimGalasyn]

@agavra, we should probably at least have a mention somewhere in KSQL Configuration Parameter Reference.

[ericqqqqq]

Hi @JimGalasyn, Yes, totally agree with you, KSQL Configuration parameter reference is a good place to show related information.