Ksql: KSQL should support object names with the - character (hyphen, dash)

[rmoff]

- is rejected but _ is valid. At minimum, we should catch and report this limitation better to the user, as the current message is very unclear:

Doesn't work:

ksql> CREATE STREAM ratings-with-customer-data WITH (PARTITIONS=1) AS \
> SELECT R.RATING_ID, R.CHANNEL, R.STARS, R.MESSAGE, \
>        C.ID, C.CLUB_STATUS, C.EMAIL, \
>        C.FIRST_NAME,  C.LAST_NAME \
> FROM RATINGS R \
>      LEFT JOIN CUSTOMERS C \
>        ON R.USER_ID = C.ID \
> WHERE C.FIRST_NAME IS NOT NULL ;
line 1:22: mismatched input '-' expecting ';'
Caused by: org.antlr.v4.runtime.InputMismatchException

Works:

ksql> CREATE STREAM ratings_with_customer_data WITH (PARTITIONS=1) AS \
> SELECT R.RATING_ID, R.CHANNEL, R.STARS, R.MESSAGE, \
>        C.ID, C.CLUB_STATUS, C.EMAIL, \
>        C.FIRST_NAME, C.LAST_NAME \
> FROM RATINGS R \
>      LEFT JOIN CUSTOMERS C \
>        ON R.USER_ID = C.ID \
> WHERE C.FIRST_NAME IS NOT NULL ;

 Message
----------------------------
 Stream created and running
----------------------------

[rmoff]

Similar issue if trying to register a stream over an existing topic:

ksql> print 'test-topic2' from beginning;
Format:JSON
{"ROWTIME":1529657665501,"ROWKEY":"null","foo":"bar"}
^CTopic printing ceased
ksql>
ksql> create stream test-topic2 (foo varchar) with (kafka_topic='test-topic2', value_format='json');
line 1:19: mismatched input '-' expecting ';'
Caused by: org.antlr.v4.runtime.InputMismatchException
ksql> create stream 'test-topic2' (foo varchar) with (kafka_topic='test-topic2', value_format='json');
line 1:15: no viable alternative at input 'create stream 'test-topic2''
Caused by: org.antlr.v4.runtime.NoViableAltException
ksql> create stream "test-topic2" (foo varchar) with (kafka_topic='test-topic2', value_format='json');

 Message
----------------
 Stream created
----------------
ksql> select * from test-topic2;
line 1:19: mismatched input '-' expecting ';'
Caused by: org.antlr.v4.runtime.InputMismatchException
ksql> select * from "test-topic2";
Invalid Expression test-topic2.ROWTIME.
ksql>

In this instance, the workaround is simply to create the stream with a non-hyphen name, e.g. :

ksql> create stream test_no_hyphens_thankyou_topic2 (foo varchar) with (kafka_topic='test-topic2', value_format='json');

 Message
----------------
 Stream created
----------------
ksql> select * from test_no_hyphens_thankyou_topic2;
1529657665501 | null | bar

[rmoff]

+1 to this again, hit it today. At very least we should be catching the exception and being more friendly and tell the user not to use hyphens in the object name.

[joefromct]

It's not only the stream/table _object_, but also the column (in the json) as well right?

This is difficult for me because although I have control over my ksql table/stream names i have a client controlling the JSON that's actually populating it... and yup they have - in the name. :/

I'm even getting a more ambiguous error such as "name is null".

[fmjrey]

It seems the KSQL grammar is hardcoded to follow the Avro spec.
I think it should follow the spec of whatever underlying value format is being used.
JSON is bound to be a successful format, and since in JSON any string can be a key, I advise KSQL to allow any string to be a field when JSON is the value format.
In my case I have JSON derived from datomic/datascript attributes, so supporting - . / would be needed.

[wlisac]

I'm also running into this issue with topic names with - in them.

[apurvam]

Thanks for the feedback folks. I think this is an important issue to fix. We have audited the code base for multiple ways in which KSQL restricts the identifiers you can use (column names, topic names, etc), and how casing affects those. There is a long list of things to become consistent on.

In that list, I think the category where input data cannot be used (becasue of the topic name or because of the columns in the input schema) are at the top of the list. We hope to have some solutions around this by the March release.

[apurvam]

related #1888

[Incara]

Using ksql server 5.2.1 and running into this. Are there plans to bring this to a 2019 release?

[mahipalrampally]

I have same issue with usage of hyphen , is there any update on the fix?

[agavra]

Hello everyone who has given a +1 on this (cc @Incara @mahipalrampally @fmjrey). There are two types of issues:

• hyphens in _field_ names (e.g. CREATE STREAM foo ("my-col" VARCHAR))
• hyphens if _collection_ names (e.g. CREATE STREAM "my-stream" (...))

The fix for the first is on master and will be available in the next KSQL release.

~I am trying to understand if the second is a problem - KSQL does not require that naming of a stream/table is the same as that of a topic, so is it acceptable to reject statements that create streams/tables with hyphens in them (i.e. not allowing quoted stream/table names)? It will still be possible to do something like CREATE STREAM my_stream WITH (kakfa_topic='my-stream'...).~

EDIT: the fix for the second is on the way!

[agavra]

The fix is checked in to master! To use something with a hyphen you will need to use quotes around the object name.