Ksql: Trying to select from K-SQL stream created o kafka topic produces end-of-input error

[visakhmurukes]

Describe the bug
Trying to select from a stream created over kafka topic fails with an error midway after displaying some initial rows
To Reproduce

• Create a kafka topic based on MySQL table.
• Create a K-SQL stream on top of Kafka topic
• Trying to select from a stream using K-SQL query

Expected behavior

stream gets created

Actual behaviour

_com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing quote for a string value
at [Source: (String)"{"row":{"columns":[1579000050672,"\u0000\u0000\u0000\u00004"; line: 1, column: 34] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["row"]->io.confluent.ksql.GenericRow["columns"]->java.util.ArrayList[1])
Caused by: Unexpected end-of-input: was expecting closing quote for a string
value
at [Source:
(String)"{"row":{"columns":[1579000050672,"\u0000\u0000\u0000\u00004"; line: 1,
column: 34] (through reference chain:
io.confluent.ksql.rest.entity.StreamedRow["row"]->io.confluent.ksql.GenericRow["
columns"]->java.util.ArrayList[1])
Caused by: Unexpected end-of-input: was expecting closing quote for a string
value
at [Source:
(String)"{"row":{"columns":[1579000050672,"\u0000\u0000\u0000\u00004"; line: 1,
column: 119]_

The offending row looks like this

'90465', 'TEST 1', '8', '990_60040_1577196000', '1577196000', '2019-12-24', '0', '2019-12-22 08:20:28.232483', '2019-12-24 19:21:02.207983', '124', '1', '1', '2019-12-24', '06:00 PM'

table definition is below

CREATE TABLE test
(
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(100) NOT NULL,
number varchar(100) NOT NULL,
s_id varchar(200) NOT NULL,
st int(10) unsigned NOT NULL,
b_date_start date NOT NULL,
status tinyint(1) NOT NULL,
created_at datetime(6) NOT NULL,
modified_at datetime(6) NOT NULL,
exp_id int(11) NOT NULL,
modified_by_id int(11) NOT NULL,
b_enabled tinyint(1) NOT NULL,
s_date date NOT NULL,
s_time varchar(200) NOT NULL,
PRIMARY KEY (id),
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

[yugensaccnt]

Got the same error message, have been working with KSQL for the past 3 months and suddenly running into this starting today.

[visakhmurukes]

Any help on this please?
The error still persists and cant understand the root cause

[yugensaccnt]

@visakhmurukes - I went on ahead by creating additional streams on the top of the stream that was originally causing this problem and did not encounter the error on the subsequents streams/tables. Let me know your experience

[Crispy1975]

I have had the exact same thing happen today, the main difference is that we are using MongoDB to stream into Kafka via Kafka Connect. Our stream is built on top of two other streams doing various transforms, and for the most part has not had an issue until today.

Query terminated
com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing quote for a string value
 at [Source: (String)"{"row":{"columns":[1582302557410,"0d364ad1-b771-49ea-b7d2-37ee088bc2c9","0d364ad1-b771-49ea-b7d2-37ee088bc2c9","60388b1049567835e143","added","item","e860b374-dab0-42ef-891d-f39a8f8932ea"; line: 1, column: 153] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["row"]->io.confluent.ksql.GenericRow["columns"]->java.util.ArrayList[6])
Caused by: Unexpected end-of-input: was expecting closing quote for a string
    value
 at [Source:
    (String)"{"row":{"columns":[1582302557410,"0d364ad1-b771-49ea-b7d2-37ee088bc2c9"
    ,"0d364ad1-b771-49ea-b7d2-37ee088bc2c9","60388b1049567835e143","added","item"
    ,"e860b374-dab0-42ef-891d-f39a8f8932ea"; line: 1, column: 153] (through
    reference chain:
    io.confluent.ksql.rest.entity.StreamedRow["row"]->io.confluent.ksql.GenericRow["
    columns"]->java.util.ArrayList[6])
Caused by: Unexpected end-of-input: was expecting closing quote for a string
    value
 at [Source:
    (String)"{"row":{"columns":[1582302557410,"0d364ad1-b771-49ea-b7d2-37ee088bc2c9"
    ,"0d364ad1-b771-49ea-b7d2-37ee088bc2c9","60388b1049567835e143","added","item"
    ,"e860b374-dab0-42ef-891d-f39a8f8932ea"; line: 1, column: 379]

[AndreaTosti]

I have the same problem, it always happens to me when I try to return a large number of rows (about 200000)

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing quote for a string value
 at [Source: (String)"{"row":{"columns":[1582909218117,"2250041",1582909218117,"User_"; line: 1, column: 58] (through reference chain: io.confluent.ksql.rest.entity.StreamedRow["row"]->io.confluent.ksql.GenericRow["columns"]->java.util.ArrayList[3])

[AndreaTosti]

EDIT: After some tests I can say that it is not a bug of ksql but it is a matter of correctly configuring ksql in order to make it work well even on slower machines. My solution is to increase the timeout in the configuration of ksqldb-server by changing an environment variable. I am using docker, in any case the instructions are here. Notice that I chose a value greater than 30000 milliseconds, in this case, 120000 milliseconds

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.7.1
   ...
    environment:
      ...
      KSQL_OPTS: "-Didle.timeout.ms=120000"
      ...

Reason:

Investigating I noticed that in the ksqldb-server logs, since I use it locally, if it is not very performing, a timeout expires, and if this timeout expires, the JSON is truncated. If the timeout does not expire, I can also return 1 million lines without problems.

Default timeout is 30000 milliseconds, and this leads to:

[2020-03-03 14:55:03,638] ERROR Exception occurred while writing to connection stream: (io.confluent.ksql.rest.server.resources.streaming.QueryStreamWriter:101) java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
My docker-compose for testing:

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.7.1
   ...
    environment:
      ...
      KSQL_OPTS: "-Didle.timeout.ms=1000"
      ...

This leads to

SEVERE: An I/O error has occurred while writing a response message entity to the container output stream. org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 1001/1000 ms