Kafka-connect-jdbc: Unable to push any connectors, due to error 500 - Request timed out

[PiotrBB]

Hello,
I am unable to PUT any of my configs into my Connect, due to error code 500.

curl http://localhost:8083/
{"version":"0.11.0.0-cp1","commit":"bba25847402e83a0"}
curl http://localhost:8083/connectors
{"error_code":500,"message":"Request timed out"}

I'm trying to create new connector with the following configuration:
curl -X POST -H "Content-Type: application/json" --data '{"name": "source-oracle-test", "config": {"connector.class":"JdbcSourceConnector", "tasks.max":"1", "connection.password":"PWD", "connection.url":"jdbc:oracle:thin:@oraclehost:1572/SERVICE", "connection.user":"USER", "table.whitelist":"TABLE", "mode":"timestamp", "timestamp.column.name":"TRANSACTION", "topic.prefix":"source-oracle-test-" }}' http://localhost:8083/connectors

but it returned same result
{"error_code":500,"message":"Request timed out"}

I somehow managed to create local file sink connector, based on confluent's sample:

$ curl -X POST -H "Content-Type: application/json" --data '{"name": "local-file-sink", "config": {"connector.class":"FileStreamSinkConnector", "tasks.max":"1", "file":"test.sink.txt", "topics":"connect-test" }}' http://localhost:8083/connectors

Source but when i tried to the same in new version (changed names to _1), i got an error.

curl -X POST -H "Content-Type: application/json" --data '{"name": "local-file-sink1", "config": {"connector.class":"FileStreamSinkConnector", "tasks.max":"1", "file":"test.sink.txt", "topics":"connect-test1" }}' http://localhost:8083/connectors

{"error_code":500,"message":"Request timed out"}

I'm not sure how to proceed with investigation. Here's connect log's tail:
[2017-10-27 08:23:57,909] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:825) [2017-10-27 08:23:57,910] INFO Sink task WorkerSinkTask{id=local-file-sink-0} finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:233) [2017-10-27 08:23:57,916] INFO Discovered coordinator localhost:9092 (id: 2147483647 rack: null) for group connect-local-file-sink. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:597) [2017-10-27 08:23:57,917] INFO Revoking previously assigned partitions [] for group connect-local-file-sink (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:419) [2017-10-27 08:23:57,918] INFO (Re-)joining group connect-local-file-sink (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:432) [2017-10-27 08:23:57,928] INFO Successfully joined group connect-local-file-sink with generation 3 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:399) [2017-10-27 08:23:57,929] INFO Setting newly assigned partitions [connect-test-0] for group connect-local-file-sink (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:262) [2017-10-27 08:28:26,696] INFO 0:0:0:0:0:0:0:1 - - [27/Oct/2017:12:26:56 +0000] "POST /connectors HTTP/1.1" 500 48 90171 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2017-10-27 08:31:34,927] INFO 0:0:0:0:0:0:0:1 - - [27/Oct/2017:12:31:34 +0000] "GET / HTTP/1.1" 200 54 13 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2017-10-27 08:32:14,898] INFO 0:0:0:0:0:0:0:1 - - [27/Oct/2017:12:32:14 +0000] "GET / HTTP/1.1" 200 54 4 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2017-10-27 08:34:13,000] INFO 0:0:0:0:0:0:0:1 - - [27/Oct/2017:12:32:42 +0000] "GET /connectors HTTP/1.1" 500 48 90008 (org.apache.kafka.connect.runtime.rest.RestServer:60) [2017-10-27 08:45:22,222] INFO 0:0:0:0:0:0:0:1 - - [27/Oct/2017:12:43:52 +0000] "POST /connectors HTTP/1.1" 500 48 90007 (org.apache.kafka.connect.runtime.rest.RestServer:60

I have also added rest.advertised.host.name and rest.advertised.port settings, but the result is exactly the same.
Any ideas?

[stjepanmatijasevic]

Hi,
I have same error. Did you solve it?

[PiotrBB]

Hello,

Unfortunately, no. Still can't use Oracle JDBC connector.

[stjepanmatijasevic]

Hi, I resolve it. It was broker configuration problem.

• I had 3 kafka node with same broker.id in server.properties
  
  
  
  • also in server properts I add advertised.listeners==PLAINTEXT://:9092
  
  
  • and added zookeeper.connect=:2181,:2181,:2181
  
  

Also in connector-distributed.properties you need to enter:

• bootstrap.servers=:9092,:9092,:9092
• rest.advertised.host.name=
• rest.advertised.port=8083

And it works,
Stjepan

[PiotrBB]

Hi Stjepan,

I already have all that set in my configs. Still doesn't work for me.

[dbranscombe]

I have two different machines with same docker-compose setup. works fine on one, but fails with this error on other. Any tips?

[hongphu8790]

I have same problem with @dbranscombe. The docker may freeze on startup after service init and I get /connectors response with error code 500.

[dbranscombe]

Not conclusive, but I think my issue was related to a Disk IO starved VM
that I was running my docker containers on...still waiting for VM to be
re-provisioned to confirm but have not seen issue on more robust VM stacks.

On Wed, Aug 22, 2018 at 2:38 AM hongphu8790 [e-mail]
wrote:

I have same problem with @dbranscombe https://github.com/dbranscombe.
The docker may freeze on startup after service init and I get /connectors
response with error code 500.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/confluentinc/kafka-connect-jdbc/issues/302#issuecomment-414974190,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0D-XWufvxaT27q33D7Pes0c4bMPvuMks5uTSajgaJpZM4QJACE
.

[balareddy]

I am trying to use connect-distributed.bat

I put kafka-connect-jdbc-3.3.0.jar, ojdbc6-12.1.0.2.0.jar, connect-file-1.1.0.jar,connect-runtime-1.1.0.jar, connect-json-1.1.0.jar in the libs folder of kafka_2.11-1.1.0\libs on windows environment

I have 3 kafka brokers ( server.properties, server2.properties, server3.properties) with unique value for broker.id. The listeners, advertised.listeners is also updated to have unique ports.
The connect-distributed has the list of all brokers updated in bootstrap.servers.

I then started the zookeeper and the kafka servers and tried to submit the below connector through REST url but it comes with the below error.

{
"error_code": 500,
"message": "Request timed out"
}

the configuration is as below: ( I have similar configuration in connect-standalone.bat and it works fine )

{
"name": "test-local-oracle",
"config": {
"tasks.max": "1",
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"poll.interval.ms":"1000",
"connection.url":"jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=6100)))(CONNECT_DATA=(SERVICE_NAME=LOCAL_LOCALHOST)))",
"connection.user":"xxxx",
"connection.password":"xxxxx",
"query":"select EVENT_ID, COMPOSE(T_ID) AS T_ID,UPDATE_TS from TEST_EVENT_LOG",
"mode":"timestamp+incrementing",
"incrementing.column.name":"EVENT_ID",
"timestamp.column.name":"UPDATE_TS",
"topic.prefix":"TEST",
"validate.non.null":"false"
}
}

Also, seeing warnings like below on starting the connect-distributed.

[2018-08-27 16:53:34,577] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.producer.Prod
ucerConfig)
[2018-08-27 16:53:34,586] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerCo
nfig)
[2018-08-27 16:53:34,593] WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.producer.P
roducerConfig)
[2018-08-27 16:53:34,602] WARN The configuration 'rest.advertised.port' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig
)
[2018-08-27 16:53:34,611] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consum
er.ConsumerConfig)
[2018-08-27 16:53:34,612] WARN The configuration 'offset.storage.file.filename' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.Consum
erConfig)
[2018-08-27 16:53:34,614] WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.C
onsumerConfig)
[2018-08-27 16:53:34,617] WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig
)
[2018-08-27 16:53:34,619] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-08-27 16:53:34,620] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-08-27 16:53:34,623] WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig
)
[2018-08-27 16:53:34,624] WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerC
onfig)
[2018-08-27 16:53:34,626] WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig
)
[2018-08-27 16:53:34,628] WARN The configuration 'rest.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-08-27 16:53:34,628] WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.C
onsumerConfig)
[2018-08-27 16:53:34,630] WARN The configuration 'offset.flush.interval.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerCo
nfig)
[2018-08-27 16:53:34,631] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-08-27 16:53:34,631] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConf
ig)
[2018-08-27 16:53:34,633] WARN The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.Consum

Could you let know on how to resolve the issue?

[hongphu8790]

Error "Request timed out" comes from many reasons, one of which is my custom plugin (connector, transform, converter,...) contains some jars dependency which belong to connect (such as connect-api, connect-json, connect-transform,...), when connect startup, it'll load them and may be caused a problem "freeze" state. You may exclude it when build a uber jar.

[rupeshpatel02]

I am also facing time out error while posting the source connector for DB2,POST API wait for almost 90 second and after that it time out with below error

[2019-06-02 00:17:17,906] INFO 192.168.1.2 - - [01/Jun/2019:18:45:47 +0000] "POST /connectors HTTP/1.1" 500 48 90004 (org.apache.kafka.connect.runtime.rest.RestServer:60)

I can see below warning also in kafka connect log just before the time out error

This member will leave the group because consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1011)

is there any configuration to increase the API time out. I have also noticed interesting behavior, when i run the kafka connect in standalone mode it works perfectly.

[lknaubkonrad]

rupeshpatel02 - How did you resolve this? I'm having the same issue.

[Fryuni]

We are also having the same issue. It only happens when there are JDBC connectors running on the Connect cluster. We had to create a separate cluster only for JDBC connectors just to keep only those with a broken api and manage the others as intended.
Any progress on solving this?