Syndesis: Unable to create Integration if Kafka is not currently running/active

[evanshortiss]

This is a...

[ ] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Documentation issue or request

The problem

Unable to create an integration with Kafka if the broker/cluster is not currently active/running. To produce this issue try the following:

1. Create a connection of any type (this will be the start connection in the in integration)
2. Create a Kafka connection to a broker URL that doesn't exist/is not running
3. Create an Integration from the connection in step 1 to Kafka. When you select "Publish" messages to the topic "Something wrong" is shown (see attachment)

Expected behavior

Should be able to create the Integration even if the Kafka broker/cluster is down.

Screenshot

Request and Response Data

400 response from https://HOSTNAME/api/v1/connections/i-M1le7YOaH8qq746bswFz/actions/io.syndesis:kafka-publish-action

{
    "inputDataShape": {
        "kind": "any"
    },
    "outputDataShape": {
        "kind": "none"
    },
    "propertyDefinitionSteps": [
        {
            "name": "Select the Kafka topic",
            "properties": {
                "topic": {
                    "order": 1,
                    "componentProperty": false,
                    "deprecated": false,
                    "displayName": "Topic Name",
                    "group": "common",
                    "javaType": "java.lang.String",
                    "kind": "path",
                    "labelHint": "Select the Kafka topic to send data to.",
                    "required": true,
                    "secret": false,
                    "type": "string"
                }
            },
            "description": "Specify Kafka topic name"
        }
    ],
    "_meta": {
        "message": "Connection to broker transactions-cluster-kafka-brokers.my-cluster.svc.cluster.local:9092 has failed.. Unable to fetch and process metadata",
        "type": "DANGER"
    }
}

Tasks involved / Steps to Reproduce

See above

[squakez]

Kafka connector is dynamically retrieving the list of topics during integration configuration. If the broker is down, then you won't be able to know which are the topics to select. I am not sure if we should consider this a bug.

[zregvart]

@squakez it's definitely a bug, we should allow the user to specify parameters even though we don't consider them valid. It's up to the user to deal with manually entered values, at the cost of failing exchanges in the integration.

[squakez]

Ok. Then, we should apply the same fix to many connectors. Most of them are recovering dynamic metadata when the connection is up or they are showing an error page as kafka connector is doing.

[zregvart]

@squakez I don't think that's the case, or it wasn't when the dynamic metadata was introduced. We should/used to have a Hystrix based fault tolerance built in. So even if the syndesis-meta pod was unavailable or an error was raised UI would receive payload without the enriched data.

[zregvart]

I think we can remove this from 1.9.0, not to block the release and address this in a patch release. Thoughts @syndesisio/backend?

[Delawen]

I agree, this is not blocking, you can use kafka connectors in all cases except this. You can wait for the broker to be online to create it.

But sure, should be fixed.

[squakez]

I will take care as part of the connector TLS refactory planned.

[zregvart]

With #8103 merged can this now be closed?

[squakez]

I expected the comment on commit was taking care of that!