Vector: [Pulsar Sink] why the pulsar sink has a consumer healthcheck?

[KannarFr]

Hi,

On pulsar sink, why the producer is not considered as a valid healthcheck?

My question is about security issues, vector should not be able to consume as a sink IMO. More, it creates a not necessary connection, right?

WDYT?

[KannarFr]

I think the pulsar binary protocol have its healthcheck system. Will check it.

[KannarFr]

Ok http://pulsar.apache.org/docs/en/develop-binary-protocol/#keep-alive,

PulsarDecoder#handlePing  PulsarDecoder#handlePong

Will open issue on pulsar-rs to use it and use it in vector.

Because currently the healthcheck consumer can be on different broker that the producer used.

[binarylogic]

Thanks @KannarFr, in the interim, you can disable the health check with healthcheck = false

[KannarFr]

The healthcheck = false has a strange behavior, by my side, on the pulsar sink the consumer healthcheck is created then stopped, right?

[KannarFr]

How can just not initiate the healthcheck? Can I?

[KannarFr]

@binarylogic^?

[binarylogic]

The healthcheck = false has a strange behavior, by my side, on the pulsar sink the consumer healthcheck is created then stopped, right?

I'm not sure what you mean? Setting healthcheck = false causes a problem?

[KannarFr]

With healthcheck = false configured on the pulsar sink. I get the following error:

Apr 29 16:35:56 db6035d1-1258-4425-b22c-2df69ff7e1a3 vector[2136]: Apr 29 16:35:56.303 ERROR vector::topology: Configuration error: Sink "pulsar_output": consumer error: Connection error: CommandError { request_id: 0, error: AuthorizationError, message: "Client is not authorized to subscribe" }

The auth token has not the right to consume, but with healthcheck false it must not tries, right?

[KannarFr]

I think the consumer is created even if the healthcheck is not ran. Possible?

[KannarFr]

@binarylogic ^

[binarylogic]

@leshow you built this sink. Maybe you understand better what is happening here? Healthchecks should be optional, so I'm curious if it's doing more than it should?

[leshow]

A shot in the dark here because I don't really follow the above thread, I think I see where a problem could be, we should move this line: https://github.com/timberio/vector/blob/master/src/sinks/pulsar.rs#L171 into a futures::lazy or something similar so that it doesn't run build inside that method.

[binarylogic]

Thanks for chiming in. Would you mind doing that? Otherwise, we'll have someone do it next week-ish.

[leshow]

I could make a PR to make this change if you'd like?

edit: just saw you commented, no problem!

[KannarFr]

Thanks to both of you.

[leshow]

does the kafka sink have the same problem? https://github.com/timberio/vector/blob/master/src/sinks/kafka.rs#L212

It calls create in it's healthcheck method also.