Ohara: [Question] Should we block the users from delete a running pipeline

[konekoya]

A good question raised by @wu87988622

The deleting pipeline feature will be soon added in #2321 (Deleting the objects like running connectors and stream apps) but should we rethink about the UI flow and block users from deleting a running pipeline? This was used to be guard by the backend API

Personally, I'm leaning toward adding a stop button in the pipeline list page so the user can control this at will. And block users from deleting the running pipelines

Open for discussion :)
/cc @vitojeng @chia7712 @eechih @wu87988622

[chia7712]

absolutely :)

[vitojeng]

should we rethink about the UI flow and block users from deleting a running pipeline this was used to be guard by the backend API?

Sure.

[chia7712]

running pipeline

BTW, all components in pipeline have state (i.e running). Which one can block us from deleting pipeline? Connector? streamapp? Or topic? Stopping topic is equal to discard all data in topic. Does it make sense?

[konekoya]

@chia7712 Good questions!

Which one can block us from deleting pipeline? Connector? streamapp? Or topic?

We currently treat a connector/streamApp as running when it has __any state__ in it. This basically means if a given connector has the state field. We count it as running. But we only treat a pipeline is running when everything (This includes connectors and streamApps, but not topics as they did not have the state back in the good old days...) in the pipeline is running. For example:

• Pipeline A has a src connector and a sink connector along with a topic and both connectors are running. <- this pipeline is marked as __running__ in the pipeline list

• Pipeline B has the same setup as the above pipeline but its sink connector is __NOT__ running. <- this pipeline is marked as __stopped__ since not all the connectors are running in this pipeline.

I think this logic would also need to tweak/update a bit since now we have to check these topics in the pipeline as well.

Stopping topic is equal to discard all data in topic.

The current implementation will stop running connectors and streamApps only, since topics can only be removed in the workspace right now

Stopping topic is equal to discard all data in topic.

Is this a safe action from the API standpoint? It sounds like a __reset__ to me?

[chia7712]

this pipeline is marked as stopped since not all the connectors

That is a bit weird to me. Do we disallow user to start a part of pipeline? Could I have a large pipeline and test a part route of it?

Is this a safe action from the API standpoint? It sounds like a reset to me?

The stop to any object is to delete its from kafka/docker, and Reset consists of STOP and START.

As discussed before, the topic is an embarrassed object to pipeline now. We can’t control topic in pipeline level and we can’t see the relationship between pipeline and topic in workspace level. Pipeline is a group/graph of connector, streamapp and topic and it does not care for topic. Topic is outside pipeline with a view to connecting different pipelines but we don’t give the such function. By contrast, being a part of pipeline is suitable way to our UI now but we are not going with this option.

[konekoya]

Do we disallow user to start a part of pipeline? Could I have a large pipeline and test a part route of it?

Nope. It's allowed. You can certainly do this with our UI. Not sure about why it's wired. Could you elaborate 😅

The stop to any object is to delete its from kafka/docker, and Reset consists of STOP and START.

Ah, okay :)

As discussed before, the topic is an embarrassed object to pipeline now. We can’t control topic in pipeline level and we can’t see the relationship between pipeline and topic in workspace level. Pipeline is a group/graph of connector, streamapp and topic and it does not care for topic. Topic is outside pipeline with a view to connecting different pipelines but we don’t give the such function. By contrast, being a part of pipeline is suitable way to our UI now but we are not going with this option.

will need some more time to read thru your last paragraph...

[chia7712]

Not sure about why it's wired. Could you elaborate

A stopped pipeline should be equal to “no data flow in this pipeline”. A stopped connector means no data produced/consumed by it. A stopped topic should be no data in/out. I don’t want to say “hi, your pipeline is stopped but it still produce/consume data from your workspace. I don’t know why we call it is stopped.”

[vitojeng]

IMO, a stopped pipeline only means all connectors and all streamapps in this pipeline are stopped.

[vitojeng]

A stopped pipeline should be equal to “no data flow in this pipeline”.

Totally agree.
IMHO, m​ore precisely, pipeline should not have state for running or stopped. Pipeline is just a Paper that we put some "components"(​with state) on it. So we say a stopped pipeline actually means that these components have no data flow.