Ohara: Able to add duplicate topics

[alan791205]

As title,
I can add 2 exactly same topic name to my configurator by using ohara restful API

# curl http://10.3.0.23:12345/v0/topics -X GET | jq                                                                                                            [
  {
    "name": "topic0",
    "brokerClusterName": "bk00",
    "lastModified": 1557139470020,
    "numberOfReplications": 1,
    "id": "20256056-f43e-4c7f-b7ce-763df0af0433",
    "metrics": {
      "meters": []
    },
    "numberOfPartitions": 1
  },
  {
    "name": "topic0",
    "brokerClusterName": "bk00",
    "lastModified": 1557139468893,
    "numberOfReplications": 1,
    "id": "1533ca7e-5c52-46a0-8ebc-6682c6181cb9",
    "metrics": {
      "meters": []
    },
    "numberOfPartitions": 1
  }
]

I know the difference is that different ID has been assigned to topics but seems the name of the topic should be unique?

see here for conclusion. written by @chia7712

[chia7712]

Renaming a topic is legal to our UI so we don't use the name to be the true kafka topic name.

[alan791205]

What I have been thinking is same topic name showing on UI will be a little confusing when users don't always know there will be 2 different topics created in kafka.
And even if you know that, it is still confusing when you tried to add one of the duplicated topic name to the pipeline or to brokers, especially when UI did not tell us the difference.

I agree that renaming topic should be legal but I do believe adding more descriptions to topics or maybe make topic name unique could be friendlier to users.

[chia7712]

@alan791205 This issue is related to UI flow. Ping @konekoya @vitojeng @eechih

We can disable users to use duplicate topic name, but the side-effect is that users can't rename an existent topic any more.

[alan791205]

We can disable users to use duplicate topic name, but the side-effect is that users can't rename an existent topic any more.

hmm, interesting trade-off. Can you elaborate?

[vitojeng]

hmm, interesting trade-off. Can you elaborate?

Because Apache Kafka does not have rename topic concept.

[vitojeng]

@chia7712 May we check duplicate name in the Ohara API level?

[alan791205]

Because Apache Kafka does not have rename topic concept.

Ohh, I thought topic names Ohara created in kafka were not using topic names provided by users.
So what @chia7712 meant is just simply disabling renaming functions in Ohara?

[chia7712]

May we check duplicate name in the Ohara API level?

This check is expensive ... Do we get a huge benefit from it?

If our UI flow may confuse user with duplicate name, we should just disable user to change the topic name and make topic's id be equal to name (so kafka can check the duplicate for us)

[vitojeng]

Currently, I would prefer to stay the same. If the user wants to keep the unique name, just avoid it.

Or maybe we add a warnning message in the UI? Let user decide.

[alan791205]

I think it could be helpful if I can know more about the topic rather than just names. Maybe first 6 letters of the topics ID?

[chia7712]

I think it could be helpful if I can know more about the topic rather than just names.

Why do you want to see the content of id ??? It is just a random string and it should be meaningless to you.

[alan791205]

Why do you want to see the content of id ??? It is just a random string and it should be meaningless to you.

If it is enough random and unique it can be meaningful to me. The point is to find out different topics with same name.

[chia7712]

The point is to find out different topics with same name.

ping @konekoya @eechih any suggestions??

[eechih]

我認為 topic_id（顯示前六碼）只能達到「區別」，無法達到有效的「識別」，對使用者而言，有識別性的名稱比較有用。
如果為了區別而要同時顯示 name + topic_id(first 6 letters) 的話，我比較建議在建立 topic 時，就不允許名稱重複。

[eechih]

make topic's id be equal to name (so kafka can check the duplicate for us)

@chia7712 會不會受到 kafka 的 renaming rule 的限制？例如：名稱中不能有空白

[chia7712]

我比較建議在建立 topic 時，就不允許名稱重複。

Checking the duplicate name is expensive since our index is build for id rather then name...

會不會受到 follow kafka 的 renaming rule 的限制？是不是就不能使用空白了呢？

yep. The naming will be restricted by kafka if we pass user-defined name to kafka.

[eechih]

我印像中，HDFS Sink 出去的檔案路徑好像有包含 topic_id
ex: /tmp/20256056-f43e-4c7f-b7ce-763df0af0433/partition0/*
如果 topic_id 和 name 相同的話，應該可提高檔案路徑的可讀性

[chia7712]

如果 topic_id 和 name 相同的話，應該可提高檔案路徑的可讀性

+1 to this :)

[konekoya]

@chia7712 My first instinct is to add a mechanism to prevent duplication from happening in the backend. But since it's expensive to do so. Both @eechih and @alan791205 's suggestions are okay to me.

One question though. If we go with @eechih's solution. What does the topic name would be like?

[alan791205]

One question though. If we go with @eechih's solution. What does the topic name would be like?

I think it would be using the name what users provide, and be following kafka rules.

[konekoya]

I think it would be using the name what users provide, and be following kafka rules.

If that's the case, +1 on this

[chia7712]

@jackyoh @vitojeng @saivirtue @wu87988622 Any suggestions? I will file another issue to address this one if we all reach the consensus.

[saivirtue]

+1 for the conclusion

using the name what users provide, and be following kafka rules.

[vitojeng]

using the name what users provide, and be following kafka rules.

+1

[jackyoh]

如果 topic_id 和 name 相同的話，應該可提高檔案路徑的可讀性

+1

[chia7712]

We have reached the consensus that ohara take user-defined name to create kafka topic. And topic APIs don't use ID any more.

The follow-up is #1080.