Pulsar: Is there any relationship between Yahoo Message Broker and Pulsar projects?

[David-Jin]

I heard about yahoo's PNUTS project long ago.
So I wandering about is this Pulsar project the open source version of Yahoo Message Broker project?

Is there any plans to open PNUTS project ?
@merlimat

[merlimat]

Hi @David-Jin, yes indeed there is a relationship.

The system described in the PNUTS paper was then productionized with the name Sherpa ([1], [2]). What was described as Yahoo Message Broker was eventually implemented as a different system and served Sherpa replication for multiple years, until it got replaced in production by Pulsar ~2 years ago.

Is there any plans to open PNUTS project ?

Since I'm not at Yahoo anymore I don't have insight or voice in that :). I'll turn the question to @joefk @msb-at-yahoo

• [1] https://en.wikipedia.org/wiki/Yahoo_Sherpa
• [2] https://yahooeng.tumblr.com/post/120730204806/sherpa-scales-new-heights

[David-Jin]

@merlimat Thanks for your information!

I'am considering using Pulsar in the following application:

1. we have 3 data centers in US, EU, and Asia;
2. we want to sync the mysql database information of above data centers.
3. we have mysql transections in some http requests.

So I want to use pulsar as message queue to do such job.
Am I right to choose Pulsar as the right option to realize the application?
would you give me some advice? thanks a lot!
@merlimat

[merlimat]

Am I right to choose Pulsar as the right option to realize the application?

That is exactly the same scenario as Sherpa use case, so I would definitely say that it is a good match. We literally designed for that use case :)

would you give me some advice? thanks a lot!

Sure, shoot up any question/doubts either here, in the mailing lists or in the Slack channel.

[David-Jin]

Hi, Merlimat:

When in the sync SQL database scenario, what do you think about CockroachDB(a mimic of google spanner) or TiDB(from China).
The above 2 databases are 3rd generation DB: NewSQL DB.
The fantastic feature of CockroachDB is ditrubuted databases, which support muti-datacenters synchronization.
Before I found "Pulsar project", I was considering using CockroachDB or TiDB in the database sychronization of different datacenters through “Internet” (Not dedicated network).
1.Do you think CockrouchDB is suitable for my application?
2.What's the best practice of such scenario?
@merlimat Thanks for your advice!

[merlimat]

I know about Spanner, though I have no insight about the replication implementation in CockroardDB or TiDB.

In Pulsar, the replication is one of the key feature of the system and it has been battle tested in several year of production usage, replicating millions of topics across 8+ datacenters in full mesh.

We have had a number of network outages especially on the Pacific ocean links that lasted for several days of seriously degraded bandwidth issues and we were able to "make it through".

[joefk]

Cockroachdb uses Raft, and the replication is strongly consistent.

Doing strict consensus across the WAN will have a penalty, does not matter
what the protocol is. Latencies will be higher.

On Fri, Feb 23, 2018 at 4:36 PM, Matteo Merli [e-mail]
wrote:

I know about Spanner, though I have no insight about the replication
implementation in CockroardDB or TiDB.

In Pulsar, the replication is one of the key feature of the system and it
has been battle tested in several year of production usage, replicating
millions of topics across 8+ datacenters in full mesh.

We have had a number of network outages especially on the Pacific ocean
links that lasted for several days of seriously degraded bandwidth issues
and we were able to "make it through".

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-pulsar/issues/1259#issuecomment-368179169,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVBZYmw-nSbIcTZjKrt0AVqT_-94Bjvzks5tX1mqgaJpZM4SMITl
.

[sijie]

Closed this issue since the question is answered.