Pulsar: PIP-31: Transactional Streaming

[kuangye098]

This is the master issue for tracking implementing PIP-31: https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.40v9f3564x7t

The PIP is broken down into following tasks:

• [x] API change #4161 #4165 #4952
• [x] Protocol change #4866 #4776
• [x] Transaction buffer
  
  
  
  • [x] In-memory transaction buffer #4165
  
  
  • [x] Persistent transaction buffer #4738
  
  
  • [x] Transaction buffer client #6544
  
  
• [x] Transaction coordinator
  
  
  
  • [x] In-memory transaction coordinator #4161
  
  
  • [ ] Persistent transaction coordinator #5570
  
  
  • [x] Transaction coordinator assignment(TC startup and failover) #5515
  
  
  • [x] Transaction coordinator client #5680
  
  
• [x] Transacdtional Acknowledgement
  
  
  
  • [x] Introduce PENDING_ACK state in acknowledgement path #4265
  
  
  • [ ] Persistence of pending acks #4557
  
  
• [x] Transaction request handling in the broker: implementation of the request in broker side.
• [x] Client and broker integration #7552 #7781
• [ ] Offloader: Make offloader aware of transactional data
• [ ] Functions: Make functions use transactions
• [ ] Integration Tests
• [ ] Chaos (Jepsen) Tests

[sijie]

@kuangye098 it is a nice feature request. I am wondering if you are interested in implementing this feature, we can collaborate on this if you are interested in contribution.

The producer pulls the pre-processing transaction message.

you mean "produce", instead of "pull", right?

[kuangye098]

@sijie You are right, I may have described it incorrectly.

[kuangye098]

@sijie I can try to implement it, because in other projects, I have implemented it.

[sijie]

@kuangye098 that’s awesome. Since it is big feature, it would be good that you can write a PIP (pulsar improvement proposal) to describe your proposal of changes. Then we can discuss in details.

Here is wiki page for all the pulsar PIPs. You can check out how other people wrote them https://github.com/apache/pulsar/wiki

[kuangye098]

@sijie OK，I will read it carefully and then write a PIP to describe the proposal of changes.

[sijie]

great thank you @kuangye098

[codelipenghui]

@sijie Is this the same as you mentioned before(Fast Transaction Message)?

[ConcurrencyPractitioner]

Hi @sijie BTW do you know how we create a PIP on Github? For example, do we need permission granted or something first before we create one.

[sijie]

@ConcurrencyPractitioner :

You don't need any permissions to start a PIP, but you do need permissions to post the PIP to Pulsar Github Wiki.

So you can just write down a PIP in gist and share it to dev@ mailing list. Any pulsar committer can help you re-post your gist to Pulsar wiki.

[sijie]

Here is the PIP for supporting transactions: https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.40v9f3564x7t

I will break down this PIP into tasks and update the tasks here. This issue will be used as an anchor for all transaction related tasks.

[sijie]

Broke the PIP into tasks and updated them in the description.

[sijie]

Marked this as for 2.5.0. since we are close to release 2.4.0 soon.

[MarvinCai]

I'm very interesting in this PIP and I could help with some subtasks.

[sijie]

@MarvinCai : @tuteng and I start working on the TC part, and @zymap is already working on the TB part. Maybe you can help us in the acknowledgement part. I will try to do the api and protocol and coordinating the work around.