Pulsar: [discussion] Pulsar release 2.6.0

[sijie]

This issue is used for collecting the features planned for release 2.6.0.

Features

• [PIP-37] Large message size support #4400 approved doc-added
• [PIP-39] Namespace change events(System Topic) #4955 merged no-need-doc
• [PIP-45] Switch ManagedLedger to use MetadataStore interface #5358 merged
• [PIP 53] Contribute DotPulsar to Apache Pulsar
• [PIP-54] Support acknowledgment at batch index level #6052 merged doc-added
• [PIP 57] Improve Zookeeper Session Timeout Handling #6347 review-required
• [PIP-60] Support SNI routing to support various proxy-server #6566 approved
• [PIP-61] Advertise multiple addresses #6903 review-required doc-added
• [PIP-65] Adapting Pulsar IO Sources to support Batch Sources #7090 review-required
• [Load balancer] Add ThresholdShedder Strategy for loadbalancer #6772 merged
• [Load balancer] Expose loadbalance metric to Prometheus #6772 merged
• 
  
  • [Key Shared] Ordering dispatch when adding consumer #7106 #7108 #7188
  
  
• [Key Shared] Add support for key hash range reading #5928 merged
• [Schema] Allow for schema reader and writer registration on SchemaDefinition #6905 merged
• Use pure-java Air-Compressor instead of JNI based libraries #5390 approved
• Support multiple pulsar clusters to use the same bk cluster #5985 merged
• Support delete inactive topic when subscriptions caught up #6077 merged
• Support consumers set custom message retry delay #6449 merged doc-added
• Add configuration to disable auto-creation of subscriptions #6456 merged
• Add a flag to skip broker shutdown on transient OOM #6634 merged
• Make zookeeper cache expiry time configurable #6668 merged doc-added
• Check replicator periodically to avoid issue due to zookeeper missing watch #6674 merged
• Expose managedLedgerCache, managedLedger, loadBalance metrics to Prometheus #6705 merged
• Optimize consumer fetch messages in case of batch message #6719 approved
• Add config to limit max partitions for a partiitoned topic #6794 merged
• Add options for the GetBookieInfo check #6800 merged
• Change default FlushEntryLogBytes to 256MB to improve bookie io throughput #6915 merged
• Make Nar Extraction Directory configurable #6933 merged
• Custom functions runtime
• Introduce precise topic publish rate limiting #7078 merged doc-required
• 
  
  • Expose new entries check delay in the broker.conf #7154
  
  
• 
  
  • Add broker interceptor for Intercepting all Pulsar command and REST API requests #7143
  
  
• 
  
  • Only close active consumer for Failover subscription when seek() #7141
  
  
• 
  
  • [Schema] Support use null key and null value in KeyValue Schema #7139
  
  
• 
  
  • Allow to delete topics that are failing to recover #7131
  
  
• 
  
  • Support set netty max frame size in bookkeeper.conf #7116
  
  
• 
  
  • Trigger rollover when meeting maxLedgerRolloverTimeMinutes #7111
  
  
• 
  
  • Cap the dispatcher batch size in bytes to fixed max #7097
  
  
• 
  
  • Support specify managedLedgerMaxSizePerLedgerMbytes in broker.conf #7085
  
  
• 
  
  • Allow to grant permissions when the authorization is disabled #7074
  
  
• 
  
  • Add messages and bytes counter stats to broker-stats/topics #7045
  
  

Pulsar Proxy

• Add REST API to get connection and topic stats #6473 merged doc-required-it is generated automatically
• Add advertised address option #6942 merged doc-added
• Add proxyLogLevel into config #6948 merged

Admin

• Add maxUnackedMessagesPerSubscription and maxUnackedMessagesPerConsumer on namespaces policies #5936 merged
• Support get a message by message ID in pulsar-admin #6331 merged
• Support delete subscription forcefully #6383 merged
• Add subscribe initial position for consumer CLI #6442 merged
• Support to get managed ledger info of a partitioned topic #6532 merged
• Support compact all partitions of a partitioned topic #6537 merged
• Support multi-hosts in PulsarAdmin #6547 merged
• Support to get internal stats for a partitioned topic #6624 merged
• Support enable or disable subscription auto-creation at namespace level #6637 merged
• Enable to set the subscription expiration time for each namespace #6851 merged doc-added

Functions

• Built-in Functions #6895 approved
• [Function-Go] Add Go Function heartbeat (and gRPC service) for production usage #6031 merged
• Add custom property option to functions #6348 merged
• Separate TLS configuration of function worker and broker #6602 merged
• Added ability to build consumers in functions and sources #6954 merged
• 
  
  • Support DLQ on sources and sinks #7032
  
  

Pulsar SQL

• KeyValue schema support #6325 merged
• Multiple Avro schema version #4847 merged
• 
  
  • Fix presto SQL does not start metrics service before queue execute #7030
  
  

Pulsar IO

• [RabbitMQ] Allow routing key per message to RabbitMQ sink connector #5890 merged
• [RabbitMQ] Add passive config options #6679 merged
• [debezium] Upgrad from v0.10.0-Final to v1.0.0-Final #5972 merged
• [debezium] Support avro schema for debezium connector #6034 merged
• [influxdb2] Add support for influxdb2 in pulsar-influxdb-sink #6601 merged
• [jdbc] Add jdbc sinks: postgres, mariadb, clickhouse #6835 merged
• Added ability for sources to publish messages on their own #6941 merged

Java Client

• Change the time unit of patternAutoDiscoveryPeriod to seconds #5950 merged
• Support waiting for inflight messages while closing producer #6648 merged
• Add support to load TLS certs/key dynamically from input stream #6760 merged
• Support return sequence ID when throwing Exception for async send message #6825 merged
• 
  
  • Change the default value of maxLookupRedirects of Java client to 20 #7126
  
  
• 
  
  • Limit the number of times lookup requests are redirected #7096
  
  
• 
  
  • Fix duplicated messages sent to dead letter topic #7021
  
  

Python Client

• Add deliver_at and deliver_after for the producer #6737 merged

CPP Client

• 
  
  • Support seek by time on partitioned topic #7198
  
  
• 
  
  • Refresh authentication credentials #7070
  
  
• 
  
  • Fix Result can't be serialized to string inside the library #7034
  
  
• 
  
  • Supo zstd and Snappy compression to C API #7014
  
  

Bug Fixes

• [Websocket] Fix incorrect topic URL parse #6630 merged
• [Broker] Make ZkBookieRackAffinityMapping work as expected #6917 merged
• [Broker] Fix backlog and backlog size stats keeps growing #7082
• [Java Client] Fix connection leak #6524 merged
• [Java Client] Fix message id compare between MessageId and BatchMessageId #6621 merged
• [Pulsar SQL] Fix problem with multiple zookeeper address #6947 merged
• 
  
  • [CPP Client] Fix deadlock of consumer for topics auto discovery #7206
  
  
• 
  
  • [Managed Ledger] Fix NPE on opening non-durable cursors on an empty managed ledger #7133
  
  
• 
  
  • [Java Client] Fix memory leak when create producer with not exsits topic #7120 #7124
  
  
• 
  
  • [Broker] Fixed increasing number of partitions with attached readers #7077
  
  

Others

• Work on JDK 11

[sijie]

Please link or comment the issues here.

[frank-dkvan]

hopefully this bug #4976 can be fixed in release 2.6.0 :-D

[murong00]

Both pulsar and bk should support ipv6, such as #5713

[Jennifer88huang]

Some users are expecting Pulsar transaction and KOP in the new release.

[Lanayx]

Custom functions runtime

[sijie]

@codelipenghui we need to make sure Pulsar 2.6.0 is released with JDK 11.

[Anonymitaet]

@codelipenghui

I've replaced some doc-required tags w/ doc-added since docs have been added by @Huanli-Meng and me.

4955, #6977: @Huanli-Meng has confirmed no doc needed for these PRs

6473: Confirmed w/ @tuteng : REST API doc is generated from code automatically, no need to update it manually.

[codelipenghui]

Thanks @Anonymitaet

[Anonymitaet]

@codelipenghui
The item Custom functions runtime does not have a PR link.

I've searched Custom functions runtime w/ the label milestone 2.6.0 and got the result.

Is [Pulsar Functions] Support config admin CLI in funtions runtime #6681 the one I'm looking for?

[Lanayx]

@Anonymitaet Looks like this item made it way to the list because of my proposal (which actually meant custom language runtime like .net) which doesn't seem to even be planned for implementation, so it should be removed.

[Anonymitaet]

@Lanayx OK, thanks.

[Anonymitaet]

@codelipenghui Does the last item Work on JDK 11 has a PR link?

[lhotari]

I wish that the bug in negative acknowledgements for batch messages (#6869) would also be fixed in 2.6.0 . It would make the improvement for batch message acknowledgements PIP-54 / #6052 more complete.

[codelipenghui]

@lhotari I have started the 2.6.0 candidate, we can plan #6869 in 2.6.1.

[Anonymitaet]

@codelipenghui shall we add the following item to 2.6.0 release notes as @sijie points here?

Feature

Admin

• Change the permission level of managing subscription from super-user to tenant admin https://github.com/apache/pulsar/pull/6122

or

Feature

2.6.0-SNAPSHOT (under development)

• Change the permission level of managing subscription from super-user to tenant admin https://github.com/apache/pulsar/pull/6122

?

[feeblefakie]

@codelipenghui @sijie @merlimat

I retied the issue I faced with 2.5.0 with pulsar-2.6.0-candidate-1.
http://mail-archives.apache.org/mod_mbox/pulsar-users/202003.mbox/%3CCAPDOW74LN3WtdhpG_cgCCSg9MuMmkNV6giCGD5p%3DW1wWji0W7w%40mail.gmail.com%3E

But I'm still facing inconsistent order in between consumers with the default way (splitting range).
I didn't face it if I switch it to use consistent hashing.

So, this seems not resolved yet.
https://github.com/apache/pulsar/issues/6554
https://github.com/apache/pulsar/pull/6977

If 2.6.0 will be released without fixing the above, consistent hashing should be the default ?

I created a how to reproduce doc, so please take a look.
https://github.com/feeblefakie/misc/blob/master/pulsar/HOW-TO-REPRODUCE.md

[codelipenghui]

@feeblefakie Thanks for your update. https://github.com/apache/pulsar/pull/7106 is to fix the order problem in the Key_Shared subscription, It looks there are some problems with the auto-split mechanism. I will take a look later.

[codelipenghui]

close this issue since 2.6.0 was released.