Rocksdb: kafka needs rocksdb 5.18.3, and it has no support for arm

[jiameixie]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev

Expected behavior

I modify the source code of rocksdb 5.18.3 and add aarch64 support according to https://github.com/facebook/rocksdb/issues/5559
https://github.com/facebook/rocksdb/pull/5258. And it works fine on arm64, can you release it as version 5.18.4.?

Actual behavior

kafka depends on rocksdb 5.18.3, and rocksdb does not support arm64 untill version 6.3.6. The difference between these two versions is really big.
The kafka issue link is https://issues.apache.org/jira/browse/KAFKA-9225

Steps to reproduce the behavior

Run kafka stream demo on arm64

[adamretter]

@siying Whilst I could do the Java release, I think the branching and back-porting is more of a question for you?

[siying]

5.18 is ancient RocksDB release. If just backporting make/cmake config, all the way back, I'm fine. But backporting any nontrivial C++/Java code would be risky. https://github.com/facebook/rocksdb/pull/5258/ looks find to me. I didn't get a clear PR list from https://github.com/facebook/rocksdb/issues/5559 . Can someone provide a list?

btw @jiameixie do you consider to upgrade to recent RocksDB release? It should be mostly compatible.

[adamretter]

@siying I think we would be looking at the set of https://github.com/facebook/rocksdb/pull/5258, https://github.com/facebook/rocksdb/pull/5565, and https://github.com/facebook/rocksdb/pull/5674

[siying]

I'm fine with those backporting those.

[siying]

Hmm... https://github.com/facebook/rocksdb/pull/5565 seems to check a flag that is only defined in https://github.com/facebook/rocksdb/pull/5221 , which is released only in 6.2.x and up. So skip it before 6.2.x.

[siying]

I mean "So skip it before 6.2.x?"

[jiameixie]

@siying The recent RocksDB release means a major change to Kafka. And Kafka might upgrade to rocksdb 6.x in the next major release. It's still a long way to go.

[jiameixie]

I modified the source code by the following patches.
https://github.com/facebook/rocksdb/pull/5258
https://github.com/facebook/rocksdb/pull/5674
git format-patch -1 5c76ba9dc4cbc676d8a28264b15af68c1bf06917
git format-patch -1 f2bf0b2d1e5cdd588026cec86c681efa7539ce64

The first can be applied directly by 'git am'. The second should be modified manually. Can I make a PR for this?

[siying]

We can backport the two commits all the way to 5.18. @adamretter let me know if you need help from me to do all the backporting.

[jiameixie]

Great! Assume that backport will be a v5.18.4 release?

[guyuqi]

We can backport the two commits all the way to 5.18. @adamretter let me know if you need help from me to do all the backporting.

@siying @adamretter
It's nice! Really thanks for your efforts to deliver a minor release for 5.18.

[adamretter]

@siying @guyuqi I created a new branch AR-5.18.4 from 5.18.fb. I have incorporated:

1. #5221 <- 03c7ae2
2. #5258 <- 5c76ba9
3. #5565 <- f064d74
4. #5674 <- f2bf0b2
5. #5836 <- 2367656
6. #6080 <- 1bf316e

I have one failing Java test on aarch64 that I am still investigating, I opened a separate issue for that here - https://github.com/facebook/rocksdb/issues/6234

[adamretter]

I just released version 5.18.4 to Maven Central

[JunHe77]

awesome! Thanks @adamretter