Pulsar: Error in starting bookie with multiple ledger directories

[krumo]

Expected behavior

When setting 2 directories as the storage directories, I expect the bookie could use 2 directories to store ledger files.

Actual behavior

When starting bookie with pulsar-daemon command, I get the error info in the log file as following:

2017-02-28 17:58:31,303 - ERROR [main:BookieServer@601] - Exception running bookie server :
java.lang.IllegalArgumentException: Db implementation only allows for one storage dir
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:93)
        at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:110)
        at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:508)
        at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:308)
        at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:116)
        at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:584)

Steps to reproduce

1. Start the zookeeper ensemble.
2. Modify the bookkeeper.conf and set ledgerDirectories=/mnt/datadir1/pulsar/bk-data,/mnt/datadir2/pulsar/bk-data, where datadir1 and datadir2 are located in different disks.
3. Run pulsar-daemon start bookie command.

System configuration

Pulsar version: 1.16.1

[agarman]

Pulsar uses a fork of Bookkeeper optimized for its workload. AFAIK complete compatibility with Apache Bookkeeper documentation is a nice-to-have not must-have.

If/when the Pulsar fork is merged into Apache Bookkeeper, then full configurations will likely be supported.

Until then refer to Pulsar Bookkeeper documentation.

@merlimat & co. would it make sense to pair down bookkeeper.conf to not have traps w.r.t. configurations that do not work with Pulsar Bookkeeper?

[merlimat]

If/when the Pulsar fork is merged into Apache Bookkeeper, then full configurations will likely be supported.

The effort of merging the fork into main branch of apache bookkeeper is actively ongoing :)

@merlimat & co. would it make sense to pair down bookkeeper.conf to not have traps w.r.t. configurations that do not work with Pulsar Bookkeeper?

All the bookkeeper options are actually supported. The issue with the multiple directories is that they are not supported in the DbLedgerStorage implementation that comes enabled in the config file shipped with Pulsar.

That doesn't prevent one to use the classic SortedLedgerStorage that support that feature.

So, for @krumo my suggestions could be to either:

1. Use LVM / software-raid / hardware-raid to provide a single disk view for the ledger storage volume
2. Enable SortedLedgerStorage and keep using multiple volumes
   ledgerStorageClass=org.apache.bookkeeper.bookie.SortedLedgerStorage
   The main reason for DbLedgerStorage is to support a large number of topics per bookie, which might not be the primary concern for every one.