Pulsar: Error open RocksDB database when 'Set up a standalone Pulsar in Docker'

[junlia]

Hello,

I am following the instruction on Set up a standalone Pulsar in Docker to setup Pular on windows 10 with the following command on both cmd.exe and powershell:

docker run -it -p 6650:6650 -p 8080:8080 -v "c:\docker\pulsar:/pulsar/data" apachepulsar/pulsar:2.4.1 bin/pulsar standalone

I am getting this error as appeared to be fixed already - per https://github.com/apache/pulsar/issues/3031:

01:24:26.913 [main] INFO  org.apache.bookkeeper.discover.ZKRegistrationManager - INSTANCEID not exists in zookeeper. Not considering it for data verification
01:24:26.998 [main] INFO  org.apache.bookkeeper.bookie.Bookie - instantiate ledger manager org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory
01:24:27.019 [main] ERROR org.apache.bookkeeper.bookie.Journal - Problems reading from data/standalone/bookkeeper0/current/lastMark (this is okay if it is the first time starting this bookie
01:24:27.021 [main] INFO  org.apache.bookkeeper.bookie.Bookie - Using ledger storage: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
01:24:27.024 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Started Db Ledger Storage
01:24:27.025 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Number of directories: 1
01:24:27.025 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Write cache size: 1024 MB
01:24:27.025 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage -  - Read Cache: 1024 MB
01:24:27.028 [main] INFO  org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Creating single directory db ledger storage on data/standalone/bookkeeper0/current
01:24:27.139 [main] INFO  org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
01:24:27.172 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
java.io.IOException: Error open RocksDB database
        at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:182) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:58) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:160) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:148) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:128) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:790) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:106) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:272) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
        at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:396) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
        at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:259) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
Caused by: org.rocksdb.RocksDBException: While fsync: a directory: Invalid argument
        at org.rocksdb.RocksDB.open(Native Method) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?]
        at org.rocksdb.RocksDB.open(RocksDB.java:231) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?]
        at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:179) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
        ... 13 more
01:24:27.182 [Thread-1] ERROR org.apache.pulsar.PulsarStandaloneStarter - Shutdown failed: null

[junlia]

Based on the comment from Sijie Guo on https://github.com/apache/pulsar/issues/4684:

Currently I don't think Pulsar support running on Windows very well. Because there are a few dependencies are using native code. The bindings we are shipping probably doesn't contain the native libraries built for windows.

[cdbartholomew]

This is a Docker configuration issue. The command in the documentation:

$ docker run -it \
  -p 6650:6650 \
  -p 8080:8080 \
  -v "$PWD/data:/pulsar/data".ToLower() \
  apachepulsar/pulsar:2.4.1 \
  bin/pulsar standalone

Is faulty. Here's why.

When the pulsar Docker image is built, it defines two volumes:

VOLUME  ["/pulsar/conf", "/pulsar/data"]

This means at run time the Docker image expects an externally mounted volume for those the two paths. When you do "docker run" without specifying any storage it will automatically create anonymous Docker volumes. It will look something like this:

        "Mounts": [
            {
                "Type": "volume",
                "Name": "c59b3a5ad98eafc8efcbe6de1f63b1a31693564996a8dcc6de60f02867015a51",
                "Source": "/var/lib/docker/volumes/c59b3a5ad98eafc8efcbe6de1f63b1a31693564996a8dcc6de60f02867015a51/_data",
                "Destination": "/pulsar/conf",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "b7600b9314de9bf8ed573ef697fa012b9d2574074fbea45c2831b3e0a854da30",
                "Source": "/var/lib/docker/volumes/b7600b9314de9bf8ed573ef697fa012b9d2574074fbea45c2831b3e0a854da30/_data",
                "Destination": "/pulsar/data",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],

The problem is the -v option in the command. This tries to create a bind mount at the same path in the container as one of the pre-specified volume mount points (/pulsar/data). This creates two mounts on the same path. Docker doesn't barf on this (for some reason), but it obviously makes the file system behave strangely, causing RocksDB to barf.

PR #3918 mentions adding the ToLower() function to -v option to "fix" the issue. This doesn't fix the issue at all. It avoids the error because it ends mangling the -v option so that it mounts a volume at /pulsar/data.ToLower(). This doesn't collide with the pre-defined path so RocksDB works, but it doesn't work at all as intended, since Pulsar is configured to expect its data to be in the /pulsar/data directory so /pulsar/data.ToLower() is never used.

The fix for this is simple, instead of messing around with bind mounts, give it a volume mount like it expects. And since we presumably want the data to persist between "docker run" commands, we just have to give the volume a name.

docker run -it \
-p 6650:6650 -p 8080:8080  \
--mount source=pulsardata,target=/pulsar/data \
apachepulsar/pulsar:2.4.1 \
bin/pulsar standalone

Here is what docker inspect gives:

        "Mounts": [
            {
                "Type": "volume",
                "Name": "6a41856b98b36f81d1bf1d1d196a59a026c82dd12aa50cee5bdf82295e7f670c",
                "Source": "/var/lib/docker/volumes/6a41856b98b36f81d1bf1d1d196a59a026c82dd12aa50cee5bdf82295e7f670c/_data",
                "Destination": "/pulsar/conf",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "pulsardata",
                "Source": "/var/lib/docker/volumes/pulsardata/_data",
                "Destination": "/pulsar/data",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],

The other advantage of using a named Docker volume is that we don't have to mess around with path specifications, so no need for the $PWD variable, which is defined in PowerShell but not CommandPrompt (CMD). I am able to run the above command using either PowerShell or CMD and it works reliably for me on Docker Desktop for Windows.

Since the image expects the config data to be persisted, we should probably specify a name for that volume too, like this:

docker run -it \
-p 6650:6650 -p 8080:8080  \
--mount source=pulsardata,target=/pulsar/data \
--mount source=pulsarconf,target=/pulsar/conf \
apachepulsar/pulsar:2.4.1 \
bin/pulsar standalone

I think just a documentation change is needed to resolve this issue. @junlia can you confirm whether my revised command above works for you?

I am happy to put in a PR with the documentation change once this is confirmed to work outside my environment.

[junlia]

@cdbartholomew It works for me with your revised command in PowerShell. A PR to update the installation guide will help others from windows world like me for sure

[cdbartholomew]

Created pull request #5419.

[kaonick]

I try below in windows:
docker run -it -p 6650:6650 -p 8080:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:latest bin/pulsar standalone

Could run ok.
But How can I mount directory in my windows like e:/data ?

[sijie]

This has been fixed by #5419