Pulsar: Mac(Mojave). Standalone quits. Failed to start pulsar service.

[RAbraham]

Describe the bug
Hi,
When I run bin/pulsar standalone it quits.

To Reproduce
Steps to reproduce the behavior:

wget https://archive.apache.org/dist/pulsar/pulsar-2.4.1/apache-pulsar-2.4.1-bin.tar.gz
wget https://archive.apache.org/dist/pulsar/pulsar-2.4.1/connectors/pulsar-io-kafka-connect-adaptor-2.4.1.nar
tar zxf apache-pulsar-2.4.1-bin.tar.gz && cd apache-pulsar-2.4.1 && mkdir connectors && cp ../pulsar-io-kafka-connect-adaptor-2.4.1.nar connectors
bin/pulsar standalone

Expected behavior
I expected it to continue to run

Desktop (please complete the following information):

• OS: MacOS(Mojave)

Additional context
The error dump is in error.txt. My etc/hosts file just in case

# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost
127.0.0.1       SomeName-MacBook-Pro-Personal.local
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

What was published in my console in addition was:

bin/pulsar standalone > error.txt
objc[29991]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java (0x1020514c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x1020d04e0). One of the two will be used. Which one is undefined.
[AppClassLoader@58644d46] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
[AppClassLoader@58644d46] info register classloader sun.misc.Launcher$AppClassLoader@58644d46
[AppClassLoader@58644d46] info using configuration file:/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-2.4.1.jar!/META-INF/aop.xml
[AppClassLoader@58644d46] info using configuration file:/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar!/META-INF/aop.xml
[AppClassLoader@58644d46] info register aspect org.apache.pulsar.zookeeper.FinalRequestProcessorAspect
[AppClassLoader@58644d46] info register aspect org.apache.pulsar.zookeeper.ZooKeeperServerAspect
[AppClassLoader@58644d46] info register aspect org.apache.pulsar.zookeeper.SerializeUtilsAspect
[AppClassLoader@58644d46] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
[ScriptLoader@517d4a0d] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
[ScriptLoader@517d4a0d] info register classloader jdk.nashorn.internal.runtime.ScriptLoader@517d4a0d
[ScriptLoader@517d4a0d] info using configuration file:/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-2.4.1.jar!/META-INF/aop.xml
[ScriptLoader@517d4a0d] info using configuration file:/Users/rabraham/Documents/dev/trial/pulsar-trial/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar!/META-INF/aop.xml
[ScriptLoader@517d4a0d] info register aspect org.apache.pulsar.zookeeper.FinalRequestProcessorAspect
[ScriptLoader@517d4a0d] info register aspect org.apache.pulsar.zookeeper.ZooKeeperServerAspect
[ScriptLoader@517d4a0d] info register aspect org.apache.pulsar.zookeeper.SerializeUtilsAspect
[ScriptLoader@517d4a0d] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
[AppClassLoader@58644d46] info processing reweavable type org.apache.zookeeper.server.ZooKeeperServer: org/apache/zookeeper/server/ZooKeeperServer.java
[AppClassLoader@58644d46] info successfully verified type org.apache.pulsar.zookeeper.ZooKeeperServerAspect exists.  Originates from org/apache/pulsar/zookeeper/ZooKeeperServerAspect.java
[AppClassLoader@58644d46] info processing reweavable type org.apache.zookeeper.server.FinalRequestProcessor: org/apache/zookeeper/server/FinalRequestProcessor.java
[AppClassLoader@58644d46] info successfully verified type org.apache.pulsar.zookeeper.FinalRequestProcessorAspect exists.  Originates from org/apache/pulsar/zookeeper/FinalRequestProcessorAspect.java

[aahmed-se]

it runs fine for me , are you saying it runs for a while and then quits ?

[RAbraham]

Yes, it just quits. If you look at the error.txt file, it says the following errors. Do you get the same errors too?

14:18:42.974 [client-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.clients.impl.channel.StorageServerChannelManager - Added range server (hostname: "127.0.0.1"
port: 4181
) into the channel manager.
14:21:12.637 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
io.grpc.StatusRuntimeException: NOT_FOUND
    at io.grpc.Status.asRuntimeException(Status.java:530) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:482) ~[io.grpc-grpc-stub-1.18.0.jar:1.18.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:699) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_05]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_05]
    at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_05]

and

14:21:12.651 [Thread-1] INFO  org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Successfully stopped storage container (1).
14:21:12.653 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x16d07cc441c0004 type:delete cxid:0x16 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/stream/controller/_c_026af608-f914-423b-9848-9f7f1958abd0-lock-0000000002 Error:KeeperErrorCode = NoNode for /stream/controller/_c_026af608-f914-423b-9848-9f7f1958abd0-lock-0000000002
14:21:12.652 [Curator-LeaderSelector-0] ERROR org.apache.curator.framework.recipes.leader.LeaderSelector - The leader threw an exception
java.lang.InterruptedException: null
    at java.lang.Object.wait(Native Method) ~[?:1.8.0_05]
    at java.lang.Object.wait(Object.java:502) ~[?:1.8.0_05]
    at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1411) ~[org.apache.pulsar-pulsar-zookeeper-2.4.1.jar:2.4.1]
    at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:880) ~[org.apache.pulsar-pulsar-zookeeper-2.4.1.jar:2.4.1]
    at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:274) ~[org.apache.curator-curator-framework-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:268) ~[org.apache.curator-curator-framework-4.0.1.jar:4.0.1]
    at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) ~[org.apache.curator-curator-client-4.0.1.jar:?]
    at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100) ~[org.apache.curator-curator-client-4.0.1.jar:?]
    at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:265) ~[org.apache.curator-curator-framework-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:249) ~[org.apache.curator-curator-framework-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:34) ~[org.apache.curator-curator-framework-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:347) ~[org.apache.curator-curator-recipes-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:124) ~[org.apache.curator-curator-recipes-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:154) ~[org.apache.curator-curator-recipes-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:449) [org.apache.curator-curator-recipes-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:466) [org.apache.curator-curator-recipes-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:65) [org.apache.curator-curator-recipes-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:246) [org.apache.curator-curator-recipes-4.0.1.jar:4.0.1]
    at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:240) [org.apache.curator-curator-recipes-4.0.1.jar:4.0.1]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_05]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_05]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_05]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_05]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_05]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_05]
14:21:12.654 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x16d07cc441c0007

[aahmed-se]

keeps running for me, probably something to do with your hosts file, but don't see any problems locally.

[RAbraham]

Thanks for your quick reply. I'll work on the docker image then.

[RAbraham]

the off chance, someone else has the same issue. I resolved it by:

• moving to openjdk8(may not be required. but it exposed the issue with a better error message than the default mac java 8, I think)
• add 127.0.0.1 <laptop name>.local to /etc/hosts or whatever name it errors out on your laptop
• dscacheutil -flushcache

[kevenYLi]

I encountered a similar problem and solved by doing as your second step. @RAbraham Thanks for your solution.

I was trying to set up a standalone Pulsar locally according to this dochttps://pulsar.apache.org/docs/en/standalone/#start-pulsar-standalone, but failed.
ERROR: caused by: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'liyi.local' after 2 queries. The details can be found in liyi_error.txt.
liyi_error.txt

[merlimat]

@kevenYLi thanks for the complete logs. it seems related to the Function worker that tries to use hostname. I think we should fix this by making sure that, in standalone mode, it will always attempt to use localhost instead.

[baptistemesta]

the off chance, someone else has the same issue. I resolved it by:

• moving to openjdk8(may not be required. but it exposed the issue with a better error message than the default mac java 8, I think)
• add 127.0.0.1 <laptop name>.local to /etc/hosts or whatever name it errors out on your laptop
• dscacheutil -flushcache

It was indeed the 2 last point: add the address in /etc/hosts and dscacheutil -flushcache
Thank you!

[marinedabare]

Thanks @RAbraham ! For me, it works only with the 3 steps (with openjdk8).