Dolphinscheduler: [BUG][DOCKER] docker compose doesn't work

[mannharleen]

For better global communication, Please describe it in English. If you feel the description in English is not clear, then you can append description in Chinese(just for Mandarin(CN)), thx! *
*Describe the bug

Out of the box - docker compose doesn't work

To Reproduce
Steps to reproduce the behavior, for example:

1. Follow the instruction at https://dolphinscheduler.apache.org/en-us/docs/1.3.4/user_doc/docker-deployment.html

Expected behavior
Master and Worker should show up under "Monitor" tab

Screenshots

zookeeper is working fine:

Which version of Dolphin Scheduler:
-1.3.4

Additional context
Logs:

dolphinscheduler-zookeeper     | 2021-02-19 11:43:38,838 [myid:1] - WARN  [NIOWorkerThread-4:NIOServerCnxn@373] - Close of session 0x0
dolphinscheduler-zookeeper     | java.io.IOException: Len error 1195725856
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:541)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:332)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
dolphinscheduler-zookeeper     |        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
dolphinscheduler-zookeeper     |        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
dolphinscheduler-zookeeper     |        at java.base/java.lang.Thread.run(Thread.java:834)
dolphinscheduler-zookeeper     | 2021-02-19 11:43:40,234 [myid:1] - WARN  [NIOWorkerThread-2:NIOServerCnxn@373] - Close of session 0x0
dolphinscheduler-zookeeper     | java.io.IOException: Len error 1195725856
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:541)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:332)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
dolphinscheduler-zookeeper     |        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
dolphinscheduler-zookeeper     |        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
dolphinscheduler-zookeeper     |        at java.base/java.lang.Thread.run(Thread.java:834)
dolphinscheduler-zookeeper     | 2021-02-19 11:43:45,268 [myid:1] - WARN  [NIOWorkerThread-1:NIOServerCnxn@373] - Close of session 0x0
dolphinscheduler-zookeeper     | java.io.IOException: Len error 1195725856
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:541)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:332)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
dolphinscheduler-zookeeper     |        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
dolphinscheduler-zookeeper     |        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
dolphinscheduler-zookeeper     |        at java.base/java.lang.Thread.run(Thread.java:834)
dolphinscheduler-zookeeper     | 2021-02-19 13:07:23,703 [myid:1] - INFO  [NIOWorkerThread-3:NIOServerCnxn@507] - Processing ruok command from /172.22.0.4:45996
dolphinscheduler-zookeeper     | 2021-02-19 13:07:23,707 [myid:1] - INFO  [NIOWorkerThread-4:NIOServerCnxn@507] - Processing srvr command from /172.22.0.4:45998
dolphinscheduler-zookeeper     | 2021-02-19 13:07:23,711 [myid:1] - INFO  [NIOWorkerThread-2:NIOServerCnxn@507] - Processing wchs command from /172.22.0.4:46000
dolphinscheduler-zookeeper     | 2021-02-19 13:07:23,714 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@507] - Processing cons command from /172.22.0.4:46002

dolphinscheduler-worker        | init env variables
dolphinscheduler-worker        | generate app config
dolphinscheduler-worker        | generate nginx config
dolphinscheduler-worker        | connect remote zookeeper
dolphinscheduler-worker        | test postgresql service
dolphinscheduler-worker        | connect postgresql service
dolphinscheduler-worker        | import sql data
dolphinscheduler-worker        |  _ _   |_  _ _|_. ___ _ |    _
dolphinscheduler-worker        | | | |\/|_)(_| | |_\  |_)||_|_\
dolphinscheduler-worker        |      /               |
dolphinscheduler-worker        |                         3.2.0
dolphinscheduler-worker        | 11:17:57.356 [main] INFO ru.yandex.clickhouse.ClickHouseDriver - Driver registered
dolphinscheduler-worker        | 11:17:58.118 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
dolphinscheduler-worker        | 11:17:58.258 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
dolphinscheduler-worker        | 11:17:58.286 [main] INFO org.apache.dolphinscheduler.dao.upgrade.DolphinSchedulerManager - The database has been initialized. Skip the initialization step
dolphinscheduler-worker        | 11:17:58.286 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - init DolphinScheduler finished
dolphinscheduler-worker        | 11:17:58.347 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
dolphinscheduler-worker        | 11:17:58.363 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
dolphinscheduler-worker        | 11:17:58.364 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - upgrade DolphinScheduler finished
dolphinscheduler-worker        | 11:17:58.364 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - create DolphinScheduler success
dolphinscheduler-worker        | start worker-server
dolphinscheduler-worker        | Begin stop worker-server......
dolphinscheduler-worker        | no worker-server to stop
dolphinscheduler-worker        | End stop worker-server.
dolphinscheduler-worker        | Begin start worker-server......
dolphinscheduler-worker        | starting worker-server, logging to /opt/dolphinscheduler/bin/../logs/dolphinscheduler-worker-server-94a7255fe9f0.out
dolphinscheduler-worker        | nohup /usr/lib/jvm/java-1.8-openjdk/bin/java -Dlogging.config=classpath:logback-worker.xml -Ddruid.mysql.usePingMethod=false -server -Xms2g -Xmx2g -Xmn1g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m  -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=dump.hprof -classpath /opt/dolphinscheduler/bin/../conf:/opt/dolphinscheduler/bin/../lib/* org.apache.dolphinscheduler.server.worker.WorkerServer > /opt/dolphinscheduler/bin/../logs/dolphinscheduler-worker-server-94a7255fe9f0.out 2>&1 &
dolphinscheduler-worker        | End start worker-server.
dolphinscheduler-worker        | start logger-server
dolphinscheduler-worker        | Begin stop logger-server......
dolphinscheduler-worker        | no logger-server to stop
dolphinscheduler-worker        | End stop logger-server.
dolphinscheduler-worker        | Begin start logger-server......
dolphinscheduler-worker        | starting logger-server, logging to /opt/dolphinscheduler/bin/../logs/dolphinscheduler-logger-server-94a7255fe9f0.out
dolphinscheduler-worker        | nohup /usr/lib/jvm/java-1.8-openjdk/bin/java  -server -Xms1g -Xmx1g -Xmn500m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m  -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=dump.hprof -classpath /opt/dolphinscheduler/bin/../conf:/opt/dolphinscheduler/bin/../lib/* org.apache.dolphinscheduler.server.log.LoggerServer > /opt/dolphinscheduler/bin/../logs/dolphinscheduler-logger-server-94a7255fe9f0.out 2>&1 &
dolphinscheduler-worker        | End start logger-server.
dolphinscheduler-worker        | tail begin
dolphinscheduler-worker        | [INFO] 2021-02-18 19:05:52.134 org.apache.dolphinscheduler.service.zk.ZookeeperCachedOperator:[55] - add listener to zk path: /dolphinscheduler

dolphinscheduler-master        | init env variables
dolphinscheduler-master        | generate app config
dolphinscheduler-master        | generate nginx config
dolphinscheduler-master        | connect remote zookeeper
dolphinscheduler-master        | test postgresql service
dolphinscheduler-master        | connect postgresql service
dolphinscheduler-master        | import sql data
dolphinscheduler-master        |  _ _   |_  _ _|_. ___ _ |    _
dolphinscheduler-master        | | | |\/|_)(_| | |_\  |_)||_|_\
dolphinscheduler-master        |      /               |
dolphinscheduler-master        |                         3.2.0
dolphinscheduler-master        | 11:17:56.831 [main] INFO ru.yandex.clickhouse.ClickHouseDriver - Driver registered
dolphinscheduler-master        | 11:17:57.389 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
dolphinscheduler-master        | 11:17:57.530 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
dolphinscheduler-master        | 11:17:57.664 [main] INFO org.apache.dolphinscheduler.dao.upgrade.DolphinSchedulerManager - The database has been initialized. Skip the initialization step
dolphinscheduler-master        | 11:17:57.665 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - init DolphinScheduler finished
dolphinscheduler-master        | 11:17:57.753 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
dolphinscheduler-master        | 11:17:57.766 [main] DEBUG com.alibaba.druid.pool.PreparedStatementPool - stmt enter cache
dolphinscheduler-master        | 11:17:57.768 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - upgrade DolphinScheduler finished
dolphinscheduler-master        | 11:17:57.768 [main] INFO org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - create DolphinScheduler success
dolphinscheduler-master        | start master-server
dolphinscheduler-master        | Begin stop master-server......
dolphinscheduler-master        | no master-server to stop
dolphinscheduler-master        | End stop master-server.
dolphinscheduler-master        | Begin start master-server......
dolphinscheduler-master        | starting master-server, logging to /opt/dolphinscheduler/bin/../logs/dolphinscheduler-master-server-be835eb5e254.out
dolphinscheduler-master        | nohup /usr/lib/jvm/java-1.8-openjdk/bin/java -Dlogging.config=classpath:logback-master.xml -Ddruid.mysql.usePingMethod=false -server -Xms4g -Xmx4g -Xmn2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m  -Xss512k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:LargePageSizeInBytes=128m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=dump.hprof -classpath /opt/dolphinscheduler/bin/../conf:/opt/dolphinscheduler/bin/../lib/* org.apache.dolphinscheduler.server.master.MasterServer > /opt/dolphinscheduler/bin/../logs/dolphinscheduler-master-server-be835eb5e254.out 2>&1 &
dolphinscheduler-master        | End start master-server.
dolphinscheduler-master        | tail begin
dolphinscheduler-master        | [INFO] 2021-02-18 19:04:56.075 org.apache.zookeeper.ClientCnxn:[1156] - Client session timed out, have not heard from server in 26750ms for sessionid 0x100028582e10000, closing socket connection and attempting reconnect

Whats going wrong?

[chengshiwen]

@mannharleen The docker image 1.3.4 of dolphinscheduler has problems and has not been updated for a long time. In 1.3.5 there many updates about docker and k8s, please visit the 1.3.5 Release Notes and the latest readme English or Chinese.
Attention: Both the official docker image in https://hub.docker.com/r/apache/dolphinscheduler and the official document in http://dolphinscheduler.apache.org/en-us/docs/1.3.5/user_doc/docker-deployment.html have not been updated due to the Chinese New Year. They will be updated soon.

[mannharleen]

Ah I see. I will try with 1.3.5 and close this issue if the same issues don’t crop up again.

Thanks!

[chengshiwen]

@mannharleen The docker image 1.3.5 of dolphinscheduler has been pushed to docker hub named apache/dolphinscheduler:latest and apache/dolphinscheduler:1.3.5. You can download the docker-compose.yml 1.3.5 and execute docker pull apache/dolphinscheduler:latest (Note that the image id is a9514c9fe1d4) and docker-compose up -d. Wait a minute to check http://127.0.0.1:12345/dolphinscheduler with username/password admin/dolphinscheduler123.

Attention: You can create a tenant named root and associate the admin user with the root tenant in docker. Next, start the joyful test.

[dailidong]

@mannharleen , hi , we created Slack workspace of Apache DolphinScheduler Community some days ago,
welcome to join through invitation url: https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-l8k90ceu-wwUfobaDkJxjzMfZp4y1Ag

[mannharleen]

@chengshiwen Still facing an error:

dolphinscheduler-zookeeper     | EndOfStreamException: Unable to read additional data from client, it probably closed the socket: address = /172.22.0.7:55722, session = 0x10007306bcf0001
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:163)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:326)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
dolphinscheduler-zookeeper     |        at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
dolphinscheduler-zookeeper     |        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
dolphinscheduler-zookeeper     |        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
dolphinscheduler-zookeeper     |        at java.base/java.lang.Thread.run(Thread.java:834)

Using the latest image now:

[chengshiwen]

@mannharleen Re-try:

1. docker-compose down -v
2. check version of docker-compose.yml is 1.3.5, it's better to download it from docker-compose.yml
3. docker-compose up -d
4. watch -n 1 docker ps until healthy if watch command exists
5. docker logs -f dolphinscheduler-api until listening :12345 occurs

[chengshiwen]

I notice that you are using Windows. If the problem still exists, please try to run it on Linux/Mac. And tell us your environment, I will try to reproduce the problem, thx

[dailidong]

@mannharleen hi, does zookeeper problem solved ? can I help you ?

[mannharleen]

Am really impressed by the fact that you guys (@dailidong and @chengshiwen) are actually following up!!

I didn’t get the chance yet, but will give it all a try again soon.

[chengshiwen]

We have tried to reproduce the issue on Windows with Docker Toolbox, but there is no such problem and it runs normally😂
If possible, please provide more environment information to help us troubleshoot the problem, thank you very much!🙏

[chengshiwen]

Close this issue, since no updats for a long time and unable to reproduce

[chengshiwen]

@mannharleen There is another possible reason for this problem, which is that the memory is too small. To solve this problem, please refer to https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/docker-deployment.html and check 0. Requirements