Webodm: Can't connect to application port from browser

[mspringfield]

So I know I am missing something here that's probably simple. WebODM is deployed and seems to be up and running just fine. It is on a VM in my lab, however, I am unable to connect to it from my laptop browser.

It appears that the container ip is bound to 0.0.0.0 on the host on the correct port. If I nmap port 8000 from the HOST then it shows open for localhost, but not for the ip (that shows filtered) [host ip is 1.1.1.29]. And when the nmap sources from my laptop (on the same subnet) it shows closed:

Containers running

Nmap from the host where webodm is running:

Nmap from laptop on same subnet

I have tried this with both Ubuntu 14.0.4 and 16.0.4. UFW is stopped and disabled.

I have tried to change the network_mode: to host in the docker-compose file but the result is the same.

Again, I know I'm probably missing something obvious here but I can't seem to nail it down. Any help is appreciated!

[pierotofy]

Hi @mspringfield !

I'm going to guess that this is an iptables configuration problem related to docker.

https://docs.docker.com/engine/userguide/networking/default_network/container-communication/

Probably the ip_forward variable (/proc/sys/net/ipv4/ip_forward), but I would read the link above and see if you find the answer there.

[mspringfield]

Hey @pierotofy! Thanks for the reply!

This was actually something that I had NOT tried so I was hopeful when I saw this. However, the ip_forward variable was already set to 1 on both 14.0.4 and 16.0.4.

[pierotofy]

I just noticed that your netstat command shows tcp6, so you would have to check the net.ipv6.conf.all.forwarding variable.

If that doesn't solve it, perhaps looking at Docker's IPv6 guide might help: https://docs.docker.com/engine/userguide/networking/default_network/ipv6/

[mspringfield]

I tried enabling forwarding on ipv6 as well with no luck. So I disabled all ipv6. Apparently the way that netstat represents the connections is "tcp6" means both 4 & 6. tcp4 just means 4. So even with v6 disabled my netstat shows the same.

Spun a new ubuntu 16.0.4 vm on azure with default docker install by the docker docs ( https://docs.docker.com/engine/installation/linux/ubuntu/#install-using-the-repository). Cloned webodm and built. Experiencing the exact same behavior. Seems to be a straight docker forwarding issue, still troubleshooting.

[pierotofy]

Mm, I've run ./webodm.sh start from a clean install on my local machine, then opened the webodm interface from a remote host and it worked OK. I'm using Arch.

On Azure the only thing that comes to mind is to make sure that the proper ports are open in the firewall settings. This looks like a problem related to docker more than WebODM, but I'll keep this open because I'd like to know what's causing the problem. If you find a solution, would you please share it with us?

[mspringfield]

Webodm.sh start on my local machine (mac) works great. Forwarding seems to be working just fine. Azure security groups are open and allowing connectivity, but with Ubuntu, still getting no luck on forwarding. Going to launch a CentOs version to see if I get any difference.

[mspringfield]

Ok, still working on this. I have ruled out the Azure network security group conflict by portmapping the webodm host from the same subnet by another vm. Same result -- shows :8000 = closed. Local nmap shows :8000 as open.

I can deploy just fine on docker Mac and connect to it from a remote host.

Back to the local lab deployment:
Default Ubuntu 16.0.4LTS install, and installed docker per Docker's instructions. Install python, pip, update pip. Then webodm.sh start -- everything seems to download, install and run the containers just fine. I receive the "Congratulations message" and can nmap port 8000 on the server and it shows open. Nmap from another machine on the same subnet shows the port as closed and I cannot connect to it with a browser from the same subnet. UFW is disabled and off. TCPDump from client shows that the host is resetting the connection (RST).

Now for the strangeness.
docker pull nginx
docker run --name docker-nginx -p 80:80 nginx
Pull and run nginx container on the same host and I can connect to it fine from client on same subnet.

Yet the iptables rules and netstat seem to be similar to webapp:

With the nginx container and webodm running:
nmap localhost (From the host):

Shows both 80 and 8000 as open.

nmap -p 80,8000 webodm (from client on same subnet)

Shows 8000 as closed.

I have tried manually setting the host ip in the docker-compose.yml, I have tried setting the port to another port (80, but while the nginx container is obviously not running). This seems to be consistent behavior for default Ubuntu and Docker install. I have installed at least 5 times in the lab.

Leave it to me to find the crazy issues! Really appreciate the assistance!

[pierotofy]

@mspringfield thanks for the detailed report of your findings. This seems really odd. I think we've covered the troubleshooting steps, and this could well be a bug in docker itself. A quick search led me to this issue https://github.com/docker/compose/issues/3055, which says the problem was fixed in docker 1.11. I think Ubuntu 16.04 is stuck at version 1.5, so perhaps updating docker to a newer version will solve the problem?

[mspringfield]

Thanks @pierotofy -- I let webodm install docker-compose so it installs the latest version. I'm currently running docker-compose version 1.13.0, build 1719ceb. I'm really surprised as succinctly as I can reproduce this, nobody else has come across this with docker or docker-compose.

I'll keep at it until I find a solution and report back. Let me know if you have any additional ideas.

Thanks!

[pierotofy]

@mspringfield docker-compose is separate from docker. What's the output of docker --version?

[mspringfield]

@pierotofy Completely understand. I posted the docker version above:

That article was posted more than a year ago. This is a brand-new fresh install of Docker as of this morning.

Thanks!

[pierotofy]

Ups, sorry I've missed that. Something else to try: if you stop WebODM and run:

docker run --name docker-nginx -p 8000:80 nginx

Does port 8000 show up as open on from the same subnet by another vm?

[mspringfield]

docker run -p 8000:80 nginx
Port shows open from another host on the vlan:

Connection to nginx works over 8000:

Is there a way we can launch the containers manually to see if it's an issue with docker-compose?

[pierotofy]

If you run docker images you should see two images, "webodm_db" and "webodm_app". You should be able to run them with docker run -d webodm_db and docker run --rm -ti -p 8000:8000 webodm_app. If the app container cannot connect to the database, you'll need to modify your local_settings.py (and mount the file via -v /path/to/webodm/webodm/local_settings.py:/webodm/webodm/local_settings.py) to point to your local host and expose the database port via docker run -d -p 5432:5432 webodm_db).

[mspringfield]

The db container starts but then stops shortly after. However, when I run the webapp:
docker run --rm -ti -p 8000:8000 webodm_app

It provides a python prompt and doesn't fully start the app (because there is no db), however, it does open the port and I can connect and see nginx running.

[pierotofy]

You shouldn't see nginx, webodm runs on the default django development server app.

[mspringfield]

Hah! Sorry about that wrong dns hostname I was connecting to....(too many vm's spun up). Any idea why the db container disappears shortly after starting?

[mspringfield]

Built a simple docker-compose.nginx.yml see if spinning up the nginx container via docker-compose would yield the same results or at least point the finger at docker-compose:
docker-compose -f docker-compose.nginx.yml up

Port 80 is open to the subnet and able to be hit from my laptop:

Nmap from the container host (both 80 and 8000 show open):

Nmap from laptop (port 8000 shows closed):

[mspringfield]

Further research into this....the nginx container I created and can connect to is actually on the same docker network that webodm is on. So I did a tcpdump on the bridge that contains these containers and I can see the traffic coming into the webodm conainer and a reset coming back from the container. The traffic destined for the nginx container flows fine.

So it seems that the reset was coming FROM the webodm container. Docker networking is passing the connection through.

So I edited the docker-compose.nodeodm.yml file and added a listen port:

And Bazinga! I'm able to nmap that port -- shows open and navigate to the node from a browser on the network.

Is there something that can be causing the Django web server on the webodm container to not allow connections due to natting issues or rfc1918 address space? Again, it works fine when deployed on a mac. However, there seems to be something that this configuration doesn't like when deployed on a default Docker implementation on Ubuntu or Centos.

I have double checked the settings.py(ALLOWED_HOSTS = ['*']) and start.sh (manage.py runserver 0.0.0.0:8000). But I am no django expert at all.

I appreciate the help!

[pierotofy]

The ALLOWED_HOSTS and bind address were the only two things that came to my mind in regard to Django, but you have those covered.

This could be a problem with Django's simple_server implementation (which hasn't been battle tested, and it's not recommended for use in production). Perhaps this is due to the uncommon IP addresses of your network (1.1.1.1).

If you download a Python3 image from docker, open port 8000 and run the following script:

from wsgiref.simple_server import make_server, demo_app

httpd = make_server('0.0.0.0', 8000, demo_app)
print("Serving HTTP on port 8000...")

# Respond to requests until process is killed
httpd.serve_forever()

# Alternative: serve one request, then exit
httpd.handle_request()

Do you get a connection reset?

A thing that needs to be done is to install a proper production server to support WebODM (such as http://gunicorn.org/). I haven't had time to do it, but it's probably important at this point.

[mspringfield]

FYI, I did think that the unannounced APNIC address space that I use in my lab could have been the issue as well, but I created a new zone today, re-ip'd and tried from an RFC 1918 space (10.0.20.0/24) and came up with the same result.

I will try the Python Image approach and provide feedback.

Thanks!

[mspringfield]

Dockerfile

FROM python

RUN mkdir /myserver
WORKDIR /myserver

COPY server.py myserver/server.py
CMD ["python", "myserver/server.py"]
EXPOSE 8000

docker run -p 8000:8000 pythonserver

I do not get a connection reset. Port is open and I can connect to it from a browser on the subnet.

Does that use the same foundation as the django simple web server?

[pierotofy]

Yes: https://github.com/django/django/blob/master/django/core/servers/basehttp.py#L14

Mm, does it work properly even if you build a docker compose file similar to webodm's to launch the image?

I wish I could reproduce this so that I could be of more help, it seems to be one of those hard bugs to catch.

[mspringfield]

version: '2'
services:
  myserver:
    image: pythonserver
    container_name: pythonserver-1
    ports:
      - "8000:8000"

docker-compose -f docker-compose.myserver.yml up
Port is open and able to connect to over the network.

It's a strange one for sure, but seems to be consistent with Docker and Ubuntu 14.0.4, 16.0.4 or even CentOS 7.3. Repeatable on my local VMWare stack and in Azure as well. I have reinstalled at least a half dozen times to make sure I'm not missing anything. I have also tried different ports on the webodm container itself, forced bridge networking, etc. Same result. For some reason I can always see the port from the local host, but it sends resets from across the network.

Leave it to me to find the weird ones!

~m

[pierotofy]

@mspringfield I've just opened a PR #175 to use gunicorn + nginx as the http server. If you try to checkout the gunicorn branch from my repo and run ./webodm.sh update && ./webodm.sh start can you connect?

[mspringfield]

I am getting a 500 server error, however, the port is open and I can see it from the local network!

Looks like it's having an issue connecting to the db:
django.db.utils.OperationalError: could not translate host name "db" to address: Name or service not known
The db container is up and running though.

[pierotofy]

Mm, does ./webodm.sh stop && ./webodm.sh start fix it?

[mspringfield]

Same result. Reboot, update again and deployed on two different docker instances: Ubuntu 16.0.4 and CentOS 7.3. Both yield same Server error (500).

Logs:
https://gist.github.com/mspringfield/03374019e4a6635c0c72f0ad40c2daeb

[pierotofy]

Try:

docker exec -ti b8d3 tail -f /tmp/nginx.error.log

Replace b8d3 with the hash of the running webodm_webapp container.

I've just tested this on Arch Linux, MacOS and things seem to work OK there.

You can also try to read /tmp/nginx.access.log

[pierotofy]

From your log, it seems like your db image is not built properly? Have you changed the name of the database at some point?

Do a docker volume ls and remove the webodm_dbdata volume with docker volume rm webodm_dbdata, then restart webodm via the webodm.sh script.

[mspringfield]

The nginx.error.log is empty. I have made no changes to any of the code or database. I removed the volume, image and container and ran webodm.sh start again and I am receiving the same error. When you execute ./webodm.sh update does it rebuild everything with a --no-cache option?

[pierotofy]

It does. django.db.utils.OperationalError: FATAL: database "webodm_dev" does not exist means that the connection to the db host succeeded (good), but for some reason the webodm_dev database is not there. It should have been created as part of https://github.com/OpenDroneMap/WebODM/blob/master/db/Dockerfile via https://github.com/OpenDroneMap/WebODM/blob/master/db/init.sql

[mspringfield]

Ok, then I'm wondering if this is the root of all the issues. I was thinking that maybe you had multiple databases (like how rails does) and a different one for dev. The "FATAL: database "webodm_dev" does not exist" is been a persistent thing across all of the deployments since the beginning.

The very first time it builds (like if I rm the containers) it fails with this error: https://gist.github.com/mspringfield/a7404ae3432f65c22df63a6f0797ccad

However, if I stop it (ctrl-c) then restart it -- it seems to go through, but it has the webodm_dev does not exist error: https://gist.github.com/mspringfield/2679943287297442a5077c3272b28319

[pierotofy]

I wonder if this boils down to some difficult timing issue... are your servers really fast by any chance?

This is my output from a fresh Ubuntu 16.04 LTS machine on Azure (2 cores, 16 GB ram):

https://gist.github.com/pierotofy/855e63555a63203326e3a6089dfb68d0

I can see that in your log you get webapp | /bin/bash: /webodm/wait-for-it.sh: /bin/bash: bad interpreter: Text file busy.

I'm not sure why, but I can't reproduce it. I would look there for clues as to why the database is not being properly initialized.

[pierotofy]

So I was actually able to get a webapp | /bin/bash: /webodm/wait-for-it.sh: /bin/bash: bad interpreter: Text file busy after I did a ./webodm.sh update && ./webodm.sh start. I stopped the process with CTRL+C, then re-run ./webodm.sh start and things work OK (Ubuntu 16.04 LTS).

So I while webapp | /bin/bash: /webodm/wait-for-it.sh: /bin/bash: bad interpreter: Text file busy prevents the app from starting, I don't think it's the root cause of the issue you are experiencing.

webapp        | 2 static files copied to '/webodm/build/static', 384 unmodified.
webapp        | [2017-05-23 23:48:36 +0000] [54] [INFO] Starting gunicorn 19.7.1
webapp        | [2017-05-23 23:48:36 +0000] [54] [INFO] Listening at: unix:/tmp/gunicorn.sock (54)
webapp        | [2017-05-23 23:48:36 +0000] [54] [INFO] Using worker: sync
webapp        | [2017-05-23 23:48:36 +0000] [61] [INFO] Booting worker with pid: 61
webapp        | 
webapp        | 
webapp        | Congratulations! └@(･◡･)@┐
webapp        | ==========================
webapp        | 
webapp        | If there are no errors, WebODM should be up and running!
webapp        | 
webapp        | Open a web browser and navigate to http://localhost:8000
webapp        | 
webapp        | NOTE: Windows users using docker should replace localhost with the IP of their docker machine's IP. To find what that is, run: docker-machine ip
webapp        | INFO Booting...
webapp        | INFO Starting background scheduler...

[mspringfield]

Ok, I figured it out. New clean install and pulled the logs from the first webodm.sh start. It seems like it's failing on accessing the init-db.sql script once it copies over: https://gist.github.com/mspringfield/d72a61d20149d43088f6d26425072500

In the db/Dockerfile I added a line to change the permissions:
RUN chmod 644 /docker-entrypoint-initdb.d/init-db.sql

Removed containers, images and volumes for the db and restarted and it works now.

Very strange that this error was consistent for me across different platforms?

[pierotofy]

That's really strange; is the location of WebODM's files on a network mount, a removable media or some more exotic file system?

Would you be able to make a PR for the changes to db/Dockerfile? I'm sure this probably happened to somebody else as well.

I'm glad you've narrowed it down! 😄 🎉

[mspringfield]

Will do! Really appreciate you sticking with me on this!

Each install was on a fresh virtual machine, local filesystem (/opt/...), run as root. VM's were 8 core, 10g RAM.

[mspringfield]

Added Pull Request 180