Ckan: Datapusher throws connection error when run via docker compose

[Nimphal]

CKAN Version if known (or site URL)

2.7.2 , using backported docker-compose as per #3983

Please describe the expected behaviour

Having enabled the datastore and datapusher plugins following the instructions, I expect that when I upload a file it is ingested by datapusher, pushed to datastore and available for preview.

Please describe the actual behaviour

Datapusher throws the following error:

datapusher    | Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
datapusher    | Traceback (most recent call last):
datapusher    |   File "/usr/local/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job
datapusher    |     retval = job.func(*job.args, **job.kwargs)
datapusher    |   File "/usr/src/app/datapusher/jobs.py", line 222, in push_to_datastore
datapusher    |     resource = get_resource(resource_id, ckan_url, api_key)
datapusher    |   File "/usr/src/app/datapusher/jobs.py", line 178, in get_resource
datapusher    |     'Authorization': api_key}
datapusher    |   File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 94, in post
datapusher    |     return request('post', url, data=data, json=json, **kwargs)
datapusher    |   File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 49, in request
datapusher    |     return session.request(method=method, url=url, **kwargs)
datapusher    |   File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
datapusher    |     resp = self.send(prep, **send_kwargs)
datapusher    |   File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
datapusher    |     r = adapter.send(request, **kwargs)
datapusher    |   File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
datapusher    |     raise ConnectionError(err, request=request)
datapusher    | ConnectionError: ('Connection aborted.', error(99, 'Cannot assign requested address'))

What steps can be taken to reproduce the issue?

• Fire up the docker-compose
• attach to the datapusher container docker-compose logs -f datapusher
• enable datastore and datapusher in the config file and restart ckan
• upload a resource and go to preview it

Additional info

I am running ubuntu 16.04 LTS, Docker version 17.12.0-ce, build c97c6d6 and docker-compose version 1.18.0, build 8dd22a9

[s-chand]

Hi @Nimphal it's probably an issue with your ckan.site_url settings.

Is ckan running in a docker container as well?
What OS are you running your docker on? Mac? Ubuntu? I see that the datapusher os is ubuntu

[Nimphal]

Hi Samuel, I am getting this error when using the docker-compose file in this repo, so everything is in docker, including ckan. The ckan site url is passed as a parameter to the ckan container, and I haven't changed it from the default localhost:5000. I am running the docker-compose on Ubuntu 16.04 LTS.

[Nimphal]

A workaround for this (many thanks to @spikeheap who actually came up with this).

First, find the IP 0f the docker0 bridge:

ip addr show | grep docker0

Example output:

7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Then edit your /etc/hosts/ file with the following:

172.17.0.1 dockerhost

The dockerhost part can be whatever you would like to call this host.

Restart the docker daemon:

sudo service docker restart

Create a docker-compose.override.yml file (so that you don't break everyone else's setups) that has the following:

# replace dockerhost with whatever you have 172.17.0.1 pointing to in /etc/hosts

version: "3"

services:
 ckan:    
   extra_hosts:
     - "dockerhost:172.17.0.1" 

 datapusher:    
   extra_hosts:
     - "dockerhost:172.17.0.1"

In your .env file or wherever you have the environment variables set the following:

CKAN_SITE_URL=http://dockerhost:5000

Start the stack by using the override:

docker-compose -f docker-compose.yml -f docker-compose.override.yml up 

Navigate to http://dockerhost:5000 and you should have a working CKAN plus a working Datapusher if you have enabled the plugin.

[spikeheap]

Another workaround, in the same vein:

1. Add 127.0.0.1 ckan to /etc/hosts on your host machine.
2. In .env set CKAN_SITE_URL=http://ckan:5000.
3. Use http://ckan:5000 from your browser.

This is pretty much @Nimphal's solution, just without the custom extra_host definition, which saves having to look up the IP address on your host.

[Vanuan]

This can be fixed by #4013.

Another approach is making datapusher aware of ckan by providing internal ckan url in datapusher setting instead of using whatever CKAN (or malicious actor) provides. This design would make datapusher much more secure.

---

Side note

Making internal service to go through public URL (and nginx/load balancing/router chain) also results in slowness.

A much better approach would be to use some internal RPC. For example, gRPC is particularly popular nowadays. RabbitMQ might also work. The problem of REST is that it's not very good with two-way communication. Lost packets might result in some portions of CKAN UI to completely hang (as datapusher-ckan communication channel is blocking). Async requests with timeout are much less intrusive.

Among other things to consider is potential self-DDoS caused by #2357

[linforestzhang]

Same issue for CKAN v2.9.0 install from Docker compose and manually downloaded zip without using git.