aiohttp and network is unreachable

[Smosker]

Long story short

I'm using aiohttp im my django docker application, when i run it outside docker - everything works fine, but if i run it inside docker - i get error (see traceback)

very simplified code:
if you run it - no error will acquire, and if i run it in docker passing one url - no error acquired. If i pass two url with diferent hosts (like https://onehost.com/someth, https://otherhost.com/someth) - no error acquired. But if i pass two url with same host https://onehost.com/someth, https://onehost.com/anything - i get an error below but only when i run it in docker.
First request with https://onehost.com/someth finished ok - i see the result, second withhttps://onehost.com/anything i got error.
Any suggestions?

import asyncio
import aiohttp
from django.http import JsonResponse
from django.views import View


class LinksView(View):
    def post(self, request):

        loop = asyncio.get_event_loop()
        result = loop.run_until_complete(
            run(request))
        print(result)
        return JsonResponse({})


class Helper:
    def __init__(self, session, urls):
        self.url_objects = urls
        self.session = session

    async def fetch(self, request, url_object):
        async with self.session.request(**request) as response:
            result = await response.json()
            response.raise_for_status()

        return result, url_object

    async def get_result(self):
        result_obj = {}
        tasks = [
            self.fetch(request=request, url_object=url_object)
            for request, url_object in self.create_requests()
        ]
        for future in asyncio.as_completed(tasks):
            response, url_object = await future
            result_obj[url_object] = response

        return result_obj

    def create_requests(self):
        requests = []
        for url_object in self.url_objects:
            request = self.create_request(url_object)
            requests.append((request, url_object))

        return requests

    def create_request(self, url_object, ):
        return dict(
            url=url_object,
            method='GET',
        )


async def run(urls):
    async with aiohttp.ClientSession(
            connector=aiohttp.TCPConnector(
                verify_ssl=False,
            ),
    ) as session:
        helpers = [Helper(session=session, urls=urls)]

        result_obj = []
        tasks = [asyncio.ensure_future(helper.get_result())
                 for helper in helpers]

        for future in asyncio.as_completed(tasks):
            result = await future
            result_obj.append(result)

    return result_obj

LinksView().post(['https://jsonplaceholder.typicode.com/users',
                  'https://jsonplaceholder.typicode.com/posts/1'])

Traceback:

Traceback (most recent call last):
 File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 796, in _create_direct_connection
 local_addr=self._local_addr)
 File "/usr/lib/python3.5/asyncio/base_events.py", line 695, in create_connection
 raise exceptions[0]
 File "/usr/lib/python3.5/asyncio/base_events.py", line 682, in create_connection
 yield from self.sock_connect(sock, address)
 File "/usr/lib/python3.5/asyncio/selector_events.py", line 402, in sock_connect
 return (yield from fut)
 File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__
 return self.result() # May raise too.
 File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
 raise self._exception
 File "/usr/lib/python3.5/asyncio/selector_events.py", line 407, in _sock_connect
 sock.connect(address)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 185, in _get_response
 response = wrapped_callback(request, *callback_args, **callback_kwargs)
 File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 68, in view
 return self.dispatch(request, *args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/django/utils/decorators.py", line 67, in _wrapper
 return bound_func(*args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
 return view_func(*args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/django/utils/decorators.py", line 63, in bound_func
 return func.__get__(self, type(self))(*args2, **kwargs2)
 File "/usr/local/lib/python3.5/dist-packages/my_app/my_module/views.py", line 25, in dispatch
 return super().dispatch(request, *args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 88, in dispatch
 return handler(request, *args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/my_app/my_module/views.py", line 46, in post
 run(urls))
 File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
 return future.result()
 File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
 raise self._exception
 File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
 result = coro.send(None)
 File "/usr/local/lib/python3.5/dist-packages/my_app/my_module/views.py", line 122, in run
 result = await future
 File "/usr/lib/python3.5/asyncio/tasks.py", line 492, in _wait_for_one
 return f.result() # May raise f.exception().
 File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
 raise self._exception
 File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
 result = coro.send(None)
 File "/usr/local/lib/python3.5/dist-packages/my_app/my_module/views.py", line 83, in get_result
 response, url_object = await future
 File "/usr/lib/python3.5/asyncio/tasks.py", line 492, in _wait_for_one
 return f.result() # May raise f.exception().
 File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
 raise self._exception
 File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
 result = coro.send(None)
 File "/usr/local/lib/python3.5/dist-packages/my_app/my_module/views.py", line 65, in fetch
 async with self.session.request(**request) as response:
 File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 692, in __aenter__
 self._resp = yield from self._coro
 File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 269, in _request
 conn = yield from self._connector.connect(req)
 File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 392, in connect
 proto = yield from self._create_connection(req)
 File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 737, in _create_connection
 _, proto = yield from self._create_direct_connection(req)
 File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 823, in _create_direct_connection
 raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host onehost.com:443 ssl:True [Network is unreachable]

Your environment

python3.5
aiohttp==2.3.1

i found same issue on stackoverflow, but there is no sutable answer where https://stackoverflow.com/questions/40347726/python-3-5-asyincio-and-aiohttp-errno-101-network-is-unreachable because i cant set family=socket.AF_INET

[asvetlov]

Hmm. No idea.

[Smosker]

I found it accidentally when making request with two url with same host, i can recreate session object before each request - in this case everything ok, but i dont think its a proper way to solve this issue

[kxepal]

Worth to check how your network configuration is different in docker and out of it.

[Smosker]

i dont specify any networks settings manually so dont know that to look for.

[asvetlov]

Don't think we can help you.
aiohttp is used in very many projects, and very often is executed in docker.
But not from django :)
You wrote: "Guys I have a code. The code doesn't work. I use aiohttp and think the problem is in your library not in my code/environment/whatever." Not sure how we can solve it even if the problem is reproducible on your box only.

[Smosker]

solve issue by passing limit=1 to connector or by passing use_dns_cache=False,

async with aiohttp.ClientSession(
                connector=aiohttp.TCPConnector(
                    verify_ssl=False,
                    limit=1, #or use_dns_cache=False
                ),
        ) as session:

Do you have any ideas why this can help? I think this changes will slow down my application, am i right? Which of this options is more prefrable to use? Or maybe i just need to create new session on each request and it will be better solution?

[Smosker]

I create demo for you (no django is used): https://github.com/Smosker/async_aiohttp

Steps to reproduce:
1) clone repo, install python3.5, aiohttp. Run main.py - no error acquired

2) run ./run.sh file - docker will build image and run same main.py file - error acquired

I hope this will help you in finding source of problem.

[asvetlov]

Thank for demonstration repo, it should help very well.
Still didn't try it yet but I want to find a time for the issue soon.

[asvetlov]

andrew•~/projects/async_aiohttp(master)» ./run.sh                                             (async_aiohttp)  
... skipped build output
Successfully built 8231882daa82
[{'https://www.google.com/?gfe_rd=cr&dcr=0&ei=I5cNWr1lh7joBIbethg': 11773, 'https://google.ru': 11756}]

I use Ubuntu 17.10.
MacOSX might provide a different output.

[Smosker]

My output on MacOs 10.12.6

Successfully built c4986b5cb295
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 796, in _create_direct_connection
    local_addr=self._local_addr)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 695, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.5/asyncio/base_events.py", line 682, in create_connection
    yield from self.sock_connect(sock, address)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 402, in sock_connect
    return (yield from fut)
  File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__
    return self.result()  # May raise too.
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 407, in _sock_connect
    sock.connect(address)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 66, in <module>
    'https://www.google.com/?gfe_rd=cr&dcr=0&ei=I5cNWr1lh7joBIbethg',
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "main.py", line 57, in run
    result = await future
  File "/usr/lib/python3.5/asyncio/tasks.py", line 492, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "main.py", line 24, in get_result
    response, url_object = await future
  File "/usr/lib/python3.5/asyncio/tasks.py", line 492, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "main.py", line 11, in fetch
    async with self.session.request(**request) as response:
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 692, in __aenter__
    self._resp = yield from self._coro
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 269, in _request
    conn = yield from self._connector.connect(req)
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 392, in connect
    proto = yield from self._create_connection(req)
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 737, in _create_connection
    _, proto = yield from self._create_direct_connection(req)
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 823, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.google.ru:443 ssl:True [Network is unreachable]

[asvetlov]

Sorry, I afraid I cannot help you.
Mac is suxx :) At least Mac+Docker

[mikenemkov]

checked on my macbook pro (10.12.6)

... skipped build output

Step 7/7 : COPY /main.py /
 ---> 5eea1112e121
Successfully built 5eea1112e121
Successfully tagged asyncdemo:latest
[{'https://www.google.com/?gfe_rd=cr&dcr=0&ei=I5cNWr1lh7joBIbethg': 11818, 'https://google.ru': 11720}]

[Smosker]

checked on my macbook pro (10.12.6)

Thanks for trying.
This is interesting, will try to run it when i will be home, maybe problem somewhere in network settings on my work.

[Smosker]

When using my home network everything is ok, thanks for help!

[AndrewBurdyug]

I have the same problem, I also get OSError: [Errno 101] Network is unreachable, but in my case the cause in IPv6, so setting family=socket.AF_INET in TCPConnector fixed it for me, tricky =)

[benelot]

I found the answer that solved my problem as well. I just document it a bit more properly here:

The solution is just to initialize the client session using:

import socket # together with your other imports

conn = aiohttp.TCPConnector(
        family=socket.AF_INET,
        verify_ssl=False,
    )

   # Create client session that will ensure we dont open new connection
   # per each request.
   async with aiohttp.ClientSession(connector=conn) as session:

Explanation based on the link:
The client session will use the connection instead of the default AsyncResolver as the resolver for the connection. It used to be the default resolver. The problem seems to be related to domains with ipv6 where the AsyncResolver has problems, so the solution is to simply specify the family to ipv4 addresses which is what we do with family=socket.AF_INET.

Taken from my stackoverflow question:
https://stackoverflow.com/questions/48007453/asynchronous-http-calls-using-aiohttp-asyncio-fail-with-cannot-connect-to-host/48008873#48008873

[Smosker]

I mention this solution in my initial comment, but unfortunately i cant set family=socket.AF_INET because of my network settings.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].