Pyzmq: ValueError: Invalid file object

[d53dave]

Hi there,

I have been playing around with your asyncio examples and I am running into the above mentioned error.

 ~  lsb_release -a
2017 16:51:45 CEST
Distributor ID: elementary
Description:    elementary OS 0.4.1 Loki
Release:    0.4.1
Codename:   loki

~  uname -a
Linux dsere-ThinkPad-T460p 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tornado
>>> tornado.version
'4.5.1'
>>> import zmq
>>> zmq.__version__
'16.0.2'

The full stacktrace is

ERROR:tornado.application:Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7fe4dd7eec80>, <tornado.concurrent.Future object at 0x7fe4dd7d40b8>)
Traceback (most recent call last):
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
    ret = callback()
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/ioloop.py", line 626, in _discard_future_result
    future.result()
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/gen.py", line 307, in wrapper
    yielded = next(result)
  File "<string>", line 6, in _wrap_awaitable
  File "/home/dave/pyzmq-asyncio-test/lib/server/tornadoasync.py", line 22, in pushing
    greeting = await server.recv()
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/eventloop/future.py", line 181, in recv
    dict(flags=flags, copy=copy, track=track)
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/eventloop/future.py", line 321, in _add_recv_event
    self._add_io_state(self._READ)
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/asyncio/__init__.py", line 297, in _add_io_state
    self.io_loop.add_reader(self, self._handle_recv)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 234, in add_reader
    key = self._selector.get_key(fd)
  File "/usr/lib/python3.5/selectors.py", line 189, in get_key
    return mapping[fileobj]
  File "/usr/lib/python3.5/selectors.py", line 70, in __getitem__
    fd = self._selector._fileobj_lookup(fileobj)
  File "/usr/lib/python3.5/selectors.py", line 224, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/lib/python3.5/selectors.py", line 39, in _fileobj_to_fd
    "{!r}".format(fileobj)) from None
ValueError: Invalid file object: <zmq.asyncio.Socket object at 0x7fe4dd7e5ac8>
Exception ignored in: <bound method Socket.__del__ of <zmq.asyncio.Socket object at 0x7fe4dd7e5ac8>>
Traceback (most recent call last):
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/sugar/socket.py", line 70, in __del__
    self.close()
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/eventloop/future.py", line 160, in close
    self._clear_io_state()
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/asyncio/__init__.py", line 319, in _clear_io_state
    self._drop_io_state(self._state)
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/asyncio/__init__.py", line 306, in _drop_io_state
    self.io_loop.remove_reader(self)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 250, in remove_reader
    key = self._selector.get_key(fd)
  File "/usr/lib/python3.5/selectors.py", line 189, in get_key
    return mapping[fileobj]
  File "/usr/lib/python3.5/selectors.py", line 70, in __getitem__
    fd = self._selector._fileobj_lookup(fileobj)
  File "/usr/lib/python3.5/selectors.py", line 224, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/lib/python3.5/selectors.py", line 39, in _fileobj_to_fd
    "{!r}".format(fileobj)) from None
ValueError: Invalid file object: <zmq.asyncio.Socket object at 0x7fe4dd7e5ac8>
ERROR:tornado.application:Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7fe4dd7eed90>, <tornado.concurrent.Future object at 0x7fe4dd7e0be0>)
Traceback (most recent call last):
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
    ret = callback()
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/ioloop.py", line 626, in _discard_future_result
    future.result()
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/tornado/gen.py", line 307, in wrapper
    yielded = next(result)
  File "<string>", line 6, in _wrap_awaitable
  File "/home/dave/pyzmq-asyncio-test/lib/server/tornadoasync.py", line 31, in pulling
    resp = await client.recv()
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/eventloop/future.py", line 181, in recv
    dict(flags=flags, copy=copy, track=track)
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/eventloop/future.py", line 321, in _add_recv_event
    self._add_io_state(self._READ)
  File "/home/dave/.virtualenvs/pyzmq-asyncio-test-gqw6x0KX/lib/python3.5/site-packages/zmq/asyncio/__init__.py", line 297, in _add_io_state
    self.io_loop.add_reader(self, self._handle_recv)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 234, in add_reader
    key = self._selector.get_key(fd)
  File "/usr/lib/python3.5/selectors.py", line 189, in get_key
    return mapping[fileobj]
  File "/usr/lib/python3.5/selectors.py", line 70, in __getitem__
    fd = self._selector._fileobj_lookup(fileobj)
  File "/usr/lib/python3.5/selectors.py", line 224, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/lib/python3.5/selectors.py", line 39, in _fileobj_to_fd
    "{!r}".format(fileobj)) from None
ValueError: Invalid file object: <zmq.asyncio.Socket object at 0x7fe4dd7e5a08>

A quick debugging session indicated that pythons selectors.py tries to squeeze out a file descriptor from the ZMQ Socket object by calling fileno(), but the object does not have this method. I monkey-patched something like this into it:

class Socket(_AsyncIO, _future._AsyncSocket):
    """Socket returning asyncio Futures for send/recv/poll methods."""

    _poller_class = Poller

    def fileno(self): 
        return self.FD

which resolved the issue.

Any idea why this is not working out of the box? Any version mismatches?

[minrk]

There are some ongoing changes to asyncio support in master, so if you are using a released version of pyzmq try looking at the examples from the same version.

The main change: pyzmq < 17 requires an initial step to tell asyncio to use zmq_poll as the underlying event loop. pyzmq 17 removes that requirement and thus removes the zmq.asyncio.install() call from the examples.

[d53dave]

Thanks for clearing that up. My bad!

Would you expect any other issues if I kept my patched version? Or would it be a better idea to install v17-dev?

[d53dave]

Hello,

I am re-opening this issue because I have tried it with the current master version and it shows the same behavior. The only way I could get the example to work is by patching the def fileno(self) method into the Socket class in zmq/asyncio/__init__.py.

[minrk]

How did you install from master? The fileno() method is defined on sockets in master. Make sure that you have recompiled the extensions.

pip uninstall pyzmq
# in repo
git clean -xfd
pip install -e .

should do it. I've also just double-checked that the examples in examples/asyncio do indeed run on master with no errors.

[urielarg]

I'm having the same issue with version 16.0.3

[minrk]

@urielarg the fileno() method is new in master (17.x), so it wouldn't be expected to work in 16.0.

[urielarg]

@minrk Thanks for the quick reply, installing 17.0 now