Tornado: RuntimeError: list changed size during iteration

[viswamy]

I am seeing this error in my project.

tornado_version: 4.3

Any thoughts?

2016-07-19 23:59:49,872 clay_kafka INFO Moving INGEST -> INGEST for Topic: ezpzpass.views.0.
  File "/home/uber/torrent/env/bin/torrent-kafka-ingester", line 9, in <module>
    load_entry_point('torrent', 'console_scripts', 'torrent-kafka-ingester')()
  File "/home/uber/torrent/env/local/lib/python2.7/site-packages/clay_kafka/ingester_main.py", line 128, in main
    process_main()
  File "/home/uber/torrent/env/local/lib/python2.7/site-packages/clay_kafka/ingester_main.py", line 90, in process_main
    io_loop.start()
  File "/home/uber/torrent/env/local/lib/python2.7/site-packages/tornado/ioloop.py", line 814, in start
    due_timeouts.append(heapq.heappop(self._timeouts))
RuntimeError: list changed size during iteration

[bdarnell]

This looks like a threading (or maybe signal-handling?) problem to me. It is unsafe to call IOLoop.add_timeout from any thread but the one that the IOLoop is running on, or from a signal handler. You must always go through add_callback, so to add a timeout from another thread you could do something like IOLoop.current().add_callback(functools.partial(IOLoop.current.add_timeout, timeout, callback))

[marvin520]

Exception in thread Tornado:
Traceback (most recent call last):

    File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner self.run()
    File "/clarifai-proxy/util/TornadoThread.py", line 34, in run
       tornado.ioloop.IOLoop.instance().start()
    File "/usr/lib64/python2.7/site-packages/tornado/ioloop.py", line 815, in start heapq.heappop(self._timeouts)
RuntimeError: list changed size during iteration

is my problem has the same resolution suggested upond?

i don't have any ideal about how to add callback function.

i didn't add any timeout in current IOLoop, my code is below,it is actually worked in a thread:
`
def run(self):
try:
startup_prompt = " {0} Proxy Started on {1}".format(self.mode,self.listen_port)

        # 单进程
        self.app.listen(self.listen_port)
        self.app.logger.info(startup_prompt)
        tornado.ioloop.IOLoop.instance().start()

    # except RuntimeError:
    #     tornado.ioloop.IOLoop.instance().
    except KeyboardInterrupt:
        tornado.ioloop.IOLoop.instance().stop()

`

[bdarnell]

@marvin520 Yes, if you're getting this error it is almost certainly from improper use of threads. You must not call any tornado-related methods except IOLoop.add_callback from any thread except the one that calls IOLoop.start().

[marvin520]

this is my code:

   class TornadoThread(threading.Thread):
        def __init__(self,threadID,name,logger,cf,mode):
            threading.Thread.__init__(self)
            self.threadID = threadID
            self.name = name
            self.logger = logger
            self.cf = cf
            self.mode = mode
            if mode == 'master':
                self.listen_port = self.cf.get_conf("app", "master_listen_port")
            elif mode == 'slave':
                self.listen_port = self.cf.get_conf("app", "slave_listen_port")
            self.app= Application(self.logger,self.cf,self.mode)
        def stop_tornado(self):
            ioloop = tornado.ioloop.IOLoop.current()
            ioloop.add_callback(ioloop.stop)
            self.logger.error('Clafifai {0} need to exit one moment'.format(self.mode))
        def run(self):
            try:
                startup_prompt = "Clarifai {0} Proxy Started on {1}".format(self.mode,self.listen_port)
                self.app.listen(self.listen_port)
                self.app.logger.info(startup_prompt)
                tornado.ioloop.IOLoop.instance().start()
                # tornado.ioloop.IOLoop.instance().start()
                # sockets = tornado.netutil.bind_sockets(self.listen_port)
                # tornado.process.fork_processes(0)
                # server = HTTPServer(self.app)
                # server.add_sockets(sockets)
                # tornado.ioloop.IOLoop.current().start()
            except RuntimeError:
                self.stop_tornado()
                startup_prompt = "Clarifai {0} Proxy Started on {1}".format(self.mode,self.listen_port)
                self.app.listen(self.listen_port)
                self.app.logger.info(startup_prompt)
                tornado.ioloop.IOLoop.instance().start()
            except KeyboardInterrupt:
                tornado.ioloop.IOLoop.instance().stop()

it works fine for some hours, and will hanged, i only call tornado-related method in this thread, could you help to piont out what is the improper code upond this comment?

[bdarnell]

@marvin520 do you have both master and slave threads in the same process? That won't work because they're both using the same IOLoop.instance(). Other than that, I don't see any problems if this is the only tornado-related code in your application (maybe you need to move the initialization of the Application into run?)

[marvin520]

i have solved this problem already.
if we use thread in our code, which one thread start tornado application ,others do some daemon or any works and use tornado async IOLoop, the code will work fine for some hours, and will crash in random time cause any thread crash will lead to tornado crash too.
the resolution is to use Process instead of thread, and use requests in other Processes instead of tornado AsyncHTTPClient.