Visdom: IndexError: list index out of range ERROR:root:ERROR: 500: {'exc_info': ...

[rdevon]

I have a bunch of experiments (18 to be exact) running on various machines connected to one visdom server. I'm viewing a bunch of these experiments together, and then suddenly the plots stop loading and the server webpage wont load. About this time, the server machine stops accepting ssh connections.

When I restart the server machine and the visdom server, I get this error on the server when I navigate to the visdom server IP:

INFO:tornado.access:101 GET /vis_socket (<MY IP>) 0.42ms
INFO:root:Opened visdom socket from ip: <MY IP>
INFO:tornado.access:101 GET /socket (<MY IP>) 0.56ms
INFO:root:Opened new socket from ip: <MY IP>
INFO:root:comparing envs
ERROR:tornado.application:Uncaught exception POST /compare/DIM_rkhs_1024_dimz_192_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_192_pen_0.1_do_pt041+DIM_rkhs_1024_dimz_512_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_512_pen_0.1_do_pt041+DIM_rkhs_1024_dimz_64_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_64_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_192_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_192_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_512_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_512_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_64_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_64_pen_0.1_do_pt041+DIM_rkhs_512_dimz_192_pen_0.0_do_pt041+DIM_rkhs_512_dimz_192_pen_0.1_do_pt041+DIM_rkhs_512_dimz_512_pen_0.0_do_pt041+DIM_rkhs_512_dimz_512_pen_0.1_do_pt041+DIM_rkhs_512_dimz_64_pen_0.0_do_pt041+DIM_rkhs_512_dimz_64_pen_0.1_do_pt041 (<MY IP>)
HTTPServerRequest(protocol='http', host='<VIZ_IP>', method='POST', uri='/compare/DIM_rkhs_1024_dimz_192_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_192_pen_0.1_do_pt041+DIM_rkhs_1024_dimz_512_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_512_pen_0.1_do_pt041+DIM_rkhs_1024_dimz_64_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_64_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_192_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_192_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_512_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_512_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_64_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_64_pen_0.1_do_pt041+DIM_rkhs_512_dimz_192_pen_0.0_do_pt041+DIM_rkhs_512_dimz_192_pen_0.1_do_pt041+DIM_rkhs_512_dimz_512_pen_0.0_do_pt041+DIM_rkhs_512_dimz_512_pen_0.1_do_pt041+DIM_rkhs_512_dimz_64_pen_0.0_do_pt041+DIM_rkhs_512_dimz_64_pen_0.1_do_pt041', version='HTTP/1.1', remote_ip='<MY IP>')
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/visdom/server.py", line 51, in _check_auth
    f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/visdom/server.py", line 1001, in post
    self.env_path)
  File "/usr/local/lib/python2.7/dist-packages/visdom/server.py", line 835, in compare_envs
    res = copy.deepcopy(envs[list(envs.keys())[0]])
IndexError: list index out of range
ERROR:root:ERROR: 500: {'exc_info': (<type 'exceptions.IndexError'>, IndexError('list index out of range',), <traceback object at 0x7f2d51c9e950>)}

What is going on here?

[JackUrb]

I'm not sure quite why but for some reason there are no envs in envs at this moment here: https://github.com/facebookresearch/visdom/blob/e34c6d5abb23ed1e3333515af714c70b570e86c1/py/visdom/server.py#L913. Are all of those experiment's envs existent? It'd be useful to check what's in envs, eids, and state right before that line.

[rdevon]

Yeh, I'm not sure. What is confusing is this is after restarting the server, even if the jobs have stopped. Which means there some files somewhere that visdom is reading that are broken.

[JackUrb]

Are you able to hit the home page of the server? And if so do those envs appear in the list in the main bar?

[rdevon]

So i can visit the page, but only the "main" env appears. On the server side, that error I posted appears.

Note also that after restarting the jobs and server, things work fine after a few steps, so whatever was broken get overwritten. However, if I try to view too many envs at once, things crash with the same issues.

Temp solution right now is just not to view too many envs at once.

[JackUrb]

The index error hits on the server side from landing on the home page? That would imply a different callsite runs into the same problem, which I wasn't expecting for the home page.

Definitely agreed that it might be an issue with comparing too many envs, seems like the right area for me to investigate when I get a chance to go into this one.