Ray: Exceptions and ResourceWarnings on ray.init (Jupyter+offline)
What is the problem?
I'm running ray in a JupyterLab notebook (AWS SageMaker), and the notebook has no internet connection (for data safety).
(instance type is ml.p3.16xlarge, so 64 vCPU and 488GiB ram).
When I simply do:
import ray
ray.init()
I get the following output:
2020-08-24 01:29:24,153 INFO resource_spec.py:212 -- Starting Ray with 323.83 GiB memory available for workers and up to 142.79 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.err' mode='a' encoding='UTF-8'>
2020-08-24 01:29:24,500 INFO services.py:1148 -- View the Ray dashboard at localhost:8265
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.err' mode='a' encoding='UTF-8'>
{'node_ip_address': '10.54.198.48',
'redis_address': '10.54.198.48:45400',
'object_store_address': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504/sockets/plasma_store',
'raylet_socket_name': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504/sockets/raylet',
'webui_url': 'localhost:8265',
'session_dir': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504'}
Do I need to modify any init settings due to the fact that I'm running in Jupyter and/or because I'm offline?
Ray version and other system information (Python version, TensorFlow version, OS):
- Python 3.6.10
- ray 0.8.7
austinmw
All 2 comments
Is there any way what exceptions are ignored in your logs?
rkooo567
on 28 Aug 2020
@austinmw this is a warning that's printed when a Python file object is created and the file isn't closed before the object is destructed. In most versions of Python this won't be printed, but it will if you're using a debug build or pass -W (and maybe some other circumstances, haven't seen it though).
This is intentional behavior because we need these files to be long-lived to write logs to, so there's nothing to worry about. Unfortunately I can't think of a good way to silence this at the moment. Leaving this issue open for others that might come across it.
edoakes
on 28 Aug 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings
Related issues
heavyinfo
路
3Comments
xudongliao
路
3Comments
robertnishihara
路
3Comments
ericl
路
3Comments
thedrow
路
3Comments
Most helpful comment
@austinmw this is a warning that's printed when a Python file object is created and the file isn't closed before the object is destructed. In most versions of Python this won't be printed, but it will if you're using a debug build or pass
-W(and maybe some other circumstances, haven't seen it though).This is intentional behavior because we need these files to be long-lived to write logs to, so there's nothing to worry about. Unfortunately I can't think of a good way to silence this at the moment. Leaving this issue open for others that might come across it.