Ray: [rllib] Weights & Biases logger cannot handle objects in configuration

[janblumenkamp]

What is the problem?

The Weights & Biases logger cannot handle object references in RLlib configurations, for example in the callback API.

Process _WandbLoggingProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "[...]/ray/tune/integration/wandb.py", line 127, in run
    wandb.init(*self.args, **self.kwargs)
  File "[...]/wandb/__init__.py", line 1303, in init
    as_defaults=not allow_val_change)
  File "[...]/wandb/wandb_config.py", line 333, in _update
    self.persist()
  File "[...]/wandb/wandb_config.py", line 238, in persist
    conf_file.write(str(self))
  File "[...]/wandb/wandb_config.py", line 374, in __str__
    allow_unicode=True, encoding='utf-8')
  File "[...]/yaml/__init__.py", line 290, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "[...]/yaml/__init__.py", line 278, in dump_all
    dumper.represent(data)
  File "[...]/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "[...]/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "[...]/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "[...]/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "[...]/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "[...]/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "[...]/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "[...]/yaml/representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
  File "[...]/yaml/representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', <class '__main__.MyCallbacks'>)

Ray version and other system information (Python version, TensorFlow version, OS):

• Ray 0.8.7
• Ubuntu 18.04
• Python 3.7

Reproduction (REQUIRED)

Please provide a script that can be run to reproduce the issue. The script should have no external library dependencies (i.e., use fake or mock data / environments):

from ray import tune
from ray.rllib.agents.ppo import PPOTrainer
from ray.rllib.agents.callbacks import DefaultCallbacks
from ray.tune.integration.wandb import WandbLogger

class MyCallbacks(DefaultCallbacks):
    def on_episode_end(self, worker, base_env, policies, episode, **kwargs):
        print("Episode ended")

tune.run(
    PPOTrainer,
    checkpoint_freq=1,
    config={
        "framework": "torch",
        "num_workers": 8,
        "num_gpus": 1,
        "env": "CartPole-v0",
        "callbacks": MyCallbacks,
        "logger_config": {
          "wandb": {
              "project": "test",
              "api_key_file": "./wandb_api_key_file",
          }
        }
    },
    stop={
        "training_iteration":10
    },
    loggers=[WandbLogger]
)

If we cannot run your script, we cannot fix your issue.

• [X] I have verified my script runs in a clean environment and reproduces the issue.
• [X] I have verified the issue also occurs with the latest wheels.

[janblumenkamp]

I noticed that the callback is already explicitly filtered out in the result handling, but of course that doesn't help during initialization. Is there any reason why objects shouldn't be generally filtered out? Probably only numbers and strings make sense to log anyway?

[amogkam]

cc @krfricke

[krfricke]

Thanks. For the moment I submitted a tiny PR to fix the callback API specifically. I agree it makes sense to filter out invalid values, I'll get back to this soon.

[rhamnett]

@krfricke I am having a similar problem trying to use APPO (PPO works fine). See https://pastebin.com/SdpNp0Pq

[krfricke]

Hey @janblumenkamp and @rhamnett this should be fixed via #10654. If this issue prevails, feel free to re-open!

[rhamnett]

@krfricke @richardliaw Afraid that I am still seeing the same error when running an APPO tune job

[krfricke]

Hi @rhamnett, did you install the latest nightly ray release? Do you have a script to reproduce the error?

[rhamnett]

@krfricke yes I am familiar with re-installing the nightly and can confirm that I have done that correctly. My script is lengthy and contains private intellectual property, so I am wondering if you can just simply adapt a PPO test you might have to APPO (should be trivial) and then let me know if you can re-produce?

If after trying that, you can't reproduce, I will seek to dedicate some time to give you an example script.

[krfricke]

Thanks, I could confirm the issue and filed a new PR to tackle it.

[rhamnett]

@krfricke appreciate it

[rhamnett]

Fixed, thanks

[rhamnett]

@richardliaw Sorry to be a pain, but there are no charts for episode_reward_{min,max,mean} etc showing (which is kind of the point of using w&b :)

[richardliaw]

Hmm thats odd! Can you post your output?

[rhamnett]

@richardliaw please can you describe what you are looking to see, are you asking for the W&B dashboard output?

[richardliaw]

ah sorry your driver stdout/stderr output?

[rhamnett]

ah sorry your driver stdout/stderr output?

https://pastebin.com/5uYwFaXq

[krfricke]

Hi @rhamnett, I could confirm the issue and file a new PR to tackle this.
It seems like the log cleaning converted numpy floats to strings, which then got filtered out before logging. The PR should fix that.

[rhamnett]

@krfricke this is fixed in 1.1.0 thanks again for all your support with these issues