Ml-agents: v0.14.0: cannot start training with a demo (for GAIL)

[Dastyn]

Hi,

When starting the training (with v0.14.0, API-14):
mlagents-learn myconfig.yaml ...
with myconfig.yaml containing demo_path(s) for pretraining and reward_signals/gail sections,
an exception is raised:

IndexError: agent_id 0 is not present in the BatchedStepResult

I don't know how to do with this!
Note: my environment worked on previous release 0.13.1

Thanks for any hint!
D.

FYI, the full exception is as follows:

2020-02-15 18:17:14.539257: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5033350 executing computations on platform Host. Devices:
2020-02-15 18:17:14.539328: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
requesting resize 84 x 84
Setting up 2 worker threads for Enlighten.
  Thread -> id: 7fea90891700 -> priority: 1 
  Thread -> id: 7fea1ffff700 -> priority: 1 
##utp:{"type":"MemoryLeaks","version":2,"phase":"Immediate","time":1581787042002,"processId":25340,"allocatedMemory":4066,"memoryLabels":[{"Default":40},{"Permanent":40},{"NewDelete":112},{"Thread":128},{"Manager":1680},{"GfxDevice":64},{"Physics":32},{"Serialization":40},{"String":298},{"GI":296},{"VR":1992},{"Subsystems":-656}]}
INFO:mlagents_envs:Environment shut down with return code 0.
Traceback (most recent call last):
  File "/home/lub/.local/bin/mlagents-learn", line 11, in <module>
    sys.exit(main())
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/learn.py", line 479, in main
    run_cli(parse_command_line())
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/learn.py", line 475, in run_cli
    run_training(run_seed, options)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/learn.py", line 320, in run_training
    tc.start_learning(env_manager)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/trainer_controller.py", line 213, in start_learning
    self._create_trainers_and_managers(env_manager, new_behavior_ids)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/trainer_controller.py", line 200, in _create_trainers_and_managers
    self._create_trainer_and_manager(env_manager, behavior_id)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/trainer_controller.py", line 180, in _create_trainer_and_manager
    policy = trainer.create_policy(env_manager.external_brains[name_behavior_id])
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/ppo/trainer.py", line 240, in create_policy
    self.load,
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/ppo/policy.py", line 51, in __init__
    self.create_reward_signals(reward_signal_configs)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/ppo/policy.py", line 136, in create_reward_signals
    self, self.model, reward_signal, config
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/components/reward_signals/reward_signal_factory.py", line 44, in create_reward_signal
    class_inst = rcls(policy, policy_model, **config_entry)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/components/reward_signals/gail/signal.py", line 48, in __init__
    _, self.demonstration_buffer = demo_to_buffer(demo_path, policy.sequence_length)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents_envs/timers.py", line 262, in wrapped
    return func(*args, **kwargs)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/demo_loader.py", line 88, in demo_to_buffer
    demo_buffer = make_demo_buffer(info_action_pair, group_spec, sequence_length)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents_envs/timers.py", line 262, in wrapped
    return func(*args, **kwargs)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents/trainers/demo_loader.py", line 56, in make_demo_buffer
    next_agent_step_info = next_step_info.get_agent_step_result(agent_id)
  File "/home/lub/.local/lib/python3.6/site-packages/mlagents_envs/base_env.py", line 121, in get_agent_step_result
    "agent_id {} is not present in the BatchedStepResult".format(agent_id)
IndexError: agent_id 0 is not present in the BatchedStepResult

[chriselion]

Hope you don't mind, I edited the formatting on your comment to make the stacktrace easier to read.

Are you able to reproduce this with any of the example scenes?

We'll look into it first thing on Tuesday (Monday is a holiday for our office).

[Xiromtz]

I have exactly the same issue, it happens with both GAIL and Behavioral Cloning, though the error message differs when using Behavioral Cloning. I made a thread in the Unity forums for it.
I tested the Unity examples folder with their GAIL implementation and interestingly enough, no errors occur there.
Did you do use old recordings or did you make new ones in version 0.14.0? This might be a bug with the recordings themselves, since I recorded mine in the new version..

[Dastyn]

Thanks @chriselion and @Xiromtz for your help and reactivity.

The same for me: this issue concerns new demo recordings with the 0.14.0 version.
(my .demo files were 0.14.0-compliant)

[harperj]

Thanks for the information @Dastyn / @Xiromtz. We've confirmed that this is a bug with v0.14.0 and are working on a fix.

[chriselion]

Just a heads up that this should be fixed in the 0.14.1 release, which came out today. Your demonstration files from 0.14.0 will load, but you probably need to re-record them because some information wasn't captured during the initial recording.

[Dastyn]

@chriselion Thank you for the fix.
I already checked with the patch given at #3464 on ml-agents/mlagents/trainers/demo_loader.py
that it works fine by now.
Thanks!