Ml-agents: Agent stops collecting observations when SceneManager() triggered

[graybob]

Hi ,

I was converting Unity FappyBird to Unity MLagent Environament and I am facing Issue as soon as SceneManager() triggered when Bird gameobject is dead,

Issue -> The UnityML stops collecting observations and agent stops responding .

Steps to Create issue.

1) Download Unity Flappybird @ https://assetstore.unity.com/packages/templates/flappy-bird-style-example-game-80330

2) Create a Empty game object (Rename to Bird_Agent) and make it child of Bird .

3) I am adding Bird_agent, Bird and GameControl modified code below , Please replace code from original project .

4) Start training , you will see that Bird fine till its dead , once Bird is dead UnityML stops collecting observations (Expecting Bird should start training again).

////////////////////////////////////////////////////////////////////////////////////////////////////////
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using MLAgents;
using MLAgents.Sensors;
using MLAgents.SideChannels;


public class Bird_Agent : Agent
{


    public override void Initialize()
    {


        //SetResetParameters();
    }


    public override void CollectObservations(VectorSensor sensor)
    {
        GameObject temp = GameObject.Find("Bird");
        //  sensor.AddObservation(gameObject.GetComponent<Bird>().rb2d.transform.localPosition);
        sensor.AddObservation(temp.transform.localPosition);
    }



    public void MoveAgent(float[] act)
    {
        var forwardAxis = (int)act[0];
        bool dirToGo;
        GameObject temp = GameObject.Find("Bird");
        Debug.Log("In MovAgent.....");
        switch (forwardAxis)

        {
            case 1:
                temp.GetComponent<Bird>().Bird_Fly = true;
                break;
            case 2:
                temp.GetComponent<Bird>().Bird_Fly = false;
                break;
        }
    }



    public override void OnActionReceived(float[] vectorAction)
    {
        MoveAgent(vectorAction);
    }

    public override float[] Heuristic()
    {
        var action = new float[0];
        GameObject temp = GameObject.Find("Bird");
        if (Input.GetKey(KeyCode.D))
        {
            temp.GetComponent<Bird>().Bird_Fly = true;
        }

        return action;
    }

    public override void OnEpisodeBegin()
    {

    }



    public void SetResetParameters()
    {

    }




}
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

using UnityEngine;
using System.Collections;

public class Bird : MonoBehaviour 
{
    public float upForce;                   //Upward force of the "flap".
    private bool isDead = false;            //Has the player collided with a wall?

    private Animator anim;                  //Reference to the Animator component.
    private Rigidbody2D rb2d;               //Holds a reference to the Rigidbody2D component of the bird.
    public bool Bird_Fly;
    void Start()
    {
        //Get reference to the Animator component attached to this GameObject.
        anim = GetComponent<Animator> ();
        //Get and store a reference to the Rigidbody2D attached to this GameObject.
        rb2d = GetComponent<Rigidbody2D>();
    }

    void Update()
    {
        Vector3 tmpPos = transform.position;
        tmpPos.y = Mathf.Clamp(tmpPos.y, -2.0f, 5.0f);
        transform.position = tmpPos;

        //Don't allow control if the bird has died.
        if (isDead == false) 
        {
            //Look for input to trigger a "flap".
            if (Bird_Fly || Input.GetMouseButtonDown(0)) 
            {
                //...tell the animator about it and then...
                anim.SetTrigger("Flap");
                //...zero out the birds current y velocity before...
                rb2d.velocity = Vector2.zero;
                //  new Vector2(rb2d.velocity.x, 0);
                //..giving the bird some upward force.
                //rb2d.AddForce(new Vector2(0, upForce));

               // transform.position = tmpPos;
                rb2d.AddForce(new Vector2(0, upForce));
               // Mathf.Clamp(tmpPos.y, 1.0f, 1.0f);
            }
        }
    }

    void OnCollisionEnter2D(Collision2D other)
    {
        // Zero out the bird's velocity
        rb2d.velocity = Vector2.zero;
        // If the bird collides with something set it to dead...
        isDead = true;
        //...tell the Animator about it...
        anim.SetTrigger ("Die");
        //...and tell the game control about it.
        GameControl.instance.BirdDied ();
    }
}


//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

using UnityEngine;
using System.Collections;
using UnityEngine.UI;
using UnityEngine.SceneManagement;

public class GameControl : MonoBehaviour 
{
    public static GameControl instance;         //A reference to our game control script so we can access it statically.
    public Text scoreText;                      //A reference to the UI text component that displays the player's score.
    public GameObject gameOvertext;             //A reference to the object that displays the text which appears when the player dies.

    private int score = 0;                      //The player's score.
    public bool gameOver = false;               //Is the game over?
    public float scrollSpeed = -1.5f;


    void Awake()
    {
        //If we don't currently have a game control...
        if (instance == null)
            //...set this one to be it...
            instance = this;
        //...otherwise...
        else if(instance != this)
            //...destroy this one because it is a duplicate.
            Destroy (gameObject);
    }

    void Update()
    {
        //If the game is over and the player has pressed some input...
        if (gameOver )//&& Input.GetMouseButtonDown(0)) 
        {
            //...reload the current scene.
            Debug.Log("Bird DIED .....................");
            SceneManager.LoadScene(SceneManager.GetActiveScene().buildIndex);
        }
    }

    public void BirdScored()
    {
        //The bird can't score if the game is over.
        if (gameOver)   
            return;
        //If the game is not over, increase the score...
        score++;
        //...and adjust the score text.
        scoreText.text = "Score: " + score.ToString();
    }

    public void BirdDied()
    {
        //Activate the game over text.
        //gameOvertext.SetActive (true);
        //Set the game to be over.
        gameOver = true;
    }
}

/////////////////////////////////////////////////////////////////////////////////////////////////

Thanks.

[surfnerd]

Hi @graybob,
Do you see anything in the Unity Console or python output when the scene is reloaded after the bird game object is destroyed?

[graybob]

Hi ,
Yes , The log is as below.. (Using ML-Agents Beta 0.15.1)

Fewer observations (0) made than vector observation size (3). The observations will be padded.
UnityEngine.Debug:LogWarningFormat(String, Object[])
MLAgents.Sensors.VectorSensor:Write(WriteAdapter) (at Assets/com.unity.ml-agents/Runtime/Sensors/VectorSensor.cs:50)
MLAgents.GrpcExtensions:GetObservationProto(ISensor, WriteAdapter) (at Assets/com.unity.ml-agents/Runtime/Communicator/GrpcExtensions.cs:222)
MLAgents.RpcCommunicator:PutObservations(String, AgentInfo, List1) (at Assets/com.unity.ml-agents/Runtime/Communicator/RpcCommunicator.cs:284) MLAgents.Policies.RemotePolicy:RequestDecision(AgentInfo, List1) (at Assets/com.unity.ml-agents/Runtime/Policies/RemotePolicy.cs:33)
MLAgents.Agent:NotifyAgentDone(DoneReason) (at Assets/com.unity.ml-agents/Runtime/Agent.cs:324)
MLAgents.Agent:OnDisable() (at Assets/com.unity.ml-agents/Runtime/Agent.cs:311)

Thanks

[surfnerd]

edit: I had the wrong code in the FixedUpdate method.

Hi @graybob,
We've logged this issue internally as MLA-874. We will work on this and update this thread when it's fixed.

In the time being, you can turn off the Academy Automatic Stepping and call RequestDecision in your Agent's FixedUpdate method for now.

In your Agent:

protected override void Initialize()
{
    Academy.Instance.AutomaticSteppingEnabled = false;
}

void FixedUpdate()
{
    Academy.Instance.EnvironmentStep();
}

Let me know if this workaround helps you move forward.

[graybob]

Hi ,

Yes this Workaround works good , the above warning messages are gone and Bird continues to train. :):)

Thanks.

[graybob]

Hi ,

I also figured out that Ray Perception Sensor Component 2D attached to bird stops working as soon as "SceneManager.LoadScene(SceneManager.GetActiveScene().buildIndex);" is triggered.

I could see raycast fine till bird died(once) , once bird dies Scecemanager is triggered to reload the scene , once this happens the raycast vanishes .

I cant train without raycast.

Thanks

[surfnerd]

Hi @graybob,
This issue was addressed in the latest release. Could you give it a try and see if the issue persists?

[graybob]

Hi ,

:) Great the fix works in latest release . Please close the issue .

Thanks 👍