Ray: [rllib] PPO Pytorch support?

[tarungog]

What is your question?

Does RLLib support APPO with PyTorch models? The docs seem to hint that it does, but this Open Issue and the attached PR's left me confused:
https://github.com/ray-project/ray/issues/3365

Moreover do Impala and APE-X have Pytorch support? Can these algorithms be used with memory/LSTM models?

[toanngosy]

There is an implementation of PG in Pytorch. I'm working on implementing PPO.

[tarungog]

@toanngosy overall is PyTorch kind of a second class citizen here? Should I just rewrite my stuff in TF? I'd like to get access to the level of distributed training offered by RLlib without haven't to move to tensorflow

[sven1977]

We are currently working on making pytorch a first class citizen across all of RLlib.

• Some work needs to be done on unifying interfaces/factories/etc.. (I'm currently working on it).
• Then we'll go through all agents and remove duplicate/almost-duplicate code (tf vs pytorch versions) as much as possible to be left with a clean backend-independent Agent-design.

[tarungog]

@sven1977 What is the timeline looking like for that?

[sven1977]

(simple) PPO is done and PR-submitted. APPO to follow. Not sure, may still take a few days or even weeks.

[tarungog]

@sven1977 awesome! does it have LSTM support?

[MadcowD]

@sven1977 Following up here re: LSTM support.

[sven1977]

I'm closing this issue.

• A PyTorch version of PPO has been added in 0.8.2 (then stabilized in 0.8.3 (fixed memory leak)).
• A PyTorch version of APPO is on its way into master via this PR: https://github.com/ray-project/ray/pull/8120