Remote Interface for Reinforcement Learning

[Brian Broll]

It would be great if 0 AD provided an interface which enabled experimentation with RL for the built-in AI (similar to OpenAI gym). This would allow people to train their own RL/ML agents and could be useful for non-ML agents as they could be prototyped in Python, if desired. This could be useful both for using 0 AD as a research sandbox to explore challenging RL/ML problems as well as benefit 0 AD by facilitating experimentation with new techniques for the built-in AI.

There are also a number of related posts on the forum that seem like they would benefit from this (or are talking about this directly):

• ​https://wildfiregames.com/forum/index.php?/topic/25691-trying-to-improve-petra-bot/&tab=comments#comment-373060
• ​https://wildfiregames.com/forum/index.php?/topic/26307-vfs-error-while-trying-to-read-a-file-javascript
• ​https://wildfiregames.com/forum/index.php?/topic/21449-openai/

I have ​implemented such an interface and would love to contribute it back to the community but wanted to open a ticket for related discussion! (I will be doing a little bit of remaining clean up and then submitting the patch here!)

[Stan]

Hello and welcome here

Itms and I have been looking forward to your patch and I've been following it from afar. It sounds like a great feature !

I'd suggest you to read SubmittingPatches and upload your patch to code.wildfiregames.com to get a review.

Thanks for the good work so far.

EDIT: Might also want to look at ​https://trac.wildfiregames.com/wiki/Coding_Conventions

[wraitii]

I've also been following from afar. I'm somewhat familiar with gRPC and this is definitely something I'm interested in at least conceptually.

Indeed, it would be good if you posted a patch to Phabricator out of formalism. However do keep the git branch as that will make it much easier to review.

Finally, just to set up expectations, don't hope that this will take a few days, as it's more in the realm of a few weeks even when things go well :)

[Brian Broll]

While making the revision for Phabricator, I realized that it included code for recording replays in nonvisual mode which is pretty useful when training an agent. I made a separate ticket and revision for these changes: ​https://trac.wildfiregames.com/ticket/5565 and ​https://code.wildfiregames.com/D2197

[elexis]

In 22991:

Save replay metadata for non-visual games too, refs #4577, #5548, fixes #5565.

Remove gui/common/ hack by going through the GUIInterface and fragile EndGame() hardcoding by saving if and only if a ReplayLogger exists following rP18613, refs #4020.
Refs D2211, D2213.

Differential Revision: ​https://code.wildfiregames.com/D2197
Based on patch by: irishninja
Tested on: clang 8.0.1, Jenkins

[wraitii]

In 23917:

Add an interface for Reinforcement Learning.

Implement a simple HTTP server to start games, receive the gamestate and pass commands to the simulation.
This is mainly intended for training reinforcement learning agents in 0 AD. As such, a python client and a small example are included.

This option can be enabled using the -rl-interface flag.

Patch by: irishninja

Reviewed By: wraitii, Itms

Fixes #5548

Differential Revision: ​https://code.wildfiregames.com/D2199