Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#5548 closed enhancement (fixed)

Remote Interface for Reinforcement Learning

Reported by: Brian Broll Owned by: wraitii
Priority: Nice to Have Milestone: Alpha 24
Component: Core engine Keywords:
Cc: Itms Patch: Phab:D2199

Description (last modified by Brian Broll)

It would be great if 0 AD provided an interface which enabled experimentation with RL for the built-in AI (similar to OpenAI gym). This would allow people to train their own RL/ML agents and could be useful for non-ML agents as they could be prototyped in Python, if desired. This could be useful both for using 0 AD as a research sandbox to explore challenging RL/ML problems as well as benefit 0 AD by facilitating experimentation with new techniques for the built-in AI.

There are also a number of related posts on the forum that seem like they would benefit from this (or are talking about this directly):

I have implemented such an interface and would love to contribute it back to the community but wanted to open a ticket for related discussion! (I will be doing a little bit of remaining clean up and then submitting the patch here!)

Change History (8)

comment:1 by Brian Broll, 5 years ago

Description: modified (diff)

comment:2 by Stan, 5 years ago

Cc: Itms added

Hello and welcome here

Itms and I have been looking forward to your patch and I've been following it from afar. It sounds like a great feature !

I'd suggest you to read SubmittingPatches and upload your patch to code.wildfiregames.com to get a review.

Thanks for the good work so far.

EDIT: Might also want to look at https://trac.wildfiregames.com/wiki/Coding_Conventions

Last edited 5 years ago by Stan (previous) (diff)

comment:3 by wraitii, 5 years ago

I've also been following from afar. I'm somewhat familiar with gRPC and this is definitely something I'm interested in at least conceptually.

Indeed, it would be good if you posted a patch to Phabricator out of formalism. However do keep the git branch as that will make it much easier to review.

Finally, just to set up expectations, don't hope that this will take a few days, as it's more in the realm of a few weeks even when things go well :)

comment:4 by Brian Broll, 5 years ago

While making the revision for Phabricator, I realized that it included code for recording replays in nonvisual mode which is pretty useful when training an agent. I made a separate ticket and revision for these changes: https://trac.wildfiregames.com/ticket/5565 and https://code.wildfiregames.com/D2197

comment:5 by Brian Broll, 5 years ago

Milestone: BacklogWork In Progress
Patch: Phab:D2199

comment:6 by elexis, 5 years ago

In 22991:

Save replay metadata for non-visual games too, refs #4577, #5548, fixes #5565.

Remove gui/common/ hack by going through the GUIInterface and fragile EndGame() hardcoding by saving if and only if a ReplayLogger exists following rP18613, refs #4020.
Refs D2211, D2213.

Differential Revision: https://code.wildfiregames.com/D2197
Based on patch by: irishninja
Tested on: clang 8.0.1, Jenkins

comment:7 by wraitii, 4 years ago

Owner: set to wraitii
Resolution: fixed
Status: newclosed

In 23917:

Add an interface for Reinforcement Learning.

Implement a simple HTTP server to start games, receive the gamestate and pass commands to the simulation.
This is mainly intended for training reinforcement learning agents in 0 AD. As such, a python client and a small example are included.

This option can be enabled using the -rl-interface flag.

Patch by: irishninja

Reviewed By: wraitii, Itms

Fixes #5548

Differential Revision: https://code.wildfiregames.com/D2199

comment:8 by wraitii, 4 years ago

Milestone: Work In ProgressAlpha 24
Note: See TracTickets for help on using tickets.