[PATCH] Visual Replay Menu

[elexis]

0ad will have the functionality to replay a game visually (see #9). To make this feature accessible to all users, a menu should be added that allows the user to select and start replays.

Features:

A new main menu item called "Replay" should be placed below the Singleplayer and Multiplayer buttons.

If clicked, it should show a list with selectable items, one for each replay file (commands.txt) found, similar to the Singleplayer->LoadGame dialog.

If an item is selected, it should display useful information of that match, including date, time, playernames, map thumbnail, victory condition and population count.

The dialog should have buttons "Cancel", "Delete" and "Play".

Also another text input line would be great, where you can enter some arbitrary strings, like playernames, mapnames, victory condition in order to filter the items displayed and find particular games quickly.

Technical Requirements:

The commands.txt must contain the engine version so that it can be checked explicitly for compability before replaying the file. It should also contain a timestamp, so that the date value won't be lost if players share their files over the internet.

The engine version is currently saved in XmppClient.cpp and should be moved to another file. Refs #2745.

m_client->disco()->setVersion("Pyrogenesis", "0.0.19");

[elexis]

Works, but the GUI should be enriched. Adds a main menu entry for listing replays and starting or deleting them. Cleans visual replay a bit. Saves engine_version.txt. Prevents the creation of empty commands.txt.

[elexis]

Works for r16777. Implements a nice list, filter capabilities and preview data. Remaining Todo: requires a check so that replays of old versions will be greyed out and not replayed. Some more data like players that resigned will be displayed if a spoiler button is pressed.

[elexis]

screenshot_v2.png: ​https://i.imgur.com/B3XIvFR.jpg

As you can see, visual replay really needs #3253.

[elexis]

Now lists are sortable. Thanks to vlad and sander for making #2405 happen. Added error message for corrupted files. Found out that 0ad crashes (because the the ifstream opening the commands.txt files is not good) if you open the menu the second time and I don't know why yet... Screenshot: ​https://i.imgur.com/4ImuMrQ.jpg

[elexis]

Fixed the crash (std::ifstream wasn't closed). After a replay, the summary screen now returns back to the replay selection screen.

[elexis]

This version has no known bugs and should be commitable. Unless you stop me though, I will do another version that makes the summary screen available from the menu directly by saving the summary screen stats at the end of the game.

General Enhancements in version 5: -moved engine version string to ps/EngineVersion.h -refactored VisualReplay::LoadReplayData -parses json in c++ instead of js -saves timestamp to replay (uses file modification date as fallback) -saves engine version to savegames and replays to add compatibility check

Tanslation Fixes: -getReplayDateTime now uses Engine.FormatMillisecondsIntoDateString -getReplayDuration now uses `timeToString -initDurationFilter now uses sprintf and translateWithContext

Whitespace Fixes: -changed some instances of "for (" to "for("

GUI: -adds compability filter and greys out incompatible replays. -two popup boxes informing the player if the replay is of an outdated version of 0ad or if the user only has to enable some mods -shows more replay info: victory condition, teams, civs, player colors, resigned players (with spoiler alert checkbox), AI details -displays filename in delete dialog (where else could the filename be displayed without having to click on delete?)

Other: -continues to use FileExists instead of VfsFileExists, because filenames are given as OsPath, i.e. an absolute pathname.

Please update ​http://trac.wildfiregames.com/wiki/ReleaseProcessDraft and replace XmppClient.cpp with EngineVersion.h after committing.

[elexis]

Saves the data that is passed to the summary screen to the commands.txt directory. Adds a link to the visual replay menu that allows browsing the summary screen (without replaying the whole game), so that you can see who won and how. Translates some more strings TODO: the game should remember all defeated players, not only those that resign

[elexis]

Now displays defeated players if spoiler checkbox is checked. Some refactoring (most notably moved ai difficulty names to a separate file). Some performance optimization. Renamed replay.js to replaymenu.js (and other related files).

[elexis]

Notice some new files:

svn add binaries/data/mods/public/gui/page_replaymenu.xml
svn add binaries/data/mods/public/gui/replaymenu/
svn add binaries/data/mods/public/gui/aiconfig/difficulties.json
svn add source/ps/VisualReplay.cpp
svn add source/ps/VisualReplay.h
svn add source/ps/EngineVersion.h

Don't forget to remove them after reverting the patch, otherwise you won't be able to compile and after applying the patch again the files will be messed up.

[elexis]

New in version 8:

Performance Enhancements:

• reduced filter initialization complexity, now O(1) instead of O(n)
• huge memory improvement after removing array copies in loadReplays()

Refactoring:

• moved constants from gamesetup.js, aiconfig.js and lobby.js to translatable json files
• moved shared translation functions to functions_utility.js
• uses const and let whereever possible

Visual Replay Enhancements:

• doesn't create empty commands.txt files anymore when returning from the gamesetup (by moving the file creation from CReplayLogger ctor to CReplayLogger::StartGame)

• fixes a bug where the ReplayFinished message was not being sent (when doing too fast forwarding on a laggy game caused by #3309) by moving the sending of that message to CNetReplayTurnManager::NotifyFinishedUpdate

Miscellaneous:

• many translation fixes
• correct handling of null in PlayerData array
• removed trailing whitespace & some useless comments

Remember to delete new files after reverting the patch:

rm binaries/data/mods/public/gui/page_replaymenu.xml
rm binaries/data/mods/public/gui/replaymenu/*
rm binaries/data/mods/public/simulation/ai/difficulties.json
rm binaries/data/mods/public/simulation/data/map_types.json
rm binaries/data/mods/public/simulation/data/population_capacity.json
rm source/ps/VisualReplay.cpp
rm source/ps/VisualReplay.h
rm source/ps/EngineVersion.h

[elexis]

Some cleanup & optimizations.

[elexis]

Pausing replays works again, refs r16871.

[elexis]

Supports each screen resolution.

[elexis]

Now supports all resolutions (1024x768 and greater). Adds clipping to COList.cpp to prevent overwriting neighboring text.

Example 1024 x 768: ​http://i.imgur.com/xuFwSQ4.png Example 1600 x 1200: ​http://i.imgur.com/wdvWmDh.jpg

Use this command to change the size of your window. If you chose a size bigger than your current resolution, you have to move the window or make a screenshot to see all areas.

wmctrl -r "0 A.D." -e 0,0,0,1600,1200

[elexis]

To reduce the size of the patch from 140kb to 60kb, it was split into several parts: r16895: Add clipping to OList cells (#3353) r16942, r16975, r16979, r17006, r17007 r17012, r17017, r17019, r17026, r17031: Move gamesetup strings to simulation/data/settings/ (#3355) r16906: Move engine version from XmppClient.cpp to Pyrogenesis.cpp and use it in savegames (#3359) r16976: Lists are sorted when clicking on the first item (not contained in the patch above, but relevant) (#3391) r17004 Don't create empty commands.txt (#3408) r16896 Allow pausing during visual replays r16897, r16898 Close and free replay streams r16902 Reliably send the ReplayFinished message at the very end of visual replays

Thanks leper, Itms and mimo for taking the time to review and commit the individual parts!

The new patch soon to be uploaded will include almost only new files implementing the replay menu.

[elexis]

Adds a getter in ReplayLogger, so that we can retrieve the directory path that contains the commands.txt file after the game started. Also adds the timestamp, g_modsLoaded, and engine_version (#3359) when the game starts. This way we will be able to filter outdated / incompatible replays.

[elexis]

Renames GameSetup::VisualReplay() to GameSetup::AutostartVisualReplay().

[elexis]

Rebased. This should be committed separately.

[elexis]

Everything needed for r17049.

[mimo]

In 17050:

rename autostart visualreplay, refs #3258, patch by elexis

[mimo]

In 17051:

move map description, refs #3258, patch by elexis

[elexis]

Everything needed for r17051. Updates the opening of the replayfiles, so that non-latin chars on unix are not supported anymore, but it should compile on windows on the other hand. Rebased as we could commit the changes to lobby.js and functions_utility.js already.

[elexis]

Adds script conversions ToJSVal<time_t> and ToJSVal<u64>. Update by Itms.

[elexis]

9.1 compiles on linux, but not on windows. 9.2 compiles on windows, but not on linux.

Notice that 9.2 loops indefintely in VisualReplay::LoadReplayData on windows, but not on linux.

[elexis]

These script conversions are required by windows. Either we need to modify this patch so that it compiles on linux too, or I need to change the code so that it doesn't need them.

[elexis]

As mentioned above, the prior version does an infinite loop on windows. Stan helped figuring out that removing replayStream->clear(); leads to that infinite loop instead of throwing the error message 5 lines below in case of !replayStream->good().

As mentioned in ​http://www.cplusplus.com/reference/ios/ios/clear/, this function resets the eof- and fail-bit.

Not sure what happens differently after that for the two operating systems yet. Couldn't test on windows yet. Above I uploaded a patch that rewrites the affected function VisualReplay::LoadReplayData, which in theory should never reach the fileending. Works for linux.

If you want to test it on windows, you will probably need t3258_missing_script_conversions_for_windows_v1.patch again.

Changes:

• For easier testing: added some optional debug_printf() calls. Enable them by setting const bool debugParser = true;. Please report errors on windows with that debug output and commands.txt.
• Does not use replayStream->clear(); to reset the failbit / eofbits when attempting to read at EOF.
• Reduced the infinite loop protection from 10 million to 1000. If it doesn't fail there are 5 iterations on average.
• Doesn't call m_Stream->close() anymore: ​http://www.cplusplus.com/reference/fstream/ifstream/close/, but SAFE_DELETE whenever leaving the stream.

Updates & test reports for windows will follow soon enough hopefully!

[Stan]

Patch 10 + Js conversion works for me on Windows 10

[elexis]

We can make it compile on both systems by not updating the script conversions and instead converting the affected variables explicitly in CReplayLogger::StartGame and VisualReplay::LoadReplayData:

< +	m_ScriptInterface.SetProperty(attribs, "timestamp", (i32) std::time(nullptr));
---
> +	m_ScriptInterface.SetProperty(attribs, "timestamp", std::time(nullptr));
1577c1577
< +	scriptInterface.SetProperty(replayData, "filemod_timestamp", std::to_string(fileTime));
---
> +	scriptInterface.SetProperty(replayData, "filemod_timestamp", fileTime);

[elexis]

Compiles on both windows and linux and doesn't freeze.

[Stan]

Works fine for me on Windows 10. Takes some time to load for the first time, maybe a loading screen ? else it's cool :)

[elexis]

Replying to stanislas69:

Works fine for me on Windows 10. Takes some time to load for the first time

---

Performance Profile:

Made some profilings with 2500 replay files.

Result: everything seems to be optimized, only HDD I/O overhead remaing. Not a mission critical problem in my opinion and the optimization can be done easily. The slow-loading occurs only the first time you load it. Instead of adding a loading screen, we should add a custom cache file to optimize.

---

C++ parsing: In order to display the players, map and duration, we have to load these files commands.txt files. As leper suggested, this is done the fastest way by parsing only the first line (header) and then read the file backwards to get the game duration from the last turn number. The C++ parsing of the files is optimized.

JS parsing & GUI updates: The javascript part loads in half a second with 2500 files on my system. I've measured the time spent in every function and it is evenly distributed (about 30ms per function). Could be improved maybe, but saving a fraction of a second is not really worth it now.

Disk I/O: The real underlying problem with the first load being slow is that we have to load a big amount of small files. While the parsing of the files and the GUI is optimized, we still have all those disk I/Os calls. Every file is located in a different place on the harddrive and thus the disk has to do mechanical work in that time. But everytime a file is read, it will cache that file in a small memory inside the HDD. This is the reason why on the second call it only takes 0,5s!.

You can reproduce the disk I/O calls on linux by clearing the HDD cache:

echo 3 | sudo tee /proc/sys/vm/drop_caches

The other game parts have the same problem. The HDD cache just needs to be loaded at some point. The game-data that doesn't change is zipped and uses VFS, see wiki:Virtual_File_System. leper suggested using it. But using it is challenging, some required functions seem missing with the current implementation. More importantly this doesn't eliminate the actual problem of having disk I/O for many scattered files once.

The ideal solution to that issue is to keep a single, custom cache file (json format) that contains the data we are interested in. We would have to do the disk I/O calls only once per replay, instead of once per "replay viewing session". This means the actual loading of the files happens only once, ever.

Implementing such a cache is a low hanging fruit. Will start working on it right now, though it would be cleaner to implement and easier to review if it is added in a separate patch.

[Stan]

I think best thing is to get that patch commited, and make another one to enhance performance.

[elexis]

Same patch above, but with profiling output. Notice that showing the output also consumes performance.

[Itms]

In 17054:

An awesome Visual Replay menu, made by elexis. Fixes #3258.

[Itms]

Thanks a lot for your hard work :D

[elexis]

We actually made it!!!

It would not have been possible to finish this,

• if leper wouldn't have committed the patches that Itms couldn't review
• if Itms wouldn't have committed the patches that leper couldn't review
• if Itms wouldn't have continued the reviewing when leper was overworked
• if mimo wouldn't have continued the reviewing when Itms was overworked
• if Stan wouldn't have helped debugging it on windows

If any of you guys were missing, we couldn't have made it! Thanks for the endurance and going through dozens of iterations of these patches!

Here the history (28 patches consuming 180kb, introducing 2792 new lines and removing 565 existing ones):

---

Feb 7 Requirements analysis Feb 12 < elexis> if i wont get struck by thunderstrike i will implement this for a19 Mar 9 Alpha 18 release Apr 3 First visual replay patch Jun 6 r16727 Itms Visual Replay from command line (#9) Jun 12 First replay menu patch Jun 14 r16765 ben Update readme (#9) Jun 18 r16790 sanderd17 Fix out-of-sync in replays (#9) Jul 8 r16839 Itms Playernames in replay files (#3253) Jul 14 r16854 wraitii Fix previous commit (#3253) Jul 30 r16895 leper Add clipping to OList cells (#3353) Jul 30 r16896 leper Pausing in visual replays (#3341) Jul 30 r16897 leper Close and free replay streams Jul 30 r16898 leper Fix previous commit Jul 30 r16902 Itms Fix "replay finished" dialog (#9) Jul 30 Splitting visual replay menu patch, as it grew too big (ticket:3258#comment:8) Aug 1 r16906 leper Move engine version to Pyrogenesis.cpp (#3359) Aug 11 First version of the gamesetup cleanup (#3355) Aug 27 Splitting gamesetup cleanup, as it grew too big Aug 27 r16942 leper Enhance i18n extractor (#3355) Sep 2 r16975 Itms Move player and team number limits (#3355) Sep 2 r16976 Itms Lists are sorted when clicking on the first item (#3391) Sep 3 r16979 Itms Fix previous commit (#3355) Sep 11 r17004 leper Don't create empty commands.txt (#3408) Sep 12 r17006 Itms Move ceasefire settings (#3355) Sep 12 r17007 mimo Move starting resources (#3355) Sep 13 r17012 leper Move population capacities (#3355) Sep 13 r17017 mimo Rename replay variables (#9) Sep 13 r17019 mimo Fast forward replays and move game speeds (#3355) Sep 16 r17026 mimo Move victory conditions and fix #3209 (#3355) Sep 17 r17031 mimo Move maptypes (#3355) Sep 18 r17034 mimo Move AI settings (#3355) Sep 20 r17049 Itms Fix move-maptypes (#3355) Sep 20 r17050 mimo Rename autostart visual replay (#9) Sep 21 r17054 Itms Visual Replay Menu! (#3258) Sep 21 r17055 Itms Finish previous commit (#3258)

---

The next task is to fill the replay list with many gg's!