Reduce the impact of lagging observers

[wraitii]

Offshoot of ​https://trac.wildfiregames.com/ticket/3752#comment:10 Per elexis:

Observers are often frowned upon because of the possibility of them delaying the game by:

1. network timeouts
2. bad network latency
3. insufficient simulation performance
4. rejoins

In 1,2,3, the main problem is the NetServerTurnManager waiting for the observer to be ready for the next turn before progressing with the simulation (this isn't specific to observers, the turn manager waits for all clients to have sent all commands for turn N to mark turn N ready).

4 also blocks the game while one client computes the game state and sends it to the server, which can add lag independently of the above problem, but compounds it.

Fundamentally, observers shouldn't send simulation-affecting commands (NB: this is not saying they do now), so it ought to be possible to disregard them in this mechanism, effectively making 1/2/3 a total non-issue.

4 remains, but might be alleviated somewhat if the state can be queried from an already existing observer.

To explore:

• How to do this.
• Should observers still have a 'max lag' before disconnect? They might otherwise comment on completely unrelated turns [this can be fixed by attaching chat to a given turn, but that opens other issues]
• UI improvements.
• it could technically be possible to send the full replay from turn 0 to rejoining observers, meaning they could rewatch the game from the start, and thus not needing to send them the game state at all. This would fix 4 elegantly.

[wraitii]

In 25156:

Netcode: allow observers to lag behind the live game.

Observers no longer lag the game for players. There is still some time to serialise the game when sending it to a joining observer, and depending on the chosen 'max lag' the game may stop while observers sufficiently catch up, but this impact too is reduced.

• Make the NetServerTurnManager ignore players marked as 'observers' for the purpose of ending a turn, effectively making it possible for observers to lag without it affecting the players in any way.
• Add a config option (network.observermaxlag) that specifies how many turns behind the live game observers are allowed to be. Default to 10 turns, or 2 seconds, to keep them 'largely live'.
• The controller is not treated as an observer.
• Implement a simple UI to show this delay & allow the game to speed up automatically to try and catch up. This can be deactivated via network.autocatchup.
• Move network options to the renamed 'Network / Lobby' options page.
• Do not debug_warn/crash when receiving commands from the past - instead warn and carry on, to avoid DOS and "coop play" issues.

Refs #5903, Refs #4210

Differential Revision: ​https://code.wildfiregames.com/D3737