[PATCH]Find a way to shrink or split OOS logs

[sanderd17]

OOS logs are getting too big to work with.

Logs of long games (when there are a big number of units) can't be opened with all text editors anymore, and there's certainly a problem with certain diff tools (like meld).

Some things could be optimised:

• TechnologyTemplateManager shouldn't serialise any data (it can always load the techs from disk when rejoining). r18100 (-> backref OOS #4239)
• SerializeVector is very ineffecient; it stores the name of the array with every element, which results f.e. in a lot of lines like. r18121
• TemplateManager stores the templates rather inefficiently. r18121
• AIInterface serializes all templates, while it isn't needed at all. r18121
• AIManager shouldn't serialiaze any data when there are no ais. r18121
• AuraManager can clean up the applied auras while running (the deletion of objects was omitted in earlier versions because it was bad for performance, but this isn't an issue anymore when using Maps). r18123
• AIManager serializes the big pathfinder grid. This should be loaded in a Deserialized method (after deserialization of the pathfinder).
• ObstructionManager a lot of information on the entities that can be derived from other components (like the Position), or from the templates
• TechnologyManager shouldn't save the autoresearch techs, they can perfectly be calculated on deserializaton (or otherwise, saving the names is enough) r20606

Next to optimising the OOS log, it could also be split. F.e. one log for the SYSTEM_ENTITY (which currently takes up 25% of the entire log), and then f.e. a log per 1000 or 2000 entities

[sanderd17]

In 18100:

Rename TechnologyTemplateManager to DataTemplateManager in order to reflect its new function. Fixes #3909. Disable serialisation of technology templates. Refs #3834

[sanderd17]

The patch above implements the following changes:

• It disables serializing if cmpAIInterface.templates, as that was only used as a cache (didn't need serialisation), while it caused a lot of data.
• A new serializer for repetitive vectors has been made, and use for the LOS and Visibility maps. Basically, instead of noting every value, it notes series of equal values. It would double the space of a random vector, but it seriously reduces the space needed of repetitive vectors (where sometimes 100 subsequent elements are the same).
• The serialization of the cmpTemplateManager follows a similar structure, which also reduces the size there because people tend to create entities in batches (place 100 trees in Atlas, produce 10 units at once, ...)

Some other things that could be improved:

• the AI still has some caches that could be removed
• the AuraManager cache could be removed, and the object could be kept cleaner while the game is running (deleting some keys instead of putting the values back to the default).

Note that the serialization test mode currently doesn't work, but it's also broken in the main game. Which means I can't test this patch completely.

[sanderd17]

Example of how the LOS array is now encoded:

    #: 1
    los state: 1
    #: 213
    los state: 0
    #: 43
    los state: 3
    #: 214
    los state: 0
    #: 42
    los state: 3
    #: 216
    los state: 0
    #: 40
    los state: 3
    #: 1
    los state: 1
    #: 217
    los state: 0
    #: 37
    los state: 3
    #: 3
    los state: 1
    #: 218
    los state: 0
    #: 35
    los state: 3
    #: 222
    los state: 0
    #: 1
    los state: 1
    #: 33
    los state: 3
    #: 1
    los state: 1
    #: 222
    los state: 0
    #: 3
    los state: 1
    #: 29
    los state: 3
    #: 3
    los state: 1
    #: 222
    los state: 0

[sanderd17]

In 18121:

Disable serialization of the AI when no AI players are present. Disable serialization of cached AI templates overall. Improve serialization of repetitive vectors and templatenames. Refs #3834

[sanderd17]

In 18123:

Clean up AuraManager objects while the game is running. Refs #3834

[elexis]

Backlogging due to lack of progress.

[Itms]

In 18752:

Fix a frequent rejoining OOS. We actually need to serialize the data loaded from the disk, because JS objects in the memory or newly loaded from the disk will not behave the same way.

Fixes #4239, refs #3834.

[mimo]

In 19242:

Cleanup of CCmprangeManager

Summary: Following discussions in ​https://code.wildfiregames.com/D60 the use of u8 to store each boolean has been cleanup up. Now only one u8 is used, thus reducing the size of the struct.

Reviewed By: leper

Differential Revision: ​https://code.wildfiregames.com/D101

refs #3834

[elexis]

In 20606:

Don't serialize the templates of autoresearched technologies.

Refs #3834, #4239
Differential Revision: ​https://code.wildfiregames.com/D1109
Reviewed By: temple
Comments By: Itms
Proposed By: sanderd17

[elexis]

In 20610:

Remove unused serialized cache of technology templates in TechnologyManager.researchedTechs.

Refs #3834, #3909, #4239, #4868, D1108
Differential Revision: ​https://code.wildfiregames.com/D1130
Reviewed By: mimo

[elexis]

In r20611:

Use Set instead of an Object with unused values in TechnologyManager.researchStarted.

Differential Revision: ​​https://code.wildfiregames.com/D1129
Reviewed By: mimo

[elexis]

In 20737:

Replace DataTemplateManager simulation component with a globalscript, refs #4868.

Removes the serialization of JSON files, shrinking savegame files and rejoin states sent across the network, refs #3834, #4239, #3909, rP18100.
Removes the AI C++ code to read JSON files from rP13225 since the AI can now use the globalscript.
Allows the AI to read Aura templates and removal of GUIInterface code to improve performance.
Serialization of the JSON objects in other simulation components was removed in r20606 / D1109, r20610 / D1130.

Serialization removal planned by sanderd17
AI part proofread by mimo
Simulation part proofread by bb
Discussed with Itms on irc

Differential Revision: ​https://code.wildfiregames.com/D1108

[bb]

In r20797

Stop serializing minimap playercolors

Patch By: temple
Comments By: elexis
Differential Revision: ​https://code.wildfiregames.com/D1124

[elexis]

In 21014:

Fix AuraManager OOS on rejoin following rP20737 / D1108.

The AuraManager serialized the affects array of the aura JSON files, while the Auras components create new affects arrays upon deserialization.
So the AuraManager of rejoined clients failed to identify and remove the according aura effects of units who received the aura effect before the rejoin.

Fix it by not serializing the array but only the strings inside the array.
It were preferable to rebuild the modification cache and remove the serialization altogether, refs #3834.

rP18100 introduced the same OOS on rejoin, refs #3909, #4239, but it wasn't noticed due to the revert in rP18804.

Differential Revision: ​https://code.wildfiregames.com/D1201
Reviewed By: bb
Comments By: gameboy
Fixes #4924

[elexis]

In r22103 by Stan:

Clean ResourceSupply up.
Add documentation
Reviewed by: @Imarok
Authored by: @Stan

Differential Revision: ​​https://code.wildfiregames.com/D1771

Removes the serialization of infinite amount of ResourceSupply.js.

[Freagarach]

In 24993:

Do not initialise null values in cmpGarrisonHolder.

Saves some serialisation data (refs #3834).

Ticket: #5979
Differential revision: ​Phab:D3468
Comments by: @Angen, @Stan