Opened 8 years ago

Last modified 3 years ago

#3834 new enhancement

[PATCH]Find a way to shrink or split OOS logs — at Version 7

Reported by: sanderd17 Owned by:
Priority: Should Have Milestone: Backlog
Component: Core engine Keywords: patch
Cc: Patch:

Description (last modified by sanderd17)

OOS logs are getting too big to work with.

Logs of long games (when there are a big number of units) can't be opened with all text editors anymore, and there's certainly a problem with certain diff tools (like meld).

Some things could be optimised:

  • TechnologyTemplateManager shouldn't serialise any data (it can always load the techs from disk when rejoining). r18100
  • SerializeVector is very ineffecient; it stores the name of the array with every element, which results f.e. in a lot of lines like. r18121
  • TemplateManager stores the templates rather inefficiently. r18121
  • AIInterface serializes all templates, while it isn't needed at all. r18121
  • AIManager shouldn't serializa any data when there are no ais. r18121
  • AIManager serialized the big pathfinder grid. This should be loaded in a Deserialized method (after deserialization of the pathfinder).
  • AuraManager can clean up the applied auras while running (the deletion of objects was omitted in earlier versions because it was bad for performance, but this isn't an issue anymore when using Maps), it also doesn't need to store the caches.
  • ObstructionManager a lot of information on the entities that can be derived from other components (like the Position), or from the templates
  • TechnologyManager shouldn't save the autoresearch techs, they can perfectly be calculated on deserializaton (or otherwise, saving the names is enough)

Next to optimising the OOS log, it could also be split. F.e. one log for the SYSTEM_ENTITY (which currently takes up 25% of the entire log), and then f.e. a log per 1000 or 2000 entities

Change History (10)

comment:1 by sanderd17, 8 years ago

In 18100:

Rename TechnologyTemplateManager to DataTemplateManager in order to reflect its new function. Fixes #3909. Disable serialisation of technology templates. Refs #3834

comment:2 by Stan, 8 years ago

Description: modified (diff)

by sanderd17, 8 years ago

Attachment: smallerOOSlog.diff added

comment:3 by sanderd17, 8 years ago

The patch above implements the following changes:

  • It disables serializing if cmpAIInterface.templates, as that was only used as a cache (didn't need serialisation), while it caused a lot of data.
  • A new serializer for repetitive vectors has been made, and use for the LOS and Visibility maps. Basically, instead of noting every value, it notes series of equal values. It would double the space of a random vector, but it seriously reduces the space needed of repetitive vectors (where sometimes 100 subsequent elements are the same).
  • The serialization of the cmpTemplateManager follows a similar structure, which also reduces the size there because people tend to create entities in batches (place 100 trees in Atlas, produce 10 units at once, ...)

Some other things that could be improved:

  • the AI still has some caches that could be removed
  • the AuraManager cache could be removed, and the object could be kept cleaner while the game is running (deleting some keys instead of putting the values back to the default).

Note that the serialization test mode currently doesn't work, but it's also broken in the main game. Which means I can't test this patch completely.

Last edited 8 years ago by sanderd17 (previous) (diff)

comment:4 by sanderd17, 8 years ago

Example of how the LOS array is now encoded:

    #: 1
    los state: 1
    #: 213
    los state: 0
    #: 43
    los state: 3
    #: 214
    los state: 0
    #: 42
    los state: 3
    #: 216
    los state: 0
    #: 40
    los state: 3
    #: 1
    los state: 1
    #: 217
    los state: 0
    #: 37
    los state: 3
    #: 3
    los state: 1
    #: 218
    los state: 0
    #: 35
    los state: 3
    #: 222
    los state: 0
    #: 1
    los state: 1
    #: 33
    los state: 3
    #: 1
    los state: 1
    #: 222
    los state: 0
    #: 3
    los state: 1
    #: 29
    los state: 3
    #: 3
    los state: 1
    #: 222
    los state: 0

by sanderd17, 8 years ago

Attachment: smallerOOSlog.2.diff added

comment:5 by sanderd17, 8 years ago

Description: modified (diff)
Keywords: patch review added
Milestone: BacklogAlpha 21
Summary: Find a way to shrink or split OOS logs[PATCH]Find a way to shrink or split OOS logs

by sanderd17, 8 years ago

Attachment: smallerOOSlog.3.diff added

comment:6 by sanderd17, 8 years ago

In 18121:

Disable serialization of the AI when no AI players are present. Disable serialization of cached AI templates overall. Improve serialization of repetitive vectors and templatenames. Refs #3834

comment:7 by sanderd17, 8 years ago

Description: modified (diff)
Keywords: review removed
Note: See TracTickets for help on using tickets.