JS out of memory error on very large Ardennes Forest rms

[zouip]

Multiplayer game setup: 2 humans versus 2 bots (qbots or jubots) on a random map. Few time after start (less than 10 seconds) a popup appears which indicates that the game is out of sync and in-game logs lists this sort of errors:

[...]
TIMER| ParseTerrain: 9.2115 ms
ERROR: JavaScript error: simulation/components/AIProxy.js line 39
out of memory
ERROR: JavaScript error: simulation/components/Player.js line 339
out of memory
ERROR: Script message handler OnGlobalOwnershipChanged failed
ERROR: JavaScript error: simulation/components/Player.js line 339
out of memory
ERROR: Script message handler OnGlobalOwnershipChanged failed
ERROR: JavaScript error: simulation/components/UnitAI.js line 1731
out of memory
WARNING: JavaScript warning: simulation/components/UnitAI.js line 3099
reference to undefined property g_Stances[this.stance]
ERROR: JavaScript error: simulation/components/UnitAI.js line 3055
TypeError: this.GetStance() is undefined
  (37)@simulation/components/UnitAI.js:3055
  ()@simulation/components/UnitAI.js:1827
  ([object Object])@simulation/components/UnitAI.js:1779
ERROR: Script message handler OnOwnershipChanged failed
ERROR: JavaScript error: simulation/components/ResourceSupply.js line 37
out of memory
ERROR: JavaScript error: simulation/components/Player.js line 339
out of memory
ERROR: Script message handler OnGlobalOwnershipChanged failed
ERROR: JavaScript error: simulation/components/ResourceSupply.js line 37
out of memory
ERROR: JavaScript error: simulation/components/AIInterface.js line 51
out of memory
ERROR: Script message handler OnCreate failed
ERROR: JavaScript error: simulation/components/AIProxy.js line 39
out of memory
ERROR: JavaScript error: simulation/components/ResourceSupply.js line 37
out of memory
ERROR: JavaScript error: simulation/components/UnitAI.js line 3055
out of memory
ERROR: Script message handler OnOwnershipChanged failed
ERROR: JavaScript error: simulation/components/Health.js line 39
out of memory
ERROR: JavaScript error: simulation/components/AIProxy.js line 256
out of memory
ERROR: Script message handler OnCreate failed
ERROR: JavaScript error: simulation/components/AIInterface.js line 51
out of memory
ERROR: Script message handler OnCreate failed
ERROR: JavaScript error: simulation/components/ResourceSupply.js line 37
out of memory
ERROR: JavaScript error: simulation/components/AIInterface.js line 51
out of memory
ERROR: Script message handler OnCreate failed
[...etc...]

Config of the 2 humans:

• host with 0ad r11863 (x86_64) on ArchLinux x86_64
• client with 0ad r11863 (win32) on Win7 x86_64

PS: there is no error without bots.

[zouip]

Log files of host and client (without oos_dump)

[historic_bruno]

Replying to zouip:

Muliplayer game setup: 2 humans versus 2 bots (qbots or jubots) on a random map. Few time after start (less than 10 seconds) a popup appears which indicates that the game is out of sync

OOS is to be expected once you get out of memory errors, that means the game is no longer working as intended (memory allocations are failing), so the host and client state will no longer match up. I think in the future we should handle the errors more gracefully (end the game).

Looks like you were playing a very large map with many entities (the random maps can have several times more than scenarios), so that's one possible reason why you'd get an out of memory error. Can you confirm that playing the same map with identical settings on the host computer will produce this error every time? Ardennes Forest, Very large, 4 players, 2 qBots.

"Densely" forested maps are currently more prone to these errors. I'll CC Spahbod to see if he has any input.

But one interesting difference is the host is playing the 64-bit version of the game and ran out of memory, while the client was using a 32-bit build and did not. The JS out of memory errors are slightly deceiving; it's not a question of how much RAM is available on the system, but it's relative to how much memory was allocated for the particular script runtime. Which defaults to 16MB currently. And I know for instance that 64-bit data structures can take up significantly more memory due to pointers, which might become an issue during serialization. Maybe we just need to increase the runtime allocation for the simulation experimentally (UI is probably fine with 16 MB or less) while being careful that we don't push the overall memory footprint of the game too high.

[zouip]

Replying to historic_bruno:

Looks like you were playing a very large map with many entities (the random maps can have several times more than scenarios), so that's one possible reason why you'd get an out of memory error. Can you confirm that playing the same map with identical settings on the host computer will produce this error every time? Ardennes Forest, Very large, 4 players, 2 qBots.

Yep, I confirm.

But one interesting difference is the host is playing the 64-bit version of the game and ran out of memory, while the client was using a 32-bit build and did not. The JS out of memory errors are slightly deceiving; it's not a question of how much RAM is available on the system, but it's relative to how much memory was allocated for the particular script runtime. Which defaults to 16MB currently. And I know for instance that 64-bit data structures can take up significantly more memory due to pointers, which might become an issue during serialization. Maybe we just need to increase the runtime allocation for the simulation experimentally (UI is probably fine with 16 MB or less) while being careful that we don't push the overall memory footprint of the game too high.

Regardless of the type of host (bin32 on win64 or bin64 on linux64) the same error still happening (if it's a very large map with lot of trees etc.).

Thanks.

[ben]

In 13659:

Increases simulation script runtime to 32 MB to reduce OOM errors, refs #1436, #1966, #2035

[leper]

In 13794:

Increase simulation script runtime size again. Refs #1436, #1966 and #2035.

[historic_bruno]

Going to close this, due to the SpiderMonkey upgrade and the simulation using a single large runtime now. Any new OOM errors should be dealt with in that context, as separate tickets.