[macOS] Memory issue (sandboxed, spindump, ReportCrash) Spidermonkey 91

[Langbart]

to consistently reproduce (1)

• add this debug message in chatHelper.js

binaries/data/mods/public/simulation/ai/petra/chatHelper.js

@@ -220 +220 @@
 
 PETRA.chatAnswerRequestDiplomacy = function(gameState, player, requestType, response, requiredTribute)
 {
+    warn(uneval(['debug', gameState, player, requestType, response, requiredTribute]))
     Engine.PostCommand(PlayerID, {
         "type": "aichat",
         "message": "/msg " + gameState.sharedScript.playersData[player].name + " " +

• start the attached replay from ticket #6654 (~2mins long)
  • commands.txt​
• the game freezes at the 97th turn
• forcefully quit the pyrogenesis app
• check activity monitor for spindump
  • For the case described above, the process is always called spindump to allocate large amounts of memory, but the process can also be called sandboxed or ReportCrash in other situations.

to consistently reproduce (2)

• pass this to the terminal and run

  pyrogenesis -replay-visual="/Users/paria\ crash"
  

• the game crashes with the following message

• check activity monitor for ReportCrash
  • same as above with spindump, if I don`t stop the process manually, the PC becomes unusable with time

to consistently reproduce (3)

• the sandboxed process occurs when I try to start a replay from the built-in command line within the VSCodium (based of VSCode) app.

  pyrogenesis -replay-visual=<here is your path to a legit replay, does not matter which one>
  

bisect

[27409]

related ticket

• #6697

related articles

• ​Why are "spindump" and "sandboxd" eating up all my CPU? (27/May/21)
• ​ReportCrash High CPU & How to Disable reportcrash in Mac OSX (28/Sep/16)
• ​What is Spindump? How to read, analysis and disable Spindump (27/Sep/18)

[Stan]

In your case it's printing the game state thousands of times that crashes the game. Is it just a way to make it faster? Can't be vulkan if it happened before. So it's Spidermonley most likely but it's probably cause we're doing something it does not like

[Langbart]

Replying to Stan‘:

In your case it's printing the game state thousands of times that crashes the game. Is it just a way to make it faster? Can't be vulkan if it happened before. So it's Spidermonley most likely but it's probably cause we're doing something it does not like

the issue is not about the lldb (ignore that), it about an external processing spinning up called spindump.

[wraitii]

@langbart -> Did this happen to you in regular play? Or just when adding some debugs?

[Stan]

spindump and sandboxd are macOS diagnostic applications apparently.

[Langbart]

Replying to wraitii:

@langbart -> Did this happen to you in regular play? Or just when adding some debugs?

• I have also noticed it in a regular game but have not found consistently reproducible steps.
  • I described some details here: Comment 1 in Ticket 6697

[Langbart]

The description was extended by another possibility to reproduce the problem consistently.

[Langbart]

• Remove bisect number
  • conflicting test results, could still have been due to spidermonkey
  • reproducing the steps with the released A26 version did not show any problems, so the problem must be with macOS or somewhere in the development cycle of A27

• lldb has been removed as it is not needed for the problem.

• Reproduction steps remain valid

[Langbart]

Installed macOS Dependencies

❯ uname -a
Darwin Paria 19.6.0 Darwin Kernel Version 19.6.0: Tue Jun 21 21:18:39 PDT 2022; root:xnu-6153.141.66~1/RELEASE_X86_64 x86_64

❯ xcode-select --version
xcode-select version 2373.

❯ rustc --version
rustc 1.66.1 (90743e729 2023-01-10) (built from a source tarball)

❯ cmake --version
cmake version 3.25.2

[Vladislav Belov]

Replying to Langbart:

• Remove bisect number

I suppose the name is outdated then.

[Stan]

Have you updated macOS recently? Both examples you mentioned produce actual crashes maybe it's a requirement. E.g you second example is just making it crash with an unsupported path

[Langbart]

• terminal output: fresh install with --force-rebuild on library folder (latest GIT [27471])
  • ​https://ttm.sh/S0n.txt
• issue remains as described in the description
• Rename the title without including the alleged changeset, since it is no longer certain that this is the reason for the problem

---

Replying to Stan‘:

Have you updated macOS recently?

• always macOS 10.15.7
• only rust/cmake are updated via homebrew when new updates are released

Both examples you mentioned produce actual crashes maybe it's a requirement. E.g you second example is just making it crash with an unsupported path

The crash doesn't matter, it's just the trigger. It's about the external process eating up my memory. Crashes happened with A26 in testing as well, but never causes this external process to allocate all my memory.

[Langbart]

two videos

A26 release version

• all good, no over allocation of memory after the crash

​https://ttm.sh/S0T.5x.mp4

current pyrogenesis build

• the problem occurs as described in the description

​https://ttm.sh/S0p.5x.mp4

[Stan]

Does it do something different in release or debug modes? I know it's tedious but since it seems reproductible can you try to bisect again?

I'll see if I can try on my macos (i have monterey)

We need to figure out why macos does weird stuff with the game. When you say A26 you mean the bundle or a version you compiled yourself?

[Langbart]

Replying to Stan‘:

When you say A26 you mean the bundle or a version you compiled yourself?

Yes, the official version downloaded with Homebrew.

spindump points at [27412], ReportCrash pointing at [27409]

I don't know how to bisect that.

---

the sandboxed issue occurs when I try to start a replay from the built-in command line within the VSCodium (based of VSCode) app.

[Stan]

Could you try a self compiled A26? What if you run outside of vscodium?

[Stan]

Just tested on macos, I cannot reproduce.

[Langbart]

Replying to Stan‘:

Could you try a self compiled A26?

is not on my wish list today, I don't think I need to. I compiled with [27408] and none of the problems listed here occur. All problems came with [27409] and above.

What if you run outside of vscodium?

then the sandboxed issue does not occur

---

for the ReportCrash process it would fill my Console (macOS default installed application) with out of bounds abstract origin messages and keeps going till i kill the process

[Stan]

Maybe vscode is sandboxed. Is there any more info about those warnings

[Langbart]

Yes, there is the possibility to create a sample from the Activity Monitor for any process. Here are the samples that are created once I triggered the issue.

• ReportCrash issue
  • ReportCrash.txt​
• sandboxed issue
  • sandboxed.txt​
• spindump issue
  • spindump.txt​

[Vladislav Belov]

Could you update to r27412 and revert the only vulkan folder from it?

[Langbart]

Replying to Stan‘:

Does it do something different in release or debug modes?

• Below is the result from running pyrogenesis_dbg
  • BuildInstructions#Buildthegame macOS debug

ReportCrash issue

• the line below that starts with Hit MOZ_CRASH()... was revealed

  ❯ pyrogenesis_dbg -replay-visual="/Users/paria\ crash"
  Path /Users/paria\ crash, separator /
  Function call failed: return value was -100303 (path contains both slash and backslash separators)
  Location: path.h:292 (DetectSeparator)
  
  Call stack:
  
  (error while dumping stack: Function not supported)
  errno = 0 (No error reported here)
  OS error = ?
  
  
  (C)ontinue, (S)uppress, (B)reak, Launch (D)ebugger, or (E)xit?
  e
  Redirecting call to abort() to mozalloc_abort
  
  Hit MOZ_CRASH() at /Users/paria/Developer/0ad/libraries/source/spidermonkey/mozjs-91.13.1/memory/mozalloc/mozalloc_abort.cpp:33
  zsh: segmentation fault  pyrogenesis_dbg -replay-visual="/Users/paria\ crash"
  

sandboxed and spindump issue

• pyrogenesis_dbg freezes or is just too slow to reproduce the problem in this mode (see #6702)

[Stan]

Too bad what about Vlad's suggestion ?

[Langbart]

Replying to Stan‘:

Just tested on macos, I cannot reproduce.

You have tried all three ways to trigger the problem and no process has appeared that has accumulated large amounts of memory?

Replying to Vladislav Belov:

Could you update to r27412 and revert the only vulkan folder from it?

so, but ends fatal.

# hash equivalent to [27412]
git checkout bec344d807e83ce3d05ffd5713c5a53dcaea7ad2

# confirm we are on the right changset
git log -1 | grep -Eo "git-svn-id: [^ ]+"
# git-svn-id: https://svn.wildfiregames.com/public/ps/trunk@27412

# revert only the vulkan folder chnage
git checkout HEAD~1 source/renderer/backend/vulkan/

# start building
build 

...

fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
3 errors generated.
make[1]: *** [obj/graphics_Release/DeviceCommandContext2.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [obj/graphics_Release/Buffer2.o] Error 1
10 errors generated.
make[1]: *** [obj/graphics_Release/DescriptorManager.o] Error 1
make: *** [graphics] Error 2

[Stan]

Only 2) For now (I'm mostly AFK during the day). Will try 1) later today. 3) Sounds like something that is because of vscode but maybe I'm wrong.

[Vladislav Belov]

Replying to Langbart:

so, but ends fatal.

It seems the added files weren't removed completely. In the source/renderer/backend/vulkan folder should be only 3 files: Device.h, Device.cpp and DeviceForward.h.

[Langbart]

• in short, [27412] did not cause any of the problems described here, to finally test spindumb you also have to restart the PC, this led me to falsely blame [27412]

• everything described here (spinddumb, sandboxed and the ReportCrash issue), came with [27409]

• The only thing left to do is to try a macOS update and test again