Opened 10 years ago

Closed 9 years ago

#2415 closed task (fixed)

Scripting transition to exact stack rooting and moving GC

Reported by: Yves Owned by: Yves
Priority: Must Have Milestone: Alpha 18
Component: Core engine Keywords: Spidermonkey
Cc: Patch:

Description (last modified by Yves)

Depends on

#2416
#1886
#2428

What is rooting?

Javascript uses garbage collection to detect unused variables and objects that can be deleted. If you use a jsval/JS::Value/CScriptVal or any other variables that point to storage managed by the garbage collector, you need to root it. Rooting tells the garbage collector that this value is still in use and can't be overwritten by something else.

What is exact stack rooting

Currently (with v1.8.5 and v24), SpiderMonkey scans the stack for anything that looks like a pointer to a GC thing. This is called conservative stack scanning. It prevents the garbage collector from deleting an unrooted jsval you use in a function for example.

Future versions will not support that stack scanning anymore and require exact rooting instead. Exact stack rooting means the programmer is responsible for keeping the garbage collector informed about every GC thing he uses on the stack.

What is a moving GC

A moving GC can change the location in memory where a script value is stored. Moving the object a JSObject* points to invalidates the JSObject pointer for example. Instead of just rooting a value once and using as many pointers to it as you need, you now have to inform the garbage collector about every pointer to the value because it has to change that pointer if it moves the value.

Why do we need this?

There's more work to do for the rooting, the code looks a little bit uglier and failing to do it properly causes crashes and security holes. Why do we want this? The reasons and some more technical background is explained very well in this blog post.

Honestly, I'm not quite convinced that the efforts are worth the benefits. Anyway, support for conservative stack scanning will be dropped in one of the next releases and we are basically forced to make this transition.

How do we tackle this?

The API needed for exact stack rooting is already available in SpiderMonkey V24. Also functionality to find issues and features requiring exact stack rooting can be enabled already in V24. After the upgrade (#1886) we can start the transition to exact rooting step by step while keeping track of the remaining number of reported rooting hazards. The linked resources explain some features that can be used to detect and report such rooting hazards. As soon as everything is done, we can enable features that rely on exact stack rooting.

Time frame

Mozilla will still support conservative stack scanning in ESR31 but at the moment it looks like it will be dropped soon after that release.

Resources

Blog post explaining technical background and reasons
SpiderMonkey MDN GC Rooting Guide
SpiderMonkey MDN Exact Stack Rooting
Partially outdated GC documentation

Change History (31)

comment:1 by Yves, 10 years ago

Description: modified (diff)

comment:2 by Yves, 10 years ago

In 14724:

Removes ValueCloner.

Fixes #2416
Refs #2415
Refs #1886

comment:3 by Yves, 10 years ago

Status: newassigned

comment:4 by Yves, 10 years ago

In 14733:

Reverts r14724. Structured clones don't support cyclic references in SpiderMonkey v1.8.5.
SpiderMonkey v24 supports it and I'll add this change to #1886 instead.

Refs #2416
Refs #2415
Refs #1886

comment:5 by Yves, 10 years ago

Description: modified (diff)

comment:6 by Yves, 10 years ago

Description: modified (diff)

comment:7 by Yves, 10 years ago

In 14878:

Third commit for the SpiderMonkey upgrade (remove v1.8.5).

This commit removes the old binaries and header files of SpiderMonkey 1.8.5 plus two patches for openbsd.
I'm closing the SpiderMonkey upgrade ticket because most of the work is done.
Of course I'll continue to work on the remaining known issues (check the ticket for a list).

Fixes #1886
Refs #2415

comment:8 by Yves, 10 years ago

In 15516:

Changes our JSNative functions to use JS::CallReceiver/JS::CallArgs.

This is the new way for working with arguments in JSNative functions. JS_THIS_VALUE, JS_ARGV, JS_SET_RVAL and direct access to vp or argc are deprecated and will probably be removed in future versions of SpiderMonkey.
CallArgs also takes care of proper rooting and you can get the values as Handles or MutableHandles. The interface changes a little bit for ESR 31, but commiting this now still makes it easier and the changes shout be straigtforward (search and replace more or less).

Refs #2462
Refs #2415

comment:9 by Yves, 10 years ago

In 15517:

Changes FromJSVal to take a JS::HandleValue instead of JS::Value.

JS::HandleValue is basically a wrapper around a JS::Value that is safe for exact stack rooting and moving GC.
I've tried to keep this changeset rather small and isolated and therefore create additional JS::Rooted<T> values at some places where the function should eventually directly take a JS::Handle<T>.
The functions "CallFunction" and "CallFunctionVoid" put their arguments inside a JS::AutoValueVector because this will be passed directly to "CallFunction_" with ESR31.

Refs #2462
Refs #2415

comment:10 by Yves, 10 years ago

In 15534:

Changes ToJSVal to take JS::MutableHandleValue instead of JS::Value&.

JS::MutableHandleValue is similar to JS::HandleValue, but the whole JS::Value it points to can be replaced.
This change is needed for support of exact stack rooting and moving GC.
Contains a few other trivial API adjustments and style improvements too.

Refs #2462
Refs #2415

comment:11 by Yves, 10 years ago

In 15541:

Implements CallFunction with JS::MutableHandle<T> return type.

Changes the CallFunction implementation to use macros because otherwise we'd have to write twice as many functions manually.
Adapts GetSavedGameData to use the new function template. Additional callers will be changed in future commits.

Refs #2415
Refs #2462

comment:12 by Yves, 10 years ago

In 15542:

Adds support for passing JS::HandleValue to SetProperty and CallFunctionVoid without using ugly casing.

Also includes one little "demo-usecase", but additional code will be changed to use that in future commits.

Refs #2415
Refs #2462

comment:13 by Yves, 10 years ago

In 15561:

Fixes compartment mismatch introduced in r15549.

Also modifies some related functions to use SpiderMonkey stack rooting types instead of CScriptValRooted.

Refs #2415

comment:14 by Yves, 10 years ago

In 15562:

Removes rooting test.

Rooting works different now and the CScriptVal types are going to be removed. We will need rooting tests again, but they will have to work completely different than this test implementation.

Refs #2415

comment:15 by Yves, 10 years ago

In 15567:

Better support for SpiderMonkey rooted types in the ScriptInterface.

  • Adds additional overloads/specializations which are required when passing JS::Handle<T>/JS::MutableHandle<T> types to different functions.
  • Replaces GetPropertyJS with a GetProperty specialization.
  • Allows us to avoid the implementation of ToJSVal specializations for JS::Value and JS::HandleValue. Such conversions should only happen if there's no way around it and if you are aware of it.
  • Adds test to make sure that all potentially required specializations with custom implementations are instantiated. This should help prevent introducing bugs in temporarily unused code.

Refs #2415

comment:16 by Yves, 10 years ago

In 15568:

Quite a lot of stack rooting related changes.

Changes GetProperty, SetProperty and HasProperty and a few other functions to take handles. The conversions to CScriptVal or CScriptValRooted at some places should be removed in the future. I've done that to avoid an even larger patch.

Refs #2415
Refs #2462

comment:17 by Yves, 10 years ago

In 15591:

Adds support for passing JS::HandleValue and JS::RootedValue& to CallFunction.

Also adds tests for these cases.

Refs #2415

comment:18 by Yves, 10 years ago

In 15592:

More exact stack rooting (CallFunction object).

Changes CallFunction and CallFunctionVoid to use a HandleValue as object parameter. Also changes some JS serialization/deserialization functions to only support the JSAPI rooted types (drop support for CScriptVal and CScriptValRooted there). Some other functions got changed too because they were closely related.

Refs #2415
Refs #2462

comment:19 by Yves, 10 years ago

In 15597:

Exact stack rooting for structured cloning functions.

Refs #2415
Refs #2462

comment:20 by Yves, 10 years ago

In 15601:

Exact rooting for CallConstructor.

Refs #2415
Refs #2462

comment:21 by Yves, 10 years ago

In 15603:

Exact stack rooting for JSON related ScriptInterface functions.

Refs #2415
Refs #2462

comment:22 by Yves, 10 years ago

In 15605:

Exact stack rooting for ScriptInterface::ToString.

I had to change a few other functions to take JS::MutableHandleValue because JS::Stringify takes a JS::MutableHandleValue as input parameter. That seems a bit strange because it should not change that value.
I assume it has historical reasons.

Refs #2415
Refs #2462

comment:23 by Yves, 10 years ago

In 15606:

Exact stack rooting for GetClass and GetPrivate.

Refs #2415

comment:24 by Yves, 10 years ago

In 15607:

Exact stack rooting for WriteStructuredClone and functions that use it.

Refs #2415

comment:25 by Yves, 10 years ago

In 15611:

Fixes crash when loading maps in Atlas.

The ScriptInterface pointer can be NULL there, so using it without checking is not safe.
This fixes the problem by continuing with the exact stack rooting changes, which makes the temporary solution unnecessary.

Fixes #2707
Refs #2415

comment:26 by Yves, 10 years ago

In 15622:

Exact stack rooting for ScriptInterface::FreezeObject.

Refs #2415

comment:27 by Yves, 10 years ago

In 15623:

Exact stack rooting for IGUIObject.

Refs #2415
Refs #2462

comment:28 by Yves, 10 years ago

In 15624:

Exact stack rooting for simulation message type conversions.

Also replaces some deprecated API (jsval typedef and OBJECT_TO_JSVAL) in that part of the code.

Refs #2415

comment:29 by Yves, 9 years ago

In 15944:

Exact stack rooting for CParamNode

Refs #2415
Refs #2462

comment:30 by Yves, 9 years ago

Milestone: BacklogAlpha 18

comment:31 by Yves, 9 years ago

Resolution: fixed
Status: assignedclosed

In 16214:

SpiderMonkey 31 upgrade

This upgrade also introduces exact stack rooting (see to the wiki: JSRootingGuide) and fixes problems with moving GC. This allows us to enable generational garbage collection (GGC).
Measurements a few months ago have shown a performance improvement of a non-visual replay of around 13.5%. This probably varies quite a bit, but it should be somewhere between 5-20%. Memory usage has also been improved. Check the forum thread for details.

Thanks to everyone from the team who helped with this directly or indirectly (review, finding and fixing issues, the required C++11 upgrade, the new autobuilder etc.)! Also thanks to the SpiderMonkey developers who helped on the #jsapi channel or elsewhere!

Fixes #2462, #2415, #2428, #2684, #1374
Refs #2973, #2669

Note: See TracTickets for help on using tickets.