Reliably serialize NaN values
|Reported by:||historic_bruno||Owned by:|
|Priority:||Nice to Have||Milestone:||Backlog|
In #1828, it was discovered the cause of serialization test failure was Spidermonkey using different values internally for NaN. According to ECMAScript-262, there is only a single NaN value which can have different internal representation depending on the implementation. Because this might vary on different systems and even the same system in JIT vs. non-JIT paths, we should probably canonize it when serializing so it doesn't cause an OOS or serialization test failure.
Related problem: the debug serializer doesn't handle NaN or Infinity, so a simple text diff won't reveal these errors. For floating-point values in the engine, we can fix that by completing the canonfloat() function. For script vals, it's a bit more difficult because we use JS_Stringify, which per the JSON spec doesn't allow NaN or Infinity (replacing them with null).