[PATCH] Improve testing tools

[Itms]

We have a lot of useful tools around, for unit tests, code coverage, documentation etc. but they are not always up-to-date or usable with Jenkins.

This ticket is for tracking some needed updates.

[Itms]

Fix the creation of binaries/data/cache by using _testcache everywhere

[Itms]

Do not print anything outside the CxxTest functions, or that breaks the XML output that Jenkins needs

[Itms]

In 19045:

Do not print things outside of the standard TS_FAIL output, or garbage will be added to the XML output when running the tests under Jenkins.

This patch has the extra benefit that the stack trace actually appears on Windows now, instead of going into the void.

Tested by Vladislav, refs #4419

[elexis]

(test_setup.js would have been slightly cleaner if the last 2 lines were merged). I guess the creation of the cache directory is the result of the mount path, since the oosdumps were also written to that directory in non-visual replaymode, but not in the visual mode (or something like that).

[Itms]

Ah that's interesting. The cache patch doesn't seem to work on Windows anyways, so I'll take a closer look at all this.

[Itms]

In 19062:

Fix the cache directory appearing during tests.

Summary: Refs #4419.

Test Plan: When running tests, make sure nothing is created.

Reviewers: leper

Reviewed By: leper

Subscribers: leper, Vulcan

Differential Revision: ​https://code.wildfiregames.com/D1

[Itms]

Do we have a ticket for the visual/non-visual inconsistency? If no it should get a separate one.

About the commit message: team members are invited to comment on its verbosity on the forums.

Not closing the ticket because other things are incoming.

[Itms]

In 19074:

Small updates to build scripts, refs #4419.

Tweak the coverage script.
Update the clean-workspaces script to be able to avoid rebuilding libraries.

Differential Revision: ​https://code.wildfiregames.com/D2

[Itms]

In 19191:

Remove all occurrences of for each in JS scripts.
This will allow us to use some linters that would otherwise crash on this non-standard SpiderMonkey feature. Refs #4419.

Special thanks to elexis for thoroughly checking and testing all the changes!
Reviewed By: leper, elexis

Differential Revision: ​https://code.wildfiregames.com/D40

[Itms]

This is not bound to an Alpha version, and I'm not going to commit updates during the A22 Feature Freeze.

[Itms]

In 20365:

Add our script for building patches with Jenkins to version control.

Refs #4419.
Reviewed By: leper, elexis
Differential Revision: ​https://code.wildfiregames.com/D18

[Itms]

In 23250:

Update the Jenkins scripts to match the current state, which uses Jenkins pipelines, Docker for reproductible build environments, and ZFS for space efficiency and robust incremental patching. refs #4419, refs #4312.

Reviewed By: Angen
Differential Revision: ​https://code.wildfiregames.com/D1910

[Stan]

Backlogging due to lack of progress and the user retiring.