[PATCH] Test UTF-8 handling of scripts and data

[historic_bruno]

Now that we support UTF-8 script and data files (e.g. JSON), it would be nice to test that we don't break it by accident. We already ​test pre-loaded JSON and script strings, but not loading them from files (beware the confusing LoadScript() call takes a filename but only for debugging). It shouldn't be any more complicated than figuring out how to add the VFS init stuff to the tests and then adding a few test files. I think we should test UTF-8 with and without ​BOM, since that seems likely to cause breakage.

[Adrián Chaves]

Since all scripts are supposed to be in UTF-8, I suggest discouraging any use of BOM, and removing it from existing files.

The Unicode Standard permits the BOM in ​UTF-8,^​[2] but does not require or recommend its use.— ​Wikipedia

[historic_bruno]

We've never really encouraged it, but as some (Windows) text editors add a BOM by default, it seemed better to simply handle it and not confuse people who know nothing about UTF-8 by rejecting their files :) As far as I know, most or all text editors that support UTF-8 can handle the BOM, and most of them save without it by default.

[Teiresias]

Proposal for utf8-support unit-tests

[Teiresias]

I've tried to implement unit-tests for all (reasonable) places where file.DecodeUTF8 is used for reading json/js (only existing calls). Test approach: Except for the ComponentManager tests which uses two new files in /binaries/dat/mods/_test.sim/simulation/components, a stub vfs implementation has been written for the utf8-support tests. This provides three advantages:

1. Less additional test files
2. Tests run faster as less test classes need to mount a "real" vfs
3. Tests are easier to analyze (in Windows environment, the BOM is hidden by most editors)

Patch tested with rev. 14713 in

• Vista/MSVC2008, appears to work ok (compiles, links, tests run ".....OK!")
• openSUSE 12.2/GCC 4.7.1, compiles and links ok, not run (./test and ./pyrogenesis segfault immediately at startup; unfortunately, a clean svn check-out shows the same behavior on that machine, so i couldn't actually test the patch).

The patch could be optimized further by extacting the StubVFS implementations from the test header and moving them into a common (separate) header. I have not done so as i found no precedence case for that in the 0ad project (e.g. where to place the StubVFS implementation)

[historic_bruno]

Sorry for the delay in responding. I had a significantly more minimal idea of tests in mind when opening the ticket. At the least, the stub VFS implementations do need to be factored out, but I don't think those are even necessary and should be proposed and discussed in a separate ticket.

As far as performance of the tests goes, we don't care too much. Obviously it shouldn't be so slow that nobody runs it, but the parts that are really slow are disabled by default and there shouldn't be so many test files involved that they impact performance.

[Itms]

I'm looking into making that compatible to the current version. I will also review it at the same time.

[Itms]

I never actually progressed on this. It's not "simple" but it is a very good entrypoint for anyone wanting to work on the engine, so I'm leaving the keyword and I'm available for guidance if someone is up for the task :)

[Cento1]

I just read the ticket and I think it is interesting. However, I'm new in the 0ad project. So, can someone give me advice for the beginning?