Specify charset in compiler commandline to avoid issues with different locales

[Inari]

I got some errors when trying to compile the code (For reference, they were those: ​https://pastebin.com/7yh0UzJf)

Noticing that the errors seem to happen with unicode stuff I did some googling and found ​this github issue. Specifically the last post in it is of interest.

I have some of my locale settings set to Japanese, so I tried what the linked GitHub issue suggested and added the "/source-charset:utf-8" to the "test" project properties (in C/C++ -> Command Line) and recompiled and that seems to have fixed the errors.

It isn't a critical thing to add I guess, but it would be nice to add that to all projects, or at least make a note in the wiki about it?

[Stan]

​https://github.com/premake/premake-core/wiki/characterset

Could use that to make a Premake5.lua patch :)

[Inari]

Hm from what I can see, the thing that changes is the "Character Set" setting in Project Properties (Configuration Properties > General). Which for me is already Unicode either way.

I tested setting it to MBCS just in case you mean that, but that doesn't seem to work. Seems it would be possible to add the /source-charset:utf-8 via ​buildoptions though, but I'm not sure if that's the best course of action?

[elexis]

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2146	syntax error: missing ';' before identifier 's2' (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	147	
Error	C2672	'TestScriptConversions::roundtrip': no matching overloaded function found (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	156	
Error	C2780	'void TestScriptConversions::roundtrip(const T &,const char *)': expects 2 arguments - 1 provided (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	156	
Error	C2017	illegal escape sequence (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	156	
Error	C2017	illegal escape sequence (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	163	
Error	C2001	newline in constant (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	146	
Error	C2001	newline in constant (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	151	
Error	C2059	syntax error: ':' (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	163	
Error	C2146	syntax error: missing ';' before identifier 'w2' (compiling source file ..\..\..\source\scriptinterface\tests\test_ScriptConversions.cpp)	test	D:\Dev\Games\0AD\source\scriptinterface\tests\test_ScriptConversions.h	152

(Upload things here so they don't get lost. Also it helps if it shows up in the search index.)

[Stan]

The best course of action is to edit the premake5.lua file so it can generate the correctly edited project the son are never committed :)

[Inari]

Yeah. However it seems VS 2013 doesn't even support the /source-charset compiler flag. Or at least MSDN only shows "Visual Studio 2015" on the page and I can't seem to find it when I got to the VS 2013 Compiler Options pages.

[Stan]

You can check for the toolset in premake. Once everything is stable we might drop support for the former. The best would be to have support for vs2017 toolset.

[Inari]

Well for now I could add it for VS 2015. Not a complete fix, but at least better?

[Itms]

Hello and sorry for not answering immediately :)

The fix looks good but I'm not completely sure whether we should patch the prebuild script or advertise the fix in the wiki for developers with non-English locales.

Once we drop VS 2013 support, this option will actually be more useful. Is the option available in VS 2017 or does that IDE assume utf-8 source charset? If the fix is both available and needed only on 2015, I think it's easier to have a wiki note.

Do you mind if I push this to the next development cycle, when we will drop 2013?

[historic_bruno]

Replying to Itms:

Once we drop VS 2013 support, this option will actually be more useful. Is the option available in VS 2017 or does that IDE assume utf-8 source charset?

Behavior seems ​unchanged through VS 2019:

By default, Visual Studio detects a byte-order mark to determine if the source file is in an encoded Unicode format, for example, UTF-16 or UTF-8. If no byte-order mark is found, it assumes the source file is encoded using the current user code page, unless you specify a character set name or code page by using the /source-charset option. Visual Studio allows you to save your C++ source code by using any of several character encodings. For more information about source and execution character sets, see Character Sets in the language documentation.

Sad that they can't do any better than that in 2019.

[Itms]

In 23383:

Set source and executable character sets to UTF-8 on Windows, removes a platform discrepancy, fixes #5094.

Patch By: Inari
Tested By: Angen

Differential Revision: ​https://code.wildfiregames.com/D1411