Version 8 (modified by 10 years ago) ( diff ) | ,
---|
Table of Contents
This topic explains the implementation details of internationalization and localization in the game. For documentation on how to use those features to either internationalize new game content or localize the game, see Internationalization and Localization.
Internationalization
Third-Party Libraries
The internationalization and localization implementation fo the game relies on two libraries:
- ICU is a common internationalization library that provides structures to handle locales, easily obtain the default system locale, handle date and number representations on each locale, or provide the localized names of locales to show them in a language combo box.
- tinygettext is a small library that we forked. This library provides the structures and methods that we use to load PO files in memory.
The Localization Singleton
The heart of the internationalization and localization implementation is at source/i18n/L10n.cpp
.
The L10n
class implemented in this file works as a singleton, and provides many functions that can be used from anywhere in the engine C++ codebase. You can get an instance of the singleton including the i18n/L10n.h
header file and calling L10n::Instance()
.
This class holds the current locale and the translations for the current locale at any given time, and its methods are used all over the C++ implementation (and published to JavaScript as well).
Initialization
When you call the singleton for the first time, its initialization:
- Creates a list of available locales based on the filenames of the PO files in the
l10n
virtual folder.
The
l10n
virtual folder holds the combination of the files in thebinaries/data/l10n
folder, which contains the engine PO files, and thel10n
folder of each enabled mod. Seesource/ps/GameSetup/GameSetup.cpp
.
You can obtain the list of available locales calculated here at any later time calling
L10n::Instance().GetSupportedLocaleCodes()
. This method returns an array of strings with the locale codes found here.L10n::Instance().GetSupportedLocaleDisplayNames()
returns a similar vector, this one with display names instead of codes, where the name of each locale is in the language of the locale itself. The special locale "Long Strings", of code "long", is only returned if yours is a development copy (if theconfig/dev.cfg
file exists in the virtual filesystem).
- Determines the current locale, first looking at the configuration value
locale
and, if undefined, looking at the system locale using icu::Locale::getDefault().
Once the locale is defined, it loads the dictionary of the target locale if it is available. The dictionary is a structure from the tinygettext library that holds the list of English strings linked to translated strings (including context and plural considerations).
You can change the locale later using
L10n::Instance().SetLocale()
. You can pass this method either a locale code as a string, such as "en_GB", or an instance of icu::Locale. The method reloads the dictionary if required.
Localization Functions
The L10n
singleton provides many localization functions that you can use from C++, some of which are also published to be used in JavaScript.
Translation functions (Internationalization explains how to use them):
std::string Translate(const std::string& sourceString); std::string TranslateWithContext(const std::string& context, const std::string& sourceString); std::string TranslatePlural(const std::string& singularSourceString, const std::string& pluralSourceString, int number); std::string TranslatePluralWithContext(const std::string& context, const std::string& singularSourceString, const std::string& pluralSourceString, int number); std::string TranslateLines(const std::string& sourceString);
Date and time functions:
// parseDateTime() and localizeDateTime() are used to convert dates from strings // into icu::UDate and the other way around. These are used in // “gui/scripting/ScriptFunctions.cpp”. UDate ParseDateTime(const std::string& dateTimeString, const std::string& dateTimeFormat, const Locale& locale); std::string LocalizeDateTime(const UDate& dateTime, DateTimeType type, DateFormat::EStyle style); // Converts a string from UNIX timestamp (in milliseconds, not seconds) into a // string with the specified format. See the ‘Internationalization’ page in the // documentation for more information. std::string FormatMillisecondsIntoDateString(int milliseconds, const std::string& formatString);
Number functions:
// Returns the specified floating-point number as a string using the current // locale. std::string FormatDecimalNumberIntoString(double number);
Path functions:
// Returns the localized version of the specified path if there is one. // Otherwise, it returns sourcePath. // This is used for image localization (see below). VfsPath LocalizePath(VfsPath sourcePath);
GUI
Text
The source/gui/CGUI.cpp
file implements the parsing of the GUI XML localization elements:
attribute
(along withkeep
andtranslate
)translatableAttribute
.
See Internationalizing GUI Files to see how to use them.
Images
The source/gui/GUIRenderer.cpp
file, responsible for rendering images on the GUI, is configured to use L10n.Instance().LocalizePath()
, which loads localized versions of images if available.
See Localizing Images for more information.
Functions Published for JavaScript
TODO: Link to svn.wildfiregames.com/docs when #67 gets committed.
The source/i18n/scripting/JSI_L10n.cpp
file publishes some of the functions that the L10n
singleton offers, as well as some custom internationalization functions based on the L10n
functionality.
The following functions from L10n
are published:
scriptInterface.RegisterFunction<std::string, &GetCurrentLocale>("GetCurrentLocale"); scriptInterface.RegisterFunction<void, std::string, &SetLocale>("SetLocale"); scriptInterface.RegisterFunction<std::vector<std::string>, &GetSupportedLocaleCodes>("GetSupportedLocaleCodes"); scriptInterface.RegisterFunction<std::vector<std::wstring>, &GetSupportedLocaleDisplayNames>("GetSupportedLocaleDisplayNames"); scriptInterface.RegisterFunction<int, &GetCurrentLocaleIndex>("GetCurrentLocaleIndex"); scriptInterface.RegisterFunction<std::wstring, std::wstring, &Translate>("Translate"); scriptInterface.RegisterFunction<std::wstring, std::string, std::wstring, &TranslateWithContext>("TranslateWithContext"); scriptInterface.RegisterFunction<std::wstring, std::wstring, std::wstring, int, &TranslatePlural>("TranslatePlural"); scriptInterface.RegisterFunction<std::wstring, std::string, std::wstring, std::wstring, int, &TranslatePluralWithContext>("TranslatePluralWithContext"); scriptInterface.RegisterFunction<std::wstring, std::wstring, &TranslateLines>("TranslateLines"); scriptInterface.RegisterFunction<std::wstring, int, std::wstring, &FormatMillisecondsIntoDateString>("FormatMillisecondsIntoDateString"); scriptInterface.RegisterFunction<std::wstring, double, &FormatDecimalNumberIntoString>("FormatDecimalNumberIntoString");
The following custom functions are also published:
// Returns the build time of the game formatted for the current locale. scriptInterface.RegisterFunction<std::wstring, int, &GetBuildTimestamp>("GetBuildTimestamp"); // in source/gui/scripting/ScriptFunctions.cpp // Translates an array of strings, avoiding what would otherwise be many calls // to the C++ engine from JavaScript. scriptInterface.RegisterFunction<std::vector<std::wstring>, std::vector<std::wstring>, &TranslateArray>("TranslateArray"); // Simply returns the string that it receives. This function is required to mark // some strings from translation in places where they should not be translated // yet, so that translate() cannot be used. The point of this function is that // strings passed to it are catched by the message extraction system (see // below). For more information, see: // http://www.gnu.org/software/gettext/manual/html_node/Special-cases.html scriptInterface.RegisterFunction<std::wstring, std::wstring, &MarkToTranslate>("MarkToTranslate");
JavaScript-Side Implementation
The following sections describe some implementation details that are specific to the JavaScript side.
JavaScript Translation Cache System
Although you can access the C++ functions listed above from JavaScript simply prefixing them with Engine
, as in Engine.Translate()
, the binaries/data/mods/public/globalscripts/l10n.js
file defines the following equivalent global internationalization functions:
translate(message); translatePlural(singularMessage, pluralMessage, number); translateWithContext(context, message); translatePluralWithContext(context, singularMessage, pluralMessage, number);
These global functions are basically wrappers for the engine internationalization functions, however they are not just that. These global functions use JavaScript-side caching to reduce the number of calls to the engine functions, because calls to engine functions require string conversions that are far from cheap.
In JavaScript, you should use these functions intead of the Engine.
whenever possible.
Object Translation Helper Function
The binaries/data/mods/public/globalscripts/l10n.js
file defines a function, translateObjectKeys
, which is a helper function that can translate specific properties of a JavaScript object:
translateObjectKeys(object, keys);
The keys
parameter is an array of strings with the names of the object properties to translate.
String Formatting Function
The binaries/data/mods/public/globalscripts/sprintf.js
file defines a function, sprintf
, that can be used for string formatting. To learn how to use it, see Using String Formatting Instead of Concatenating Strings in JavaScript.
Message Extraction
Message extractions is the process of parsing the source files searching for strings that need to be translated, and generating a translation template file (POT) from them that translators can use.
The script responsible for generating POT files is source/tools/i18n/updateTemplates.py
. This scripts goes through the l10n
folders of the sources (binaries/data/l10n
and l10n
in mod folders) and it reads the messages.json
file there, which defines where to extract the strings from, and how to extract them.
The format of the messages.json file is a custom format. Check existing messages.json
files to learn the syntax.
The resulting POT files are generated on the same l10n
folder that contains the messages.json
file that define their configuration.
updateTemplates.py
relies on a Python library to create POT files, potter
, which is the name of our fork of babel-messages. It is located at source/tools/i18n/potter
.
Localization
Transifex Integration
Currently, the translation of the game happens in Transifex, at https://www.transifex.com/projects/p/0ad/
The l10n
folders of the game sources contain a hidden folder, .tx
, which contains a config
file. This config file determines where in Transifex are the PO files that must be downloaded to that specific l10n
folder.
To download the translation files from Transifex, you can call source/tools/i18n/pullTranslations.py
, which uses those .tx/config
files and the Transifex client (source/tools/i18n/tx
) and library (source/tools/i18n/txclib/
) to automatically download the latest PO files into their respective folders.
To use this script, you need a Transifex account.
Long Strings Locale
The source/tools/i18n/generateLongStringTranslations.py
is a special script that generates a PO file for an artificial language with code "long". This PO file consists of the longest strings (as in number of characters) of every available language for each message.
The generateLongStringTranslations.py
script parses the PO files in the l10n
folders of the sources, and generates the new PO file in each of those l10n
folders as well. Remember to download the real PO files before you call the script, so that the script can read the translations from them in order to find the longest strings.
Language Selection Menu
The dialog box that you open when you select Options → Language in the main menu is defined in the following files:
gui/page_locale.xml
gui/locale/locale.js
gui/locale/locale.xml