Changes between Version 5 and Version 6 of Implementation_of_Internationalization_and_Localization


Ignore:
Timestamp:
Apr 13, 2014, 10:32:38 PM (10 years ago)
Author:
Adrián Chaves
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Implementation_of_Internationalization_and_Localization

    v5 v6  
     1[[PageOutline(1-100, Table of Contents)]]
     2
    13This topic explains the implementation details of internationalization and localization in the game. For documentation on how to use those features to either internationalize new game content or localize the game, see [wiki:Internationalization] and [wiki:Localization].
    24
    3 = The Localization Singleton =
    4 
    5 The heart of the internationalization and localization implementation is at `source/i18n/L10n.cpp`. The `L10n` class implemented in this file works as a [http://en.wikipedia.org/wiki/Singleton_pattern singleton], and provides many functions that can be used from anywhere in the engine C++ codebase. You can get an instance of the singleton including the `i18n/L10n.h` header file and calling `L10n::instance()`.
    6 
    7 == Initialization ==
     5= Internationalization =
     6
     7== Third-Party Libraries ==
     8
     9The internationalization and localization implementation fo the game relies on two libraries:
     10
     11* '''[http://site.icu-project.org/ ICU]''' is a common internationalization library that provides structures to handle locales, easily obtain the default system locale, handle date and number representations on each locale, or provide the localized names of locales to show them in a language combo box.
     12
     13* '''tinygettext''' is a small library that we forked from the the [https://github.com/supertuxkart/stk-code/tree/master/src/tinygettext SuperTuxKart repository]. This library provides the structures and methods that we use to load PO files in memory.
     14
     15== The Localization Singleton ==
     16
     17The heart of the internationalization and localization implementation is at `source/i18n/L10n.cpp`.
     18
     19The `L10n` class implemented in this file works as a [http://en.wikipedia.org/wiki/Singleton_pattern singleton], and provides many functions that can be used from anywhere in the engine C++ codebase. You can get an instance of the singleton including the `i18n/L10n.h` header file and calling `L10n::instance()`.
     20
     21This class holds the current locale and the translations for the current locale at any given time, and its methods are used all over the C++ implementation (and published to JavaScript as well).
     22
     23=== Initialization ===
    824
    925When you call the singleton for the first time, its initialization:
    … …  
    2137        You can change the locale later using `L10n::instance().setLocale()`. You can pass this method either a locale code as a string, such as "en_GB", or an instance of [http://www.icu-project.org/apiref/icu4c/classicu_1_1Locale.html icu::Locale]. The method reloads the dictionary if required.
    2238
    23 == Localization Functions ==
     39=== Localization Functions ===
    2440
    2541The `L10n` singleton provides many localization functions that you can use from C++, some of which are also published to be used in JavaScript.
    … …  
    6783=== Text ===
    6884
    69 TODO:
    70 * source/gui/CGUI.cpp
     85The `source/gui/CGUI.cpp` file implements the parsing of the GUI XML localization elements:
     86* `attribute` (along with `keep` and `translate`)
     87* `translatableAttribute`.
     88
     89See [wiki:Internationalization#InternationalizingGUIFiles Internationalizing GUI Files] to see how to use them.
    7190
    7291=== Images ===
    7392
    74 TODO:
    75 * source/gui/GUIRenderer.cpp
     93The `source/gui/GUIRenderer.cpp` file, responsible for rendering images on the GUI, is configured to use `L10n.instance().localizePath()`, which loads localized versions of images if available.
     94
     95See [wiki:Localization#LocalizingImages Localizing Images] for more information.
    7696
    7797== Functions Published for JavaScript ==
    7898
    79 TODO:
    80 * source/gui/scripting/ScriptFunctions.cpp
    81 
    82 == Third-Party Libraries ==
    83 
    84 TODO:
    85 * tinygettext (third_party/tinygettext)
    86 * ICU
    87 
    88 
    89 == Message Extraction System ==
    90 
    91 TODO:
    92 * source/tools/i18n/updateTemplates.py
    93 * <mod>/l10n/messages.json
    94 * source/tools/i18n/potter/
    95 
    96 == Long Strings Locale ==
    97 
    98 TODO:
    99 * source/tools/i18n/generateLongStringTranslations.py
    100 
    101 == Transifex Integration ==
    102 
    103 TODO:
    104 * .tx/config
    105 * source/tools/i18n/pullTranslations.py
    106 * source/tools/i18n/tx
    107 * source/tools/i18n/txclib/
    108 
    109 = JavaScript =
     99The `source/gui/scripting/ScriptFunctions.cpp` file publishes some of the functions that the `L10n` singleton offers, as well as some custom internationalization functions based on the `L10n` functionality.
     100
     101The following functions from `L10n` are published:
     102{{{
     103scriptInterface.RegisterFunction<std::string, &getCurrentLocale>("getCurrentLocale");
     104scriptInterface.RegisterFunction<void, std::string, &SetLocale>("SetLocale");
     105
     106scriptInterface.RegisterFunction<std::vector<std::string>, &GetSupportedLocaleCodes>("GetSupportedLocaleCodes");
     107scriptInterface.RegisterFunction<std::vector<std::wstring>, &GetSupportedLocaleDisplayNames>("GetSupportedLocaleDisplayNames");
     108scriptInterface.RegisterFunction<int, &GetCurrentLocaleIndex>("GetCurrentLocaleIndex");
     109
     110scriptInterface.RegisterFunction<std::wstring, std::wstring, &translate>("translate");
     111scriptInterface.RegisterFunction<std::wstring, std::string, std::wstring, &translateWithContext>("translateWithContext");
     112scriptInterface.RegisterFunction<std::wstring, std::wstring, std::wstring, int, &translatePlural>("translatePlural");
     113scriptInterface.RegisterFunction<std::wstring, std::string, std::wstring, std::wstring, int, &translatePluralWithContext>("translatePluralWithContext");
     114scriptInterface.RegisterFunction<std::wstring, std::wstring, &translateLines>("translateLines");
     115
     116scriptInterface.RegisterFunction<std::wstring, int, std::wstring, &formatMillisecondsIntoDateString>("formatMillisecondsIntoDateString");
     117
     118scriptInterface.RegisterFunction<std::wstring, double, &formatDecimalNumberIntoString>("formatDecimalNumberIntoString");
     119}}}
     120
     121The following custom functions are also published:
     122{{{
     123// Returns the build time of the game formatted for the current locale.
     124scriptInterface.RegisterFunction<std::wstring, int, &GetBuildTimestamp>("GetBuildTimestamp");
     125
     126// Translates an array of strings, avoiding what would otherwise be many calls
     127// to the C++ engine from JavaScript.
     128scriptInterface.RegisterFunction<std::vector<std::wstring>, std::vector<std::wstring>, &translateArray>("translateArray");
     129
     130// Simply returns the string that it receives. This function is required to mark
     131// some strings from translation in places where they should not be translated
     132// yet, so that translate() cannot be used. The point of this function is that
     133// strings passed to it are catched by the message extraction system (see
     134// below). For more information, see:
     135// http://www.gnu.org/software/gettext/manual/html_node/Special-cases.html
     136scriptInterface.RegisterFunction<std::wstring, std::wstring, &markToTranslate>("markToTranslate");
     137}}}
     138
     139== JavaScript-Side Implementation ==
    110140
    111141The following sections describe some implementation details that are specific to the JavaScript side.
    112142
    113 == JavaScript Translation Cache System ==
    114 
    115 TODO:
    116 * l10n.js (only cache stuff)
    117 
    118 == Object Translation Helper Function ==
    119 
    120 TODO:
    121 * l10n.js (translateObjectKeys)
    122 
    123 == String Formatting Function ==
    124 
    125 TODO:
    126 * sprintf.js
    127 
    128 = Public Mod =
    129 
    130 == Language Selection Menu ==
    131 
    132 TODO:
    133 * gui/page_locale.xml
    134 * gui/locale/*
    135 
    136 
    137 
    138 
    139 
    140 
    141 
    142 
    143 
    144 TODO:
    145 = File Tree =
    146 == binaries/data/mods/public/ ==
    147 === globalscripts/ ===
    148 ==== l10n.js ====
    149 Implements the following global internationalization functions:
     143=== JavaScript Translation Cache System ===
     144
     145Although you can access the C++ functions listed above from JavaScript simply prefixing them with `Engine`, as in `Engine.translate()`, the `binaries/data/mods/public/globalscripts/l10n.js` file defines the following equivalent global internationalization functions:
    150146
    151147{{{
    … …  
    154150translateWithContext(context, message);
    155151translatePluralWithContext(context, singularMessage, pluralMessage, number);
     152}}}
     153
     154These global functions are basically wrappers for the engine internationalization functions, however they are not just that. These global functions use JavaScript-side caching to reduce the number of calls to the engine functions, because calls to engine functions require string conversions that are far from cheap.
     155
     156In JavaScript, you should use these functions intead of the `Engine.` whenever possible.
     157
     158=== Object Translation Helper Function ===
     159
     160The `binaries/data/mods/public/globalscripts/l10n.js` file defines a function, `translateObjectKeys`, which is a helper function that can translate specific properties of a !JavaScript object:
     161
     162{{{
    156163translateObjectKeys(object, keys);
    157164}}}
    158 The first four functions are simply wrappers for the engine internationalization functions (such as `Engine.translate`). These global functions use caching to reduce the number of calls to the engine functions, because calls to engine functions require string conversions that are far from cheap.
    159 
    160 `translateObjectKeys` is a helper function that can translate specific properties (`keys` array) of a !JavaScript object.
    161 
    162 ==== sprintf.js ====
    163 Implements the `sprintf()` function, used for [wiki:Internationalization#UsingStringFormattingInsteadofConcatenatingStringsinJavaScript string formatting].
    164 
    165 === gui/ ===
    166 ==== page_locale.xml ====
    167 Represents the dialog box that you open when you select '''Options → Language '''in the main menu. The actual content of the dialog box is defined in `gui/locale/`.
     165
     166The `keys` parameter is an array of strings with the names of the object properties to translate.
     167
     168=== String Formatting Function ===
     169
     170The `binaries/data/mods/public/globalscripts/sprintf.js` file defines a function, `sprintf`, that can be used for string formatting. To learn how to use it, see [wiki:Internationalization#UsingStringFormattingInsteadofConcatenatingStringsinJavaScript Using String Formatting Instead of Concatenating Strings in JavaScript].
     171
     172= Message Extraction =
     173
     174Message extractions is the process of parsing the source files searching for strings that need to be translated, and generating a translation template file (POT) from them that translators can use.
     175
     176The script responsible for generating POT files is `source/tools/i18n/updateTemplates.py`. This scripts goes through the `l10n` folders of the sources (`binaries/data/l10n` and `l10n` in mod folders) and it reads the `messages.json` file there, which defines where to extract the strings from, and how to extract them.
     177
     178The format of the messages.json file is a custom format. Check existing `messages.json` files to learn the syntax.
     179
     180The resulting POT files are generated on the same `l10n` folder that contains the `messages.json` file that define their configuration.
     181
     182`updateTemplates.py` relies on a Python library to create POT files, `potter`, which is the name of our fork of [http://babel.edgewall.org/log/trunk/babel/messages babel-messages]. It is located at `source/tools/i18n/potter`.
     183
     184= Localization =
     185
     186== Transifex Integration ==
     187
     188Currently, the translation of the game happens in Transifex, at https://www.transifex.com/projects/p/0ad/
     189
     190The `l10n` folders of the game sources contain a hidden folder, `.tx`, which contains a `config` file. This config file determines where in Transifex are the PO files that must be downloaded to that specific `l10n` folder.
     191
     192To download the translation files from Transifex, you can call `source/tools/i18n/pullTranslations.py`, which uses those `.tx/config` files and the Transifex client (`source/tools/i18n/tx`) and library (`source/tools/i18n/txclib/`) to automatically download the latest PO files into their respective folders.
     193
     194To use this script, you need a Transifex account.
     195
     196== Long Strings Locale ==
     197
     198The `source/tools/i18n/generateLongStringTranslations.py` is a special script that generates a PO file for an artificial language with code "long". This PO file consists of the longest strings (as in number of characters) of every available language for each message.
     199
     200The `generateLongStringTranslations.py` script parses the PO files in the `l10n` folders of the sources, and generates the new PO file in each of those `l10n` folders as well. Remember to download the real PO files before you call the script, so that the script can read the translations from them in order to find the longest strings.
     201
     202= Language Selection Menu =
     203
     204The dialog box that you open when you select '''Options → Language '''in the main menu is defined in the following files:
     205* `gui/page_locale.xml`
     206* `gui/locale/locale.js`
     207* `gui/locale/locale.xml`