In general, I completed most of the tasks from my proposal, but not all of them. Some (many) details changed during implementation, became harder than I thought, while others were simpler. I won’t get into details (there are plenty of week summaries for that purpose ;-) ), I’d only like to mention that I’m looking forward to participating in the FLOW3 project in my free time.

I would like to especially thank my mentor Karsten Dambekalns for all the help he provided me during the coding period, and much earlier too. Also I’d like to thank the members of TYPO3 community for support during GSoC and for making great software, which FLOW3 is, available as open source! You rock! :-)

After that I returned to the Validators which I were writing last week – NumberValidator and DateTimeValidator. They both accepts additional options: locale, strictMode (to work in strict or lenient matching), formatType, formatLength. These options are closely related to the arguments accepted by Parsers (NumberParser and DatetimeParser), as Validators use them to do their job.

Currently I’m still working on the Internationalization and Localization chapter for FLOW3 documentation. Its almost finished – I just need to format the text nicely ;-). XML Mind is very convenient tool for writing DocBooks.

Well, this is the last week summary as tomorrow is the day when GSoC coding period ends. I’m very glad for being able to participate in such great open source project like FLOW3. It was great time for me. But let’s not hijack a week summary post! I will write separate post about the GSoC & FLOW3 seen from my point of view very soon! :-)

I planned to create the Validators this week and I even have some code in my working copy, but I need more time. As the Parsers created earlier provide pretty useful functionality, I would like to support all of it in Validators using options property of AbstractValidator class. Initially I wanted to do quite a few simple Validators (like DecimalNumberValidator, PercentNumberValidator, and similar for date and time), but now I think that two classes (NumberValidator and DatetimeValidator) with options for configuring how they work will be a better solution.

I also need to remove few more issues (like magic strings as parameters, and lack of validation for these – why PHP still has no Enum type?).

There is a pencils down deadline after the next week (August 16th). I won’t be allowed to write any more code after. Fortunately almost everything what I planned to do and what I described in my proposal is done. I will write a nice documentation chapter and all should be OK ;-).

This week was full of frequent, small commits (two more I didn’t mention: r4916, r4922). But the most important task was the lenient parsing (NumberParser: r4925, DatetimeParser: 4928).

This is not a full featured lenient parsing as described in some documents on the Internet, it’s rather a “light” variant. But I’m pretty happy with results and I think that both strict and lenient mode complements each other nicely. Strict parsing is meant to be used when pattern needs to be matched exactly (even whitespace counts :-) ), while lenient parsing ignores everything that can be ignored in order to extract important elements from the string.

I rewrote the tests for parsers in order to test both parsing modes thoroughly. You can see in these tests that when strict mode fails and returns FALSE, lenient mode does just fine :-).

I gathered significant list of various code-cleaning tasks in my FLOW3 TODO file ;-), so I plan to review my code extensively next week. I will also implement some Validators for FLOW3 Validation subsystem (they will use NumberParser and DatetimeParser to do the job).

I managed to implement the strict modes for both NumberParser and DatetimeParser (r4903). They use parsed formats accordingly from NumbersReader and DatesReader, and try to match particular elements of the format. If something goes wrong, FALSE is returned.

Parses do not follow full specification of CLDR for numbers and datetime, as Readers don’t do it either. But the most important features are supported.

Next week I will improve some parts of existing code (especially DatetimeParser as I have some ideas, and I think I will change Readers too). Also, the tests for parsers need to be extended. Then I will work on the lenient parsing in order to provide at least a little bit of loose matching.

According to the timeline less than 4 weeks of GSoC left. When I finish the parsers, I will work on the number and datetime Validators for FLOW3. After that, if all goes right I will have almost 2 weeks for documentation and for improvements in various parts of the code. For example, I need to think about the Translate Fluid viehelper (Bastian helped me with this a lot, thank you!).

I implemented the Translator class this week. The Translator class is meant to be used for messages’ translation (of course ;-)), and it has pretty interesting functionality. This class is a place where almost all functionality I wrote up to today meets.

First of all, the Translator has two modes of operation: translating by original label, and by ID.

• by original label – searching for translated label is done for original (untranslated) label
• by ID – searching is done for particular identification key

These modes seem to be very similar (as original label and ID are just used as a key), but there is an important difference between them. In general, placing untranslated labels in view templates is more readable (for example, “You should log in before doing this action” is better than “user.notLoggedIn“). But using IDs is probably more maintainable. So the Translator has API for both modes, but whether they are both supported depends on translation provider class used by Translator.

The translation provider is a class which implements TranslationProviderInterface (described in previous summary). Concrete provider doesn’t have to support two modes. The default provider, XliffTranslationProvider, supports both modes as either source labels and IDs are stored in XLIFF files.

Translator requires a $source parameter to be set for every use of translate*() methods. Source is a string identifier understandable by translation provider which uniquely denotes a source catalog of translation labels to use. For XliffTranslationProvider, it’s just a name of XLIFF file.

I think that source parameter should be optional, set by default to “main” for example, as defining it for every method call is pretty redundant. This needs to be discussed by core devs though.

Translator supports plural forms. In order to use them, you need to set the $quantity parameter. It is a number (integer or float, unsigned) used for evaluating a plural-form expression defined for particular language. Of course, the source label has to be translated into all plural forms used in the target language.

It is also worth to note, that when plural forms are used together with the “by original label” translation mode, you need to use singular form for source message. It will be translated to correct plural form if feasible. Please note that it is a subject to change as for example avoiding translation (when source and target language are the same) is not done yet.

The Translator class also uses the FormatResolver for substituting placeholders with formatted values. So you can define a placeholder (like {0,number}) and provide array of values, and they will be formatted and inserted into translated label.

TranslateViewHelper

I also committed preliminary version of the TranslateViewHelper, a view helper for Fluid templating system. It is kind of a wrapper for Translator class, and is also subject to change. Some usage examples:

<f:translate mode="originalLabel" source="main">Untranslated label</f:translate>
<f:translate mode="originalLabel" source="someLabelsCatalog" locale="de_DE">Untranslated label</f:translate>
<f:translate mode="originalLabel" source="main" values="{0: 'foo', 1: '99.9'}">Untranslated {0} and {1,number}</f:translate>
<f:translate mode="id" source="main">user.unregistered</f:translate>

Other

Robert decided to rename the Locale subpackage I’m working on to I18n, as it is better name for the functionality it will provide. So I did it, breaking things in other subpackages by the way :-). Fortunately Karsten was near and fixed all of them.

I also fixed few bugs Bastian noticed, providing me good advices. Thank you!

By the end of this week I was thinking about the next task, a locale-aware input parser for date/time and for numbers. This is my goal for next two weeks, and I hope I will be able to provide a nice solution. Parsing user input can be very hard task. I read about Lenient Parsing. There will probably be few modes of operation, eg. “strict” and “loose”. We will see ;-).

As Gettext natively supports plural forms, and XLIFF does not (it can be done in many ways), I decided to take it into account during XLIFF parser implementation. There is a specification defining how the conversion from .po to .xlf should be done (with plurals), and it’s what the parser conforms. I think it’s the best way for storing plural forms in XLIFF files, as it is possible to convert .xlf to .po, edit it using .po editor (like very good Poedit), and then convert it back to .xlf without any data loss. There is the Translate Toolkit project which has good converters.

However, the way how Gettext defines plural forms is different to the method used in CLDR. In .po files, you just write a C language expression in header which evaluates to integer number, and this number is an index of plural form. In CLDR, there are separate expressions for every form.

Example from Gettext’s .po file (for Polish language):

"Plural-Forms: nplurals=3; plural=(n==1 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2);\n"

And the message is defined like this:

msgid "Source singular"
msgid_plural "Source plural"
msgstr[0] "Translated singular"
msgstr[1] "Translated plural 1"
msgstr[2] "Translated plural 2"

When you convert this to .xlf file, it looks as follow (unimportant tags were stripped):

<?xml version='1.0' encoding='utf-8'?>
<xliff xmlns="urn:oasis:names:tc:xliff:document:1.1" version="1.1">
  <file original="foo.po" source-language="en-US" datatype="po">
    <body>
      <group restype="x-gettext-plurals" id="1" xml:space="preserve">
        <trans-unit id="1[0]">
          <source>Source singular</source>
          <target>Translated singular</target>
        </trans-unit>
        <trans-unit id="2[1]">
          <source>Source plural</source>
          <target>Translated plural 1</target>
        </trans-unit>
        <trans-unit id="3[2]">
          <source>Source plural</source>
          <target>Translated plural 2</target>
        </trans-unit>
      </group>
    </body>
  </file>
</xliff>

In CLDR, there are no indices, but keys: “one”, “zero”, “two”, “few”, “many”, “other”. Particular language uses only subset of these keys. I simplified this and the keys are mapped to indices for particular language. For example, if language has “one”, “few”, and “many” forms (like Polish), they are mapped to 0, 1, and 2, so correct form is chosen from XLIFF file, and automatic .po – .xlf conversion is possible.

The Translator class (which is not done yet) will use implementations of TranslationProviderInterface in order to translate strings. I implemented only one concrete translation provider (the XliffTranslationProvider), but it will be easy to add other providers, to support different file formats (or even other storage methods).

When implementing the XLIFF parser, I noticed that some class hierarchy could be improved. Actually, the CldrModel written few weeks earlier was an abstract for XML file, same as the XliffModel which I created this week. So I created an abstract class, AbstractXmlModel, which is common base for former two classes. Also, they all use parsers in order to convert CLDR or XLIFF file to an internal representation. So there is now an AbstractXmlParser, from which CldrParser and XliffParser extends. I think it reduced the redundant code quite a lot.

Then I came up with “brilliant” idea to get it even further and make HierarchicalCldrModel extend CldrModel, and change the readers (DatesReader, PluralsReader, NumbersReader) so they are models extending HierarchicalCldrModel. When I was almost done I realized that it complicated things and the hierarchy was vague, so I reverted it all back ;-).

This weeks’ changes were commited in r4827.

Next week I will implement Translator class, and Fluid Viewhelper for message translation feature. Then the next important feature to do will be an input parsing, and validators. So far, I’m good with my proposal’s timeline ;-).

During the end-of-semester period I managed to do simple things. I refactored some classes in order to conform FLOW3 Coding Guidelines, updated CLDR version in FLOW3 package, did more minor code refactoring. All of the tasks were Karsten’s notices / suggestions.

With the beginning of July my holidays started, so I focused on GSoC project. The most important part I implemented was message formatting (r4694). These classes will be responsible for formatting messages (labels, strings), or to be more precise, replacing the “placeholders” in these messages with formatted values (numbers, date/time, etc). Placeholders have very simple syntax and look like this:

• {0}, {1} etc – Places string-casted version of a value (number in brackets defines the value’s index from values array provided)
• {0,number,decimal} or {0,number} – Format as decimal number using pattern from CLDR defined for current locale
• {0,datetime} – Format as date and time
• {0,datetime,time,full} – Format only time part, choose the “full” version of pattern from CLDR

Second element in the placeholder defines name of formatter to use. Remaining elements are passed directly to the formatter (for example, NumberFormatter uses only one additional element, defining if format value as decimal, or percentage, but DatetimeFormatter accepts two elements – style (datetime, date or time) and length (full, long, medium, short).

DatetimeFormatter is still subject to change (as you can see, there is some duplication in placeholders, when you write {0,datetime,time,full}, it would be nicer to write {0,time,full}. But this can be changed very easily as formatters are simple classes implementing FormatterInterface, and are automatically loaded if needed.

Formatter classes are the place where the data from CLDR is used. Numbers and date / time are formatted with rules from CLDR.

I managed to do few more commits this week (improved exception hierarchy in Locale, defined public API as Karsten suggested, but it’s not finished yet of course), but I didn’t manage to commit changes I have in my working copy right now. It’s still a lot to be done but I think it will be pretty important commit which will clear up any concerns about the classes already done and how the Locale subpackage is meant to be used.

What I’m doing right now is the Translator class which will translate texts (also using the information about plural forms). But this class needs some kind of messages repository. I’m implementing XLIFF parser so messages will be stored in XLIFF files, but adding support for other file formats in the future will be pretty easy. The new code also imposed changes on older parts (mainly on CLDRModel etc) so I think the class hierarchy will be more robust now.

DatesReader parses most data from dates tag, that is: months, days, quarters, dayPeriods, eras, dateFormats, timeFormats and dateTimeFormats. The most important functionality of DatesReader is of course, date and time formatting. Syntax for date / time format defined in CLDR is pretty extensive (see the specification). Most of it is supported by DatesReader class. As in NumbersReader, the solution for dateTime formatting in Yii Framework was very useful, although I extended it, so the specification is supported even more precisely. These are examples of supported formats (one format per line):

yyyy.MM.dd G
HH:mm:ss zzz
EEE, MMM d, 'yy
hh o'clock a, zzzz

Formats are parsed and stored in cache. DatesReader can format date, time or dateTime (both date and time in one string) – it uses formats defined in CLDR for particular locale. It can also format dateTime using custom format pattern. This class uses DateTime PHP object as a representation of date and time.

DatesReader was commited in Revision 4504 (update: actually in r4512 as I forgot to add new files :-)).

I also thoroughly changed CLDRModel class (and HierarchicalCLDRModel). First of all, parsing of CLDR data is changed – internal array generated from CLDR is now much simpler, which I found much more robust and useful. It was easier to support alias tag which is used in CLDR to point places where actual data is placed (in order to not duplicate data in different parts of XML file). I also changed API for CLDRModel, and I think it’s now more useful.

DatesReader is the last of CLDR classes I planned to implement. CLDR is very extensive and a lot more data could be used, but I’m afraid I won’t have time for the rest of my proposal. So, now I will proceed to the next task, which is support for XLIFF files. XLIFF is a XML format for storing translation messages (labels). If I will have time at the end of coding period, I will return to CLDR.

I started with PluralsReader, a class which will be used in messages (labels) translation process. This class basically returns a form-name (a string, one of: zero, one, two, few, many, other) for a pair of locale and number. In some languages form of noun depends on a number expressed in a sentence – for example, unit such as time or currency. English has only two rules (forms “one” and “other”), for example:

form "one": 1 day
form "other": 0 days, 2 days, 5 days, ...

A rule used here is simple, and can be written like this:

<pluralRule count="one">n is 1</pluralRule>

(it is actually how rules are defined in CLDR)

For other languages, rules can be very complicated:

<pluralRules locales="hr ru sr uk be bs sh">
    <pluralRule count="one">n mod 10 is 1 and n mod 100 is not 11</pluralRule>
    <pluralRule count="few">n mod 10 in 2..4 and n mod 100 not in 12..14</pluralRule>
    <pluralRule count="many">n mod 10 is 0 or n mod 10 in 5..9 or n mod 100 in 11..14</pluralRule>
    <!-- rest are plurals -->
</pluralRules>

PluralsReader class can parse these rules and define which form should be used for particular number. I committed this class (among others related) in Revision 4399.

When I wrote plurals reader, I started with another, similar class – NumbersReader. This one is more complicated. It can format a number (float or integer) using a format string. Syntax of format strings (patterns) is defined in CLDR. It’s very flexible and pretty complicated.

I didn’t implement all features described in CLDR (or to be precise: in Unicode Technical Standard #35), although I wanted the NumbersReader to support as much of the syntax as possible. I came across a solution of similar problem in Yii Framework codebase – it was very good starting point for me.

These are examples of formats supported by NumbersReader (one format per line):

#,##0.###
##0%
#,##0.00
00000.0000
'#,##0.0;(#)
¤ #,##0.00;¤ #,##0.00-
#,##0.05

NumbersReader class can parse format – it stores all parsed formats in the cache – and then can format a number using parsed representation of the format. It has methods for formatting decimal / percent / currency numbers (formats are extracted from CLDR). Formatting with custom format is also possible.

NumbersReader was commited in Revision 4445.

Next week I will work on similar Reader class for date and time. Hopefully I will also have time to do something with currency formatting – as for now, NumbersReader can format a value with currency sign, but it is simplified (just replacing currency placeholder with currency sign provided). CLDR has pretty extensive data for currencies.