2020 is shaping up to be an amazing year for JavaScript Internationalization API.
After many years of careful design we’re seeing a lot of the work now coming close to completion with a number of high profile APIs on track for inclusion in ECMAScript 2020 standard!
Let’s cut to the chase. Here’s the list of APIs coming to the JavaScript environment of your choice:
Between 2017 and 2018 we refactored a major component of the Gecko Platform – the intl/locale module. The main motivator was the vision of Multilingual Gecko which I documented in a blog post.
Firefox 65 brought the first major user-facing change that results from that refactor in form of Locale Selection. It’s a good time to look back at the scale of changes. This post is about the refactor of the whole module which enabled many of the changes that we were able to land in 2019 to Firefox.
All of that work led to the following architecture:
It’s very rare in software engineering that projects of scale (a Gecko module with close to 500 000 lines of code) go through such a major transition. We’ve done it a couple times, with Stylo, WebRender etc., but each case is profound, unique and rare.
There are good reasons not to touch an old code, and there are good reasons against rewriting such a large pile of code.
There’s not a lot of accrued organizational knowledge about how to handle such transitions, what common pitfalls await, and how to handle them.
That makes it even more unique to realize how smooth this change was. We started 2017 with a vision of putting a modern localization system into Firefox, and a platform that required a substantial refactor to get there.
We spent 2017 moving thousands of lines of 20 years old code around, and replacing them with a modern external library – ICU. We designed a new unified locale management and regional preferences modules, together with new internationalization wrappers finishing the year by landing Fluent into Gecko.
To give you a rough idea on the scope of changes – only 10% of ./intl/locale directory remained the same between Firefox 51 and 65!
In 2018, we started building on top of it, witnessing a reduced amount of complex work on the lower level, and much higher velocity of higher level changes with three overarching themes:
We then ended up Q1 2019 with a Fluent 1.0 release, close to 50% of DTD strings migrated to Fluent, with all major new JavaScript ECMA402 APIs based on our Gecko work and what’s even more important, with proven runtime ability to select locales from the Preferences UI.
What’s amazing to me, is that we avoided any major architectural regression in this transition. We didn’t have to revisit, remodel, revert or otherwise rethink in any substantial way any of the new APIs! All of the work, as you can see above, was put into incremental updates, deprecation of old code, standardization and integration of the new. Everything we designed to handle Firefox UI that has been proposed for standardization has been accepted, in roughly that form, by our partners from ECMA TC39 committee, and no major assumption ended up being wrong.
I believe that the reason for that success is the slow pace we took (a year of time to refactor, a year to stabilize), support from the whole organization, good test coverage and some luck.
On December 26th 2016 I filed a bug titled “Unify locale selection across components and languages“. Inside it, you can find a long list of user scenarios which we dared to tackle and which, at the time, felt completely impossible to provide.
Two years later, we had the technical means to address the majority of the scenarios listed in that bug.
Three new centralized components played a crucial role in enabling that:
LocaleService was the first new API. At the time, Firefox locale management was decentralized. Multiple components would try to negotiate languages for their own needs – UI chrome API, character detection, layout etc.
They would sometimes consult nsILocaleService / nsILocale which were shaped after the POSIX model and allowed to retrieve the locale based on environmental, per-OS, variables.
There was also a set of user preferences such as general.useragent.locale and intl.locale.matchOS which some of the components took into account, and others ignored.
Finally, when a locale was selected, the UI would use OS specific calls to perform internationalization operations which depended on what locale data and what intl APIs were available in the platform.
The result was that our UI could easily become inconsistent, jumping between POSIX variables, user settings, and OS settings, leading to platform-specific bugs and users stuck in a weird mid-air state with their UI being half in one locale, and half in the other.
The role of the new LocaleService was to unify that selection, provide a singleton service which will manage locale selection, negotiation and interaction with platform-specific settings.
LocaleService was written in C++ (I didn’t know Rust at the time), and quickly started replacing all hooks around the platform. It brought four major concepts:
The result, combined with the introduction of IPC for LocaleService, led us in the direction of cleaner system that maintains its state and can be reasoned about.
In the process, we kept extending LocaleService to provide the lists of locales that we should have in Gecko, both getters and setters, while maintaining the single source of truth and event driven model for reacting to runtime locale changes.
That allowed us to make our codebase ready for, first locale selection, and then runtime locale switching, that Fluent was about to make possible.
Today LocaleService is very stable, with just 8 bugs open, no feature changes since September 2018, up-to-date documentation and a ~93% test coverage.
With the centralization of internationalization API around our custom ICU/CLDR instance, we needed a new layer to handle interactions with the host environment. This layer has been carved out of the old nsILocaleService to facilitate learning user preferences set in Windows, MacOS, Gnome and Android.
It has been a fun ride with incremental improvements to handle OS-specific customizations and feed them into LocaleService and mozIntl, but we were able to accomplish the vast majority of the goals with minimum friction and now have a sane model to reason about and extend as we need.
Today OSPreferences is very stable, with just 3 bugs open, no feature changes since September 2018, up-to-date documentation and a ~93% test coverage.
With LocaleService and OSPreferences in place we had all the foundation we needed to negotiate a different set of locales and customize many of the intl formatters, but we didn’t have a way to separate what we expose to the Web from what we use internally.
We needed a wrapper that would allow us to use the JS Intl API, but with some customizations and extensions that are either not yet part of the web standard, or, due to fingerprinting, cannot be exposed to the Web.
We developed `mozIntl` to close that gap. It exposes some bits that are internal only, or too early for external adoption, but ready for the internal one.
mozIntl is pretty stable now, with a few open bugs, 96% test coverage and a lot of its functionality is now in the pipeline to become part of ECMA402. In the future we hope this will make mozIntl thinner.
In 2019, we were able to focus on the migration of Firefox to Fluent. That required us to resolve a couple more roadblocks, like startup performance and caching, and allowed us to start looking into the flexibility that Fluent gives to revisit our distribution channels, build system models and enable Gecko based applications to handle multilingual input.
Gecko is, as of today, a powerful software platform with a great internationalization component leading the way in advancing Web Standards and fulfilling the Mozilla mission by making the Web more accessible and multilingual. It’s a great moment to celebrate this achievement and commit to maintain that status.
Welcome to the fourth edition of Multilingual Gecko Status Update!
In the previous update we covered the work which landed in Firefox 61-64.
At the time, we were landing Fluent DOM Localization APIs, still adding mozIntl features, and we had close to 800 strings migrated to Fluent.
I indicated that 2019 should be quieter, and in result I reduced the update frequency to just one this year.
Let’s dive into what happened:
Welcome to the third edition of Multilingual Gecko Status Update!
In the previous update we covered the work which landed in Firefox 59 and Firefox 60.
At the time, we’ve been finalizing the platform work to support Fluent localization system, and we were in the middle of migration of the first Firefox UI component – Preferences – to it.
Today, we’ll pick up right where we left off!
Firefox 61 fits into the trend of things calming down in Intl – and it’s a great news! It means that we are reaching platform maturity after all the groundbreaking refactors in 2017, and we all can focus on the work on top of the modules, rather than playing whack-a-mole fixing bugs and adding missing features.
The biggest platform change is really just an update to ICU 61 which Andre landed in March and my work on adding mozIntl.RelativeTimeFormat and mozIntl.getLocaleDisplayNames.
The former gave us a stable unified API for presenting relative time (such as “In 5 minutes” or “10 days ago”) while the latter unified how we present language, region and combinations of those in our user interface based on the Unicode CLDR representation (example: “English (United States)”).
As I explained in my earliest posts, one of the things I’m particularly proud of is that we go the extra mile to use every such opportunity to not only fix the immediate Firefox UI need, but also push such proposals for standardization and in result make the Web Platform more complete.
In this case, Intl.RelativeTimeFormat has been proposed and thanks to amazing work by Daniel Ehrenberg is now in Stage 3 and soon will be exposed to all web developers in all browsers! Intl.getLocaleDisplayNames is less mature but the work on it just picked up.
Firefox migration to Fluent reached its next milestone moving from just a couple messages to over 100 Fluent messages in Firefox!
Notable changes [my work] [intl module]:
Another calm cycle! The biggest feature was the introduction of the pseudolocalization in Firefox which I blogged about, and landing of the developer documentation.
The documentation has been insanely useful in distributing knowledge and helping engineers feel more comfortable working with the new stack, and I’m very happy we reached a stage where landing a documentation is the big news 🙂
In the Fluent land we spent the March-May timeframe moving from a 100 messages to 500 Fluent messages in Firefox!
Notable changes [my work] [intl module]:
This cycle was quite similar to the previous one, with a bulk of work going into regular maintenance and cleanups.
For my work, the trend is to start integrating Fluent deeper into Gecko with more work around L10nRegistry v1 limitations and getting deeper DOM integration to improve XUL performance.
In this cycle I landed a XPCOM mozIDOMLocalization API which allows us to call Fluent from C++ and was required for a bigger change that we landed in Firefox 64.
One new theme is Kris Maglione who started working on reducing the performance and memory overhead coming from the old StringBundle API (used for .properties). With all the new work centralized around Fluent, but with a large portion of our strings still using StringBundle, it becomes a great target for optimizations by cutting out everything we don’t use and now we know – we never will.
Notable changes [my work] [intl module]:
This release, is still getting stabilized and will get released in December, but the work cycle on it happened between September and October, so I can already provide you an account of that work!
Besides of a regular stack of cleanups coming from Henri and me, we’ve seen Ehsan Akhgari taking over from Kris to remove more lines of unused code.
The really big change was the introduction of DocumentL10n API which is a C++ API with its own WebIDL tightly integrated into the very core of DOM module in Gecko – nsIDocument.
Before that Fluent lived in Gecko in some form of a glorified javascript library. While it is Fluent’s goal to target the web platform, Firefox UI is inherently different from the web content and benefits from better integration between the DOM and its localization component.
This change allowed us to better integrate localization into the document’s life cycle, but what’s even more important, it allowed us to expose Fluent to documents that usually do not have special privileges and could not access Fluent before.
As for migration, we moved along nicely bumping from 500 to around 800 messages thanks to hard work of a number of students mentored by Jared Wein and Gijs Kruitbosch. The students picked up work on the migration as their Capstone project.
Notable changes [my work] [intl module]:
2018 has been much “easier” for the intl module than 2017 was. It’s great to see the how all pieces fit together and for me personally, it enabled me to focus on getting Fluent better integrated into Gecko.
There’s still a lot of work but it now is fully focused on Fluent and localization, while our intl module as a whole goes through a more well earned peaceful period.
Between now and the next status update, I hope to publish a summary post about the last two years of work. Stay tuned!
As promised in my previous post, I’d like to do a better job at delivering status updates on Internationalization and Localization technologies at Gecko at shorter intervals than once per year.
In the previous post we covered recent history up to Firefox 58 which got released in January 2018. Since then we finished and shipped Firefox 59 and also finished all major work on Firefox 60, so this post will cover the two.
Firefox 58 shipped with a single string localized using Fluent. In 59 we made the next step and migrated 5 strings from an old localization system to use Fluent. This allowed us to test all of our migration code to ensure that as we port Firefox to the new API we preserve all the hard work of hundreds of localizers.
In 59 we also made several improvements to performance of how Fluent operates, all of that while waiting for the stylo-chrome to land in Firefox 60.
In LocaleService, we made another important switch. We replaced old general.useragent.locale and intl.locale.matchOS prefs with a single new pref intl.locale.requested.
This change accomplished several goals:
defaultLocale, we can now skip it and just treat its non-presence as the default mode in which we follow the default locale. That allowed us to remove some code."fr-CA, fr-FR, en-US", in line with our model of handling locale lists, rather than single locales.This is important particularly because it took us very long time to unify all uses of the pref, remove it from all around the code and finally be able to switch to the new one which should serve us much better.
Next come a usual set of updates, including update to ICU 60 by Andre, and cleanups by Makoto Kato – we’re riding the wave of removing old APIs and unifying code around ICU and encoding_rs.
Lastly, as we start looking more at aligning our language resources with CLDR, Francesco started sorting out our plural rules differences and language and region names. This is the first step on the path to upstream our work to CLDR and reuse it in place of our custom data.
Notable changes [my work] [intl]:
Although Firefox 60 has not yet been released as of today, the major work cycle on it has finished, and it is currently in the beta channel for stabilization.
In it, we’ve completed another milestone for Fluent migrating not just a couple, but over 100 strings in Firefox Preferences from the old API. This marks the first release where Fluent is actually used to localize a visible portion of Firefox UI!
As part of that work, we pushed our first significant update of Fluent in Gecko, and landed a special chrome-only API to get Fluent’s performance on par with the old system.
With an increase in the use of Fluent, we also covered it with Mozilla eslint rules, improved missing strings reporting, and wrote an Introduction to Fluent for Firefox Developers.
On the Locale Management side, we separated out mozilla::intl::Locale class and further aligned it with BCP47.
But the big change here is the switch of the source of available locales from the old ChromeRegistry to L10nRegistry.
This is the last symbolic hand-over from the old model to the new, meaning that from that moment the locales registered in L10nRegistry will be used to negotiate language selection for Firefox, and ChromeRegistry becomes a regular customer rather than a provider of the language selection.
We’re very close to finalize the LocaleService model after over a year of refactoring Gecko!
Regular healthy number of cleanups happened as well. Henri switched more code to use encoding_rs, and updated encoding_rs to 0.7.2, Jeff Walden performed a major cleanup of our SpiderMonkey Intl source code, Andre added caches for many Intl APIs to speed them up and Axel updated compare-locales to 2.7,
We also encountered two interesting bugs – Andre dove deep into ICU to fix `Intl.NumberFormat` breaking on roundings in Persian, and I had to disable some of our bidirectionality features in Fluent due to a bug in Windows API.
Notable changes [my work] [intl]:
With all that work in, we’re slowly settling down the new APIs and finalizing the cleanups and the bulk of work now is going directly into switching away from DTD and .properties to Fluent.
As Firefox 60 is getting ready for its stable release, we’re accelerating the migration of Preferences to Fluent hoping to accomplish it in 61 or 62 release. Once that is completed, we’ll evaluate the experience and make recommendations for the rest of Firefox.
Stay tuned!