Categories
main mozilla tech

L20n gets tangible

While Firefox 4 was the main focus for the last weeks and months, I’ve been also making progress with the next iteration of Mozilla localization technology – L20n.

Here are three things that constitute a milestone for me and should make it much easier to test and play with the features of it.

Toolbox

Toobox guides you through the examples of various localization scenarios and how L20n solves them. It blends incremental learning of features available to both developer and localizer. At the bottom it contains several more complex example that should rarely happen but constitute the latter part of Pike’s “easy things easy, complex things possible” mantra.

XPCShell tests

It’s a small (two at the moment) set of tests that run l20n code. It’s a great start point to play with how the library works and how the format works. You can adjust the compiled code, or the library code and see if it gives the expected result.

Live toolbox

What can be better than a toolbox for a geek? Yes. A live toolbox. A toolbox you can not only read, but one that you can actually touch, change, hack on and see the result live.

It’s a hack itself, so don’t be harsh pls, but it does the job. I even included a set of 7 examples (example1 to example7) that correspond to what you can find in the Toolbox. Feel free to modify the L20n code, see if it compiles properly, play with the compiled code, change the HTML or JS and see the results live!

Also, if you encounter a bug, you can save your code and send it my way so that I can investigate it. The compiler is just an initial approach, and a moving target right now as we still don’t have a finalized JS structure schema, but it works for most simple and medium complexity cases, so I’d say it’s ready for you to play with it!

Next steps

Now, that we have Firefox 4 released (yay!), and mozilla-central is open again, I hope to work on landing the initial set of L20n Gecko bindings which requires some updates to the patches themselves first. With that part, we’ll be able to start investigating migration away from current DTD/properties format into the wonderland of L20n.

 

Categories
main mozilla tech

Crazy times for the localizers

March of 2011 is special for all of us. It’s the Firefox 4 release month and the whole project is operating in it’s “crazy” mode.

One particular plane on which it’s pretty visible is localization as we’re involved in almost every aspect of the release, every little detail, every step of the product release, every engagement campaign and of course in every bit of discussion about the state of the web that happens in our respective countries.

The story

As an organization that has glocalisation imprinted on the very fabric of its universe, we always feel responsible for making sure that everything we do is localizable, so that our communities around the world can translate and adjust everything that Mozilla project creates and use it to promote the Mozilla mission in their regions.

That’s easy to say, much harder to do. Especially as we grow, and the number of projects that we touch goes up. Firefox 4 release brings an unprecedented number of web projects that are part of our push to make bring the web to the next level.

Those projects vary from HTML5 demo sites, through Mozilla Developer Network Demo Studio, social web projects for twitter and facebook, web games to new Mozilla Foundation campaign – Join Mozilla. In total, I believe, we’re preparing 12 separate projects that we call “special projects” that are going to more or less closely accompany Fx4 release (not to mention Firefox 4 l10n itself, and our major projects like SUMO, MDN, AMO).

Each of the projects is very different and unique, uses different set of technologies and is embedded into different environment and yet we need to unify it for our localization community so that they don’t have to learn how to localize each of them separately.

Given the usual amount of challenges multiplied by 12 different setups it could have been a mess and it should have been impossible, but thanks to a great work done mostly by Staś Małolepszy, our team of l10n-drivers unified the experience as much as possible and localizers are right now doing an amazing job by bringing those special projects to every corner of the Earth, to billions of people.

Below is a summary of what happens there.

Categories
main mozilla tech

Hard Blockers Counter 1.0

A little bit more of coding, a few bug reports later, HBC is ready for its prime time. Version 1.0 fixes the nasty toolbar height problem, it gives a user an indication of the interval covered by his plot and is just overall better.

It can be downloaded from an addons.mozilla.org listing, and the source code is available at builder.addons.mozilla.org. 🙂

A few of the lessons learned and thoughts:

  • builder is awesome, but it needs more real life users. A lot of bugs are only reproduced after you write your extension for some time, hundreds of revisions etc.
  • AddonSDK is excellent for this kind of extensions. It has everything you may want and makes the whole code extremely clean and simple to write and maintain. Just look at it – about 50 lines of core code – your cat could read that.
  • AddonSDK needs more real life users. Like with the builder, bugs show up only when you really use the extension you created.
  • AMO is an excellent developer friendly platform – it gives me a lot of satisfcators in a form of stats, and ability to manage my extension release process.
  • AMO-builder bindings need more real life users. I felt like I’m the first to try to push builder based extension to AMO – many trivial bugs that can be only revealed if you try to go through the whole thing.
  • AMO’s review/release process is excellent for the extension of the Old Days. It gives us a pool of high quality, verified extensions, that are easy to find and safe to use. It does not work with agile development. Builder and AddonSDK makes creating ad-hoc extensions super simple and quick (literally, 2 hours and you’re done with the first version, every new version is about 15 minutes of work). When you then push it to AMO it feels like Matrix slow motion then – you suddenly wait days for a preliminary review, not to mention almost two weeks you have to wait for a full review. My last revised version is super old comparing to what I claim to be the “stable” one now 🙁

This issue requires a little bit of description. I do not try to say here, that what AMO reviewers are doing is wrong – quite the opposite, I believe the whole process is excellent and anything that is exposed to the millions of users should get some time to season and be tested and be reviewed. It’s just that AddonSDK/Builder gives you a totally different setup that fits different needs. I believe AMO will need a workflow for extensions that are created in 10 minutes, distributed in 20 minutes, updated 5 times during 4 hours and are becoming useless after one or two days.

Think of a conference where schedule is updated often and people have hard time to track it. Using AddonSDK/Builder you can create an extension for it in literally 20 minutes (xhr, panel, widget). AMO is excellent for distributing it, updating your users etc., but it requires very different approach than say, AdBlock or Firebug. Then, you add a feature (ability to mark the talk you want to attend and get a notification when its room/time changes) and upload a new version 15 minutes later. You want to switch all your users to the new one now. Then you fix a small bug affecting linux users, and update users once again.

It’s amazing that Firefox is becoming a platform where it is possible, and I can’t wait for such application for AMO 🙂

  • AddonSDK requires a lot of users with their use cases. Myk’s approach is to iterate often which means to get version 1 ASAP and then add new features for version 2 instead of trying to build an ultimate solution without a release. I love this approach and it serves AddonSDK well, but now we need version 2 of many of the packages there – it can only be done if people start using the packages for a real life extensions and report what they miss. Like – Widget content cannot be easily themed. Or, you cannot control Panel’s scrollbar appearance. Jetpack team cannot plan for those use cases, they have to come from jetpack users. So be brave! Try things, report everything you need! 🙂
  • There is a group of at least 500 people who deeply care about our release process. They’re ready to increase the amount of items on their screen to have a continuous updates on our progress toward Firefox 4. And it’s been just two days. It can motivate people to help, make them feel the sense of progress, help them understand the challenges better. It sucks outsiders closer to the inner circle. I believe we can do much more and the nightly users, mozilla planet readers and the audience of my extension deserve the chance to get more info which can help them start contributing! 🙂

Categories
main mozilla tech

Hard Blockers Counter 0.8 (0.9) – with a plot!

Tremendous success of the Hard Blockers Counter extension (it got over 300 daily users and 4 positive reviews – yet they were lost in the transition :() motivated me to spend another too-early-morning on adding a history plot for the counter.

It was a real pleasure and a good lesson to go through – simpleStorage, timer, panel, and contentScript communication stuff. I reported some bugs in the toolkit, in the builder and in the builder->amo release process. It’s all been fixed 😀
The only real unresolved issues is that the widget affects the height of the toolbar. Reported as bug 626728.

So, 0.8 is ready and waiting, enjoy and fight that bug number down! 🙂

Source code as always available.

[update] Actually, Builder provided me an old version of my jetpack and I submitted it to AMO as 0.8. So now I added 0.9, but it has to wait to get a review – so if you want colorful plot, download it directly from here.

Categories
main mozilla tech

Firefox 4 hard blockers counter

Last night was a usual “15 hours on a plane brings your sleep cycle into the fifth dimension” one for me.

I was pondering through the planet mozilla and realized that despite great effort from our UX Team, Firefox 4 nightly does not show up the single, most important information piece about the state of my browser, it does not answer the question that our whole project is living now:

How many blockers do we have until we reach Release Candidate

Well, you probably can imagine everything from there – now, I present you the missing piece – Fx4 Hard Blockers Counter extension in its full glory.

Source code available.

Categories
main mozilla tech

Using Common Pool – Tutorial

In this post I’m going to show you how to make your extension localizable via Common Pool and how it can be localized.

early adopter warning: It’s the first attempt, it’s a new concept, things may brake, things may be a little rough, stay calm. It’s not a replacement for a well known solutions yet, but it may be one day. Play with it, hack the service and the library, suggest improvements, report the issues, but stay calm whatever happens – there may be dragons.

Using the library

Since Common Pool is currently implemented in Addon SDK, I assume you’re using it for writing your extension. First thing is to add my localization module to your extension. It’s available at http://builder.addons.mozilla.org.

Once you have it in your dependencies, you can just include it into your code.

Let’s take an example Hello World code:

var notifications = require("notifications");

var menuItem = contextMenu.Item({
  label: "Hello World",
  contentScript: 'on("click", function (node, data) {' +
    '  postMessage(node.src);' +
  '});',
  onMessage: function (imgSrc) {
    notifications.notify({
      title: "Hello World",
      text: "Welcome to the new, wonderful world!",
    });
  }
}

Now, in order to make it localizable, all we need to do is to add one require, and change the string initialization:

var notifications = require("notifications");
var _ = require("localization").get;

var menuItem = contextMenu.Item({
  label: _("Hello World"),
  contentScript: 'on("click", function (node, data) {' +
    '  postMessage(node.src);' +
  '});',
  onMessage: function (imgSrc) {
    notifications.notify({
      title: _("Hello World"),
      text: _("Welcome to the new, wonderful world!"),
    });
  }
}

Done! That’s what I meant when I wrote that Common Pool is asking as little attention from the developer as possible. If it reminds you of common patterns used by GetText, you’re right on spot. We like this notation, we reuse it.

With such code, you’re ready to go 🙂 As a developer, you can move along, do your work, deploy your extension on AMO and never ever bother about this again, while we will make sure that the localizers can localize, and users will receive the localized version dynamically. 🙂

Early adopter warning: For the time of the field testing of this library, we do not actually have any integration with AMO, so when you want your addon to be localized, you need to “cfx xpi” it, or if you’re using the builder just click on “Download” and upload the result xpi here: https://l10n.mozillalabs.com/projects/p/mozilla-jetpacks/upload/. This is a temporary step

Using the web service

Once a developer uploaded his addon and his job is done, the camera shifts onto another hero – a localizer. This brave soul, can just go to https://l10n.mozillalabs.com, log in and start localizing. As a localizer, you can just select your locale and start localizing. You can filter by an extension and you can view how other locales localized it.

You can also select overrides and add one for a given jetpack or occurrence. When you finish localizing your jetpack, you’re done. We will spread your translation to all users of the jetpack who use your locale. If you later update your translation, all users will receive an update on the next day. Isn’t that awesome? 🙂

Early adopter warning: Until we fix bug 10, we refer to jetpacks by their unique ID, instead of the human readable name. It should be fixed soon 🙂

The beauty of early adopting

Well, that being said, you know you would not be happy to receive a complete, ready to use, bug free solution. That’s not what hackers life is about. In fact, we decided to blow quite a big hole through the middle of the road that’s called “First Run Experience“.

As we were working on the library, and the webservice, and how nicely they’re tied, and we enjoyed the experience of a user staying up to date with localization of his extensions, and being able to download an extension in en-US and then see it localized on the day when a localizer finishes localizing it… we actually did not solve a pretty important case. Here’s the issue:

When a user installs an extension it contains no translation resources bundled with it. On the first run, we asynchronously ask the localization web service for the entities used by the extension. So far so good.

The server asynchronously sends the user all the entities and they’re cached. From that moment every request for a string will work by providing a translation of the string. But what happens before we cache?

Well, for now, nothing. Bummer 🙁 Basically, until we cache the strings, all we have is an empty cache and the localization library will just return the english strings which is the default fall back.

The bug for that is reported, bug 619807, and we’re working on a solution for that. We may decide to bundle entities that we have available for the extension together with it at the time of download, but that will be just a workaround. An ideal solution would be to pull the localizations together with the addon (as a part of addon installation experience) and then check if there are any localization updates on a normal daily routine.

If you have any ideas, suggestions or code snippets that solves it – feel free to jump in and help! 🙂

Real Life Example

Take a look at this example extension, Translate Selection with l10n. Now, go to l10n.mozillalabs.com to see its localization and localize it live.

Next time you run this extension you should be able to see it working with localization (assuming your browser is using your locale and minus the First Run Experience bug).

That’s all. 🙂

Summary

Common Pool is a simple, yet very powerful solution. It solves a lot of issues that we’ve been struggling with since the beginning of times. It makes internationalization easier than ever, localization easier than ever and synchronization easier than ever. Bare in mind, it’s intended as a solution for simple strings, for developers who don’t have much time to think through their UI’s and plan it for localization, and for massive localization of tons of thousands of extensions.

We believe it nails it perfectly (with the few early adopters hiccups that we’re working on right now, and an unlimited number of issues that are awaiting to be discovered by you!), and we want to improve it until we feel it to be the right one.

At the same time, please, remember that for a more complex and sophisticated UI’s, the ones where a developer has to think about entities, and the localizer wants to fine tune each and every sentence – we’re going to work on a complementary, L20n based solution.

Now, it’s your turn to play with it, hack on it, give us feedback and help us evaluate the solution. We’re available at irc.mozilla.org #l10n and #jetpack channels, and we use mozilla-labs-jetpack group that you can use. For bugs in the l10n web service, please, use its bitbucket issue reporting system. Thanks! 🙂

Categories
main mozilla tech

Jetpack Localization – Common Pool approach

Jetpack project is taking an aim at rewriting the extension system, learning from the past, responding to the feedback and thinking outside of the box. So when the Jetpack team approached L10n-Drivers and asked for support with building localization infrastructure for new addons, we decided to go the same way. 🙂

Common Pool

The first approach we’re presenting you is called Common Pool (yup, that’s not a very creative name) – it’s basic assumption is that most jetpacks are small applications with a minimal UI. And there are going to be tons of them. We already have tens of thousands of extensions, and Jetpack with its nice API and builder are going to make it much, much easier to write a good extension.

We also made some observations – There are common phrases, like “Cancel“, “Ignore“, “Log in” that are replicated over and over in almost every extension. It’s becoming hard to localize it, hard to keep it consistent and maintain the localization. On the other hand, extension authors are struggling to work with localizers as they need to iterate fast and  string freeze phases are adding a lot of cost to the release process.

For that reasons we decided to build on a basic concept of entities shared across extensions, stored on a server, and pulled by the jetpack itself.

Shared entities are powerful. For a localizer it means that we can show you the list of mostly used entities and if you localize 20 of them, you localized 50, 100 or maybe 500 extensions. You can update a translation and have it translated the same way across all extensions. You can localize only one extension and your work will probably benefit other extensions as well. You can localize today, and tomorrow all users will get updated localizations. How cool is that? 🙂

For a add-on developer it also means more good. You can pick entities that are already localized into many languages and have you extension Just Work. You don’t have to worry about releases and localization because if you only make thing localizable, users will be able to pull localizations as they appear. It means less to think about.

Now, there are of course some trade offs. One of them is that not all translations of a word should be equal. For that reason we propose a concept of an exception. Actually, two of them.

First one is a per-addon exception which allows localizers to localize a given string differently for this particular addon. It’s important when the addon uses some specific vocabulary.

Second one is a per-case exception which allows localizers to divert from the common translation for a specific use case. Say, the use case in jetpack X, file main.js line 43 should be translated differently.

Library and the Web Service

So, that’s the concept, and it has been implemented as a library. Yahoo! Now, for such a solution to work, we knew we need a partner in crime. We need a web service that will support Common Pool and allow for addon localization. We knew we will want to merge it with AMO and the Builder, so django was a preferred technology, and one other requirement was to be able to support a localization storage model that is substantially different from the industry standard.

Fortunately, at the time when we conceptualized Common Pool, our friends from Transifex were reworking their storage model away from gettext-specific to much more generic database driven one. Well then – good time to see how generic it can be, right?

Together with Transifex team, mainly Dimitris Glezos and Seraphim Mellos we’ve been working on the Common Pool web service that can back up the library and become a localization web tool. Today, the first iteration of the tool is alive at – https://l10n.mozillalabs.com! The source code is GPL and available at transifex-commonpool bitbucket repository and that’s where we’re collecting bugs as well.

What’s next?

Now we have to prove that the concept in the field. We’re challenging the status quo, so it’s on us to validate it. We’re going to reach out to extension authors and localizers alike to get support and testing coverage and if we that it works, we’ll include common pool library into Addon SDK 1.0. It’s up to you to help us evaluate it!

Then, it’ll be time to complement Common Pool with another architecture, that does not solve the simple cases that easily, and require much more active work from the developer, but will provide better level of details for both parties. The solution we want to offer is L20n.

Bundle pack ahead

Together, Common Pool and L20n should provide everything that localizers and developers may need pushing the limits and stretching what’s possible with the localization. We think of a Common Pool as a great generic, easy solution for most common strings, while L20n complements it with more sophisticated, more powerful approach for the developers whose extension reached the level when they want it to be localized with more attention and where the UI requirements are higher.

Let me know what you think – I know it’s early, I gave many talks about those technologies to our Mozilla communities, but I realize that it’s hard to evaluate without actual code in hand, so now is the time for feedback, feedback, and even more feedback 🙂

In the next blog post I’ll explain the steps to create your first common-pool driven extension.

Categories
main mozilla tech

Mozilla Summit 2010 – Localization 2.0 talk

Here come slides I used for Summit 2010 Localization 2.0 talk.

It was a very tough talk to give. Hard to grasp, hard to explain. I originally wanted to devote it exclusively to L20n, and make it as a form of tech talk, but eventually figured out it will not work and there’s much broader vision I need to explain. Thus a few hours before the talk I started rewriting it and end up with what you can find here.

Of course slides alone will tell you just a small part of the story, but it’s better than nothing. 😉

Thanks to all who participated in this! I know it eats a lot of brain cycles to process and it was already 4pm, but I hope you enjoyed it! 🙂

Categories
main mozilla tech

Pontoon – more details and 0.2 plans

Catching up from the blog post that introduced Pontoon, I’d like to present the project in a little more detail and the vision for a near future.

Clients

As stated previously, Pontoon 0.1 introduces two types of clients – web client and jetpack client. Both share majority of code base (Pontoon, jQuery, jQueryDomec and heavily modified editableText plugin), but because of the limitations of jetpack prototype I had to merge them into one JS file.

user experience

For user interface of Pontoon 0.2 I’d like to focus on a few key aspects that may make Pontoon a tool ready to use in work environment:

  • Introduce identification system that will sign contributions made by a given user. We may use Verbatim accounts for now.
  • Figure out a way to recognize the state of localization and allow for continuing localization.
  • Add ability to view/edit entity in a source view. (much like wordpress’s HTML view)
  • Fix some common cases like entity with a single anchor should treat the anchor as an entity, not its parent
  • Improve UI to look nicer and inform user on the outcome of his actions.

With those six improvements (including two top ones which are major), Pontoon will become, in my opinion, ready for real life use and we may offer it for one of the upcoming small websites for those brave souls who would like to test it.

code base

On the code base front, I will definitely want to clean up the sources, which will be easier when jetpack client get updated to new jetpack architecture. editableText plugin will require a cleanup and will be extended to allow source view, so it may make sense to fork it as a separate project, while other classes should be refactored and site-dependent code should be moved away from libraries. Also, CSS inclusions should be merged into libraries.

One thing I’d like to see soon is a third type of a client – one that may be embedded into any website so that the website author can turn on and off the “localization” mode.

Server

Pontoon server will require adaptation to the client side changes, like ability to identify/authenticate users, and then two new things should be possible with the data received from a client. First, server should be able to add translations as suggestions in Verbatim. This way users will be able to suggest translations, leaving up to the registered localizers what to do with it. This workflow is an intended one for Pontoon.

Second thing, is that we should be able to generate a pure, localized HTML file out of a source one and translation list. This way we will be able to localize things like SUMO documents or plain HTML sites that do not use Gettext or even PHP.

Hooks

Hooks will go through a major rewrite, because I want to test the concept of providing an external meta file with information like lists of entities for the client to read. Such meta information is the most natural way to also add ability to pick up localization from a midpoint. The issue here is that of course this will not help the client with localization of websites that do not use pontoon hooks.

It would be also great to add Django hooks, since so many websites this days are migrating to django, and it would also allow us to test how cross-linguistic the hook system is.

Summary

This is my plan for a short term, and I’ll try to get as many of the features described above into Pontoon 0.2 as possible. I prepared a wiki article describing the plan, and I’m open to feedback, ideas, and of course, contribution. 🙂

Categories
main mozilla tech

Pontoon – introduction

One of the three core types of content I described in my previous blog post is what I call rich content.

This type of localizable content has certain characteristics that are very different from the common UI entities. Starting from different goal (to provide information), through different size (long sentences, paragraphs), different l10n flexibility (needs ability to reorder, extend, shrink long texts) to much richer syntax (sentences can be styled with HTML, CSS).

Almost a year ago I started playing around some type of a web tool that would allow for WYSIWYG style localization for the web, later picked up by Ozten in his blog post about potential use cases, and finally landed on Fred Wenzel’s mind as he decided to give this idea a try. He ignited the project, and created first iteration of the UI, where I joined him, and added server side.

Now, the idea itself is neither new nor shocking. Many projects were experimenting with some forms of live localization for web content but the solutions we know so far are hardly adaptable for generic purposes. Some require a lot of prerequisites, others are gettext only, and they come as black-box solution which requires you to follow all the procedures and lock your process in.

Pontoon is different in the sense that it is a toolset that allows for rich content localization with grading amount of prerequisites and varying strictness of output. It’s a very alpha mode tool, so be kind, but I think it’s ready to get the first round of feedback 🙂

Pontoon – components

Pontoon is composed of three elements:

  • client – pontoon client is an application that allows for content navigation and localization. It’s responsible for analyzing content and providing ways for the user to translate it.
  • server – pontoon server is an application that the client communicates with in order to send/update translations provided by the localizer and store them later for the website to use.
  • hook – pontoon hook is a small library that can be hooked into a target application in order to provide additional information for the client to improve its abilities.

There are various clients possible, for now we have two – HTML website as a client, and Jetpack 0.8.1 based one. They share a lot of code and they operate quite similarly.

We have one server – django based one – that can receive translations provided by the client, and uses Silme to store them into a file (currently .po), and we have one hook – for PHP – that adds special meta headers and provides simple API for modified gettext calls that gives client direct information on where the entities are located, so that the client does not have to guess.

I’ll dig into details in a separate post, there’s much more into how Pontoon can operate and what are the possible enhancements for each of the three components, but for now, I’d like to present you a short video of Pontoon used to localize two old versions of our website projects.

The video is a coffee-long, so feel free to grab one and schedule 6 minutes 🙂

urls used in the video:

notes:

  • the locale codes that are supported by the server are usually ab_CD, not ab – for example de_DE, fr_FR etc.
  • pontoon does not support multiple users at the same time, so you may observe strange results when many people will try it at the same time. Enjoy alpha mode!
  • “0.1 alpha” means – I’m looking for feedback, comments, ideas and contributions 🙂