Sandboxing the Mozilla Firefox Sandbox

by Justin Scott (fligtar)

Wednesday, March 24, 2010 at 07:04 AM EDT

With almost 2 billion downloads, add-ons have proven to be a huge part of Firefox’s growth and popularity over the last 5 years. As Firefox continues to be adopted by non-technical, mainstream users, the security and consumer experience of installing third party add-ons becomes increasingly more important. It’s with these users in mind that I propose some major changes to the way add-ons are submitted and distributed through Mozilla’s official add-ons gallery.

First, some background

This month marks the three year anniversary of the “sandbox” review model being introduced on addons.mozilla.org. Veteran Mozilla contributors and add-on fans may remember what the process was like prior to the sandbox: add-ons were inaccessible until they were reviewed, new add-ons and updates were listed together in the same queue, and the quality of some of the reviewed add-ons was questionable.

The current add-on submission and review process was designed to surface unreviewed add-ons in a “sandbox” to testers who wanted to try them out and write reviews, while still keeping them far away from casual Firefox users just looking to customize their browser. We hoped this would alleviate developer frustration over long review times and raise the quality bar, as not every add-on would have to be “public” and reviewed in order to be distributed on the site. In its original incarnation, the sandbox excelled at keeping untested add-ons from everyday users, but was found lacking in usability for advanced users: no one could figure out the process of signing up for an account and then opting in to sandbox access in order to see the unreviewed add-ons.

In the years since there have been two iterations affecting the discoverability of the sandbox: the first removed the opt-in requirement and allowed anyone logged in with an account to see the sandbox. This made the sandbox a bit more usable and easier to understand; however, feedback from developers indicated that more needed to be done. The second iteration, currently in use today, went even further by letting logged-out users browse and install unreviewed add-ons by simply checking a box.

We get a lot of community feedback on our ideas and site features, but almost all of that comes from developers who, of course, want to make their add-ons as easy to get as possible. We rarely hear from the non-technical users installing these add-ons, which can sometimes lead to decisions that are too developer-focused. I think the current state of unreviewed add-ons is an example of one of those decisions, and that we’ve gone too far in making them widely available. Unreviewed add-ons, which are potentially harmful to your computer and your data, are trivial to find and too easy to install without understanding the risk.

Since the inception of the sandbox, we’ve had very few incidents involving unreviewed add-ons, and have been quick to investigate and respond to any reports we’ve received from our users concerning these add-ons. But as we strive to make Firefox add-ons and our website more consumer friendly and encourage the hundred million users already enjoying add-ons to tell their friends about customization, “very few” incidents is a few too many.

What are the issues?

I helped design the original sandbox and had a part in all of its subsequent iterations, and I’ve heard from developers, users, and security experts on it. I’ve agreed with many of the issues brought forward, and we think it’s time to make some big changes. In my opinion, the biggest issues with the current system are:

• We use misleading terminology. Untested, unapproved add-ons are called “experimental”, which hints that the add-on is not ready for prime time, but doesn’t evoke enough (or any) levels of alarm or warning for a casual user. In computer vernacular, when something is in a sandbox, it is trapped in a secure container and can do no harm. This is not the case at all with the addons.mozilla.org sandbox.
• The interface doesn’t say what it means. For logged-in users, unreviewed add-ons have a green install button, the same as reviewed add-ons. People like to click on green, shiny buttons. For logged-out users, the only thing standing in the way is a checkbox that says “Let me install this experimental add-on.” We don’t explain what this actually means (“not reviewed”) or let the user make an informed decision unless they click on “What’s this?”.
• Untested add-ons are too popular. Yesterday there were 40,000 downloads of add-ons in the sandbox. While this makes up less than 3% of all downloads yesterday, I have serious doubts that all 40,000 downloads came from advanced testers who fully understand the risks of installing unreviewed add-ons.

We need to limit the exposure of unreviewed add-ons so that if an incident does occur, only the few people who have have made an educated decision and accepted the risk will be affected.

What can we do to fix this?

Some have proposed removing the sandbox and unreviewed add-ons entirely. It would certainly solve all of the above issues, but would leave us back where we were in 2006 when developers had to wait a week or two before anyone could try out their add-on. As an add-on developer myself, I know that after I finish working on something, I want to let my friends try it out immediately.

Others have proposed spinning off a separate website for add-ons that have not been reviewed, or encouraging a third party not affiliated with Mozilla to create a website for such creations to live. This may address most of the issues listed above, but I don’t think it would be in the best interest of users. If a user arrives at the only add-on they can find that has a certain feature, they’ll probably install it regardless of any warnings the site may have (keeping in mind a third party would not have to have any warnings at all). I think it’s important that Mozilla provide some form of home for these add-ons until they can be reviewed so that we can take the appropriate cautions for users that do wish to try out the add-ons.

I think there’s a middle ground, and that’s what I’m proposing today. There are three main components:

1. Unapproved add-ons will not be shown in the add-ons gallery.

Users can be sure that every add-on they come across browsing around the website has been tested and approved by Mozilla’s editors. This would mainly affect browsing through categories, search results, and collections.

2. Users who are linked directly to an unapproved add-on by its developer can install the add-on.

After submission, developers can send the link to their unapproved add-on to anyone they’d like to try it out. This can be on their blog, website, email, instant message, etc. However, these direct links will be disjointed from the normal add-ons gallery and make it clear that this add-on has not been reviewed, as well as offer the user the option to leave and go to the gallery with only reviewed add-ons.

3. There will be a maximum amount of time an unapproved add-on can be hosted on the site.

There are 7,500 add-ons currently in the sandbox that have never been reviewed by a human because the developers have never nominated them to show up in the “public” site. Some of these add-ons have been there for years, and are usually the source of the problems we do have.

To address this, we think a critical part of fixing the review process is setting the expectation that all add-ons submitted to the site must undergo review and using a time limit to enforce that.

Under this proposal, an add-on’s submission to addons.mozilla.org will begin its 30-day incubation period. As noted above, the add-on will not appear in the gallery but can be accessed through a direct link from the developer during this period.

The add-on must apply to the gallery sometime during this incubation period, whether on day 1 or day 30. Reminder emails will be sent at certain intervals to remind the developer of this timeframe, but if the add-on has not applied to the gallery during the 30-day window, its listing will expire and no longer be available on the website, even with a direct link.

If the developer has allowed an add-on to expire, he or she may still submit a gallery application after the 30-day window, but the add-on will not be re-activated for download until it has been approved.

The add-ons team tried to come up with details that we think are reasonable and not too complicated, yet still address many of the edge-cases we’re sure to encounter. I haven’t included all the details of this part of the propsal here, but those interested should read the full proposal and give feedback in the discussion thread.

Some additional important details that are worth mentioning in this summary are:

• This policy will apply to add-ons currently in the sandbox, and those add-ons will expire 30 days after this new process is implemented on the website, if it is adopted. Those authors will be given months of advanced notice in order to nominate their add-ons for review before the deadline rush.
• Currently, users who install “experimental” add-ons do not automatically update to new versions in order to limit the exposure of unreviewed code. Many users and developers do not realize or expect this, and it’s not adequately explained. Under this proposal, unapproved add-ons already have a limited exposure window, so automatic updates will be turned on for all add-ons, even if unapproved.
• New versions of add-ons will not be available for download until they have been approved. Currently, 95% of add-on updates are reviewed in under 5 days.

If implemented, this new system will make some big changes to the add-ons website, but I think they’re needed. We’re asking for community feedback on all parts of the proposal, although if you’d like to comment, please take the time to read the full proposal and then add your thoughts in the discussion thread.

This article originally appeared on fligtar's blog.