Addons-server: Improve source-viewer i/o performance, make sure it's not DOSable

[EnTeQuAk]

In our last push (https://mail.mozilla.org/pipermail/dev-addons/2017-April/002713.html) we had to disable our source-viewer temporarily to make sure our servers are not impacted too much by it. It's only available to reviewers and admins for now.

This issue is to track progress on fixing potential issues or thinking about other ways of re-enabling the source-viewer, e.g make it more performant, pre-extract source-code to avoid extraction or similar ideas.

[Gitoffthelawn]

What is the ETA to fix this? It's rather important.

[diox]

Through it would be good to fix it... maybe we should consider replacing it with just a link to a source repo, entered by the developer (stored on the Addon model, not the Version or File). It would be much, much simpler and just as good for users IMHO.

[wagnerand]

Another idea: Rename "View Source" to "Download Source" and force download the latest public XPI file on click.

[Gitoffthelawn]

Having an additional Download Source button makes sense.

But being able to quickly view the source within the browser is a much better option for a common task. Otherwise, just to check if an extension is e10s-compatible,users will have to:

1. Go to AMO
2. Find a link to download the source
3. Download the source archive
4. Open the source in an unzipper (many users will need to rename the file first to realize this is possible)
5. Find the appropriate file within the source archive
6. Extract the appropriate file from within the source archive
7. Open the file from item 5 in a text editor
8. Examine the file to see if the extension is e10s-compatible

Of course, the real solution is for AMO to indicate whether or not an extension is e10s-compatible.

[wagnerand]

Of course, the real solution is for AMO to indicate whether or not an extension is e10s-compatible.

Exactly. But we are moving towards a webextension-only world, so the importance of e10s compatibility for legacy add-ons keeps decreasing...in the way that at some point all add-ons will be webextensions and therefore e10s compatible by default.

[Gitoffthelawn]

Exactly. But we are moving towards a webextension-only world, so the importance of e10s compatibility for legacy add-ons keeps decreasing...in the way that at some point all add-ons will be webextensions and therefore e10s compatible by default.

Very true. But at the same time, WebExtensions won't be universal for more than half a year... at the earliest. (When was the last time Mozilla didn't delay a major technological shift?)

In the meantime, Mozilla will gather much better telemetry data and crash reports if more people are using e10s... especially from those users who actually use extensions. But right now, it is a huge PITA for users to determine whether or not extensions are e10s-compatible. The current (and sudden) lack of a source viewer on AMO is making this even more difficult.

[diox]

But right now, it is a huge PITA for users to determine whether or not extensions are e10s-compatible. The current (and sudden) lack of a source viewer on AMO is making this even more difficult.

Finding e10s information is irrelevant to this discussion: the source viewer was never a good solution for this, since it's up to the add-on developer to enable that link or not. Plus, as you yourself noted, it's really not practical for this usage anyway. Fortunately, displaying e10s information on AMO has its own issue, https://github.com/mozilla/addons-server/issues/4975

[Gitoffthelawn]

@diox If https://github.com/mozilla/addons-server/issues/4975 has a chance of getting resolved, I agree. But if not, the AMO source viewer has been the only semi-reasonable solution.

[EnTeQuAk]

So, from our discussion on IRC I think there are a few options we have.

Re-opening the current file-viewer for public usage isn't one of them. It still has performance problems even if only used internally and they are not easy to fix. Aside these there's no real good reason why we should provide a file-viewer interface to the public.

So, if we want to allow people to download the add-on files that are hosted on AMO we should provide them with a link to download the XPI file (basically a 'right-click, Save as' shortcut).

We can give developers the option to enter a public-repository url so that people can find that information too, that'd be actually a cool new feature for open source projects imho.

I personally strongly vote against a public file-viewer functionality and vote to implement the 'Download XPI file' functionally and additionally (optionally) a 'View source on … Github/BitBucket" link or something.

Any opinions on that @diox @muffinresearch @jvillalobos @eviljeff @wagnerand?

[wagnerand]

What I suggested above would still work for me (basically your third paragraph).

[Gitoffthelawn]

Aside these there's no real good reason why we should provide a file-viewer interface to the public.

I have to respectfully disagree. I'm "the public" and I have used AMO's file-viewer interface quite a bit for years. I have found it to be very useful. (BTW, as a usability expert, I do recommend consulting with your userbase before making assumptions about their needs.)

...a 'View source on … Github/BitBucket" link or something

I like this idea. It sort of exists, as "Add-on home page" and "Support site" links, but a real "View Source" link could be a worthwhile addition.

[jvillalobos]

I agree that giving developers the option to link to a repository, separate from a homepage and support link, would be valuable, and maybe sufficient to replace View Source. There is some added value in allowing to inspect the package directly in case users don't trust the package matching the sources. However, in that case they shouldn't trust us showing the right sources either and it would be best to download the package themselves and inspect it with their own tools. And then there's minification and obfuscation.

Moreover, the public file viewer was a relatively easy win given that we already need it for the reviewer tools. However, we're probably moving to different code inspection tools that we won't be able to publish in the same way. So it sounds like its days are numbered. But this issue still matters while we use the existing reviewer tools.

What I have in mind for the new front-end is that the All Versions pages would have download links rather than install buttons (and also view source links, but I'll change that given the situation). Later on we can add a way to point to a source repo and show it in the listing page as well.

[hckr]

Oops! Not allowed.

You tried to do something that you weren't allowed to.

Why does the message suggest a lack of permissions? Wouldn't it be better to change the message and/or hide "view the source" link for anyone but reviewers and admins?

[jvillalobos]

See the recently filed #5431

[EnTeQuAk]

I'm going to close this in favor of https://github.com/mozilla/addons/issues/689 and many other issues that are filed and will be tackled regarding the file-viewer.

I personally don't think we should enable the current file-viewer for public access and we will be working on a replacement for our reviewers eventually anyway.