Addons-server: Add external API to collections

[wagnerand]

In order for Personas Plus to be migrated to WebExtensions, we need an external API to access a user's collections, at least the Favorites one.

[wagnerand]

@muffinresearch I am marking this P2, so we don't lose track. We don't need this right away, but soon so we can start migrating PP.

[eviljeff]

Fwiw, we have the collections API now. It's the authentication that's the issue - our current external auth, JWT, isn't particularly well regarded. The method of getting the keys is possibly a bit beyond the average Personas Plus user too.

(I'd like to see all of our APIs be available externally, ideally)

[diox]

From IRC discussion, we're probably going to have to implement oauth2...

[EnTeQuAk]

Which would have many advantages for us to be honest:

• It's (very) widely used, wether it's good or not, many people know it and many libraries implement it
• I think oauth2 as excellent client support, especially from various javascript libraries
• It's relatively simple to hook in a 3rd party authentication provider (fxa) into the pipeline if desired

There's lots more but I think these are the most important points that should be noted.

[muffinresearch]

In order for Personas Plus to be migrated to WebExtensions, we need an external API to access a user's collections, at least the Favorites one.

@wagnerand Can you point to a specific list of requirements for this please? It will help us to know what's missing.

[wagnerand]

@muffinresearch Do you need more information beyond https://docs.google.com/document/d/1SYTfAbeHvQ_G-7eiB_oyIfKIAI4M8jCGvkQvdzW5rWQ/ ?

[eviljeff]

"AMO needs to provide an API for collections." which links back to here 😆

[wagnerand]

Hm, ok so I'm thinking of the following (MVP):
Retrieve the themes of the "Favorites" collection of a user. I guess a theme already contains a bunch of information, the least bit the add-on would need would be whatever the API in https://bugzilla.mozilla.org/show_bug.cgi?id=1369209 is consuming. A link to a "preview image" would be nice to have.

Of course, that can be split up into something like

1. Retrieve all collections of a user (The add-on would need to figure out which one the "Favorites" one is though. Is there a way to do that?)
2. Retrieve all items of a collection (and the add-on does the filtering of themes client-side)

[wagnerand]

I'm still not sure how the logging-in mechanism would work in case the "Favorites" collection is private.

[eviljeff]

Retrieve all collections of a user

http://addons-server.readthedocs.io/en/latest/topics/api/collections.html#list

(The add-on would need to figure out which one the "Favorites" one is though. Is there a way to do that?)

It always has the slug 'favorites'. So it'd be /api/v3/accounts/account/(int:user_id|string:username)/collections/favorites/

Retrieve all items of a collection (and the add-on does the filtering of themes client-side)

http://addons-server.readthedocs.io/en/latest/topics/api/collections.html#collection-add-ons-list
The detail object contains the add-on detail object which contains a whole lot of information - all that's needed for addons-frontend to display the detail page so I assume enough for this too.

[wagnerand]

Alright, what about the authentication to AMO?

[muffinresearch]

Andreas and I had a call yesterday to explore different possibilities for solutions to the current issues related to getting the necessary data from AMO.

Using the new collections APIS

Initially we were looking at what the web-extension would look like if it just used the public collection APIS. There were a few issues with this.

• The legacy add-on uses cookie auth - the web-ext would need a way to get the username to use the new APIS. Without an auth flow this becomes a UX pain point because a user might not necessarily know the connection between their login (email address) and username.
• The list endpoint for the new collections API doesn't allow getting the list of public collections - it would be nice if without auth this endpoint could return the public collections or maybe this could/should be a separate endpoint?
• Assuming the extension could get a list of public collections it would need to inform the user they'd need to edit their favourites collection to make it public if not present in the list - or alternative use a different collection - we could link directly to the relevant page to carry out that change.
• All of the above creates some friction points in the add-on from a UX perspective.

Adding auth to the new APIs

Another option to make the APIs be able to access private data (collections still have the concept of public and private even if in the future this might change).

• We'd need an estimate of effort on this piece of work. Any estimate would need to include any requirements around pages to handle the authorization flows and any revocation etc.
• Time is tight in Q3 currently so this is unlikely to be possible this quarter.
• We'd need to prioritize this with @jvillalobos against all the other things in q4

Continue using the current legacy APIs

• Needs investigating if the web-ext version would have any technical blockers to re-using the same APIs the current add-on uses.
• Going this route would require an agreement that we don't deprecate the old APIs until we have implemented external auth for the new APIS

Actions:

• @wagnerand is to look at the possibility of using the old apis from a web-ext
• AMO engineering will look into providing an idea of how much time is needed to implement the external auth solution.
  
  
  
  • There's this bug to track that discussion https://github.com/mozilla/addons-server/issues/5964
  
  

[eviljeff]

The list endpoint for the new collections API doesn't allow getting the list of public collections - it would be nice if without auth this endpoint could return the public collections or maybe this could/should be a separate endpoint?

Not being able to list user's collections other than your own was a specific requirement of the collection changes @jvillalobos wanted. The change wouldn't be difficult to make, but then we're adding something _soley_ for this add-on. (In addition to the other friction points mentioned).

If the webext keeps using the existing APIs we'd need to know exactly what's used, because the plan was to remove all the existing views that the addons-server pages use once addons-frontend takes over.

[wagnerand]

Not being able to list user's collections other than your own was a specific requirement of the collection changes @jvillalobos wanted. The change wouldn't be difficult to make, but then we're adding something soley for this add-on. (In addition to the other friction points mentioned).

Apart from being specified in the PRD, I don't see a reason why public collections should not be accessible using the API anonymously when they are publicly visible on the website. I would have assumed API and website behave the same.

Maybe @jvillalobos can shed some light on this.

[eviljeff]

They are _accessible_ you just can't list them.

[wagnerand]

Sorry for not being precise. My point is that you can list them on the website if you're not authenticated (e.g. https://addons.mozilla.org/en-US/firefox/collections/mozilla/) bot not using the API.

[wagnerand]

If the webext keeps using the existing APIs we'd need to know exactly what's used, because the plan was to remove all the existing views that the addons-server pages use once addons-frontend takes over.

Here are all requests (I could find) PersonasPlus makes:

https://services.addons.mozilla.org/en-US/api/1.5/addon/%ADDON_SLUG%/?format=json
to get persona details like images

https://addons.mozilla.org/collections/mozilla/featured-personas/format:json
to get all Mozilla featured personas

https://addons.mozilla.org/collections/mine/favorites/format:json
to get the current user's (own) favorites. The user is probably determined by the cookie that is sent along.

https://versioncheck.addons.mozilla.org/en-US/themes/update-check/%ADDON_ID%
to check for theme updates and apply them. Although I am not sure why this is necessary. Doesn't the regular update ping include themes too?

https://addons.cdn.mozilla.net/user-media/addons//%ADDON_ID%/header.png
https://addons.cdn.mozilla.net/user-media/addons//%ADDON_ID%/header.png
to get header and footer images in order to apply them

Some are probably a non issue like the ones that hit the CDN.
@muffinresearch Are there any plans on keeping the JSON output for the new frontend as well?

[eviljeff]

https://addons.mozilla.org/en-US/firefox/collections/mozilla/ and the like are the existing pages. The collection detail pages will be replaced by new addons-frontend pages but there won't be any list pages, there won't be listed on an account page (other than your own) and you won't be able to search for them.

See https://docs.google.com/document/d/1DutmS8yco-hNiKJ1-0MrkAkze7Ue88xpA4b-MWx6MLI/edit and https://docs.google.com/document/d/1cJ1k8HsAb6taauvJMDLOiYMa1McfUXNgEtax3hoToPc/edit#heading=h.z5wk3ws26iay

[jvillalobos]

Like @eviljeff pointed out, the plan is to make all collections publicly _accessible_, but there won't be a way to _browse_ to them on AMO, with the possible exception of featured collections, which might show up in the homepage. For the same reason, there shouldn't be an API to list a user's collections, unless you're that user.

Now, those are all plans for the new front-end, but given the timing it's best to focus the work on that future rather than how things work now.

Personas Plus is an important add-on, so if any work needs to be prioritized early in Q4 to make it work with WebExtensions, I hope we can make that happen. However, I don't know how much work it is to add auth, or whether it will be needed only for this API.

[eviljeff]

https://services.addons.mozilla.org/en-US/api/1.5/addon/%ADDON_SLUG%/?format=json

This is the old api. It's deprecated already and will probably be removed at some point but it's not part of the collections changes and the move to addons-frontend. There is equivalent functionality in the new api.

https://versioncheck.addons.mozilla.org/en-US/themes/update-check/%ADDON_ID%
to check for theme updates and apply them. Although I am not sure why this is necessary. Doesn't the regular update ping include themes too?

Theme updates are whole separate thing it seems... psa: reading theme_update.py will give you nightmares. But it will be staying around until static themes replace personas.

https://addons.mozilla.org/collections/mozilla/featured-personas/format:json
https://addons.mozilla.org/collections/mine/favorites/format:json

These are the two that would go because they're part of the old views that will be obsolete in Q4. But the former can be replaced with:
https://addons.mozilla.org/api/v3/accounts/account/mozilla/collections/featured-personas/ (doesn't require auth because its a public collection).
And the latter is:
https://addons.mozilla.org/api/v3/accounts/account/<username|id>/collections/favorites/ (doesn't require auth if your favorites collection is public. e.g. https://addons.mozilla.org/api/v3/accounts/account/eviljeff/collections/favorites/ )

The current url requires auth anyway to get your own personas, as you said, so it's actually pretty similar in terms of requirements if those are the only urls - there aren't any list requests.

[wagnerand]

Now, those are all plans for the new front-end, but given the timing it's best to focus the work on that future rather than how things work now.

Aha!

Theme updates are whole separate thing it seems... psa: reading theme_update.py will give you nightmares. But it will be staying around until static themes replace personas.

My question was more why the add-on does its own update checks, doesn't AOM check for theme updates already?

https://addons.mozilla.org/api/v3/accounts/account//collections/favorites/ (doesn't require auth if your favorites collection is public. e.g. https://addons.mozilla.org/api/v3/accounts/account/eviljeff/collections/favorites/ )

and that's fine as it seems it's the direction collections are going into. But it still leaves us with the problem of how the add-on figures out the username, quoting @muffinresearch:

The legacy add-on uses cookie auth - the web-ext would need a way to get the username to use the new APIS. Without an auth flow this becomes a UX pain point because a user might not necessarily know the connection between their login (email address) and username.

[muffinresearch]

Like @eviljeff pointed out, the plan is to make all collections publicly accessible, but there won't be a way to browse to them on AMO, with the possible exception of featured collections, which might show up in the homepage. For the same reason, there shouldn't be an API to list a user's collections, unless you're that user.

That's a good reason to leave it how it is then 👍

[muffinresearch]

@wagnerand where did we end up with this? Did you manage to achieve what you needed to?

[wagnerand]

Yes, I am currently working around this by accessing the AMO session cookie.

[davidhedlund]

@wagnerand

https://addons.mozilla.org/collections/mozilla/featured-personas/format:json
to get all Mozilla featured personas

That's useful, thanks. Is it possible to list all collections by https://addons.mozilla.org/collections/mozilla/ ? Unfortunately this didn't work:

https://addons.mozilla.org/collections/mozilla/format:json

[diox]

@davidhedlund : http://addons-server.readthedocs.io/en/latest/topics/api/collections.html#list

[davidhedlund]

@diox Thanks but I have already read that, can you just give me the URL? https://addons.mozilla.org/collections/mozilla/results/format:json didn't work either.

http://addons-server.readthedocs.io/en/latest/topics/api/collections.html#list relies on authentication, https://addons.mozilla.org/collections/mozilla/featured-personas/format:json doesn't.

[diox]

Issues are not the right place to ask for help. Try discourse forums, mailing lists or IRC. You're right though, I forgot that you can't list collections belonging to somebody else than you (and that to access you own you need auth). I don't think there is any way around this.

[davidhedlund]

@diox

I don't think there is a way around this though.

Should I submit a new issue for that? Or perhaps you can do it even better?

[diox]

I think that's by design. Feel free to open a new issue over mozilla/addons, then we can ping product to find out.

[davidhedlund]

@wagnerand I submitted https://github.com/mozilla/addons/issues/722, you can ignore what I asked you about.