Invidious: [RFC] Authentication and Authorization API

[omarroth]

Related: #427, #469

For various applications it is desirable to be able to read or write user data. This proposal describes a format for tokens, a method for authenticating requests, and endpoints for authorizing tokens.

Token format

Each token is a JSON object with the following schema:

{
  "session": String,
  "expire": Int64?,
  "scopes": Array(String)
  "signature": String
}

Tokens will be created with a given session, which allows them to be revoked by the user.

Tokens may be issued with an expire timestamp after which the token will be considered invalid.

Tokens will have a list of one or more scopes determining their permissions. The format for scopes is described below.

Tokens will have a signature to ensure integrity of the other fields.

Example token:

{
  "session": "v1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
  "expires": 1554680038,
  "scopes": [
    ":notifications",
    ":subscriptions/*",
    "GET:tokens*",
  ],
  "signature": "f//2hS20th8pALF305PJFK+D2aVtvefNnQheILHD2vU="
}

Authentication

All sub-routes of /api/v1/auth must be authenticated.

Endpoints requiring authentication are authenticated with an Authentication: Bearer <token> header.

Tokens are validated by taking all key-value pairs, minus signature, and creating the following string:

key1=value1
key2=value1,value2,value3

Values are alpha-sorted. Keys are alpha-sorted and joined by newlines. There is no trailing newline.

The signature is created by signing the string using SHA256-HMAC with the instance's HMAC_KEY and Base64 encoded.

Using the above token as an example:

expires=1554680038
scopes=GET:tokens*,:notifications,:subscriptions/*
session=v1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Signed using SECRET_KEY provides the signature f//2hS20th8pALF305PJFK+D2aVtvefNnQheILHD2vU=.

Scoping

In order to provide a means of limiting permissions for an application, tokens must provide one or more scopes.

Each scope refers to a sub-route of /api/v1/auth. Scopes are defined as zero or more HTTP methods (semicolon-separated), colon (:), followed by an endpoint and optional wildcard for matching sub-routes.

A scope _X_ is said to be containing scope _Y_ if _X_ can match _Y_, but _Y_ cannot match _X_. For example, :subscriptions* contains GET:subscriptions/subscribe.

Example scope allowing access to /api/v1/auth/subscriptions using any method:

{ "scopes": [":subscriptions"] }

Example scope allowing access to /api/v1/auth/subscriptions and any sub-routes using any method:

{ "scopes": [":subscriptions*"] }

Example scope allowing access to any sub-routes of /api/v1/auth/subscriptions/ using only GET and POST:

{ "scopes": ["GET;POST:subscriptions/*"] }

Example scope allowing access to any endpoint using any method:

{ "scopes": [":*"] }

A SID cookie is equivalent to a token with "scopes": [":*"].

POST /api/v1/auth/tokens/register

The /api/v1/auth/tokens/register endpoint would support the following body (Content-Type: application/json):

{
    "scopes": Array(String), // List of scopes (comma-separated)
    "callbackUrl" : String?, // URL for redirecting after successful authorization, optional
    "expire": Int64?  // Int64, optional
}

Calls to /api/v1/auth/tokens/register must _already_ be authenticated in order to receive a new token. This can be accomplished by a signed in user or with a Bearer token with a scope containing POST:tokens/register.

Calls made using a Bearer token can only create tokens with equivalent or subsets of their own scopes.

If calls are made using a SID cookie, the user will be presented with a page presenting the requested scopes, and must then authorize the request before being redirected to callbackUrl. If no callbackUrl is provided, then after authorizing the token the user will be redirected to a page on the instance containing the newly created token.

If a callbackUrl is provided, then after successful authorization the instance will redirect to callbackUrl with ?access_token=<token> set as the query.

POST /api/v1/auth/tokens/unregister

The /api/v1/auth/tokens/unregister endpoint would support the following body:

{
    "session": String?
}

Tokens can be unregistered by calling /api/v1/auth/tokens/unregister. A token must have a scope containing POST:tokens/unregister in order to unregister itself. In order to unregister other tokens a token must have a scope containing GET:tokens and POST:tokens/unregister.

If the response body is empty, or session is not provided the token is assumed to be unregistering itself.

Users would also be provided a /list_tokens endpoint for viewing and degistering tokens they had authorized.

Example endpoints

All endpoints below are sub-routes of /api/v1/auth.

• GET/POST /preferences - Would allow GET of current preferences, or POST with updated preferences
• GET /notifications - Same as /api/v1/notifications with support for ?since=, see #469
• GET /subscriptions - Shows list of currently subscribed channels by the user
  
  
  
  • POST /subscriptions/:ucid - Subscribe to given ucid
  
  
  • DELETE /subscriptions/:ucid - Unsubscribe to given ucid
  
  
• GET /tokens - Lists tokens authorized by user
  
  
  
  • POST /tokens/register
  
  
  • POST /tokens/unregister
  
  

Example of use

For an application with a shared pool of users, such as FreeTube or CloudTube, an example token could be:

{
  "session": "v1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
  "scopes": [":notifications", "POST:subscriptions/*"],
  "signature": "fNvXoT0MRAL9eE6lTE33CEg8HitYJDOL9a22rSN2Ihg="
}

This would allow clients to use ?since=TIMESTAMP to receive any notifications missed when offline, and subscribe to new channels. DELETE:subscriptions/* is not included here to prevent unsubscribing to channels that still may need to be tracked by other clients.

[omarroth]

Technically, an application only has to subscribe in order for a desired channel to be tracked by the instance (see #469). It would be possible for a client to subscribe and then unsubscribe so that the subscriptions for the given account (and corresponding feed) doesn't become too large.

[omarroth]

POST and DELETE should be used for /subscriptions/subscribe and /subscriptions/unsubscribe, and similarly for /tokens/register and /tokens/unregister, respectively.

Just updated the above, DELETE is not used for /tokens/unregister since DELETE with a body is unsupported in some HTTP implementations.

[omarroth]

Added with 2a6c81a89dd64c00dff0b37af1c6637771aad27e. There are a couple changes from what's proposed above, and clarification on anything that's missing:

• /api/v1/auth/tokens/register supports application/x-www-form-urlencoded so it's possible to request a token using something similar to the following:

<form action="https://invidio.us/api/v1/auth/tokens/register" method="post">
<input type="text" name="scopes[0]" value="GET:subscriptions*">
<input type="text" name="scopes[1]" value="POST:tokens/register">
<input type="text" name="scopes[2]" value="POST:tokens/unregister">
<input type="text" name="callbackUrl" value="https://www.example.com/callback">
<input type="submit" value="submit">
</form>

Which will require the user to authorize the token as expected.

• callback_url was renamed to callbackUrl for consistency with the rest of the API.

• /list_tokens was renamed to /token_manager to match with /subscription_manager, and allows a user to revoke API tokens from the UI

• POST to /api/v1/auth/tokens/register with a Bearer <token> and no callback URL will return the requested token as the response body. Requesting with callback URL will return a 302 redirect.

• /api/v1/auth/tokens/unregister will return 204 on successful response

• Content-Type: application/json is required when POSTing with JSON body, otherwise you may encounter an unexpected result

• Bearer tokens will only be valid authentication for /api/v1/auth/*

---

Currently the only API endpoints that require authentication are for tokens themselves. Unless there is any unexpected/unspecified behavior I'll close this and work on adding the other endpoints ( /preferences, /subscriptions, /notifications, etc).

[omarroth]

Added GET /subscriptions, POST /subscriptions/:ucid, and DELETE /subscriptions/:ucid with 250860d92c166b06d4af5d713ec2640a5ad4e701.

[omarroth]

Added GET /preferences, POST /preferences with 8a525bc13109e9bee3a59d3082d763f91b1dc7f6.

[cloudrac3r]

Please provide a web form for the registration of new tokens. I added this simple HTML to a file and opened that in the browser so I could easily create tokens for testing purposes, but it would be really nice if Invidious had its own page for this. Maybe link to this page from the token manager page?

<!-- TEMPLATE pre -->
<title>Invidious token generator</title>
<style>
    label {
        display: block;
    }
</style>
<script>
    let nextScopeIndex = 0;
    function addScope() {
        let ne = new ElemJS("label").text("Scope: ").child(
            new ElemJS("input").attribute("name", `scopes[${nextScopeIndex++}]`)
        )
        let form = q("form");
        form.insertBefore(ne.element, form.lastElementChild);
    }
</script>
<!-- TEMPLATE header -->
<button onclick="addScope()">Add a scope</button>
<form action="https://invidio.us/api/v1/auth/tokens/register" enctype="application/x-www-form-urlencoded" target="_blank" method="post">
    <input type="Submit">
</form>
<!-- TEMPLATE end -->

I cannot simply use curl to test because the initial token creation request redirects to a "confirm authorisation" web form with a CSRF token, and it would be very inconvenient to simulate the completion of that second form with curl.

[cloudrac3r]

It would be convenient if the token manager page also showed the list of scopes that the token is authorised to access, to allow me to easily tell tokens apart. Or, allow some sort of optional "label" to be provided during the token creation process, and display this label next to its token.

[cloudrac3r]

Added GET /subscriptions, POST /subscriptions/:ucid, and DELETE /subscriptions/:ucid with 250860d.

Since this is apparently the thread to be in:

cloud@future ~> curl -i -H 'Authorization: Bearer extremely_sexy_token' 'https://invidio.us/api/v1/auth/subscriptions/UC9-y-6csu5WGm29I7JiwpnA' -X POST
HTTP/1.1 204 No Content
Content-Type: application/json
Access-Control-Allow-Origin: *
X-Frame-Options: sameorigin
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src blob: data: 'self' https://invidio.us 'unsafe-inline' 'unsafe-eval'; media-src blob: 'self' https://invidio.us https://*.googlevideo.com:443
Referrer-Policy: same-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Transfer-Encoding: chunked

(a long pause of over a minute without curl exiting, then finally prints this and exits)

curl: (18) transfer closed with outstanding read data remaining

[omarroth]

Please provide a web form for the registration of new tokens.

You might try using GET /authorize_token?callback_url=https://example.com&scopes=SCOPE,SCOPE although it's currently not well documented.

Or, allow some sort of optional "label" to be provided during the token creation process, and display this label next to its token.

Sounds good.

curl: (18) transfer closed with outstanding read data remaining

I'm having trouble reproducing this locally, testing both https://invidio.us and https://dev.invidio.us with curl --trace:
https://invidio.us:

[snip]
<= Recv header, 28 bytes (0x1c)
0000: 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e Transfer-Encodin
0010: 67 3a 20 63 68 75 6e 6b 65 64 0d 0a             g: chunked..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a                                           ..
<= Recv SSL data, 5 bytes (0x5)
0000: 15 03 03 00 12                                  .....
== Info: TLSv1.2 (IN), TLS alert, close notify (256):
<= Recv SSL data, 2 bytes (0x2)
0000: 01 00                                           ..
<= Recv data, 0 bytes (0x0)
== Info: transfer closed with outstanding read data remaining
== Info: Closing connection 0
=> Send SSL data, 5 bytes (0x5)
0000: 15 03 03 00 12                                  .....
== Info: TLSv1.2 (OUT), TLS alert, close notify (256):
=> Send SSL data, 2 bytes (0x2)
0000: 01 00         

https://dev.invidio.us

[snip]
<= Recv header, 28 bytes (0x1c)
0000: 54 72 61 6e 73 66 65 72 2d 45 6e 63 6f 64 69 6e Transfer-Encodin
0010: 67 3a 20 63 68 75 6e 6b 65 64 0d 0a             g: chunked..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a                                           ..
<= Recv data, 6 bytes (0x6)
0000: 31 0d 0a 0a 0d 0a                               1.....
<= Recv SSL data, 5 bytes (0x5)
0000: 17 03 03 00 1d                                  .....
<= Recv data, 5 bytes (0x5)
0000: 30 0d 0a 0d 0a                                  0....
== Info: Connection #0 to host dev.invidio.us left intact

I'll try testing it out a bit more to see if there's an issue with Content-Length or some other bug with an empty response, although currently I suspect the issue is with HAProxy or some middlewhere somewhere. Probably better to open this as a new issue.

[omarroth]

Appears to be a bug in curl, reported here: curl/curl#3968.

[omarroth]

Added /api/v1/auth/feed with 27e032d10dce0bd657e7b705d3dc8b24f1b55178.

[thatsatoffee]

POST /api/v1/auth/tokens/register

[...]
If calls are made using a SID cookie, the user will be presented with a page presenting the requested scopes, and must then authorize the request before being redirected to callbackUrl.

While it is certainly desirable to allow the user to control access to their account, this behaviour limits the capability of applications using the API to make authentication as painless as possible to the user. Especially in an environment where including or accessing a webbrowser is not possible, this makes it necessary to either store the credentials in the application (to allow for cookie-based authentication) or to have the user create the token on another device and then copy it over to the application in some way.

To circumvent this, it may be a good idea to take a page out of Google's book. To authorize the YouTube addon for Kodi to access features of a user's account, the addon requests a code from YouTube and then prompts the user to visit a webpage at google.com (formerly at youtube.com) and enter that code there. The webpage then allows the user to decide whether they want to authorize the addon to access the requested features.

It would be great if Invidious would include such a page as well as this would allow for, say, the development of a matching Kodi addon. :)

Note that this approach differs from that described by @cloudrac3r above in that the latter still requires the user to copy the token over to the application.

[trymeouteh]

This will be amazing and will allow third party apps to be made for Invidious accounts such as NewPipe.