Vscode_deno: discussion: multi registry import IntelliSense

[lucacasonato]

We added support for IntelliSense autocomplete a week ago. It only has support for deno.land/x right now. I would like this feature to support more registries (like nest.land or jspm.dev). This is currently not practical to do because each registry requires a separate code path in the extension.

As a solution to this I propose a dynamic import IntelliSense that is not specific to any one registry. You should be able to dynamically add more registries without updates to the VS Code Deno extension.

How does this work for the user?

All registries that want to support this feature must host a /.well-known/deno-import-intellisense.json file on their domain (contents of this file are described below). The extension will request this file when it encounters a domain in an import statement for the first time. If the extension finds this file and recognizes the syntax, a popup is shown to the user:

Do you want to enable import IntelliSense for https://deno.land? Only do this if you trust https://deno.land.

Learn more

[No] [Yes]

If the user enables this, this domain will be added to an array of allowlisted domains in the deno.import_intellisense setting in VS Code user settings (override per project possible). If a domain is part of this allowlist, autocomplete for that domain will be enabled (how this works is described below).

How does this technically work?

When an import is found with a matching registry it is parsed according to the schema defined in /.well-known/deno-import-intellisense.json. Depending on where in the URL the cursor is, the correct auto complete should be triggered. The actual IntelliSense logic should be the same as the one we use now.

Schema of /.well-known/deno-import-intellisense.json

This is approximately what the schema could look like for https://deno.land:

{
  "version": 1,
  "registries": [
    {
      "schema": "/x/:module@:version/:path*",
      "variables": {
        "module": {
          "url": "https://api.deno.land/importintellisense/modules"
        },
        "version": {
          "url": "https://api.deno.land/importintellisense/modules/${module}"
        },
        "path": {
          "url": "https://api.deno.land/importintellisense/modules/${module}/versions/${version}"
        }
      }
    },
    {
      "schema": "/std@:version/:path*",
      "variables": {
        "version": {
          "url": "https://api.deno.land/importintellisense/modules/std"
        },
        "path": {
          "url": "https://api.deno.land/importintellisense/modules/std/versions/${{version}}"
        }
      }
    }
  ]
}

So what does this mean?

The file contains a list of registries under this domain, each with a url schema it containing variables. These variables have a specified URL endpoint that is used to request the information for auto complete. These urls can have replacement variables which will be populated with the extracted values from matched the URL schema. There are two types of replacement variables: ${} which inserts the literal value of the variable and ${{}} which inserts the URL encoded value of the variable. These URLs are expected to return an array of strings in JSON format when sent a HTTP GET request. They will cached (respecting the cache-control headers).

cc @kitsonk @t8 @CGQAQ @callionica @David-Else

[kitsonk]

I like this a lot better. It allows users to add registries without modifying the code of the extension, and as important too, it doesn't pick a horse in the race.

[t8]

I completely agree with @kitsonk on this. In my personal opinion, this would be the best approach because it doesn't force any specific registry on a user (no horse picking), while also supporting any new service that comes along and wants to take advantage of this feature.

Don't get me wrong though, we are totally flattered about @CGQAQ's work to add support for Nest.land on the extension, but I can speak for the team and say that this new solution would be ideal for the Deno ecosystem as a whole.

[CGQAQ]

Don't forget to unifying the result of these API like I mentioned in https://github.com/denoland/vscode_deno/pull/209#issuecomment-693124218
Personally, I like denoland one, which you give me the entry is dir or file

[lucacasonato]

These URLs are expected to return an array of strings in JSON format when sent a HTTP GET request

@CGQAQ Currently they should all just return an array of strings. But in a later version we can add a version: 2 which has dates for versions, types and file sizes for dir listing, and star count (or similar) and descriptions for module names. But I think we should start out with the bare minimum features first. Let's just get this module schema working :-D

[CGQAQ]

These URLs are expected to return an array of strings in JSON format when sent a HTTP GET request

@CGQAQ Currently they should all just return an array of strings. But in a later version we can add a version: 2 which has dates for versions, types and file sizes for dir listing, and star count (or similar) and descriptions for module names. But I think we should start out with the bare minimum features first. Let's just get this module schema working :-D

I know it's array of string, what I'm saying is the file structure, for example: the denoland one is contains both file and folder and have a type property to indicate what it is, while the nestland one only contains files.

[CGQAQ]

Also, IMO, Star count is not that important to me, I want readme.md url. The vscode is capable of showing markdown in completion details see this

[callionica]

As I understand it, the proposal says that the VS Code extension will not send web requests to 3rd party registries (any registries) until the user enters at least the protocol+domain of that registry while in an import context in the editor. At that point a request will be made to that newly observed domain for a well-known file describing the registry. If found and valid, the user will then be prompted to enable autocomplete for that registry. If the user accepts, the extension will then make future web requests to the registry during editing without prompting the user. If the user decides not to enable autocomplete for that registry, future uses of that registry's URLs in an import context will neither trigger autocomplete nor any prompts to the user to enable autocomplete.

That sounds like a fair system that doesn't privilege particular registries. It also sounds like it makes a minimal number of web requests and only when users could reasonably expect them to be made.

[callionica]

Q1. It's not explicitly called out in the description, but the design calls for a block list as well as an allow list, right? So users could manually add domains to the block list to prevent any web requests including the web request for the well-known file?

Q2. How many requests for the well-known file will a domain that doesn't provide the file see? Can domains enable tracking by not providing the file?

Q3. If a user or organisation plans to use a single registry, can they use this system to enable autocomplete for that registry while still preventing all web requests to other locations (including the requests for the well-known file)?

[lucacasonato]

Q1. It's not explicitly called out in the description, but the design calls for a block list as well as an allow list, right? So users could manually add domains to the block list to prevent any web requests including the web request for the well-known file?

SGTM - so a deno.import_intellisense_ignore setting.

Q2. How many requests for the well-known file will a domain that doesn't provide the file see? Can domains enable tracking by not providing the file?

I would say we try max once every few days? A user can prevent this by adding the domain to the deno.import_intellisense_ignore setting.

Q3. If a user or organisation plans to use a single registry, can they use this system to enable autocomplete for that registry while still preventing all web requests to other locations (including the requests for the well-known file)?

What about a "deno.import_intellisense_discovery": false setting to disable the requests for the well-known file? It would be true by default. Users can still manually add registries via the deno.import_intellisense setting.

[callionica]

You've addressed my concerns about predictability and control over the web requests made. Your plan for that sounds good. Thanks!

I'll leave it to others to provide feedback on the details of 3rd party registries' integrating with autocomplete.

The only related question I have is whether import autocomplete will prefer or visually highlight previously used imports over imports suggested by the registry? It seems like an important feature to avoid honest mistakes and typo hijacking. And previously used imports could provide a local source of suggestions for when the network is slow. Maybe something like that is already implemented?

[lucacasonato]

@callionica Autocomplete for files already cached in $DENO_DIR is already implemented.

[callionica]

@lucacasonato If I've previously imported Customer, but registry contains Custard, I'd want Cust to expand to Customer, not Custard. Does the lookup on the cached files make that happen?

[lucacasonato]

@lucacasonato If I've previously imported Customer, but registry contains Custard, I'd want Cust to expand to Customer, not Custard. Does the lookup on the cached files make that happen?

Not yet. This is definitely something we could add down the road.