Deno: Caching and loading non-importable resources

[KSXGitHub]

Problem

I often find myself needing to load non-JavaScript non-TypeScript assets, and various problems arise:

• Local module (in git repo) has to use fs but remote module has to use fetch.
• The assets cannot be cached.
• The assets cannot be bundled.

Suggestion

Add some way to annotate assets using comments, and fetch should support those assets. For example:

// @deno-asset './assets/foo.wasm'
// @deno-asset './assets/bar.png'
await fetch(new URL('./assets/foo.wasm', import.meta.url))
await fetch(new URL('./assets/bar.png', import.meta.url))
// --allow-net is not required since it reads from cache

Alternatives

There is asset reference proposal that is likely not going to get to stage 3 anytime soon.

[nayeemrmn]

Resolved

I don't think fetch() has room for these special identifiers since those are valid URLs, and it wouldn't be browser compatible. Also those URLs are relative to the current module, right? Regular runtime functions can't determine or change behaviour based on the current module URL, so you need to resolve it explicitly:

// deno-asset "./assets/foo.wasm"
// deno-asset "./assets/bar.png"
await fetch(new URL("assets/foo.wasm", import.meta.url));
await fetch(new URL("assets/bar.png", import.meta.url));
// The above skip net permission because they fetch cached, statically dependent
// resources -- similar to using imports from a static code dependency.
// As a special case they should work for `file://` URLs so both local and
// remote modules/resources will work.

---

Ref #4001. Nevertheless, the problem of "modules needing to statically depend on non-code resources" exists in web browsers. Even though the proposal you linked is most likely going nowhere, it's still risky for Deno to invent its own solution.

[KSXGitHub]

I don't think fetch() has room for these special identifiers since those are valid URLs, and it wouldn't be browser compatible. Also those URLs are relative to the current module, right? Regular runtime functions can't determine or change behaviour based on the current module URL, so you need to resolve it explicitly:

// deno-asset "./assets/foo.wasm"
// deno-asset "./assets/bar.png"
await fetch(new URL("assets/foo.wasm", import.meta.url));
await fetch(new URL("assets/bar.png", import.meta.url));
// The above skip net permission because they fetch cached, statically dependent
// resources -- similar to using imports from a static code dependency.
// As a special case they should work for `file://` URLs so both local and
// remote modules/resources will work.

This is simpler, let's use it.

Nevertheless, the problem of "modules needing to statically depend on non-code resources" exists in web browsers

I suppose bundlers have that problem too? Maybe we could use some references.

it's still risky for Deno to invent its own solution.

We can hide it behind --unstable flag.

[ry]

I think we'll add import foo from "foo.wasm" soon.

We had this feature before, but removed it before 1.0 because we felt it had not seen enough usage to commit it to it.

[KSXGitHub]

I think we'll add import foo from "foo.wasm" soon.

This is good to hear. What about other kinds of resources (such as images)?

[ry]

@KSXGitHub I'm quite open to adding support for JSON - we also removed that before 1.0 due to some security concerns. I don't know about images - we need to investigate what's happening in the web standards proposals.

[nayeemrmn]

Plugins, too. I think the point is being able to statically depend on arbitrary non-importable resources, not discussing what resources can be imported.

the problem of "modules needing to statically depend on non-code resources" exists in web browsers. Even though the proposal you linked is most likely going nowhere, it's still risky for Deno to invent its own solution.

I've thought more about the Web vs Deno on this issue...

• The Web provides no way for scripts to do this -- _statically_. They have to fetch() them at runtime. The same is currently true for Deno.
• On the web you can pre-fetch resources in the HTML context:
  html <link rel="preload" href="bar.png" as="fetch">
  Deno's analogy to this would be listing some pre-fetched URLs in a command line flag... not very useful. What's requested is a way for a module to distributively declare it's own dependency on a resource as it could for an import.
• Normally I would say "Why? Modules are for code-splitting. Modules shouldn't be managing resources in a local way." but the practical uses are too many.
• The reason Deno might especially need this is because those runtime fetches require --allow-net. Why should they? Those are statically identified resources. Ideally, they're viewed the same as code dependencies from a sandbox perspective. The proposal here uses comments as a way to download them at compile time instead.

[KSXGitHub]

• On the web you can pre-fetch resources in the HTML context:
  html <link rel="preload" href="bar.png" as="fetch">

I think we can use this (in triple-slash comment) instead of // @deno-asset. Example:

/// <link rel='preload' href='./assets/foo.wasm' as='fetch' />
/// <link rel='preload' href='./assets/bar.png' as='fetch' />
await fetch(new URL('./assets/foo.wasm', import.meta.url))
await fetch(new URL('./assets/bar.png', import.meta.url))
// --allow-net is not required since it reads from cache

Does this still count as "Deno inventing its own solution"?

[kitsonk]

My opinion. We should only allow importing resources that logically represent code which Deno can execute and have a statically determinable binding surface. Deno can execute Web Assembly and there are browser standards in flight to talk about how it exposes itself in JavaScript. It can execute plugins and Deno's own APIs determine the bindings.

Images are not code that Deno can execute. CSS is not code that Deno can execute.

JSON _can_ be interpreted as code, and there are established conventions on how to determine binding patterns, though it isn't part of current standards tracks.

Making fetch() cacheable and working with local files (both which have open issues) should be the primary API used for non-code assets. Potentially plugins processors for the runtime compiler APIs and the support of custom compilers (which also both have existing issues) would be ways users could extend Deno in a supportable fashion to support other resources they wish to treat as code.

[KSXGitHub]

Making fetch() cacheable

This is what this issue is based on. The difference is that Deno should preload resources even before executing code.