Deno: proposal: support a comprehensive meta-data file

[kitsonk]

There is discussion around "lockfiles" and sub-resource integrity (see: #200).

There are also other "external" data that might be needed to describe a workload in Deno, like what the main module is, a custom TypeScript configuration, permissions and other command line options, import-maps, etc. While Deno should always _just work_ without a meta data file, there are lots of use cases where external meta data is necessary or desired.

Providing a single file way of describing that meta data would be a desirable feature.

I propose the following:

• Deno supports a single file meta-data schema that fully describes a workload.
• That the format of this file would be .json
• That when a command resolves an argument to a .json file, it assumes that it adheres to the meta-data schema, meaning that deno run app.json would read the meta data file and utilise that to determine how to cache and execute the app. This would also mean something like deno run https://example.com/app.json would also _just work_.

As this proposal evolves, the definition of the schema would be detailed, but would include all the functional areas mentioned above. Specifically for the "lockfile" mechanism, if an application is loaded and the .json file is writable (local) and there is no cache of the hashes of the modules, then Deno will write out the hashes of all dependent modules to the file. Upon subsequent invocations where the cache of the hashes is present, Deno will validate that all modules loaded match that cache, and if they don't they will throw and the workload would terminate.

Support a command like deno write main.ts app.json would scaffold out the meta data file, but not run the workload. Any arguments passed on the CLI could be automatically written to the file as well.

[nayeemrmn]

So one of the ideas here is to consume the meta-file by using it as the entrypoint of the app rather than implicitly finding and reading it based on obscure logic. Nice.

[7040934]

อีเมลข้อมูลและข้อมูลส่วนตัวหายไปไหนใครมาสวมแทน

[7040934]

อีเมลข้อมูลและข้อมูลส่วนตัวหายไปไหนใครมาสวมแทน

[7040934]

มีการอภิปรายเกี่ยวกับ "lockfiles" และความสมบูรณ์ของทรัพยากรย่อย (ดู: # 200 )

นอกจากนี้ยังมีข้อมูล "ภายนอก" อื่น ๆ ที่อาจจำเป็นต้องใช้ในการอธิบายปริมาณงานใน Deno เช่นเดียวกับที่โมดูลหลักคือการกำหนดค่า TypeScript ที่กำหนดเองสิทธิ์และตัวเลือกบรรทัดคำสั่งอื่น ๆ แผนที่การนำเข้า ฯลฯ ในขณะที่ Deno ควร_ทำงาน_เสมอหากไม่มีไฟล์ข้อมูลเมตามีหลายกรณีที่จำเป็นต้องใช้หรือต้องการข้อมูลเมตาภายนอก

ให้วิธีเดียวในการอธิบายว่าข้อมูลเมตาจะเป็นคุณสมบัติที่ต้องการ

ฉันเสนอดังต่อไปนี้:

• Deno สนับสนุนสกีมา meta-data ไฟล์เดียวที่อธิบายถึงเวิร์กโหลดอย่างสมบูรณ์
• รูปแบบของไฟล์นี้น่าจะเป็น .json
• That when a command resolves an argument to a .json file, it assumes that it adheres to the meta-data schema, meaning that deno run app.json would read the meta data file and utilise that to determine how to cache and execute the app. This would also mean something like deno run https://example.com/app.json would also _just work_.

As this proposal evolves, the definition of the schema would be detailed, but would include all the functional areas mentioned above. Specifically for the "lockfile" mechanism, if an application is loaded and the .json file is writable (local) and there is no cache of the hashes of the modules, then Deno will write out the hashes of all dependent modules to the file. Upon subsequent invocations where the cache of the hashes is present, Deno will validate that all modules loaded match that cache, and if they don't they will throw and the workload would terminate.

Support a command like deno write main.ts app.json would scaffold out the meta data file, but not run the workload. Any arguments passed on the CLI could be automatically written to the file as well.
ไม่เคยมีใครมอบหมายถึงอะไรไอ้ทั้งสิ้น

[rsp]

Any arguments passed on the CLI could be automatically written to the file as well.

@kitsonk Do you mean the --allow-* arguments here as well?

I'm wondering how the permission system would work when running, e.g.:

deno run https://example.com/app.json

with app.json having been generated with:

deno write --allow-all main.ts app.json

More specifically, what would this mean:

deno write --allow-read=X --allow-read=Y main.ts app.json
deno run --allow-read=Y --allow-read=Z app.json

Would it:

• run with read access to X and Y
• run with read access to Y and Z
• run with read access to X, Y and Z (sum)
• run with read access to only Y (intersection)
• throw a runtime exception on access to X
• exit with error on startup

Edit - other options that come to mind:

• require running app.json with explicit --allow-* arguments that are the same as those required by app.json file
• require running app.json with explicit --allow-* arguments that are compatible (a superset) of those required by app.json file
• add --allow-meta argument to use privileges defined in the meta-data json file (could lead to problems if people get used to do it all the time)

[aliabolhassani]

Some other use cases:

• Putting icon URI or encoded base64 data for the GUI apps #3234
• Device compatibility: control who runs the app

[brandonkal]

The issue with this is that directories will end up looking like this:

README.md
args.ts
args.json
dedent.test.ts
dedent.test.json
dedent.ts
dedent.json
doc-gen.ts
doc-gen.json
go.ts
go.json
http-cache-semantics.ts
http-cache-semantics.json
kite
kite.ts
kite.json
kubernetes
kubernetes.ts
kubernetes.json
kx.ts
kx.json
merge.ts
merge.json
promise-object.ts
promise-object.json
proxymise.ts
proxymise.json
quokka-shim.ts
quokka-shim.json
resolve-promise-object.ts
resolve-promise-object.json
runtypes.ts
runtypes.json
sourcemap-codec.ts
sourcemap-codec.json
tsconfig.json
unraw.ts
unraw.json
utils.ts
utils.json
yaml-formatter.ts
yaml-formatter.json
yaml-tag.ts
yaml-tag.json
yaml.test.ts
yaml.test.json
yarn.lock

As there isn't much info that would be provided in such a file, I'd like to propose that this information is instead embedded in the .ts file itself as suggested here: https://github.com/denoland/deno/issues/3675#issuecomment-583691889
My other issue is that it is intuitively odd to execute a json file.

[aliabolhassani]

@brandonkal I agree.

For the private artifactories, we definitely need a separate file for authentication and config purposes like .npmrc file.

Also, repeating the remote module URI in every single file is against DRY principle. So, I propose to have a different file to assign an alias name to each remote module. This way, we can have different implementation of each functionality as well. For instance:

import { select } from "sql";

instead of

import { select } from "https://deno.land/[email protected]/db/sql.ts";

Having this, modules can easily get replaced with the compatible ones just by keeping the alias name and changing the URI in the manifest file. (Like a dependency injection)

We need to have a .ts file as an entry point of the app which can re-export other config/modules like, a file for artifactory registry and another for the alias names. For instance:

mod.ts

export { registry } from ‘./manifests/registry';
export { alias } from ‘./manifests/alias’;

[brandonkal]

@aliabolhassani You agree but then you suggest using a manifest file. I don't understand. What you describe is already possible with a importmap.

[aliabolhassani]

@brandonkal The top module, exports the manifests to Deno. Since any application only needs one configuration like permission and artifactory, no need to have manifest files for every single .ts file. This seems to be a clean way to declare manifests, however, need for the reserved words like 'registry', 'alias', 'permission', ... could be criticized.

Although, your suggestion in #3675, seems to be smart, but presenting configs as comments looks odd.

An alternate way would be to behave configs using Triple-Slash Directives in TypeScript, like:

/// <reference path="..." />
/// <permission path="..." />
/// <registry path="..." />

[KSXGitHub]

Is this metadata file supposed to be written by human? If it is, then I suggest Deno should support formats other than JSON file.

[mathiasrw]

@KSXGitHub I agree. The JSON regime has been a pain for so many.

I would strongly advocate supporting the hjson file format. Relaxed syntax, fewer mistakes, more comments. See http://hjson.org. there is a rust implementation at https://crates.io/crates/serde-hjson

An alternative is json5 from http://json5.org (https://crates.io/crates/json5) or even a .yml file (even if the indent things is a bit error-prone in my opinion)

I guess it really depends on the speed of each implementation.

[KSXGitHub]

@mathiasrw HJSON is not as complicated as YAML while supporting proper string indentation. Good choice. 👍

[hayd]

toml is another obvious candidate.

IMO the way forward here is for someone to make a proof of concept as a deno installable script/separate repo, that way it could be battle tested prior to anything being included in mainline deno (I don't think we're close to that yet).

[brandonkal]

I'm not a fan of this proposal for reasons stated above. This should really be implemented in userland. However if Deno implements this it should be JSON. HJSON doesn't really add much value.

Write the config in whatever language you want and have it compile down to JSON before feeding it to the system.

[KSXGitHub]

@hayd I'm not a big fan of TOML. It is quite obvious that this metadata file would have nested object, which is TOML weakness. Aside from that, indentation of multi-line string is not properly handled like in YAML and HJSON.

[KSXGitHub]

Write the config in whatever language you want and have it compile down to JSON before feeding it to the system

I disagree. Deno supports TypeScript to avoid intermediate compilation steps.

[andyfleming]

While TOML is preferable for some people, I believe there's still a lot of javascript users that aren't familiar with the syntax.

I think JSON5 would be a great option. It's more flexible and robust than JSON while still remaining intuitive for javascript users.

[brandonkal]

I disagree. Deno supports TypeScript to avoid intermediate compilation steps.

So write it in TypeScript...
It handles JSON output perfectly.
Users really don't want code that requires configuration to run.

[mathiasrw]

@KSXGitHub Good point regarding TOML. I must say I also like the idea that you can cenerate the file as regular .json, save it and it just works.

@brandonkal I disagree. A config file like this will act like the package.json does for node - it will be hand edited often and removing dumb problems for users must be a goal for a new ecosystem. Why limit people from commenting in the config file? Why refusing to run because you added an extra , at the end of the last element? Dare I compare it to why they moved from assembly to C? It does the same but it's easier for the developer not to mess things up.

Valid json is both valid hjson and json5

[hayd]

IMO the way forward here is for someone to make a proof of concept as a deno installable script/separate repo, that way it could be battle tested prior to anything being included in mainline deno (I don't think we're close to that yet).

This first and then we can bike shed about which exts/syntax to support...

[brandonkal]

The first step would be to consider what such a file would contain:

1. URL to an import map (which must be JSON)
2. URL to a tsconfig.json
3. A string of command line arguments for deno run

That about sums it up. That can all be contained in a 3-line comment at the top of my application code.
No need for another file or a different configuration language.

So again, there is no value in duplicating package.json here. We don't need that.

[KSXGitHub]

@brandonkal That is short-sighted.

Config file is not for only Deno to read, but also for all kind of tools in the ecosystem to work with. Just like in Node.js, package.json can be read by npm, webpack, jest, etc. Encode them as comments in JavaScript/TypeScript file would force all these tools to use an AST parser.

Config file is not read-only either, it can be modified by machine. Embed them as comments in code would lead to all kinds of problem, such as code style not being preserved.

Furthermore, comments were never meant to be read. Using comments as config would be clumsy.

there is no value in duplicating package.json here

So you duplicate package.json 3 times instead? (importmap, tsconfig.json, deno run arguments).

[brandonkal]

Other tools have their own config files. Deno doesn't need a config file.

You can try to have 10-20 tools reading and writing to the same config file, but that isn't scalable. It also encourages copy-and-paste. You are better off linking to your tsconfig.json which is required for other tools such as VSCode.

package.json isn't scalable. It's the reason we have a seperate tsconfig.json. I understand you are familiar with package.json, but it is not something you should look to model.
Importmap is a web standard, so it gets its own file.

So that just leaves run arguments. It is trivial to write a tool that parses JSDoc and extracts Deno run arguments. Do you really want to litter your projects with yet another file, deno-config.json to document some deno run arguments? I won't be doing that. There's a lot of value in having documentation with the code. It's why we use tools such as JSDoc, rustdoc, godoc, etc. Why not also document which run arguments are required to execute that file with the rest of your module's documentation?

[KSXGitHub]

It is trivial to write a tool that parses JSDoc and extracts Deno run arguments

Please don't tell me you would use RegExp.

[KSXGitHub]

Also, your suggestion would lock user to a particular file type: JavaScript/TypeScript while Deno can also run WASM.

[brandonkal]

Nice try. I don't have anything more to add here. My point still stands: https://github.com/denoland/deno/issues/3179#issuecomment-603673018

• There are three strings that such a metadata file would contain.
• I've shown a package.json clone does not scale
  Three values does not warrant its own file. Parse it however you like.

Also, placing metadata in the entrypoint module does not lock people out of using WASM.

[mathiasrw]

Deno doesn't need a config file.

I agree. I prefer no config file. But IF we are going to have one (because x) I still suggest its a .hjson

[aaronhuggins]

Would there be any interest in an approach that uses a TypeScript file for a manifest? https://deno.land/x/deno_run

I have written this because I see a place where users may need to install or run a project on the command line, but having to write out lengthy arguments may become tedious and can lead to just throwing --allow-all as short-hand.

A project could define a manifest.ts file which contains the necessary information to compose a deno run or deno install command.

So a lengthy, complex command that a user may need to copy/paste or write out:

deno install --allow-net=deno.land,github.com,gitlab.com,raw.githubusercontent.com,x.nest.land --allow-read --allow-run https://deno.land/x/deno_run/deno_run.ts

becomes instead:

deno_run install https://deno.land/x/deno_run/

Which then simply prompts users to accept permissions.