Assemblyscript: First-class package management support

[vgrichina]

What is the best way to add package management support to AssemblyScript?

I think that using existing package structure created by asinit (i.e. root of sources in assembly within npm package) is pretty good and we can go with it.

However I don't think there is any support for actually using the package without specifying whole path.
E.g. let's say I use https://github.com/MaxGraey/bignum.wasm. My understanding is that currently I'll have to specify full path to the module I need, i.e. something like:

import u128 from '../node_modules/bignum.wasm/assembly/integer/u128.ts';

I want to be able to do something like:

import u128 from 'bignum.wasm';

What is the best way to achieve this? Is there any way to specify module search paths when calling compiler?

[jtenner]

See my library https://github.com/as2d/as2d.
The developer has to require it from a relative path and use the typescript source directly for now.

Also, they need to add an included glue.ts source file in their compilation.

[dcodeIO]

(i.e. root of sources in assembly within npm package

Agreed, that seems like something we can do by means of implementing actual include paths with a node-like ./node_modules, ../node_modules etc. traversal that skips directly to the assembly directory within an npm package. Maybe together with an asMain entry in package.json that indicates the actual main file, falling back to assembly/index if not specified.

[MaxGraey]

It will be great also support URL as path to source like deno does (may be in future)

[vgrichina]

It will be great also support URL as path to source like deno does (may be in future)

I think the best would be if module resolver is abstracted out. So that I can provide custom function which for a given module path / URL returns module contents.

[vgrichina]

@dcodeIO @MaxGraey is it possible to prioritize this?

Another related question:

1. We generate binding files with .near.ts extension for some of our .ts files (using forked assemblyscript).
2. Ideally I think we want to hook into import handling, so that e.g. if imported module name is model.near we can kick off our binding generation for module model.ts and import resulting file. What's the best way to do it?
3. Binding generation is currently hacked together in definition.ts on our assemblyscript fork. I wonder if it makes more sense to have pluggable ExportsWalker implementation (or some other way to plug this in). WDYT?

[MaxGraey]

@vgrichina AssemblyScript support --transform flag which provide access to full AST tree after parsing. So I don't see any reason for fork. See usage example for applyTransform and after that you could attach this like asc ... --transform ./transforms/zome_function.js

[vgrichina]

@MaxGraey nice, I didn't realize that's a thing.

What if there is any error during parsing. e.g. some of dependencies cannot be resolved?

[vgrichina]

@MaxGraey @dcodeIO

What if there is any error during parsing. e.g. some of dependencies cannot be resolved?

In particular I'm thinking of following:
1) When module is imported – resolve it a-la Node.js looking into node_modules folders (assuming assembly prefix).
2) If module is somehow marked as needing bindings generations (ideally just by using annotations on classes) – it is post-processed using our custom transform (i.e. what happens now in definitions.ts in our forked AssemblyScript) and only after that imported.

What's the best way to implement this? At what stage is --transform applied, can it replace module being imported? How to hook into module resolution process?

[dcodeIO]

What's the best way to implement this? At what stage is --transform applied

The idea of transforms is that you can hook into anything you'd like to, with the only hook implemented currently being afterParse. We can add more such hooks of course, like one that is called once the program has been created with all compile-time information being present, one that is called to intercept command line arguments, you name it.

can it replace module being imported? How to hook into module resolution process?

That should be possible using some sort of hook as well, like one that intercepts any imports or a fallback that is called if an import is not found, so it can be filled in by the transform.

I think we should start by implementing node-like module resolve and once that's done add a suitable set of hooks for your use case.

[MaxGraey]

@dcodeIO When we talked about module resolving last time we decide use assembly/index.ts in first stage with fallback to asmain value in package.json. Also possible do it via our own asconfig.json I guess

[jtenner]

We should make a decision that is forward thinking and gives us options later. Some kind of asconfig might be useful.

[vgrichina]

@dcodeIO @MaxGraey is this still blocked by some other issues?

[dcodeIO]

Not specifically. Initially I planned to focus on this one next, but then decided to tackle https://github.com/AssemblyScript/assemblyscript/pull/535 first which aims at getting a managed runtime up and running. Turned out runtime is quite a hard nut to crack, and it's taking longer than expected :( Sorry!

[MaxGraey]

@vgrichina I could deal with this issue, but after finished research about new better candidate for our sorting algorithm which use currently unstable approach for non-references and temporary slow fallback insertion-sort for reference elements which is not good.

[vgrichina]

@MaxGraey https://en.wikipedia.org/wiki/Timsort is pretty solid.

[MaxGraey]

@vgrichina I compared GrailSort (it's block sort family), Timsort, nearly optimal mergesort family (one of the latest research paper) and one of the new algorithm from Gregorius van den Hoven. And I can say that Timsort is not the fastest one and btw have some pitfalls and really complicated (have large size of code)

[MaxGraey]

@vgrichina
Early results for sorting random 1e6 numbers on js (node v11.12.0):

sort: 541.060ms (standard, stable since v8 7.0)
<new algo>: 157.619ms
TimSort: 226.613ms
QuadSort: 254.894ms
GrailSort: 296.994ms

So as we see TimSort is not fastest. I'll share my bench code after porting <new algo> to AS soon.

[vgrichina]

@MaxGraey which algorithm is fastest highly depends on what data is sorted. E.g. different algorithms win on partly pre-sorted vs randomly shuffled data. In most of real world use cases you don't deal with randomly shuffled data.

[MaxGraey]

yes, I know, so that's why I mentioned about early results. But 100% random shuffled array is usually hardest case. I'll add different bench cases. btw I really happy about all rest cases as well. <new algo> still better in that cases

[vgrichina]

Any update on this?

We have to go into ridiculous hacks without this feature:
https://github.com/nearprotocol/near-shell/blob/master/gulp-utils.js#L55

[willemneal]

I think the best way to handle this to add to the compiler that if a file isn't found to call out to a plugin that uses the same method that typescript does to resolve the dependency.

[vgrichina]

@willemneal yeah, probably even can use something like https://github.com/paulirish/node-resolve

[trusktr]

Personally I like putting my source inside the src folder; it's conventional in JS projects. I feel like it could be funky to see both a src/ folder and an assembly/ folder in a project with both TS and AS.

Everything outside of src is most JS projects is generally meta, infra settings, etc, related to managing the project, but the line isn't always clear as some configs are source code.

What I've been toying with so far is having src/as/ and src/ts/ folders to differentiate between the two types of TS. Stuff in src/ts/ might import portable code from src/as/. I'm not sure if I want to move some stuff out to src/common/ folder to make it obvious.

I'm currently trying out src/as/one, src/as/two, etc, with one and two being separate wasm modules in the same project.

I'm also not sure I want to keep src/as/ and src/ts/. I might try src/browser/, src/one/, src/two/, etc, where one and two may be separate AS modules, while browser is regular TS, etc.

It might make sense to have src/one/ts/ and src/one/as/, src/two/ts/ and src/two/as/, etc, where each ts folder contains glue code for the as stuff.

falling back to assembly/index if not specified.

What about falling back to index.ts in the root of the project, and if that doesn't build then throw?

Another options could be .as extensions to make thing unambiguous, but not sure tsc can handle other extensions (in order to have the benefits in IDEs).

[ExE-Boss]

Well, VS Code can be configured to have file type associations for unknown file extensions, which also adds IntelliSense to files with said extensions.

[dcodeIO]

The respective PR https://github.com/AssemblyScript/assemblyscript/pull/594 has been merged (thanks @willemneal !). The compiler will now search through node_modules/ automatically and allows to specify additional node_modules-like paths with the --path compiler option. It prefers an ascMain entry in package.json that works like node's main entry but for AS files, and otherwise falls back to the default assembly/ directory. Let us know if there are any issues!

[iwasaki-kenta]

The respective PR #594 has been merged (thanks @willemneal !). The compiler will now search through node_modules/ automatically and allows to specify additional node_modules-like paths with the --path compiler option. It prefers an ascMain entry in package.json that works like node's main entry but for AS files, and otherwise falls back to the default assembly/ directory. Let us know if there are any issues!

Tried it out here: https://github.com/perlin-network/smart-contract-as

Testing an example project, including smart-contract-as as a dependency gave me this error:

ERROR: Source file '~lib/smart-contract-as/index.ts' not found.

Example source code:

import {Parameters, Tag, Transfer, send_transaction} from "smart-contract-as";

export function _contract_init(): void {}

export function _contract_on_money_received(): void {
  const params = new Parameters();

  const tx = new Transfer(params.sender_id, params.amount / 2);
  send_transaction(Tag.TRANSFER, tx.marshal());
}

Might I be missing anything? The library's code is within a single file in assembly/index.ts.

[willemneal]

Hmm weird. I just cloned your repo and added a test project and ran the following:

asc assembly/index.ts --noEmit --traceResolution
Looking for '~lib/smart-contract-as' imported by '/Users/willem/c/tmp/smart-contract-as/test'
  in 'node_modules/assembly/smart-contract-as'
  -> 'node_modules/smart-contract-as/assembly/index.ts'

This was using my global asc, which is the most up to date one. I then tried using the assemblyscript in your main repo and got the same error. I then ran yarn upgrade, which upgraded to the newest version of assemblyscript and it worked!

[iwasaki-kenta]

Hmm weird. I just cloned your repo and added a test project and ran the following:

asc assembly/index.ts --noEmit --traceResolution
Looking for '~lib/smart-contract-as' imported by '/Users/willem/c/tmp/smart-contract-as/test'
  in 'node_modules/assembly/smart-contract-as'
  -> 'node_modules/smart-contract-as/assembly/index.ts'

This was using my global asc, which is the most up to date one. I then tried using the assemblyscript in your main repo and got the same error. I then ran yarn upgrade, which upgraded to the newest version of assemblyscript and it worked!

Thank you for that! Looks like getting the dependency by default via yarn add -D AssemblyScript/asssemblyscript gave me an older version than expected. Upgrading worked splendidly :smile:.