Typescript: test-262 integration

[mheiber]

Search Terms

test-262, test262

Suggestion

Run test-262 on TS downlevel

Use Cases

What do you want to use this for?

• Verify that downleveling is spec-compliant to avoid issues when changing the test target.

• Possible benefits for TS team:

  • Reduce triage overhead by catching semantic errors quickly
  • Reduce duplicate effort by putting creative testing work into the test-262 project

Examples

gulp runtests  # runs relevant test-262 tests

Checklist

• [x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
• [x] This wouldn't change the runtime behavior of existing JavaScript code
• [x] This could be implemented without emitting different JS based on the types of the expressions
• [x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
• [x] This feature would agree with the rest of TypeScript's Design Goals.

[weswigham]

We did that once, in the past - there's even a defunct test262 runner, and our internal rwc suite had some _very_ old test262 tests alongside it. It was neglected for a very long time and is now unused -from what I gather, the runtime was long and since it wasn't inspected regularly and baseline updated, wasn't particularly valued.

But, enough with the history: Our downlevel is occasionally very _intentionally_ non spec-compliant in some areas - for example, for of loops are downleveled to only handle arrays and not iterators as a whole (unless --downlevelIteration is opted in to).

[mheiber]

Thanks for the background.

My understanding is that it's possible to use test-262 while marking certain things as intentionally non-compliant.

Is that correct, @littledan ?

---

From: Wesley Wigham [e-mail]
Sent: Sunday, August 18, 2019 9:18:52 AM
To: microsoft/TypeScript [e-mail]
Cc: Max Heiber [e-mail]; Author [e-mail]
Subject: Re: [microsoft/TypeScript] test-262 integration (#32954)

We did that once, in the past - there's even a defunct test262 runner, and our internal rwc suite had some very old test262 tests alongside it. It was neglected for a very long time and is now unused -from what I gather, the runtime was long and since it wasn't inspected regularly and baseline updated, wasn't particularly valued.

But, enough with the history: Our downlevel is occasionally very intentionally non spec-compliant in some areas - for example, for of loops are downleveled to only handle arrays and not iterators as a whole (unless --downlevelIteration is opted in to).

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/TypeScript/issues/32954?email_source=notifications&email_token=ABDZJFKEQ3TLZI4TOMPIG7LQFEAWZA5CNFSM4IMQFHHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Q3ADQ#issuecomment-522301454, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABDZJFLMNBCF6BM4PO5SGQTQFEAWZANCNFSM4IMQFHHA.

[RyanCavanaugh]

I like this idea because we occasionally get the very reasonable request to document all the known differences in runtime behavior our downlevel could produce, and having a .json file or whatever that describes which test262 tests are OK to fail would be an easy accounting of that. We wouldn't even need to run this on a regular basis since we change downlevel emit so rarely.

[mheiber]

It sounds like a fun project for TS contributors.

---

From: Ryan Cavanaugh [e-mail]
Sent: Monday, August 19, 2019 5:16:04 PM
To: microsoft/TypeScript [e-mail]
Cc: Max Heiber [e-mail]; Author [e-mail]
Subject: Re: [microsoft/TypeScript] test-262 integration (#32954)

I like this idea because we occasionally get the very reasonable request to document all the known differences in runtime behavior our downlevel could produce, and having a .json file or whatever that describes which test262 tests are OK to fail would be an easy accounting of that. We wouldn't even need to run this on a regular basis since we change downlevel emit so rarely.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/TypeScript/issues/32954?email_source=notifications&email_token=ABDZJFPA5MOF47UCVYT5CSLQFLBMJA5CNFSM4IMQFHHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4TPTDY#issuecomment-522647951, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABDZJFIJ53RPG3HGN6LBOMLQFLBMJANCNFSM4IMQFHHA.

[weswigham]

Hm, aight. The complex thing we're missing in our runner is a definition of the runtime harness of required test262 assertions/helpers (with type definitions~). (And some code to execute every test262 test under that harness within a new context via node's vm library) It's fairly extensive nowadays, and part of the reason our old test262 output was so unused was because we never actually _executed_ the tests, just parsed and transpiled the containing js, so we have no progress on that front.

It's also worth noting that TS is going to throw semantic errors (eg, adding an object) for some code that is probably tested, so we'll need to log that difference, too - it's definely another bit of test output that's not captured in the test expectations.

[littledan]

Enumerating known failures (including different kinds of failures) is really typical for test262. I don't know of any project that passes all of them. This issue sounds like a great effort to me, though keep in mind that it will be a lot of work to maintain the known failures list.

[mheiber]

Would this issue be appropriate for a "Help Wanted" tag? Not sure what level of difficulty to assign to it.

[RyanCavanaugh]

'Backlog' is the new 'help wanted'; anyone can pitch in if they like. I think this is reasonably straightforward as long as you're familiar with our test setup + test262 itself.

[jbhoosreddy]

Hey so, we had a catch that involved @DanielRosenwasser, @RyanCavanaugh, @mheiber, @robpalme, @joeywatts and others (sorry I couldn't find github usernames).

We discussed taking on this particular issue for external contributions as discussed with @DanielRosenwasser and @RyanCavanaugh. In particular, @mkubilayk and I would like to work on it based on a solution I worked on for babel/babel using babel-test262-runner along with babel maintainers @nicolo-ribaudo and @JLHwung.

We now already have a similar test262 runner for this which transpiles TypeScript code and outputs results. The interesting part here would be to integrate with existing CI infrastructure.

They recommended @sandersn and @weswigham to be point of contact for help guiding this work. Thanks!

[weswigham]

There's an antiquated test262 runner in the testRunner project that could be re-purposed. The interesting thing, IMO, is keeping the test-262 tests up-to-date with an everchanging test-262. It makes me think it might even be better suited as, like, an npm install script on a repo we reference for a user test, or a docker test.

[nicolo-ribaudo]

The problem with auto-updates is that, after any update, new tests will probably start failing and you don't know if it is expected or not. In Babel, we have hard-coded a commit sha of the test262 version we want to test, and every few months we manually update it.

[weswigham]

Ick, big-bang reintegrations? I'd much rather flag breaks as they come in and then baseline them as "dealt with", like we do for "docker" suite tests.

[littledan]

Very happy to see this work moving forward. Sounds like @weswigham 's idea is a kind of automated version of @nicolo-ribaudo 's. I think they can both work, and I've seen both patterns used in browsers. The important thing is that the tree doesn't suddenly become red when test262 adds a new test, but that the failure is visible if there was a regression caused by a TypeScript change; I think you can achieve this through either approach.

[mkubilayk]

Hey @weswigham - we have a test262 runner for TypeScript that @jbhoosreddy published at https://github.com/jbhoosreddy/TypeScript/tree/typescript-test262-runner.

$ npm i
$ node lib/download-node
$ node lib/run-tests arrow-function

should run a subset of tests transpiled by TypeScript 3 on node 0.1.0 (which doesn't have newer language features). We would like to get this integrated with your workflow and CI.

I see defunct test runner you have for test262 doesn't actually run the code but we can have something similar that calls out the this runner and collects the results. We will need some hand-holding to figure out the CI pipelines, though.