Jest: Implement async code transformations

[SimenB]

EcmaScript Modules can (actually must, but easy to wrap the sync stuff in a promise) resolve asynchronously. While it's in no way a blocker for ESM support in Jest, it would be nice to add support for async transforms to be used.

This logic should for now not impact anything in Jest beyond @jest/transform as Jest wouldn't use it until ESM support is in place.

I'm thinking we can add a .transformAsync and .transformSourceAsync to the ScriptTransformer class.

There are some checks and verifications that process is a function - we need to make sure there is one clear code path for sync and one for async transformation that verify the provided reporter implements the correct interface.

Sort of reopening #5556 and all its linked issues.

[the-spyke]

@SimenB I was looking into babel-jest because it can't load ESM config files: Babel demands to load them asynchronously. And looks like not only Transformer.process() needs to be async, but Transformer.getCacheKey() too.

[SimenB]

Ah, to load Babel's config? That makes sense.

[ahnpnl]

Does this affect to current custom transformers’ implementation ? I’m curious about the performance can be gained with async

[the-spyke]

ScriptTransformer also should load transformers with import() to support ESM transformers (just bumped into this).

[SimenB]

@ahnpnl not unless they want to, it'll be opt-in. If the transformer doesn't export async functions, we'll fallback to the synchronous ones.

@the-spyke good point! I'm starting to wonder if we should export both SyncScriptTransformer (the one we have today) and AsyncScriptTransformer. I started work on this in #9597, and it already feels a bit hairy trying to juggle sync vs async. Might be cleaner with two separate classes (who could both extend from some base class with the base of implementation, or import shared helpers)

[ahnpnl]

I did some experiment yesterday with jest-worker, end up I noticed I had to wait for this. This can help a lot for ts-jest to separate type checking and compiling.

[ychi]

I am interested in this, can I help?

[SimenB]

Yes please! It's somewhat invasive, but I'll be happy to work on it with you.

I haven't figured out if it makes the most sense to add transformAsync and tranformSourceAsync to ScriptTransformer class, or if an AsyncScriptTransformer class should be created. Probably the former?

Regardless, these functions should then call transformer.processAsync and transformer.getCacheKeyAsync if they exist, falling back to the synchronous methods.

Let's start with just adding support to @jest/transform somehow, then separately tackle calling this new API from the ESM parts of jest-runtime and adding support to babel-jest

[ychi]

Thank you, @SimenB!

Just so I understand it correctly:
There are 3 major steps of ES module instance graph building: construction, instantiation and evaluation, and each step can be asynchronous. Transform happens during construction phase, after fetching the file; while parsing the file, it will be transformed/transpiled.

We want to enable transformation itself to be async, that means when client calls ScriptTransformer.transformAsync() or ScriptTransformer.transformSourceAsync(), promises will be returned. If processAsync and getCacheKeyAsync of Transformer are not in place, resolved promise will be returned.

No matter we are going transformAsync or AsyncScriptTransformer, I think we will need an AsyncTransformer that extends Transformer interface? 🙂

[SimenB]

Just so I understand it correctly:

yup, that summary sounds correct.

No matter we are going transformAsync or AsyncScriptTransformer, I think we will need an AsyncTransformer that extends Transformer interface? 🙂

I'm not sure if we should force transformers to work in sync mode - there might be cases where a transformer won't work synchronously, and we shouldn't ban that use case.

So I'm thinking 2 separate interfaces, one with optional async methods, and one with optional sync methods.

[ahnpnl]

maybe my question here won't be related to this feature but is it possible to have a post process hook in ScriptTransformer which is called once all the files have been transformed ?

[ychi]

So I'm thinking 2 separate interfaces, one with optional async methods, and one with optional sync methods.

Do you mean something like the following?

interface STransformer {
  ...
  process: (
    ...
  ) => TransformedSource;
  processAsync?: (
    ...
  ) => Promise<TransformedSource>;
}

interface ATransformer {
  ...
  process?: (
    ...
  ) => TransformedSource;
  processAsync: (
    ...
  ) => Promise<TransformedSource>;
}

And transformAsync() always check if processAsync is there, if not, it will use process and wrap the result in a promise. But it looks like STransformer will just do the job?

maybe my question here won't be related to this feature but is it possible to have a post process hook in ScriptTransformer which is called once all the files have been transformed ?

transform() only handle one file at a time, maybe there is a client of ScriptTransformer that calls ScriptTransformer.transform() for each file?

[SimenB]

maybe my question here won't be related to this feature but is it possible to have a post process hook in ScriptTransformer which is called once all the files have been transformed ?

There is no point at which "all files have been transformed" as there might be import() or require() calls all over the place, even as the last thing that happens in a test file.

---

Do you mean something like the following?

Yup!

[ahnpnl]

Is there a way for transformer to access some sort of “after test run finish” hook ? I know there are some hooks which are available for watch plugin but transformer doesn’t have.

[SimenB]

Not really at the moment. You could implement it as a reporter, but I'm guessing you want to access some state or some such... Transformers can run in different workers though, so even that might not work.

Regardless, not really related to this issue. Could you open up a new one, and describe your use case for it?