Vscode: Unit testing extensions is quite difficult

[eamodio]

Refs: #93604

Maybe we could provide a mock library for certain vs code types, and guidance (docs, examples, blogs, etc) on how it can be done effectively.

/cc @jrieken

[jrieken]

Adding @octref since you have been making good work here

[bfitzpat]

Our QE team at Red Hat has been doing great work with the https://github.com/redhat-developer/vscode-extension-tester if anybody wants to take a look at it.

[octref]

@bfitzpat Thanks for mentioning. I'll take a look. Do you have specific issues you ran into? What specifics you were trying to test that you cannot test using our integration test?

@eamodio There's nothing actionable. Please be more precise. What's the issue they run into?

[eamodio]

@octref the request here is about pure unit-tests -- not integration or UI driven. Basically providing anything we can do to make it easier as well as guidance for best practices.

@Almenon Can you provide any more specific details here?

[octref]

Prescribing one pattern excludes others. I don't think it's good to include "how you can write your extension so it's unit-testable".
If there's anything specific to vscode extension that makes unit testing hard I can address it. But if it's just "split your code into parts where some requires vscode API the others can be unit-testable", I don't see much value advising everyone doing that.

[Almenon]

This originates from https://github.com/microsoft/vscode/issues/82471. I never got around to responding and the issue got locked, sorry. Rearchitecturing my code to decouple it from vscode would be difficult, or at least take a while. I tried mocking but that was nightmarish. Getting the mocks to behave like vscode in some cases basically required rewriting vscode logic.

Maybe if I tried a blend of all three approaches - decoupling what can easily be decoupled for easy unit testing, mocking that which is lightly coupled, and integration testing that which is heavily coupled, maybe that would work effectively. But I would strongly prefer plain old unit testing if it was possible.

Is there any technical solution that might solve #6586 to let vscode be imported during unit tests? Maybe there's some way it might be achieved (or maybe it fundamentally goes against the core architecture, idk)

If there's no possible way it can be realistically done then the only action item I can think of would be adding documentation / code samples. See https://github.com/microsoft/vscode-extension-samples/issues/91

[tamuratak]

To my experience with LaTeX Workshop, the following steps are an easy way to write unit tests.

1. Export objects for which you want to write unit tests as a return value of activate function.
2. In test codes, you can obtain the exported objects with vscode.extensions.getExtension.
3. Write unit tests for the objects.
4. If you do not want to export the objects in the release, you can use an environment variable to enable it only on tests.

I do not think this is an appropriate way. However, it is super easy.

[Almenon]

@tamuratak thanks, but I believe those are integration tests - they run with a live vscode instance and it looks like you run them through vscode-test so you can't choose a individual test to run. I'm talking about unit tests - the kind where you could run individual tests through the test explorer sidepanel.

Also just curious but why are you exporting objects that way? In arepl I just import the objects for my integration tests directly.

[Almenon]

While I was writing my above comment I realized that there are some more possibilities here. If there's a way to run vscode in headless mode the tests could be run faster and without the UI popping up. Unfortunately electron doesn't support headless mode yet 😢 (even though chrome supports it which is kinda weird).

Another thing that would be nice is if you could run tests individually through the test explorer.

[iliazeus]

@eamodio

To make unit testing easier, it would be nice to have certain classes and enums from the extension API to be available without VSCode itself.

Like, for example, Ranges and Locations are nice to use internally, not only when interfacing with VSCode, but we can't unit test that code in a generic nodejs environment.

This has already been done for Uris with the vscode-uri package.

[dcermak]

Of the top of my head I can think of the following pain points when trying to write unit tests:

• If you want to check the results of an Event firing, you have to somehow wait for the event to resolve and all listeners to be processed. I've come up with these hacky helper functions for that:

export async function waitForEvent<T>(
  event: vscode.Event<T>
): Promise<vscode.Disposable> {
  return new Promise((resolve) => {
    const disposable = event((_) => {
      resolve(disposable);
    });
  });
}

export async function executeAndWaitForEvent<T, ET>(
  func: () => Thenable<T>,
  event: vscode.Event<ET>
): Promise<T> {
  const [res, disposable] = await Promise.all([func(), waitForEvent(event)]);
  disposable.dispose();
  return res;
}

They are unfortunately only doing the "right thing" if none of your listeners is async. Once one of them is asynchronous, it will not wait for it to complete (I guess this might be actually a limitation in vscode).

• A convenient mock of the ExtensionContext would be nice to have: something that creates a completely isolated environment (no shared globalState, a new home directory, maybe even a mocked filesystem via mock-fs)

• A good sample how to get code coverage out of unit tests. I have managed to get it to work with nyc but that required a lot of searching around, trying things out and eventually I took the code from @connor4312 (see https://github.com/microsoft/vscode-docs/issues/1096#issuecomment-545630839).

[connor4312]

Since I was mentioned, just dropping one more rather annoying case I had to test recently: quickpicks

• I was able to stub the vscode.window.createQuickPick with Sinon easily enough
• With that I can set the selectedItems, and I dug around and found the workbench.action.acceptSelectedQuickOpenItem command I can run to accept those items
• However, it seems that setting items is not synchronous or ordered relative to running the command. I had to put a manual delay/timeout between setting the selectedItems and running the command
• But that was still flakey. Eventually I made this mock setup which seemed to work fairly reliably:

createQuickPick = stub(vscode.window, 'createQuickPick').callsFake(() => {
  const picker = original();
  acceptQuickPick = new EventEmitter<void>();
  stub(picker, 'onDidAccept').callsFake(acceptQuickPick.event);
  return picker;
});

Then in my tests, run a command and wait for the picker to be created:

vscode.commands.executeCommand(Contributions.StartProfileCommand, session.id);

const typePicker = await eventuallyOk(() => { // test helper that retries unti it doesn't throw
  expect(createQuickPick.callCount).to.equal(1);
  const picker: vscode.QuickPick<vscode.QuickPickItem> = createQuickPick.getCall(0).returnValue;
  expect(picker.items).to.not.be.empty; // needed since here we set items asynchronously
  return picker;
}, 2000);

typePicker.selectedItems = typePicker.items.filter(i => /CPU/i.test(i.label));
acceptQuickPick.fire();

[dcermak]

I've been using Sinon stubs to test QuickPicks roughly like this:

this.fixture.vscodeWindow.showQuickPick.onCall(0).resolves(userInput);
await myProvider.cmdFunc(element).should.be.fulfilled;
this.fixture.sandbox.assert.calledOnce(
  this.fixture.vscodeWindow.showQuickPick
);
this.fixture.sandbox.assert.calledWithMatch(
  this.fixture.vscodeWindow.showQuickPick.firstCall,
  [option1, option2, option3]
);

That will however probably break when you try to use it via commands.

Given the amount of setup code this requires, I am currently migrating most of these tests to vscode-extension-tester, which is perfect for testing UI elements (albeit a bit flaky from time to time).

[connor4312]

Yea showQuickPick is easier to test. createQuickPick is the fancier object-based one, e.g: https://github.com/microsoft/vscode-js-debug/blob/4b5cba447468cd60e949dd2d089b3608e38d5dc1/src/ui/profiling/breakpointTerminationCondition.ts#L28-L43

[grconrad]

Thanks @Almenon for raising the issue. Agree there is too much friction right now. I've only created simple stuff so far, but ran into the exact issue you described where nothing in my extensions (or libraries) was unit testable because everything depended directly or indirectly on vscode. Eventually I had some success with the following:

1. identify those parts of the API surface my library is using
2. in entry point, pluck what i use from vscode into my own object
3. pass that object down to function calls defined in lower-level modules
4. unit-testing those lower-level modules by making mock objects in the tests

With this strategy only the entry point (index.js in my library, but could be extension.js for an extension) imports vscode so the other modules are unit testable. I was able to get 90% code coverage that way. But it feels dirty and would not scale well in logic of more complexity.

[dcermak]

What might help (at least it would help me) would be a "test/mock mode" for vscode or some other library that you'd inject via dependency injection. This would allow you to setup expected user input and give you convenience functions to check that certain things happened.
For example:

it("inputBox rejects a number", async () => {
  const someObj = new Something();
  vscodeTest.setUserInput(InputBox, "123");
  await someObj.askForName();
  vscodeTest.assertError(InputBox, "Invalid name: it contains only numbers");
});

The API that I've written down is of course terrible, but something along those lines would be certainly helpful although it overlaps a bit with vscode-extension-tester.

[Lennon925]

Refs: #93604

Maybe we could provide a mock library for certain vs code types, and guidance (docs, examples, blogs, etc) on how it can be done effectively.

/cc @jrieken

Hi, Any way to get extentionContext in unit test?

[DonJayamanne]

Today in the Python extension we mock most of the VS Code API.

We (Python Extension) find that to be very simple, however it could be a lot simpler if we were using SinonJs.
I have also used sinonjs (and mocked VS Code) in another extension, see here https://github.com/DonJayamanne/gitHistoryVSCode/tree/master/test/extension.

As for full blow UI tests, the Python extension did go down this route. Basically we used the VS code Smoke Tests infrastructure and built on top of that to run full blown UI tests. However we stopped using that simply because we didn't see much value in it (i.e. it didn't get used much).

If anyone is interested in the UI tests we had, it can be found in this PR (where it was remove).
https://github.com/microsoft/vscode-python/pull/10215/files
We used puppeteer + BDD (cucumberjs) for our tests.