Deno: snapshot testing

[brandonkal]

A nice to have feature would be Jest-like snapshot testing in the std/testing library.

[ry]

I don't know what that means... can you explain more?

[bartlomieju]

Ref https://github.com/denoland/deno_std/issues/346

[brandonkal]

@ry Basically when you call expect(x).toMatchSnapshot() it serializes whatever x is and stores it in a state file. Here's an example.

The premise is that you take a snapshot of the state when things are how they should be, and if they later differ, the test fails. You also have the option to update the snapshot if you desire the change or to define a custom serializer.

[brandonkal]

Also the option to run a specific test by name (filter) would be nice. Failfast marks all other tests as failed which is confusing. I'm finding myself commenting tests out to iterate on broken ones.

[KSXGitHub]

What do you think the API should look like?

I propose this:

import { matchSnapshot } from 'https://deno.land/std/testing/snapshot.ts'
matchSnapshot('filename.snapshot', myObject)

deno test # would fail if snapshot is outdated
deno test -u # would update snapshot if it is outdated

[brandonkal]

I would propose the API be closer to Jest's .toMatchSnapshot(). So there is no need to specify where to store the snapshot.

A similar structure would be implied

my_test.ts
__snapshots__
|- my_test.ts.denosnap

[KSXGitHub]

How would a std lib knows where the test file or snapshot file is located? The only way to know is for the test file to provide import.meta.url, which is just snapshot location with extra steps.

[brandonkal]

import.meta.url gives you the filepath of the test file. From there you easily convert it to containing folder + __snapshots__/filename.denosnap

You could also do something like:

import { makeSnapshotter } from 'https://deno.land/std/testing/snapshot.ts'
const snap = makeSnapshotter('./__snapshots__')
snap(myObject)

[KSXGitHub]

@brandonkal That's what I said: "snapshot location with extra steps".

import.meta.url gives you the filepath of the test file

import.meta.url only gives filepath of the test file if it was the test file that invoke it. Within an /std lib, it will gives you a URL starts with https://deno.land/std. The library can only get import.meta.url of the test file if the test file give it, which is equivalent to giving location of the snapshot file.

---

Both my design (the test file give location of the snapshot file) and yours (the test file give import.meta.url) have a limitation: The test file may decide to give a URL that does not match its location, making detection of redundant snapshot files impossible.

I think we better come up with a better design which may allow testing lib to determine snapshot file location without help from the test file, but it would require change in Deno core:

• deno test runs every test file in isolation from each other, with separate globalThis.
• Every isolate has a way to tell testing lib location of the test file, be it window.location, Deno.scriptLocation(), Deno.env().DENO_TEST_LOCATION, or what have you.

[nayeemrmn]

Just create explicit paths before you pass to std.

matchSnapshot(new URL(import.meta.url, "testdata/filename.denosnap").pathname, myObject);
// or
matchSnapshot("__snapshots__/filename.denosnap", myObject); // from cwd

Implicitly adding a directory (other than cwd) offers little control and resolving from main module seems like bad practice.

[KSXGitHub]

@nayeemrmn What you proposed was my idea. However, as I also wrote in https://github.com/denoland/deno/issues/3635#issuecomment-599023737, it has limitation: Deletion of redundant snapshot file is impossible (whereas in Jest, detecting outdated snapshot file is easy, because test file cannot control where to write snapshot to).

[brandonkal]

The point is you are not supposed to have control on where the snapshot is stored. It's an implementation detail. All files have a unique import.meta.url. From there all you need is a function which maps a import.meta.url to its snapshot storage folder. It is up to the testing library where to store the snapshot. Typically it is right next to the test file so it can be committed in git.

matchSnapshot("__snapshots__/filename.denosnap", myObject)

That is verbose and gets messy quickly.

[KSXGitHub]

@brandonkal Are we just repeating ourselves now? Yes, all import.meta.url are unique, but test file can choose to pass an arbitrary string instead of import.meta.url. How could a testing lib know if a string is truly import.meta.url or not?

If you so value the lack of control of snapshot location (just like I do), then I beg you to focus the discussion on https://github.com/denoland/deno/issues/3635#issuecomment-599023737 (the second part)

[brandonkal]

I see what you are saying now. That would be a good way to solve the issue.

Just spitballing here but you could also solve this by having each test file could look like this:

import { makeSnapshotter } from 'https://deno.land/std/testing/snapshot.ts'

export default function run(name) {
  const snap = makeSnapshotter(name)
  /// The tests
  snap(myObject)
}

if (import.meta.main) run(import.meta.main)

Then the .deno.test.ts file that is generated changes from:

import "file:///the_test.ts"

to

import test1 from "file:///the_test.ts"
test1("file:///the_test.ts")

It's a bit more explicit but you are not injecting magic globals into the test environment. Though I wouldn't mind that either.

[nayeemrmn]

It's an implementation detail.

But we go as far as to commit them to git, I kind of don't buy it as an implementation detail nor does automatic deletion sound so important... anyway I didn't know anything about snapshot testing before yesterday so I'll back out :p but the more I've learnt about them the more they seem like a crap jest thing. Ignore me.

[KSXGitHub]

@nayeemrmn

You seem to dislike snapshot. Yet even Deno itself uses something similar to snapshots (cli/tests/*.out).

[KSXGitHub]

Anyway, snapshot testing may or may not be in the standard library, either way is fine to me. But I think isolating every test from each other is a good idea in general, wouldn't you agree? With tests being isolated, it would then be possible for others to create a snapshot testing library that does not ask test file for import.meta.url.

[nayeemrmn]

similar to snapshots (cli/tests/*.out).

Hardly!

[kitsonk]

Yeah, that is like saying any assertion against a string is like snapshotting. I don't think it is a good comparison at all.

I haven't voiced an opinion yet, because I am very well aware of snapshotting, but have never seen it deliver value. Developers simply rebaseline when something unexpected happens. That are, in my opinion, a false security blanket.

It certainly feels like something that should start in the wider community versus being in std in my opionion.

[bartlomieju]

It certainly feels like something that should start in the wider community versus being in std in my opionion.

I completely agree with @kitsonk on this one; it should start as third-party lib instead of std/ lib. Only then if it's very desirable we can think about introducing it into std/.

I'm gonna close this issue now, but feel free to revisit if such library surfaces!