jest 🚀 - New shape matcher proposal

@Spy-Seth Thank you for proposing match shape. Good idea to make it a new issue.

Can you give some examples of JSX that are incorrect for the assertion?

Can you confirm if these examples are correct for the assertion, or not:

“extra” props, for example <Bar who="John" what="person" />
Foo does not need to be root of rendered tree, for example <App><Foo><Bar who="John" /></Foo></App>
Bar can contain children? for example, <Foo><Bar who="John"><X>…</X></Bar></Foo>
Foo can contain other children? for example, <Foo><Bar who="John" /><X>…</X></Foo>

pedrottimark on 1 Dec 2016

I wonder if this could be done using expect.any(…) which we are adding soon. See https://github.com/facebookincubator/create-react-app/pull/1127#issuecomment-264290075

cpojer on 1 Dec 2016

@cpojer Interesting, a quick test suggests that Jest .toEqual already supports:

https://jasmine.github.io/edge/introduction#section-Custom_asymmetric_equality_tester

Although I am not suggesting solutions yet, is that relevant to the use case in this issue?

pedrottimark on 1 Dec 2016

Thanks for taking time for discutting this.

“extra” props, for example <Bar who="John" what="person" />

Extra props should be allowed by default. If needed it could exist an option/syntax that will be more strict.

Foo does not need to be root of rendered tree

Exactly, as we match a shape no warranty on the exact level on the DOM.

Bar can contain children? for example, …
Foo can contain other children? for example, …

Good questions! As we allow props, I guess that we have to allow sibling and child.

Here come incorrect vdom for the previously proposed shape (<Foo><Bar who="John" /></Foo>):

// No <Bar> component
<Foo />

// <Foo> in not a parent of <Bar>
<Bar who="John" /> 

// Or:
<div>
  <Foo />
  <Bar who="John" />
</div>

// No props `who`
<Foo>
  <Baz />
</Foo>

// Wrong value for props `who`
<Foo>
  <Bar who="Michael" />
</Foo>

Spy-Seth on 2 Dec 2016

👍 Love the idea, in particular I see a use case when we want to ensure that a HOC (which is tested separately with snaps) effectively decorates another component.

shouze on 2 Dec 2016

Just for fun, tried a minimal shape match for deep renderer from react-test-renderer by copying the code of compare helper in toMatchObject assertion (merged in https://github.com/facebook/jest/pull/1843 but not released in Jest at time of this comment).

As you would expect, an object subset match deals with extra props:

import React from 'react';
import renderer from 'react-test-renderer';

const Foo = ({children, ...rest}) => (<div {...rest}>{children}</div>);
const Bar = ({children, ...rest}) => (<span {...rest}>{children}</span>);

describe('object subset', () => {
  const pattern = renderer.create(<Foo><Bar who="John" /></Foo>).toJSON();

  it('does match extra props', () => {
    const tree = renderer.create(<Foo><Bar who="John" what="person" /></Foo>).toJSON();
    //expect(tree).toMatchObject(pattern);
    expect(compare(pattern, tree)).toBe(true);
  });

As you might expect, an object subset match does not automatically deal with:

pattern as descendant instead of root of tree
any subset matching of the children prop, including: siblings of nodes or children of leaves

pedrottimark on 2 Dec 2016

Too much fun, so went a bit farther. Extended a copy of compare with the following:

any function matches any other function
expected children match adjacent subsequence of received children: option?
no children in expected leaf node match children in corresponding received node: option?
find expected subtree within received tree: option?

Original code is 42 lines, extended code is 73 lines with naive traversal algorithms.

Given expected subtree, matcher returns object with asymmetricMatch method: https://jasmine.github.io/edge/introduction#section-Custom_asymmetric_equality_tester

EDIT: Forgot to say that error output for asymmetricMatch is not helpful :(

import React from 'react';
import renderer from 'react-test-renderer';

const A1 = ({children, ...rest}) => (<article {...rest}>{children}</article>);
const A2 = ({children, ...rest}) => (<section {...rest}>{children}</section>);
const P = ({children, ...rest}) => (<div {...rest}>{children}</div>);
// Components render flow elements above and phrasing elements below
const C = ({children, ...rest}) => (<span {...rest}>{children}</span>);
const D1 = ({children, ...rest}) => (<strong {...rest}>{children}</strong>);
const D2 = ({children, ...rest}) => (<em {...rest}>{children}</em>);

describe('function props', () => {
  it('match too exactly in default object subset match', () => {
    const subtree = renderer.create(<P><C who="John" how={() => {}} /></P>).toJSON();
    const tree = renderer.create(<P><C who="John" how={() => {}} /></P>).toJSON();
    //expect(tree).not.toMatchObject(subtree);
    expect(compare(subtree, tree)).toBe(false);
  });
  it('must match as equivalent in a useful subtree match', () => {
    const treeFalse = renderer.create(<P><C who="John" how={() => false} /></P>).toJSON();
    const treeTrue = renderer.create(<P><C who="John" how={() => true} /></P>).toJSON();
    expect(treeFalse).toEqual(matcher(treeTrue));
  });
});

describe('subtree', () => {
  const subtreeDefault = renderer.create(<P><C who="John" /></P>).toJSON();

  it('does match extra props in parent and child', () => {
    const tree = renderer.create(<P who="Marie"><C who="John" what="person" /></P>).toJSON();
    expect(tree).toEqual(matcher(subtreeDefault));
  });
  it('does not match missing prop in child', () => {
    const tree = renderer.create(<P><C /></P>).toJSON();
    expect(tree).not.toEqual(matcher(subtreeDefault));
  });
  it('does not match different prop in child', () => {
    const tree = renderer.create(<P><C who="Michael" /></P>).toJSON();
    expect(tree).not.toEqual(matcher(subtreeDefault));
  });
  it('does not match missing child', () => {
    const tree = renderer.create(<P />).toJSON();
    expect(tree).not.toEqual(matcher(subtreeDefault));
  });
  it('does not match missing parent', () => {
    const tree = renderer.create(<C who="John" />).toJSON();
    expect(tree).not.toEqual(matcher(subtreeDefault));
  });
  it('does not match missing missing child but another occurs', () => {
    const tree = renderer.create(<P><D1 /></P>).toJSON();
    expect(tree).not.toEqual(matcher(subtreeDefault));
  });
  it('does not match missing parent which occurs as sibling', () => {
    const tree = renderer.create(<A1><P /><C who="John" /></A1>).toJSON();
    expect(tree).not.toEqual(matcher(subtreeDefault));
  });
  it('does not match separated parent and child', () => {
    const tree = renderer.create(<P><D1><C who="John" /></D1></P>).toJSON();
    expect(tree).not.toEqual(matcher(subtreeDefault));
  });
  it('does match parent as descendant in tree', () => {
    const tree1 = renderer.create(<A1><P><C who="John" /></P></A1>).toJSON();
    expect(tree1).toEqual(matcher(subtreeDefault));
    const tree2 = renderer.create(<A1><A2>preceding</A2><A2><P><C who="John" /></P></A2><A2>following</A2></A1>).toJSON();
    expect(tree2).toEqual(matcher(subtreeDefault));
  });
  it('can match child which has children', () => {
    const tree = renderer.create(<P><C who="John"><D1>grandchild</D1></C></P>).toJSON();
    expect(tree).toEqual(matcher(subtreeDefault));
  });
  it('can match child which has preceding sibling', () => {
    const tree = renderer.create(<P><D1>preceding</D1><C who="John" /></P>).toJSON();
    expect(tree).toEqual(matcher(subtreeDefault));
  });
  it('can match child which has following sibling', () => {
    const tree = renderer.create(<P><C who="John" /><D2>following</D2></P>).toJSON();
    expect(tree).toEqual(matcher(subtreeDefault));
  });
  it('can match adjacent children which have siblings', () => {
    const subtreeDefault = renderer.create(<P><C who="Paul" /><C who="Mary" /></P>).toJSON();
    const tree = renderer.create(<P><C who="Peter" /><C who="Paul" /><C who="Mary" /><C who="John" /></P>).toJSON();
    expect(tree).toEqual(matcher(subtreeDefault));
  });
});

pedrottimark on 2 Dec 2016

I'm following this here (a bit busy with other stuff) and like what you guys are coming up with. Happy to review PRs for Jest if that makes sense.

cpojer on 2 Dec 2016

@pedrottimark

EDIT: Forgot to say that error output for asymmetricMatch is not helpful :(

Created a special issue https://github.com/facebook/jest/issues/2215 about making this better to read.

just-boris on 4 Dec 2016

👍1

Thanks you @pedrottimark for taking so many time to play around this! 👍

I'm not entering too much in the implementation discussions as I do not manage any of the internals details of jest or react.

Spy-Seth on 6 Dec 2016

Selecting or traversing has simmered in my mental slow cooker for several days after reading:

https://github.com/facebook/jest/issues/2197#issuecomment-264078856
https://github.com/facebook/jest/issues/2197#issuecomment-264664134 which lead to https://github.com/adriantoine/enzyme-to-json#focused-tests

Now it seems like I mistakenly mentioned matching flexibility not implied in the original comment.

If we assume that you select or traverse in the tree to the relevant subtree for expect(…), then the soon-to-come .toMatchObject(shape) assertion seems like a candidate for this use case.

Following up on https://github.com/facebook/jest/pull/2213#issuecomment-264733764 I would welcome any thoughts from @just-boris or examples from experience with jasmine.objectContaining which sounds similar to .toMatchObject

pedrottimark on 8 Dec 2016

The .toMatchObject assertion can match a subset of the props of rendered DOM nodes:

more declarative than prop assertions
less automated than snapshots, but you might edit a copy of a “checksum” snapshot (which tests for all changes) to delete irrelevant details for an interaction test from the JSX and then paste into .toMatchObject(renderToObject(…)) see below

Examples are for the following components:

1. content without functions

Here is a better solution than I suggested in https://github.com/facebook/jest/issues/2197#issue-192688463

describe('Todo', () => {
  it('renders text and uncompleted style', () => {
    const text = 'Install Jest';
    const completed = false;
    const textDecoration = 'none';

    // react-test-renderer
    expected = renderer.create((<li style={{textDecoration}}>{text}</li>).toJSON();
    received = renderer.create(<Todo onClick={onClick} text={text} completed={completed} />).toJSON();
/*
    // or Enzyme shallow rendering (for this component, it renders only DOM elements)
    received = enzymeToJSON(shallow(<Todo onClick={onClick} text={text} completed={completed} />));
    // or Enzyme full DOM rendering
    received = mountedToObject(mount(<Todo onClick={onClick} text={text} completed={completed} />));
*/
    expect(received).toMatchObject(expected); // expected omits onClick
  });
});

helper functions

We need these now. Don’t take too seriously (not recommending as the future design :)

mountedToObject returns just DOM elements, because enzymeToJSON(mount(…)) returns composite tree of React component and DOM elements. I will ask maintainer about current behavior. It requires a helper here and might make it hard to refactor child components. EDIT: Found https://github.com/adriantoine/enzyme-to-json/issues/9
renderToObject is a shortcut for renderer.create(…).toJSON() in the rest of the examples.
literalToObject adds $$typeof to {type, props, children} elements (like react-test-renderer). Unlike React.createElement or JSX, an object literal can omit children prop in elements of expected object if you need to ignore descendants in received object.

RE-EDIT: in the following https://github.com/facebook/jest/issues/2202#issuecomment-268347718

mountedDomToObject supersedes mountedToObject
renderAsRelevantObject with JSX markup supersedes literalToObject

2. result of interaction

describe('TodoList', () => {
  it('changes completed style' => {
    const state0 = [todo0, todo1, todo2];
    const index = 1;
    const {text} = state0[index];
    const textDecoration = 'line-through';

    let state = state0;
    const onTodoClick = jest.fn((index) => { state = todos(state, toggleTodo(index)); });

    const wrapper = mount(
      <TodoList
        onTodoClick={onTodoClick}
        todos={state0}
      />
    );
    const li = wrapper.find('li').at(index);
    li.simulate('click');
    expect(onTodoClick).toHaveBeenCalledTimes(1);
    expect(onTodoClick).toHaveBeenCalledWith(index);

    // simulate re-render because props changed
    wrapper.setProps({todos: state});

    // ordinary assertion via Enzyme
    expect(li.prop('style').textDecoration).toEqual(textDecoration);

    // 3 alternative ways to declare the assertion
    expect(mountedToObject(li)).toMatchObject(renderToObject(
      <li style={{textDecoration}}>{text}</li>
    )); // JSX can omit onClick but not text, if received has text
    expect(mountedToObject(li)).toMatchObject(renderToObject(
      React.createElement('li', {style: {textDecoration}}, text)
    )); // React.createElement can omit onClick but not text, same song, second verse
    expect(mountedToObject(li)).toMatchObject(literalToObject({
      type: 'li',
      props: {style: {textDecoration}},
    })); // object literal can omit onClick and text
  });
});

A more convincing example when to write a declarative subset assertion would have:

multiple relevant changes: to show why not write ordinary assertions
multiple irrelevant details: to show why not take a snapshot, even a diff https://github.com/facebook/jest/issues/2197#issuecomment-266059285

Here are some assertions just to illustrate how the matching works:

expect(mountedToObject(li)).not.toMatchObject(renderToObject(
  <li style={{textDecoration}}></li>
)); // JSX cannot omit text from expected, if received has text

expect(mountedToObject(li)).toMatchObject(renderToObject(
  React.createElement('li', null, text)
)); // JSX or React.createElement can omit all props from expected
expect(mountedToObject(li)).not.toMatchObject(renderToObject(
  React.createElement('li')
)); // but cannot omit text, same song, second verse

expect(mountedToObject(li)).toMatchObject(literalToObject({
  type: 'li',
})); // object literal can omit props and children
expect(mountedToObject(li)).toMatchObject(literalToObject({
  props: {style: {textDecoration}},
})); // or could even omit the type of the element

pedrottimark on 14 Dec 2016

Goal of this iteration: easily delete irrelevant parts from expected object for subset match.

Let’s take a minute to look at the bugbig picture :)

https://daveceddia.com/what-to-test-in-react-app/

The main reason to write tests is to ensure that your app works the way it should…test the high-value features. For example, let’s say you have a shopping cart. You click an “Add to Cart” button. The app had better damn well add that item to the cart.

http://reactkungfu.com/2015/07/approaches-to-testing-react-components-an-overview/

When you test React components, there are at least two concerns that must be tested:
Given properties and state, what structure our rendered tree will have?
Given an output of render, is there a possibility to transition from state A to state B?

https://blog.callstack.io/unit-testing-react-native-with-the-new-jest-i-snapshots-come-into-play-68ba19b1b9fe

The bigger picture (full render) is complicated to create, tedious to maintain and very repetitive (copy-paste).
The second approach (using find, contains…) is unstable: We need to remember to implement all the cases since the new ones won’t break those tests.

Example table inspired by Whinepad from React Up & Running

Here is a balanced diet of variations on snapshot testing for aspects of a component.

Read operation: some snapshots for a representative variety of props, so developers and reviewers can see and confirm any changes to structure and content. For example:

test('Table renders received fields and records: Read of CRUD', () => {
  const store = createStore(reducer);
  store.dispatch(receiveData(fieldsReceived, recordsReceived));
  const rendered = renderer.create(
    <Provider store={store}>
      <Table />
    </Provider>
  ).toJSON();
  expect(rendered).toMatchSnapshot();
});

Given some tests for complete details above, invest effort on purposefully incomplete snapshots to maximize relevant and minimize irrelevant details, so tests below focus on correct interaction.

Update operation: one or more diff snapshots at the “scene of the change” to test the rendered delta from state changes, as illustrated in https://github.com/facebook/jest/issues/2197#issuecomment-268275949

Photograph analogy: subject in focus and cropped close

Create or Delete operation: subset matches of the surrounding context to test that the change is in the correct place or affects the correct item.

ADDED to test content in View operations with minimal irrelevacy and redundancy:

expect(wrapper.find('tbody tr').map((node) => node.find('td').not('.delete').map((node) => node.text()).join(''))).toBe(/*map of joined field text for records*/)

Sort is similar to Read for content and Update for UI feedback (for example, ascending/descending indicator in most-recently clicked column heading).
Search is similar to Delete (when typing) and Create (when deleting) for content.

Photograph analogy: subject at focal point within depth of field and surroundings are blurred

In the following two tests, the heart is a toMatchObject assertion.

mountedDomToObject encapsulates enzymeToJSON(mount(…)) to omit React components and return only DOM elements, as discussed in preceding https://github.com/facebook/jest/issues/2202#issuecomment-267073209
renderAsRelevantObject encapsulates react-test-renderer to improve subset match:
- omit props properties whose values are an empty object
- omit children properties whose values are [irrelevant] which is a sentinel value
- copy $$typeof properties as enumerable, related to https://github.com/facebook/jest/issues/2351

Here is a recipe to edit an internal snapshot for the expected object:

If you can use same data as a Read test, copy from the complete snapshot
Paste into relevantTestObject(…)
If descendants of an element are irrelevant, replace them with {irrelevant}
Delete irrelevant props from remaining elements
Optional: If value of a text or number child is in scope, replace literal with {value}

Here is a chunk of complete details so you can see the motivation for incomplete details :)

…
  <tbody
    onDoubleClick={[Function]}>
    <tr>
      <td
        className="delete"
        onClick={[Function]}
        title="delete row">
        ❌
      </td>
      <td
        data-field-key="when"
        data-record-id={0}>
        2016
      </td>
…

test('Table creates a default record in a non-empty table: Create of CRUD', () => {
  const store = createStore(reducer);
  store.dispatch(receiveData(fieldsReceived, recordsReceived));
  const wrapper = mount(
    <Provider store={store}>
      <Table />
    </Provider>
  );

  const recordCreated = {
    when: 2016,
    what: '',
    whimper: '',
  }
  // dispatch an action because the add button is outside the table
  store.dispatch(createRecord(recordCreated));
  invariant(recordsReceived.length === 2, 'added 1 record to 2 records');
  // the irrelevant cell contains the delete button for the row
  expect(mountedDomToObject(wrapper.find('tbody'))).toMatchObject(renderAsRelevantObject(
    <tbody>
      <tr>
        <td>{irrelevant}</td>
        <td>{recordCreated.when}</td>
        <td></td>
        <td></td>
      </tr>
      <tr>{irrelevant}</tr>
      <tr>{irrelevant}</tr>
    </tbody>
  ));
  expect(theadCount(wrapper)).toBe('3');
});

test('Table deletes the middle, first, and then only record: Delete of CRUD', () => {
  const store = createStore(reducer);
  store.dispatch(receiveData(fieldsReceived, recordsInitial));
  invariant(recordsInitial.length === 3, 'delete 3 records in a specific order');
  const wrapper = mount(
    <Provider store={store}>
      <Table />
    </Provider>
  );
  const wrapperBody = wrapper.find('tbody');

  // Delete the middle record
  wrapperBody.find('tr').at(1).find('td').at(0).simulate('click');
  expect(mountedDomToObject(wrapperBody)).toMatchObject(renderAsRelevantObject(
    <tbody>
      <tr>
        <td>{irrelevant}</td>
        <td>{recordsInitial[0].when}</td>
        <td>{recordsInitial[0].what}</td>
        <td>{recordsInitial[0].whimper}</td>
      </tr>
      <tr>
        <td>{irrelevant}</td>
        <td>{recordsInitial[2].when}</td>
        <td>{recordsInitial[2].what}</td>
        <td>{recordsInitial[2].whimper}</td>
      </tr>
    </tbody>
  ));
  expect(theadCount(wrapper)).toBe('2');

  // Delete the first record
  // Delete the last record, which is now also the first and only
});

Hypotheses to be challenged or confirmed:

Your initial investment is repaid by rarely needing to update because of changes that are irrelevant to the purpose of the test (false positives).
Because the pattern is more declarative than multiple assertions on values, you have a higher chance to see when you need to update the test for relevant changes (avoid false negatives when an attempted enhancement is incorrect).
Especially for enhancements to existing components, you can write tests first if you want to follow TDD order.

By the way, I support traditional traversal assertions when their context isn’t in focus:

const theadCount = (wrapper) => wrapper.find('thead th').at(0).text();

Instead of yet more devDependencies, where might the functions and sentinel value come from?

mountedDomToObject could become a named import from enzyme-to-json
renderAsRelevantObject could become a method of renderer and irrelevant could become a named import from react-test-renderer

pedrottimark on 20 Dec 2016

ReactTestComponent plugin for pretty-format works for elements that omit props or children. Such forward thinking supports JSX diff when relevant subset does not match!

Example for Create https://github.com/facebook/jest/issues/2202#issuecomment-268347718 if record is incorrectly inserted as last row instead of first:

 <tbody>
   <tr>
     <td />
     <td>
-      2017
+      2016
     </td>
     <td>
-      ECMAScript 2017
+      ECMAScript 7
     </td>
     <td>
-      async or swim, not much longer to await
+      small but powerful: ha * ha === ha ** 2
     </td>
   </tr>
   <tr />
   <tr />
 </tbody>

pedrottimark on 23 Dec 2016

@cpojer What do you think about the following difference between react-test-renderer compared to react-dom and react-addons-test-utils as a baseline for what to expect:

renderer.create(…).toJSON() returns different result for  and {''} that is, first span has no children and second span has one child: the empty string
reactDOM.render(…) and TestUtils.renderIntoDocument(…) return both span with no children

Where I ran into the difference was tests comparing received from mountToJson or shallowToJson to expected from react-test-renderer discussed in preceding comments. Although it might not be possible for different libraries to agree on all edge cases, this stuck out from other differences, because enzyme might follow what react-addons-test-utils does.

On the other side of the argument, this seems like a breaking change for snapshot tests:

exports[`empty string is rendered by react-test-renderer 1`] = `
<span>

</span>
`;

exports[`if empty string is not rendered by react-test-renderer 1`] = `<span />`;

To take it one step farther, a codemod to fix this case could make a subtle (though perhaps unlikely in practice) incorrect change to the identical snapshot which is arguably correct for elements which have conditional logic like {false}{''} or {''}{null}

The reason I say that empty string as an explicit child could be considered consistent in these cases: when there are two or more child expressions, react-dom and react-addons-test-utils render a pair of comment nodes per string or number value expression.

For the preceding cases, they render just the 2 comment nodes but no text node.
For {''}{''} they render just the 4 comment nodes but no text nodes.
For {'hello'}{''}{13} they render 6 comment nodes and 2 text nodes.

By the way, the conditional logic example is what put me onto this issue. The mountToJson and shallowToJson methods from enzyme-to-json use lodash.compact to get rid of all empty strings always, which is inconsistent in the other direction. If it looks like this subset matching proposal moves forward, I can discuss with the maintainer of that project. In my sample project, the testing problem made me think harder about a better way to express the conditional logic :)

In summary, what do you think about this case when the children on a React element consists of exactly one empty string:

for snapshots as they currently exist
for potential future relevant subset matching
for helpful analogy with renderers, including React Native, and fiber (see below :)

If you think there is a case to propose a change, are you willing to take the role of “champion” if I am willing to do the work? Aha, at this very moment, the situation has become more complex:

As an alternative, the proposed renderAsRelevantObject method or function could take up the slack by changing children: [''] to children: null as part of its responsibility. The reason to discuss a change to the output from react-test-renderer is fewer “magic” transformations seems like less confusion in the long run.

pedrottimark on 11 Jan 2017

@cpojer While reading the code for react-test-renderer something else hit my eye.

Flow type omits number in disjunction for children and has only string for the values in props:

What do you think? Does it matter?

EDIT: Submitted https://github.com/facebook/react/pull/8816

pedrottimark on 11 Jan 2017

I'm not super up-to-date on this discussion and react related conversation should probably happen somewhere else (react repo?) but here is one pointer: https://github.com/airbnb/enzyme/pull/742 enzyme is looking into this, too.

cpojer on 13 Jan 2017

@cpojer Thank you for that link. For now, the helper function will take care of the difference.

pedrottimark on 13 Jan 2017

@pedrottimark I'm a bit unclear on this issue what the action items are. What can we do in Jest to improve this? The shape matcher proposal seems to make sense to me; it appears to me that pretty-printing React component diffs through toMatchSnapshot would help us with this?

I understand you have concerns with the behavior of the react-test-renderer. I would like to ask you to discuss that in the react repo, as the Jest team isn't responsible for this piece :)

cpojer on 24 Jan 2017

Yes, it’s time to make things clear enough for action! Please pardon the length of this comment.

@cpojer Would you like to craft an API from the following summary from discussion items?

Is the item needed?
What to call it?
From where will test files import it?

After that, I will work through the remaining details for each item that survives.

Part A seems useful to test update operations. Part B seems useful to test create or delete operations. So they are independent decisions, but the whole is more than the sum of the parts :)

Part A: differential snapshot

Decision 1: Is diffSnapshot needed? What to call it? From where to import it?

expect(diffSnapshot(prev, next)).toMatchSnapshot();

Decision 2: Is diffSnapshotSerializer needed? What to call it? From where to import it?

expect.addSnapshotSerializer('path/to/diffSnapshotSerializer');

Alternative 2: a standard built-in plugin? For example, pretty-format/plugins/diffSnapshot.js

In case it helps you to know, the serializer does the hard work. The function returns an identifiable object which consists of the args prev, next, and who knows, maybe options some day.

Part B: expected test object for relevant subset match

Although Jest team doesn’t decide, it seems like you are co-stakeholder with React team concerning developer experience in Jest. According to guidance here, I can open an issue there.

Opinion 3: Is render function or method needed? What to call it? How to import it?
Opinion 4: Is irrelevant descendants sentinel value needed? What to call it? How to import it?

// Given an element, for example:
const element = <tr><td>{irrelevant}</td><td>{recordCreated.when}</td><td /><td /></tr>;

// A possibility is a separate named export
import {irrelevant, renderAsRelevantObject} from 'react-test-renderer';
expect(…).toMatchObject(renderAsRelevantObject(element));

// Another possibility is a method of the default export
import renderer, {irrelevant} from 'react-test-renderer';
expect(…).toMatchObject(renderer.create(element).toRelevantObject());

The function calls renderer.create(element).toJSON() as its implementation to get a test object, and either function or method returns a test object that is transformed as follows:

make $$typeof enumerable to solve #2351
omit props property if it’s an empty object
omit children property if it consists of sentinel value: [irrelevant]
in children convert number to string items for consistency, especially with future fiber

Among the reasons to use react-test-renderer for the expected value: avoid warning from React about element like tr as an invalid child when the following mount it to a div element:

renderIntoDocument from react-addons-test-utils
mount from enzyme

Part B: received test object for relevant subset match

Although Jest team doesn’t decide, we can have friendly influence on enzyme-to-json maintainer concerning developer experience in Jest. According to guidance here, I can reopen an issue there.

Opinion 5: Is mountedDomToObject function needed? What to call it?

import {mount} from 'enzyme';
import {mountedDomToObject} from 'enzyme-to-json';
const wrapper = mount(/* component in its initial state */);
// simulate interaction, and then compare component to relevant subset:
expect(mountedDomToObject(wrapper)).toMatchObject(…);

The new named export transforms as follows an enzyme wrapper returned either directly from mount or indirectly from traversal methods:

only DOM nodes, unlike mountToJson which returns tree of components and DOM nodes
in children convert number to string items for consistency, especially with future fiber
in children omit null but keep empty strings returned from component render methods

In summary of Part B, a delicate interaction that Jest depends on but doesn’t control: react-test-render and enzyme-to-json must compute compatible expected and received test objects for toMatchObject assertion. We’ll need to provide solid unit tests of edge cases for both projects.

P.S. Update to previous comment about react-test-renderer as my ignorance decreases, it looks like root of problem might be mountToJson where compact from lodash eliminates empty strings.

pedrottimark on 24 Jan 2017

I can't speak as much about Part B/C but for the first one:

How do we identify a previous snapshot? Since we already keep state, should we just have go with the flow here?

const apple = new Apple();
expect(apple).toMatchDIffSnapshot()
const applieModified = apple.eat();
expect(appleModified).toMatchDiffSnapshot()

You are also bringing up the question on how we should diff: it seems better to do that on the object level, but it may be slow because it has to be recursive.

cpojer on 25 Jan 2017

An example came to mind this week why to let the test take full responsibility for state.

As a warm up, pseudo code for a test of a Redux reducer. Although we don’t assume everyone uses Redux, many intermediate level React devs have probably heard about immutability.

it('reducer changes state in response to action', () => {
  const prev = /* initial state */
  const next = reducer(prev, /* action */);
  expect(diffSnapshot(prev, next)).toMatchSnapshot();
});

Imagine that you double-click a table cell to edit it. A test for this update operation might have 3 states: prev, updating, next (that is, the edited result). Because a mounted enzyme wrapper is mutable (as far as I know) calling the hypothetical mountedDomToObject function to get an object representing the DOM nodes has a beneficial side-effect to save the UI state.

it('table updates a cell', () => {
  const wrapper = mount(/* Table component in its initial state*/);
  const prev = mountedDomToObject(wrapper.find(/* cell selector */);
  wrapper.find(/* cell selector */).simulate('doubleClick');
  const updating = mountedDomToObject(wrapper.find(/* cell selector */);
  // interaction to edit the input element in the cell
  const next = mountedDomToObject(wrapper.find(/* cell selector */);
  expect(diffSnapshot(prev, updating)).toMatchSnapshot();
  expect(diffSnapshot(prev, next)).toMatchSnapshot(); // instead of (updating, next)
});

What I did at first was diff the adjacent pairs of states. As if Jest diffs the states implicitly.

Positive result is 2 diffs instead of 3 reduces size and irrelevant changes to cell.
Negative result is 2 diffs involving the form (first adding it, second removing it) which misses an opportunity to reduce size and irrelevant changes to form.

It hit me this week that the second diff is clearer and shorter if its purpose is to compare the initial and final cell contents, independent of how the interaction happened.

So this is a long way around to suggesting it is better for the test to handle state explicitly.

How the trial implementation works: the serializer calls its serialize argument on both prev and next state (which can be any type) and then uses diffLines from diff package, imitated from the code to display results when a snapshot fails to match. So I didn’t expect slow to be a problem.

pedrottimark on 25 Jan 2017

What do you think about API:

assertion is toMatchDiffSnapshot as you [Christoph] suggested in https://github.com/facebook/jest/issues/2202#issuecomment-275070495
argument of expect is pair of states as array tuple

Here are examples for the scenarios in the preceding comment:

it('reducer changes state in response to action', () => {
  const prev = /* initial state */
  const next = reducer(prev, /* action */);
  expect([prev, next]).toMatchDiffSnapshot();
});

it('table updates a cell', () => {
  const wrapper = mount(/* Table component in its initial state*/);
  const prev = mountedDomToObject(wrapper.find(/* cell selector */);
  wrapper.find(/* cell selector */).simulate('doubleClick');
  const updating = mountedDomToObject(wrapper.find(/* cell selector */);
  // interaction to edit the input element in the cell
  const next = mountedDomToObject(wrapper.find(/* cell selector */);
  expect([prev, updating]).toMatchDiffSnapshot();
  expect([prev, next]).toMatchDiffSnapshot(); // instead of [updating, next]
});

pedrottimark on 1 Feb 2017

In this proposal, how will we customize diffing? Will toMatchDiffSnapshot be able to take an optional "diff" argument?

Also, I think this is a lot about how we can highlight this in the output as well. Should diff snapshots show different output than regular toMatchSnapshot.

cpojer on 2 Feb 2017

Am happy to receive your direction on any or all of:

developer experience to understand output in .snap files and console errors (see below)
developer experience to write tests (I don’t have strong preference between APIs below)
how, where, and whether it fits into Jest code

What is an example an optional diff argument? That sounds more like the original API:

expect(diffSnapshot(prev, next, options?)).toMatchSnapshot();

in which diffSnapshot assumes that a serializer does all of the hard work:

serializes prev and next args
calls diff.diffLines
proposed different output is a 2-character col inspired by (shudder :) merge conflicts:
- < for removed to contrast with - for expected in Jest or removed in GitHub
- > for added to contrast with + for received in Jest or added in GitHub
- space for unchanged

Here’s an example of snap and error from https://github.com/facebook/jest/issues/2197#issuecomment-266059285 to get your critique of format:

exports[`CheckboxWithLabel changes attribute and text after click (diff) 1`] = `
  <label>
    <input
<     checked={false}
>     checked={true}
      onChange={[Function]}
      type="checkbox" />
<   Off
>   On
  </label>
`;

   <label>
     <input
-<     checked={false}
->     checked={true}
+      checked={false}
       onChange={[Function]}
       type="checkbox" />
-<   Off
->   On
+    Off
   </label>

The original API could be a third-party extension to Jest. To become a standard part of Jest:

How to make diffSnapshot function available in test files.
Whether to define corresponding diffSnapshot serializer in plugins of pretty-format?
If yes to preceding question, which of the following:
- Include it in jest-snapshot built-in list with [ReactElementPlugin, ReactTestComponentPlugin]
- Let people who need it include it via snapshotSerializers in Jest configuration
- Let people who need it include it via expect.addSnapshotSerializer in test files

The alternative API with toMatchDiffSnapshot answers how to use it (except maybe for options). To avoid duplicated code, I think it would imply refactoring in:

packages/jest-snapshot/src/index.js
packages/jest-snapshot/src/State.js

Current status:

Replaced confusing regex with split-join, so diffSnapshot algorithm is now reviewable.
Am building a superset of potential tests for the serializer.
Will think about potential tests of output from failures, in case we need them.
Finding what and where the new docs explain snapshot testing to write first draft of content.

pedrottimark on 2 Feb 2017

Hm, interesting, I actually thought we could push it into toMatchDiffSnapshot:

expect([prev, current]).toMatchDiffSnapshot(options?)

However, diffSnapshot might be more flexible overall, but I'm not sure yet. It really depends on how flexible we want this to be and what kind of user experience we want to make work here. One example could also be to have a mode (using --expand?) where we'd actually show the prev/next snapshots as well. I think there is some UX exploration to do here.

The question is, how would the differ work? Would it receive two pretty-printed strings and operate on those (line-by-line) or would it receive the plain data and require users to diff them, based on what object they are? If so, we could have a default recursive-diff feature in toMatchDiffSnapshot.

I also encourage you to see past what we have now with toMatchSnapshot. Assume that jest-snapshot is the low-level building block for this new feature but we may want an entirely different CLI output for the diffs. I know @rogeliog has been thinking about this. I think there is a ton of value in visualizing snapshots over time/state.

cpojer on 3 Feb 2017

Whilst this discussion has moved from the original suggestion to diffing diffs, I thought it might be worth mentioning another project that does exactly what was originally suggested - maybe you can borrow some ideas.

From the issue:

expect(vdom.toJSON()).toMatchShape(
  <Foo>
    <Bar who="John" />
  </Foo>
);

unexpected-react does exactly this, and can ignore extra props and child elements if desired.

expect(vdom, 'to have rendered',
  <Foo>
    <Bar who="John" />
  </Foo>
);

You can also select parts of the tree to compare against the jest snapshot. e.g. with the above example

expect(vdom, 'queried for', <Bar />, 'to match snapshot');

This will validate the snapshot, but only of the Bar subcomponent. Works on shallow and full DOM rendering too.

Zooming in like this is also a neat way to limit the scope of tests.

bruderstein on 3 Feb 2017

@bruderstein Thank you for the link. I am still interested in partial match.

@cpojer On the subject of partial match, in addition to enzyme issue you gave me recently, do you know if there is any issue about goals for react-test-renderer in 2017?

I actually thought we could push it into toMatchDiffSnapshot … but I'm not sure yet

Yes indeed, we could. Optional options and snapshot name made me timid. Me either.

how would the differ work?

Trial implementation is a hybrid of the 2 suggestions: https://gist.github.com/pedrottimark/24f182ff6249f83658b1f68a77b2b125

I will compare if it has (or could have) similar algorithm to jest-diff with alternative formatting options, also considering the thought about output in https://github.com/facebook/jest/issues/2471

we could have a default recursive-diff feature

Sounds interesting, can you tell me more?

pedrottimark on 3 Feb 2017

I guess I didn't consider this so much an extension for pretty-format as I consider the implementation of this to exist on top of jest-snapshot in some way. Instead of diffing line-by-line I thought it makes sense to diff JavaScript objects before printing. I see value in both approaches, unsure which one is the best yet.

cpojer on 3 Feb 2017

@cpojer +1 for diffing the object - that seems likely to produce better output than the stringified version from pretty-format.

developit on 3 Feb 2017

@cpojer unexpected-react diffs objects, and there are a number of advantages:

You can have some "intelligence" around things that are added or removed (e.g. is an added attribute less important than an added child)
Easier to pretty-print without forcing everything onto a new line to ease the diffing.
More "intelligence" around different attributes - e.g. the order of classes in className doesn't matter, and if you're comparing objects you can apply that logic.

The only disadvantage is serialisation and particularly deserialisation for comparison in snapshots - that's obviously a bit trickier, and much more difficult to do generically.

bruderstein on 3 Feb 2017

👍1

@bruderstein sorry for what is probably an uninformed question - but aren't snapshots already stored as JSON?

developit on 3 Feb 2017

@developit In jest, they're stored as strings in the snapshot files. If you're using unexpected-react with jest, they're stored as JavaScript objects, with a string representation as a comment to assist in seeing the diff in version control.

bruderstein on 3 Feb 2017

👍1

Ah interesting.

developit on 4 Feb 2017

I think this concept is quite interesting and it is really worth exploring... I'm a bit worried of how complex the API can get here.

I kind of like the idea of just starting by using the latest snapshot as a base diff for the next snapshot, with as close to zero options as possible. This will allow us to just create the minimum API to support the most common use case of this feature and then incrementally make it more flexible as it is needed. I would love to leak the lest amount of implementation detail on how snapshots are stored or how the .snap files works.

rogeliog on 5 Feb 2017

Yes, it sounds super to diff objects and serialize separately as late as possible.

To diff the state of a component or store might also apply to React or Redux dev tools.
The concept becomes similar to reconciliation.
It might become possible for a test to combine diff and subset match:
- A diff focuses attention on the expected change in state.
- A subset match (soon with asymmetrical matchers :) minimizes irrelevant details.

I am exploring what it would look like to write a test that combines diff and subset match.

pedrottimark on 7 Feb 2017

Reading through this thread is a bit of a trip, since the past week I've independently followed pretty much every bit of the same winding path, starting from wanting a shape matcher, moving to a diff of diffs approach, then trying to just pretty-format to serialize and js-diff to diff the strings, then realizing that ends up either excluding all context (if you render unchanged hunks as "...") or keeping a lot of unnecessary context (if you just render the unchanged hunks as-is), then trying to diff the objects, writing that code, then finding subtle edge cases with keys and reordering, then trying to take two objects, collapse equivalent props to ...="..." and children to '...' and string-diff that, then discovering other subtle edge cases, then going back to tree diffing and looking into tree-diff, then realizing that didn't handle keys at all... phew. The best version I have so far looks like https://gist.github.com/0x24a537r9/9bc7b40aab1ef5a6860db42ce9cf06b4.

Most of the time, it ellipsizes common props, children, and subtrees and produces lovely minimal-context diffs as desired like:

  <View …="…">
    <View …="…">…</View>
-   <View
-     a="1"
-     b="2"
+   <Image
+     c="3"
+     d="4"
    />
    <View …="…">…</View>
  </View>"

Unfortunately, without going into the details, this approach of collapsing the trees before diffing is inherently flawed and has some subtle bugs and poorly-handled edge-cases. During the process though it helped me to realize that reinventing the wheel is not the right approach here. As @pedrottimark mentioned, React already has a reconciliation algorithm that takes keys into account and, most importantly, is actually representative of the transformations React will perform in prod. So what we probably want is to actually a custom React renderer that rather than performing reconciliations on updates like a regular renderer, simply serializes the set of changes React has already found for us. We can then use those serialized changes as snapshots.

Apart from meaning we don't have to write our own diffing algorithm (surprisingly tricky in practice), this also adds a completely new value proposition to the tool: the ability to test the optimality of rendering changes. For example, if I don't specify keys in a list of elements, I might be incurring unnecessary reconciliation operations, but doing the kinds of diffs described above probably won't actually highlight those issues, since the differs will act differently than React's. By using React's intrinsic VirtualDOM diffing, we gain the ability to test not only what state we're changing to, but also the means by which React gets us there. So if a diff snapshot seems unnecessarily verbose, it can clue us into the fact that we might not be designing our DOM efficiently.

Useful references for creating a new renderer: https://github.com/iamdustan/tiny-react-renderer/tree/master/src/stack and https://github.com/facebook/react/blob/master/src/renderers/testing/ReactTestRendererFiber.js

0x24a537r9 on 5 Apr 2017

@0x24a537r9 That sounds really interesting and further proof that we need to do something here. Are you planning on creating a proof-of-concept renderer for React?

cpojer on 9 Apr 2017

@cpojer So, yes and no.

Yes: I'm most of the way through building a diffReact module that uses the simpler virtual-dom project for the diffing (rather than React's virtual DOM diffing via a custom renderer). I already have tree canonicalization and diffing working--I'm just working through the last, most complex part: serialization in a way that shows the context of each change without unnecessary detail.

No: I'm not planning a proof-of-concept renderer--I tried looking into it, but quickly found that it's way above my current React abilities. I only started using React two months ago. That said, the core of my diffReact proof-of-concept (the smart-serialization algorithm) should be fairly transferable into a React renderer by someone familiar with the innards of React. As input it just needs some virtual DOM representation of the two trees and an object with the details of what changed between them (e.g. props changes, element insertion, element removal, children reordering, text changing).

I'll be sure to post a gist back to this thread once it's ready--it might be worth bringing into Jest even in that form, since it should actually be consistent with React's transformations in theory (virtual-dom appears to use the same constraints for diffing as React, for example its treatment of keys and type changes).

0x24a537r9 on 10 Apr 2017

Starting from Fiber (available as react@next on npm), writing custom renderers really is a bliss and doesn't really require any React knowledge anymore (everything can fit in a single file!). I've documented which functions need to be implemented here¹, if you're interested to take a look.

¹ Some functions might have had additional arguments added to them since I wrote this, but it worked fine with [email protected] a few weeks ago, so there shouldn't be too much delta :)

arcanis on 10 Apr 2017

👍1

Although I see much potential in the diff path of this discussion, the original subset match seems to make the biggest difference (pardon pun :) as a next step forward.

The small amount of remaining work is outside the control of Jest. Here is the current status:

For the proposed mountToDeepJson and mountToShallowJson functions described in https://github.com/pedrottimark/whimper#proposed-to-import-from-enzyme-to-json, I submitted a pull request: https://github.com/adriantoine/enzyme-to-json/pull/56
For the proposed relevantTestObject function and irrelevant sentinel string described in https://github.com/pedrottimark/whimper#proposed-to-import-from-react-test-renderer, @cpojer can you recommend anyone on React team who could act as a “champion” to guide me through the process of contributing? For example, to react-test-render or somewhere else?

pedrottimark on 27 Apr 2017

Component shape matching would be really powerful for testing required relationships between ancestor/child HTML elements. For example, a menuitem element is required to exist in the context of a group, menu or menubar. I'd love to be able to declare generic shapes for these semantic relationships and test them against component in a library.

jessebeach on 28 Apr 2017

👍1

@jessebeach What do you think about a custom tester as expected value of toEqual in Jest?

https://jasmine.github.io/edge/introduction#section-Custom_asymmetric_equality_tester

Could you make a gist of a minimal realistic example, so that I can make sure to understand?

pedrottimark on 28 Apr 2017

👍1

Interesting, I hadn't considered an asymmetric equality tester. I guess it's always possible to write a custom matcher. I guess I was hoping for something a little more templatic :) Keep in mind that I'm not asking you to write this. It just strikes me that I might get what I need for free from the work you're doing.

The relationship between DOM fragment and the Accessibility Tree (AX Tree) looks like this:

<div role="menubar">
  <div>
    <div>
      <div role="menuitem">Save</div>
      <div>
        <div role="menuitem">Edit</div>
      </div>
    </div>
  </div>
</div>

<AXMenuBar>
  <AXMenuItem>Save</AXMenuItem>
  <AXMenuItem>Edit</AXMenuItem>
</AXMenuBar>

DOM elements that have no impact on the AX Tree are ignored. So for the purposes of asserting ancestor --> child relationships for valid accessible components, we really just want to assert that an element with a role menuitem is a child of a menbuar with no intervening elements with a role attribute regardless of depth:

Valid:

<div role="menubar">
  ...
  <div role="menuitem">Save</div>
  ...
    <div role="menuitem">Edit</div>
</div>

Invalid:

<div role="menubar">
  <div role="article">
    ...
    <div role="menuitem">Save</div>
    ...
      <div role="menuitem">Edit</div>
  </div>
</div>

As I write all of this, I'm coming to the conclusion that this is way out of scope for what you're trying to achieve, right?

jessebeach on 30 Apr 2017

❤1

@jessebeach Thank you for the example. It suggests a descriptive test for a component.

Imagine writing a generic accessibleSubset function that, given a React test object, returns the subset of nodes which have counterparts in the corresponding accessibility tree.

A snapshot test of the subset seems like a regression test for accessibility relationships.

Prevent unexpected regression. If change is incorrect, then fix code.
Confirm expected progress. If change is correct, then update snapshot.

import React from 'react';
import renderer from 'react-test-renderer';

describe('Component', () => {
  describe('for … set of props', () => {
    const element = <Component {...props} />;
    const received = renderer.create(element).toJSON();

    it('renders …', () => {
      expect(received).toMatchSnapshot();
    });

    it('renders … (accessibility)', () => {
      expect(accessibleSubset(received)).toMatchSnapshot();
    });
  });
});

Oversimplified baseline snapshot for test:

<div role="menubar">
  <div role="menuitem">Save</div>
  <div role="menuitem">Edit</div>
</div>

Oversimplified diff when test fails:

  <div role="menubar">
+   <div role="article">
      <div role="menuitem">Save</div>
      <div role="menuitem">Edit</div>
+   </div>
  </div>

So, as is typical of snapshot testing, it’s specific to a component and therefore must be reviewed in context by humans, instead of being a generic “template” applied to a specific instance.

P.S. If the accessibleSubset function also omits props which are irrelevant for this type of test (for example, event handler functions) then you might not even need to consider replacing toMatchSnapshot assertion with toMatchObject assertion as discussed in this issue :)

pedrottimark on 1 May 2017

🎉2 👍2

As a quick follow up for those that are interested, I finished my diffing module and have released it as an npm package, diff-react. I didn't end up using a custom Fiber renderer, but it does all that I planned it to do at first using virtual-dom and should work for React, React Native, and React VR (the purpose for which I created it).

I don't want to distract from @pedrottimark's point though, because I agree that the original subset matcher has merit of its own, so if anyone has any questions on diff-react feel free to open an issue on its Github. Thanks!

0x24a537r9 on 2 May 2017

The sort of thing that @jessebeach was suggesting is _exactly_ what unexpected-react does (just not with the accessibility tree), and can confirm it makes for great non-brittle tests
e.g. a component that renders

<Connect(MyComponent)>
  <div className="foo bar baz">
    <BigText>
        <span>one</span>
    </BigText>
    <BigText>
        <span>two</span>
    </BigText>
  </div>
</(Connect(MyComponent)>

Can be asserted to match

<div className="bar">
   <span>one</span>
   <span>two</span>
</div>

And if it fails, you get a JSX diff (note the components identified as just wrappers are greyed out)

screen shot 2017-05-02 at 23 37 18

bruderstein on 2 May 2017

🎉5

I'm going to close this issue but feel free to keep the discussion going and create new issues for actionable things that come out of this.

cpojer on 15 Dec 2017

👍1

Jest: New shape matcher proposal

Most helpful comment

All 48 comments

1. content without functions

helper functions

2. result of interaction

Let’s take a minute to look at the bugbig picture :)

Example table inspired by Whinepad from React Up & Running

Part A: differential snapshot

Part B: expected test object for relevant subset match

Part B: received test object for relevant subset match

Related issues