Testcafe: Snapshot property warnings work incorrectly with loops

[Ogurecher]

What is your Test Scenario?

Assert an unawaited snapshot property multiple times in a loop.

What is the Current behavior?

The warning is raised as if the snapshot was awaited.

What is the Expected behavior?

TestCafe should not raise warnings and the test should pass correctly.

Your complete test code (or attach your test files):

import { Selector } from 'testcafe';
fixture `Loop`;
test.only("looped assertion warning test", async t => {
    for (let i = 0; i < 2; i++) {
        await t
            .expect(Selector('body').innerText).ok();
    }
});

Your complete test report:

Running tests in:
 - Chrome 84.0.4147.105 / Windows 10

 Loop
 √ looped assertion warning test


 1 passed (1s)

 Warnings (1):
 --
  You passed a DOM snapshot property to the assertion's 't.expect()' method. The property value is assigned when the snapshot is resolved and this value is no longer updated. To ensure that the assertion verifies an
  up-to-date value, pass the selector property without 'await'.

     1 |import { Selector } from 'testcafe';
     2 |fixture `Loop`;
     3 |test.only("looped assertion warning test", async t => {
     4 |    for (let i = 0; i < 2; i++) {
     5 |        await t
   > 6 |            .expect(Selector('body').innerText).ok();
     7 |    }
     8 |});

     at <anonymous> (D:\Work\tests\expectTest\loop-test.js:6:21)
     at <anonymous> (D:\Work\tests\expectTest\loop-test.js:3:1)

Your Environment details:

• testcafe version: 1.9.0-rc.1
• node.js version: v12.16.1
• command-line arguments: npx testcafe chrome .loop-test.js
• browser name and version: Chrome 84.0.4147.105
• platform and version: Windows

[philga7]

I was noticing that this same behavior appears to be prevalent within helpers functions as well.

If in my test I include a reference to something like helpers.clickElement() and then pass in a Selector as an argument, I will forever get the same warning, because I have an await statement within the helpers.clickElement() method.

I can provide further details if needed.

[Ogurecher]

Hello @philga7,
Thank you for your input. Yes, we would appreciate further details about your case.

[philga7]

@Ogurecher : Absolutely; details provided as properly anonymized for confidentiality purposes.

TestCafe v.1.9.0
Chrome 84.0.4147.125
macOS 10.15.6

I call a helper within my given test function:

test.meta( { test: 'regression' } )
('Sample test case', async (t: any) => {
     await helpers.validateElementText(await myPage.findCompanyName(expectedMyPageText.companyName.name),
          expectedMyPageText.companyName.name);
});

The helper is set up accordingly:

public async validateElementText(element: Selector, label: string): Promise<void> {
     await t
          .expect(element.innerText).contains(label);
}

The above when ran with the rest of my awaits, etc. within the test function run normally both pre- and post-v1.9.0. However, as mentioned in my previous response, here is the warning I get:

You passed a DOM snapshot property to the assertion's 't.expect()' method. The property value is assigned when the snapshot is resolved and this value is no longer updated. 
To ensure that the assertion verifies an up-to-date value, pass the selector property without 'await'.

     .
     .
     .
     188 |    public async validateElementText(element: Selector, label: string): Promise<void> {
     189 |        await t
   > 190 |            .expect(element.innerText).contains(label);
     191 |    }

The challenge here is that I use lots of nested functions in order to accomplish what we do. Hopefully TestCafe isn't inadvertently trying to take this ability away or, better yet, can provide a reasonable work-around to this type of functionality if such warnings are insistent upon change :)

Let me know if you require additional information.

[Ogurecher]

@philga7, Thank you for additional details. I couldn't reproduce this issue using your example. To reproduce it, I need more information about the findCompanyName function (feel free to redact sensitive data). Could you please share more information about this function or create a simple example that demonstrates the issue?

Regarding your concerns: these warnings were implemented to clarify some aspects of our API and to improve User Experience. If your tests work as expected, you can leave them as they are. We are not planning to restrict this functionality.

Also, could you please create a separate thread for this issue?

[philga7]

@Ogurecher: Let me first bring all the functions together before opening up a separate thread, as I think we're dealing with recursive logic that TestCafe is now interpreting as a non-updating snapshot value (per the warning). If we need to make this into a separate thread, I can certainly do so.

Here is the complete picture (apologies for not being more specific originally):

I call a helper within my given test function:

test.meta( { test: 'regression' } )
('Sample test case', async (t: any) => {
     await helpers.validateElementText(await myPage.findCompanyName(expectedMyPageText.companyName.name),
          expectedMyPageText.companyName.name);
});

The myPage.findCompanyName() variable and function are as follows:

public companyName: string = '.companyName';

public async findCompanyName(name: string): Promise<any> {
        return Selector(this.companyName).withText(`${name}`);
    }

The helper is set up accordingly:

public async validateElementText(element: Selector, label: string): Promise<void> {
     await t
          .expect(element.innerText).contains(label);
}

The above when ran with the rest of my awaits, etc. within the test function run normally both pre- and post-v1.9.0. However, as mentioned in my previous response, here is the warning I get:

You passed a DOM snapshot property to the assertion's 't.expect()' method. The property value is assigned when the snapshot is resolved and this value is no longer updated. 
To ensure that the assertion verifies an up-to-date value, pass the selector property without 'await'.

     .
     .
     .
     188 |    public async validateElementText(element: Selector, label: string): Promise<void> {
     189 |        await t
   > 190 |            .expect(element.innerText).contains(label);
     191 |    }

I hope this helps; let me know!

[Ogurecher]

@philga7, Thank you for the clarification. Unfortunately, I still can't reproduce the issue.

TestCafe 1.9.0
Chrome 84.0.4147.125
Windows 10 1803

My test page index.html:

<!DOCTYPE html>
<html>
    <body>
        <div class='a'>a1</div>
        <div class='a'>a2</div>
    </body>
</html>

My test.ts file:

import { Selector } from 'testcafe';
import { validateElementText } from './helpers';

async function findCompanyName(name: string): Promise<any> {
    return Selector('.a').withText(`${name}`);
}

fixture `helpers`
    .page `./index.html`

test.meta( { test: 'regression' } )
('Sample test case', async t => {
    await validateElementText(await findCompanyName('a2'), 'a2');
});

My helpers.ts file:

import { t } from 'testcafe';

export async function validateElementText(element: Selector, label: string): Promise<void> {
    await t.expect(element.innerText).contains(label);
}

Console command: testcafe chrome ./test.ts

Please share a repository with a reproducible example.

[philga7]

@Ogurecher: Thanks for sharing your test. I noticed off the top that you're running under Windows (whereas mine is macOS), and you've explicitly declared your functions (whereas mine are in classes), and your test case is using explicit values (whereas mine are variables).

For the declared function aspect, I honestly do believe that if I were re-writing these tests, I'd likely use the constructor object (if that's what it's called -- I'm pretty green at dev!) in the future.

Therefore, as originally requested, I'll open up a separate thread (and reference this one) to describe my particular situation. I'll also endeavor to get into as much detail as I can with exactly what I have.

Thanks again for the great assist.

[omartaghlabi]

I'm also running into this issue which is making the test logs longer than I'd like them to be.
Is there a way to disable warnings? I couldn't find documentation on this.

[Ogurecher]

@omartaghlabi, There is currently no option to disable these warnings. We'll look into ways to fix the incorrectly raised warning messages in the nearest future so that a workaround will not be necessary. We'll update this thread once the issue is fixed. Thank you for your patience.

[costag1982]

Hi

I am getting the same error when looping through:

Here is the code:

```async checkPredictions(predictions) {
await t.click(this.editPredictions)

let i = 1
for await (let prediction of predictions) {
  await t.expect(this.homeTeamScoreInput(i).value).eql(prediction.homeScore.toString())
  await t.expect(this.awayTeamScoreInput(i).value).eql(prediction.awayScore.toString())
  i+=1
}

}

[Ogurecher]

@costag1982, Thank you for the example. It will help us test the fix once it's ready.

[Khartir]

I ran into this problem outside of loops. Simply calling a function that does some assertions twice produces this warning.

Here is an example:

test('Checkout', async() => {
    await addToCart(item); //just some setup
    await validatePrice(item.price);
    await validatePrice(item.price);
});

const validatePrice = (price) => {
    const total = Selector('.class');
    return t.expect(total.textContent).contains(price);
}

Obviously this usually makes no sense, however the warning also occurs if there are some actions between the two calls (including navigating to different pages)

If the assertion is not extracted into a function, there will be no warnings:

test('Checkout', async(t) => {
    await addToCart(item);  //just some setup
    let total = Selector('.class');
    await t.expect(total.textContent).contains(item.price);
    total = Selector('.class'); //this is optional, it works with or without it
    await t.expect(total.textContent).contains(item.price);
});

This is with testcafe 1.9.3 and node 12.18.4

[Ogurecher]

@Khartir, Thank you for the example. We're working on a fix for the issue you outlined in the context of this issue.

[Ogurecher]

For the team:
This issue is caused by the same reason as #5449.

When the snapshot property promise is resolved, its callsite is saved to a callsite storage. We check this storage in the .expect assertion call. If the storage contains a callsite with the same filename and lineNumber as the .expect's callsite, we delete it and raise a warning.

If the property was not awaited, the .expect would execute before the property promise's .then method. Thus, some callsites would remain in the storage after they were asserted (we delete them only in .expect). It works fine if we use our assertion only once, but causes the warnings to raise if we call .expect from the same line of code again.

[GrahamLea]

Hi @Ogurecher. I've run into this too with my Page Objects. How do I figure out when the fix will be available?

[Farfurix]

@GrahamLea

Hello,

We are working on the next release version. At the moment, I cannot give you any ETA. However, you can run your tests using our current release candidate version: npm i [email protected].