Testcafe: Implement per-test, not per-browser hooks

[chadmyers]

Are you requesting a feature or reporting a bug?

Requesting a feature

What is the current behavior?

Fixture hooks are run once per fixture, test hooks are run once per test per browser (if using the all option or something like chrome,firefox).

What is the expected behavior?

I'd like a third type of hook which runs once per test for all browsers. So if I'm running a fixture with 2 tests in it against chrome,firefox browsers, I'd like it to do:

• fixture setup
• test1 pre-browser setup hook
• test1 setup hook for Chrome
• test1 against Chrome
• test1 setup hook for Firefox
• test1 againt Firefox
• test2 pre-browser setup hook
• test2 setup hook for Chrome
• test2 against Chrome
• test2 setup hook for Firefox
• test2 against Firefox

[inikulin]

Hello @chadmyers,

We've had discussion before about this feature in context of #903, but decided to postpone it's implementation since where was no demand for it at the time. The question is should this hooks have access to TestController and thus to tested page or not (like fixture.before/after hooks)? Can you share your use case, please, so we'll be able to figure out basic capabilities of this feature?

BTW, for currently you can use the following workaround to achieve desired functionality:

function once(fn) {
    var executed = false;

    return function() {
          if (!executed) {
               executed = true;
               return fn.apply(this, arguments);
          }
    };
}


fixture `My fixture`
     .beforeEach(once(async t => {
           // ...
      }));

test
.before(once(async t => {
//...
}))
('My test', async t => {
//...
});

[chadmyers]

@inikulin Thank you for the response! I'm loving TestCafe, btw.

I'll definitely check out the "once()" trick. That may do what I need.

You asked for my use case: Our testing philosophy for UI tests (what we call "regression tests" since we follow the unit/integration/regression test hierarchy scheme) is that we reset the DB back to a known state before each test. So before each test, I want to run our "clear()" routine. The problem I ran into is that if 4 browsers are running the test simultaneously and they fire off clear() at various times (due to asynchronous execution) they can stomp on each other. So I'd like to run the clear once for all four browsers, run the test for all browsers, then move onto the next test.

The workaround I used was to do all the test setup in the fixture hook, but that means I can only have one test per fixture and we usually have more tests per fixture.

It's possible I may be bringing my prior-tool philosophy into this tool and that's why I'm having this difficulty, so feel free to tell me I'm "doing it wrong" and I'll readjust my understanding accordingly :)

Thanks again for a great tool!

[chadmyers]

I understand the conundrum about the TestController access in the "beforeAll" (or whatever it would be called) method. For my purposes, I do _not_ need access to the browser in the beforeAll(), in fact, that would be the entire point of this method would be "Before each test, before the browsers are spawned". The before hook satisfies the "before each test, after the browser is spawned" use case very well.

[inikulin]

@chadmyers Thank you for the feedback. We need to figure out naming now. I wonder if test.beforeOnce and fixture.beforeEachOnce fits well.

[djbreen7]

If I'm not mistaken, it looks like the once workaround above will cause everything in the fixture.beforeEach to run only once. I wasn't able to accomplish what I wanted with that solution. I _was_, however, able to implement a similar solution that worked really well for me:

fixture(`My Fixture`)
    .before(async ctx => {
        ctx.hasExecuted = false;
        // Server-side actions
    })
    .beforeEach(async t => {
        // Before each test

        if (!t.fixtureCtx.hasExecuted) {
            t.fixtureCtx.hasExecuted = true;
            // Client side actions to run once.
        }
    })

[AlexanderMoskovkin]

@djbreen7 The approach with fixtureCtx looks well.

[qualityshepherd]

I'm trying to use djbreen7's solution to login _once_ and run multiple tests while logged in. Ie. behavior that would normally be handled in a beforeAll. Sadly, Testcafe seems to log me out between tests, and thus, this solution fails.

I tried using Roles as well... no joy.

Any ideas on how to work around this? I really just want to login _once_, then run a couple tests (same file).

Thanks!

[djbreen7]

@qualityshepherd Do you have sample code somewhere? I've had a lot of luck with Roles.

[AlexKamaev]

@qualityshepherd
The Roles mechanism can help you solve this issue. If you have some problems with it, feel free to create a separate ticker using this form
We would really appreciate if you provide a sample project which demonstrates the issue.

[qualityshepherd]

Ahhh... figured out the problem I was having with Roles. I'm using page objects and had a signInAsRole method that I was calling t.useRole(admin) in... that was no bueno. it seems useRole must be called in the test or beforeEach to work properly (ie. re-use one login for multiple tests).

My solution is now to have a roles.js that I export various roles from and import _that_ file into my tests and call useRole with reckless abandon within. Eg.

// roles.js
import { Role } from 'testcafe';
import signInPage from '../pages/signInPage';

export const admin = Role(signInPage.url, async t => {
        await signInPage.signIn('[email protected]', 'password');
    }, { preserveUrl: true });

And then...

// test.js
import { admin } from 'roles';

fixture `Sign In`
    test('should sign in', async t => {
        await t.useRole(admin);
...

[miherlosev]

fixture.beforeAll
fixture.afteAll
or
fixture.beforeEachOnce
fixture.afterEachOnce

[alexschwantes]

Regarding the question of should the is should the Fixture .before and .after hooks have access to TestController, I would say that it would be very useful to have this.

Our example is as follows:

1. Say we have a test suite with 6 tests that test a particular function of the site. All of these tests require the same prerequisite.
2. Currently the prerequisite setup in the Fixture.beforeEach() which runs a series of steps to set it up. This is run for all 6 tests. This takes about 20 seconds for each test.

Now we could of course develop a custom hook to insert this into the database directly but this has other security implications. We could also go direct to the database, but again security issues and it means the test can't be run in production or controlled environments. Also the solutions suggested to abuse the .beforeEach() hook to run only once don't seem like they would work as the setup takes 20 seconds and you could enter a situation where other tests start running before the prerequisite has been setup.

Allowing .before and .after to have access to the TestController will save the execution time of 100 seconds for this fixture. When you add it up across all the fixtures, it does result in TestCafe running slower than had the tests been run in other frameworks that support this.

[qualityshepherd]

Please... I _BEG_ of you... give me a beforeAll. PRETTY PLEASE.

But seriously, do.

[LavrovArtem]

@qualityshepherd, based on your previous comment, you seem to have found a solution using Roles. Therefore, it's not quite clear why you need the beforeAll functionality. Could you please describe your use case in as many details as possible and give us a test example with an API you like to see. It will help us understand your requirements.

[qualityshepherd]

Roles solves logging in over and over again but does not alleviate the need for a beforeAll and afterAll. These are crucial for writing DRY, non-coupled tests that are easy to reason about.

Here's an example of what I would _like_ to do:

import employeePage from "../pages/employeePage";
import { admin } from "../data/roles";
const employeeName = 'Jane Doe';

fixture`Edit Employees`.beforeAll(async (t) => {
  await t.useRole(admin);
  await employeePage.goto();
  await employeePage.addNewEmployee(employeeName);
});

test("should add an employee", async (t) => {
  await t.expect(await employeePage.employeeExists(employeeName)).ok();
});

test("should delete an employee", async (t) => {
  await employeePage.deleteEmployee(employeeName);

  await t.expect(await employeePage.employeeExists(employeeName)).notOk();
});

Here I'm creating a new employee via my UI, asserting that the employee exists, then deleting via the UI and asserting the delete. The tests are not coupled, very DRY, and easy to reason about. Lovely.

Now, because Testcafe only has a beforeEach, I have to add a new employee at the beginning of each test, do the test, and delete the employee after each test. This is obviously much slower and repeats steps/tests.

Could I delete both users in an after via an api call? That assumes an api :) Could I write both tests into one larger test? Maybe... but I prefer singular tests whenever possible, and also, _this is just a small, imperfect example of the overall issue_. Larger, more complex tests would most definitely benefit from a beforeAll

[qualityshepherd]

Honestly... I'm surprised I need to explain wanting a beforeAll. Perhaps y'all can explain your reluctance towards adding it? I mean, in my mind, it is a fundamental part of a test runner.

[chadmyers]

What about my use case of resetting the data source to known good state before each test?

The workaround of having only one test per fixture is onerous and makes me miss out on a lot of the other features of testcafe.

[AndreyBelym]

@qualityshepherd I don't think that a new hook is required for your example.

Please note, that each test starts from a clean state, with no cookies and storages saved from a previous test. So if you want to apply a role for each test in a fixture, t.useRole should be called in each test directly in or in the fixture.beforeEach hook. However, roles execute their initialization actions once on the first time t.useRole and only restores cookies and storages on consecutive calls. It means that roles do not significantly increase the test execution time when called in the beforeEach hook.

For your non-login actions you can use the once function from lodash:

import employeePage from "../pages/employeePage";
import { admin } from "../data/roles";
const employeeName = 'Jane Doe';

const addEmployees = () => {
  await employeePage.goto();
  await employeePage.addNewEmployee(employeeName);
};

const addEmpoyeesOnce = once(addEmployees);

fixture`Edit Employees`.beforeEach(async t => {
  await t.useRole(admin);
  await addEmployeesOnce();
});

//...

If you don't want to use the once function, you can also create a special role that will add an employee along with logging:

function signInAsAdmin () {
    await t.navigateTo(loginPage.url);
    await loginPage.signIn(adminLogin, adminPass);
}

const adminRole = Role(loginPage.url, t => signInAsAdmin());
const adminRoleWithTestUser = Role(employeePage.url, async t => {
   await signInAsAdmin();
   await employeesPage.addNewUser();
});

Now let's talk about the reasons what problems we have to solve to allow using the test controller in hooks that run only once for all browsers, like fixture.before.

1. TestCafe can run each test in a number of browsers in parallel at the same time. Which browser we should use to execute actions specified in the fixture.before hook? The first one? A random one? We can't run them in all browsers without significantly changing our API, because test controllers and browsers have a one-to-one relationship. It allows scaling your tests to any number of browsers while keeping the test syntax simple. When you specify multiple browsers, TestCafe just starts a copy of a test for each browser. In other words, TestCafe parallel flow is a test-based, not an action-based one. Thanks to this, you can use await element.textContent to get the textContent property as a string and use simple ifs to handle different possible values in different browsers. If TestCafe adopted an action-based parallel flow, you would have dealt with an array or a map of values which is significantly harder.

2. Let's take a closer look at the test that deletes an employee:

test("should delete an employee", async (t) => {
  await employeePage.deleteEmployee(employeeName);

  await t.expect(await employeePage.employeeExists(employeeName)).notOk();
});

What if some browser manages to delete an employee before the others? Likely your test will fail in all browsers except the first one. It means that using different accounts and beforeEach hooks is required to run your tests in parallel and significantly decrease your test execution time.

@chadmyers if even we do not allow using test controllers in such all browsers hooks, there is still a problem with managing the global state for multiple browsers. Only cleaning the global state seems harmless enough, but I'm afraid that there will be a lot of tests that also try to populate the test data in such hooks and thus getting in a corrupted state during parallel execution. It's also hard to find a good name for this hook: fixture.beforeAll is more suitable for some global hook that runs before all fixtures, beforeEachOnce is an oxymoron because each and once have contradictory meanings in this context.

If you really want to avoid initializing each browser, you can take the following approach:

const getUserAgent = ClientFunction(() => navigator.userAgent);

fixture`...`
   .beforeEach(t => {
       const userAgent = await getUserAgent();

       if (!t.fixtureCtx.activeUserAgent)
            t.fixtureCtx.activeUserAgent = userAgent;
       else if (t.fixtureCtx.activeUserAgent !=== userAgent)
            return;

       //... Init steps
   });

[chadmyers]

I don't understand what parallelism has to do with fixture setup. It should be run synchronously and all the browsers should wait until the fixture setup is done.

Back to my original post, something like this:

sync( * fixture setup [no text context/browser available] )
parallel(
 * test1 pre-browser setup hook
 * test1 setup hook for Chrome
 * test1 against Chrome
 * test1 setup hook for Firefox
 * test1 against Firefox
 * test2 pre-browser setup hook
 * test2 setup hook for Chrome
 * test2 against Chrome
 * test2 setup hook for Firefox
 * test2 against Firefox
)

[AndreyBelym]

The fixture.before already works this way, it blocks all browsers until its execution is finished. We discuss the test pre-browser setup hook in this thread. It' won't have big parallelism issues if we don't allow to use test controllers in it, same as in the fixture.before hook. All explanations about parallelism were addressed to @qualityshepherd who wants to enable browser actions in pre-browser hooks. I agree that such test pre-browser setup hook that has no test controller access could be useful to perform cleanup operations. The problem is that it can be used by some people to set up test data for the app's global state if we don't find a good name for this hook. A test that tries to set up global state only once will not work properly in a multi-browser environment, because the first browser that executes the test will modify the global state thus affecting test environments in other browsers. It will cause the test to always fail in the best case and fail sometimes without a clearly visible reason in the worst case.

[qualityshepherd]

Yeah, that's the problem with examples... they always become strawman (strawcode?) arguments :)

Honestly, I find the role functionality confusing. Nobody looking at it in use understands it without explanation. I wish I didn't have to use it, which is part (but _only_ a part) of my issue here.

Making tests run in parallel is _my_ job. In my case, I'm use a randomly generated, unique username to avoid test collisions... but now I'm worried that the way Testcafe works (ie _starts a copy of a test for each browser_) that y'all might thwart that? Eg. if I set the random username _outside_ the test, will each browser use the same username? 🤔 That would suck.

[AndreyBelym]

Yes, if you define a variable outside a test in the module scope, it will be shared among all parallelized copies of the test. Honestly, I don't like this part too, but I'm afraid that we can't do much without breaking backward compatibility.

I think that the reason why this issue exists is that we try to adapt the abstraction that is too simple and doesn't reflect our testing flow. It's not a secret that our hooks were designed with mocha hooks in mind, but mocha testing flow is a lot simpler because it doesn't have to deal with multiple test execution contexts for different browser. I guess that life will be easier if we implement an event emitter that will emit events that honestly represent TestCafe's testing flow, like:

• A browser was started
• All browsers are ready
• The test session started
• A fixture started
• A test started
• A test was dispatched to a browser
  <-- here we execute test bodies -->
• A test was executed in a browser
• A test was executed in all browsers
• A fixture was executed
• The test session completed
• A browser was terminated
• All browsers are terminated

Or is it too low-leveled and complex for an end-user API?

[qualityshepherd]

Yeah... bummer... but good to know, and now I will be updating a few of my tests accordingly. This will _almost certainly_ cost someone(s) some debug time in the future ;)

I would love to see emitter/hooks at all of those stages. I don't think it's too low level. I mean, I get that people do some _crazy_ shite with tests that they probably _shouldn't_... but there are always legit reasons to hook into various stages of the flow.

[jaredvincent]

Using something like once() works alright for page actions that you want to perform before all tests, but it doesn't solve the use case where you want to perform page actions after all tests in a fixture.

My use case: I have buttons on a page that only exist when a file has been uploaded. In a beforeAll() I would like to upload said file. Then I would run several tests that are all dependent on that file being uploaded. Finally I would run an afterAll() that would delete that file so as not to leave testing detritus behind.

This can be accomplished with a combination of once() and tacking a regular after() on the last test in the fixture, but that's not very robust.

[AlexKamaev]

@jaredvincent
Thank you for the additional use case.
Please clarify whether you need to emulate file uploading with the TestCafe API or you just need the file to exist on the server.
If you want to emulate file uploading on beforeAll, how are you going to check that the button is visible on each test? Each test starts from the clear page, so you need to upload the file before every test.
Do you want the page to not load between the tests? Please correct me if I misunderstood you.

[elimence]

@AlexKamaev
I have a similar use case as pointed out by @jaredvincent.
Before each test in the fixture is run, I create data (event data) and delete after the test is run.

Currently the best options to implement this is to create a setup test that runs first and a cleanup test that runs last, but this is rather inelegant.

The other option, mentioned above, is to tack on a n after hook to the last test, equally inelegant.
I know next to nothing about testcafe's internals, but it would be super useful to have these once-before and once-after hooks

[LavrovArtem]

@elimence, Thank you for your opinion.

[vicentesaettone09]

Hello everybody. Do you have any news on this ticket?

Thanks in advance.

[miherlosev]

Hi @vicentesaettone09

Thank you for your inquiry. At present, the feature is not implemented.
The current development sprint is already planned. If this feature is important for you, the PR will be greatly appreciated.

[qualityshepherd]

With every test I write... I get more an more annoyed by the lack of a beforeAll/afterAll. It forces you to make bad decisions... _do I write long, multi-assert tests or do I couple them_. I always opt for the latter because the former is more problematic. I _really_ want to see this on the roadmap.

[omartaghlabi]

Is there currently a workaround to login once at the fixture level, and execute multiple tests within that fixture?
I'm currently using the beforeEach which is not ideal as it adds significant time during test runs.

[qualityshepherd]

@omartaghlabi try out Testcafe's User Roles. They're a _little_ janky (sorry guys, but #fact) but they do work. I have an example project where I use them. Take a peek and see if that helps...

[omartaghlabi]

I'll try that out. Thanks @qualityshepherd!

[AlexKamaev]

I researched the issue and created a separate testcafe-once-hook module, which allows you to emulate the required behavior.

Please check the following github repository to see how the module works: https://github.com/AlexKamaev/testcafe-once-hook-example

The testcafe-once-hook module exports two functions:

• oncePerFixture runs in the first specified browser before and after all tests in a fixture. Unlike the fixture.before and fixture.after hooks designed for server-side operations, oncePerFixture allows you to execute test actions in the browser.
• oncePerTest runs in the first specified browser before and after each test in a fixture, while regular beforeEach and afterEach hooks run in every browser.

[qualityshepherd]

Nice... will give it a go... only thing is I also need an onceAfterFixture for cleanup...

[qualityshepherd]

Oh... I get how you're working this now that I've looked at the tests...

[AlexKamaev]

I'll close the issue since now we have a separate module to support the described behavior. If there are any issues that cannot be covered by the module, feel free to describe them here.

[lock[bot]]

This thread has been automatically locked since it is closed and there has not been any recent activity. Please open a new issue for related bugs or feature requests. We recommend you ask TestCafe API, usage and configuration inquiries on StackOverflow.