Testcafe: Allow roles preserve page

[inikulin]

Are you requesting a feature or reporting a bug?

feature

What is the current behavior?

Test run redirected back to original URL after role switch.

What is the expected behavior?

Add option to preserve last URL that was used after Role initialization. See: https://github.com/DevExpress/testcafe/issues/243#issuecomment-287876990

[p-bakker]

Should it preserve the url the login action ended on and go back to that everytime useRole(role) is called?

Or should the url be preserved on which the role was before switching to another role and using the url on which the login action ended only when it's the first time switching to that role?

This would imho make more sense or at least be another useful option. Think of stateful apps: going back to the url as it was after login might not be desired or even wanted

[p-bakker]

To add to my comment above: I realized when testing out the workaround https://github.com/DevExpress/testcafe/issues/243#issuecomment-288326324 that before each test, TestCafe navigates back to the 'main' page.

This does not work for the scenario in which I'm trying to use TestCafe: I'm trying to automate what I call a process test, which is testing a webapp front to back, going through dozens of steps, where each step builds on the previous step. That approach, combined with the framework I'm using, means that each test needs to continue from the URL where the previous test ended.

This currently isn't TestCafe's behavior and the behavior described in this case to switch back to the URL on which the login action ended before each test would also not give me the behavior I need.

I've tried to work around this using the beforeEach and afterEach fixture hooks, by storing the current URL to the t.ctx in the afterEach hook and navigating to t.ctx in the beforeEach hook, but TestCafe first executed the beforeEach hook and then navigates to the 'main' page, so the next test starts at the wrong URL for me.

The only way I currently see to automate my process test is to just have one test function that contains all the steps involved in the entire process test, which I don't like, since:
a: this makes is very difficult to read (the test method would become hundred if not thousands LOC long)
b: doesn't give the the execution time per individual step (which I would also like to track).

So, long story short: implementing this feature as (currently) described would not solve my problem. Instead, I'd need an option to only provide a starting url per fixture or role) and have TestCafe just continue each test from the URL where the previous test (or the login action) ended.

I hope you see this as a useful and possible addition, although I do understand that from a generic testing point of view it makes sense to start of each test from the same main/base url, since tests should not affect eachother. However, for the particular type of test I'm trying to automate, this won't work

[p-bakker]

@inikulin Curious to know if you/the TestCafe team sees options/are willing to support our testing approach, where each test within the fixture should continue where the previous test finished (state/url-wise).

[inikulin]

@p-bakker I believe that we shouldn't implement state sharing between tests. If we do so, we will introduce ambiguity:

• what should happen if one test fails, does it makes sense to continue the task? What if you have tests that doesn't share URL, should we continue running them?
  
  
  
  • which API should we use to show that tests should be run in sequence on the same page without isolation from each other?
  
  

Therefore, taking in the account this requirements:

a: this makes is very difficult to read (the test method would become hundred if not thousands LOC long)
b: doesn't give the the execution time per individual step (which I would also like to track).

I believe we should implement named sections for tests:

test('Some test')
   .section('Login', async t => {
       //...
   })
   .section('Do smthg', async t => {
      //...
   })
   .section('Do smthg else', async t => {
      //...
   });

It will still be a single tests, however each section will be reported separately on completion as they progress. Failing single section will fail whole test (which I guess makes sense in your scenario).

[inikulin]

@p-bakker What do you think about this approach?

[p-bakker]

@inikulin Hi Ivan, been thinking about it and have a couple of thoughts:

1. your approach works for me from a testcode structuring point of view
2. I wonder if such a section structure isn't going to cause issues with output formats like xunit for example. I rather not loose the ability to have the output of tests generated in 'standard' formats that can be consumed by external tools
3. how would this interact with the useRole feature? I'd still need the ability to continue after a useRole call on the url where that role was last.

Regarding the questions you had on my suggestion:

1. what should happen when tests fails: there is already a separate FR for bailing out on first failure https://github.com/DevExpress/testcafe/issues/1323
2. what API to use: I'd have suggested something like a continueWithPreviousUrl flag on the fixture

[inikulin]

@p-bakker

I wonder if such a section structure isn't going to cause issues with output formats like xunit for example. I rather not loose the ability to have the output of tests generated in 'standard' formats that can be consumed by external tools

For xunit it will be reported as `'.

how would this interact with the useRole feature? I'd still need the ability to continue after a useRole call on the url where that role was last.

You just login in the first section of the test and subsequent steps happen on the same page without reload.

what should happen when tests fails: there is already a separate FR for bailing out on first failure #1323

Yes, but it's runner option, what if will not enable it and I have mixed tests; some of them use the same page and others not. How we will TestCafe that tests should reuse the page?

what API to use: I'd have suggested something like a continueWithPreviousUrl flag on the fixture

The question is "previous to what"? It depends on what set of test files/tests (there can be skipped ones) were run. Too much ambiguity.

[p-bakker]

@inikulin
I think something went missing from your answer on xunit, so not sure what your answer is

As for your answer on how your .section approach will work in combination with useRole: so you suggest to also still add the feature described in your first comment on this case, besides the .section approach?

I think those two features combined would make it possible to do what i need to do in a maintainable fashion. At least I can't think of any loopholes right now

[inikulin]

p-bakker Yeah, forgot to close code section and it was removed by sanitizer =)

For xunit it will be:

<testcase classname="fixtureName" name="testName-sectionName" ...>

As for your answer on how your .section approach will work in combination with useRole: so you suggest to also still add the feature described in your first comment on this case, besides the .section approach?

You will not need this Role feature, since sections are technically a single test (no page reloads happen in between), they just help you arrange test code and you will have them reported separately.

[mjhea0]

+1 for the sections

[p-bakker]

@inikulin

Sounds good for the xunit handling.

As for the useRole: I still see requirements to be able to switch roles during the a single test, in order to test certain things from different security/role perspectives. But I guess I can handle that with the work-around you suggested earlier by capturing and storing the url at the appropriate times against a role and navigating back to the captured url when switching to the Role again.

It all takes some coordination, but I think I could get everything done and as long as I can get my testers to write tests using my domain-specific API in which I handle all the intricacies of dealing with statefull urls, things would be good

[inikulin]

But I guess I can handle that with the work-around you suggested earlier by capturing and storing the url at the appropriate times against a role and navigating back to the captured url when switching to the Role again.

Oh, I see now. Makes sense. I'll create separate ticket for sections then and advance this one to the current sprint.

[p-bakker]

k, great!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs or feature requests. For TestCafe API, usage and configuration inquiries, we recommend asking them on StackOverflow.