Winappdriver: Large application support

[kfrajtak]

Hello,

I have started using the WinAppDriver for UI testing of a larger application. Locating the element using XPath can take up to 10 seconds (yes, the UI structure is quite complicated). I am using the attach feature since the application does not start very quickly. Currently, I have just a few tests written and I can imagine that the test execution time will grow when more tests are added.

The caching of elements helps quite a lot, but this does not have any effect when the tests are rerun.

So I got the idea that there can be an intermediate driver sitting between the tested application and the test runner that will be kept alive as long as the tested application is alive and do the caching. The new test run will be able to get the elements from the cache.

Can this be achieved using the WinAppDriver only? I assume not and the application in the middle has to implement the JSON wire protocol or behave as a proxy and relay the requests that cannot be cached (and cache misses) to the real driver. Is it possible to reuse the session and cache the elements using this approach?

This is not an issue, just an idea.

Thanks,

Karel

[timotiusmargo]

Hi @kfrajtak,

The idea of caching an element (keeping a reference to a located element) is useful when all of the criteria below are true:

1. It is non-trivial or time consuming to locate
2. It doesn't get invalidated (the UI element doesn't get destroyed) over the entire test scenario
3. It is frequently accessed or it helps locating its children elements

When any of the criteria above is not true, it becomes difficult and unjustifiable to cache a relevant UI element. Internally, WinAppDriver DOES cache any located UI element that is previously located through find element queries. For example, a find element by XPath successfully locates and returns a valid UI element in the application UI tree. This element would then be cached and have an element id that WinAppDriver will keep track of until the application session ends or the element is invalidated.

Would you be able to make use of these behaviors to modify your test scripts such that the sessions and certain located elements are kept alive?

In our WebDriverAPI sample tests, we kept single sessions of Calculator and AlarmClock applications to be shared throughout many test scenarios. As a result, we don't need to spawn new instance of the test application for every test classes or test cases. When you look closer to CalculatorBase application session class, you can also see how we keep a reference to header element that we expect to stay valid throughout the lifetime of the application. As a result, we don't need to locate the same element repeatedly when we are using the same application session. Hope this helps.

[kfrajtak]

Hi @timotiusmargo,

thanks for your reply, I'll try to rewrite the test code base.

Internally, WinAppDriver DOES cache any located UI element that is previously located through find element queries

Does it apply to XPath queries too? I have created a small "benchmark" and the query takes roughly the same time to finish (5-6 seconds each time it is executed).

[timotiusmargo]

Hi @kfrajtak,

You made an excellent point. WinAppDriver does keep a cached reference to the element you successfully located through XPath query. So if you do a nested element search from the element you previously located as you can see in the example here, you can save time by not having to traverse the entire application UI tree.

When searching using XPath query however, WinAppDriver currently constructs the XML document reflecting current UI tree state which is not a cheap operation. While constructing the XML from the given node (XPath search from a given element) is less intensive than constructing it for the entire application, this operation is still time consuming which may have contributed to the 5-6 seconds you saw.

[kfrajtak]

The problem with the application I'm testing is that it has quite complicated UI and even getting the PageSource XML document for a page (MDI tab) with a filter panel can take up to 1 minute and that is the case when the data is not displayed (when PageSource is saved the file is almost 200kB). With the data, the query usually times out after 2 minutes (session timeout). And also, in this case, even the SendKeys operation times out after 2 minutes.

This is an excellent tool, but not everybody is testing Notepad application and the speed will become a real concern here.

Thanks for your time.

[timotiusmargo]

Hi @kfrajtak,

Thank you for your valuable feedback. It is absolutely a valid concern and we will look for ways to improve our XPath performance. For the time being if you really have to rely on XPath to locate the element, you can override the timeout value in Appium client. For example, when writing your test script in C# using appium-dotnet-driver, you can specify the commandTimeout argument when creating a new session as follows:

c# TimeSpan commandTimeout = TimeSpan.FromMinutes(5); session = new WindowsDriver<WindowsElement>(new Uri(WindowsApplicationDriverUrl), appCapabilities, commandTimeout);

This will allow each command to complete before the timeout occurs. By default a 60 seconds timeout is enforced for each command.

[kfrajtak]

I have noticed that timeout set in WebDriverWait does not override the session timeout, i.e. when session timeout is 20 seconds and wait operation timeout is 30 seconds, then the operation times out after 20 seconds.

[dhapolapankaj]

One of the best way to reduce time taken by XPath search is to take reference from nearest parent element.

Suppose you have below heirarchy
P1
P2
P3
Element1, TargetElement, Element3, Element4

So when you use function ParentObject.FindElementByXPath("SomeXPath")
Make sure ParentObject is instance of P3

Regards
PD

[hassanuz]

Hi @kfrajtak,
correct, webdriverwait does not impact session timeout.

[kfrajtak]

Hi,

can you let me know what is the status of XPath query performance? Can you share any details? Are you creating the XML first, then querying it and the locating the element? Or is the tree traversed dynamically without the XML interlude?

Thanks

[hassanuz]

Hi @kfrajtak,

Yes, your understanding is correct--this is something we're looking into.

[kfrajtak]

Thank you.

[kfrajtak]

Do you have any information on when the project will be open sources and source code available? I bet number of people interested in WinAppDriver will contribute to solve this issue (unless it's already done :))

[kfrajtak]

I gave WinAppDriver another try and a very simple XPath query in the large app I'm trying to test is timing out after 60 seconds. This time with the latest version 1.1 and it was working in previous version though it was quite slow.

So I put together (https://github.com/kfrajtak/WinAppDriver) some projects (Nancy, XPathParser) and the same query is executed in about 2 seconds. Similar query broken into separate steps and executed step by step using AutomationElement basic methods (FindAll and FindFirst) is executed under 1 second.

My question is where is that overhead in WinAppDriver query execution coming from and if it will be adressed? I have the feeling the driver is creating XML document from all the elements in the app (or form) and then perform a XPath query on this document.

Thank you.

Edited:
My apologies, I have also tried WinAppDriver on Outlook to get the action buttons of the currently open message (Reply, Reply All, Forward) with following XPath query /Window/Pane/Pane/Pane/Pane/Pane/Pane/Group/Button and got the response back in 12 seconds. However, there's some overhead when executing that query against my application when the UI gets more complicated, but don't know how to find out where ...

[hassanuz]

Hi @kfrajtak,

If you're using UI Recorder to generate your XPath queries, then we recommend updating to the latest WinAppDriver v1.1.1 patch as that greatly improves XPath handling(specifically for Absolute XPath that is generated by UI Recorder).

[kfrajtak]

Hi, no I am not using UI recorder.

[kfrajtak]

This thread is unfortunately not going to be resolved. In the meantime I have written my own driver, the source code is available under my account and it is open for anyone.

[ashkaps]

Can the maintainers explore use of another xpath query like @kfrajtak indicated. It looks like its greatly improves the efficiency of xpath search.