Pywinauto: Unable to retrieve all information from ComboBox selection

[harryhobson]

I'm new to pywinauto and uia, but I'm hoping you can help me with an issue.

I'm trying to retrieve information about the selected item in some sort of ComboBox. I've noticed that the ComboBox items don't populate until I run expand(), but I'm only interested in what's currently displayed (see image below). Inspect.exe shows me that ComboBoxEdit has three children, all three children correspond to the selected item it seems - not to a combobox list. But pywinauto only detects the first child 'Image', not the other two pieces of text.

Is there any way I can retrieve the other two children "FC 123-456789" and "123-456789"? I'm wondering if this is a non-standard combobox that pywinauto doesn't understand.

The following code gives control_count = 1. I thought the missing children might be accessible in the 'Image' control but I haven't found anything.

from pywinauto.application import Application

app = Application(backend="uia").connect(path="Program.exe")

# be picky about the title in case two windows are open
dlg = app.window(title_re=".*Program.*")

cbo = dlg.ComboBox

print(cbo.wrapper_object().control_count())

Inspect.exe:

How found:  Mouse move (299,838)
    hwnd=0x000A0E1E 64bit class="HwndWrapper[Program.exe;;15de3ec8-8cf3-4e77-a2af-ef9b415fa0a8]" style=0x17CF0000 ex=0x40100
RuntimeId:  "[7.12152.24717373]"
BoundingRectangle:  {l:39 t:824 r:360 b:864}
ProcessId:  12152
ControlType:    UIA_ComboBoxControlTypeId (0xC353)
LocalizedControlType:   "combo box"
Name:   ""
AcceleratorKey: ""
AccessKey:  ""
HasKeyboardFocus:   true
IsKeyboardFocusable:    true
IsEnabled:  true
AutomationId:   ""
ClassName:  "ComboBoxEdit"
HelpText:   ""
ClickablePoint: {x:199 y:844}
IsControlElement:   true
IsContentElement:   true
IsPassword: false
ItemType:   ""
IsOffscreen:    false
Orientation:    0
FrameworkId:    "WPF"
IsRequiredForForm:  false
ItemStatus: ""
ProviderDescription:    "[pid:12152,hwnd:0x0 Main(parent link):Unidentified Provider (managed:MS.Internal.Automation.ElementProxy, PresentationCore, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)]"
Value.IsReadOnly:   false
Value.Value:    "MIP.Platform.Domain"
Selection.Selection:    
Selection.CanSelectMultiple:    false
Selection.IsSelectionRequired:  false
ExpandCollapse.ExpandCollapseState: Collapsed (0)
IsDockPatternAvailable: false
IsExpandCollapsePatternAvailable:   true
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsMultipleViewPatternAvailable: false
IsRangeValuePatternAvailable:   false
IsScrollPatternAvailable:   false
IsScrollItemPatternAvailable:   false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    true
IsTablePatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTextPatternAvailable: false
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsValuePatternAvailable:    true
IsWindowPatternAvailable:   false
IsItemContainerPatternAvailable:    false
IsVirtualizedItemPatternAvailable:  false
FirstChild: "" image
LastChild:  "123-456789" text
Next:   "Incubator: Off" text
Previous:   "" custom
Other Props:    Object has no additional properties
Children:   "" image
    "FC 123-456789" text
    "123-456789" text
Ancestors:  "" custom
    "Session tree" group
    "" custom
    "PART_ContentPresenter" group
    "script v0.1" group
    "DocumentGroup" tab
    "LayoutGroup" group
    "DockLayoutManager" group
    "BarManagerbarManager" group
    "" custom
    "script v0.1 - Program" window
    "Desktop" pane
    [ No Parent ]

dump_tree excerpt:

   |    |    |    |    |    |    |    |    |    | Custom - ''    (L39, T824, R360, B995)
   |    |    |    |    |    |    |    |    |    | ['99', 'Custom41']

   |    |    |    |    |    |    |    |    |    |    | 
   |    |    |    |    |    |    |    |    |    |    | Custom - ''    (L0, T0, R0, B0)
   |    |    |    |    |    |    |    |    |    |    | ['100', 'Custom42']

   |    |    |    |    |    |    |    |    |    |    | 
   |    |    |    |    |    |    |    |    |    |    | ComboBox - ''    (L39, T824, R360, B864)
   |    |    |    |    |    |    |    |    |    |    | ['101', 'ComboBox', 'MIP.Platform.Domain']

   |    |    |    |    |    |    |    |    |    |    |    | 
   |    |    |    |    |    |    |    |    |    |    |    | Image - ''    (L44, T829, R74, B859)
   |    |    |    |    |    |    |    |    |    |    |    | ['102', 'Image20']

[airelil]

First, we have to include "CombBoxEdit" to list of control types to create ComboBoxWrapper automatically. However, you can subclass it like in the example below. Once you have the ComboBox functionality you can try to select the item before accessing its children.
Something like this:

from pywinauto.application import Application
from pywinauto.controls.uia_controls import ComboBoxWrapper


class ComboBoxEditWrapper(ComboBoxWrapper):

    """Wrap a UIA CoboBoxEdit control"""

    _control_types = ['ComboBoxEdit']

    def __init__(self, elem):
        """Initialize the control"""
        super(ComboBoxEditWrapper, self).__init__(elem)


app = Application(backend="uia").start(r"apps\WPF_samples\x64\WpfApplication1.exe")
dlg = app.window(title_re=".*WPF Sample.*")
cbo = dlg.ComboBox
dlg.ComboBox.select(0)  # index of a combobox item
selection = cbo.get_selection()
for itm in selection:  # the returned selection is a list
    for c in itm.children():
        print(c)
app.kill()

BTW, ComboBoxWrapper.select method will make expand/collapse operations that you mentioned, this is to populate the combobox.

[airelil]

@vasily-v-ryabov, it seems that there is a bunch of WPF controls from DevExpress. Should we add these third-party controls like: 'ComboBoxEdit' or 'TreeListControl' to our control types ?

[vasily-v-ryabov]

@airelil it looks like ControlType: UIA_ComboBoxControlTypeId (0xC353) is correct to detect it as ComboBox. Class name string matters for Win32 backend only as I remember.

Hi @harryhobson maybe .children(content_only=True) would help. Because these texts are content elements, not a control ones.

[harryhobson]

Thanks for the comments and suggestions

When I try dlg.ComboBox.select(0) I get the below exception. I believe this is due to item 0 being the currently selected item, but I will try to test that later this week.

runfile('C:/Users/harry/Documents/python/combobox.py', wdir='C:/Users/harry/Documents/python')
Traceback (most recent call last):

  File "<ipython-input-56-78d736bbd92d>", line 1, in <module>
    runfile('C:/Users/harry/Documents/python/combobox.py', wdir='C:/Users/harry/Documents/python')

  File "C:\Users\harry\AppData\Local\Continuum\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 710, in runfile
    execfile(filename, namespace)

  File "C:\Users\harry\AppData\Local\Continuum\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/harry/Documents/python/combobox.py", line 23, in <module>
    cbo = dlg.ComboBox.select(0)

  File "C:\Users\harry\AppData\Local\Continuum\Anaconda3\lib\site-packages\pywinauto\controls\uia_controls.py", line 160, in select
    self._select(item)

  File "C:\Users\harry\AppData\Local\Continuum\Anaconda3\lib\site-packages\pywinauto\controls\uiawrapper.py", line 606, in _select
    wrp.iface_selection_item.Select()

  File "C:\Users\harry\AppData\Local\Continuum\Anaconda3\lib\site-packages\pywinauto\controls\uiawrapper.py", line 129, in __get__
    value = self.fget(obj)

  File "C:\Users\harry\AppData\Local\Continuum\Anaconda3\lib\site-packages\pywinauto\controls\uiawrapper.py", line 221, in iface_selection_item
    return uia_defs.get_elem_interface(elem, "SelectionItem")

  File "C:\Users\harry\AppData\Local\Continuum\Anaconda3\lib\site-packages\pywinauto\uia_defines.py", line 219, in get_elem_interface
    raise NoPatternInterfaceError()

NoPatternInterfaceError

Similarly, when I run dlg.ComboBox.get_selection() I get an empty list [], even though the ComboBox has a selection as per the image in my first post.

Both dlg.ComboBox.children() and dlg.ComboBox.children(content_only=True) return:
[]
Inspect.exe detects three children (1 image field and 2 text fields) but I can only see the image field in pywinauto. Any more suggestions to detect the other two children? I don't think these children are Combobox items (part of a combobox list), since they are all related to the selected item.

A side note that may be related to the above. Expanding the ComboBox actually creates a listbox:

# this creates a listbox
dlg.ComboBox.expand()

# the created listbox can be accessed like this
lbo = dlg.ListBox

Listbox in Inspect.exe

How found:  Mouse move (271,900)
    hwnd=0x009D0826 64bit class="HwndWrapper[Program.exe;;6b71aa24-c1f5-428a-9628-9b632947d733]" style=0x96000000 ex=0x8080088
RuntimeId:  "[7.6560.63585586]"
BoundingRectangle:  {l:39 t:869 r:358 b:911}
ProcessId:  6560
ControlType:    UIA_ListControlTypeId (0xC358)
LocalizedControlType:   "list view"
Name:   ""
AcceleratorKey: ""
AccessKey:  ""
HasKeyboardFocus:   false
IsKeyboardFocusable:    false
IsEnabled:  true
AutomationId:   "PART_Content"
ClassName:  "ListBox"
HelpText:   ""
ClickablePoint: {x:198 y:890}
IsControlElement:   true
IsContentElement:   true
IsPassword: false
ItemType:   ""
IsOffscreen:    false
Orientation:    0
FrameworkId:    "WPF"
IsRequiredForForm:  false
ItemStatus: ""
ProviderDescription:    "[pid:6560,hwnd:0x0 Main(parent link):Unidentified Provider (managed:MS.Internal.Automation.ElementProxy, PresentationCore, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)]"
Scroll.HorizontalScrollPercent: -1.000000
Scroll.HorizontalViewSize:  100.000000
Scroll.VerticalScrollPercent:   -1.000000
Scroll.VerticalViewSize:    100.000000
Scroll.HorizontallyScrollable:  false
Scroll.VerticallyScrollable:    false
Selection.Selection:    "MIP.Platform.Domain" list item
Selection.CanSelectMultiple:    false
Selection.IsSelectionRequired:  false
IsDockPatternAvailable: false
IsExpandCollapsePatternAvailable:   false
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsMultipleViewPatternAvailable: false
IsRangeValuePatternAvailable:   false
IsScrollPatternAvailable:   true
IsScrollItemPatternAvailable:   false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    true
IsTablePatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTextPatternAvailable: false
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsValuePatternAvailable:    false
IsWindowPatternAvailable:   false
IsItemContainerPatternAvailable:    true
IsVirtualizedItemPatternAvailable:  false
FirstChild: "MIP.Platform.Domain" list item
LastChild:  "MIP.Platform.Domain" list item
Next:   "Button" button
Previous:   [null]
Other Props:    Object has no additional properties
Children:   "MIP.Platform.Domain" list item
Ancestors:  "" window
    "Harry1 (1) - Program" window
    "Desktop" pane
    [ No Parent ]

[airelil]

Did you try to work with the listbox ? What children and descendants of the list are returned ?

[harryhobson]

Here is the listbox using dlg.ListBox.dump_tree():

ListBox - ''    (L0, T0, R0, B0)
['', 'ListBox', '0', '1']
child_window(auto_id="PART_Content", control_type="List")

   | 
   | ListItem - 'MIP.Platform.Domain'    (L0, T0, R0, B0)
   | ['ListItem', 'MIP.Platform.Domain', 'MIP.Platform.DomainListItem', 'ListItem0', 'ListItem1', 'MIP.Platform.Domain0', 'MIP.Platform.Domain1', 'MIP.Platform.DomainListItem0', 'MIP.Platform.DomainListItem1']
   | child_window(title="MIP.Platform.Domain", control_type="ListItem")

   |    | 
   |    | Image - ''    (L0, T0, R0, B0)
   |    | ['2', 'Image', 'Image0', 'Image1']
   |    | child_window(auto_id="PART_Image", control_type="Image")

   |    | 
   |    | Static - 'FC 123-456789'    (L0, T0, R0, B0)
   |    | ['FC 123-456789', 'Static', 'FC 123-456789Static', 'Static0', 'Static1']
   |    | child_window(title="FC 123-456789", control_type="Text")

   | 
   | ListItem - 'MIP.Platform.Domain'    (L0, T0, R0, B0)
   | ['ListItem2', 'MIP.Platform.Domain2', 'MIP.Platform.DomainListItem2']
   | child_window(title="MIP.Platform.Domain", control_type="ListItem")

   |    | 
   |    | Image - ''    (L0, T0, R0, B0)
   |    | ['3', 'Image2']
   |    | child_window(auto_id="PART_Image", control_type="Image")

   |    | 
   |    | Static - 'SIMULATOR_FC'    (L0, T0, R0, B0)
   |    | ['SIMULATOR_FC', 'Static2', 'SIMULATOR_FCStatic']
   |    | child_window(title="SIMULATOR_FC", control_type="Text")

   | 
   | ListItem - 'MIP.Platform.Domain'    (L0, T0, R0, B0)
   | ['ListItem3', 'MIP.Platform.Domain3', 'MIP.Platform.DomainListItem3']
   | child_window(title="MIP.Platform.Domain", control_type="ListItem")

   |    | 
   |    | Image - ''    (L0, T0, R0, B0)
   |    | ['4', 'Image3']
   |    | child_window(auto_id="PART_Image", control_type="Image")

   |    | 
   |    | Static - 'FC 234-567890'    (L0, T0, R0, B0)
   |    | ['FC 234-567890', 'Static3', 'FC 234-567890Static']
   |    | child_window(title="FC 234-567890", control_type="Text")

When I run print(dlg.ListBox.texts()) I get:

[['', 'FC 123-456789'], ['', 'SIMULATOR_FC'], ['', 'FC 234-567890']]

However when I try to select the first item lbo = dlg.ListBox.item(0).select() the listbox collapses and I receive the Exception:

  File "C:\Users\harry\AppData\Local\Continuum\Anaconda3\lib\site-packages\pywinauto\controls\uiawrapper.py", line 535, in select
    self.iface_selection_item.Select()

COMError: (-2146233079, None, (None, None, None, 0, None))

Most functions I call will collapse the ListBox before they can complete, giving an error that the control cannot be found. Therefore I can't run dlg.dump_tree() because the ListBox disappears beforehand. dlg.ListBox.dump_tree() works and gives the above control information.

So there are two issues at the moment. Firstly it's retrieving the information of the selected item in the ComboBox (the image and two text children), the second is doing actions with the listview (like selecting items).

I'll check if the DevExpress demos have the ComboBoxEdit control, they may be easier to analyse.

[airelil]

Yes, a DevExpress demo with a set of their controls could be interesting to check, but I didn't find a standalone pack of their binaries. Looks as it's coming only as a 30-day trial, together with a full blown installation. Now, I'd suggest you to continue working with Inspect.exe in UIA mode. It should help you to locate the ListBox control. As a side note (and you probably noticed it), often, ListView (or ListBox) is not a direct descendant of ComboBox and can reside in a completely different pane/window. Also, from the Inspect report we can see IsSelectionItemPatternAvailable: false this may explain why elf.iface_selection_item.Select() fails. I see that you tried to get an item of the list box, but I'm not sure how it works as we don't have item() method for base_wrapper and UIAWrapper or wrappers deriving from it. However, you could try to leverage IsItemContainerPatternAvailable: true with dlg.ListBox.get_item(0). Also it's may worth keeping a wrapper of the ListBox and trying to work through it:

wrp = dlg.ListBox.wrapper_object()
itm = wrp.get_item(0)
print(itm.texts())

I believe, that to be able to select an item you have to verify with Inspect utility if ListItem (not ListView, but one of its items) has IsSelectionItemPatternAvailable: true

[harryhobson]

I began experimenting with FlaUI and might have made some progress on this issue.

FlaUI has a wiki section that was able to fix the problem of the 2 TextBlocks not appearing in the image in my first post. My problematic textblocks have IsControlElement=false and IsContentElement=false, but the RawViewWalker is able to find these elements. The relevant code for finding these elements with FlaUI is here.

A few weeks back, I tried to replicate the pywinauto element finding method in FlaUI. This involved running the AutomationElement.FindAll method like from here. I think I found that the FindAll method does not return these TextBlock elements, but RawViewWalker does. So it's an issue with automationelement.findall, not a bug in pywinauto.

Any chance you would be able to implement RawViewWalker for finding elements? I have no idea what other issues this may cause... I would like to give this a crack myself but with job changes I haven't found the time. I would like to move my automation project from FlaUI/C# to pywinauto/Python at some stage in the next 6 months. Will give it a crack then if you don't find time.

Don't have an update on the listbox item selection error at this stage

[vasily-v-ryabov]

Hi @harryhobson thanks for your findings. It should be pretty easy to re-implement pywinauto's search algorithm using RawViewWalker. It's available by pywinauto.uia_defines.IUIA().iuia.RawViewWalker. Not sure about full replacement of the current algorithm: need more tests for different GUI samples and some performance comparison probably. If you were able to add some test(s) with DevExpress sample app (I think a separate line in test matrix with one Python only should be enough for future wide variety of apps compatibility testing), this would be really useful for the project.

We were too busy at the main jobs this half of the year. But we are usually able to review new pull requests on a few days basis.

P.S. At the first half of 2018 the main focus is on Linux AT-SPI support and text based "record-replay" feature.

[vasily-v-ryabov]

By the way, pywinauto.uia_defines.IUIA().iuia.CreateTreeWalker can help to optimize descendants(...) with conditions.

[jethro33]

Hello there,

I've been using pywinauto for a couple of days now and I've actually encountered a similar situation as @harryhobson since I have troubles accessing the items within a ComboBox. @vasily-v-ryabov are there any news about this issue?

[vasily-v-ryabov]

Hi @jethro33 no news yet. We had no chance to try RawTreeWalker. Currently we're working on Linux backend finalization. Also I spent this summer on macOS code that is ~40% implemented.

RawTreeWalker relates mostly to speed optimization. So it's not a high priority for now.

[jethro33]

Hi @vasily-v-ryabov thank you for your reply. I was able to use the alternate solution proposed by @harryhobson anyway by doing something like this:

dlg["ComboBox control ID name"].expand()
dlg["ComboBox item control ID name"].select()

It works, it's slow but it work. Just remarking in case somebody encounters a similar issue. The only limitation is the fact that you need the text information of the ComboBox which can not always be the case.