Pywinauto: Pywinauto doesn't see most of controls while connecting to Skype

[dstepanenko]

On pywinauto 0.6.6 this simple code snippet fails in one of 2 scenarious:

    app = Application(backend="uia").start("skype.exe")
    sleep(8)

    skp = app.window(title_re='Skype.*')
    sleep(8)

    search = skp["Search for people, groups & messages"]
    search.click()

It either fails on line skp = app.window(title_re='Skype.*'):

Traceback (most recent call last):
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\application.py", line 256, in __resolve_control
    criteria)
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\timings.py", line 458, in wait_until_passes
    raise err
pywinauto.timings.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "skype.py", line 300, in <module>
    status = add_friends(prefix, min_mutuals, invitation_msg)
  File "skype.py", line 130, in add_friends
    search.click()
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\application.py", line 362, in __getattribute__
    ctrls = self.__resolve_control(self.criteria)
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\application.py", line 259, in __resolve_control
    raise e.original_exception
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\timings.py", line 436, in wait_until_passes
    func_val = func(*args, **kwargs)
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\application.py", line 201, in __get_ctrl
    dialog = self.backend.generic_wrapper_class(findwindows.find_element(**criteria[0]))
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\findwindows.py", line 87, in find_element
    raise ElementNotFoundError(kwargs)
pywinauto.findwindows.ElementNotFoundError: {'title_re': 'Skype.*', 'backend': 'uia', 'process': 9112}

or fails in the moment when code trying to reach search control using line search = skp["Search for people, groups & messages"]

C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\application.py",` line 256, in __resolve_control criteria)
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\timings.py", line 458, in wait_until_passes
    raise err
pywinauto.timings.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "skype.py", line 300, in <module>
    status = add_friends(prefix, min_mutuals, invitation_msg)
  File "skype.py", line 130, in add_friends
    search.click()
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\application.py", line 362, in __getattribute__
    ctrls = self.__resolve_control(self.criteria)
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\application.py", line 259, in __resolve_control
    raise e.original_exception
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\timings.py", line 436, in wait_until_passes
    func_val = func(*args, **kwargs)
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\application.py", line 220, in __get_ctrl
    ctrl = self.backend.generic_wrapper_class(findwindows.find_element(**ctrl_criteria))
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\findwindows.py", line 84, in find_element
    elements = find_elements(**kwargs)
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\findwindows.py", line 303, in find_elements
    elements = findbestmatch.find_best_control_matches(best_match, wrapped_elems)
  File "C:\Users\Bravissimo\AppData\Local\conda\conda\envs\py36\lib\site-packages\pywinauto\findbestmatch.py", line 533, in find_best_control_matches
    raise MatchError(items = name_control_map.keys(), tofind = search_text)
pywinauto.findbestmatch.MatchError: Could not find 'Search for people, groups & messages' in 'dict_keys(['', 'Pane', 'TitleBar', '0', '1', '2', 'System', 'SystemMenu', 'Menu', 'System0', 'System1', 'System2', 'MenuItem', 'SystemMenuItem', 'Button', 'MinimalizujButton', 'Minimalizuj', 'Button0', 'Button1', 'Button2', 'Maksymalizuj', 'MaksymalizujButton', 'Button3', 'Zamknij', 'ZamknijButton', 'Custom', '3', 'Custom0', 'Custom1', 'Custom2', '4', 'Custom3', '5', 'Custom4', '6', 'Custom5', '7'])

Also in this message you can see that there're only few controls visible, while most of them aren't here. 2 weeks ago this code worked, but now it works very rarely and in most cases fails.

Do you have any idea why it's happening?

[vasily-v-ryabov]

Hi @dstepanenko if pywinauto wasn't updated, the only assumption is Skype update or Windows update changed something.

The code can be customized with more flexible timings this way:

app = Application(backend="uia").start("skype.exe")

skp = app.window(title_re='Skype.*') # actual search is NOT happening here
skp.wait('ready', timeout=20) # wait up to 20 sec. (actual search is done, return early if found)

search = skp["Search for people, groups & messages"]
search.click() # actual search of the sub-element is done here (inside of call to .click())

[dstepanenko]

Hi @dstepanenko if pywinauto wasn't updated, the only assumption is Skype update or Windows update changed something.

The code can be customized with more flexible timings this way:

app = Application(backend="uia").start("skype.exe")

skp = app.window(title_re='Skype.*') # actual search is NOT happening here
skp.wait('ready', timeout=20) # wait up to 20 sec. (actual search is done, return early if found)

search = skp["Search for people, groups & messages"]
search.click() # actual search of the sub-element is done here (inside of call to .click())

Vasily, thank you for the reply. The question isn't what changed, but how to fix it. The thing is that in Inspect I see many controls, edits and other elements, they're visible and enabled, but due to some reasons I can't reach them via pywinauto. Do you have any idea on how to do it? Or maybe you know the reason why it's not possible?

thanks,
Dmitry

[vasily-v-ryabov]

Maybe it's worth to re-install comtypes so that comtypes cache is cleared. If UIAutomationCore.dll was updated, the cache might be outdated.

[dstepanenko]

Maybe it's worth to re-install comtypes so that comtypes cache is cleared. If UIAutomationCore.dll was updated, the cache might be outdated.

Today I installed python 3.6.1 from scratch with all the libraries and rerun the code - still same issue. Any other thoughts?

[dstepanenko]

Vasily, I missed your comment about using waits everywhere. I tried it and it worked (didn't expect my timeouts wasn't enough but it seems they were). Thank you for your help!

P.S.: may I ask you not to close this issue for the next 24 hours so that in case this fix won't work in a stable manner we can go back to this issue?

P.P.S: thank you once again!

[dstepanenko]

Upd: After first successful attempt with waits all the next attempts failed. I increased timeout from 20 seconds for 5 minutes it didn't help

[airelil]

Let me comment with, probably, the most frequent recommendation on this project. Can you see the control and its parent hierarchy in Inspect.exe running in uia mode?

[dstepanenko]

Let me comment with, probably, the most frequent recommendation on this project. Can you see the control and its parent hierarchy in Inspect.exe running in uia mode?

yes, I do. Please go ahead

[vasily-v-ryabov]

Well, the interesting thing is that Skype version 8.45.0.41 runs 4 (four!) processes Skype.exe as you can check in "Details" tab of Task Manager. So connect(path='Skype.exe') won't help.

This code is working (pretty slowly) for me:

from pywinauto.application import Application

app = Application(backend="uia").start(r'"C:\Program Files (x86)\Microsoft\Skype for Desktop\Skype.exe"')
app = Application(backend='uia').connect(title_re='Skype.*') # takes 40.5 seconds!

skp = app.window(title_re='Skype.*')
skp.wait('ready', timeout=20) # takes 1.2 sec.

search = skp["Search for people, groups & messages"]
search.click() # takes 7.1 sec.

The third timing seems OK because Skype main window contains 422 elements and this step can be faster this way: skp.child_window(title="Search for people, groups & messages", control_type='Button', found_index=0) which takes 0.5 sec.

The first timing (~40.5 sec.) looks magic because Desktop(backend="uia").windows() takes 0.4 sec. only. Looks like the problem should be in method .connect().

[vasily-v-ryabov]

If you add .start("skype.exe"), then method Desktop(backend="uia").windows() takes ~40 seconds! So probably the magic happens when Skype launcher checks for existing Skype instance. And this definitely looks like a bug in UIAutomationCore.dll or in other Windows DLLs.

P.S. Inspect.exe hangs for the same time approximately.

[vasily-v-ryabov]

A-ha-ha! The fifth Skype.exe process with pid = 32 eats 100% of one CPU core for this period of time! Come on, Skype team! What are you doing?

[vasily-v-ryabov]

Reported the issue to Skype through standard feedback form with the link to this issue. Let's wait for a month at least for any reaction.

[dstepanenko]

Well, the interesting thing is that Skype version 8.45.0.41 runs 4 (four!) processes Skype.exe as you can check in "Details" tab of Task Manager. So connect(path='Skype.exe') won't help.

This code is working (pretty slowly) for me:

from pywinauto.application import Application

app = Application(backend="uia").start(r'"C:\Program Files (x86)\Microsoft\Skype for Desktop\Skype.exe"')
app = Application(backend='uia').connect(title_re='Skype.*') # takes 40.5 seconds!

skp = app.window(title_re='Skype.*')
skp.wait('ready', timeout=20) # takes 1.2 sec.

search = skp["Search for people, groups & messages"]
search.click() # takes 7.1 sec.

The third timing seems OK because Skype main window contains 422 elements and this step can be faster this way: skp.child_window(title="Search for people, groups & messages", control_type='Button', found_index=0) which takes 0.5 sec.

The first timing (~40.5 sec.) looks magic because Desktop(backend="uia").windows() takes 0.4 sec. only. Looks like the problem should be in method .connect().

I should've mentioned that Skype has several processes - I also noticed it.

I created the script that iterates over all Skype processes trying to connect to each of them and it haven't suceeded also:

    for i in range(4):
      os.system("taskkill /f /im Skype.exe")
    sleep(8)
    app0 = Application(backend="uia").start(r'"C:\Program Files (x86)\Microsoft\Skype for Desktop\Skype.exe"')
    sleep(8)
    skype_processes = []
    c = wmi.WMI ()
    for process in c.Win32_Process ():
      if process.Name == "Skype.exe":
        skype_processes.append(process.ProcessId)

    for i, process in enumerate(skype_processes):
      try:
        app = Application(backend='uia').connect(process=process) # takes 40.5 seconds!
      except:
            print("Failed to find process {0} while connecting. {1}-th process out of {2}".format(process, i, len(skype_processes)))
            continue

      winsound.Beep(frequency, duration)

      skp = app.window(title_re='Skype.*')
      try:
        skp.wait('ready', timeout=40) # takes 1.2 sec.
      except:
            print("Failed to find skype in {0} skype process. {1}-th process out of {2}".format(process, i, len(skype_processes)))
            continue
      winsound.Beep(frequency // 3, duration)

      try:
        search = skp["Search for people, groups & messages"]
        search.wait('ready', timeout=20)
        search.click() # takes 7.1 sec.
        winsound.Beep(frequency // 9, duration)
        print("Sucess with {0} skype process. {1}-th process out of {2}".format(process, i, len(skype_processes)))
        break
      except:
        print("Failed to find search in {0} skype process. {1}-th process out of {2}".format(process, i, len(skype_processes)))

Here are the output of it:

Failed to find search in 9464 skype process. 0-th process out of 5
Failed to find skype in 11720 skype process. 1-th process out of 5
Failed to find skype in 9664 skype process. 2-th process out of 5
Failed to find skype in 14728 skype process. 3-th process out of 5
Failed to find process 13704 while connecting. 4-th process out of 5

so, after connecting to the first skype process (9464) the script is able to find window, but not able to find search button. For the next 3 it's not able to find window with title "Skype.*". The last process is seems to be dead process of previously killed skype.

[dstepanenko]

Reported the issue to Skype through standard feedback form with the link to this issue. Let's wait for a month at least for any reaction.

If you'll receive from microsoft something like link to the opened issue, please share it with me

[vasily-v-ryabov]

Nothing yet. Just a workaround to check already running Skype instance on Python side can be applied so far.

[vasily-v-ryabov]

Application's method .connect(title_re='...', visible_only=False) is what you need. It can raise an exception by timeout if Skype window is not found.