Pywinauto: ctypes.ArgumentError @ click_input

[Enteleform]

Windows 10.0.15063 x64
Python 3.6.2
pywinauto 0.6.3

 
Setup (functional):

hwnd = my_kivy_app.get_hwnd()

app = pywinauto.Application()
app.connect(handle=hwnd)
window = app.window(handle=hwnd).wrapper_object()

 
Exception @:

window.click_input(button='left', pressed='', coords=(100, 100), double=False, absolute=False)

Traceback (most recent call last):
  File "Test_Window_Commands.py", line 114, in <module>
    app.run()
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\app.py", line 828, in run
    runTouchApp()
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\base.py", line 504, in runTouchApp
    EventLoop.window.mainloop()
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\core\window\window_sdl2.py", line 663, in mainloop
    self._mainloop()
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\core\window\window_sdl2.py", line 405, in _mainloop
    EventLoop.idle()
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\base.py", line 339, in idle
    Clock.tick()
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\clock.py", line 581, in tick
    self._process_events()
  File "kivy\_clock.pyx", line 367, in kivy._clock.CyClockBase._process_events (kivy\_clock.c:7700)
  File "kivy\_clock.pyx", line 397, in kivy._clock.CyClockBase._process_events (kivy\_clock.c:7577)
  File "kivy\_clock.pyx", line 395, in kivy._clock.CyClockBase._process_events (kivy\_clock.c:7498)
  File "kivy\_clock.pyx", line 167, in kivy._clock.ClockEvent.tick (kivy\_clock.c:3483)
  File "C:\OneDrive\_Projects\Python\dev\kivy\basic\_app.py", line 140, in _wrapped
    function(*args, **kwargs)
  File "Test_Window_Commands.py", line 104, in ctrl_shift_alt
    app._window.click_input(button='left', pressed='', coords=(100, 100), double=False, absolute=False)
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\pywinauto\base_wrapper.py", line 667, in click_input
    coords = self.client_to_screen(coords)
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\pywinauto\base_wrapper.py", line 333, in client_to_screen
    rect = self.element_info.rectangle
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\pywinauto\win32_element_info.py", line 142, in rectangle
    return handleprops.rectangle(self.handle)
  File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\pywinauto\handleprops.py", line 200, in rectangle
    win32functions.GetWindowRect(handle, ctypes.byref(rect))
ctypes.ArgumentError: argument 2: <class 'TypeError'>: expected LP_RECT instance instead of pointer to RECT

[airelil]

@Enteleform, does it work if you enable absolute coordinates ?
window.click_input(button='left', pressed='', coords=(100, 100), double=False, absolute=True)

[Enteleform]

That also causes an exception:

 Traceback (most recent call last):
   File "Test_Window_Commands.py", line 161, in <module>
     app.run()
   File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\app.py", line 828, in run
     runTouchApp()
   File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\base.py", line 479, in runTouchApp
     EventLoop.start()
   File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\base.py", line 166, in start
     provider.start()
   File "C:\OneDrive\_Frameworks\Python\3\lib\site-packages\kivy\input\providers\wm_touch.py", line 68, in start
     self.hwnd, GWL_WNDPROC, self.new_windProc)
 ctypes.ArgumentError: argument 3: <class 'TypeError'>: wrong type

[vasily-v-ryabov]

Hi @Enteleform do you use some side libraries along with pywinauto? Can you try the same on clean Python + pywinauto and its dependencies only? Our users saw similar issue when using Lackey + pywinauto.

[Enteleform]

Just checked & it works fine by itself. I'm attempting to use it with Kivy.

I had some similar issues with the keyboard library related to pywin32 & how it handles threads, but managed to resolve them by using Kivy's clock module.

The same approach did not work for pywinauto.

Was a working solution found for Lackey?

[airelil]

I don't know about Lackey, @vasily-v-ryabov sure has more info about it, but I'd suggest you to give a try with another backend: app = Application(backend='uia'). Notice though, UIA has problems with threading, so all pywinauto operations should be run from the same thread.

[vasily-v-ryabov]

OK, Kivy is installed on the same Python version as pywinauto? I'd like to try it on my side. Are you using pywinauto inside your Kivy app? I mean the same Python process.

@airelil another backend won't help because .click_input() implementation doesn't depend on a backend. It uses common function pywinauto.mouse.click(...).

[Enteleform]

OK, Kivy is installed on the same Python version as pywinauto?

Yes.

 

I'd like to try it on my side. Are you using pywinauto inside your Kivy app?

Yes, here's an MVCE:

import random
import pywinauto
from kivy.app           import App
from kivy.lang          import Builder
from kivy.core.window   import Window
from kivy.uix.boxlayout import BoxLayout


class DemoLayout(BoxLayout): pass
Builder.load_string("""
#: import datetime  datetime.datetime
<DemoLayout>:
  padding: 75

  Button:
    on_press: print(f"PRESSED @ {datetime.now()}")
""")


class Demo(App):

  def build(self):
    self.root = DemoLayout()

  def on_start(self):
    title = f"__KIVY_APP__{random.getrandbits(128)}"
    Window.set_title(title)
    hwnd = pywinauto.findwindows.find_window(title=title)
    app = pywinauto.Application()
    app.connect(handle=hwnd)
    window = app.window(handle=hwnd).wrapper_object()
    window.click_input(button="left", pressed="", coords=(100, 100), double=False, absolute=False)


Demo().run()

[vasily-v-ryabov]

Thanks for the snippet! Will try to take a look. Can you also try to import pywinauto after kivy? I'm curious if it helps. Suspect Kivy changes something at import time. At the same time I'm working to fix the first issue using win32gui.GetWindowRect function instead of ctypes.windll.user32.GetWindowRect.

The issue in your second post looks like a crash inside Kivy... The traceback doesn't highlight pywinauto modules.

[Enteleform]

Can you also try to import pywinauto after kivy?

Tried it, and the same issue occurs.

 

I'm working to fix the first issue using win32gui.GetWindowRect function instead of ctypes.windll.user32.GetWindowRect.

That seems like it should work. I managed to get the same functionality working in Kivy without any issues using:

• win32gui.GetWindowRect and/or the GetWindowRect method of pywin32's PyCWnd object
• the mouse library, which uses
  user32 = ctypes.WinDLL('user32', use_last_error=True)
user32.mouse_event

Maybe creating a user32 instance with ctypes.WinDLL instead of using ctypes.windll.user32 will help?

[vasily-v-ryabov]

I'm getting back to the old issues and I could reproduce the original issue with GetWindowRect. The fix is here (in master branch already, just verified it): https://github.com/pywinauto/pywinauto/commit/3a24b360220515f0e84b69af8962096bb28bccac#diff-a0da689e9deac595ceaf85dd5d6e8fa1

@Enteleform if it's still relevant, do you have any more pywinauto issues related to this project?

[Enteleform]

@vasily-v-ryabov

Haven't been Pythoning much lately, can't even remember what I was doing when I ran into that issue or what project it's from lol.

Actually just started getting back into it a few days ago though, so I'll let you know if I run into anything.

[CristiFati]

• ctypes.ArgumentError: argument 2: <class 'TypeError'>: expected LP_RECT instance instead of pointer to RECT
  

  (worked around by win32gui.GetWindowRect) was because both modules had their own RECT (in terms of) definition, and prototype specified one but the other was passed.
  I think if both classes would have extended wintypes.RECT (without redefining _fields_, of course), and windll.user32.GetWindowRect's last argtype would have been POINTER(wintypes.RECT), both could have worked with ctypes

  • ```python
    ctypes.ArgumentError: argument 3: : wrong type
    ````
    is a bug on Kivy's side: [GitHub]: kivy/kivy - SetWindowLongPtrW ctypes prototype bug

[vasily-v-ryabov]

Hi @CristiFati thanks for nice investigation! It's time to re-open this bug. Re-using wintypes.RECT._fields_ makes sense for sure.

Post on StackOverflow would be useful for history:
https://stackoverflow.com/q/55928463/3648361

[CristiFati]

@vasily-v-ryabov: Thanks for the heads-up.

I also have a question: I'm not sure what the policy regarding dependent modules is, but I noticed that pywin32 is used in many other places (it wasn't introduced specifically for this fix). I mean that both approaches (pywin32 vs. ctypes) do the same thing, but with the former there's a lot of code that needs to be written (as it was already written there). Of course, there are WINAPIs that pywin32 doesn't wrap, and for those ctypes is the only way.

[airelil]

@CristiFati the project has been started a long time ago and historically it was using pywin32 (as you mentioned, usually it's an easier option. And maybe ctypes wasn't mature enough at that time?...) Since than, different contributors used different approaches. I think, our current goal, would be just make sure uniform usage of same WinAPI calls.

Also, since introducing UIA backend the win32 getting a bit less attention as we consider switching to UIA as a default backend.

[vasily-v-ryabov]

@CristiFati pywinauto 0.4.0 written by Mark McMahon didn't use pywin32 at all but it worked with 32-bit apps only. At some moment I decided to use pywin32 as it behaved more stable as far as I had known several years ago. Later some users requested Jython support but pywin32 can't be installed on Jython. It can be useful to Cucumber BDD framework users which is written in Java and can use Jython.

But getting back to ctypes only usage requires plenty of work and it's not a high priority for now. We'd better finish UI recorder for UIA backend and then full Linux backend (without recorder).

[CristiFati]

Thank you both (@airelil, @vasily-v-ryabov ) for clarifying things. I suppose you won't reject any potential (ctypes related) PRs, that I might submit when I'll have the time.

[vasily-v-ryabov]

@CristiFati thanks, we appreciate any help. There is very draft Contribution Guide (maybe a bit outdated): https://github.com/pywinauto/pywinauto/wiki/Contribution-Guide-(draft)

[vasily-v-ryabov]

@CristiFati it's fixed in master branch. Could you please verify on your side?