Pywinauto: Cannot Import pywinauto (comtypes passes a union by value)

[MyerFire]

Expected Behavior

When running from pywinauto import Application, the Python terminal should just reset to >>>.

Actual Behavior

When running from pywinauto import Application, Python throws an error.

Steps to Reproduce the Problem

1. Open Python terminal.
2. Run from pywinauto import Application

Specifications

• Pywinauto version: 0.6.1
• Python version and bitness: Python 3.8.1 64 bit
• Platform and OS: Windows 10 64 bit

[MyerFire]

It's worth mentioning that I used to have no problems while using Python 3.7.6, but after I installed Python 3.8.1 and also PyCharm (unlikely to be the problem), it stopped working. Since then, I've reinstalled Python 3.7.6 and tried again, but it still threw the same error.

[MyerFire]

Alright, I've fixed this by reverting to 3.7.4.

[vasily-v-ryabov]

It's most likely comtypes issue. Anyway I'd prefer to keep it open until I have a chance to look at this.

[sharanraj63]

Hi @vasily-v-ryabov,

Facing the same issue in python 3.8.1

[fengerzh]

Confirmed, we can only use Python 3.7.4. Do not use higher version of Python if you want to use pywinauto now.

[FallDownTheSystem]

3.8.0 was working, after updating to 3.8.1 same issue.

[vasily-v-ryabov]

It seems this fix in CPython caused this error:
https://bugs.python.org/issue26628

But it's not a bug in Python. We have to adapt something to the new reality. Going to consider it as critical priority.

[airelil]

FYI, a similar issue reported on comtypes: https://github.com/enthought/comtypes/issues/195 The actual fix might be made in comtypes code.

[vasily-v-ryabov]

It seems the very basic VARIANT type is implemented as Union in comtypes. Not a good news.

[kdschlosser]

I thought that in python a union and a structure are the same thing....

In the python bug report they mentioned both structures and unions but the majority of the conversation was geared at unions and there seems to be only an exception being thrown if a union is being passed. which is strange because I had thought that in python a structure and a union are essentially the same thing. But if it is only throwing exceptions for passing a union then the code below should work. The other thing that is not right is that they provide no solution of existing code. they made the change and left people having to scramble and search out a solution.. Kinda messed up actually.

import ctypes

class SomeStruct(ctypes.Structure):
    class _DUMMYUNION(ctypes.Union):
        _fields_ = [
            ('SomeField', ctypes.c_ulong)
        ]    

    _anonymous_ = ('DUMMYUNIONNAME',)
    _fields_ = [
        ('DUMMYUNIONNAME', _DUMMYUNIONNAME)
    ]

[vasily-v-ryabov]

This link tells that the problem is not in undefined behavior:
https://stackoverflow.com/q/11373203/3648361

So the reason of the prohibited functionality is that this libffi issue is still open:
https://github.com/libffi/libffi/issues/33

[vasily-v-ryabov]

Regarding VARIANT implementation in comtypes I can't imagine how to implement it another way. All Windows COM objects are based on it: https://docs.microsoft.com/en-us/windows/win32/api/oaidl/ns-oaidl-variant There are 2 sub-unions inside.

I feel it's time to contribute to CPython, learn PEP-7 and so on. Very interesting...

[vasily-v-ryabov]

comtypes VARIANT implementation (just for reference): https://github.com/enthought/comtypes/blob/master/comtypes/automation.py#L141

[airelil]

Regarding VARIANT implementation in comtypes I can't imagine how to implement it another way. All Windows COM objects are based on it: https://docs.microsoft.com/en-us/windows/win32/api/oaidl/ns-oaidl-variant There are 2 sub-unions inside.

A possible workaround would be, as mentioned in the original libffi issue, specifying the largest union element as argument but enhancing it according to required calling convention. It's far from ideal, but at least the problem is not as wide as in libffii and can be tailored to VARIANT only.
More details in the excellent comment of Thomas: https://github.com/libffi/libffi/issues/33#issuecomment-204694387

[airelil]

BTW, start thinking, some of these cryptic comtypes crashes could be related to this since the whole thing seems to be worked on undefined behaviour.

[kdschlosser]

well see that's all fine and dandy.. But no matter what change you make to the thing I think if there is a Union involved it is going to get kicked out..

The issue is that Python was altered in a way that is going to break a plethora of software that uses the Windows API and specifically COM objects... The issue was placed 4 years ago.. and then poof... out of the blue they fix something that honestly was never really broken in the first place.

How many people here can say they have seen this bug first hand???? probably 0. if it was such an important change to make that caused a complete disregard for killing software then how come there is not a whole lot of people complaining about this problem.??? I am almost wondering if this is a NIX related issue, the OP of the bug report was using a NIX based OS when the report was made. and I also think that other people that were involved in that bug report were also running a NIX based OS..

Now I am not denying that there may be a problem.. But the bad impact of fixing it far exceeds the benefit of fixing it. it is not a bug that produced a large number of complaints. so it is not really effecting anything. It is so important a bug to fix and to allow it to break all kinds of code that the report sat for 4 years before anything was done.. and then NO notification that this change was going to cause such a massive impact especially on Windows and using COM objects.

I have never had any crashes due to comtypes it's self for no apparent reason.. and you wouldn'y even know if comtypes caused this crash.. Because it is an application crash.. there is no trace back. you would have to be running a debugging version of python and have a crash happen so it could be debugged and to see where the iossue is.. but that is only going to get to so far.. the issue could still b from how comtypes is being used.. the debugger for the c code is not going to give you a walk through of the application code...

However if you compiled both comtypes and the application code using cython you can then debug the whole process..

I just stumbled upon a bug in ctypes dealing with structures and passing them through a function call. for some reason the damned thing would want to get GC's but something got screwed upo with the reference count... solution.. move the code so i didn't have to pass the thing.. This is an issue within python code.. I do not know as to why this was happening.. but python was trying to gc something it shouldn't have been. some how some way with something that i was doing was causing it to get flagged for a GC in a local namespace of a function when there was actually still a reference to it in another function..

[vasily-v-ryabov]

Some initial learning of libffi code made me thinking this is too big feature (in libffi) which probably requires implementations for all possible ABI's (architectures). Also it will take too much time (at least for my level of expertise).

I'm going to start from simple example provided by @tilsche in https://github.com/libffi/libffi/issues/33#issuecomment-204694387. Also I will try another workaround based on .from_address() method of the Structure which also allows re-interpreting memory content without direct usage of a union (this is what I have experience with).

After trying all workarounds on this example, I can proceed with comtypes.

[kdschlosser]

so you found away around the problem?

I had a thought if you didn't.. I wonder if compiling the python code using cython would solve the issue.. comtypes is what would have to be compiled I would imagine.

[vasily-v-ryabov]

@kdschlosser not yet. But I know the direction to dive into.

Cython compilation is a last resort. If other methods don't work, we may think about it. In any case our priority is eliminating compilation on user side since it's a source of problems. What we can do with compilation is just building binary .whl package with pre-compiled cython libraries.

[kdschlosser]

I guess my question was no so much about distribution because that's the the hard part. Having to write a bridge in c code would be fairly annoying to have to do. But we know 100% that it will work..

If using cython can we make it easier and also faster to get to the finish line. will cython convert the code properly so it can be compiled using msvc.. and even tho the code has been converted to c will ctypes still pitch a fit about the union?. or would ctypes have to be completely removed from the equation??.. That is where I was going with the question.

Now my other question is this.. the way the bug fix was explain is that it does not allow a Union to be passed to a c function call directly... Well. the VARIANT is a Structure not a Union. there are unions contained within the structure as does probably a good 70% of the other structures in the Windows API....

another question I have is., How does this traceback occur. How is the type checking done? I would imagine there is a way to "trick" it into thinking what is being passed is not a Union and does not contain a Union.

[kdschlosser]

well would you look at that...

https://bugs.python.org/issue16575

A more comprehensive fix would be to tweak ctypes to reject unions and bit-fields when running on non-x86 (does this work for ARM and other non-Intel archs?)

it explicitly states non x86 systems. It appears that there may be a problem on non x86 systems.

Now here is another issue created for the same bug.,..

https://bugs.python.org/issue26628

Also I would strongly argue to generally prohibit this with an exception instead of just trying if libffi maybe handles this on the current architecture.

It seems as tho the lazy route was taken.. No investigation was properly done to ascertain the extent of the problem. and they simply blocked all unions and structs having bit fields.

and as it turns out this bug has existed for a lot longer then 4 years. It goes back 7 years at least.. and not a whole lot of people actually encountering a problem.... strange. why fix it now??? And technically speaking this is a not a bug in Python it is a problem with libffi

[kdschlosser]

also why would this even be fixed to only have to be undone latter down the road??

as you pointed out this issue that was created for libffi https://github.com/libffi/libffi/issues/33
If you read over the comments.. It has been added to the 4.0 milestone. That means that a fix/support is going to be added to libffi... Then why the patch to Python??

[kdschlosser]

When reading the issues the test cases are always identical. Not done using any other OS except linux. not done with real world unions/structures they always use a union that has 11 c_double fields.

I have not seen a single bitfield test nor have I need any tests with unions nested in a structure

This is what would need to be used in order to to test for the validity but also the scope of the bug.

Scope of issue: Unions being pass to a c function using ctypes, Structures with bitfields being passed to a c function using ctypes

Needed Tests: Tests to be conducted on all supported platforms for each platform architecture using each Python architecture for each Python version

Scope of Test:
Tests should be random in nature for the number of fields and random in nature for the data types for those fields,
Each test with the above rule applied to each test
1: ctypes.Union
2: ctypes.Structure
3: ctypes.Union contained within a ctypes.Structure
4: ctypes.Structure contained within a ctypes.Union
5: above tests to be repeated with random fields having a bit fields and that bitfield size is random as well.

This is what they used as a test case in order to confirm the bug for all system types and python versions.

System CPU: ???
Platform architecture: ???
Platform: Linux
Platform: version ???
Platform kernel: ???
Python architecture: ????
Python versions: 2.7.11, 3.5.1
Tests performed: Single ctypes.Union containing a fixed 11 fields of ctypes.c_double.
No other tests performed.

is it me or are the tests somewhat lacking???

[kdschlosser]

On another note..

We are talking about 1's and 0's here.. It's always 1's and 0's. Windows is all it's glory really doesn't give a crap what is passed to it as long as there are enough bytes allocated to fit the information needed.

I have used this kind of an approach on more then one occasion because of the Windows API being goofy. or simply because of an unknown amount of data being placed.

I recently ran into this issue when writing a pure python eHome CIR binding.

There is not a huge amount of documentation for this portion of the Windows API. and what is available conflicts with other documentation.

ULONG_PTR data type has a double meaning when using it with ctypes. some of the fields containing it. when reading what the ULONG_PTR is.. it either a ULONG or a ULONGLONG depending on architecture.. This is where the documentation failed... some of the fields needed to be set based on the bitness of python while others had to be set based on the bitness of windows. and others were just flat out wrong and are not ULONG's at all.

class _IR_RECEIVE_PARAMS(ctypes.Structure):
    _fields_ = [
        ('DataEnd', ULONG_PTR),  # out
        ('ByteCount', ULONG_PTR),  # in
        ('Data', ULONG_PTR * 1),  # out
    ]

So I used code like this to deal with the problem at hand.,

class _IR_RECEIVE_PARAMS(ctypes.Structure):
    _fields_ = [
        ('DataEnd', ULONG_PTR),  # out
        ('ByteCount', ULONG_PTR),  # in
    ]
obj = _IR_RECEIVE_PARAMS()
# LONG or ULONG does not matter they are of the same size
offset = ctypes.sizeof(_IR_RECEIVE_PARAMS) + ctypes.sizeof(LONG)
size = offset + (ctypes.sizeof(LONG) * 100)
func(ctypes.byref(obj), size)

data = ctypes.cast(ctypes.byref(obj), POINTER(ctypes.c_ubyte))

It was something to this nature.. I can't exactly remember. I do know that I had found a bug in ctypes with the garbage collection of ctypes objects being passed between function calls because of having to deal with this.
Now I know in this case I am passing a pointer. but windows does not really care what the data type is so long as there is enough space or any data it needs can be found at specific locations in memory. I would imagine that anything can be pass so long as it, A: is the correct size and B: data is positioned properly. if that is the case then it should be possible to do the 2 step around that exception that they have added. Now I am not saying it would be easy to do. This is a tad above my head so I really do not know how else to explain it.. could lead to an idea of a fix.. who knows.. I thought it worth mentioning.

[kdschlosser]

@vasily-v-ryabov you may want to chime in here https://github.com/python/cpython/pull/16799 we have finally managed to get the person that write the patch to respond..

You might be able to provide a really simple example to them for reproducing this issue. or provide more details into some of the underlying behavior. There is another guy in there that is also pointing out to the python developer of another library that this broke as well,

[airelil]

I was thinking about just giving an example with WinAPI call, but honestly, I couldn't find any WinAPI function that passes a union by value. At least, from my quick go through WindSDK includes... The VARIANT looks more as an exemption here, but unfortunately it's the core structure for the COM. Providing a scenario based on VARIANT (for example this call: https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/nf-uiautomationclient-iuiautomation-createpropertycondition), will require the definition of the whole VARIANT in "its glory" (refer to @vasily-v-ryabov above link to comtypes)

[vasily-v-ryabov]

@airelil I see 23 COM methods in UIAutomationCore.dll generated wrapper take VARIANT by value.

@kdschlosser thanks! I've provided an example there. Yes I saw 0.4 milestone label in libffi, but it was labeled almost 2 years ago. libffi has been moved from 3.2 to 3.3 release since this moment. Not sure waiting for 2-3 years (in good scenario) is a good solution. Of course we can try to raise the priority.

[airelil]

@airelil I see 23 COM methods in UIAutomationCore.dll generated wrapper take VARIANT by value.

Sure. That's what I was saying. Passing union by value can be found only in calls using VARIANT. But how many other Win API functions have the union-by-value argument ? I couldn't find any.

[kdschlosser]

what do you think about this as a possibility.

I have not tested this and I am not sure if I am grabbing the correct attribute or not.. It's more like an idea then anything else..

The ctypes c code only looks for a flag that has been set
TYPEFLAG_HASUNION = 0x400

it only seems to do this when an argument is added using func.argtypes

so if we create the structure. replace the one in comtypes.automation. we unset the flag import the dynamic module and then add the flag back in.

import ctypes
import _ctypes
import sys

from comtypes.automation import *
from comtypes.automation import (
    _VariantChangeType,
    _VariantCopyInd,
    _midlSAFEARRAY,
    _vartype_to_ctype,
    _ctype_to_vartype,
    _byref_type,
    _VariantCopy,
    _Pointer,
    _arraycode_to_vartype,
    _SysAllocStringLen,
    _com_null_date,
    _VariantClear,
    _safearray
)
TYPEFLAG_HASUNION = 0x400


class UnionType(_ctypes.UnionType):

    def __init__(cls, name, bases, dct):
        super(UnionType, cls).__init__(name, bases, dct)

        def set_union_flag():
            cls.__flags__ |= TYPEFLAG_HASUNION

        def remove_union_flag():
            cls.__flags__ ^= TYPEFLAG_HASUNION

        setattr(cls, 'set_union_flag', set_union_flag)
        setattr(cls, 'remove_union_flag', remove_union_flag)


class Union(metaclass=UnionType):
    pass

class tagVARIANT(Structure):
    class U_VARIANT1(Union):
        class __tagVARIANT(Structure):
            # The C Header file defn of VARIANT is much more complicated, but
            # this is the ctypes version - functional as well.
            class U_VARIANT2(Union):
                class _tagBRECORD(Structure):
                    _fields_ = [("pvRecord", c_void_p),
                                ("pRecInfo", POINTER(IUnknown))]
                _fields_ = [
                    ("VT_BOOL", VARIANT_BOOL),
                    ("VT_I1", c_byte),
                    ("VT_I2", c_short),
                    ("VT_I4", c_long),
                    ("VT_I8", c_longlong),
                    ("VT_INT", c_int),
                    ("VT_UI1", c_ubyte),
                    ("VT_UI2", c_ushort),
                    ("VT_UI4", c_ulong),
                    ("VT_UI8", c_ulonglong),
                    ("VT_UINT", c_uint),
                    ("VT_R4", c_float),
                    ("VT_R8", c_double),
                    ("VT_CY", c_longlong),
                    ("c_wchar_p", c_wchar_p),
                    ("c_void_p", c_void_p),
                    ("pparray", POINTER(POINTER(_safearray.tagSAFEARRAY))),

                    ("bstrVal", BSTR),
                    ("_tagBRECORD", _tagBRECORD),
                    ]
                _anonymous_ = ["_tagBRECORD"]
            _fields_ = [("vt", VARTYPE),
                        ("wReserved1", c_ushort),
                        ("wReserved2", c_ushort),
                        ("wReserved3", c_ushort),
                        ("_", U_VARIANT2)
            ]
        _fields_ = [("__VARIANT_NAME_2", __tagVARIANT),
                    ("decVal", DECIMAL)]
        _anonymous_ = ["__VARIANT_NAME_2"]
    _fields_ = [("__VARIANT_NAME_1", U_VARIANT1)]
    _anonymous_ = ["__VARIANT_NAME_1"]

    def __init__(self, *args):
        if args:
            self.value = args[0]

    def __del__(self):
        if self._b_needsfree_:
            # XXX This does not work.  _b_needsfree_ is never
            # set because the buffer is internal to the object.
            _VariantClear(self)

    def __repr__(self):
        if self.vt & VT_BYREF:
            return "VARIANT(vt=0x%x, byref(%r))" % (self.vt, self[0])
        return "VARIANT(vt=0x%x, %r)" % (self.vt, self.value)

    def from_param(cls, value):
        if isinstance(value, cls):
            return value
        return cls(value)
    from_param = classmethod(from_param)

    def __setitem__(self, index, value):
        # This method allows to change the value of a
        # (VT_BYREF|VT_xxx) variant in place.
        if index != 0:
            raise IndexError(index)
        if not self.vt & VT_BYREF:
            raise TypeError("set_byref requires a VT_BYREF VARIANT instance")
        typ = _vartype_to_ctype[self.vt & ~VT_BYREF]
        cast(self._.c_void_p, POINTER(typ))[0] = value

    # see also c:/sf/pywin32/com/win32com/src/oleargs.cpp 54
    def _set_value(self, value):
        _VariantClear(self)
        if value is None:
            self.vt = VT_NULL
        elif (hasattr(value, '__len__') and len(value) == 0
                and not isinstance(value, basestring)):
            self.vt = VT_NULL
        # since bool is a subclass of int, this check must come before
        # the check for int
        elif isinstance(value, bool):
            self.vt = VT_BOOL
            self._.VT_BOOL = value
        elif isinstance(value, (int, c_int)):
            self.vt = VT_I4
            self._.VT_I4 = value
        elif isinstance(value, long):
            u = self._
            # try VT_I4 first.
            u.VT_I4 = value
            if u.VT_I4 == value:
                # it did work.
                self.vt = VT_I4
                return
            # try VT_UI4 next.
            if value >= 0:
                u.VT_UI4 = value
                if u.VT_UI4 == value:
                    # did work.
                    self.vt = VT_UI4
                    return
            # try VT_I8 next.
            if value >= 0:
                u.VT_I8 = value
                if u.VT_I8 == value:
                    # did work.
                    self.vt = VT_I8
                    return
            # try VT_UI8 next.
            if value >= 0:
                u.VT_UI8 = value
                if u.VT_UI8 == value:
                    # did work.
                    self.vt = VT_UI8
                    return
            # VT_R8 is last resort.
            self.vt = VT_R8
            u.VT_R8 = float(value)
        elif isinstance(value, (float, c_double)):
            self.vt = VT_R8
            self._.VT_R8 = value
        elif isinstance(value, (str, unicode)):
            self.vt = VT_BSTR
            # do the c_wchar_p auto unicode conversion
            self._.c_void_p = _SysAllocStringLen(value, len(value))
        elif isinstance(value, datetime.datetime):
            delta = value - _com_null_date
            # a day has 24 * 60 * 60 = 86400 seconds
            com_days = delta.days + (delta.seconds + delta.microseconds * 1e-6) / 86400.
            self.vt = VT_DATE
            self._.VT_R8 = com_days
        elif npsupport.isdatetime64(value):
            com_days = value - npsupport.com_null_date64
            com_days /= npsupport.numpy.timedelta64(1, 'D')
            self.vt = VT_DATE
            self._.VT_R8 = com_days
        elif decimal is not None and isinstance(value, decimal.Decimal):
            self._.VT_CY = int(round(value * 10000))
            self.vt = VT_CY
        elif isinstance(value, POINTER(IDispatch)):
            CopyComPointer(value, byref(self._))
            self.vt = VT_DISPATCH
        elif isinstance(value, POINTER(IUnknown)):
            CopyComPointer(value, byref(self._))
            self.vt = VT_UNKNOWN
        elif isinstance(value, (list, tuple)):
            obj = _midlSAFEARRAY(VARIANT).create(value)
            memmove(byref(self._), byref(obj), sizeof(obj))
            self.vt = VT_ARRAY | obj._vartype_
        elif isinstance(value, array.array):
            vartype = _arraycode_to_vartype[value.typecode]

            typ = _vartype_to_ctype[vartype]
            obj = _midlSAFEARRAY(typ).create(value)
            memmove(byref(self._), byref(obj), sizeof(obj))
            self.vt = VT_ARRAY | obj._vartype_
        elif npsupport.isndarray(value):
            # Try to convert a simple array of basic types.
            descr = value.dtype.descr[0][1]
            typ = npsupport.numpy.ctypeslib._typecodes.get(descr)
            if typ is None:
                # Try for variant
                obj = _midlSAFEARRAY(VARIANT).create(value)
            else:
                obj = _midlSAFEARRAY(typ).create(value)

            memmove(byref(self._), byref(obj), sizeof(obj))
            self.vt = VT_ARRAY | obj._vartype_
        elif isinstance(value, Structure) and hasattr(value, "_recordinfo_"):
            guids = value._recordinfo_
            from comtypes.typeinfo import GetRecordInfoFromGuids
            ri = GetRecordInfoFromGuids(*guids)
            self.vt = VT_RECORD
            # Assigning a COM pointer to a structure field does NOT
            # call AddRef(), have to call it manually:
            ri.AddRef()
            self._.pRecInfo = ri
            self._.pvRecord = ri.RecordCreateCopy(byref(value))
        elif isinstance(getattr(value, "_comobj", None), POINTER(IDispatch)):
            CopyComPointer(value._comobj, byref(self._))
            self.vt = VT_DISPATCH
        elif isinstance(value, VARIANT):
            _VariantCopy(self, value)
        elif isinstance(value, c_ubyte):
            self._.VT_UI1 = value
            self.vt = VT_UI1
        elif isinstance(value, c_char):
            self._.VT_UI1 = ord(value.value)
            self.vt = VT_UI1
        elif isinstance(value, c_byte):
            self._.VT_I1 = value
            self.vt = VT_I1
        elif isinstance(value, c_ushort):
            self._.VT_UI2 = value
            self.vt = VT_UI2
        elif isinstance(value, c_short):
            self._.VT_I2 = value
            self.vt = VT_I2
        elif isinstance(value, c_uint):
            self.vt = VT_UI4
            self._.VT_UI4 = value
        elif isinstance(value, c_float):
            self.vt = VT_R4
            self._.VT_R4 = value
        elif isinstance(value, c_int64):
            self.vt = VT_I8
            self._.VT_I8 = value
        elif isinstance(value, c_uint64):
            self.vt = VT_UI8
            self._.VT_UI8 = value
        elif isinstance(value, _byref_type):
            ref = value._obj
            self._.c_void_p = addressof(ref)
            self.__keepref = value
            self.vt = _ctype_to_vartype[type(ref)] | VT_BYREF
        elif isinstance(value, _Pointer):
            ref = value.contents
            self._.c_void_p = addressof(ref)
            self.__keepref = value
            self.vt = _ctype_to_vartype[type(ref)] | VT_BYREF
        else:
            raise TypeError("Cannot put %r in VARIANT" % value)
        # buffer ->  SAFEARRAY of VT_UI1 ?

    # c:/sf/pywin32/com/win32com/src/oleargs.cpp 197
    def _get_value(self, dynamic=False):
        vt = self.vt
        if vt in (VT_EMPTY, VT_NULL):
            return None
        elif vt == VT_I1:
            return self._.VT_I1
        elif vt == VT_I2:
            return self._.VT_I2
        elif vt == VT_I4:
            return self._.VT_I4
        elif vt == VT_I8:
            return self._.VT_I8
        elif vt == VT_UI8:
            return self._.VT_UI8
        elif vt == VT_INT:
            return self._.VT_INT
        elif vt == VT_UI1:
            return self._.VT_UI1
        elif vt == VT_UI2:
            return self._.VT_UI2
        elif vt == VT_UI4:
            return self._.VT_UI4
        elif vt == VT_UINT:
            return self._.VT_UINT
        elif vt == VT_R4:
            return self._.VT_R4
        elif vt == VT_R8:
            return self._.VT_R8
        elif vt == VT_BOOL:
            return self._.VT_BOOL
        elif vt == VT_BSTR:
            return self._.bstrVal
        elif vt == VT_DATE:
            days = self._.VT_R8
            return datetime.timedelta(days=days) + _com_null_date
        elif vt == VT_CY:
            return self._.VT_CY / decimal.Decimal("10000")
        elif vt == VT_UNKNOWN:
            val = self._.c_void_p
            if not val:
                # We should/could return a NULL COM pointer.
                # But the code generation must be able to construct one
                # from the __repr__ of it.
                return None # XXX?
            ptr = cast(val, POINTER(IUnknown))
            # cast doesn't call AddRef (it should, imo!)
            ptr.AddRef()
            return ptr.__ctypes_from_outparam__()
        elif vt == VT_DECIMAL:
            return self.decVal.as_decimal()
        elif vt == VT_DISPATCH:
            val = self._.c_void_p
            if not val:
                # See above.
                return None # XXX?
            ptr = cast(val, POINTER(IDispatch))
            # cast doesn't call AddRef (it should, imo!)
            ptr.AddRef()
            if not dynamic:
                return ptr.__ctypes_from_outparam__()
            else:
                from comtypes.client.dynamic import Dispatch
                return Dispatch(ptr)
        # see also c:/sf/pywin32/com/win32com/src/oleargs.cpp
        elif self.vt & VT_BYREF:
            return self
        elif vt == VT_RECORD:
            from comtypes.client import GetModule
            from comtypes.typeinfo import IRecordInfo

            # Retrieving a COM pointer from a structure field does NOT
            # call AddRef(), have to call it manually:
            punk = self._.pRecInfo
            punk.AddRef()
            ri = punk.QueryInterface(IRecordInfo)

            # find typelib
            tlib = ri.GetTypeInfo().GetContainingTypeLib()[0]

            # load typelib wrapper module
            mod = GetModule(tlib)
            # retrive the type and create an instance
            value = getattr(mod, ri.GetName())()
            # copy data into the instance
            ri.RecordCopy(self._.pvRecord, byref(value))

            return value
        elif self.vt & VT_ARRAY:
            typ = _vartype_to_ctype[self.vt & ~VT_ARRAY]
            return cast(self._.pparray, _midlSAFEARRAY(typ)).unpack()
        else:
            raise NotImplementedError("typecode %d = 0x%x)" % (vt, vt))

    def __getitem__(self, index):
        if index != 0:
            raise IndexError(index)
        if self.vt == VT_BYREF|VT_VARIANT:
            v = VARIANT()
            # apparently VariantCopyInd doesn't work always with
            # VT_BYREF|VT_VARIANT, so do it manually.
            v = cast(self._.c_void_p, POINTER(VARIANT))[0]
            return v.value
        else:
            v = VARIANT()
            _VariantCopyInd(v, self)
            return v.value

# these are missing:
##    getter[VT_ERROR]
##    getter[VT_ARRAY]
##    getter[VT_BYREF|VT_UI1]
##    getter[VT_BYREF|VT_I2]
##    getter[VT_BYREF|VT_I4]
##    getter[VT_BYREF|VT_R4]
##    getter[VT_BYREF|VT_R8]
##    getter[VT_BYREF|VT_BOOL]
##    getter[VT_BYREF|VT_ERROR]
##    getter[VT_BYREF|VT_CY]
##    getter[VT_BYREF|VT_DATE]
##    getter[VT_BYREF|VT_BSTR]
##    getter[VT_BYREF|VT_UNKNOWN]
##    getter[VT_BYREF|VT_DISPATCH]
##    getter[VT_BYREF|VT_ARRAY]
##    getter[VT_BYREF|VT_VARIANT]
##    getter[VT_BYREF]
##    getter[VT_BYREF|VT_DECIMAL]
##    getter[VT_BYREF|VT_I1]
##    getter[VT_BYREF|VT_UI2]
##    getter[VT_BYREF|VT_UI4]
##    getter[VT_BYREF|VT_INT]
##    getter[VT_BYREF|VT_UINT]

    value = property(_get_value, _set_value)

    def __ctypes_from_outparam__(self):
        # XXX Manual resource management, because of the VARIANT bug:
        result = self.value
        self.value = None
        return result

    def ChangeType(self, typecode):
        _VariantChangeType(self,
                           self,
                           0,
                           typecode)

VARIANT = tagVARIANT
VARIANTARG = VARIANT


sys.modules['comtypes.automation'].VARIANT = VARIANT
sys.modules['comtypes.automation'].VARIANTARG = VARIANT

# remove the union flag
VARIANT.remove_union_flag()


# import dynamically generated comtypes module

# reset the union flag after import
VARIANT.set_union_flag()

[kdschlosser]

There is also another way.

Here is the quick and dirty of a struct and a union

A struct is a block of memory that stores several data objects, where those objects don't overlap. A union is a block of memory that stores several data objects, but has only storage for the largest of these, and thus can only store one of the data objects at any one time.

so for all intents and purposes they are the same thing except for memory allocation. So technically you can change out the Unions for Structures use ctypes.c_byte as the data type for a single field. make that datatype an array where the field would be the same size as the largest dataype originally stored in the union.

You would have to use the above method to cast to the correct datatype using offset for where the data starts. It can be done.. It would require a huge amount of code to handle doing that.

[kdschlosser]

This also may work.

import ctypes
import comtypes
import comtypes.automation

VARIANT = ctypes.c_byte * ctypes.sizeof(comtypes.automation.VARIANT)

comtypes.automation.VARIANT = VARIANT
comtypes.automation.VARIANTARG = VARIANT
# import generated module
import comtypes.gen.stdole

# in order to pass the variant structure you would use the following

variant = comtypes.automation.tagVARIANT()

# set whatever attributes ned to be set

passable_variant = comtypes.cast(VARIANT, ctypes.byref(variant))

# pass the passable_variant to whatever is needed.

I have not tested these solutions as of yet.

[mrginglymus]

Happily the changes appear to have been reverted for now https://github.com/python/cpython/pull/17960

[jakewaxman]

So does that mean the problem is fixed using the latest 3.8 version of python or should we stick with 3.74?

[mrginglymus]

It should be fixed in 3.8.2, due to be released later today.

[Andrylast]

It should be fixed in 3.8.2, due to be released later today.

I have the latest version of Python (3.8.2) installed.
When trying to import pywinauto, a slightly different error appears:

Traceback (most recent call last):
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\ctypes\__init__.py", line 123, in WINFUNCTYPE
    return _win_functype_cache[(restype, argtypes, flags)]
KeyError: (<class 'ctypes.HRESULT'>, (<class 'comtypes.POINTER(IUIAutomationElement)'>, <class 'ctypes.c_long'>, <class 'comtypes.automation.tagVARIANT'>), 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/HomePC/PycharmProjects/etp_autotests/test_runner.py", line 2, in <module>
    import tests
  File "C:\Users\HomePC\PycharmProjects\etp_autotests\tests.py", line 26, in <module>
    import pywinauto
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pywinauto\__init__.py", line 89, in <module>
    from . import findwindows
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pywinauto\findwindows.py", line 42, in <module>
    from . import controls
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pywinauto\controls\__init__.py", line 36, in <module>
    from . import uiawrapper # register "uia" back-end (at the end of uiawrapper module)
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pywinauto\controls\uiawrapper.py", line 47, in <module>
    from ..uia_defines import IUIA
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pywinauto\uia_defines.py", line 181, in <module>
    pattern_ids = _build_pattern_ids_dic()
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pywinauto\uia_defines.py", line 169, in _build_pattern_ids_dic
    if hasattr(IUIA().ui_automation_client, cls_name):
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pywinauto\uia_defines.py", line 50, in __call__
    cls._instances[cls] = super(_Singleton, cls).__call__(*args, **kwargs)
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pywinauto\uia_defines.py", line 60, in __init__
    self.UIA_dll = comtypes.client.GetModule('UIAutomationCore.dll')
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\comtypes\client\_generate.py", line 110, in GetModule
    mod = _CreateWrapper(tlib, pathname)
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\comtypes\client\_generate.py", line 184, in _CreateWrapper
    mod = _my_import(fullname)
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\comtypes\client\_generate.py", line 24, in _my_import
    return __import__(fullname, globals(), locals(), ['DUMMY'])
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\comtypes\gen\_944DE083_8FB8_45CF_BCB7_C477ACB2F897_0_1_0.py", line 843, in <module>
    IUIAutomationPropertyChangedEventHandler._methods_ = [
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\comtypes\__init__.py", line 329, in __setattr__
    self._make_methods(value)
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\comtypes\__init__.py", line 698, in _make_methods
    prototype = WINFUNCTYPE(restype, *argtypes)
  File "C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\ctypes\__init__.py", line 125, in WINFUNCTYPE
    class WinFunctionType(_CFuncPtr):
TypeError: item 3 in _argtypes_ passes a union by value, which is unsupported.

Process finished with exit code 1

[vasily-v-ryabov]

@Andrylast are you sure it's proper clean installation of Python 3.8.2? Maybe comtypes cache wasn't cleared somehow. Please uninstall all packages, uninstall Python, delete Python folder and install everything from scratch.

I've just tried Python 3.8.2 64-bit and import pywinauto works.

[vasily-v-ryabov]

@Andrylast in your case the comtypes cache folder is here:

C:\Users\HomePC\AppData\Local\Programs\Python\Python38-32\lib\site-packages\comtypes\gen\