Python-zeep: Order of elements in response: XMLParseError: Unexpected element

[macrojames]

Hi,
i am running in an Issue with the order of elements in a response.
It's a SOAP API of RSA Authentication Manager.
It returns the reponse elements alphabetically (looks like Java does that automatically) in a different order as the request. It works with strict=False and I don't know if any RFC specifies the order of elements.

Version: zeep==2.2.0

I am pasting the relevant parts of request and the full response:

Request:

<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
  <soap-env:Body>
    <ns0:executeCommand xmlns:ns0="http://webservice.rsa.com">
      <in1 xmlns:ns1="http://authn.rsa.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns1:LoginCommand">
        <backgroundCommand>false</backgroundCommand>
        <commandJobId>a1</commandJobId>
        <schedule xsi:nil="true"/>
        <agentId xsi:nil="true"/>
        <authenticateUnregisteredUser>false</authenticateUnregisteredUser>
        <authenticationMethodId>RSA_Password</authenticationMethodId>
        <authenticationState xsi:nil="true"/>
        <authenticationStep xsi:nil="true"/>
...

Response:

<?xml version="1.0" encoding="utf-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Body>
    <ns1:executeCommandResponse xmlns:ns1="http://webservice.rsa.com" soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
      <executeCommandReturn href="#id0"/>
    </ns1:executeCommandResponse>
    <multiRef xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:ns2="http://authn.rsa.com" id="id0" soapenc:root="0" soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xsi:type="ns2:LoginCommand">
      <agentId xsi:type="xsd:string" xsi:nil="true"/>
      <authenticateUnregisteredUser xsi:type="xsd:boolean">false</authenticateUnregisteredUser>
      <authenticationMethodId xsi:type="xsd:string">RSA_Password</authenticationMethodId>
      <authenticationState xsi:type="xsd:string">authenticated</authenticationState>
      <authenticationStep xsi:type="xsd:string" xsi:nil="true"/>
      <backgroundCommand xsi:type="xsd:boolean">false</backgroundCommand>
      <commandJobId xsi:type="xsd:anyType" xsi:nil="true"/>
      <emergencyAuthentication xsi:type="xsd:boolean">false</emergencyAuthentication>
      <hiddenParameters xmlns:ns3="http://data.authn.rsa.com" soapenc:arrayType="ns3:AbstractParameterDTO[0]" xsi:type="soapenc:Array"/>
      <identitySourceGuid xsi:type="xsd:string">ims.000000000000000000001000d0011000</identitySourceGuid>
      <message xmlns:ns4="http://data.authn.rsa.com" xsi:type="ns4:MessageDTO" xsi:nil="true"/>
      <netAddress xmlns:ns5="http://net.java" xsi:type="ns5:InetAddress" xsi:nil="true"/>
      <parameters xmlns:ns6="http://data.authn.rsa.com" soapenc:arrayType="ns6:AbstractParameterDTO[0]" xsi:type="soapenc:Array"/>
      <policyGuid xsi:type="xsd:string" xsi:nil="true"/>
      <principalGuid xsi:type="xsd:string">ims.8a384c0f0ce3xxxxxxx</principalGuid>
      <principalRegistered xsi:type="xsd:boolean">true</principalRegistered>
      <schedule xsi:type="xsd:anyType" xsi:nil="true"/>
      <sessionId xsi:type="xsd:string">e62775e00xxxxcfd-CzxxxxjptThb</sessionId>
      <usingTransientSession xsi:type="xsd:boolean">false</usingTransientSession>
    </multiRef>
  </soapenv:Body>
</soapenv:Envelope>

Ends with:

  File "/home/.../venv/local/lib/python2.7/site-packages/zeep/xsd/types/complex.py", line 173, in parse_xmlelement
    raise XMLParseError(exc.message)
zeep.exceptions.XMLParseError: Unexpected element u'agentId', expected u'backgroundCommand`

I call the endpoint like that:

client = Client('https://xxxxxxx:7022/ims-ws/services/CommandServer?wsdl', transport=transport, strict=True)
loginCommand = client.get_type("{http://authn.rsa.com}LoginCommand")(backgroundCommand=False, commandJobId='a1')
loginCommand.authenticationMethodId = "RSA_Password"
loginCommand.authenticationStatus = "initial"
# ...
loginCommand = client.service.executeCommand(in0=xsd.SkipValue, in1=loginCommand)

If you need the wsdl I could upload it but it shouldn't be relevant.
I would be happy if I could still use strict mode but i am unsure if element order has to be a part of strict mode.
You can close this issue if you say order does matter for strict mode.

[macrojames]

I must correct myself. It does not work with strict=False. The reponse will not be parsed correctly:
I added the following in zeep/xsd/elements/element.py

            # Only compare the localname
            print element_tag, self.qname, element_tag.localname == self.qname.localname

The output was:
executeCommandReturn executeCommandReturn True agentId backgroundCommand False agentId commandJobId False agentId schedule False agentId agentId True authenticateUnregisteredUser authenticateUnregisteredUser True authenticationMethodId authenticationMethodId True authenticationState authenticationState True authenticationStep authenticationStep True backgroundCommand emergencyAuthentication False backgroundCommand hiddenParameters False backgroundCommand identitySourceGuid False backgroundCommand message False backgroundCommand netAddress False backgroundCommand parameters False backgroundCommand policyGuid False backgroundCommand principalGuid False backgroundCommand principalRegistered False backgroundCommand sessionId False backgroundCommand usingTransientSession False
It is skipping and missing elements if the order differs

[macrojames]

Found #250 :(
Same issue.
The wsdl specifies

         <complexType abstract="true" name="TargetableCommand">
            <sequence>
               <element name="backgroundCommand" type="xsd:boolean"/>
               <element name="commandJobId" nillable="true" type="xsd:anyType"/>
               <element name="schedule" nillable="true" type="xsd:anyType"/>
            </sequence>
         </complexType>

         <complexType name="LoginCommand">
            <complexContent>
               <extension base="tns2:TargetableCommand">
                  <sequence>
                     <element name="agentId" nillable="true" type="xsd:string"/>
                     <element name="authenticateUnregisteredUser" type="xsd:boolean"/>
                     <element name="authenticationMethodId" nillable="true" type="xsd:string"/>
                     <element name="authenticationState" nillable="true" type="xsd:string"/>
                     <element name="authenticationStep" nillable="true" type="xsd:string"/>
                     <element name="emergencyAuthentication" type="xsd:boolean"/>
                     <element name="hiddenParameters" nillable="true" type="impl:ArrayOf_tns632_AbstractParameterDTO"/>
                     <element name="identitySourceGuid" nillable="true" type="xsd:string"/>
                     <element name="message" nillable="true" type="tns21:MessageDTO"/>
                     <element name="netAddress" nillable="true" type="tns62:InetAddress"/>
                     <element name="parameters" nillable="true" type="impl:ArrayOf_tns632_AbstractParameterDTO"/>
                     <element name="policyGuid" nillable="true" type="xsd:string"/>
                     <element name="principalGuid" nillable="true" type="xsd:string"/>
                     <element name="principalRegistered" type="xsd:boolean"/>
                     <element name="sessionId" nillable="true" type="xsd:string"/>
                     <element name="usingTransientSession" type="xsd:boolean"/>
                  </sequence>
               </extension>
            </complexContent>
         </complexType>

[mvantellingen]

If you set strict=False then all other elements should be collected in a separate attribute, can you dump the object returned by zeep?

[macrojames]

{
    'backgroundCommand': None,
    'commandJobId': None,
    'schedule': None,
    'agentId': None,
    'authenticateUnregisteredUser': False,
    'authenticationMethodId': 'RSA_Password',
    'authenticationState': 'authenticated',
    'authenticationStep': None,
    'emergencyAuthentication': None,
    'hiddenParameters': None,
    'identitySourceGuid': None,
    'message': None,
    'netAddress': None,
    'parameters': None,
    'policyGuid': None,
    'principalGuid': None,
    'principalRegistered': None,
    'sessionId': None,
    'usingTransientSession': None,
    '_raw_elements': deque([<Element backgroundCommand at 0x7fcff0fe5ea8>, <Element commandJobId at 0x7fcff0fe5cf8>, <Element emergencyAuthentication at 0x7fcff0fe5e60>, <Element hiddenParameters at 0x7fcff0fe5cb0>, <Element identitySourceGuid at 0x7fcff0fe5200>, <Element message at 0x7fcff0fe54d0>, <Element netAddress at 0x7fcff0fe5758>, <Element parameters at 0x7fcff0fe5c20>, <Element policyGuid at 0x7fcff0fe5a70>, <Element principalGuid at 0x7fcff0fe57e8>, <Element principalRegistered at 0x7fcff0fe5998>, <Element schedule at 0x7fcff0fe5b48>, <Element sessionId at 0x7fcff0fe53b0>, <Element usingTransientSession at 0x7fcff0fe50e0>])
}

Accessing single elements works and content is correct.

[mvantellingen]

As you can see elements which zeep couldn't resolve are available in _raw_elements. Those are just generic lxml elements

[macrojames]

I saw that. I opened that issue because I hoped for a way to circumvent manual parsing as every element is existant and otherwise totally valid.
Main issue seems like the Java Weblogic Backend is combining the two sequences defined in the wsdl into a single, alphabetically ordered sequence?

[mvantellingen]

It's probably an axis soap server which are buggy as hell. xsd:sequences are always ordered, since axis doesn't/didn't respect that you will need to use strict=false and manually parse it.

[macrojames]

My slow and ugly workaround is in
macrojames/python-zeep@e2dc96ef53b6548a79e13e6ff935fb5e06d1739d
Maybe you will consider some SKIP_AXIS_FAIL (can confirm on that) flag some time.