Azure-sdk-for-python: ADF V2 Create or update pipeline error using datafactory 0.3, AttributeError: 'str' object has no attribute 'values'

[saulcruz]

after upgrading the python library from .2.2 to .3 I started getting the following error when creating/updating pipelines:

 File "C:\ProgramData\Anaconda3\lib\site-packages\azure\mgmt\datafactory\operations\pipelines_operations.py", line 162, in create_or_update
    body_content = self._serialize.body(pipeline, 'PipelineResource')
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 443, in body
    errors = _recursive_validate(data_type, data)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 122, in _recursive_validate
    return data.validate()
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 169, in validate
    validation_result += _recursive_validate(attr_type, value)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 117, in _recursive_validate
    result += _recursive_validate(attr_type[1:-1], content)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 122, in _recursive_validate
    return data.validate()
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 169, in validate
    validation_result += _recursive_validate(attr_type, value)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 122, in _recursive_validate
    return data.validate()
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 169, in validate
    validation_result += _recursive_validate(attr_type, value)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 119, in _recursive_validate
    for content in data.values():
AttributeError: 'str' object has no attribute 'values'

Process finished with exit code 1

This is how I create a pipeline, not sure I'm completely missing something, I've reviewed my code and looks exactly as when it was working with .2.2

            pipeline_name = self.pipeline_name 
            params_for_pipeline = {}
            # activities = [list of CopyActivity created prior to this process]
            pl_resource_object = PipelineResource(additional_properties={},
                                                  activities=activities,
                                                  parameters=params_for_pipeline)

            self.adf_client.pipelines.create_or_update(resource_group_name=self.resource_group_name,
                                                       factory_name=self.data_factory_name,
                                                       pipeline_name=pipeline_name,
                                                       pipeline=pl_resource_object)

Did the way to create PipelineResources change? or something else?

Thanks for your help and efforts on this library :)

[lmazuel]

This error means that the model is expecting a dictionary, but it got a string (i.e. we call values() because the meta-data says it will be a dict, but we got a str and str does not have value). Note to myself: improve the error message with the name of the property validated.

Since it's a serialization issue, your PipelineResource object is defined incorrectly: you pass a string where a dict is expected. If it was working before, I guess this object changed its signature in the latest version.

Pipeline resource doc is available here:
https://docs.microsoft.com/en-us/python/api/azure.mgmt.datafactory.models.pipelineresource?view=azure-python

I also did a small change in msrest (the low level library) recently, that may create the issue. Could you check with both 0.4.19 and 0.4.21 to see if something changes? (likeliness 0.1% but just in case :))

FYI @hvermis

[hvermis]

It could also be caused by your activities since PipelineResource serialization includes everything in the pipeline. Our version .3 has some breaking changes compared to .2.2, so you could see some issues in your scripts. What sources and sinks are you using in Copy activities?

[saulcruz]

@hvermis SqlSource and SqlSink CopyActivities, however i downgraded it to azure-mgmt-datafactory-0.2.2 and it worked so not sure if this is related to the copy activities unless there was a change in the way we define CopyActivities

@lmazuel I think I'm using the right signature (as specified in the link you provided) which is

pl_resource_object = PipelineResource(additional_properties={}, activities=activities, parameters={})

And regarding the msrest, I tested it with 0.4.21 and with datafactory-0.2.2 and it worked meaning that we can rule this out

[lmazuel]

@saulcruz Being the number of recursive call in your stacktrace, it's likely not in PipelineResource directly, but deep inside the model, in one of the submodel.
Being that you don't provide additional_properties or parameters, it's probably deep inside activities.

Note that each model has a validate method, like this:
pl_resource_object.validate()

You can try to validate each subpart:

for act in activities:
   act.validate()

and so on on all inside attributes, until you find out who's is causing trouble.

[lmazuel]

@saulcruz , could you replace your msrest with this one from a branch, and try to see if the exception is better now?

pip uninstall -y msrest
pip install git+https://github.com/Azure/msrest-for-python@validate_fix#egg=msrest

[saulcruz]

@lmazuel sorry about the delay and thanks for the suggestion, and yeah I'm actually getting a better exception now.

i'm getting the error: msrest.exceptions.ValidationError: Parameter 'additional_properties' must be of type '{object}'

so, I added additional_parameters={} and an empty dictionary as I thought it was like the pipeline parameters where I need to pass the empty dict. I'm doing that in all the objects that require additional_parametres (SqlSource, SqlSink, CopyActivity, LookupActivity, PipelineResource), is this really required, if so, what would be the right definition of: additional_properties dict[str, object]

and the full traceback is:

  File "C:\ProgramData\Anaconda3\lib\site-packages\azure\mgmt\datafactory\operations\pipelines_operations.py", line 162, in create_or_update
    body_content = self._serialize.body(pipeline, 'PipelineResource')
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 465, in body
    errors = _recursive_validate(data_type, data_type, data)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 124, in _recursive_validate
    return data.validate()
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 176, in validate
    validation_result += _recursive_validate(attr_name, attr_type, value)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 117, in _recursive_validate
    result += _recursive_validate(attr_name+" content", attr_type[1:-1], content)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 124, in _recursive_validate
    return data.validate()
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 176, in validate
    validation_result += _recursive_validate(attr_name, attr_type, value)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 124, in _recursive_validate
    return data.validate()
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 176, in validate
    validation_result += _recursive_validate(attr_name, attr_type, value)
  File "C:\ProgramData\Anaconda3\lib\site-packages\msrest\serialization.py", line 120, in _recursive_validate
    raise ValidationError("type", attr_name, attr_type)
msrest.exceptions.ValidationError: Parameter 'additional_properties' must be of type '{object}'

[hvermis]

You should be able to not specify additional properties and it should work as they have default value None

[saulcruz]

@hvermis that's how my code was originally and that's when I got the error for the first time after upgrading to datafactory=0.3.0

[lmazuel]

Additional_properties was added in msrest 0.4.20 (it's why I was talking about that). Could you try with msrest 0.4.19 and adf 0.3.0 (I think you tried 0.4.21 and 0.2.2, I'd like the contrary now :)). My guess is it will fail with an horrible error.

Additional_properties was added in adf 0.3.0, but the ADF team (I mean @hvermis ) was not aware that it was not supported in Python. Coincidence is I released the first version of Python runtime that supports this feature at the same time I released ADF 0.3.0...

It's then probably a runtime bug since this is really recent work (0.4.21 was for instance just a release to fix an additional_properties bug).

[lmazuel]

@saulcruz I pushed another commit to the previous branch, if you could try this should solve all your problems.

[saulcruz]

@lmazuel awesomeness!!!! it works, I really appreciate your help, and thanks for the recommendations @hvermis

[lmazuel]

Thank you for your testing @saulcruz ! I release this version as msrest 0.4.22. Next fresh install of ADF will pick it up automatically.
Thanks again!