Azure-docs: How to deal with outputs of Data Factory Validation activity?

[bertrandpons]

Hello,

Question based on the following documentation: https://docs.microsoft.com/en-us/azure/data-factory/control-flow-validation-activity

I have a Validation activity in my pipeline. Depending on it, I want to add a If condition activity with a different behavior in function of the output _exists_ field of the Validation activity.

The output of the Validation activity is as follow:

{
    "PipelineName": "pipeline3",
    "JobId": "XXXXXXXXXXXx",
    "ActivityRunId": "XXXXXXXXXXXXXXXXx",
    "Status": "Succeeded",
    "StatusCode": 200,
    "Output": {
        "exists": true,
        "childItems": [
            {
                "name": "XXXXXX.png",
                "type": "File"
            }
        ],
        "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (West Europe)",
        "executionDuration": 250
    },
    "Error": null,
    "ExecutionStartTime": "2019-05-15T13:14:55.2872631Z",
    "ExecutionEndTime": "2019-05-15T13:14:56.6833996Z",
    "ExecutionDetails": {
        "integrationRuntime": [
            {
                "name": "DefaultIntegrationRuntime",
                "type": "Managed",
                "location": "West Europe"
            }
        ]
    },
    "Duration": "00:00:01.3961365",
    "iterationCount": "1"
}

However the If Condition activity keeps having the same error message:

{
    "errorCode": "InvalidTemplate",
    "message": "The expression 'activity('ValidateIfFileExistsInInput').output.exists' cannot be evaluated because property 'output' cannot be selected.",
    "failureType": "UserError",
    "target": "If Condition1"
}

How to deal with _exists_ value then?

Thank you !

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 81948373-1590-26a4-179f-4c01fa09e60e
• Version Independent ID: b03d5df8-097d-e7cd-6a9e-3a4201333959
• Content: Validation activity in Azure Data Factory
• Content Source: articles/data-factory/control-flow-validation-activity.md
• Service: data-factory
• GitHub Login: @sharonlo101
• Microsoft Alias: shlo

[mimckitt]

Thanks for the feedback! We are currently investigating and will update you shortly.

[MartinJaffer-MSFT]

Hello @bertrandpons and thank you for your inquiry. I have tried for myself, and all my attempts to grab the 'output' of the Validation activity have failed. I think this is a bug, as the Validation activity has an output as visible in the debug mode. I will attempt to contact the product group.

[MartinJaffer-MSFT]

@bertrandpons I do have one piece that can possibly help. Instead of using the output to test for success/fail, use the Validation activity's Red/Green paths. When the Validation times out , the red failure path is taken.

[bertrandpons]

@MartinJaffer-MSFT Thank you for your investigation. I hope this bug will be fixed soon!

[MartinJaffer-MSFT]

The product group has been made aware. I will update this when I have more information.

[bertrandpons]

Thank you

[MartinJaffer-MSFT]

@bertrandpons I have great news! This issue will be fixed in the next release cycle. Thank you for helping us improve our products.

[bertrandpons]

@MartinJaffer-MSFT thank you for that ! Do you know when the release is planned?

[MartinJaffer-MSFT]

Last I heard was roughly 2 weeks.

[seanbetts]

I've been experiencing the same problem here and when the validation times out it doesn't take the red failure path. All I get in the debug output is Status = Timed Out and the pipeline fails.

[MartinJaffer-MSFT]

Hello @seanbetts . Could you please share with me the logic and dataset type you are using?

I have set up a test pipeline, with a validation, and a set variable activity on success, and a set variable activity on failure.

When I do debug run using a blob...
and the blob does not exist, and no minimum size, failure
and the blob does not exist, and a minimum size is set, failure
and the blob does exist, and minimum size is greater than blob, failure

[seanbetts]

Hi @MartinJaffer-MSFT, thanks for getting back to me. Below is an image of my pipeline logic:

The data I’m validating is a simple JSON file and I’d like the pipeline to check if these files exist in the two above locations and then stop if it doesn’t find any or proceed if it finds one or both.

[MartinJaffer-MSFT]

Okay. I see what is happening. Let me use this picture to help with example:

Here, both validation activities ValidX and ValidY are connected to SetVar by 'on failure' dependencies. SetVar waits for all 'incoming' activities before executing. When _both_ ValidX and ValidY time out, SetVar executes. When only ValidX times out, and ValidY finds the file, SetVar does not execute, because it is waiting for a response that never comes. Furthermore, the pipeline reports failure.

Here, both validation activities ValidX and ValidY are connected to SetVar by 'on completion' dependency. This means no matter whether success, or failure, time out or found file, execution will continue along this path. Inside SetVar I use the dynamic expression @{or(activity('ValidX').output.exists,activity('ValidY').output.exists)}. This sets the variable to True when any of the validation activities find the file.

From the images you provided, it looks like the 'Combine TSCRPTS' is expecting success from the 'GET __-TSCRPTS' and expecting failure from the '__ Validation' .
Since you want the 'Combine TSCRPTS' to happen only when at least one of the 'GET __-TSCRPTS' succeeds, I suggest you create an 'If' activity. Inside the 'If' activity, place the 'Combine TSCRPTS' and everything that follows.

In this picture, I have the 'Get __-TSCRPTS' only activating when the respective Validation succeeds. Both of the 'Get __-TSCRPTS' are connected to the If condition by both 'on success' AND 'skipped' lines. This is significant, because it prevents the If Condition from activating before the copy completes. It is also significant, because it does not stop the If Condition from activating if the copy does not happen. Because the Validation activities are ancestors of the 'Get __-TSCRPTS' which are ancestors of the If Condtion, the If Condition has access to the 'output' of the Validation activities. This allows us to use the logic I demonstrated in the previous example.

[seanbetts]

Thanks @MartinJaffer-MSFT for taking the time to look at this for me. I'm trying to replicate your suggestions and if I'm right in understanding your advice, I should include the following expression in my If Condition: @or(activity('Get WC-TSCRPTS').output.exists,activity('Get WC-TSCRPTS').output.exists). Is that correct?

Am I also right in understanding that all the subsequent actions in the pipeline (such as Combine TSCRPTS, Delete Staging etc.) should be included in the If True Activities in the If Condition?

[seanbetts]

I've just tried implementing the expression above in the If condition and I'm getting the following error: The expression 'or(activity('Get WC-TSCRPTS').output.exists,activity('Get FB-TSCRPTS').output.exists)' cannot be evaluated because property 'exists' doesn't exist, available properties are 'dataRead, dataWritten, filesRead, filesWritten, sourcePeakConnections, sinkPeakConnections, rowsRead, rowsCopied, copyDuration, throughput, errors, effectiveIntegrationRuntime, usedDataIntegrationUnits, usedParallelCopies, executionDetails'.

[MartinJaffer-MSFT]

@seanbetts try activity('WC Validation') instead of activity('Get WC_TSCRPTS').
The exists is from the output of the validation activity.

[seanbetts]

Thanks @MartinJaffer-MSFT - that seems to have worked and my pipeline is up and running! I now just need to work out an if statement around the deleting of the original WC-TSCRPTS/FB-TSCRPTS depending on whether they're picked up at the start of the pipeline or not. Thanks for all your help!

[seanbetts]

Hi @MartinJaffer-MSFT - quick question: If I wanted to add a third validation what is the best way to construct the OR statement? Should I nest the OR statement i.e.@ or(Validate A, or(Validate B, Validate C)). As the OR operator only takes two items I'm not sure of the best way of doing it? Thank you!

[MartinJaffer-MSFT]

@seanbetts 'Or' follows the 'associative property' (see mathematics). It does not matter whether you do:
A or (B or C)
(A or B) or C
Both are valid. Nesting is the way to go.