Azure-docs: Getting BES to work in ADF UI

[LHTech-Analytics]

Trying (For the last 2 days) to run a ML Batch Execution activity in latest version of ADF in a pipeline. I'm finding that the interface is pretty non intuitive and it's far from clear what I need to do. I've been referencing the only docs I can find that relate to carrying out this specific task:

https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-machine-learning

But it just provides a bunch of JSON snippets that result in errors when I try to use them. I'm assuming that the docs have not been updated properly as there are clear differences between what you're advising to do in the article and what the interface is asking/expecting me to do.

When I go through the steps in the interface, it produces JSON but the parameters differ from what you advise to use in your article.

When I run the BES module, it errors out saying:

{
"errorCode": "2600",
"message": "Batch Execution failed. The response from the Machine Learning service at endpoint https://europewest.services.azureml.net/workspaces/3b558adebdec45bb921d258fc00bde83/services/93726a7d7ae3464e842cb3814c0f27a0/jobs was 'Invalid argument provided. Invalid service input name(s) specified in the request: RemovedKey. Please map the input data to the correct service inputs and try again.'.",
"failureType": "UserError",
"target": "ML Batch Execution1"
}

When I try to google this error, there are 0 results returned. What am I supposed to do here?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 87aead2a-319e-0c71-1d5b-8a42ebdca892
• Version Independent ID: 52c87a50-923b-464b-d126-78e944aebe2f
• Content: Create predictive data pipelines using Azure Data Factory
• Content Source: articles/data-factory/transform-data-using-machine-learning.md
• Service: data-factory
• GitHub Login: @nabhishek
• Microsoft Alias: abnarain

[LHTech-Analytics]

This is the JSON produce by the ADF UI after I step through creating the BES activity. Can anyone see any obvious issues with this?:

{
"name": "pipeline1",
"properties": {
"activities": [
{
"name": "ML Batch Execution1",
"type": "AzureMLBatchExecution",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"webServiceInputs": {
"KEYREMOVED": {
"filePath": "adftutorial/[email protected]",
"linkedServiceName": {
"referenceName": "AzureStorageLinkedService",
"type": "LinkedServiceReference"
}
}
},
"webServiceOutputs": {
"KEYREMOVED": {
"filePath": "azureml",
"linkedServiceName": {
"referenceName": "AzureStorageLinkedService",
"type": "LinkedServiceReference"
}
}
}
},
"linkedServiceName": {
"referenceName": "MachineLearningLinkedService",
"type": "LinkedServiceReference"
}
}
],
"annotations": []
}
}

[LHTech-Analytics]

Also, noticing that when I import data from blob storage, it sets the type of all the fields to String. With a copy data or data flow acitivity, I have the option to change the data type but in the ML BES activity, I don't have the option to change this which is the only reason I can think that I may be getting the error relating to invalid service inputs.

My input data set has all the same attributes as specified in the web service input in ML studio. When configuring the BES activity, there are 4 steps, the third of which is called Settings. It asks for a Web service input. I'm using a linked service I set up to connect to my blob storage. It then requires me to browse to a file in that storage account and asks for an input key (I'm assuming the key should be Key1 for my storage account but again this is not particularly clear so I could be wrong).

As I say, the file I specify is a CSV that contains only the columns required by the endpoint.

For my output, I've specified just a folder in the same storage account with the same key. I'd assume that I'd need to specify the columns as I would only want the scroed labels and probabilities but there's no way to specify any column definitons for the output in the Settings section of the BES activity.

I'm sure I'm doing something wrong here but in lieu of any kind of accurate docs or support from yourselves, I'm stuck and don't see any way of being able to resolve this.

[LHTech-Analytics]

@MicahMcKittrick-MSFT, can you provide any guidance? How am I supposed to obtain support when I find myself in scenarios like this? I'm completely stuck and have been for 2 days now :(

[LHTech-Analytics]

@MicahMcKittrick-MSFT Was hoping this tutorial would show the way but the trainers interface is different from mine and he creates all his pipeline elements using JSON only. The visual UI I have is completely different.

In This other tutorial, the trainer is using the same interface as the first video and says that pipelines, activities etc can only be created using Javascript.

I'm happy to just use JS if it works better than the UI but I'm massively confused. How do I get the same interface these two people are usin in their workflow. Are these old versions of DF?

I seem to have exhausted all documentation whether Microsoft or third party and still have no idea why this won't work in ADF UI

[LHTech-Analytics]

You don't have permission to create a support request
To get permission, ask your subscription administrator or owner to assign you ‘Support Request Contributor’ role for the selected subscription. Learn more about role assignments in the portal.

Brilliant

[mimckitt]

@LHTech-Analytics this will get assigned to one of our engineers who work with data-factory sometime today. That being said, the purpose of this repo is to resolve documentation issues. So if this issue is outside of a doc issue, we will request you either reach out to the MSDN or Stack Overflow community or open a technical support request. As per the support request, if your subscription is managed by a CSP (Cloud Service Provider) you would need to contact them to open that ticket for you.

[LHTech-Analytics]

@MicahMcKittrick-MSFT as per my previous comment, I am not allowed to raise a support request as I do not have permissions. This is an issue with documentation as I have explained in great detail above. There is a link in the ADF UI app that links to this page but the page just has a load of JSON snippets and mentions absolutely nothing about running the batch execution module in the visual interface.

I'm actually quite annoyed now. If I created an application and you were using it, I wouldn't expect you to just understand exactly how it works. And I would be willing to provide support to you as a paying customer. I wouldn't be happy with you being stuck for several days because my documentation doesn't marry with the task you were trying to complete.

[LHTech-Analytics]

[LHTech-Analytics]

This is the visual interface I'm working with. As you can see, it does not involve writing raw JSON, it instead prompts you to go through the wizard style steps to specify inputs and outputs. This generates JSON but as I have explained many times now, the format of the produced JSON does not match the format in the documentation article that is linked to from here.

[LHTech-Analytics]

Again, the error I get is:

{
"errorCode": "2600",
"message": "Batch Execution failed. The response from the Machine Learning service at endpoint https://europewest.services.azureml.net/workspaces/3b558adebdec45bb921d258fc00bde83/services/93726a7d7ae3464e842cb3814c0f27a0/jobs was 'Invalid argument provided. Invalid service input name(s) specified in the request: RemovedKey. Please map the input data to the correct service inputs and try again.'.",
"failureType": "UserError",
"target": "ML Batch Execution1"
}

There is literally zero information online in what error code 2600 means or more importantly how to resolve it. I'm looking at other options now because I can't afford to waste any more days on this :(

[mimckitt]

Thanks for all the details. Again, I don't work on data-factory but am part of the community docs team. I simply added the correct tag to this issue so we can route it correctly. One of our engineers will investigate and get back to you.

[LHTech-Analytics]

The annoying thing @MicahMcKittrick-MSFT is that this is probably a simple fix and I just need to be made aware of simple change that needs to be made. Very irritating that MS and other companies these days seek to commoditise their support function instead helping their customers with basic understandings such as this. According to your CEO, the success of MS/Azure customers = the ongoing success of MS/Azure but the nature of the relationship actually seems a lot more one sided to me right now.

[LHTech-Analytics]

HI @MartinJaffer-MSFT, if you have any assistance to provide, it would be much appreciated. I'm now looking down the route of just doing all this in a Flask App if possible as I am able to call BES from a python script using the API. I just don't understand why I'm having such a headache with a supposedly simple alternative such as ADF UI. Really hoping you can help here.

[MartinJaffer-MSFT]

Hello @LHTech-Analytics and apologies for your distress. I will assign to author, @nabhishek .

I created an activity and compared your pipeline JSON to mine, it looks pretty much the same, other than the referenceName.

I also compared both to the longer JSON sample in the doc and the format looks similar.

I will attempt to work through the tutorial and will provide an update as soon as I can.

[LHTech-Analytics]

Hi @MartinJaffer-MSFT, thanks for the update and my apologies for coming across as distressed. Just can be somewhat frustrating getting stuck like this.

The crux of the issue is that I am using the visual interface (ADF UI as per screenshot above) that does not require manually writing the JSON to create linked services to the web service and the input and output files. The docs however just go over writing Raw JSON and I think the doc article is talking about V1 as it doesn't mention the visual interface at all and does not step through any examples of how to do this in the visual interface which is specifically what I am trying to do.

What I'd expect to see on the docs page is instructions of how to complete the wizard for the Batch Execution activity in the UI visual editor. This generates the underlying JSON but I should not need to actually edit the JSON directly. here's the screenshot of the visual UI again with the four wizard sections:

1. General - Specify the name and timeout parameters of the BES activity
2. Azure ML - Specify the linked service for the web service (This linked service definition is also created visually without needing to write JSON)
3. Settings - Where you specify the linked service to your data storage and specify the input data and the output data (These linked service definitions have also been created visually without needing to write JSON)
4. User properties - Nothing needs to be done in this section from what I can see

I'm going to try using an instance of DF V1 today to see if it is more inline with your currents docs. I look forward to your response :)

Thanks,

Luke

[LHTech-Analytics]

Hi @nabhishek, I see you've been assigned to look at this. I've just added some more info above so hoping you can provide some clarity.

Thanks,

Luke

[MartinJaffer-MSFT]

@LHTech-Analytics After conferring with my colleage, this is what I have found. @nabhishek can you confirm I have this correct?

Under the Settings section...

The Web Service Inputs and Outputs correspond to the Datastores or data references. The Global parameters (not expanded at bottom of the image) would be any parameters or environment variables passed to the python script.

[LHTech-Analytics]

Hi @MartinJaffer-MSFT and @nabhishek , yes this is the interface I am using. The error I'm getting is 2600 as above and says that there is an issue with the input. When I use the same input with the python SDK though it runs fine so I know there's no issue with the input.

My main problem is that the docs (That you are sent to by clicking the documentation link in the general section of the visual UI) don't relate to the visual interface as shown in your (And my previously added) screenshots. They are advising to perform these actions with JSON alone which is more in line with old version of ADF which does not have a visual interface.

I need to see docs that relate to using the visual interface as per your screenshot and not the olsd ADF V1 JSON only version.

Thanks,

Luke

[LHTech-Analytics]

Hi @MartinJaffer-MSFT and @nabhishek, any updates on this? Why does the documentation not cover how to run the BES in the visual UI? And why am I getting the 2600 error?

Thanks,

Luke

[KranthiPakala-MSFT]

@nabhishek Can you please have a look at this? - Thanks!

[LHTech-Analytics]

Thanks @KranthiPakala-MSFT

[LHTech-Analytics]

Hi @nabhishek, I'd really appreciate a response if possible considering this was raised 7 days ago now. The docs do not match the workflow in ADF UI. They appear to be wrong and relate to ADF V1 which is not a visual interface and has a JSON editor only. I am trying to use the visual interface (For which I can find no docs whatsoever) and it is failing with error 2600.

I'd really like to use DF but if I can't rely on the docs or quick responses when I point out errors in the docs, I have no option but to look at alternative ways to do this. Again @MartinJaffer-MSFT shared a screenshot of the same UI I am using but again the docs do not describe this interface. It describes ADF V1. there are no docs for the visual UI interface as per @MartinJaffer-MSFT's screenshot from what I can see and there is no information online relating to the 2600 error.

[MartinJaffer-MSFT]

Linking to MSDN issue