Azure-docs: Documentation is using azureml.monitoring Which is Retiring Soon

[mkrdip]

In the documentation, the image used to show code for logging into AppInsights uses functions from azureml.monitoring. It's misleading. Please include examples of storing predict params and results in AppInsights instead of ModelDataCollector from azureml.monitoring

Thanks!

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 12f83c8d-8ec9-0b56-00c3-cdafc53c90f1
• Version Independent ID: 9fb32434-54da-9789-ec8e-1d192343feb5
• Content: Monitor and collect data from Machine Learning web service endpoints - Azure Machine Learning
• Content Source: articles/machine-learning/how-to-enable-app-insights.md
• Service: machine-learning
• Sub-service: core
• GitHub Login: @Blackmist
• Microsoft Alias: larryfr

[RohitMungi-MSFT]

@mkrdip Thanks for the feedback. We are investigating the issue and will update you shortly.

[RohitMungi-MSFT]

@Blackmist Can this be reviewed and added to your backlog for an update? The SDK for azureml.monitoring says it will be retiring soon but the page is old too. Can the image be updated to use the app insights instead?

[Blackmist]

Thanks for the feedback @mkrdip ! I've created a work item in our internal DevOps to track this. We're in the middle of some other doc work currently, so this probably won't be updated immediately.

[mkrdip]

Thanks for tracking this. It's really confusing right now. What to use for model data collection. Here you go: https://github.com/MicrosoftDocs/azure-docs/issues/9406#issuecomment-638497479

[Blackmist]

Hi @mkrdip, working on a related issue and found a way to log the predict/result data using the current 'print to stdout' method recommended in the doc; create a JSON structure and print it. Here's an example:

def run(raw_data):
    try:
        data = json.loads(raw_data)['data']
        data = numpy.array(data)
        result = model.predict(data)
        info = {
            "input": raw_data,
            "output": result.tolist()
            }
        print(json.dumps(info))
        # you can return any datatype as long as it is JSON-serializable
        return result.tolist()

With this, in the analytics log view, you can expand customDimensions, Content, and the Input and Output fields will show up as child items. Since they are also JSON data in this example, they will also be expandable (to show all elements in the list for example.)

I'll update the doc to talk about this, and provide an example in a screenshot.

[mkrdip]

Thanks for the suggestion @Blackmist, adding to official documentation would be a great help for others!

Would you please explain this in a bit, please? What's recommended to use for model-data collection? AppInsights or AML Monitoring (Stores to Blob Storage).

From this conversation, you can see https://github.com/MicrosoftDocs/azure-docs/issues/9406#issuecomment-638525797

we have removed the message that talked about retiring model data collector. Model data collector is the recommended option to log model inference data in AzureML

this documentation used to say it'll be retiring soon but removed now after I mentioned it in that issue.

But it's under testing: https://pypi.org/project/azureml-monitoring/ and the documentation say it'll be retiring soon.

[Blackmist]

@mkrdip The core problem is that logging to AppInsights has a limit of 64kb. So if your inference data is > than 64kb, the logging to AppInsights will fail. Not ideal if you're relying on it as a way to collect data for future training. That's when you want to use Azure ML Monitoring to store to blob.

I don't know the story on the retirement of azureml-monitoring. There's another person on my team working with engineering on the doc story for it currently. It may be that we received a lot of customer feedback asking to keep it around. Or the engineering team may have tested a replacement in private preview and received feedback that it wasn't enough of an improvement over the current experience.

[mkrdip]

Thanks, @Blackmist. Please keep us updated on Azure ML Monitoring SDK.

[Blackmist]

@mkrdip the latest status on the monitoring SDK is that it is not being retired.

The note about retirement has been removed from almost all the docs as of today. There's one SDK page that still has it, and we're working through getting that removed now.

please-close