Azure-sdk-for-python: Add support to get metrics data of vm using resource manager.

[samshinde]

Add support to get metrics data of vm using resource manager In REST as well as SDK.

[siddheshwar-more]

Hey this seems nice idea- currently powershell supports cmdlets(classic ASM) to add diagnostic extension to azure resources(VM's) and to retrieve data from diagnostic storage account.
Where as with help ARM powershell cmdlets you can get metrics directly no need to query diagnostic storage table.

@lmazuel could you please help here!

[lmazuel]

Hi,

To be more specific, could you give here the list of Powershell commands you know that you would like to use from Python?

[siddheshwar-more]

@lmazuel

For ARM cmdlets helps you to get metrics data.
https://msdn.microsoft.com/en-us/library/mt282452.aspx

For ASM, retrieve metric data from diagnostic storage-

$table = Get-AzureStorageTable -Name $table_name -Context $diagnostic_storage_context

$query = New-Object Microsoft.WindowsAzure.Storage.Table.TableQuery

#Define columns to select.
$list = New-Object System.Collections.Generic.List[string]
$list.Add("RowKey")
$list.Add("IntValue")

#Set query details.
$query.SelectColumns = $list
$query.TakeCount = 20

#Execute the query.
$entities = $table.CloudTable.ExecuteQuery($query)

[lmazuel]

Unfortunately, we will not put effort on ASM in the future.
However, for ARM your point is totally something we should provide, but I can't give you an ETA for now.

I keep this issue open for the record.

[miketonks]

+1 For ARM, it would be nice to get list of metric definitions for a virtual machine, and to query for metric data, using the python API.

I managed to get a list of metric definitions from the REST API using ComputeManagementClient with url:

/subscriptions//resourceGroups//providers/Microsoft.Compute/virtualMachines//metricDefinitions

but unfortunately /metrics doesn't work and I can't find any useful documentation to help.

[solvingj]

The issue is affecting most of the third party SDK's right now, including java.

Situation is best documented here:
https://github.com/Azure/azure-xplat-cli/issues/2698

In short, VM's behave differently than other resources types right now, and it will continue to change. The metrics for VM's are stored on azure storage tables, which means you have to use a completely different API/library to get the data. As the article above says, powershell can get the metrics, but does so by invoking the .NET library for Azure storage.

Then, this was recently announced:
https://azure.microsoft.com/en-us/documentation/articles/event-hubs-streaming-azure-diags-data/

It seemed almost like diagnostics and metrics would only be available via eventhubs and storage, but I emailed one of the REST API developers about the issue with the /metrics endpoint and they said this:

"we are working on the REST api to access metrics. They will be stored in memory, so it has much better latency. You will also have the option to export these metrics to blobs and eventhubs."

I hope this helps somehow.

[lmazuel]

Thank you @jerrywiltse, yes it helps :)

[solvingj]

Finally got an example working in Java/Groovy thanks to some help on StackOverflow.

http://stackoverflow.com/questions/39112101/java-8-time-equivalent-of-net-datetime-maxvalue-ticks/39112535?noredirect=1#comment65572581_39112535

In short, from a /metricDefinition result, you can take the : partition key , table name, sasToken, and tableEndpoint. With those, you can construct most of the HTTP parameters you need to query the storage API for the actual metric values.

The hard part (outside of .NET), is replicating the math needed to generate the "RowKey" parameters. You can see the solution Andreas came up with in Java.

The RowKey column essentially acts as an indexed timestamp column. You really need to include RowKey parameters in your call, because it's the only indexed column (other than partitionKey). If you don't include it, each query does a full partition scan, and queries can take several minutes.

Note: The azure storage table has an actual timestamp column, but it's not indexed. You need to include RowKey.

http://stackoverflow.com/questions/19715885/azure-tables-query-by-rowkey-as-condition

Good luck to you!

[lmazuel]

Hi all,

I'm glad to announce the first beta release of the azure-monitor package to access in Python:

• Activity log
• Metrics of ARM resources (if resource compatible)
• Quotas of ARM resources (if resource compatible)

(Azure Monitor is the new name of Azure Insights.)

I published a quick sample to help start on this here.

Please provide feedback and do not hesitate to create new issues if necessary. I leave this issue open for a few days, but will close it since the initial question of package availability is done (bugs in usage should be a new issue)

Thanks!

[Gopalvenu]

Hi,

Thanks for the info. Is this possible to get the memory usage from azure ?

Thanks,
Gopal.

[lmazuel]

Hi @Gopalvenu
I asked the Monitor about your question:

Memory usage is only available within the guest VM, not from host OS so it won't be available via the metrics API. Current guest agent story is around WAD/LAD. There is no REST API today to retrieve the memory metric but customer can configure WAD/LAD to collect the data to their storage account.

I hope this helps,

[lmazuel]

I close this issue for now, since the initial question about "metrics as a package" is solved. Do not hesitate to open a new issue for more specific questions on the package features or issues found.
Thank you,