Azure-sdk-for-go: Error on List function of Consumption Usagedetails

[svaroop]

import path 'services/consumption/mgmt/2018-03-31/consumption'

--pseudo code

List, err := usageClient.List(ctx, expand, filter, skipToken, &top, "")
if err != nil {
    log.Println("Error In Getting Usage List : ")
    log.Println(err.Error())
    log.Println("")
    break
}

getting this error after first iterations using next token.
using top value as 1000

consumption.UsageDetailsClient#List: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="400" Message="id query parameter is not a valid Guid (Request ID: 80c46708-2afa-4d1a-b155-f1c692bd0fa6)"

[jhendrixMSFT]

My internal subscription doesn't have access to this functionality so I can't repro this yet.

[zbynekbotlo]

Any update on this? I'm getting similar error when trying to request consumption data with more than 1000 results. Fails on the second page (page.Next()) with following error:
consumption.UsageDetailsClient#listNextResults: Failure responding to next results request: StatusCode=400 -- Original Error: autorest/azure: error response cannot be parsed: "The request is badly formed." error: invalid character '<' looking for beginning of value

[joshgav]

We're working on getting a repro still, sorry for the delay!

cc @parisa-naeimi

[edesai]

Any updates here ? What was the resolution or workaround ?

[jhendrixMSFT]

Sadly there hasn't been any resolution yet. :( If you have a repro it would help out to set AZURE_GO_SDK_LOG_LEVEL to Debug and provide the complete request/response so we can investigate. Note that this requires go-autorest version v10.15.0 or greater to work.

[edesai]

Enabled the debug logs.

(2019-02-25T14:23:56.2246739-08:00) INFO: REQUEST: GET https://management.azure.com//subscriptions/<subscriptionId>//providers/Microsoft.Consumption/usageDetails?%24expand=properties%2FmeterDetails&%24filter=properties%2FusageStart+eq+%272019-02-01T00%3A00%3A00Z%27+and+properties%2FusageEnd+eq+%272019-02-14T00%3A00%3A00Z%27&%24skiptoken=317C313030307C313030307C46616C73657C547275657C30&api-version=2019-01-01
User-Agent: Go/go1.10.3 (amd64-darwin) go-autorest/v11.4.0 Azure-SDK-For-Go/v25.1.0 consumption/2019-01-01
Authorization: **REDACTED**
(2019-02-25T14:23:56.6790713-08:00) INFO: RESPONSE: 400 https://management.azure.com//subscriptions/<subscriptionId>//providers/Microsoft.Consumption/usageDetails?%24expand=properties%2FmeterDetails&%24filter=properties%2FusageStart+eq+%272019-02-01T00%3A00%3A00Z%27+and+properties%2FusageEnd+eq+%272019-02-14T00%3A00%3A00Z%27&%24skiptoken=317C313030307C313030307C46616C73657C547275657C30&api-version=2019-01-01
Expires: -1
Session-Id: cbb0bdba-61c9-467c-92b4-e607b8e352c3
Odata-Version: 4.0
X-Ms-Correlation-Request-Id: e0a5d84d-9427-40a8-b9dd-9cee7d34b4ed
X-Ms-Client-Request-Id: dd96dc44-f717-4129-bf3e-0538db98f29c
X-Content-Type-Options: nosniff
X-Ms-Routing-Request-Id: WESTUS:20190225T222356Z:e0a5d84d-9427-40a8-b9dd-9cee7d34b4ed
Date: Mon, 25 Feb 2019 22:23:56 GMT
Cache-Control: no-cache
Content-Type: application/json; odata.metadata=none
X-Ms-Request-Id: b3daed2c-f032-4ad6-9d98-b76ccda277df
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11996
Pragma: no-cache
Content-Length: 126
Access-Control-Allow-Origin: *
{"error":{"code":"400","message":"id query parameter is not a valid Guid (Request ID: b3daed2c-f032-4ad6-9d98-b76ccda277df)"}}
ERRO[0011] Error during usage listconsumption.UsageDetailsClient#List: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="400" Message="id query parameter is not a valid Guid (Request ID: b3daed2c-f032-4ad6-9d98-b76ccda277df)"
exit status 255

PS: I have noticed that the skiptoken seems to be a little different than what it was returning last week. The response of my first call (without the skiptoken ofcourse) is returning first 1000 results correctly but the second call with the skiptoken set is returning the above error.

The skiptoken is being parsed from the Response().NextLink.
For eg, skiptoken is being set to "317C313030307C313030307C46616C73657C547275657C30" for the Response().NextLink below:
https://management.azure.com/subscriptions//providers/Microsoft.Consumption/usageDetails?$expand=properties/meterDetails&$filter=properties/usageStart eq '2019-02-01T00:00:00Z' and properties/usageEnd eq '2019-02-14T00:00:00Z'&api-version=2019-01-01&sessiontoken=9:12577158&$skiptoken=317C313030307C313030307C46616C73657C547275657C30&skiptokenver=v1&id=d36f096f-7044-4c4e-8238-35e8a12da276

Let me know if you need more info.

[jhendrixMSFT]

Thanks this is helpful. Can you please post the code you're using?

[edesai]

Code snippet below:

usageClient := consumption.NewUsageDetailsClient(*res.SubscriptionID)

        usageClient.Authorizer = authorizer

        filterStr := "properties/usageStart eq '2019-02-01T00:00:00Z' and properties/usageEnd eq '2019-02-14T00:00:00Z'"

    next:
        ctx := context.Background()
        usageList, err := usageClient.List(ctx, "/subscriptions/"+*res.SubscriptionID+"/", "properties/meterDetails", filterStr, skiptoken, nil, "")
        if err != nil {
            logrus.Error("Error during usage list", err)
            os.Exit(-1)
        }

        skiptoken = ""
        if usageList.Response().NextLink != nil {
            if *usageList.Response().NextLink != "" {
                splitStrRegexp := regexp.MustCompile("\u0026")
                splitStr := splitStrRegexp.Split(*usageList.Response().NextLink, -1)
                for _, v := range splitStr {
                    skiptokenRegexp := regexp.MustCompile("skiptoken=")
                    skiptokenStr := skiptokenRegexp.Split(v, -1)
                    if len(skiptokenStr) > 1 {
                        skiptoken = skiptokenStr[1]
                    }
                }
            }
        }

        printValues(usageList.Values())

        for true {
            if skiptoken == "" {
                break
            }
            goto next
        }

[edesai]

Another code snippet that I tried out today using NextWithContext(). I am not sure if I am using it the right way though. It returned a different error.

usageClient := consumption.NewUsageDetailsClient(*res.SubscriptionID)

        usageClient.Authorizer = authorizer

        filterStr := "properties/usageStart eq '2019-02-01T00:00:00Z' and properties/usageEnd eq '2019-02-14T00:00:00Z'"
        fmt.Println("filterStr=", filterStr)

        ctx := context.Background()
        usageList, err := usageClient.List(ctx, "/subscriptions/"+*res.SubscriptionID+"/", "properties/meterDetails", filterStr, skiptoken, nil, "")
        if err != nil {
            logrus.Error("Error during usage list", err)
            os.Exit(-1)
        }

        printValues(usageList.Values())

        for usageList.NotDone() {
            err = usageList.NextWithContext(ctx)
            if err != nil {
                fmt.Println(err, "Could not go to move to next result page")
                return
            }
            printValues(usageList.Values())
        }

The request and response as below:

(2019-02-26T17:50:22.7401609-08:00) INFO: REQUEST: GET https://management.azure.com/subscriptions/<subscriptionid>/providers/Microsoft.Consumption/usageDetails?$expand=properties/meterDetails&$filter=properties/usageStart eq '2019-02-01T00:00:00Z' and properties/usageEnd eq '2019-02-14T00:00:00Z'&api-version=2019-01-01&sessiontoken=22:12415896&$skiptoken=317C313030307C313030307C46616C73657C547275657C30&skiptokenver=v1&id=7f41b78c-486e-453e-92c5-f912e31a6091
Content-Type: application/json
User-Agent: Go/go1.10.3 (amd64-darwin) go-autorest/v11.4.0 Azure-SDK-For-Go/v25.1.0 consumption/2019-01-01
Authorization: **REDACTED**
(2019-02-26T17:50:22.7493430-08:00) INFO: RESPONSE: 400 https://management.azure.com/subscriptions/<subscriptionid>/providers/Microsoft.Consumption/usageDetails?$expand=properties/meterDetails&$filter=properties/usageStart eq '2019-02-01T00:00:00Z' and properties/usageEnd eq '2019-02-14T00:00:00Z'&api-version=2019-01-01&sessiontoken=22:12415896&$skiptoken=317C313030307C313030307C46616C73657C547275657C30&skiptokenver=v1&id=7f41b78c-486e-453e-92c5-f912e31a6091
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 27 Feb 2019 01:50:22 GMT
Content-Length: 311
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request</h2>
<hr><p>HTTP Error 400. The request is badly formed.</p>
</BODY></HTML>

consumption.UsageDetailsClient#listNextResults: Failure responding to next results request: StatusCode=400 -- Original Error: autorest/azure: error response cannot be parsed: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITLE>Bad Request</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Bad Request</h2>\r\n<hr><p>HTTP Error 400. The request is badly formed.</p>\r\n</BODY></HTML>\r\n" error: invalid character '<' looking for beginning of value Could not go to move to next result page 

[svaroop]

Error occurred when i was trying to get usage of enterprise account using csp rest api. got correct values when i used rest api specified in below link

https://docs.microsoft.com/en-us/rest/api/billing/enterprise/billing-enterprise-api-usage-detail

[edesai]

^Not using for an enterprise account.

[jhendrixMSFT]

@edesai thanks I was going to suggest you use the methods on the returned page type to iterate over the results.
It looks to me that the problem here is the query params in the returned URL for the next link aren't properly encoded. In the consumption/models.go file could you replace the existing method with this version.

func (udlr UsageDetailsListResult) usageDetailsListResultPreparer(ctx context.Context) (*http.Request, error) {
    if udlr.NextLink == nil || len(to.String(udlr.NextLink)) < 1 {
        return nil, nil
    }
    u, err := url.Parse(*udlr.NextLink)
    if err != nil {
        return nil, err
    }
    q, err := url.ParseQuery(u.RawQuery)
    if err != nil {
        return nil, err
    }
    u.RawQuery = q.Encode()
    return autorest.Prepare((&http.Request{}).WithContext(ctx),
        autorest.AsJSON(),
        autorest.AsGet(),
        autorest.WithBaseURL(u.String()))
}

This will properly encode the query parameters and will prove/disprove my theory.

[edesai]

That did the work! Thanks. So how does this work .. will a new version come out with the fix ?

[jhendrixMSFT]

So to summarize I think there are several issues here.

• The List API taking just a skip token isn't sufficient (you need to pass the ID and likely the skip token version as well).
• Parsing the skip token out of the next URL is not user-friendly (there might be some ways to simplify this, like getting the skiptoken value directly from the HTTP response object but it's still not ideal).
• The nextLink URL provided by the service doesn't encode the query params so it's effectively useless.

I would like to avoid having to always round-trip the nextLink URL like this as most services return a valid URL so I will follow up with the service team about getting it fixed. I'll also bring up the other points to see what can be done about them.
As for a more immediate fix it will depend on how fast the service team can turn around fixing the nextLink URL. I'll discuss with my team and follow up.

[hapticPaper]

Im having the same issue using the python SDKs. (ConsumptionManagementClient)
b'{"error":{"code":"400","message":"id query parameter is not a valid Guid (Request ID: ab3d3db2-42d3-40db-bea3-e99d1cb692ec)"}}'
What was the solution so that I can try to reproduce it?

[hotsnow]

Already two years. I still get this problem. Any update on this?

[hotsnow]

I'm sending a PR for this bug: https://github.com/Azure/azure-sdk-for-go/pull/12641

fix id missing for consumption

1. when requesting the usage detail api with consumption package, it's ok
for the first request, but it will failed when request for the next page
with only skiptoken. The return error:
{"error":{"code":"400","message":"id query parameter is not a valid Guid
    (Request ID: xxxxxxx
    consumption.UsageDetailsClient#List: Failure responding to request:
    StatusCode=400 -- Original Error: autorest/azure: Service returned
    an error. Status=400 Code="400" Message="id query parameter is not a
    valid Guid (Request ID: xxxxxxx
Add another parameter id from the nextLink will fix this bug.

2. this error also happens in the "try it" document page:
https://docs.microsoft.com/en-us/rest/api/consumption/usagedetails/list#code-try-0
it will respond the same error if didn't provide the id parameter

[jhendrixMSFT]

Sorry for the delay, I have a fix to encode the query parameters in autorest.WithBaseURL().

[jhendrixMSFT]

The query params fix is in module autorest/v0.11.8 if somebody can try it out.

[hotsnow]

@jhendrixMSFT It doesn't works for me. Still respond the same error:
consumption.UsageDetailsClient#List: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="400" Message="id query parameter is not a valid Guid (Request ID: 95ba4e09-0a08-4c1a-ae43-d6a23e15e8cb)"

my related go.mod
require (
¦ github.com/Azure/azure-sdk-for-go v46.4.0+incompatible
¦ github.com/Azure/go-autorest/autorest v0.11.8 // indirect
¦ github.com/Azure/go-autorest/autorest/azure/auth v0.5.2
¦ github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
¦ github.com/Azure/go-autorest/autorest/validation v0.3.0 // indirect

)

[jhendrixMSFT]

@hotsnow thanks for trying it out. Would you mind stepping through the code to see if the query params aren't getting encoded for some reason? Maybe there's something else at work here. I don't have a repro of this at present (working on it).

[jhendrixMSFT]

Per offline conversation with @hotsnow this has been resolved.

I've opened an issue about the missing skiptokenver and id query params in the swagger for manual paging.

https://github.com/Azure/azure-rest-api-specs/issues/11351