Azure-sdk-for-js: Commerce API breaks on deserializing BOM character in RateCard response?

[keesschollaart81]

• Package Name:
  @azure/arm-commerce

• Package Version:
  2.1.1

• Operating system:
  macOS Mojave 10.14.6

• [x] nodejs

  • version: v10.16.3
• [ ] browser
  
  
  
  • name/version: N/A
  
  
• [x] typescript
  
  
  
  • version: 3.4.1
  
  
• Is the bug related to documentation in
  
  
  
  • [ ] README.md
  
  
  • [ ] source code documentation
  
  
  • [ ] SDK API docs on https://docs.microsoft.com
  
  

Describe the bug
When trying to download the rate card data via the commerce package, an error occurs. This has always worked but since december 20th a JSON deserialization exception occurs.

Error: Error SyntaxError: Unexpected token  in JSON at position 0 occurred in deserializing the responseBody - {"OfferTerms":[{"Credit":150.0,"EffectiveDate":"2013-12-23T00:0<<full json payload removed>>

Here in the Issue it's not showing but there is a 'weird' character at the start of the response:

I think it's a BOM (Byte order mark) that causes the deserialization to fail?

To Reproduce
Steps to reproduce the behavior:

1. Create a small app:

``` index.ts
import * as msRestAzureNode from "@azure/ms-rest-nodeauth" // "2.0.4"
import * as commerce from "@azure/arm-commerce"; // "2.1.1"
import * as fs from "fs";

const subscriptionId = "*-3703-4687-9d4e-826010b85e24";
var tenantId = "*-384b-4d25-aaba-4e6c8b0762d5";
var offerId = "MS-AZR-0063P"
var currency = 'USD'
var locale = 'en-US'
var regionInfo = 'US'

msRestAzureNode.interactiveLogin({ domain: tenantId }, (err, creds) => {
this.usageManagementClient = new commerce.UsageManagementClient(creds, subscriptionId);
this.usageManagementClient.rateCard.get(OfferDurableId eq '${offerId}' and Currency eq '${currency}' and Locale eq '${locale}' and RegionInfo eq '${regionInfo}').then((rates) => {
console.info(Done: ${rates})
}).catch((e) => {
console.log("Error, see log.txt")
fs.writeFileSync("log.txt", JSON.stringify(e));
});
});

```

Expected behavior
Work with the price card data

Additional context
I'm using this package/api to power the Azure Cost Insights DevOps Extension. Therefor, my users run this every day as part of a scheduled nightly Azure Pipeline. First failures started ±19-20 december. Did they change the API's, or...?

Also reproduced on Windows 10 with Node 12.9.1 and TS 3.7.4.

[ramya-rao-a]

Thanks for reporting @keesschollaart81

@qiaozha Can you take a look if this is an sdk issue or a service side issue?

[qiaozha]

I think it's a service side issue. the original responseBody has already contained the BOM.

[rithinskaria]

I am also facing the same issue with node sdk, tested this call in python and I got this values added to the expected output:

root@UbuntuDocker:~# od -c test1.txt | head -5
0000000 b ' \ x e f \ x b b \ x b f { "
0000020 O f f e r T e r m s " : [ ] , "
0000040 M e t e r s " : [ { " E f f e c
0000060 t i v e D a t e " : " 2 0 1 9 -
0000100 1 0 - 2 8 T 0 0 : 0 0 : 0 0 Z "

root@UbuntuDocker:~# cat test1.txt
b'\xef\xbb\xbf{"OfferTerms":[],"Meters":[{"EffectiveDate":"2019-10-28T00:00:00Z","IncludedQuantity":0.0,"MeterCategory":"Azure App Service","MeterId":"dee311f4-b516-4e64-9158-df610306a632","MeterName":"S1","MeterRates":{"0":0.1021},"MeterRegion":"NO East","MeterStatus":"Active","MeterSubCategory":"Standard Plan","MeterTags":[],"Unit":"1 Hour"},{"EffectiveDate":"2018-10-11T00:00:00Z","IncludedQuantity":0.0,"MeterCategory":"Storage","MeterId":"7baf71c7-13a4-469d-ba60-1dc5b675b90b","MeterName":"Cool Read Additional IO","MeterRates":{"0":0.002184},"MeterRegion":"AE Central","MeterStatus":"Active","MeterSubCategory":"Azure Data Lake Storage Gen2 Hierarchical Namespace","MeterTags":[],"Unit":"10K"},{"EffectiveDate":"2016-02-01T00:00:00Z","IncludedQuantity":0.0,"MeterCategory":"Virtual Machines","MeterId":"705ac022-f360-4a96-9f32-a2a982d7b70e","MeterName":"D14 v2/DS14 v2","MeterRates":{"0":1.944},"MeterRegion":"US East 2","MeterStatus":"Active","MeterSubCategory":"Dv2/DSv2 Series Windows","MeterTags":[],"Unit":"1 Hour"},{"EffectiveDate":"2019-10-28T00:00:00Z","IncludedQuantity":0.0,"MeterCategory":"Storage","MeterId":"6acfb317-2b22-48ca-b518-6161fe94e8f9","MeterName":"Protocol Operations",

[rithinskaria]

Seems like the issue is fixed!

From yesterday's curl call dump:

root@MININT-N7C590P:/mnt/c/Users/riskaria# cat ratecard.json | od -c | head -10
0000000 357 273 277 { " O f f e r T e r m s "
0000020 : [ ] , " M e t e r s " : [ { "
0000040 E f f e c t i v e D a t e " : "
0000060 2 0 1 9 - 1 0 - 2 8 T 0 0 : 0 0
0000100 : 0 0 Z " , " I n c l u d e d Q
0000120 u a n t i t y " : 0 . 0 , " M e
0000140 t e r C a t e g o r y " : " A z
0000160 u r e A p p S e r v i c e "
0000200 , " M e t e r I d " : " d e e 3
0000220 1 1 f 4 - b 5 1 6 - 4 e 6 4 - 9

When I checked today, I don't see the stream at the beginning of the response:

root@MININT-N7C590P:/mnt/c/Users/riskaria# curl --location --request GET 'https://management.azure.com/subscriptions/548f7d26-b5b1-468e-ad45-6ee12accf7e7/providers/Microsoft.Commerce/RateCard?api-version=2016-08-31-preview&%24filter=OfferDurableId+eq+%27MS-AZR-0003P%27+and+Locale+eq+%27en-US%27+and+Regioninfo+eq+%27US%27+and+Currency+eq+%27USD%27' --header 'Authorization: Bearer $token > ratedump.json

root@MININT-N7C590P:/mnt/c/Users/riskaria# cat ratedump.json | od -c | head -10
0000000 { " O f f e r T e r m s " : [ ]
0000020 , " M e t e r s " : [ { " E f f
0000040 e c t i v e D a t e " : " 2 0 1
0000060 9 - 1 0 - 2 8 T 0 0 : 0 0 : 0 0
0000100 Z " , " I n c l u d e d Q u a n
0000120 t i t y " : 0 . 0 , " M e t e r
0000140 C a t e g o r y " : " A z u r e
0000160 A p p S e r v i c e " , " M
0000200 e t e r I d " : " d e e 3 1 1 f
0000220 4 - b 5 1 6 - 4 e 6 4 - 9 1 5 8

Can somebody check and confirm this?

[qiaozha]

I think it's fixed. @keesschollaart81 please check and verify.

[keesschollaart81]

That is correct, this now works in all my test cases/customers. Would you mind to share what the root cause was?

[ramya-rao-a]

@qiaozha, There was no changes done to the packages, so my guess is that something was fixed in the service side?

[qiaozha]

@qiaozha, There was no changes done to the packages, so my guess is that something was fixed in the service side?

Yes. we haven't done anything in the package side.

[lilyjma]

Hi, since the issue was resolved and there hasn't been any activity on this issue for a while, I'll close it. Thanks!