Azure-sdk-for-java: [BUG] Export API schema - running into BOM complications

[d2baker]

Describe the bug
Attempting to use
apiManagementManager.inner().apiExports()
.get(apimResourceGroupName, apimServiceName, id, exportFormat);

within SpringBoot

Causes an exception which indicates it fails due to inclusion of a Byte Order Mark (BOM) on Jackson transformation to an ApiExportResultInner object.

Exception or Stack Trace
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('' (code 65279 / 0xfeff)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: (String)'{'link':'https://apimgmtstbp9kg0aataxe9es.blob.core.windows.net/api-export/Aircraft Configuration.json?sv=2017-04-17&sr=b&sig=YwMImrNL62TK67PxmMUESoNpuo0937R0JKhvsfRRKVI%3D&se=2019-09-23T17:43:54Z&sp=r'}'; line: 1, column: 2]

To Reproduce
Run the code snippets to a valid API in API Manager, with any valid ExportFormat, in my case I used swagger-link, then will see the exception, instead of the expected returned object.

Code Snippet
pom.xml snippet:

com.microsoft.azure
azure-mgmt-apimanagement
1.10.0-beta

code snippet:
...
public ApiExportModel findExportByIdAndFormat(final String id, final String format) {
ExportFormat exportFormat = ExportFormat.fromString(format);
ApiExportResultInner apiExportResultInner = apiManagementManager.inner().apiExports()
.get(apimResourceGroupName, apimServiceName, id, exportFormat);
...

controller code:
...
@RequestMapping(value = '/{id}/export/{format}', method = RequestMethod.GET, produces = { MediaType.APPLICATION_JSON_VALUE )
public ResponseEntity findExportByIdAndFormat(@PathVariable('id') final String id, @PathVariable('format') final String format) {
try {
return new ResponseEntity<>(apisService.findExportByIdAndFormat(id, format), HttpStatus.OK);
} catch (Exception e) {
return new ResponseEntity<>('Nothing found', HttpStatus.NOT_FOUND);
}
}
...

calling method with
id = 'aircraft-configuration' // an API ID that works for other requests
format ='swagger-link' // (or any ExportFormat)

Expected behavior
I expected the ApiExportResultInner object to be returned from the call, but instead see the exception.

Setup (please complete the following information):

• OS: CentOS 7.6.1810 (core)
• IDE : STS4 - 4.4.0.RELEASE
• Version of the Library used: openjdk 64bit 1.8.0_222-b10, azure-mgmt-apimanagement:1.10.0-beta

[d2baker]

A note from the Azure representative helping me with the issue:
...

com.fasterxml.jackson.core.JsonParseException: Unexpected character ('' (code 65279 / 0xfeff)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: (String)'{'link':'https://apimgmtstbp9kg0aataxe9es.blob.core.windows.net/api-export/Aircraft Configuration.json?sv=2017-04-17&sr=b&sig=YwMImrNL62TK67PxmMUESoNpuo0937R0JKhvsfRRKVI%3D&se=2019-09-23T17:43:54Z&sp=r'}'; line: 1, column: 2]

Looking at the error message.. there is a BOM (byte order mark) at line 1, column 2 which is between the two visible characters.. '{ ..I verified it using https://www.soscisurvey.de/tools/view-chars.php .. it resolves to U+FEFF which represents BOM character..

I discussed this with the product group and they also think this may be an issue on our side and have asked me to request you if it may be possible for you to open an issue here.. https://github.com/Azure/azure-sdk-for-java/issues
...

[yaohaizh]

@ChenTanyi Could you help take a look at this issue? Thanks.

[ChenTanyi]

@d2baker This error is due to the return string has unexpected char making JSON parse error, which needs service team to fix it.

@mjconnection Could you please take a look at this issue? The following request would return unexpected chars 'xefxbbxbf', which makes JSON parse error.

Sample Example:

• request url:

https://management.azure.com/subscriptions/1c638cf4-608f-4ee6-b680-c329e824c3a8/resourceGroups/testapiUeW1v/providers/Microsoft.ApiManagement/service/testapiUeW1v/apis/echo-api?format=swagger-link&export=true&api-version=2018-01-01

• response body:
  
  
  
  • string format:
    
    \xef\xbb\xbf{"link":"https://apimstenhwfq8phslpgbb7kg.blob.core.windows.net/api-export/Echo API.json?sv=2017-04-17&sr=b&sig=laOSRM6LNbAo2syNOz9x7y97G2ksoF04Iw%2BmLhaHvqM%3D&se=2019-09-25T03:28:23Z&sp=r"}
  
  
  • hex format:
    
    efbbbf7b226c696e6b223a2268747470733a2f2f6170696d7374656e68776671387068736c70676262376b672e626c6f622e636f72652e77696e646f77732e6e65742f6170692d6578706f72742f4563686f204150492e6a736f6e3f73763d323031372d30342d31372673723d62267369673d6c614f53524d364c4e62416f3273794e4f7a39783779393747326b736f46303449772532426d4c68614876714d2533442673653d323031392d30392d32355430333a32383a32335a2673703d72227d
  
  

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @mjconnection

[solankisamir]

@ChenTanyi we tried to reproduce the issue by directly calling the REST endpoint, I cannot see these characters in the response Content. Can this UTF-8 BOM characters be handled in the Java SDK?

We are not seeing this issue in the .NET SDK, which is able to parse the same Json response fine.

[ChenTanyi]

@solankisamir The JSON parse was used in base of the structure, I'm afraid it would affect other code. @yaohaizh @weidongxu-microsoft Could you take a look?

Not only the Java, python would also fail to parse this JSON by default.

[weidongxu-microsoft]

@solankisamir
Is the problem that you cannot reproduce the BOM, or you want to have the BOM and force SDK handle it?

If first, you can call
curl "https://management.azure.com/subscriptions/1c638cf4-608f-4ee6-b680-c329e824c3a8/resourceGroups/testapiUeW1v/providers/Microsoft.ApiManagement/service/testapiUeW1v/apis/echo-api?format=swagger-link&export=true&api-version=2018-01-01" -H "your auth header" > output.txt
Then
od -bc output.txt
You can see the BOM clearly

0000000 357 273 277 173 042 154 151 156 153 042 072 042 150 164 164 160
        357 273 277   {   "   l   i   n   k   "   :   "   h   t   t   p
0000020 163 072 057 057 141 160 151 155 163 164 145 156 150 167 146 161
          s   :   /   /   a   p   i   m   s   t   e   n   h   w   f   q
0000040 070 160 150 163 154 160 147 142 142 067 153 147 056 142 154 157
          8   p   h   s   l   p   g   b   b   7   k   g   .   b   l   o
0000060 142 056 143 157 162 145 056 167 151 156 144 157 167 163 056 156
          b   .   c   o   r   e   .   w   i   n   d   o   w   s   .   n
0000100 145 164 057 141 160 151 055 145 170 160 157 162 164 057 105 143
          e   t   /   a   p   i   -   e   x   p   o   r   t   /   E   c
0000120 150 157 040 101 120 111 056 152 163 157 156 077 163 166 075 062
          h   o       A   P   I   .   j   s   o   n   ?   s   v   =   2
0000140 060 061 067 055 060 064 055 061 067 046 163 162 075 142 046 163
          0   1   7   -   0   4   -   1   7   &   s   r   =   b   &   s
0000160 151 147 075 071 061 107 104 161 101 106 113 123 131 107 172 155
          i   g   =   9   1   G   D   q   A   F   K   S   Y   G   z   m
0000200 144 065 122 156 125 070 117 151 045 062 102 144 172 105 141 110
          d   5   R   n   U   8   O   i   %   2   B   d   z   E   a   H
0000220 124 171 101 160 155 160 144 067 167 164 165 160 167 143 070 131
          T   y   A   p   m   p   d   7   w   t   u   p   w   c   8   Y
0000240 045 063 104 046 163 145 075 062 060 061 071 055 060 071 055 062
          %   3   D   &   s   e   =   2   0   1   9   -   0   9   -   2
0000260 066 124 060 062 072 061 061 072 065 071 132 046 163 160 075 162
          6   T   0   2   :   1   1   :   5   9   Z   &   s   p   =   r
0000300 042 175
          "   }
0000302

If second, you really really want to have the BOM unlike other services, please get API board person to sign off, so all SDK can adjust accordingly.

[solankisamir]

@d2baker we have located the issue and will deploy the fix within the next 2 weeks.

[yaohaizh]

@solankisamir is this issue fixed on the service side.

[solankisamir]

yes, we have resolved the issue. By 10/21/2019 all service in national clouds should also have the fix

[yaohaizh]

@d2baker, close the issue as it has been fixed. Feel free to re-open it if it still persists.

[msftbot[bot]]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.