Azure-cli: deployment: handle deployment templates and parameters with json comment and metadata

[yugangw-msft]

So far, CLI will fail on loading such files
For context, CLI 1.0 leverages json-minify package for better error reporting, and that package is able to strip away comments during the parsing. So coincidentally, CLI 1.0 is able to do better than 2.0.

[williexu]

@yugangw-msft
Here's a equivalent package for python we can use:
https://github.com/getify/JSON.minify/tree/python
Essentially, using this package, we should be able to strip away JS-style comments from the contents of a file before parsing it into JSON by doing:
json.loads(json_minify(contents))

[adewaleo]

@yugangw-msft, by json metadata, do you also mean comments?

[adewaleo]

@marstr, this is the issue we discussed earlier on.

Given our discussion and conclusion, a work around is to use a js minifier tool to remove comments from their json.
e.g. python -m jsmin myfile.js

[marstr]

To elaborate a little on our thinking, we want to ensure a consistent experience with ARM templates across all Microsoft properties. For example, if the Azure CLI allows for JSON with C-style comments but https://portal.azure.com doesn't, it may be jarring to have the same template work one place and not another.

Now, that's not to say that ARM templates shouldn't support C-style comments. But, instead of changing our behavior alone, it'd be better to change it server-side or coordinate with all of the other Azure clients to roll-out this functionality at the same time.

On top of all of that, as @adewaleo points out, it'd be more true to the Linux-philosophy to have a separate tool sanitize your template before passing it to az anyway.

If you're reading this and it fills you with rage, feel free to chime in and tell me why I'm wrong. We love feedback. :)

[tjprescott]

I believe the issues is that an ARM template that worked in CLI 1.0 that had comments will not work in CLI 2.0.

[adewaleo]

P.S
If you try adding comments to a quickstart template on Portal, you get the following error:
"Error parsing template. Please ensure template is valid JSON and has no comments."

This was from attempting to add a comment to one of the quickstart templates.

[onionhammer]

So why was this closed?

Why does it make sense that New-AzResourceGroupDeployment handles my azuredeploy.json file fine but az deployment sub doesn't?

[zhoxing-ms]

@onionhammer Hi, New-AzResourceGroupDeployment command is used to deploy resources under the resource group scope, but az deployment sub is used to deploy resources under the subscription scope. If you want to use CLI to deploy resources under the scope of resource group, please use az deployment group.
If you still have any questions, please feel free to let us know~

[onionhammer]

@zhoxing-ms so you're saying az deployment group does support comments in JSON?

[zhoxing-ms]

@onionhammer Yes, all the scope deployments of CLI now support comments in JSON.

The reason why CLI cannot deploy your azuredeploy.json successfully is that the command used is incorrect: when deploying under the scope of resource group, you should use the command az deployment group.
Documentation about the deployment scope

[onionhammer]

@zhoxing-ms Just tried it and see this:

az deployment group validate -g next-c2c --template-file azuredeploy.json
Failed to parse 'azuredeploy.json', please check whether it is a valid JSON format

resources> az --version
azure-cli                          2.7.0

command-modules-nspkg              2.0.3
core                               2.7.0
nspkg                              3.0.4
telemetry                          1.0.4

Extensions:
azure-devops                      0.17.0

...

Your CLI is up-to-date.

[zhoxing-ms]

@onionhammer Could you please email me the template file azuredeploy.json you used? My email address is Zhou.[email protected], thanks~

[zhoxing-ms]

@onionhammer Thanks for your email~
This problem is actually caused by the mutil-line string in the ARM template (such as resources.Configuration.WebTest )
This is a known issue. Issue #12909 is already following up on this problem.
It is recommended that you change all the mutil-line strings to single line first.

[zhoxing-ms]

@onionhammer By the way, May I ask is the configuration of mutil-line string in the resources you successfully deploy by PowerShell valid? (such as such as resources.Configuration.WebTest)
If PowerShell does not modify the ARM template and send it directly to the service ( It should be so in design) , I think the parsing logic might be break due to the newline character added by the Service.
For example:
In ARM template:

"meta": {
    "type": "string",
    "metadata": {
        "description": "meta of the network security group."
    },
    "defaultValue": "ARM
    template
    test
    newline"
}

After service processing:

"meta": {
    "type": "String",
    "value": "ARM\r\n      template\r\n      test\r\n      newline"
}

[onionhammer]

@zhoxing-ms Yep, the web tests work fine on deployment via powershell.

[zhoxing-ms]

In fact, I found out through the debug mode of PowerShell that PowerShell already adds line breaks to a multi-line string when requesting a Service, Such as:
resources.Configuration.WebTest becomes

"WebTest": "[concat(\r\n                '<WebTest Name=\"Health check\" Id=\"', guid(concat('health check-', variables('partnerInsights'))), \r\n                '\" Enabled=\"True\" ......

I think modifying the ARM template of customers before requesting a Service is not in line with the design and may destroys the customers's input to some extent
So waiting for the reply from the service team in #12909

[onionhammer]

@zhoxing-ms what's the status on this? It's been a few months, but this is still definitely an issue. Is this discrepancy a "wont fix"? Is Microsoft's preferred solution an 900+ character long string on a single line filled with variables when a user wants to create a WebTest in ARM?

If so, a ticket should be opened here https://github.com/Microsoft/vscode-azurearmtools to make multiline strings invalid, because the VS Code tools and Az PowerShell handle them fine, the azure-cli is the odd man out currently.

Also #12909 it seems -j says it's deprecated, not that it works anyway

[zhoxing-ms]

@onionhammer Because this is mainly a Service issue, not a CLI problem, I reminded relevant personnel to pay attention in this issue. #12909 (The reason why PowerShell can support it is that they have pre-processed the ARM templates, but this does not conform to our design)

[onionhammer]

Consistency of developer experience is presumably more important than some principle of not wanting to do preprocessing in the CLI tool

[zhoxing-ms]

@onionhammer Hi, this issue has been fixed by #15502
We will publish it in the new CLI version on 12-29, please try to update CLI and have a try at that time~