Cli: Release v51 breaks mdapi deployment

[aly76]

Summary

In release v51, force:mdapi:deploy now uses the REST API by default, which is failing with a JSON parser error. See below for error stack.

Steps To Reproduce:

1. Run the mdapi deployment without the --soapdeploy parameter

$ sfdx force:mdapi:deploy -u <env> -d <dir> --json -l RunSpecifiedTests -r skip

Expected result

Deployment is queued

Actual result

{
  "status": 1,
  "name": "Error",
  "message": "JSON_PARSER_ERROR: Can not deserialize: unexpected array at [line:1, column:73]",
  "exitCode": 1,
  "commandName": "MdapiDeployCommand",
  "stack": "Error: JSON_PARSER_ERROR: Can not deserialize: unexpected array at [line:1, column:73]\n    at Request._callback (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/salesforce-alm/dist/lib/core/force.js:590:29)\n    at Request.self.callback (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/request/request.js:185:22)\n    at Request.emit (events.js:315:20)\n    at Request.EventEmitter.emit (domain.js:467:12)\n    at Request.<anonymous> (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/request/request.js:1154:10)\n    at Request.emit (events.js:315:20)\n    at Request.EventEmitter.emit (domain.js:467:12)\n    at IncomingMessage.<anonymous> (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/request/request.js:1076:12)\n    at Object.onceWrapper (events.js:421:28)\n    at IncomingMessage.emit (events.js:327:22)\n    at IncomingMessage.EventEmitter.emit (domain.js:467:12)\n    at endReadableNT (internal/streams/readable.js:1327:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21)\nOuter stack:\n    at Function.wrap (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/@salesforce/core/lib/sfdxError.js:171:27)\n    at MdapiDeployCommand.catch (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/salesforce-alm/dist/ToolbeltCommand.js:248:46)\n    at MdapiDeployCommand._run (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/@salesforce/command/lib/sfdxCommand.js:85:13)\n    at Config.runCommand (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/@oclif/config/lib/config.js:173:24)\n    at Main.run (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/@oclif/command/lib/main.js:27:9)\n    at Main._run (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/node_modules/@oclif/command/lib/command.js:43:20)\n    at Object.run (/Users/alan.ly/.local/share/sfdx/client/7.88.4-3b2e55c3f1/dist/cli.js:121:21)",
  "warnings": []
}

Additional information

_Feel free to attach a screenshot_.

SFDX CLI Version(to find the version of the CLI engine run sfdx --version):
Version 7.88.4
SFDX plugin Version(to find the version of the CLI plugin run sfdx plugins --core)
Version 51.0.4
OS and version:
macOS Catalina Version 10.15.7

[github-actions[bot]]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[azlam-abdulsalam]

The current workaround is to use --soapdeploy and everything works as expected

[zerkz]

force:source:deploy also seems to be defaulting to REST API deploys , but this command does NOT seem to have the --soapdeploy switch.

[aheber]

This seems to also be a failure to follow a proper procedure for deprecating the restDeploy configuration property. Because of bugs in the REST Metadata API we now have scripts failing and our existing method to manage this has failed without alternative.

We do several force:source:deploy actions in our scratch org setup script and depending on the mood of the REST api they are failing. Converting to using force:mdapi:deploy would be a huge effort for us.

[uip-robot]

This issue has been linked to a new work item: W-8936983

[shetzel]

There are 2 bugs here. 1 for the JSON parse error and 1 for the unintended switch of source commands to use REST. @aheber You're absolutely right that this was a failure to properly follow the deprecation procedure. We'll be switching the source commands back to SOAP asap. Apologies for this regression!

[shetzel]

@aheber can you please post the errors you're seeing with force:source:deploy using the REST API, either here or as separate github issues? It's easy to switch it back to SOAP and provide another way to use REST, but we'd like to address the REST bugs you're seeing now.

[aheber]

@shetzel #860

[zerkz]

For those who do not want to rely on this new default behavior till things are fixed, I think this works well for the non-npm (and possibly npm installed?) sfdx-cli.

sfdx plugins:install [email protected]

This rolls back the core plugin back to a version. Weirdly enough trying to downgrade the CLI package itself did not work.

Even if your CI is set to run sfdx update or you have the related env set, the plugin will not roll forward from my experience. Once you are confident you want to upgrade, then you can run `sfdx plugins:install salesforcedx@latest) to update to latest.

[shetzel]

The REST API default change coincided with the Salesforce Spring 2021 release. I.e., salesforcedx v50.x --> salesforcedx v51.x. If you specifically install a v50.x version of that plugin it'll use SOAP instead. However, you're now pinned to that plugin version and updating your CLI will not update the salesforcedx plugin. You'll have to uninstall that specific plugin version to go back to the core version bundled with the CLI, or explicitly install a newer version.

[shetzel]

@aly76 - This specific error is due to the CLI sending the runTests property as a string but the REST API expects an array. That is an easy fix and we'll let you know when it's in latest-rc.

[shetzel]

This has been fixed and is now in production