Cli-microsoft365: Bug report: Error thrown when parsing JSON output of `spo sites classic list` command

[garrytrinder]

Description

Error is thrown when executing o365 spo site classic list -o json command and attempting the parse the output in bash and PowerShell Core

Steps to reproduce

Bash: Execute o365 spo site classic list -o json | jq
PowerShell Core: o365 spo site classic list -o json | ConvertFrom-Json

Expected result

JSON should be parsed

Actual result

Error is thrown

bash
parse error: Unfinished string at EOF at line 1, column 65536

PowerShell Core

ConvertFrom-Json : Conversion from JSON failed with error: Unterminated string. Expected delimiter: ". Path '[28].RestrictedToRegion', line 1, position 65534.
At line:1 char:38
+ o365 spo site classic list -o json | ConvertFrom-Json
+                                      ~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

Environment

MacOS Catalina, bash, 2.4.0
Ubuntu 18.04, bash, 2.3.0

Additional Info

This is not an issue on Windows 10 when using the standard PowerShell ConvertTo-Json cmdlet, so it would appear that this is only an issue on Unix based systems.

Executing o365 spo site classic list -o json command displays the full JSON response in the terminal window as you would expect.

It is possible to redirect the JSON output the file system in bash by executing o365 spo site classic list -o json > output.json, the resulting output file contains full JSON output which is displayed in the terminal window stored on a single line. However executing the same command in PowerShell Core results the output file containing a truncated version of the JSON output.

In both bash and PowerShell Core, assigning the JSON output directly to a variable results in the variable containing truncated JSON output which is invalid JSON.

[garrytrinder]

I have a branch containing an update to the Vorpal library which changes the output to the terminal from a string to a buffer which is then streamed to stdout so we can handle large outputs.

https://github.com/garrytrinder/vorpal/tree/output-buffer

[garrytrinder]

Some further investigation on this on v2.5.0...

Zsh

Node 10 & 12

✅o365 spo site classic list -o json => returns full output in console on single line
✅o365 spo site classic list -o json > sites.json => returns full output in file on single line
✅o365 spo site classic list -o json | jq => JSON is parsed correctly and returned to console

Node 8

❌o365 spo site classic list -o json | jq => Returns parse error: Unfinished string at EOF at line 1, column 65536

PowerShell Core (MacOS)

Node 12

✅o365 spo site classic list -o json => Returns full output in console on single line
❌o365 spo site classic list -o json > pwshsites.json => Returns trimmed output in file on single line
❌o365 spo site classic list -o json | ConvertFrom-Json => Returns Conversion from JSON failed with error: Unterminated string. Expected delimiter: ". Path '[28].RestrictedToRegion', line 1, position 65534.

PowerShell Core (Windows 10)

Node 12

✅o365 spo site classic list -o json => returns full output in console on single line
✅o365 spo site classic list -o json > pwshwindowssites.json => returns full output in file on single line
✅o365 spo site classic list -o json | ConvertFrom-Json => JSON is parsed correctly and returned to console

Windows PowerShell (Windows 10)

Node 12

✅o365 spo site classic list -o json => returns full output in console on single line
✅o365 spo site classic list -o json > powershellsites.json => returns full output in file on single line
✅o365 spo site classic list -o json | ConvertFrom-Json => JSON is parsed correctly and returned to console

JQ Parsing

✅cat sites.json | jq => JSON is parsed correctly and returned to console
❌cat pwshsites.json | jq => Returns parse error: Unfinished JSON term at EOF at line 2, column 0
✅cat powershellwindows.json | jq => JSON is parsed correctly and returned to console
✅cat powershellsites.json | jq => JSON is parsed correctly and returned to console

[waldekmastykarz]

Hey @garrytrinder, anything that we can do with this issue or should we close it for now?

[garrytrinder]

PowerShell 7 has just been released, so I have a task to check to see if it's still an issue.

[garrytrinder]

Tested this on PowerShell 7 and still no change.

Therefore we should recommend that for now people should use the --query option to pass in a JMESPath query to return only the values that they require from the response and convert that to JSON, should they run into the above issue.

The example below will filter the sites by the GROUP#0 template and only return the Title and Url of the site.

$sites = o365 spo site classic list -o json --query '[?Template==`GROUP#0`].{Title:Title, Url:Url}' | ConvertFrom-Json

[waldekmastykarz]

I've found the reason of the issue with the output of CLI being truncated in PowerShell. The culprit is us calling process.exit() after executing the command. It turns out, that console.log is not synchronous and when we call process.exit(), node doesn't wait on all the output being sent to stdout and instead closes the process immediately. I don't have a solution for it just yet, but at least we have a direction where to look.

This is not related in any way to the length of the string we output. It's pure timing that it takes for stdout to process all output sent by console.log.

Additional find (https://stackoverflow.com/a/27900423):

The console functions are synchronous when the destination is a terminal or a file (to avoid lost messages in case of premature exit) and asynchronous when it's a pipe (to avoid blocking for long periods of time).
That is, in the following example, stdout is non-blocking while stderr is blocking:

[waldekmastykarz]

And seems like adding:

if (process.stdout._handle) {
  process.stdout._handle.setBlocking(true);
}

anywhere in the CLI would fix fixes the issue!