Azure-cli: az ad sp list --query will traceback while attempting to filter a nullable field

[kirk-spragg]

Describe the bug

Trying to filter service principles based on publisherName (or indeed any other field that is nullable) results in a traceback. For example:

az ad sp list --query "[?contains(publisherName, 'someName')]

produces the following error:

The result is not complete. You can still use '--all' to get all of them with long latency expected, or provide a filter through command arguments
The command failed with an unexpected error. Here is the traceback:

In function contains(), invalid type for value: None, expected one of: ['array', 'string'], received: "null"
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\knack\knack\cli.py", line 206, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\azure-cli-core\azure\cli\core\commands\__init__.py", line 572, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\knack\knack\cli.py", line 162, in raise_event
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\knack\knack\query.py", line 46, in filter_output
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\parser.py", line 509, in search
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\visitor.py", line 94, in visit
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\visitor.py", line 180, in visit_filter_projection
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\visitor.py", line 94, in visit
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\visitor.py", line 171, in visit_function_expression
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\functions.py", line 80, in call_function
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\functions.py", line 91, in _validate_arguments
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\functions.py", line 98, in _type_check
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-dsvrkf2x\jmespath\jmespath\functions.py", line 113, in _type_check_single
jmespath.exceptions.JMESPathTypeError: In function contains(), invalid type for value: None, expected one of: ['array', 'string'], received: "null"

To Reproduce:

Run the command above. Appears to happen if you attempt to filter on ANY nullable field. So the following will also break as 'homepage' is null in may default service principles:

az ad sp list --query "[?contains(hompage, 'Seequent')]" --all

Expected Behavior

The command produces a list of matching service principles.

Environment Summary

Windows-10-10.0.17763-SP0
Python 3.6.6
Shell: cmd.exe

azure-cli 2.0.65

Additional Context

[adewaleo]

@kirk-spragg

Try this, you misspelled 'homepage' :

az ad sp list --query "[?contains(homepage, 'Seequent')]"

and if it still fails then try this:

az ad sp list --query "[?contains(to_string(homepage), 'Seequent')]"

Explanation:

The reason for to_string() is that some service principals have their homepage set to null which is not an array or string. So the jmespath library fails.

This can be remedied by calling to_string on each object's homepage field so that homepage is always a string for the purposes of contains()

Try this query to see which of your service principals don't have a homepage:

az ad sp list --query "[?contains(to_string(homepage), 'null') == \`true\`].{appId: appId, homepage: homepage}"

[adewaleo]

Still the CLI should not throw a stack trace. So I will mark this as a bug. The CLI should gracefully handle this error.

[kirk-spragg]

@adewaleo I did not realize that i needed to do a string conversion in my JMESPath query, thank you for pointing that out. I've got it working my making the change you suggested.

Having the az cli report some sort of useful error instead of a traceback would be useful for users like me who are not so familiar with the az cli or JMESPath query syntax.

[adewaleo]

You are right @kirk-spragg. I will add some logic to try to catch these jmespath errors and redirect users to CLI and JMESPATH documentation.

[smereczynski]

This bug is more than annoying. :)

[ghost]

az ad sp list --query "[?contains(objectId, 'b020932b-b2df-43ed-9c9f-27b7e91af5ad')].servicePrincipalNames" --output tsv --all

And after HOURS, I received this error message:

_The command failed with an unexpected error. Here is the traceback: Traceback (most recent call last):
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\knack\knack\cli.py", line 206, in invoke
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\azure-cli-core\azure\cli\core\commands__init__.py", line 575, in execute
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\azure-cli-core\azure\cli\core\commands__init__.py", line 633, in _run_jobs_serially
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\azure-cli-core\azure\cli\core\commands__init__.py", line 624, in _run_job
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\azure-cli\azure\cli\command_modules\role\commands.py", line 69, in graph_err_handler
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\azure-cli-core\azure\cli\core\commands__init__.py", line 611, in _run_job
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\azure-cli\azure\cli\command_modules\role\commands.py", line 49, in transform_graph_objects_with_cred
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\paging.py", line 143, in __next__
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\paging.py", line 129, in advance_page
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\azure-graphrbac\azure\graphrbac\operations\service_principals_operations.py", line 156, in internal_paging
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\service_client.py", line 336, in send
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\pipeline__init__.py", line 197, in run
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\pipeline__init__.py", line 150, in send
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install76yvurxi\msrest\msrest\pipeline\requests.py", line 72, in send
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\pipeline\requests.py", line 137, in send
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\pipeline__init__.py", line 150, in send
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\pipeline\requests.py", line 193, in send
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\msrest\msrest\universal_http\requests.py", line 333, in send
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-
76yvurxi\msrest\msrest\universal_http\requests.py", line 142, in send File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\requests\requests\sessions.py", line 533, in request
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\requests\requests\sessions.py", line 686, in send
File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-76yvurxi\requests\requests\models.py", line 828, in content
MemoryError_

I tried also with the to_string but happens the same...

Any solution or workaround?