Azure-cli: Command `az acs create` always returns error: `Incorrect padding`

[minherz]

Problem Description
When a command az acs create is called to create ACS service, it _always_ returns an error Incorrect padding.
Environment
OS: Windows 10 Version 1607 (build 14393)
Python: 3.5.2
AZ CLI:

PS C:\Program Files> az --version
azure-cli (2.0.0)

acs (2.0.0)
appservice (0.1.1b5)
batch (0.1.1b4)
cloud (2.0.0)
component (2.0.0)
configure (2.0.0)
container (0.1.1b4)
core (2.0.0)
documentdb (0.1.1b2)
feedback (2.0.0)
iot (0.1.1b3)
keyvault (0.1.1b5)
network (2.0.0)
nspkg (2.0.0)
profile (2.0.0)
redis (0.1.1b3)
resource (2.0.0)
role (2.0.0)
sql (0.1.1b5)
storage (2.0.1)
vm (2.0.0)

Python (Windows) 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)]

[brendandburns]

Looking into this. Can you add the specific stack trace that you see?

[brendandburns]

I can't seem to reproduce this @ head. Can you provide more details about how/where you are running this?

[minherz]

@brendandburns , there is no stracktrace. Just the error. Running the command with --debug produces the following output:

PS C:\WINDOWS\system32> az acs create --debug --orchestrator-type=kubernetes --resource-group=acs-k8s-group --name=mck-k8s-cluster-test1 --dns-prefix=mck-k8s-test1
Command arguments ['acs', 'create', '--orchestrator-type=kubernetes', '--resource-group=acs-k8s-group', '--name=mck-k8s-cluster-test1', '--dns-prefix=mck-k8s-test1']
Installed command modules ['acs', 'appservice', 'cloud', 'component', 'configure', 'container', 'feedback', 'network', 'profile', 'resource', 'role', 'sql', 'storage', 'vm', 'batch', 'documentdb', 'iot', 'keyvault', 'redis']
Current active cloud 'AzureCloud'
{'active_directory': 'https://login.microsoftonline.com',
 'active_directory_graph_resource_id': 'https://graph.windows.net/',
 'active_directory_resource_id': 'https://management.core.windows.net/',
 'batch_resource_id': 'https://batch.core.windows.net/',
 'gallery': 'https://gallery.azure.com/',
 'management': 'https://management.core.windows.net/',
 'resource_manager': 'https://management.azure.com/',
 'sql_management': 'https://management.core.windows.net:8443/'}
{'azure_datalake_analytics_catalog_and_job_endpoint': 'azuredatalakeanalytics.net',
 'azure_datalake_store_file_system_endpoint': 'azuredatalakestore.net',
 'keyvault_dns': '.vault.azure.net',
 'sql_server_hostname': '.database.windows.net',
 'storage_endpoint': 'core.windows.net'}
Registered application event handler 'CommandTableParams.Loaded' at <function add_id_parameters at 0x000001E5932B86A8>
Registered application event handler 'CommandTable.Loaded' at <function add_id_parameters at 0x000001E5932B86A8>
Loaded module 'acs' in 0.413 seconds.
Loaded module 'appservice' in 0.097 seconds.
Loaded module 'cloud' in 0.002 seconds.
Loaded module 'component' in 0.002 seconds.
Loaded module 'configure' in 0.003 seconds.
Loaded module 'container' in 0.004 seconds.
Loaded module 'feedback' in 0.003 seconds.
Loaded module 'network' in 0.188 seconds.
Loaded module 'profile' in 0.002 seconds.
Loaded module 'resource' in 0.057 seconds.
Loaded module 'role' in 0.003 seconds.
Loaded module 'sql' in 0.055 seconds.
Loaded module 'storage' in 0.061 seconds.
Loaded module 'vm' in 0.007 seconds.
Loaded module 'batch' in 0.110 seconds.
Loaded module 'documentdb' in 0.014 seconds.
Loaded module 'iot' in 0.033 seconds.
Loaded module 'keyvault' in 0.054 seconds.
Loaded module 'redis' in 0.016 seconds.
Loaded all modules in 1.124 seconds. (note: there's always an overhead with the first module loaded)
Application event 'CommandTable.Loaded' with event data {'command_table': OrderedDict([('network vnet subnet create', <azure.cli.core.commands.CliCommand object at 0x000001E594776A20>), ('vmss nic list', <azure.cli.core.commands.CliCommand object at 0x000001E594B04978>), ('iot hub show-stats', <azure.cli.core.commands.CliCommand object at 0x000001E594CF4860>), ('appservice web config container update', <azure.cli.core.commands.CliCommand object at 0x000001E594507F28>), ('vmss list-instances', <azure.cli.core.commands.CliCommand object a
t 0x0000 [...]
Application event 'CommandParser.Loaded' with event data {'parser': AzCliCommandParser(prog='az', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True)}
Application event 'CommandTableParams.Loaded' with event data {'command_table': OrderedDict([('network vnet subnet create', <azure.cli.core.commands.CliCommand object at 0x000001E594776A20>), ('vmss nic list', <azure.cli.core.commands.CliCommand object at 0x000001E594B04978>), ('iot hub show-stats', <azure.cli.core.commands.CliCommand object at 0x000001E594CF4860>), ('appservice web config container update', <azure.cli.core.commands.CliCommand object at 0x000001E594507F28>), ('vmss list-instances', <azure.cli.core.commands.CliCommand ob
ject at 0x0000 [...]
Application event 'CommandParser.Parsed' with event data {'args': Namespace(_command_package='acs', _jmespath_query=None, _log_verbosity_debug=False, _log_verbosity_verbose=False, _output_format='json', _parser=AzCliCommandParser(prog='az acs create', usage=None, description='Create a new Acs.', formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True), _validators=[<function validate_ssh_key at 0x000001E5932C6EA0>, <function generate_deployment_name at 0x000001E5944FB400>], admin_username='azureuser', agent_coun [...]
Use existing SSH public key file: C:\Users\leonidy\.ssh\id_rsa.pub
Incorrect padding

The id_rsa.pub has a form:

---- BEGIN SSH2 PUBLIC KEY ----
Comment: "rsa-key-20170307"
AAAAB3NzaC1yc2EAAAABJQAAAQEA9y1L7ZDjFLJk4AJv3NXPqgxB2i0AyIfKS1sy
nMwU1RPAvQp+wm8E4xIWyQ+DLHso5safCyapPF8Dwoy9j2ElZujwqe3h5h0MkByG
qrERl1iV50PUswLmgRHFB1/mEC+BZGIX1TmcpldHETIktj8gxc4sxYILv8GeLY1V
Yo7G6ts59z7mHQvoNmdHhOToqqGW1JSsFj+eARpOvt8QZHnskS3xOoRAW5b69Y+2
2ZpnO2S1TzjeJQzP+k3jnoLCmQ/aGhN48pwIHunbJs8zDxMn9fdqQk7ZkjfOPy7y
iVkQbFgAWCpSgNjPwb5oghcXwwwzIUvWonhPVFU40grlloSrrQ==
---- END SSH2 PUBLIC KEY ----

[brendandburns]

What happens if you delete that "Comment" line at the top of the key, perhaps Parimiko doesn't like that?

[minherz]

@brendandburns , I tried it already. The result (and debug output) are exactly the same.
Although the error does not point to that direction, can the reason be in the azure account i am using? I learned that my account is somehow limited to do operation on Azure AD. For example, I cannot create service principal using CLI.

[brendandburns]

@minherz ah, I see the problem...

The code expects the "compact" form of the .ssh key

id-rsa <base64encoded> <user@server>

This is a legit bug in the code. For now, you can work around it by transforming your key to match that form using:

 ssh-keygen -i -f existing_key.pub > formatted_key.pub

[minherz]

Thank you.
Do you mark this issue as a bug?

[brendandburns]

Yeah this is bug.

[minherz]

Can you mark this issue as a bug?

[sudharsanmit]

I too faced this. I believe this is a legit bug. The bug exists both places - When trying to create a cluster thru az-cli and Azure UI. Overcome by using the single line RSA key.

[minherz]

However, it is still not marked as a bug or meant to be fixed either... :)

[brendandburns]

Marked as bug, I will be looking to fix it. In the meantime you can use:

 ssh-keygen -i -f existing_key.pub > formatted_key.pub

to convert to a format that works.

[minherz]

The workaround works. Hope the original will be handled as well. @brendandburns feel free reopen it when you plan to implement the fix

[MrkCrng]

Is there a workaround for this Incorrect Padding bug when using the BASH or Power Shell CLI? If so please specify.

[brendandburns]

@MrkCrng work around is here:

https://github.com/Azure/azure-cli/issues/2386#issuecomment-292580482

[dazinator]

Just hit this now. Using Azure CLI 2.0 and following the quickstart: https://docs.microsoft.com/en-gb/azure/container-service/dcos-swarm/container-service-swarm-mode-walkthrough

My public key looks like this:

---- BEGIN SSH2 PUBLIC KEY ----
Comment: "imported-openssh-key"
AAAAB3NzaC1yc2EAAAABJQAAAQEAuhI5BgWbrZX11/DameE1UxJ17EexMAD5kfcg
hR0w8ckTsg2hPe7XME0erGCKD5DTVs8khTwcwjpFA2yaED5clUASejNmFYzcXzlI
FuRKTOsvtiW8c97yX8sLh+s4nSkuwEnRe7ugLr6KQ9uBIeR+QNnnMKQm9f0gM8TD
W3E/MXhRzchjETqWOZC/kM/HLIYfUsNsH4zS6bqXD74NQ1UShqXmOYYYqu/RzXEe
8jRrg8xzM/zNrhc6j/ypK6sUjEEwxM8iyNL0aTRDAM3A+43UNEtb72aG1VyoGpLh
AxZmOOt3BB+Peeflz5kvsmLFzlshmMMsey9VjpB24iQ+5noVew==
---- END SSH2 PUBLIC KEY ----

[dazinator]

Transforming it to the short format worked.

[notjames]

I'm not sure if anyone else is getting this problem. The workaround specified:

ssh-keygen -i -f existing_key.pub > fixed_key.pub

Does not work. It errors with uudecode failed

My current public key is potentially already in the "compact" format which looks similar to what @brendandburns states the pubkey needs to look like except that instead of the uuencoded string preceded with "id_rsa", mine is preceded with "ssh-rsa". The rest of the key seems correct unless the keypair is supposed to be a different key type. Mine was created as an RSA protocol 2 key pair. I believe this is likely a problem with the type of key I created. However, I haven't seen any documentation that is explicit about that. I'm going to continue working with some ideas and if anything works, I'll repost.

[jimsander]

Had the same uudecode failed (running from macos fwiw)
Workaround: stored the pubkey in a variable and referenced instead

foo="ssh-rsa AAAyadayadayada desc"
az vm create . . . --ssh-key-value "$foo" . . .

[ImmutableVector]

Very new to all of this. Can anyone tell me how I use ssh-keygen -i -f existing_key.pub > formatted_key.pub?

I don't understand the work around. I am very new to all of this and trying to follow a tutorial on aks and they didn't have this problem.

[wmarcuse]

I have a hands-on workaround answer posted in #6142