Aws-cli: No clear documentation for creating a aws-cli plugin

[RichardBronosky]

I've done code spelunking and found interesting ideas in:

https://github.com/aws/aws-cli/blob/c0d866a86329343b1dab11dc853f6adedcdb9c4a/awscli/plugin.py#L25-L27

https://github.com/aws/aws-cli/blob/5ce60bea0634065bb396e7e300eae315c9d11a1d/awscli/customizations/s3/s3.py#L23-L26

https://github.com/aws/aws-cli/blob/37aac2542dba9e86fffc87b17445bab9b3b850a0/awscli/customizations/commands.py#L47-L49

and finally
https://github.com/aws/aws-cli/blob/c0d866a86329343b1dab11dc853f6adedcdb9c4a/awscli/clidriver.py#L55

which lead me to try:

$ aws configure set plugins '{"mycommand": "awsplugins.mycommand"}'
$ cat ~/.aws/config
[default]
region = us-east-1
output = json
plugins = {"mycommand": "awsplugins.mycommand"}
$ aws help | grep mycommand
$ aws mycommand
usage: aws [options] <command> <subcommand> [parameters]
aws: error: argument command: Invalid choice, valid choices are:

autoscaling                              | cloudformation
cloudfront                               | cloudhsm
cloudsearch                              | cloudsearchdomain
cloudtrail                               | cloudwatch
cognito-identity                         | cognito-sync
datapipeline                             | directconnect
dynamodb                                 | ec2
ecs                                      | elasticache
elasticbeanstalk                         | elastictranscoder
elb                                      | emr
glacier                                  | iam
importexport                             | kinesis
kms                                      | lambda
logs                                     | opsworks
rds                                      | redshift
route53                                  | route53domains
sdb                                      | ses
sns                                      | sqs
ssm                                      | storagegateway
sts                                      | support
swf                                      | s3api
s3                                       | configure
deploy                                   | configservice
help

Are there any examples of a working aws-cli plugin external to the aws-cli package? Is it supposed to be possible?

[jamesls]

The big caveat is that the plugin API is subject to change which is the main reason it's not documented yet. We do have on our backlog plans to flesh out the APIs and get proper documentation around plugins. But here's the main points if you want to try it out:

In the ~/.aws/config file you'll need to add a plugins section that has the name of the package to import:

[plugins]
foo = myplugin

Then you'll need to make it so that the right hand side, in this case myplugin is importable, (i.e import myplugin works).

You then need to define an entry point, awscli_initialize that takes a single argument.

At this point, you can look at some of the built in customizations to get an idea for how these work. Just about all the code in awscli/customizations uses this interface (though instead of awscli_initialize, they typically use their own function names). For example, the cloudtrail customization has a good example of how to add a new command. The only difference is instead of initialize, you'd need to call the function awscli_initialize.

Feel free to experiment and give us feedback. Just keep in mind there's no backwards compatibility guarantee for the plugin API (yet) and it will likely change in the future. Let me know if you have any other questions.

[jamesls]

Like there was actually a regression in a recent release for the "plugins" section being set via the configure set command (https://github.com/aws/aws-cli/pull/1263). Once this is fixed, you can also configure plugins via: aws configure set plugins.name modulename

[RichardBronosky]

Thanks, @jamesls! That helps a lot. I've run into a challenge that I hope you can help me figure out. I've packaged my work up into https://github.com/RichardBronosky/aws-cli-plugins

The good news is that my awscli_initialize is being called! Unfortunately, my call to cli.register seems to do nothing. I've tried passing an inject_commands function like I've seen in https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudtrail.py#L35 and tried passing the add_command method that my class inherits from BasicCommand like I've seen in https://github.com/aws/aws-cli/blob/master/awscli/customizations/configure.py#L29-L31

I'm trying to get my head around https://github.com/boto/botocore/blob/develop/botocore/hooks.py#L297 but it seems like it's going to require a complete understanding of botocore. I think my problem may have something to do with my use of register vs. register_first or register_last. Or maybe the event_name. I did try calling-command.helloworld as seen in https://github.com/aws/aws-cli/blob/master/awscli/customizations/cloudsearchdomain.py#L22 with no effect.

Can you point me in the right direction to get my command registered?

[RichardBronosky]

https://github.com/RichardBronosky/aws-cli-plugins/commit/d45711e35f15cdb5a2f0b0450f88906f5971fd8d
Not sure if this is the right thing to do, but I found that calling cli.register with 'building-command-table.main' gets all the way through inject_commands and HelloWorld.__ini__

It seems that I have some confusion about where "helloworld" and "say-hello" commands and subcommands are separated. A little more Trial & Exception should get me there.

I'm seeing a lot of results here https://github.com/aws/aws-cli/search?q=building-command-table.main but, if you can confirm to me that 'building-command-table.main' is the right approach, that'd help.

[jamesls]

Yes, building-command-table.main is the event you'll need to add a command to the main command table (i.e the commands available after aws). After that, to add subcommands to a command (the commands available after aws <firstcommand>), then the event name will be building-command-table.yourcommandname.

As for the way the events works, the main difference between this system and other publish/subscribe event systems is that the CLI events are hierarchical. It still works with exact event names as well. So for example, if I register for an event, when that specific event is emitted, my handler will be called:

>>> import botocore.session
>>> s = botocore.session.get_session()
>>> def myhandler(**kwargs):
...   print("Called with kwargs:", kwargs)
...
>>>
>>> s.register('foo.bar', myhandler)
>>>
>>> s.emit('foo.bar', arg1='foo', arg2='bar')
('Called with kwargs:', {'event_name': 'foo.bar', 'arg1': 'foo', 'arg2': 'bar'})
[(<function myhandler at 0x108e55050>, None)]

However, I can also just register for foo, and my handler will be called when foo is emitted, or when foo.bar is emitted, or foo.bar.anything.else:

>>> s.register('foo', myhandler)
>>>
>>> s.emit('foo.bar', arg1='foo', arg2='bar')
('Called with kwargs:', {'event_name': 'foo.bar', 'arg1': 'foo', 'arg2': 'bar'})
('Called with kwargs:', {'event_name': 'foo.bar', 'arg1': 'foo', 'arg2': 'bar'})
[(<function myhandler at 0x108e55050>, None), (<function myhandler at 0x108e55050>, None)]

>>> s.emit('foo.bar.bar.anything', arg1='foo', arg2='bar')
('Called with kwargs:', {'event_name': 'foo.bar.bar.anything', 'arg1': 'foo', 'arg2': 'bar'})
('Called with kwargs:', {'event_name': 'foo.bar.bar.anything', 'arg1': 'foo', 'arg2': 'bar'})
[(<function myhandler at 0x108e55050>, None), (<function myhandler at 0x108e55050>, None)]

>>> s.emit('foo.anything.else', arg1='foo', arg2='bar')
('Called with kwargs:', {'event_name': 'foo.anything.else', 'arg1': 'foo', 'arg2': 'bar'})
[(<function myhandler at 0x108e55050>, None)]

The main reason do this is because event names in the CLI/botocore have names such as before-call.s3.ListObjects, which is emitted before an s3.list_objects() call. So if I want a specific handler just for this operation I can register a handler with before-call.s3.ListObjects. But if I wanted to register a handler that was called before _any_ operation (say for logging/debugging/auditing purposes), I could just register a handler for before-call, and the handler would be called before any API call, regardless of the service operation. I could also register a handler just for any S3 call by registering for before-call.s3.

Hope that helps let me know if you have any other questions.

[jcmcken]

Are there any plans to make the plugin system a public, documented API? It seems like this issue was closed without answering that question.

We use a custom IAM role federation service that ties to our corporate services for authentication/authorization. Currently, users have to develop separate tools to acquire and manage credentials which is somewhat awkward. Implementing a common tool for interacting with that middleware as a plugin within awscli would make the experience much better, but we're concerned that the plugin system isn't stable or supported.

[nitrocode]

@jcmcken check out https://github.com/RichardBronosky/aws-cli-plugins or https://github.com/nitrocode/aws-cli-plugins . If you find a way to add a subcommand without showing the help dialogue, let me know 😄

[ozbillwang]

could we have same features (plugin) for AWS SDK, such as python , nodejs, java, etc?

For example, with python boto3, seems the codes ignore the part of [plugin] in ~/.aws/config, which make the incompatible and inconsistant codes between aws cli and sdk

https://github.com/aws/aws-cli/issues/1270#issuecomment-585489375

Should this feature (plugin) to be moved to botocore, more than put here? So it can be supported in aws cli and python sdk as well.