Cloud-custodian: azure - c7n-org report error with azure

[cmedley]

Using c7n-org, the run command works with azure.

However when using report command it only works with aws. Looking at command line help and code, it looks like report command is only for aws.

$ c7n-org run -c ./policies/8cf56bc1-3a08-4aef-9462-42f4c6351bed-config.yaml -u ./policies/8cf56bc1-3a08-4aef-9462-42f4c6351bed.yaml -s ./results/8cf56bc1-3a08-4aef-9462-42f4c6351bed/

2019-06-18 14:42:26,966: c7n_org:INFO Ran account:mysub region:global policy:vms-with-public-ip2 matched:1 time:1.96
2019-06-18 14:42:26,998: c7n_org:INFO Policy resource counts Counter({'vms-with-public-ip2': 1})

$ c7n-org report -c ./policies/8cf56bc1-3a08-4aef-9462-42f4c6351bed-config.yaml -u ./policies/8cf56bc1-3a08-4aef-9462-42f4c6351bed.yaml -s ./results/8cf56bc1-3a08-4aef-9462-42f4c6351bed/
Traceback (most recent call last):
  File "/usr/local/bin/c7n-org", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/c7n_org/cli.py", line 362, in report
    factory.resource_type,
AttributeError: 'NoneType' object has no attribute 'resource_type'

Looking at the resource_registry around line 362, it has the following (does not have azure resources)

{  
   'plugin_type':'resources',
   '_factories':{  
      'iam-group':<class 'c7n.resources.iam.Group'>,
      'iam-role':<class 'c7n.resources.iam.Role'>,
      'iam-user':<class 'c7n.resources.iam.User'>,
      'iam-policy':<class 'c7n.resources.iam.Policy'>,
      'iam-profile':<class 'c7n.resources.iam.InstanceProfile'>,
      'iam-certificate':<class 'c7n.resources.iam.ServerCertificate'>,
      'account':<class 'c7n.resources.account.Account'>,
      'acm-certificate':<class 'c7n.resources.acm.Certificate'>,
      'ami':<class 'c7n.resources.ami.AMI'>,
      'rest-account':<class 'c7n.resources.apigw.RestAccount'>,
      'rest-api':<class 'c7n.resources.apigw.RestApi'>,
      'rest-stage':<class 'c7n.resources.apigw.RestStage'>,
      'rest-resource':<class 'c7n.resources.apigw.RestResource'>,
      'rest-vpclink':<class 'c7n.resources.apigw.RestApiVpcLink'>,
      'shield-protection':<class 'c7n.resources.shield.ShieldProtection'>,
      'shield-attack':<class 'c7n.resources.shield.ShieldAttack'>,
      'app-elb':<class 'c7n.resources.appelb.AppELB'>,
      'app-elb-target-group':<class 'c7n.resources.appelb.AppELBTargetGroup'>,
      'ec2':<class 'c7n.resources.ec2.EC2'>,
      'launch-template-version':<class 'c7n.resources.ec2.LaunchTemplate'>,
      'ec2-reserved':<class 'c7n.resources.ec2.ReservedInstance'>,
      'asg':<class 'c7n.resources.asg.ASG'>,
      'launch-config':<class 'c7n.resources.asg.LaunchConfig'>,
      'lambda':<class 'c7n.resources.awslambda.AWSLambda'>,
      'lambda-layer':<class 'c7n.resources.awslambda.LambdaLayerVersion'>,
      'backup-plan':<class 'c7n.resources.backup.BackupPlan'>,
      'batch-compute':<class 'c7n.resources.batch.ComputeEnvironment'>,
      'batch-definition':<class 'c7n.resources.batch.JobDefinition'>,
      'cfn':<class 'c7n.resources.cfn.CloudFormation'>,
      'distribution':<class 'c7n.resources.cloudfront.Distribution'>,
      'streaming-distribution':<class 'c7n.resources.cloudfront.StreamingDistribution'>,
      'cloudsearch':<class 'c7n.resources.cloudsearch.CloudSearch'>,
      'cloudtrail':<class 'c7n.resources.cloudtrail.CloudTrail'>,
      'codecommit':<class 'c7n.resources.code.CodeRepository'>,
      'codebuild':<class 'c7n.resources.code.CodeBuildProject'>,
      'codepipeline':<class 'c7n.resources.code.CodeDeployPipeline'>,
      'identity-pool':<class 'c7n.resources.cognito.CognitoIdentityPool'>,
      'user-pool':<class 'c7n.resources.cognito.CognitoUserPool'>,
      'config-rule':<class 'c7n.resources.config.ConfigRule'>,
      'alarm':<class 'c7n.resources.cw.Alarm'>,
      'event-rule':<class 'c7n.resources.cw.EventRule'>,
      'event-rule-target':<class 'c7n.resources.cw.EventRuleTarget'>,
      'log-group':<class 'c7n.resources.cw.LogGroup'>,
      'directory':<class 'c7n.resources.directory.Directory'>,
      'cloud-directory':<class 'c7n.resources.directory.CloudDirectory'>,
      'directconnect':<class 'c7n.resources.directconnect.DirectConnect'>,
      'dlm-policy':<class 'c7n.resources.dlm.DLMPolicy'>,
      'dms-instance':<class 'c7n.resources.dms.ReplicationInstance'>,
      'dms-endpoint':<class 'c7n.resources.dms.DmsEndpoints'>,
      'dynamodb-table':<class 'c7n.resources.dynamodb.Table'>,
      'dynamodb-backup':<class 'c7n.resources.dynamodb.Backup'>,
      'dynamodb-stream':<class 'c7n.resources.dynamodb.Stream'>,
      'dax':<class 'c7n.resources.dynamodb.DynamoDbAccelerator'>,
      'datapipeline':<class 'c7n.resources.datapipeline.DataPipeline'>,
      'kms':<class 'c7n.resources.kms.KeyAlias'>,
      'kms-key':<class 'c7n.resources.kms.Key'>,
      'ebs-snapshot':<class 'c7n.resources.ebs.Snapshot'>,
      'ebs':<class 'c7n.resources.ebs.EBS'>,
      'ecr':<class 'c7n.resources.ecr.ECR'>,
      'ecs':<class 'c7n.resources.ecs.ECSCluster'>,
      'ecs-service':<class 'c7n.resources.ecs.Service'>,
      'ecs-task':<class 'c7n.resources.ecs.Task'>,
      'ecs-task-definition':<class 'c7n.resources.ecs.TaskDefinition'>,
      'ecs-container-instance':<class 'c7n.resources.ecs.ContainerInstance'>,
      'efs':<class 'c7n.resources.efs.ElasticFileSystem'>,
      'efs-mount-target':<class 'c7n.resources.efs.ElasticFileSystemMountTarget'>,
      'cache-cluster':<class 'c7n.resources.elasticache.ElastiCacheCluster'>,
      'cache-subnet-group':<class 'c7n.resources.elasticache.ElastiCacheSubnetGroup'>,
      'cache-snapshot':<class 'c7n.resources.elasticache.ElastiCacheSnapshot'>,
      'elasticbeanstalk':<class 'c7n.resources.elasticbeanstalk.ElasticBeanstalk'>,
      'elasticbeanstalk-environment':<class 'c7n.resources.elasticbeanstalk.ElasticBeanstalkEnvironment'>,
      'elasticsearch':<class 'c7n.resources.elasticsearch.ElasticSearchDomain'>,
      'elb':<class 'c7n.resources.elb.ELB'>,
      'eks':<class 'c7n.resources.eks.EKS'>,
      'emr':<class 'c7n.resources.emr.EMRCluster'>,
      'gamelift-build':<class 'c7n.resources.gamelift.GameLiftBuild'>,
      'gamelift-fleet':<class 'c7n.resources.gamelift.GameLiftFleet'>,
      'glacier':<class 'c7n.resources.glacier.Glacier'>,
      'glue-connection':<class 'c7n.resources.glue.GlueConnection'>,
      'glue-dev-endpoint':<class 'c7n.resources.glue.GlueDevEndpoint'>,
      'health-event':<class 'c7n.resources.health.HealthEvents'>,
      'cloudhsm-cluster':<class 'c7n.resources.hsm.CloudHSMCluster'>,
      'hsm':<class 'c7n.resources.hsm.CloudHSM'>,
      'hsm-hapg':<class 'c7n.resources.hsm.PartitionGroup'>,
      'hsm-client':<class 'c7n.resources.hsm.HSMClient'>,
      'iot':<class 'c7n.resources.iot.IoT'>,
      'kafka':<class 'c7n.resources.kafka.Kafka'>,
      'kinesis':<class 'c7n.resources.kinesis.KinesisStream'>,
      'firehose':<class 'c7n.resources.kinesis.DeliveryStream'>,
      'kinesis-analytics':<class 'c7n.resources.kinesis.AnalyticsApp'>,
      'lightsail-instance':<class 'c7n.resources.lightsail.Instance'>,
      'lightsail-db':<class 'c7n.resources.lightsail.Database'>,
      'lightsail-elb':<class 'c7n.resources.lightsail.LoadBalancer'>,
      'ml-model':<class 'c7n.resources.ml.MLModel'>,
      'message-broker':<class 'c7n.resources.mq.MessageBroker'>,
      'opswork-stack':<class 'c7n.resources.opsworks.OpsworkStack'>,
      'opswork-cm':<class 'c7n.resources.opsworks.OpsworksCM'>,
      'rds':<class 'c7n.resources.rds.RDS'>,
      'rds-subscription':<class 'c7n.resources.rds.RDSSubscription'>,
      'rds-snapshot':<class 'c7n.resources.rds.RDSSnapshot'>,
      'rds-subnet-group':<class 'c7n.resources.rds.RDSSubnetGroup'>,
      'rds-param-group':<class 'c7n.resources.rdsparamgroup.RDSParamGroup'>,
      'rds-cluster-param-group':<class 'c7n.resources.rdsparamgroup.RDSClusterParamGroup'>,
      'rds-cluster':<class 'c7n.resources.rdscluster.RDSCluster'>,
      'rds-cluster-snapshot':<class 'c7n.resources.rdscluster.RDSClusterSnapshot'>,
      'redshift':<class 'c7n.resources.redshift.Redshift'>,
      'redshift-subnet-group':<class 'c7n.resources.redshift.RedshiftSubnetGroup'>,
      'redshift-snapshot':<class 'c7n.resources.redshift.RedshiftSnapshot'>,
      'hostedzone':<class 'c7n.resources.route53.HostedZone'>,
      'healthcheck':<class 'c7n.resources.route53.HealthCheck'>,
      'rrset':<class 'c7n.resources.route53.ResourceRecordSet'>,
      'r53domain':<class 'c7n.resources.route53.Route53Domain'>,
      's3':<class 'c7n.resources.s3.S3'>,
      'sagemaker-notebook':<class 'c7n.resources.sagemaker.NotebookInstance'>,
      'sagemaker-job':<class 'c7n.resources.sagemaker.SagemakerJob'>,
      'sagemaker-transform-job':<class 'c7n.resources.sagemaker.SagemakerTransformJob'>,
      'sagemaker-endpoint':<class 'c7n.resources.sagemaker.SagemakerEndpoint'>,
      'sagemaker-endpoint-config':<class 'c7n.resources.sagemaker.SagemakerEndpointConfig'>,
      'sagemaker-model':<class 'c7n.resources.sagemaker.Model'>,
      'secrets-manager':<class 'c7n.resources.secretsmanager.SecretsManager'>,
      'step-machine':<class 'c7n.resources.sfn.StepFunction'>,
      'simpledb':<class 'c7n.resources.simpledb.SimpleDB'>,
      'snowball-cluster':<class 'c7n.resources.snowball.SnowballCluster'>,
      'snowball':<class 'c7n.resources.snowball.Snowball'>,
      'sns':<class 'c7n.resources.sns.SNS'>,
      'storage-gateway':<class 'c7n.resources.storagegw.StorageGateway'>,
      'sqs':<class 'c7n.resources.sqs.SQS'>,
      'ssm-parameter':<class 'c7n.resources.ssm.SSMParameter'>,
      'ssm-managed-instance':<class 'c7n.resources.ssm.ManagedInstance'>,
      'ssm-activation':<class 'c7n.resources.ssm.SSMActivation'>,
      'support-case':<class 'c7n.resources.support.SupportCase'>,
      'vpc':<class 'c7n.resources.vpc.Vpc'>,
      'subnet':<class 'c7n.resources.vpc.Subnet'>,
      'security-group':<class 'c7n.resources.vpc.SecurityGroup'>,
      'eni':<class 'c7n.resources.vpc.NetworkInterface'>,
      'route-table':<class 'c7n.resources.vpc.RouteTable'>,
      'transit-gateway':<class 'c7n.resources.vpc.TransitGateway'>,
      'transit-attachment':<class 'c7n.resources.vpc.TransitGatewayAttachment'>,
      'peering-connection':<class 'c7n.resources.vpc.PeeringConnection'>,
      'network-acl':<class 'c7n.resources.vpc.NetworkAcl'>,
      'network-addr':<class 'c7n.resources.vpc.NetworkAddress'>,
      'customer-gateway':<class 'c7n.resources.vpc.CustomerGateway'>,
      'internet-gateway':<class 'c7n.resources.vpc.InternetGateway'>,
      'nat-gateway':<class 'c7n.resources.vpc.NATGateway'>,
      'vpn-connection':<class 'c7n.resources.vpc.VPNConnection'>,
      'vpn-gateway':<class 'c7n.resources.vpc.VPNGateway'>,
      'vpc-endpoint':<class 'c7n.resources.vpc.VpcEndpoint'>,
      'key-pair':<class 'c7n.resources.vpc.KeyPair'>,
      'waf':<class 'c7n.resources.waf.WAF'>,
      'waf-regional':<class 'c7n.resources.waf.RegionalWAF'>,
      'fsx':<class 'c7n.resources.fsx.FSx'>,
      'fsx-backup':<class 'c7n.resources.fsx.FSxBackup'>,
      'workspaces':<class 'c7n.resources.workspaces.Workspace'>
   },
   '_subscribers':{  
      0:[  
         <bound method ConfigCompliance.register_resources of <class 'c7n.filters.config.ConfigCompliance'>>,
         <bound method HealthEventFilter.register_resources of <class 'c7n.filters.health.HealthEventFilter'>>,
         <bound method SecurityHubFindingFilter.register_resources of <class 'c7n.filters.securityhub.SecurityHubFindingFilter'>>,
         <bound method CopyRelatedResourceTag.register_resources of <class 'c7n.tags.CopyRelatedResourceTag'>>,
         <bound method JsonDiff.register_resources of <class 'c7n.filters.revisions.JsonDiff'>>
      ],
      1:[  
         <function register_action_tag_user at 0x7f922eb661e0>,
         <function register_action_invoke_lambda at 0x7f922eb666a8>,
         <function register_action_put_metric at 0x7f922eb668c8>,
         <bound method OtherResourcePostFinding.register_resource of <class 'c7n.actions.securityhub.OtherResourcePostFinding'>>
      ]
   }
}

Should it work and missing something?

[logachev]

I will take a look closer later today, but I think you're right.
c7n-org might support only run at the moment for Azure.

[kapilt]

there's a separate resource registry for each provider. we'll need to wire cross-provider report support into c7n-org.

[cmedley]

I was able to get it working by adding the following

$ git diff tools/c7n_org/c7n_org/cli.py
diff --git a/tools/c7n_org/c7n_org/cli.py b/tools/c7n_org/c7n_org/cli.py
index 07b154c80..5a8a88693 100644
--- a/tools/c7n_org/c7n_org/cli.py
+++ b/tools/c7n_org/c7n_org/cli.py
@@ -40,7 +40,6 @@ from c7n.config import Config
 from c7n.policy import PolicyCollection
 from c7n.reports.csvout import Formatter, fs_record_set
 from c7n.resources import load_resources
-from c7n.manager import resources as resource_registry
 from c7n.utils import CONN_CACHE, dumps

 from c7n_org.utils import environ, account_tags
@@ -375,7 +374,16 @@ def report(config, output, use, output_dir, accounts,
     prefix_fields = OrderedDict(
         (('Account', 'account'), ('Region', 'region'), ('Policy', 'policy')))
     config = Config.empty()
-    factory = resource_registry.get(list(resource_types)[0])
+
+    cloud, resource = list(resource_types)[0].split(".")
+    if cloud == "azure":
+        from c7n_azure.provider import resources as resource_registry
+    elif cloud == "gcp":
+        from c7n_gcp.provider import resources as resource_registry
+    else:
+        from c7n.manager import resources as resource_registry
+
+    factory = resource_registry.get(resource)

probably not the right way to solve it though

[kapilt]

I'll have a pr up shortly