Aws-cli: Docker - local variable 'process' referenced before assignment

[josephwibowo]

I am running a local docker container (which is built from python3.7) that installs the v2 AWS CLI using CURL, unzip, .aws/install as indicated by the guide. I am also mounting my IAM role credentials as a volume. aws2 configure seems to locate it correctly.

I am able to get output from aws2 --version, but when I try to run any command such as aws2 iam get-user, it returns the following:

local variable 'process' referenced before assignment

There is a new line if that helps in debugging. Cannot figure out why this would ever happen...

[jamesls]

Can you post the --debug output? The only place I'm seeing a process var is in a codedeploy customization so I'm not sure where that's coming from without see the full stack trace.

[josephwibowo]

oops, didn't realize that was an option...
here you go...

2019-12-12 20:36:31,302 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.0.0dev2 Python/3.7.3 Linux/4.9.184-linuxkit botocore/2.0.0dev1
2019-12-12 20:36:31,303 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['iam', 'get-user', '--debug']
2019-12-12 20:36:31,303 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f4e86d5e730>
2019-12-12 20:36:31,304 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f4e876f3950>
2019-12-12 20:36:31,305 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f4e87648ae8>
2019-12-12 20:36:31,307 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f4e86ecda60>
2019-12-12 20:36:31,307 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7f4e86f08488>
2019-12-12 20:36:31,318 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.0dev2/dist/botocore/data/iam/2010-05-08/service-2.json
2019-12-12 20:36:31,339 - MainThread - botocore.hooks - DEBUG - Event building-command-table.iam: calling handler <function _add_wizard_command at 0x7f4e86d10598>
2019-12-12 20:36:31,340 - MainThread - botocore.hooks - DEBUG - Event building-command-table.iam: calling handler <function add_waiters at 0x7f4e86d64c80>
2019-12-12 20:36:31,352 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.0dev2/dist/botocore/data/iam/2010-05-08/waiters-2.json
2019-12-12 20:36:31,353 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('user-name', <awscli.arguments.CLIArgument object at 0x7f4e86bdd198>)])
2019-12-12 20:36:31,354 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.get-user: calling handler <function add_streaming_output_arg at 0x7f4e86d5ec80>
2019-12-12 20:36:31,354 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.get-user: calling handler <function add_cli_input_json at 0x7f4e87650158>
2019-12-12 20:36:31,355 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.get-user: calling handler <function add_cli_input_yaml at 0x7f4e87650378>
2019-12-12 20:36:31,355 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.get-user: calling handler <function unify_paging_params at 0x7f4e86f0ac80>
2019-12-12 20:36:31,363 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.0dev2/dist/botocore/data/iam/2010-05-08/paginators-1.json
2019-12-12 20:36:31,364 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.get-user: calling handler <function add_generate_skeleton at 0x7f4e86dd1378>
2019-12-12 20:36:31,365 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.iam.get-user: calling handler <function add_auto_prompt at 0x7f4e86d10e18>
2019-12-12 20:36:31,365 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.iam.get-user: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7f4e86be2ba8>>
2019-12-12 20:36:31,366 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.iam.get-user: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7f4e86be2c50>>
2019-12-12 20:36:31,367 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.iam.get-user: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f4e86be2c88>>
2019-12-12 20:36:31,367 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.iam.get-user: calling handler <bound method AutoPromptArgument.override_required_args of <awscli.customizations.autoprompt.AutoPromptArgument object at 0x7f4e86b71ba8>>
2019-12-12 20:36:31,369 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.get-user.user-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4e86c9e9e8>
2019-12-12 20:36:31,369 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.get-user.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4e86c9e9e8>
2019-12-12 20:36:31,370 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.get-user.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4e86c9e9e8>
2019-12-12 20:36:31,370 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.get-user.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4e86c9e9e8>
2019-12-12 20:36:31,370 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.iam.get-user.cli-auto-prompt: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4e86c9e9e8>
2019-12-12 20:36:31,371 - MainThread - botocore.hooks - DEBUG - Event calling-command.iam.get-user: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7f4e86be2ba8>>
2019-12-12 20:36:31,371 - MainThread - botocore.hooks - DEBUG - Event calling-command.iam.get-user: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7f4e86be2c50>>
2019-12-12 20:36:31,371 - MainThread - botocore.hooks - DEBUG - Event calling-command.iam.get-user: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f4e86be2c88>>
2019-12-12 20:36:31,372 - MainThread - botocore.hooks - DEBUG - Event calling-command.iam.get-user: calling handler <bound method AutoPromptArgument.auto_prompt_arguments of <awscli.customizations.autoprompt.AutoPromptArgument object at 0x7f4e86b71ba8>>
2019-12-12 20:36:31,372 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2019-12-12 20:36:31,372 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2019-12-12 20:36:31,373 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2019-12-12 20:36:31,373 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2019-12-12 20:36:31,373 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2019-12-12 20:36:31,380 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2019-12-12 20:36:31,396 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.0dev2/dist/botocore/data/endpoints.json
2019-12-12 20:36:31,400 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f4e886b4620>
2019-12-12 20:36:31,406 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.iam: calling handler <function add_generate_presigned_url at 0x7f4e8869ce18>
2019-12-12 20:36:31,407 - MainThread - botocore.regions - DEBUG - Using partition endpoint for iam, us-west-2: aws-global
2019-12-12 20:36:31,407 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2019-12-12 20:36:31,411 - MainThread - botocore.endpoint - DEBUG - Setting iam timeout as (60, 60)
2019-12-12 20:36:31,415 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.0.0dev2/dist/botocore/data/_retry.json
2019-12-12 20:36:31,415 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: iam
2019-12-12 20:36:31,417 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.iam.GetUser: calling handler <function generate_idempotent_uuid at 0x7f4e88660510>
2019-12-12 20:36:31,418 - MainThread - botocore.hooks - DEBUG - Event before-call.iam.GetUser: calling handler <function inject_api_version_header_if_needed at 0x7f4e8865fea0>
2019-12-12 20:36:31,418 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetUser) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.0.0dev2 Python/3.7.3 Linux/4.9.184-linuxkit botocore/2.0.0dev1'}, 'body': {'Action': 'GetUser', 'Version': '2010-05-08'}, 'url': 'https://iam.amazonaws.com/', 'context': {'client_region': 'aws-global', 'client_config': <botocore.config.Config object at 0x7f4e86888278>, 'has_streaming_input': False, 'auth_type': None}}
2019-12-12 20:36:31,419 - MainThread - botocore.hooks - DEBUG - Event request-created.iam.GetUser: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f4e86888240>>
2019-12-12 20:36:31,420 - MainThread - botocore.hooks - DEBUG - Event choose-signer.iam.GetUser: calling handler <function set_operation_specific_signer at 0x7f4e88660400>
2019-12-12 20:36:31,424 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2019-12-12 20:36:31,424 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:iam.amazonaws.com
x-amz-date:20191212T203631Z

content-type;host;x-amz-date
63cdc6da6b76dfdbbd82fe53059afd80e08926740ab5436bd2d4196da1af20db
2019-12-12 20:36:31,425 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20191212T203631Z
20191212/us-east-1/iam/aws4_request
cd6f1fd2bad60a7cf6925d71c3ea555ad74e0531e6145904a257c900ee538cfe
2019-12-12 20:36:31,426 - MainThread - botocore.auth - DEBUG - Signature:
a4b56929a9f38942cf7cda132ca495d715042a32a9f1a63550f4959411af43b0
2019-12-12 20:36:31,427 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://iam.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.0.0dev2 Python/3.7.3 Linux/4.9.184-linuxkit botocore/2.0.0dev1', 'X-Amz-Date': b'20191212T203631Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAXSSGVFIIRWWVOUNG/20191212/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=a4b56929a9f38942cf7cda132ca495d715042a32a9f1a63550f4959411af43b0', 'Content-Length': '33'}>
2019-12-12 20:36:31,429 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): iam.amazonaws.com:443
2019-12-12 20:36:31,966 - MainThread - urllib3.connectionpool - DEBUG - https://iam.amazonaws.com:443 "POST / HTTP/1.1" 200 539
2019-12-12 20:36:31,968 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '2f1d67da-05ce-4ee4-be85-7efd04569052', 'Content-Type': 'text/xml', 'Content-Length': '539', 'Date': 'Thu, 12 Dec 2019 20:36:31 GMT'}
2019-12-12 20:36:31,969 - MainThread - botocore.parsers - DEBUG - Response body:
b'<GetUserResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">\n  <GetUserResult>\n    <User>\n      <Path>/</Path>\n      <PasswordLastUsed>2019-12-12T17:43:52Z</PasswordLastUsed>\n      <UserName>joseph.wibowo</UserName>\n      <Arn>arn:aws:iam::520912972305:user/joseph.wibowo</Arn>\n      <UserId>AIDAXSSGVFII7D2D5XSNS</UserId>\n      <CreateDate>2019-04-16T16:32:24Z</CreateDate>\n    </User>\n  </GetUserResult>\n  <ResponseMetadata>\n    <RequestId>2f1d67da-05ce-4ee4-be85-7efd04569052</RequestId>\n  </ResponseMetadata>\n</GetUserResponse>\n'
2019-12-12 20:36:31,979 - MainThread - botocore.hooks - DEBUG - Event needs-retry.iam.GetUser: calling handler <botocore.retryhandler.RetryHandler object at 0x7f4e86888550>
2019-12-12 20:36:31,979 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2019-12-12 20:36:31,979 - MainThread - botocore.hooks - DEBUG - Event after-call.iam.GetUser: calling handler <function json_decode_policies at 0x7f4e8865f268>
2019-12-12 20:36:31,983 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "site-packages/awscli/clidriver.py", line 313, in main
  File "site-packages/awscli/clidriver.py", line 464, in __call__
  File "site-packages/awscli/clidriver.py", line 642, in __call__
  File "site-packages/awscli/clidriver.py", line 764, in invoke
  File "site-packages/awscli/clidriver.py", line 785, in _display_response
  File "contextlib.py", line 112, in __enter__
  File "site-packages/awscli/utils.py", line 290, in get_pager_stream
UnboundLocalError: local variable 'process' referenced before assignment
2019-12-12 20:36:31,984 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

local variable 'process' referenced before assignment

[jamesls]

Oh right! I was checking the CLI v1 branch not CLI v2. Yeah this is definitely a bug we need to fix, labeling as a bug (cc @kyleknap):

    @contextlib.contextmanager
    def get_pager_stream(self, preferred_pager=None):
        if not preferred_pager:
            preferred_pager = self._get_configured_pager()
        popen_kwargs = self._get_process_pager_kwargs(preferred_pager)
        try:
            process = self._popen(**popen_kwargs)
            yield process.stdin
        except IOError:
            # Ignore IOError since this can commonly be raised when a pager
            # is closed abruptly and causes a broken pipe.
            pass
        finally:
            process.communicate()   # <--- this line is causing the error

[ming-apt-mel]

hi just wondering whats the eta on this? i am facing the same issue right now, probably revert to v1 for interim ?

[kyleknap]

I sent a PR to fix this issue: https://github.com/aws/aws-cli/pull/4810. Once merged, it will be available in the next release we cut of v2.

[kyleknap]

We just cut a release of v2, with this issue fixed. You should no longer be seeing the reference before assignment error and be able to actually see what is causing the pager to fail to open. The version you will need is 2.0.0.dev3:

$ aws2 --version
aws-cli/2.0.0dev3 Python/3.7.4 Darwin/17.7.0 botocore/2.0.0dev2