Aws-cli: aws s3 cp command will stop further execution in batch

[AntoniusGolly]

This minimal Windows batch script test.bat

$ echo yes
$ aws s3 cp s3://mybucket/index.html test.html
$ echo no

will download the file successfully but will not echo the "no".

The output is this:

d:\_webdev\_aws>test.bat
d:\_webdev\_aws>echo yes
yes

d:\_webdev\_aws>aws s3 cp s3://mybucket/index.html test.html
download: s3://mybucket/index.html to .\test.html

d:\_webdev\_aws>

The file is downloaded successfully. Any further commands are not executed. Any ideas?

[JordonPhillips]

To be clear, are you talking about Windows batch scripts or AWS Batch? Did the cp command succeed or fail? Could you post some --debug logs?

[AntoniusGolly]

Thanks for clarifying. This is a Windows batch script. What is an aws batch?
The --debug log was unsuspicious, only showing DEBUG lines. I'll post tomorrow.

[AntoniusGolly]

This is the output when running with --debug flag. Note, I replaced a key in the log with _______DELETED IN THIS LOG_____

d:\_webdev\_aws>aws s3 cp s3://mybucket/index.html test.html --debug
2018-02-22 11:42:18,037 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.14.37 Python/3.7.0b1 Windows/10 botocore/1.8.41
2018-02-22 11:42:18,038 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', 's3://mybucket/index.html', 'test.html', '--debug']
2018-02-22 11:42:18,039 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x0000028E3EABE730>
2018-02-22 11:42:18,042 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,042 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x0000028E3E6402F0>
2018-02-22 11:42:18,043 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,043 - MainThread - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2018-02-22 11:42:18,044 - MainThread - botocore.session - DEBUG - Loading variable config_file from defaults.
2018-02-22 11:42:18,044 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,045 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2018-02-22 11:42:18,045 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,046 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2018-02-22 11:42:18,047 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,047 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x0000028E3E861378>
2018-02-22 11:42:18,049 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,051 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x0000028E3EAC59D8>
2018-02-22 11:42:18,052 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,052 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cp: calling handler <function add_waiters at 0x0000028E3EAC59D8>
2018-02-22 11:42:18,055 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,055 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,056 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000028E3E6065F8>
2018-02-22 11:42:18,056 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,057 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000028E3E6065F8>
2018-02-22 11:42:18,058 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,059 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,061 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,062 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000028E3E6065F8>
2018-02-22 11:42:18,062 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,063 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,064 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,064 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,065 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,065 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,066 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,066 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,069 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,069 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,073 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,073 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,074 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,074 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,075 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,075 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,076 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,076 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,076 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000028E3E6065F8>
2018-02-22 11:42:18,077 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,078 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,078 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,082 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000028E3E6065F8>
2018-02-22 11:42:18,083 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.force-glacier-transfer: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,083 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x0000028E3E6065F8>
2018-02-22 11:42:18,084 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,084 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,085 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,085 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x0000028E3E5CA488>
2018-02-22 11:42:18,086 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,087 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,088 - MainThread - botocore.session - DEBUG - Loading variable region from config file with value 'eu-central-1'.
2018-02-22 11:42:18,088 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,089 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-02-22 11:42:18,091 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,093 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-02-22 11:42:18,093 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2018-02-22 11:42:18,094 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2018-02-22 11:42:18,094 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2018-02-22 11:42:18,095 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2018-02-22 11:42:18,096 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Python\Python37\lib\site-packages\botocore\data\endpoints.json
2018-02-22 11:42:18,098 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,098 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x0000028E3DDDC158>
2018-02-22 11:42:18,100 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Python\Python37\lib\site-packages\botocore\data\s3\2006-03-01\service-2.json
2018-02-22 11:42:18,109 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x0000028E3E3001E0>
2018-02-22 11:42:18,109 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x0000028E3E2FAF28>
2018-02-22 11:42:18,110 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2018-02-22 11:42:18,116 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2018-02-22 11:42:18,117 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2018-02-22 11:42:18,117 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2018-02-22 11:42:18,117 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,119 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,119 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,122 - MainThread - botocore.session - DEBUG - Loading variable region from config file with value 'eu-central-1'.
2018-02-22 11:42:18,123 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,123 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-02-22 11:42:18,124 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,124 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-02-22 11:42:18,125 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,125 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x0000028E3DDDC158>
2018-02-22 11:42:18,126 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x0000028E3E3001E0>
2018-02-22 11:42:18,126 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x0000028E3E2FAF28>
2018-02-22 11:42:18,127 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2018-02-22 11:42:18,132 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2018-02-22 11:42:18,132 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2018-02-22 11:42:18,133 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2018-02-22 11:42:18,133 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,134 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,134 - MainThread - botocore.session - DEBUG - Loading variable region from config file with value 'eu-central-1'.
2018-02-22 11:42:18,134 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,135 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-02-22 11:42:18,135 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,136 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-02-22 11:42:18,136 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-02-22 11:42:18,138 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x0000028E3DDDC158>
2018-02-22 11:42:18,140 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x0000028E3E3001E0>
2018-02-22 11:42:18,141 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x0000028E3E2FAF28>
2018-02-22 11:42:18,142 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2018-02-22 11:42:18,144 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2018-02-22 11:42:18,144 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2018-02-22 11:42:18,145 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2018-02-22 11:42:18,146 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2018-02-22 11:42:18,147 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x0000028E3E34A950>
2018-02-22 11:42:18,148 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x0000028E3E34A8C8>
2018-02-22 11:42:18,148 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x0000028E3EE60898>>
2018-02-22 11:42:18,151 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x0000028E3E34A510>
2018-02-22 11:42:18,152 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x0000028E3E34AD90>
2018-02-22 11:42:18,152 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x0000028E3EE60898>>
2018-02-22 11:42:18,152 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=HeadObject) (verify_ssl=True) with params: {'url_path': '/mybucket/index.html', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'aws-cli/1.14.37 Python/3.7.0b1 Windows/10 botocore/1.8.41'}, 'body': b'', 'url': 'https://s3.eu-central-1.amazonaws.com/mybucket/index.html', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x0000028E3EE600B8>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'mybucket'}}}
2018-02-22 11:42:18,153 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000028E3EE55FD0>>
2018-02-22 11:42:18,154 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x0000028E3E34A400>
2018-02-22 11:42:18,154 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x0000028E3DDD6510>
2018-02-22 11:42:18,155 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-02-22 11:42:18,155 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
HEAD
/mybucket/index.html

host:s3.eu-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180222T104218Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2018-02-22 11:42:18,156 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180222T104218Z
20180222/eu-central-1/s3/aws4_request
297b12e22b6718b246dc5320381ff896ce72589766c1d8a5d1c7769e603d78d4
2018-02-22 11:42:18,159 - MainThread - botocore.auth - DEBUG - Signature:
5341b2ed88ccc69c43b8cd88a23ea599f8c5640157c2f4746dd8350d402e5628
2018-02-22 11:42:18,162 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [HEAD]>
2018-02-22 11:42:18,163 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): s3.eu-central-1.amazonaws.com
2018-02-22 11:42:18,592 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "HEAD /mybucket/index.html HTTP/1.1" 200 0
2018-02-22 11:42:18,593 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '_______DELETED IN THIS LOG_____=', 'x-amz-request-id': 'F57837BE754D6A2A', 'date': 'Thu, 22 Feb 2018 10:42:17 GMT', 'last-modified': 'Tue, 13 Feb 2018 12:53:19 GMT', 'etag': '"49b463fced04f96c5bda02c6026df19d"', 'accept-ranges': 'bytes', 'content-type': 'text/html', 'content-length': '74', 'server': 'AmazonS3'}
2018-02-22 11:42:18,594 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2018-02-22 11:42:18,600 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x0000028E3EE602E8>
2018-02-22 11:42:18,600 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2018-02-22 11:42:18,601 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x0000028E3EE60898>>
2018-02-22 11:42:18,601 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x0000028E3EABE268>
2018-02-22 11:42:18,605 - MainThread - s3transfer.utils - DEBUG - Acquiring 0
2018-02-22 11:42:18,608 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x0000028E3EED9DA0>}) about to wait for the following futures []
2018-02-22 11:42:18,608 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x0000028E3EED9DA0>}) done waiting for dependent futures
2018-02-22 11:42:18,609 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x0000028E3EED9DA0>}) with kwargs {'client': <botocore.client.S3 object at 0x0000028E3EE092E8>, 'config': <s3transfer.manager.TransferConfig object at 0x0000028E3EE60B00>, 'osutil': <s3transfer.utils.OSUtils object at 0x0000028E3EE60B70>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x0000028E3EE60CF8>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x0000028E3EED9DA0>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x0000028E3EE60F28>}
2018-02-22 11:42:18,610 - ThreadPoolExecutor-1_0 - s3transfer.futures - DEBUG - Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'mybucket', 'key': 'index.html', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x0000028E3EE60CF8> for transfer request: 0.
2018-02-22 11:42:18,610 - ThreadPoolExecutor-1_0 - s3transfer.utils - DEBUG - Acquiring 0
2018-02-22 11:42:18,611 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'mybucket', 'key': 'index.html', 'extra_args': {}}) about to wait for the following futures []
2018-02-22 11:42:18,611 - ThreadPoolExecutor-1_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-02-22 11:42:18,612 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'mybucket', 'key': 'index.html', 'extra_args': {}}) done waiting for dependent futures
2018-02-22 11:42:18,614 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'mybucket', 'key': 'index.html', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x0000028E3EE092E8>, 'bucket': 'mybucket', 'key': 'index.html', 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x0000028E3EED9160>, 'extra_args': {}, 'callbacks': [functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.utils.ProvideSizeSubscriber object at 0x0000028E3EED90F0>>, future=<s3transfer.futures.TransferFuture object at 0x0000028E3EED9DA0>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.utils.DirectoryCreatorSubscriber object at 0x0000028E3EED90B8>>, future=<s3transfer.futures.TransferFuture object at 0x0000028E3EED9DA0>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.utils.ProvideLastModifiedTimeSubscriber object at 0x0000028E3EED9B70>>, future=<s3transfer.futures.TransferFuture object at 0x0000028E3EED9DA0>), functools.partial(<bound method BaseResultSubscriber.on_progress of <awscli.customizations.s3.results.DownloadResultSubscriber object at 0x0000028E3EED9BA8>>, future=<s3transfer.futures.TransferFuture object at 0x0000028E3EED9DA0>)], 'max_attempts': 5, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x0000028E3D3EE940>, 'io_chunksize': 262144, 'bandwidth_limiter': None}
2018-02-22 11:42:18,617 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x0000028E3E34A950>
2018-02-22 11:42:18,618 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x0000028E3E34A8C8>
2018-02-22 11:42:18,618 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x0000028E3EE09C18>>
2018-02-22 11:42:18,619 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x0000028E3E34A510>
2018-02-22 11:42:18,620 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x0000028E3E34AD90>
2018-02-22 11:42:18,620 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x0000028E3EE09C18>>
2018-02-22 11:42:18,621 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetObject) (verify_ssl=True) with params: {'url_path': '/mybucket/index.html', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/1.14.37 Python/3.7.0b1 Windows/10 botocore/1.8.41'}, 'body': b'', 'url': 'https://s3.eu-central-1.amazonaws.com/mybucket/index.html', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x0000028E3EE094A8>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'mybucket'}}}
2018-02-22 11:42:18,622 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x0000028E3E6F0D08>
2018-02-22 11:42:18,622 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000028E3EE09400>>
2018-02-22 11:42:18,623 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x0000028E3E34A400>
2018-02-22 11:42:18,626 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-sign.s3.GetObject: calling handler <function fix_s3_host at 0x0000028E3DDD6510>
2018-02-22 11:42:18,627 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-02-22 11:42:18,628 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest:
GET
/mybucket/index.html

host:s3.eu-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180222T104218Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2018-02-22 11:42:18,629 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180222T104218Z
20180222/eu-central-1/s3/aws4_request
9084ffdf3c8d63f39e84e00500b9d282390ed57e51f9d16ad16ead5c722954d5
2018-02-22 11:42:18,630 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Signature:
af9b0cc6864ff2d56b2c03e1e5164d85630b6bfd1ec595ed3ed1deb70a258ce1
2018-02-22 11:42:18,632 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x0000028E3E707048>
2018-02-22 11:42:18,633 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [GET]>
2018-02-22 11:42:18,651 - ThreadPoolExecutor-0_0 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): s3.eu-central-1.amazonaws.com
2018-02-22 11:42:18,887 - ThreadPoolExecutor-0_0 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /mybucket/index.html HTTP/1.1" 200 74
2018-02-22 11:42:18,887 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'zC96JqlK2FVWFvF87T3VI09wG2pTL3Gz0OKIryTymnSFHxxrsYK1nuoQY7h0J3dy4TA+6Xmv1hk=', 'x-amz-request-id': 'D1B298FA70C64E8A', 'date': 'Thu, 22 Feb 2018 10:42:17 GMT', 'last-modified': 'Tue, 13 Feb 2018 12:53:19 GMT', 'etag': '"49b463fced04f96c5bda02c6026df19d"', 'accept-ranges': 'bytes', 'content-type': 'text/html', 'content-length': '74', 'server': 'AmazonS3'}
2018-02-22 11:42:18,888 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response body:
<botocore.response.StreamingBody object at 0x0000028E3DCD1780>
2018-02-22 11:42:18,891 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x0000028E3EE09748>
2018-02-22 11:42:18,892 - ThreadPoolExecutor-0_0 - botocore.retryhandler - DEBUG - No retry needed.
2018-02-22 11:42:18,892 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x0000028E3EE09C18>>
2018-02-22 11:42:18,893 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <function enhance_error_msg at 0x0000028E3EABE268>
2018-02-22 11:42:18,893 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
2018-02-22 11:42:18,894 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
2018-02-22 11:42:18,894 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x0000028E3EED9160>, 'offset': 0}
2018-02-22 11:42:18,896 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': 'd:\\_webdev\\_aws\\test.html'}) about to wait for the following futures []
2018-02-22 11:42:18,896 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': 'd:\\_webdev\\_aws\\test.html'}) done waiting for dependent futures
2018-02-22 11:42:18,896 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IORenameFileTask(transfer_id=0, {'final_filename': 'd:\\_webdev\\_aws\\test.html'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x0000028E3EED9160>, 'final_filename': 'd:\\_webdev\\_aws\\test.html', 'osutil': <s3transfer.utils.OSUtils object at 0x0000028E3EE60B70>}
2018-02-22 11:42:18,901 - ThreadPoolExecutor-0_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download: s3://mybucket/index.html to .\test.html
2018-02-22 11:42:18,902 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

d:\_webdev\_aws>

[lushenwu]

I had the same issue and solved it by adding "& ^" after each line, e.g.
aws s3 cp source1 target2 & ^
aws s3 cp source2 target2 & ^
aws s3 cp source2 target3

per one of the solutions on this page:
https://stackoverflow.com/questions/4036754/why-does-only-the-first-line-of-this-windows-batch-file-execute-but-all-three-li

[justnance]

@AntoniusGolly - Thanks for reaching out. @lushenwu - Thanks for posting a response.

AWS Batch is a managed service that allows you to easily and efficiently run hundreds of thousands of batch jobs on AWS. There is no additional charge for AWS Batch. You only pay for the AWS Resources (e.g. EC2 Instances) created to store and run the batch jobs. Here's some online documentation to dig deeper:

Getting Started with AWS Batch - AWS Batch
AWS Batch FAQs

It appears the aws s3 cp command is working as expected and the behavior is isolated to Windows. It also appears @lushenwu's response provides a good solution

Please reply if this is still an issue.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[C0lumbo]

I found that using call aws instead of aws in .bat files was a decent workaround.

[luis02lopez]

Exactly @C0lumbo ! That's was the error for me. The aws s3 command in a .bat file was stopping after the first command.

[ukwatte]

Call worked for me as well, thanks!