Aws-sdk-java: com.amazonaws.AmazonClientException: Unable to complete transfer: Task java.util.concurrent.FutureTask@fb8627a rejected from java.util.concurren

Created on 22 Sep 2017  路  11Comments  路  Source: aws/aws-sdk-java

com.amazonaws.AmazonClientException: Unable to complete transfer: Task java.util.concurrent.FutureTask@fb8627a rejected from java.util.concurrent.ThreadPoolExecutor@34a22437[Shutting down, pool size = 6, active threads = 6, queued tasks = 0, completed tasks = 0]
at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:281)
at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265)
at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.waitForCompletion(AbstractTransfer.java:103)
at com.alacriti.reports.etl.service.S3FileDAO.pushToS3(S3FileDAO.java:49)
at com.alacriti.reports.etl.service.EtlFilesManagerService.rotateFiles(EtlFilesManagerService.java:184)
at com.alacriti.reports.etl.service.EtlFilesManagerService.run(EtlFilesManagerService.java:111)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@fb8627a rejected from java.util.concurrent.ThreadPoolExecutor@34a22437[Shutting down, pool size = 6, active threads = 6, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadPartsInParallel(UploadCallable.java:295)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInParts(UploadCallable.java:188)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123)
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139)
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more

We are running this process, which pushes files into S3, for every 5 minutes. This is a sequential task and nowhere, it tries to push 2 files simultaneously.

I wonder , why is this TransferManager is getting shutdown all of a sudden.
sdks used:
aws-java-sdk-s3-1.11.76.jar
aws-java-sdk-kms-1.11.76.jar
aws-java-sdk-core-1.11.75.jar

code

    PutObjectRequest putRequest = new PutObjectRequest(this.bucketName, this.key, file)
                    .withSSEAwsKeyManagementParams(new SSEAwsKeyManagementParams(this.kmsKey));

            TransferManager tm = new TransferManager(s3Client);
            Upload upload = tm.upload(putRequest);
                          upload .waitForCompletion();
guidance response-requested
Source
picture anilkumar763

Most helpful comment

We encountered the same RejectedExecutionException here. But a little different. In our scenario, we create a new TransferManager every time once a file download request was submitted.
We fix our issue with calling
transferManager.shutdownNow(false);
right after download.waitForCompletion()
Since we found out that hundreds of s3-transfer-manager-worker thread are up and waiting for GC in the background.
I write here to see if this could help anyone.

picture pyzhou-talend on 14 Feb 2019
👍2

All 11 comments

Can you point a breakpoint on the shutdown method of TransferManager to see what's closing it?

shorea picture shorea on 22 Sep 2017

Hi @shorea ,
I want to wait for the upload to complete and calling the upload object method,

                      upload .waitForCompletion();

This is the point, where I saw this error.
I didn't get what your question is. I'm not calling the shutdown method of TransferManager.

anilkumar763 picture anilkumar763 on 23 Sep 2017

@anilkumar763 something appears to be shutting down the transfer manager. It might also be getting garbage collected and calling shutdown on finalize. https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/TransferManager.java#L2085

shorea picture shorea on 25 Sep 2017

Do you still have the issue?

varunnvs92 picture varunnvs92 on 5 Oct 2017

Feel free to reopen if you still face the issue.

varunnvs92 picture varunnvs92 on 6 Oct 2017

@anilkumar763 this error can also occur when you exhaust a bounded thread pool (i.e. submit more tasks then the queue backlog allows). How are you configuring the transfer manager? Are you passing in a custom executor?

shorea picture shorea on 30 Oct 2017

Hi @shorea ,
Everytime we want to push file to S3, we are creating AmazonS3 object and passing it to TransferManager. 

this.s3Client = AmazonS3ClientBuilder.standard().withRegion(region).build();
PutObjectRequest putRequest = new PutObjectRequest(this.bucketName, this.key, file)
                    .withSSEAwsKeyManagementParams(new SSEAwsKeyManagementParams(this.kmsKey));

            TransferManager tm = new TransferManager(s3Client);
            Upload upload = tm.upload(putRequest);
                          upload .waitForCompletion();

For every 5 minutes, we push around 2-5 files to S3.

We got this error again today, after 22 Sep, when this question is posted.

anilkumar763 picture anilkumar763 on 16 Jan 2018
👍2

We encountered the same RejectedExecutionException here. But a little different. In our scenario, we create a new TransferManager every time once a file download request was submitted.
We fix our issue with calling
transferManager.shutdownNow(false);
right after download.waitForCompletion()
Since we found out that hundreds of s3-transfer-manager-worker thread are up and waiting for GC in the background.
I write here to see if this could help anyone.

pyzhou-talend picture pyzhou-talend on 14 Feb 2019
👍2

I am trying to upload a file to AWS s3,i am getting the below exception :

Message=AmazonClientException occurred while Uploading to s3 com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286) com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265) com.amazonaws.services.s3.transfer.internal.AbstractTransfer.waitForCompletion(AbstractTransfer.java:103) com.expedia.www.ews.jobs.service.ws.expediaDataPull.utility.ProcessData.uploadToS3(ProcessData.java:363) com.expedia.www.ews.jobs.service.ws.expediaDataPull.utility.ProcessData.writDataToCsvFile(ProcessData.java:414) com.expedia.www.ews.jobs.service.ws.expediaDataPull.utility.ProcessData.processFile(ProcessData.java:174) com.expedia.www.ews.jobs.service.ws.expediaDataPull.service.ExpediaDataFetchServiceImpl.extractDataForExpedia(ExpediaDataFetchServiceImpl.java:44) com.expedia.www.ews.jobs.service.ws.expediaDataPull.controller.DataFetchController.expediaDataFetcher(DataFetchController.java:63) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760) org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) javax.servlet.http.HttpServlet.service(HttpServlet.java:687) org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) javax.servlet.http.HttpServlet.service(HttpServlet.java:790) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:104) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) com.expedia.www.ews.jobs.service.init.InitRequestContextFilter.doFilterInternal(InitRequestContextFilter.java:26) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Thread.java:745)

Please find my code for uploading :

` public boolean uploadToS3(String bucketName, String fileName, String extension,
String s3uploadLocation) {
File fileLocation = new File(LOCAL_FILE_PATH + fileName + extension);
eventLogger.logEvent(EventLevel.INFO, EventLogIds.UPLOADING_FILE_TO_AWS_S3,
"Uploading the File to s3 " + "");
TransferManager manager = null;
boolean uploadStatus = false;
try {
manager = TransferManagerBuilder.standard().withS3Client(getS3Client()).build();
final Upload upload = manager.upload(bucketName, s3uploadLocation + fileName + extension,
fileLocation);
upload.waitForCompletion();
manager.shutdownNow(false);
uploadStatus = true;
eventLogger.logEvent(EventLevel.INFO, EventLogIds.UPLOADING_FILE_TO_AWS_S3,
"Uploading status : " + uploadStatus);
} catch (AmazonServiceException exc) {
eventLogger.logEvent(EventLevel.ERROR, EventLogIds.FILE_UPLOAD_EXCEPTION_FROM_S3,
"AmazonServiceException occurred while Uploading to s3 ", exc.getErrorMessage());
} catch (AmazonClientException exc) {
eventLogger.logEvent(EventLevel.ERROR, EventLogIds.FILE_UPLOAD_EXCEPTION_FROM_S3,
"AmazonClientException occurred while Uploading to s3 ", exc.getStackTrace());
} catch (InterruptedException e) {
eventLogger.logEvent(EventLevel.ERROR, EventLogIds.FILE_UPLOAD_EXCEPTION_FROM_S3,
"InterruptedException occurred while Uploading to s3 ", e.getStackTrace());
} catch (Exception e) {
eventLogger.logEvent(EventLevel.ERROR, EventLogIds.FILE_UPLOAD_EXCEPTION_FROM_S3,
"Exception occurred while Uploading to s3 ", e.getStackTrace());
} finally {
//manager.shutdownNow();
}
return uploadStatus;

}`

Also find my s3 client code :

` public static AmazonS3 getS3Client() {

    return AmazonS3ClientBuilder.standard()
            .withRegion(AWS_REGION) // The first region to try your request against
            .withForceGlobalBucketAccessEnabled(true)
            .withClientConfiguration(new ClientConfiguration().withMaxConnections(100)
                    .withConnectionTimeout(120 * 1000)
                    .withMaxErrorRetry(15))
            .build();
}`

I even tried transferManager.shutdownNow(false);
But still the problem exist,can some one help to resolve the issue?

abinmathew4 picture abinmathew4 on 24 Jun 2019

@abinmathew4
Can you open a new issue with all the info you provided in the last comment? Please also say which version of the SDK you're using.
This original issue has been closed for some time now, and after a quick look it seems like the errors may be different.

@anilkumar763
I see we didn't reply back after the issue was closed, were you able to figure out the problem? If not, just leave a comment here and we will reopen it.
As @pyzhou-talend mentioned, calling transferManager.shutdownNow() will release the resources once the transfer is complete.

debora-ito picture debora-ito on 24 Jun 2019

This issue is due to a fundamental flaw in how TransferManager works.
Typically the flow is a TransferManager is created, then a method that returns something that implements Transfer is returned. eg. TransferManager.downloadDirectory returns a MultipleFileDownload. Calling code then calls downloader.waitForCompletion on the MultipleFileDownload object. At this point its also typical for no other reference to TransferManager exist, allowing it to be garbage collected, but unfortunately TransferManager has a finalizer that destroys the ThreadPool which causes Transfer.waitForCompletion() to blow up
IMPORTANT: The GC only considers references it does not consider scope. Thus even through TransferManager may still be in scope it can be garbage collected, unless there is future code references it.

A workaround is to add a line of logging that references TransferManager AFTER the call to waitForCompletion() call. This will prevent the TransferManager being garbage collected before the transfer has completed.

eg.

upload.waitForCompletion();
Log.info("Logging to keep a reference to TransferManager:" + mgr.hashCode());

AWS could fix this by adding a reference to the TransferManager in all implementations of Transfer, such as MultipleFileDownload etc. This would also prevent the TransferManager being garbage collected, while it's still in use.
I wish AWS would fix this. It's crazy it's behaved like this for so long.

gitkrakenacct picture gitkrakenacct on 11 Dec 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

DaichiUeura picture DaichiUeura  路  3Comments
Yeggstry picture Yeggstry  路  3Comments
martinth picture martinth  路  4Comments
pagil picture pagil  路  4Comments
verkhoro picture verkhoro  路  5Comments