Payara: Default queue size of PayaraExecutorService too small depending on the environment

[svendiedrichsen]

# Description #

There are some tickets that have error descriptions or comments that suggest that the default queue size of the PayaraExecutorService is _sometimes_ too small.

Issue: #3360 Comment: https://github.com/payara/Payara/issues/3360#issuecomment-444692778
Issue: #3493

Workaround currently would be to configure the PayaraExecutorService to a queue size of 0 (unlimited) or something large like 1000.

Expected Outcome

Payara starts and deploys consistently.

Current Outcome

Depending on the machine performance (slow harddrive, single core processor or high processor load at deployment or startup) the submitted tasks fill up the threadpool queue and may eventually lead to java.util.concurrent.RejectedExecutionException.

Steps to reproduce (Only for bug reports)

Startup or deploy on a _slow_ machine. This can involve:

• single core
• old, slow hard drive
• low availability for both or any of them

Samples

Please see the mentioned tickets above.

Context (Optional)

Server startup or application deployment.

Environment

• Payara Version: 5.184
• Edition: see the ticket mentioned above
• JDK Version: see the ticket mentioned above
• Operating System: see the ticket mentioned above
• Database: see the ticket mentioned above

[smillidge]

The queue size can be fixed using the asadmin command
asadmin set-payara-executor-service-configuration --threadpoolexecutorqueuesize 1000

This is a known issue with ID PAYARA-3328

[svendiedrichsen]

@smillidge Maybe besides increasing queue default size the RejectExecutionHandler could also by default be set to https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.CallerRunsPolicy.html

This would prevent these Exceptions even in case a bigger queue gets filled.

[smillidge]

Yeah will do. Weird thing is noone saw this in dev or testing even with huge ear files. (sigh)

[svendiedrichsen]

Maybe tests on a machine with limited resources for IO and cores could help?

[marcreichman]

The queue size can be fixed using the asadmin command
asadmin set-payara-executor-service-configuration --threadpoolexecutorqueuesize 1000

This is a known issue with ID PAYARA-3328

Is there a way to apply this with Payara micro on the commandline? Or would we need to use the domaindir approach?

I'm trying to figure out if it's worth it to use 5.184 with workarounds or just use 5.183.

[dhubbard-ag]

Just an addition for others - I hit the same problem on payara start after upgrading to 5.184. As the app server instance wouldn't start I was unable to run asadmin set-payara-executor-service-configuration --threadpoolexecutorqueuesize 1000.

However I was able to edit domain,xml file with

    <config name="server-config">
        ...     
          <payara-executor-service-configuration thread-pool-executor-queue-size="1000"></payara-executor-service-configuration>

and successfully start.