Failed to acquire a permit within 5 MINUTES exception in EAP 7

We are seeing the following exception, leading the server to become unresponsive. 

ERROR

11:46:16,829 ERROR [org.jboss.as.ejb3.invocation] (default-threads – 9) WFLYEJB0034: EJB Invocation failed on component mycomp_MySampleCMTEJB for method public abstract void aaa.bbb.ccc.MySampleCMTEJBLocal.methodOne(aaa.bbb.ccc..myVarOne,aaa.bbb.ccc.myVarTwo): javax.ejb.EJBException: javax.ejb.EJBTransactionRolledbackException: WFLYEJB0378: Failed to acquire a permit within 5 MINUTES
at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:188) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

Solution

• The exception you have cited indicates exhaustion in the EJB’s slsb-strict-max-pool. You need to increase the number of instances in the slsb-strict-max-pool to cater to the demand using the max-pool-size attribute. Please increase the slsb-strict-max-pool to 100 or 200 and observe the outcome. You need to define an ideal max-pool-size based on the load of the EAP server. Here’s a sample configuration snippet for your reference:

<bean-instance-pools>
<strict-max-pool name="slsb-strict-max-pool" max-pool-size="200" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
....
....
</bean-instance-pools>

Root Cause

• The EJB container creates a new bean instance each time a bean is accessed until the container reached the configured max pool size. Then, the container would block incoming instance requests up to 5 minutes, and wait for the bean instances to be returned to the pooled by the clients. If the clients take longer than 5 minutes to return the bean instances to the pool, the incoming bean requests cannot be accommodated by the EJB container and throw the reported exception. The defaults are adequate for most deployments. However, you need to increase the instance pool size should the clients occupy bean instances for more than a few minutes.