Data Source Connection pool running out of connections in JBoss EAP

Application is configured to use a DataSource connection pool and during run time from the logs we see below error’s

• Unable to get a new connection from a JBoss EAP managed connection pool
• In JBoss EAP 6 and later IJ000453 and IJ000655 messages such as the following appear together in the server.log

... javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/MyDS
...
... javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])

... No ManagedConnections available within configured blocking timeout ( 30000 [ms] ]

"ajp-127.0.0.1-8009-1" daemon prio=10 tid=0x00002aaabde44000 nid=0x106c waiting on condition [0x000000004d2b7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000078f4a3140> (a java.util.concurrent.Semaphore$FairSync)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
    at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)
    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:193)
    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:747)
    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)
    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:424)
    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
    ...

FiX

• Increase the value for max-pool-size for the connection pool (the default is 20)
• See the diagnostics section below for details on checking for connection usage and leak issues.
• In rare cases, it may be sufficient to increase the value for blocking-timeout-millis (default is 30000 milliseconds or 30 seconds).
  • The blocking-timeout-millis element within the datasource timeout settings (see docs/schema/jboss-as-datasources_1_*.xsd) indicates the maximum time in milliseconds that a thread will block while waiting for a connection before throwing an exception.
  • Note that this setting only impacts the block while waiting for an existing connection, and will not impact the block when creating a new connection takes an inordinately long time.
  • This is not usually feasible to resolve IJ000655 errors encountered by end users but may be suitable to address errors encountered during batch processing

Root Cause

• The IJ000453 exception (in this case) results from the IJ000655 exception
• The IJ000655: No managed connections available within configured blocking timeout exception (or the EAP 5 and earlier equivalent) is usually seen due to 1 of 3 reasons:

1. The datasource connection pool has not been tuned (e.g. max-pool-size and blocking-timeout-millis) correctly for the maximum load on the application.
2. The application is leaking connections because it is not closing them and thereby returning them to the pool.
3. Threads with connections to the database are hanging and holding on to the connections or slow performance (e.g. due to CPU or memory usage issues) may delay (otherwise rapid) return of connections by worker threads and lead to pool exhaustion.

Sample Datasource tags from Domain or standalone config looks like below

<datasource jta="false" jndi-name="java:/prov2DS" pool-name="prov2DS" enabled="true" use-ccm="false">

     <connection-url>jdbc:oracle:thin:@dbhost:1521/testdb</connection-url>

     <driver-class>oracle.jdbc.OracleDriver</driver-class>

     <driver>OracleJDBCDriver</driver>

     <pool>

       <min-pool-size>10</min-pool-size>

       <max-pool-size>30</max-pool-size>

       <prefill>true</prefill>

       <use-strict-min>false</use-strict-min>

     </pool>

     <timeout>

       <idle-timeout-minutes>30000</idle-timeout-minutes>

     </timeout>

     <security>

         <user-name>username</user-name>

         <password>password</password>

     </security>

     <validation>

         <validate-on-match>false</validate-on-match>

         <background-validation>false</background-validation>

     </validation>

     <statement>

         <share-prepared-statements>false</share-prepared-statements>

     </statement>

</datasource>

If you get this error monitor the connection pool usage during hight load and plan the max connection pool accordingly . In the lower environments during the load test one should be able to judge these value based on the load test results .