Symptoms of performance or hang issues may be observed with any number of threads executing on java.net.SocketInputStream.socketRead for a prolonged period of time. The following will detail how to determine if such threads are problematic and offer some recommendations for further investigation.
Many performance issues are not due to an issue with the application server (or JVM) itself but with one or more of the remote resources to which the application server is connected. Threads executing in socketRead for a prolonged period of time may be a symptom of such an issue. The following is an example of a thread executing in socketRead:
The bottom part of the stack leading up to the socketRead will vary depending on what type of remote resource is being used and what code is initiating the activity.
Diagnosing The Problem
Not every thread found in a state such as the above is problematic. The stack above is in fact a normal and expected state for any number of scenarios where the application server is reading data from a remote resource. For example, it is common to see threads in a similar state while they are retrieving results from a database or an LDAP server. Any time the application server is reading data from a remote resource, it is likely that socketRead is being used.
Suspicion only arises when we observe threads executing on socketRead for a prolonged period of time. That being said, it is also important to note that some prolonged socketRead operations may be expected or necessary (reader threads, large but expected result sets, …).
There are a number of different ways to determine if a thread has been executing on socketRead for a prolonged period of time. Here are two specific methods:
Regardless of the method used above, once a thread of interest has been identified, the thread stack leading up to the socketRead must be analyzed in order to determine more information about the remote resource. What is the remote resource? Is the remote resource a database, an LDAP server, another WebSphere Application Server process, or something else altogether? What code or ‘use case’ is initiating the activity?
With an understanding of what type of remote resource is being used and how the activity is being initiated, steps to resolve the problem can be taken.
Resolving The Problem
The socketRead method is classified as a blocking operation. As such, this method will execute until all the data has been read from the remote resource or until the connection is terminated by the remote resource. There are any number of reasons why an executing socketRead could take longer than expected and result in a hang or performance issue. The points below outline a number of aspects to keep in mind while resolving this type of issue.
- What is the health of the remote resource at the time of the issue?
- Health of database, LDAP, …
- Is the remote resource responding to any other requests?
- What is the health of the connection between the application server and the remote resource at the time of the issue?
- Health of networking, firewalls, routers, …
- Is there anything impeding the free flow of data from one end to the other?
- How much data is being requested / sent by the application server / remote resource at the time of the issue?
- Could it be that the application just requested a massive amount of data that is simply taking a long time to transfer?
- Did the application server just execute a query for all the records in a database?
- Did the application server just request all the users from an LDAP server?
Again, each situation is different and there are a number of different ways in which issues like these can come about. Using these points as a guide should assist in addressing performance issues with threads executing in socketRead for a prolonged period of time.