Concurrency Testing in JAVA Applications

Testing and optimizing Java code to handle concurrent activities is difficult without test automation. Even with test automation, being able to correlate the test activity from the client side to observations of thread, memory, object and database connection use on the server side is difficult at best. In this article, Frank Cohen describes methods for concurrency testing in Java applications and shows a new technique to correlate what a Java application server is doing on the server side while a load test automation tool drives a test on the client side.

Introduction

This article picks up where Brian Goetz’s article Testing Concurrent Programs ends. Goetz describes the need to use concurrency mechanisms in Java programs and gives tips to test concurrency mechanisms. For instance, Goetz article introduces a technique to avoid Java’s inherent concurrency problems – mutable and static objects – by shielding the mutable objects by wrapping them in immutable objects. Goetz says “all this sounds like a lot of work, and it is.” At this point in your concurrency testing, an investigation into test automation tools may help your effort.

Concurrency testing is well known, but hard to do

Most IT managers agree that concurrency testing is the appropriate way to determine many performance bottlenecks, resource contention issues, and service interruptions. However, in my experience few ever do concurrency testing because the available test tools are not satisfactory.

Consider the minimum functions a test automation tool needs to support to expose concurrency issues:

Unit test of a specific function.

Variations on the input data the unit test sends to a function.

Business flows through a system function using sequences of functional unit tests.

Operating the business flows in concurrently running threads. Additionally, in large-scale concurrency tests the flows may need to be operated across multiple test machines.

Variations in the mix of business flows in the concurrent running threads. For instance, the first mix may test at 20% of the business flows operating a create-a-new-user flow while 80% operate a sign-in flow and a second test reverses the mix to 80% create and 20% sign-in.

Variations in the test parameters based on usage factors. For instance, changing the test parameters based on the time-of-day or end-of-quarter.

There is no concurrency testing heaven here. You can never create a deterministic test to uncover a non-deterministic concurrency issue. The goal instead is to get close to the problem and to get lucky. Operating a concurrency test with as many of the above operational parameters might not guarantee that the test will surface concurrency issues but it will work the odds in your favor to run across the right conditions to trigger a failure.

From Unit Tests to Concurrency Tests

There are some things you should expect in today’s test automation tools. For instance, many test automation tools are now using unit tests in ways that aid in surfacing concurrency problems. Each unit test operates a particular method or class. Stringing unit tests together into a sequence forms a functional test of a business flow.

You should also expect the test tool to provide libraries that embellish unit tests to use protocol handlers to speak the native protocols of the application or service under test, including HTTP, SOAP, Ajax, REST, Email, and custom network protocols. Additionally, you should expect the test tool embellishes functional unit tests with a Data Production Library (DPL) to provide test data as input to the unit test and data to validate the service or class response.

It is reasonable to find a test automation framework wrapping unit tests into functional tests that may be operated concurrently to identify concurrency issues in an application. I coined the term TestScenario in my open-source test tool (TestMaker) to name this wrapper.

A TestScenario operates the unit tests as a functional test by operating a sequence of unit tests once, as a load test by operating sequences of the unit tests in concurrently running threads, and as service monitors by running the unit tests periodically. The TestScenario gives us the flexibility to try multiple sequences of business flows concurrently.

The TestScenario defines which TestNodes will operate the test. I coined the term TestNode for the distributed test environment operating on a remote machine or a rack of machines in a QA lab. When the TestScenario begins it moves the unit test and any other resource files to the defined TestNodes. The unit test speaks the native protocols of the application under test. The TestNodes provide the unit test data through the DPL, operates the unit tests, and logs the results.

As the TestScenario operates the test, a monitor watches the application host and logs usage and resource statistics. The test tools then correlate the monitored data to the unit test operation to identify performance bottlenecks and available optimizations.

The key to using a test automation platform for concurrency testing is to identify what is happening on the server side. I find it valuable to observe at least the following:

Java Virtual Machine statistics (heap, memory, thread, object instantiations, and garbage collection)

Service call statistics (time to serve request, external service and database query times, properties settings, and service interface request values and schema)

Subsystem statistics (if the system uses a database then I want to see the transaction logs, input parameters, and query operations)

Eventually all of these observations are going to come with the manual expense of the time to correlate the observed data to what the test automation tool is operating on the client-side. Recently I began finding agent and agentless tools to monitor server-side activity in a way that can be snapshot correlated to the test operation. One such agent-based tool is Glassbox.

Tools like Glassbox watch an application server receive requests, operate objects and EJBs, communicate with databases, and interact with message buses. These tools provide a service interface that test environments can use to automate the correlations. For instance, Glassbox uses the Java Management Extensions (JMX) interface to communicate performance observations.

Glassbox installs an agent named the Glassbox Inspector to monitor traffic through the Java virtual machine and send data to an analysis application called Glassbox Troubleshooter. The system diagnoses problems by constructing a virtual application model describing the application components and how they connect to each other using aspects. Using a singly directed graph the system represents a component or resource within the application and at each edge a usage relationship. Using aspects Glassbox discovers the components dynamically as the application runs and constructs a model automatically.

The model allows Glassbox to go beyond simple correlation analysis to trace causality through the system. For example, a correlation analysis might reveal that a slow operation occurred at the same time the database was running slowly overall, suggesting a database tuning problem. Because the virtual application model allows walking the usage relationships in both directions, the model reveals that the slow database was in fact caused by a long-running report query and identifies that as the cause.

I recently used Glassbox to observe the Sun Blueprint’s Java Petstore and correlated the observations to a load tests. Some things became clear. While I had thought the browser-to-server communication would lead to performance bottlenecks at parsing the incoming requests and marshalling the responses, we found that more often communication between the EJBs and the database caused the biggest hotspots in performance.

I was able to operate a load test of the PetStore and identify the slowest performing functions by correlating the load test to an alert for slow database operations.

Integration gives correlation capability

Some of the test automation tools and server-side monitors are going further to provide correlations between the test activity and observations at the Thread and object level on what happens in the backend.

For instance, most test automation tools produce a Scalability Index chart that shows Transaction Per Second (TPS) ratings for increasing levels of concurrent requests to a service. The following chart shows the TPS values for two use cases.

In the above chart the red bar shows the service operating at 0.17 TPS when the test use case operates for 10 minutes at 20 concurrent virtual users (threads.) The blue bar shows the service operating at 0.08 TPS at 40 concurrent virtual users. I modified the chart to also show a Performance Alert when Glassbox observes an operation that falls outside of a normal time to complete the operation. (Glassbox uses properties files to set the parameters for normal time operation.) Clicking on the Performance Alert button brings you directly to the Glassbox Inspector for an explanation of the problem and details needed to understand and solve the performance problem.

While concurrency testing is often arduous, difficult to stage, and lengthy there are a new crop of test automation tools that make development and operation of concurrency tests relatively easy. Combining these test tools with monitoring software enables correlations of load testing activity on the front-end to thread, object, and service activity to surface concurrency issues. This is not guaranteed to surface concurrency issues but it will work the odds in your favor to run across the right conditions to trigger a failure and remediate a solution.