Planning Automated Regression Testing for Integration Projects

What is Regression Testing?

Software regression testing is the means of identifying defects that may have been introduced as a result of changing a component within your system. The system regresses by no longer working as it used to, or to put it more accurately, it no longer works as specified. Regression tests help identify changes between a selected release and a previous release – we will refer to this as a baseline. A baseline is a recorded snapshot of desirable product “behavior”. This expected behavior is then used to ensure that nothing has been broken in the system as a result of changes introduced in a module or component. Establishing a regression testing framework is crucial for building a reliable and stable SOA infrastructure.

Regression testing can be exercised at any point during the software lifecycle – during unit, functional and non-functional testing. For the purposes of this section, we shall examine its use in end-to-end system testing.

Why it is important

Testers face unique challenges in performing regression testing of an SOA. The fundamental advantage of a Service Oriented Architecture is reuse of services across a distributed, technology agnostic infrastructure. In a successful SOA deployment, the number and re-use of services should continue to increase. As the number of services and their re-use within a SOA increases, the difficulty in regression testing of services increases dramatically owing to the interdependencies of the services within a distributed environment. If the behaviour of one of the services changes, all the dependent services are at risk of exhibiting faulty behavior. As a result, SOA Architects, Developers and Testers are now responsible for adapting their regression testing techniques.

Environments and controls

In an ideal model, end-to-end regression testing should take place in an environment that is controlled by a central authority, such as a Testing Center of Excellence (COE). The introduction of components into the environment is tightly controlled and the environment should best represent the end-to-end and distributed nature of the system under test.

How to begin to build the regression test set

As discussed previously, an asset sharing, “embellishment” approach works best when moving through the test phases. Automated test cases are shared between development and test teams and they gain more business-like logic added to them as they mature. These automated test cases propagate into the controlled end-to-end test environment and form the basis of your regression tests.

No doubt you will also have manual test cases. This is where the analysis comes in. Which of these tests should be introduced into your regression pack?

Consider the following:
· Are these tests likely to be run frequently?
· Will they be reasonably easy to maintain (data)?
· Do they exercise “busy” areas of the system?
· Are there lots of downstream impacts?

If the answer to some of those questions is yes, then you have an extremely good candidate for your automated regression pack. So, when a new component is promoted into the end-to-end environment all of its automated test cases are introduced into the regression set. But this is not the end of the story.

Running the set

Test cases may be a combination of UI and non-UI automated tests that are run as a scheduled suite of tests. They must be choreographed in order to satisfy end-to-end business process and ensure data integrity. In order to maintain this integrity, it may be necessary to include automated scripts whose sole purpose is to create and tear down data, both for driving the test cases and for static, referential data.

Adjusting the set

The best automated regression packs are fully scalable both in terms of breadth and depth of coverage. The output from each of the runs should feed into later runs to allow the regression testing to become more focussed and surgical. Put simply, there is more depth, less breadth. Firstly, consider how you are going to measure the efficiency of the regression set. The most obvious measure would be timings. How long does it take to execute each section and the entire set?Defects can be tracked on a per test case basis and shown up in a heat map. Test cases that frequently uncover defects are particularly effective test cases.

Consider building a matrix that maps the test cases against functionality. The incremental approach to integration should mean the existence of granular test cases that can make up end-to-end business functionality. This can greatly assist with impact analysis. A change to a granular piece of functionality may only require the execution of a limited regression test set. This can be more easily determined through assessment of the matrix with a visual representation of the system under test. The best SOA automated test tools now provide visualization of the architecture, allowing easy identification of downstream dependencies in order to better focus your regression tests.

In continually adjusting your regression test set, you should also consider the following:
• Which are the tests that cover core functionality and which are those that are less frequently exercised?
• What data needs to be run to exercise the functionality? Can this be scaled down from the original test set provided with the service component? Can typical data coverage assumptions such as boundary value analysis or equivalence partitioning be applied?
• What are the integration points around the change?
• Can you prioritize execution of the tests within the suites? Can you run the core functionality tests first as a form of smoke test to determine if the system is up and running correctly and quickly find defects?
• How are the tests prioritized in terms of business need and technical risk?

The most important part of all this is to standardize the process – ensure that you are provided with the minimum of inputs to determine the scope of your execution. As you have no doubt determined already, the more inputs you have into determining the scope of your regression testing the easier your job will become.