|
||||||
|
What is Regression Testing? Software regression testing is the means of identifying defects that may have been introduced as a result of changing a component within your system. The system regresses by no longer working as it used to, or to put it more accurately, it no longer works as specified. Regression tests help identify changes between a selected release and a previous release – we will refer to this as a baseline. A baseline is a recorded snapshot of desirable product “behavior”. This expected behavior is then used to ensure that nothing has been broken in the system as a result of changes introduced in a module or component. Establishing a regression testing framework is crucial for building a reliable and stable SOA infrastructure. Regression testing can be exercised at any point during the software lifecycle – during unit, functional and non-functional testing. For the purposes of this section, we shall examine its use in end-to-end system testing. Why it is important Testers face unique challenges in performing regression testing of an SOA. The fundamental advantage of a Service Oriented Architecture is reuse of services across a distributed, technology agnostic infrastructure. In a successful SOA deployment, the number and re-use of services should continue to increase. As the number of services and their re-use within a SOA increases, the difficulty in regression testing of services increases dramatically owing to the interdependencies of the services within a distributed environment. If the behaviour of one of the services changes, all the dependent services are at risk of exhibiting faulty behavior. As a result, SOA Architects, Developers and Testers are now responsible for adapting their regression testing techniques. Environments and controls In an ideal model, end-to-end regression testing should take place in an environment that is controlled by a central authority, such as a Testing Center of Excellence (COE). The introduction of components into the environment is tightly controlled and the environment should best represent the end-to-end and distributed nature of the system under test. How to begin to build the regression test set As discussed previously, an asset sharing, “embellishment” approach works best when moving through the test phases. Automated test cases are shared between development and test teams and they gain more business-like logic added to them as they mature. These automated test cases propagate into the controlled end-to-end test environment and form the basis of your regression tests. No doubt you will also have manual test cases. This is where the analysis comes in. Which of these tests should be introduced into your regression pack? Consider the following: If the answer to some of those questions is yes, then you have an extremely good candidate for your automated regression pack. So, when a new component is promoted into the end-to-end environment all of its automated test cases are introduced into the regression set. But this is not the end of the story. Running the set Test cases may be a combination of UI and non-UI automated tests that are run as a scheduled suite of tests. They must be choreographed in order to satisfy end-to-end business process and ensure data integrity. In order to maintain this integrity, it may be necessary to include automated scripts whose sole purpose is to create and tear down data, both for driving the test cases and for static, referential data. The best automated regression packs are fully scalable both in terms of breadth and depth of coverage. The output from each of the runs should feed into later runs to allow the regression testing to become more focussed and surgical. Put simply, there is more depth, less breadth. Firstly, consider how you are going to measure the efficiency of the regression set. The most obvious measure would be timings. How long does it take to execute each section and the entire set?Defects can be tracked on a per test case basis and shown up in a heat map. Test cases that frequently uncover defects are particularly effective test cases. Consider building a matrix that maps the test cases against functionality. The incremental approach to integration should mean the existence of granular test cases that can make up end-to-end business functionality. This can greatly assist with impact analysis. A change to a granular piece of functionality may only require the execution of a limited regression test set. This can be more easily determined through assessment of the matrix with a visual representation of the system under test. The best SOA automated test tools now provide visualization of the architecture, allowing easy identification of downstream dependencies in order to better focus your regression tests. In continually adjusting your regression test set, you should also consider the following: The most important part of all this is to standardize the process – ensure that you are provided with the minimum of inputs to determine the scope of your execution. As you have no doubt determined already, the more inputs you have into determining the scope of your regression testing the easier your job will become. 4th August 2011 saw the Green Hat team presenting at the London Tester Gathering. These gatherings are ably run by Tony Bruce and provide a relaxed environment for testers to network and discuss new ideas. There were about 50 testers packed into the Shooting Star function room, which proved for an intoxicating mix of pheromones and ISTQB capability. With only 15 minutes to present, it can be difficult to decide what to leave out. One thing is for certain, you need to get the first few minutes spot-on to keep everyone’s attention in such a relaxed atmosphere. Handy hint: paying for the beer helps to some degree. The objective of the demo was to get testers to understand the difference between traditional Automated Testing against the UI and Automated Testing with Green Hat Tester. One of Green Hat’s Technical Account Managers, Chris Haggan, took to the floor to dazzle the crowd with his GH Tester expertise. The presentation went very well. Chris used our mocked-up flight booking system to demonstrate the end-to-end process of test creation, execution and results analysis. What surprised me was the quality of the questioning throughout. Even though we only presented for 15 minutes it was clear that the audience had grasped the more technical nature of the testing being demonstrated. When it comes to GUI-less test automation it can sometimes be difficult for newbies to visualize the flow of the test. That did not prove to be the case here. Architecture School view helps greatly with this. A logical view of the system under test allows the tester to understand the inputs and outputs of a test case in the context of the architectural layout. Indeed, Architecture School synchronization is always my favourite part of demos (in this instance we were synchronizing with a webMethods Integration Server). The moment the crowd realises that we have automatically pulled in the logical layout, schemas, dependencies and physical environment details into GH Tester, it always goes a bit quiet. I call this the “zone of realization”. As in, “I’ve just realized how much simpler you’ve made my job”. So that’s always a good place to focus for short presentations. Synchronization kick-starts the test case creation process as we can now easily create tests from the schemas, and GH Tester now understands how the schemas are used within the operations contained in the system. Chris then took us through the test creation wizard and ran some simple tests against the flight booking system. After the presentation, there followed questions on our GUI testing and our Virtualization capabilities. So what did we learn from the London Tester Gathering? To keep things simple for short presentations and let the crowd visualize what you are about to test. And let the tool do the talking. Look out for the Green Hat crew at more events of this nature. If you have an idea as to any topics you would like to see discussed, do drop us a line. Below are the remaining tips which I hope you find useful: 6 Use Virtualization Make use of the SOA automated test tools’ capabilities for virtualizing services, databases and other components. This approach is not be confused with hardware virtualization – we are only concerned with virtualizing the behavioural aspects of the component in order to generate a realistic load. Not only will this provide you with the flexibility to execute when components are unavailable, it may also allow you to scale up beyond performance levels that your standard test environment could typically deliver. 7 Account for security layers An over-engineered SOA security plan can be damaging to performance as it will introduce tight coupling and performance bottlenecks, bringing the enterprise performance to a crawl. Security layers may add run-time performance overheads associated with the required encryption and decryption of message data. Plan to perform additional performance tests after the introduction of security layers. 8 Include GUI response times (but only test for them in isolation) When considering the end-to-end performance of a SOA implementation, front end rendering times on the UI must be included. To determine the rendering times, an isolated test on a representative client machine should be all that is necessary. These transaction response time results can then simply be appended to your UI-less performance test scenarios. 9 Report your findings A performance test tool has to be supported by extensive reporting capabilities that organize and present the information gathered during performance testing into a comprehensive report. Without this capability, valuable insight into the performance of the system is wasted. Expect to run your performance tests scenarios repeatedly. Results baselines are essential for comparison against new builds and configurations. Analysis of the statistics produced by runs will take up a good deal of the tester’s time. Experienced performance testers will understand that the process of analysis involves looking for relationships between the data sets. For example, how did the ramp up of messages affect the CPU utilization and transaction times? 10 Think about and visualize the impact of queuing Slow response times can be the result of bottlenecks in the system that may essentially be a result of architectural design. Simply put, if more messages are being produced onto a queue than are being consumed then this results in queuing. The greater the delta between consumption and production then the quicker that queue will grow. Queuing results in system bottlenecks that impair response times. This blog makes no attempt to describe the more complex elements of queuing theory as that would require a further 1000 pages, but it is well worth investigating and understanding the basic concepts. Have I missed any? Please let me know! A customer recently asked me how we approached performance testing for SOA at Green Hat, and I thought it made sense to share some of our proven approaches with a wider audience. 1) Simulate the projected real world usage by reusing test cases created for functional testing One of the advantages in using a dedicated automated test tool for SOA testing is that functional tests can easily be rerun within a performance scenario. Inexperienced users may need to modify their functional tests to make use of them in this way but experienced users will have planned ahead: as with coding, the fewer and simpler the test cases, the easier it is to respond to system changes in future releases. It is insufficient to merely send a request and measure the time it takes for a service to come back with a response. If an operation simply rejects the input and sends back a fault message, the time it takes to do this is significantly different from the time it takes to properly process a request and return a valid response. If a test does not validate the outcome of an operation, it can provide an inaccurate view of the true system performance. 2) User think time is the staggering of payloads The majority of testing for SOA occurs at the process-to-process or transaction level where user think time is often redundant. Contrast this with traditional end-to-end performance test approaches, where think time will exist between transactions within an individual script. With SOA Performance Testing there is little need to consider where to place think times within scripts, it is more of a question of how you stagger scripts and payloads to accurately represent the usage profile. 3) Model the nature of the distributed architecture for your scenarios With services deployed over the Internet, clients can potentially access them from all over the world. When evaluating the performance of these services it is therefore necessary to correctly model and measure network latency and throughput. The ideal solution is to simulate the traffic across different locations by distributing load generation through multiple machines set in the corresponding target locations. Alternatively (or in conjunction with the above), judicious use of sleep time and staggering of the scripts will allow you to accommodate these scenarios. 4) Perform load and stress tests using bell curve and linear increase scenarios Stress tests push the system beyond predefined operational limits to see how the system behaves and if the system recovers within operational parameters. It can be very difficult to predict web service traffic, the “openness” means that there are rarely constraints on access and availability. The possibility of surges and exponential growth in traffic should not be dismissed. Thus, if you possibly can, test for the ultimate scenario and push it to its limits. 5) Make good use of probes (not in an alien abduction type way) With complex and heterogeneous platforms, it can be difficult to understand what to measure apart from transaction response times, but it is also important to note that in a reasonably complex environment it will be impossible to measure everything. A performance probe measures statistics from a particular component in the system under test. These may include an app server, broker, process engine or even a database. Test automation tools often possess the capability to deploy probes that gather statistics for all of the messaging and integration components with your SOA implementation. System and Database resource stats are useful in determining when components are stressed and what transactions may be causing the stress: as this is an integration project, the delays are far more likely to come from existing systems subjected to new loads (e.g. by now providing customer self-service), than they are to be caused by the processes themselves. We must make sure that we performance testing to a sufficiently granular level of detail in order to pinpoint the cause and effect. Look out for tips 6-10 in my next blog post which appears later next week. If you’d like to share your top SOA performance testing tips, then please submit a response and we will do our best to publish it. A few people have been asking us recently what factors influence SOA and Web Service test phase estimation. We’ve gathered some thoughts here for you on this topic. Firstly, you’re going to need a tool to execute your tests. The power and capabilities of the tool will greatly influence how long the testing will take. If the tool is cheap (or free), like some open-source offerings, testing estimation can be a fraught process. There can be so many unknowns with free test tools that the margin of error around your estimations will have to be very wide. The more established and powerful the tool is, the more confidence you will have around your estimations. Put simply, testing can take far longer (and therefore cost far more) using open source tools. We recently built a test framework for common customer scenarios in a SOA project using Green Hat’s GH Tester. The framework was built before lunchtime on the first day. After that it became more of a traditional data preparation exercise, some of which the tool can help you with and some of which it cannot, depending on how thorough the service design has been. But one thing was very clear after that initial half day of scenario creation, we then had a solid understanding of how long testing preparation would take. Here are some factors which can influence the time it takes to create test cases and thus influence your estimations:
This is not an exhaustive list of factors but should get you pointed in the right direction and provoke some more discussion. Remember: SOA testing benefits from quick iterations and an agile approach. So, you may find that you are revisiting (and refining) your estimations more often than on a typical testing effort. That’s one of the agile mantras – “Always be planning”. If a car fails to start, you know it has “failed” one important function. But where do you begin to identify the cause of failure, which could be lack of petrol, flat battery, or any number of other reasons? Knowing the car won’t start is just part of the story and it can take hours of investigating to discover the root cause of the problem. It’s the same with testing. There is little point in knowing a test failed if you do not know where and, more precisely, why it failed and what was going on at the time with the system you were testing. It is all about context, and when a developer comes to investigate a defect, they want to quickly establish what the context was when the defect occurred. Ideally, all of this information would be accessible directly from the defect tracking system itself. The good news is, this is exactly what we have, baked into the core of GH Tester. Green Hat, unlike other vendors, does not charge you extra for the features you ought to be expecting from your test automation platform and we certainly do not pretend it is a separate product that you need to buy. With GH Tester you can very simply record the system’s end- to-end transactions, see the inputs and outputs at each point along the way, together with database and other activities and then select the appropriate events to quickly and easily turn those recordings into a repeatable end-to-end regression test. This allows for the swift, simple creation of code free tests of the entire process without the need for a deep technical knowledge of the system under test. When the test is run as part of a regression suite, the test will show you which part of the process failed and the input and output data including any differences from the baseline. GH Tester also has the ability to validate log files, interrogate databases, examine output files, perform reconciliations, record performance times and capture stack traces that all assist in the root cause analysis of a defect. This helps the developer to understand the exact system state at the time the defect occurred. The ability to identify and pin-point defects as well as capture stack traces and information from log files greatly reduces the time to correct a defect. All of this context information is captured in a link and easily added into a defect tracking system. GH Tester has built-in integration with major defect tracking systems such as RTC and JIRA, making the task even simpler. All that is required is a mouse-click. If the end-to-end test is used in conjunction with another part of the GH Tester Suite, GH Performance, you will also be able to capture system statistics as the process runs. This helps to quickly identify system bottle necks or hotspots in process definitions, catching problems and enabling tuning even before the heavyweight performance testing is carried out. In this way, critical performance problems are identified much earlier in the lifecycle and are thus prevented from reaching production. Returning to the car analogy, it’s like having a fuel indicator gauge, low oil indicator and much more letting you know what is not working as it should. Last Wednesday saw the return of Green Hat to a London event we attended late last year, TestExpo. Once again, we saw around 200+ attendees from both large enterprises and consultancies at a busy and lively show. Around 90% of visitors were from the testing community, with test leads, test managers and project managers all present. Most told us they wanted to expand their toolset to handle their increasingly changing environments. Many were looking to expand test coverage, improve test quality, and test earlier in the development lifecycle (as a new release is assembled), well ahead of the UI stage. Green Hat was also a presenter at the event and I had the opportunity to share our approach to accessing systems that are unavailable for testing, featuring GH VIE (Virtual Integration Environment). GH VIE takes the established practice of stubbing but then dismantles the traditional view of stubs as complex, requiring coding and therefore solely the preserve of the development team. Green Hat’s view is very different - we make stubs simple to create, codeless, and (much to the approval of our audience) something testing teams can now own. GH VIE is part of the GH Tester suite and a key component of our vision of 2nd generation test automation. I talked about how we create production level performance simulations, minimize risks by creating an automated emulation of an organization’s environment, and model an existing system’s behaviors, however complex. This created quite a buzz amongst the audience who flocked to the booth during the break. We were asked lots of interesting questions, such as what we can and cannot stub, how GH VIE integrates with our other products (answer: seamlessly), and if we can support industry specific requirements such as financial feeds using the FIX protocol (yes we can). There was quite a queue around the stand! One visiting company had the challenge of how they integrate their software with their client’s systems and how 3rd party license rental costs can be reduced. GH VIE helps eliminate “extras” for making an environment available just for testing. Another delegate told us he saw a virtual environment fitting well into their Agile development strategy as it cuts the delays in waiting for systems to become available. OK, the location was Milan not Turin, and the local welcome was much warmer, but both Brit visits left an impression! On March 15th Green Hat and our Italian partner Primeur co-hosted a briefing at IBM’s Italy HQ at Segrate, just outside Milan. Having achieved several “Ready for Rational” validations in December last year, this was the perfect opportunity for Green Hat to showcase the integration between GH Tester and the Rational Suite. By combining the technologies, enterprises can extend their test and change management processes to the complex test environments that BPM and SOA projects often involve. IBM opened the event with an overview of their Rational suite and its role in driving Enterprise Quality Management. Green Hat and Primeur then explained how GH Tester and Rational work together. In a nutshell, the user-created GH Tester assets (test cases, suites or performance tests) are retrieved from a repository by Rational Quality Manager (RQM); RQM then runs the test using an RQM Agent which produces the results; these are then reported and analysed within Rational Quality Manager. Software defects (detected by execution from within or without RQM) can then be raised in Rational Team Concert to enable the software development team to plan and develop fixes. The quality lifecycle is then easily repeated. Green Hat and Primeur showed attendees how they can simply build code-free automated tests for their integrated systems using GH Tester’s comprehensive palette of actions while leveraging Rational Quality Manager for test planning and management. The ability to execute tests on demand or when scheduled is another plus. We concluded the session with a detailed demo showing the products in action together. Delegates were pleased to see the integration was far from the “slideware” offered by some vendors. We were asked about integrating the Green Hat suite with HP’s Quality Center and as well IBM Rational. The value Green Hat’s suite can offer when compared to traditional suites was another popular subject. One standout item was the deep analysis capabilities of the solution, such as how to analyze and verify the status of the tools and the quality of applications. Analysis of a different kind was also mentioned; chiefly how other quality practices (API, TQM, Deming etc) are handled in relation with the test phase. The creation of a virtual environment to enable the access of unavailable systems was of great interest, and we referenced our innovative GH VIE (Virtual Integration Environment) offering. Integration was never far from the minds of attendee and we explored the integration and analysis of existing systems. The event concluded with very positive feedback from the delegates and a number of agreed follow-ups. We’ll be posting additional details on the Rational integration and some of the questions raised over the coming months, including a demo and more about building in quality during the test phase. Green Hat left the “Job” in a white Mercedes taxi; the coach loaded with gold bars being apparently delayed elsewhere…  Cycles waiting for action outside our offices I’ve watched them dig the holes and wondered why. I’ve watched them fill them in again leaving funny plates behind. I’ve watched them install the bike stands and understood. I’ve registered and watched the bikes arrive in obsessive excitement and today I took one of the bikes for a test ride. But how well tested is this system which, I’ve been told, we’ve imported from Canada? I’ve never eagerly anticipated the launch of something quite so much in all my life. As soon as I could I registered for my key and watched in amusement as it arrived two days later in a handwritten envelope. “That won’t scale” I thought to myself. Reminder emails badgered me to activate my new key, I guess some kind of security thing to make sure the credit cardholder had the key. Failure #1: Why then does my online account actually display the key number anyway, even though I haven’t yet completed the activation process? This failure enabled me to activate the key whilst at the office, without having the key in my possession. Today marks the first day of the Barclays Cycle Hire scheme in London. I am a keen cyclist, and arrived later than usual into my normal station and made for the nearest bike stand. From afar it looked like a lot of bikes had already gone, and worried about a cluster of people around the stands, I broke into a run. As the distance between me and my new mode of transport shortened, allowing my myopic vision to discern events correctly, I realised the cluster of people were in fact journalists. I approached a stand with a bike and began the process of removing one. “Do you mind if I ask you a few questions” someone from the Guardian asked me. “Not at all” I replied, as I continued the process of obtaining a bike. A few questions later, I had managed to insert and withdraw the key, but was struggling to remove the bike. This is much harder when someone is waving a boom microphone in your face. Then disaster struck, the stand locked itself up again and then reinsertion of the key into that stand or any other (all accompanied by the microphone, possibly to detect explitives) failed to elicit the warm and friendly glow of a green LED. Instead I got a red LED. At that point, I’m not sure if it was me, but I swear the journalist smiled smugly when she said that this had happened to a few people and maybe the people round the corner could help at the official press opening. I didn’t clarify if all the people having problems also had microphones in their faces but did take the advice. I got the feeling there were lots of people hoping for this to go wrong. Round the corner there were more bikes, but none of these could be unlocked. “Not until 09:30″ an official said. Another helpful pair gave me a phone number, which I called. “Due to high call volume …” said the recorded message. I persevered, determined not to be put off. Michael soon answered the phone and after a few putting on holds said I just needed to wait five minutes before trying again. Failure #2: I agree red LEDs are cheap, but hardly an effective way of communicating any kind of complex message as to WHY I was being denied a bike and the fact that if I waited five minutes I would get one. If every one of these events generates a phone call to the call centre for each user the first time they try and get a bike, that’s gonna be expensive. Failure #3: What is the point of having bike stations manned during this initial adoption period if all they do is tell you to call the call centre? Training would have been nice. I returned to my original bike stand to find more journalists and TV people who now wanted to film me getting a bike out. This time it happened so quick they missed it so I had to repeat it a few times, and then again for a different crew. By now I’m late for the office but enjoying my five minutes of celebrity, and conscious this may be it for me, milk it for all its worth. The bike are too heavy for me to do a wheely or any other stunts for that matter and I have to satisfy myself with my new skill of docking and undocking without getting locked out by the computer! Finally the journalists tire of this and I get to cycle towards the office. Apparently I’m not the only one who has managed to get a bike, and there is a quiet smug feeling of riding a bike that belongs to someone else, even if you do get overtaken by proper cyclists on meaner machines, the smugness remains of having paid 12p today for this remarkable privilege. I’m surprised by the presence of gears and they seem fine for the purpose of keeping up with traffic, although I wouldn’t like to tackle Ditchling Beacon with them! The bike stand at the office (pictured) is almost too full for me to park, although I do get a spot agreeably close to my final destination, dock the bike and grin. Despite the initial hiccups, most likely caused by user training anyway, it looks like being a success and I cannot wait until I get to cycle back to the station again tonight. Some lessons then for all of us building systems:
I’m a big fan of this scheme and look forward to seeing it blossom.
So now, Java programmers and testers of Winchester, Southampton, Portsmouth, Basingstoke and beyond, if you’re fed up with the commute to London, want a fresh challenge, want to make a difference to a company big enough to be safe but small enough to be free of bureaucracy, where the code you develop will end up very rapidly on the desks of people in banks, electronics companies, telecommunications companies, hospitals and so on all over the world, this is the place to be. Drop us a line through our recruitment pages and come and join the fun, we’ve got jobs waiting for you! To steal a phase from CrimeWatch: we’re waiting for your call. |
||||||
|
Copyright © 2012 The Green Hat Blog - All Rights Reserved
|
||||||
After months of preparation, the Winchester office opened this week in the heart of Hampshire and in the old capital of the Kingdom of England. It’s been really busy getting everything sorted. The furniture movers did an excellent job, and I was glad they turned up as apparently other people in the building were worried about the strange man sitting on the floor with a laptop on his lap talking into his iPhone. Hopefully they are now reassured that yes, we can afford chairs and desks too! A big thanks to Rebecca and Nadia who have made this process one of the smoothest new office openings I have ever been involved with - we couldn’t have done it without you.