The Importance of Benchmark Testing

benchmarkingWhen creating a web application, the questions that have to be answered are “What will it do?” and “How well will it work?”. The answer the first question lies with the developers that are creating the application, the answer to the second question can be found through benchmark testing.  Benchmark testing is the part of the software development life cycle where several teams – developers, database administrators and a solid group of QA testers combine to determine current performance and make changes to improve the application’s performance.

Goals of Benchmark Testing

There are several goals to be met when benchmark testing. The first is to ensure that the application meets the minimum specifications in terms of response time, number of concurrent users, and any other performance requirements as specified in the SLAs that are mandated. This requires an iterative process of testing, improving, and retesting until performance levels are met.  The second is to retest the application after upgrades and changes are made. The third goal is to determine the breaking point of the application, to increase load and demand until it breaks. This not only marks the ‘top end’ of the application’s performance for both operations and support, it allows the developers to see how the application fails. Having an application that fails gracefully, or that can self-regulate to keep from failing provides a better experience for both users and the support team.

Benchmark testing is performed using the same software and hardware components that will run in the production network. The hallmark of this type of testing is the repeatability of the test suites. Each test run should be exactly the same to ensure apples-to-apples comparisons across test runs. There should also be no other applications running in the test bed than the one under test. Problems found should be well documented including the failure symptoms, the state of the test bed when the failure occurred (number of connections, amount of traffic being moved, what portion of the application failed), and how far into testing the suite had progressed. You want to ensure that the development team can duplicate the failure to find the fault and fix it.

Test Tools for Benchmark Testing

One key component to successful benchmark testing is the test tool. You need a tester that can mimic the types of actions that a user will be trying to do in the real world. It needs to be able to simulate any number of concurrent users from one to thousands so you can meet, and then exceed, the application’s specifications. It also needs to be able to capture the transactions, as well as the test results, so when failures are uncovered they can be traced back to their source.

The payoff at the end of benchmark testing is an application whose characteristics are well understood and that can stand up to the rigors of real-world users. You will be able to launch with confidence because you know your users will be satisfied with their experience.