Are You Crazy, Baseline Performance Without Testing!!!

What Is A System Performance Baseline – Before we dive into the challenges of defining a System Performance Baseline, let’s look at the definition of Baseline Performance including how it fits within the context of this article. Baselining Performance or the art of Baselining System Performance is defined as process of measuring the Performance of a system along various relevant dimensions for a given workload. There are a few key considerations that one should keep in mind when referring to the process of System Performance Baselining.

  • One always Baselines System Performance is always for a given workload e.g.
    • User concurrency e.g. 1500 Real Concurrent Users
    • 100 TPS (Transactions Per Second)
    • 7 years historical data (Within the Database)
  • System Performance Baseline can me measured along various dimensions e.g.
    • End User Response Times
    • Transaction Response Times measured at the server
    • Infrastructure Utilization Metrics
    • Application Performance Metrics
    • Database Performance Metrics
    • Middleware Performance Metrics, etc.
  • System Performance Baseline is specific to a given system configuration i.e. compute, storage, network, etc. As the System Configuration changes so does the System Performance Baseline.

A System Performance Baseline can be obtained at different production workload configurations and it’s entirely upto the Architect, Developer or Performance Engineer to determine the workload the Baseline would need to be measured at which in-turn is dependent on the nature of questions being asked.

Why Do We Need A System Performance Baseline – System Performance Baseline is measured for various reasons. Let’s look at a few relevant ones –

  • A new release is supposed to go into production in a few weeks. It would be good to baseline system performance so that one could measure performance of the new release to that of the current release
  • A system upgrade (infrastructure) is being performed and you have been tasked with ensuring that the performance of the system does not degrade post completion of the upgrade. A system performance baseline for the old system at the current production workload would be a measure which you would compare the performance of the upgraded system against.
  • You are migrating from your internal system to a system hosted within the cloud whose performance is managed by the vendor. A baseline of the current system performance (Which your business is accustomed to) for your current peak production workload is a good way to set the Non Functional Requirements for the target system.

Am sure you will find many more reasons to put together a system performance baseline. With the list of potential scenarios above we have attempted to provide a view of some of the reasons an Architect, Developer or a Performance Engineer would want to invest in building a System Performance Baseline.

Challenge – Now that we have defined what is a System Performance Baseline looks like, why one would want to obtain a System Performance Baseline including some of the reasons why you would want to obtain a System Performance Baseline let’s look at some of the challenges you might face when creating the baseline.

  • Challenge obtaining relevant performance testing tools
  • Challenges setting up a relevant sized environment for executing performance tests
  • Challenges obtaining resources or capability required to support the baselining exercise
  • Challenges due to time constraints that prohibit execution of a comprehensive baseline exercise

There could be many reasons why you are constrained and unable to execute a comprehensive baseline performance test to understand the behavior of the system as a whole. Please do note, while there are many other more approaches to measure the System Performance Baseline the one that stands out the most is the “Performance Benchmark” approach which involves injection of a production equivalent workload into the test environments. But it is not always practical to execute a System Performance Benchmark, requiring you to be innovative in your approach to obtaining a system performance baseline.

How-To Obtain A System Performance Baseline – In this section we will look at how-to put together a system performance baseline. Measuring a system Performance baseline requires that you measure some or all of the relevant performance dimensions (mentioned above e.g. Infrastructure Performance metrics, Application Performance Metrics, End User Performance, Server Side Transaction Performance, etc. ) across the existing system first and then validate the same performance metrics on the new/upgraded/migrated system. This can be accomplished in a few different ways –

  • Testing based approach – Benchmark the current system for the given production workload. Then re-validate the performance of the newly upgraded system through an extensive performance benchmark for similar workload.
  • Measurement based approach – Measure the existing system performance through a combination of performance tools and processes for a given production workload. Repeat the same on the newly upgraded system and compare performance of the two. You could perform the measurements along the following dimensions –
    • End User Response Times
    • Transaction Response Times measured at the server
    • Infrastructure Utilization Metrics
    • Application Performance Metrics
    • Database Performance Metrics
    • Middleware Performance Metrics, etc.

The first one you would agree is probably the most straight forward (in terms of approach) and sounds most reasonable as well providing a set of results a Performance Engineer or Architect can easily stand behind. However it’s also an approach that is the most expensive due to nature of resources and time required. The second approach on the other hand sounds slightly dodgy and might not give you all the confidence you need leaving you and rest of the program with some residual risk to manage.

While option 2 is not probably going to leave you with the best outcome along with the residual risk you are happy living it, it’s possibly the only practical approach you have given the constraints you are surrounded with. Keep in mind though that you want to measure the System Performance Baseline at a given Workload (i.e. when the system is being put through its paces in production) and not when the system is idle doing nothing. Option 2 is probably something you and rest of the project will have to live with due to the nature of limited resources/time/environments available and the magnitude of the task at hand. Experience suggests that it’s not always feasible to obtain the resources, funding, environments required to perform a thorough performance benchmark.

Conclusion – In this piece we’ve covered a couple of different approaches to measuring the baseline of a given system. The Measurement based approach to identifying the performance baseline of a system and the testing based approach to identifying a performance baseline for a given system. Both of the approaches have their own merits and your choice of approach will depend a lot on the nature of resources/constraints you are working with combined with the risk appetite the program is willing to live with. While the gold plated option is best it’s not always the one that will fly with your customers and program leads.

Trevor Warren (Linked In) loves hacking with open source, designing innovative solutions and building trevor_warrencommunities. Trevor is inquisitive by nature, loves asking questions and some times does get into trouble for doing so. He’s passionate about certain things in life and building solutions that have the ability to impact people’s lives in a positive manner is one of them. He believes that he can change the world and is doing the little he can to change it in his own little ways. When not hacking open source, building new products, writing content for Practical Performance Analyst, dreaming up new concepts or building castles in the air, you can catch-him bird spotting (watching planes fly above his house). You can reach trevor at –  trevor at practical performance analyst dot com. The views expressed on this web site are his own.

Related Posts

  • Use of Quantitative Models For Performance Testing & EngineeringUse of Quantitative Models For Performance Testing & Engineering Introduction : It is usually a misconception that performance-testing activities are limited to using a load-testing tool, scripting of  business scenarios, execution of the test, analysis and finally submission of test results. Many are unaware of the importance of basic Quantitative […]
  • Proactive Performance Engineering or Proactive Performance Testing – Whats the Sensible Choice ?Proactive Performance Engineering or Proactive Performance Testing – Whats the Sensible Choice ? WOPR22 is scheduled for the 21-23 of May 2014. The host or in the case of WOPR, the Content Owner, Eric Proegler, along with the WOPR Organizers, has a post at the WOPR website calling for invites to submit proposals for WOPR22. WOPR22 will be hosted by Maria Kedemo of Verisure in Malmö, […]
  • Load Tests : Setting The GoalsLoad Tests : Setting The Goals A load test either achieves a certain goal, or it dies trying.  If your computing world crumbles half way to that goal, then your performance meters should contain some good clues as to what to fix, reengineer, or upgrade before you try again. The goals for the load test are often based […]
  • Defining User Concurrency – A Battle Worth Fighting For?Defining User Concurrency – A Battle Worth Fighting For? Defining User Concurrency in context of your application is not always an easy task. In my limited experience over the last decade, going down the path of defining or in some cases re-defining something as simple as User Concurrency has started flame wars, got architects and developers […]
  • jasonk

    I guess I would only add one point to that Trevor, which is that often a baseline will be used to compare against any new changes for example new infrastructure. If your baseline is based on real production, and you are testing with a synthetic workload, the results may need a lot of interpretation.

    • You do have a point and that’s a reason why we would recommend that all workload models be built using data from production. Workload models also need to be constantly refreshed as the application functionality changes and customer access patterns change. In the end your workload models for synthetic or virtual user testing (Transaction or OLTP systems) is only as good as your understanding of the production workload that your performance testing scenarios are based on – Trevor