Use of Quantitative Models For Performance Testing & Engineering

Introduction : It is usually a misconception that performance-testing activities are limited to using a load-testing tool, scripting of  business scenarios, execution of the test, analysis and finally submission of test results. Many are unaware of the importance of basic Quantitative Analysis methodologies used in conjunction with performance testing to build performance models which could be used for the following:

  • Forecast system performance for change in business workload
  • Forecast system performance for change in system configuration
  • Identify performance constraints across the system and validate design decisions
  • Many more….

Performance testers are passionate about what they do but they might be completely oblivious to how the Performance Testing results might be used by the Performance Engineers responsible for managing the overall system performance. Performance Testers at times also assume that application of these modelling techniques t0 achieve the above outcomes is quite beyond their responsibility.  In this series of articles we will attempt to explore some of the basic performance modelling approaches and how Performance Testers might use them to increase the effectiveness of Performance Testing they perform.

When the basics of these methodologies are understood and appreciated to the core they form the real essence of performance thinking, and aid both during testing and engineering activities. Let’s ask ourselves the question though, “But what advantages do application of the above approaches provide?”. This is possibly a question that springs into the mind of many a performance tester. Let us go through some of the ways the quantitative methodologies might assist the Performance Engineering teams in the early stages of a performance-testing exercise including during requirements gathering and analysis.

Advantages:

  • Application of Performance Modelling techniques could help contribute in verifying and validating against the subjective statements defined during the requirements stage. Validating Non Functional Requirements is definitely an activity that could be performed using simple quantitative approaches.
  • Basic modelling techniques could help convert client requirements into numbers that are essential during the scripting of the business scenarios.
  • Having the numbers assessed during requirement gathering stage can help understand the expected system scalability. Thus helping the design team make the appropriate design decisions.
  • Quantitative methodologies can be used to create relevant Workloads Models which help characterize the application workload used as input into performance testing. This information could be used to design more relevant and practical performance models within the Performance Testing tool.

Let us look at an example to understand the advantages using modelling techniques:

Assuming that there are 100 users and in one hour they need to complete 500 transactions, then the pacing that would need to be applied between the iterations or loops for each simulated-user would be around 12 minutes. By incorporating this value into the run-time settings of your load test tool, the required simulation conditions can be achieved (if not necessarily the desired outcome).

  • The numbers provide simplified correlation between requirements, translation and validation.
  • The numbers provide increased confidence when implemented both at the beginning, during and after test execution, providing improved overall validation
  • The quantitative approach can be applied not only to simple systems, but provide foundations for performance-testing of more complicated systems and applications

Relevant Modelling Techniques: Some of the terms like Response Time, Utilization, Throughput, Queuing Time, Time to First Byte or Time to First Buffer, Service Time, Think Time, etc. can also be correlated by means of Quantitative Methodologies. For an introduction to the various SPE (Systems Performance Engineering) concepts and SPE Fundamentals please Click Here.

There are quite a few different modelling techniques but some of the most frequently used are described below:

  • Utilization Law – During a period of time (T), tasks arrive at a server, the server processes them, and completed tasks leave the server. An observer of the server during (T) can identify :
    • The number of tasks arriving at the server during (T), identified by (A)
    • The amount of time, during (T), that the server is busy processing tasks (B)
    • The number of tasks completed during (T), called (C). These measurements result in the following calculations :
    • The mean service time for a completed task, measured in a unit of time : S = B/C :
    • The output rate or throughput of the server, measured in tasks per unit of time : X = C/T
    • The percentage of the observation period that the server is processing tasks, called the utilization of the server : U = B/T
    • Multiplying mean service time by output rate results in the utilization of the server : B/C x C/T = B/T
    • The utilization law is : U(i) = X(i) x S(i) , where i identifies the server.

 

  • Forced Flow Law – The forced flow law captures the relationship between the different components within a system.
    • The Forced Flow law states : The throughputs or flows, in all parts of a system must be proportional to one another. The forced flow law captures the relationship between the different components within a system.
    • It states that the throughputs or flows, in all parts of a system must be proportional to one another : Xi = XVi.
    • The throughput at the i th resource is equal to the product of the throughput of the system and the visit count at that resource.
  • Response Time Theory – The Response Time model is a simple way of calculating the overall Queue Length and Queuing Time for a given system for a given set of workload conditions. The Response time Model is generally used for back of the envelope calculations and to get a sense of direction. We wouldn’t recommend using Response Time models to make investment or major architectural design decisions. For more detailed performance modelling when validating architectural decisions or investment decisions we would recommend investing in building thorough Queuing Theory models.
    • To be able to use the Response Time model at minimum you would need to know two of the three variables mentioned below:
      • Uavg – Average System Utilization
      • X – System Throughput
      • St – Service Time
    • To read more about Response Time Theory at the SPE (Systems Performance Engineering) Fundamentals Section here at Practical Performance Analyst please Click Here.
  • Little’s Law – Little’s Law basically states that the long-term average number of customers in a stable system N is equal to the long-term average effective arrival rate, λ, multiplied by the average time a customer spends in the system, W or Rt.
    • The relationship can be expressed algebraically as: N = λ * Rt  ………………..  [ N = Number of Users in the System, Rt = Response Time, λ = Arrival Rate ] –
    • To read more on Little’s Law at the SPE (Systems Performance Engineering) Fundamentals section here at Practical Performance Analyst please Click Here.

We hope you’ve enjoyed the first part of this series. Please send through any feedback to feedback at practical performance analyst dot com. In the second part of this series we’ll define each of these laws in a lot more detail and along with some examples demonstrate how they can be mapped to the requirements discussed above.


Ramesh Iyer (LinkedIn) is a Senior Performance Test Architect, works for a large US MNC and is currently based out of Bangalore. Ramesh is creative, innovative & a results driven technology leader, with 13 years of experienRamesh_Iyerce in Performance Testing, providing Test-Solutions, Performance – Project Management, Delivery Management. He is a self-motivated testing professional with expertise in driving product vision and product road mapping across various assignments keen on providing business test-solutions from ground-up. Ramesh also happens to be certified on Siebel7.5 and an ISTQB Professional.  Ramesh is a problem solver with a passion for technology; skilled in grasping the big picture, conceptualizing, translating project plans into action, developing, implementing solutions for the product & managing process setup for performance testing applications / products & partnering closely with business leaders & stakeholders within budgets and deadlines within time for deliverables.

Related Posts

  • Use of Quantitative Models For Performance Testing & Engineering – Part IIUse of Quantitative Models For Performance Testing & Engineering – Part II This article is the second in a two part series by Author Ramesh Iyer on the 101 around Use of Quantitative Models for Performance Testing & Engineering. If you haven't read the first part we encourage you to check it out at the following link - Use of Quantitative Models for […]
  • Look At The Bigger Picture : Focus On The Process Not Just The ToolLook At The Bigger Picture : Focus On The Process Not Just The Tool Introduction: With the emphasis on Digital Business and the increasing understanding of the impact of poor application performance on the business’s bottom line there has been a sudden increase in the number of vendors providing tools for various aspects of Application Performance […]
  • Modelling System Performance Using Fundamental Laws – Littles LawModelling System Performance Using Fundamental Laws – Littles Law In the first article of this series we reviewed some of the most basic Performance Modelling techniques using fundamental performance equations. The first part of this series covered a Performance Modelling technique we call "Response Time Theory" and leveraged easy to use performance […]
  • Application Of Performance Modeling Techniques Application Of Performance Modeling Techniques What is Performance Modeling – Performance Modeling is the process of modeling application performance considering various growth factors with the objective of proactively predicting potential breach of SLA’s. Performance Modeling is also used to validate design decisions and […]