Use of Quantitative Models For Performance Testing & Engineering – Part II

This article is the second in a two part series by Author Ramesh Iyer on the 101 around Use of Quantitative Models for Performance Testing & Engineering. If you haven’t read the first part we encourage you to check it out at the following link – Use of Quantitative Models for Performance Testing & Engineering : Part 1.

Picking up where we left off – In the first part of this series on Use of Quantitative Models for Performance Testing & Engineering : Part 1 we looked at how some elementary performance modelling techniques can be used to validate our assumptions for Performance Testing or Performance Engineering. We looked at the following Performance Modelling techniques:

  • Utilization Law
  • Forced Flow Law
  • Little’s Law
  • Response Time Theory

We would like to clarify that the above modelling techniques are not necessarily the only analytical modelling techniques available to Performance Engineers or Performance Testers. You have a plethora of other modelling techniques that are available and this article focuses on some of the more relevant ones based on the experiences I’ve had over the years.

Digging into detail – Let’s look at practical application of the various analytical models we’ve defined in the first part of this series i.e. Utilization model, Forced Flow law, Little’s Law and Response Time Theory.

Utilization Law : Let us consider a resource ‘r’. The utilization of this resource is defined mathematically as the product of mean service time per completion in that resource ‘Sr’ and throughput of that resource ‘Xr’ and that is given by the equation Ur = Sr × Xr.

Let’s now consider the example as to how we can implement this law.

Assume the bandwidth of a communication link is 1 Mbps and it is used to transmit 20 Kbyte packets that stream through the link at a rate of 5 packets / second. What is the utilization of the link?

Let us first understand what is given:

  • Resource of the communication link is ‘r’
  • Bandwidth which is the maximum packets of the communication link = 1000000 bits per second
  • The data transmitted by the link is 20 * 1000 byte –> 20 * 1000 * 8 bits / packet
  • Throughput of the resource Xr is 5 packets / sec

Let us now calculate the service time Sr = data transmitted / bandwidth = 20 * 1000 * 8 / 1000 * 1000 = 0.16sec / packet.

Applying the utilization law gives Ur = Sr × Xr   –>  0.16 * 5 = 0.80 or 80% as the utilization of the link.

Forced Flow Law : Let us consider the resource ‘r’,  the relation between throughput of that resource (Xr), the average number of visits (Vr) made by a request to that resource and the throughput of the system (X0) is expressed by the formulae as Xr = Vr × X0.

Consider a database system which is under constant load of transactions and assuming that all database transactions have similar resource demands and a few metrics provided in the table below:

Quantitative Models For Performance Testing & Engineering - Table

It is observed that in one hour almost 14,400 transactions are completed. Find the average number of visits for I/Os on each disk?

Mapping to what is given:

  • The resource ‘r’ maps to each disk
  • The observation period is one hour = 3600 sec
  • The number of transactions completed in that one hour = 14,400
  • The throughput of each disk is given under the column Total I/Os per second
  • The visits on each of the disks needs to be calculated

Now, calculate the throughput of the database server X0 = 14,400 / 3600 = 4 transaction per second

Therefore, referring the formulae Xr = Vr × X0   calculate V1, V2 and V3

  • V1 = X1 / Xo = 32 / 4 = 8 visits to disk 1 per database transaction;
  • V2 = X2 / Xo = 36 / 4 = 9 visits to disk 1 per database transaction;
  • V3 = X3 / Xo = 44 / 4 = 11 visits to disk 1 per database transaction.

Service Demand Law : Let us consider the resource as ‘r’ and accordingly we have the product of mean service time per completion at resource r Sr and throughput of that resource r, Xr is expressed by the formulae Ur = Dr × Xr.

A Web server is monitored for 10 minutes and its CPU is observed to be busy 80% of the monitoring period. The Web server log reveals that around 36,000 requests are processed in that monitored period. What is the CPU service demand of requests to the Web server?

From this let us understand the given values:

  • Resource is the CPU Service ‘r’
  • The monitored period is 10 minutes = 600 seconds
  • The number of requests processed = 36,000 requests

The CPU utilization is around 80% = .80

Need to calculate the CPU service demand?

  • The throughput of the webserver = 36,000 / 600 = 6 requests per second.
  • Therefore the CPU service demand Dr = Ur / Xr = .80 / 6 = 0.133 sec per request.

Little’s Law : Let us consider a closed system*. The relation between the number of customers, the arrival rate of the customer or throughput of the system and the average time spent by the customer in the system is mapped by the formulae : Nr = Ar × Tr.

The system is monitored for one hour during which 10,800 transactions were and an unknown number of jobs were executed. The average time spent by a job in the system is 10secs; find the number of jobs that were executed?

Let us find the given details:

  • Resource is computer system that is ‘r’
  • The monitoring period is 3600seconds
  • The number of transactions executed were 10,800 transactions

Need to calculate the number of Jobs executed?

  • Throughput of the system in the observed period =  10,800 / 3600 = 3 transactions per second.
  • Therefore we have Nr = Ar × Tr = 3 * 10 = 30 Jobs.

Response Time Theory : This is similar to the Little’s law but with a new variable called think time incorporated providing relationship between the average response time R, the number of clients which is either in the think state or waiting for a reply to a submitted request (M) and the system throughput (Xo) provided by R = (M / Xo) – Z.

For a system that has processing speed of 3,600 requests during one hour by an interactive computer system with 50 clients and an average think time of 10secs, calculate the average response time? Given details:

  • Observation period as 1 hour = 3600secs
  • Requests processed = 3,600 requests
  • Number of clients = 50
  • Think time = 5secs

We need to calculate throughput as 3600 / 3600 = 1 request per second.

Now applying all the values provided in the formulae R = (M / Xo) – Z we have (50 /1) – 10 = 40sec

The above laws provide the required basics to get started and shall form the foundation to understand performance-testing requirements and also to provide support during the performance-engineering phase. For the systems the numbers of users are impossible to estimate is mapped to an Open System and for system where the numbers of users are possible to estimate is mapped to a Closed System.

Application of the basics to a real world scenario : Now let us think about an e-commerce application and try to see how best these laws defined shall assist. Consider an e-commerce application; it was observed in one business scenario that the number of  transactions completed during a 1hr period is around 28,000, which is almost the same number of transactions arriving at the system during the same period. During this time the CPU of the web server was observed to be busy for almost 30 minutes. The response of the system was required to be 8secs for that business scenario with a think time of 15secs. Analyze the performance of the system?

When this type of application needs to be analyzed, start by making a note of the measured quantities (also known as operational variables) and obtain the derived quantities which based on the details discussed in the performance testing laws above: The measured quantities for the scenario above are as follows:

T is the observation period (1hour that is 3600 sec)
Bt is the busy time of the system observed in time T (30 minutes that is 1800 sec)

  • Aa is the number of arrival requests to the system in time T (28,000 requests)
  • As is the number of requests that was submitted to the system in the period T (28,000 requests)
  • Cc is the number of requests completions by the system in the observation period T (28,000 requests)
  • Co is the total number of requests completed by the system in the observation period T (28,000 requests)
  • R is the response time as 8 sec and Z is the think time as 15 secNow by applying the operational variables, let us attempt to understand the performance of the system.

Let us find the system throughput from the formulae given by:

  • Xo = Co / T
  • 28,000 / 3600 = 7.78 requests per second:
  1. Let us apply the Utilization Law to understand the utilization of the system, applying the operational variables to the formulae Us = Bt / T  –> 1800 / 3600 = .50 arriving at the result utilization is 50%
  2. Applying the Service Demand Law to understand the demand request, applying the operational variables to the formulae DSr = Us / Xo –> 0.50 / 7.78 = 0.064 sec per request. “But how do we verify the calculations are correct?” is a question that arises in the mind of many testers, so here is the alternate way of checking this applying the mean service team per completion: Sc = Bt / Cc –> 1800 / 28000 = 0.064 sec per request
  3. Applying Interactive Response Time law to calculate the number of users, and the operational variables to the formulae R = (M / Xo) – Z , we need to find M;

The equation becomes:

  • M / Xo  = R + Z
  • M = Xo (R + Z) thereby we have 7.78 (8 + 15) –>  7.78 * 23
  • M = 178.94 approximately 180 users

Thus by applying both measurable and derived variables the performance of the system is mathematically obtained, providing a set of quantitative requirements that can be validated by the load test tool during the test-scripting, test-execution and test-reporting phases.

References:

  • Computer Capacity Planning: Quantifying Performance Models  By Virgilio A.F. Almeida, Lawrence, W. Dowdy, Daniel A. Menasce May 13, 2004
  • Capacity Planning for Web Services: Metrics, Models, and Methods by Daniel A. Menasce, Virgilio A. F. Almeida, Prentice Hall, 2002
  • “Operational Laws” The Art of Computer Systems Performance Analysis, by Raj Jain Wiley, April 1991

We hope you’ve enjoyed this two part of this series. Please send through any feedback to feedback at practical performance analyst dot com.

Through this two part article we intended to introduce you to the power of Quantitative models for purposes of validating assumptions and outcomes on Performance Testing or Performance Engineering activities.


Ramesh Iyer (LinkedIn) is a Senior Performance Test Architect, works for a large US MNC and is currently based out of Bangalore. Ramesh is creative, innovative & a results driven technology leader, with 13 years of experienRamesh_Iyerce in Performance Testing, providing Test-Solutions, Performance – Project Management, Delivery Management. He is a self-motivated testing professional with expertise in driving product vision and product road mapping across various assignments keen on providing business test-solutions from ground-up. Ramesh also happens to be certified on Siebel7.5 and an ISTQB Professional.  Ramesh is a problem solver with a passion for technology; skilled in grasping the big picture, conceptualizing, translating project plans into action, developing, implementing solutions for the product & managing process setup for performance testing applications / products & partnering closely with business leaders & stakeholders within budgets and deadlines within time for deliverables.

Related Posts

  • Use of Quantitative Models For Performance Testing & EngineeringUse of Quantitative Models For Performance Testing & Engineering Introduction : It is usually a misconception that performance-testing activities are limited to using a load-testing tool, scripting of  business scenarios, execution of the test, analysis and finally submission of test results. Many are unaware of the importance of basic Quantitative […]
  • Modelling System Performance Using Fundamental Laws – Littles LawModelling System Performance Using Fundamental Laws – Littles Law In the first article of this series we reviewed some of the most basic Performance Modelling techniques using fundamental performance equations. The first part of this series covered a Performance Modelling technique we call "Response Time Theory" and leveraged easy to use performance […]
  • Applications Have A Usage Volume – Part IIApplications Have A Usage Volume – Part II About This Series - This series of articles is a how-to manual on applying performance modelling techniques without any needless discussion of internals or mathematics (with the exception of the necessary but simple Little’s formula which will be included in this article which happens to […]
  • Application Of Performance Modeling Techniques Application Of Performance Modeling Techniques What is Performance Modeling – Performance Modeling is the process of modeling application performance considering various growth factors with the objective of proactively predicting potential breach of SLA’s. Performance Modeling is also used to validate design decisions and […]