This article is the second in a two part series by Author Ramesh Iyer on the 101 around Use of Quantitative Models for Performance Testing & Engineering. If you haven’t read the first part we encourage you to check it out at the following link – Use of Quantitative Models for Performance Testing & Engineering : Part 1.
Picking up where we left off – In the first part of this series on Use of Quantitative Models for Performance Testing & Engineering : Part 1 we looked at how some elementary performance modelling techniques can be used to validate our assumptions for Performance Testing or Performance Engineering. We looked at the following Performance Modelling techniques:
- Utilization Law
- Forced Flow Law
- Little’s Law
- Response Time Theory
We would like to clarify that the above modelling techniques are not necessarily the only analytical modelling techniques available to Performance Engineers or Performance Testers. You have a plethora of other modelling techniques that are available and this article focuses on some of the more relevant ones based on the experiences I’ve had over the years.
Digging into detail – Let’s look at practical application of the various analytical models we’ve defined in the first part of this series i.e. Utilization model, Forced Flow law, Little’s Law and Response Time Theory.
Utilization Law : Let us consider a resource ‘r’. The utilization of this resource is defined mathematically as the product of mean service time per completion in that resource ‘Sr’ and throughput of that resource ‘Xr’ and that is given by the equation Ur = Sr × Xr.
Let’s now consider the example as to how we can implement this law.
Assume the bandwidth of a communication link is 1 Mbps and it is used to transmit 20 Kbyte packets that stream through the link at a rate of 5 packets / second. What is the utilization of the link?
Let us first understand what is given:
- Resource of the communication link is ‘r’
- Bandwidth which is the maximum packets of the communication link = 1000000 bits per second
- The data transmitted by the link is 20 * 1000 byte –> 20 * 1000 * 8 bits / packet
- Throughput of the resource Xr is 5 packets / sec
Let us now calculate the service time Sr = data transmitted / bandwidth = 20 * 1000 * 8 / 1000 * 1000 = 0.16sec / packet.
Applying the utilization law gives Ur = Sr × Xr –> 0.16 * 5 = 0.80 or 80% as the utilization of the link.
Forced Flow Law : Let us consider the resource ‘r’, the relation between throughput of that resource (Xr), the average number of visits (Vr) made by a request to that resource and the throughput of the system (X0) is expressed by the formulae as Xr = Vr × X0.
Consider a database system which is under constant load of transactions and assuming that all database transactions have similar resource demands and a few metrics provided in the table below:
It is observed that in one hour almost 14,400 transactions are completed. Find the average number of visits for I/Os on each disk?
Mapping to what is given:
- The resource ‘r’ maps to each disk
- The observation period is one hour = 3600 sec
- The number of transactions completed in that one hour = 14,400
- The throughput of each disk is given under the column Total I/Os per second
- The visits on each of the disks needs to be calculated
Now, calculate the throughput of the database server X0 = 14,400 / 3600 = 4 transaction per second
Therefore, referring the formulae Xr = Vr × X0 calculate V1, V2 and V3
- V1 = X1 / Xo = 32 / 4 = 8 visits to disk 1 per database transaction;
- V2 = X2 / Xo = 36 / 4 = 9 visits to disk 1 per database transaction;
- V3 = X3 / Xo = 44 / 4 = 11 visits to disk 1 per database transaction.
Service Demand Law : Let us consider the resource as ‘r’ and accordingly we have the product of mean service time per completion at resource r Sr and throughput of that resource r, Xr is expressed by the formulae Ur = Dr × Xr.
A Web server is monitored for 10 minutes and its CPU is observed to be busy 80% of the monitoring period. The Web server log reveals that around 36,000 requests are processed in that monitored period. What is the CPU service demand of requests to the Web server?
From this let us understand the given values:
- Resource is the CPU Service ‘r’
- The monitored period is 10 minutes = 600 seconds
- The number of requests processed = 36,000 requests
The CPU utilization is around 80% = .80
Need to calculate the CPU service demand?
- The throughput of the webserver = 36,000 / 600 = 6 requests per second.
- Therefore the CPU service demand Dr = Ur / Xr = .80 / 6 = 0.133 sec per request.
Little’s Law : Let us consider a closed system*. The relation between the number of customers, the arrival rate of the customer or throughput of the system and the average time spent by the customer in the system is mapped by the formulae : Nr = Ar × Tr.
The system is monitored for one hour during which 10,800 transactions were and an unknown number of jobs were executed. The average time spent by a job in the system is 10secs; find the number of jobs that were executed?
Let us find the given details:
- Resource is computer system that is ‘r’
- The monitoring period is 3600seconds
- The number of transactions executed were 10,800 transactions
Need to calculate the number of Jobs executed?
- Throughput of the system in the observed period = 10,800 / 3600 = 3 transactions per second.
- Therefore we have Nr = Ar × Tr = 3 * 10 = 30 Jobs.
Response Time Theory : This is similar to the Little’s law but with a new variable called think time incorporated providing relationship between the average response time R, the number of clients which is either in the think state or waiting for a reply to a submitted request (M) and the system throughput (Xo) provided by R = (M / Xo) – Z.
For a system that has processing speed of 3,600 requests during one hour by an interactive computer system with 50 clients and an average think time of 10secs, calculate the average response time? Given details:
- Observation period as 1 hour = 3600secs
- Requests processed = 3,600 requests
- Number of clients = 50
- Think time = 5secs
We need to calculate throughput as 3600 / 3600 = 1 request per second.
Now applying all the values provided in the formulae R = (M / Xo) – Z we have (50 /1) – 10 = 40sec
The above laws provide the required basics to get started and shall form the foundation to understand performance-testing requirements and also to provide support during the performance-engineering phase. For the systems the numbers of users are impossible to estimate is mapped to an Open System and for system where the numbers of users are possible to estimate is mapped to a Closed System.
Application of the basics to a real world scenario : Now let us think about an e-commerce application and try to see how best these laws defined shall assist. Consider an e-commerce application; it was observed in one business scenario that the number of transactions completed during a 1hr period is around 28,000, which is almost the same number of transactions arriving at the system during the same period. During this time the CPU of the web server was observed to be busy for almost 30 minutes. The response of the system was required to be 8secs for that business scenario with a think time of 15secs. Analyze the performance of the system?
When this type of application needs to be analyzed, start by making a note of the measured quantities (also known as operational variables) and obtain the derived quantities which based on the details discussed in the performance testing laws above: The measured quantities for the scenario above are as follows:
T is the observation period (1hour that is 3600 sec)
Bt is the busy time of the system observed in time T (30 minutes that is 1800 sec)
- Aa is the number of arrival requests to the system in time T (28,000 requests)
- As is the number of requests that was submitted to the system in the period T (28,000 requests)
- Cc is the number of requests completions by the system in the observation period T (28,000 requests)
- Co is the total number of requests completed by the system in the observation period T (28,000 requests)
- R is the response time as 8 sec and Z is the think time as 15 secNow by applying the operational variables, let us attempt to understand the performance of the system.
Let us find the system throughput from the formulae given by:
- Xo = Co / T
- 28,000 / 3600 = 7.78 requests per second:
- Let us apply the Utilization Law to understand the utilization of the system, applying the operational variables to the formulae Us = Bt / T –> 1800 / 3600 = .50 arriving at the result utilization is 50%
- Applying the Service Demand Law to understand the demand request, applying the operational variables to the formulae DSr = Us / Xo –> 0.50 / 7.78 = 0.064 sec per request. “But how do we verify the calculations are correct?” is a question that arises in the mind of many testers, so here is the alternate way of checking this applying the mean service team per completion: Sc = Bt / Cc –> 1800 / 28000 = 0.064 sec per request
- Applying Interactive Response Time law to calculate the number of users, and the operational variables to the formulae R = (M / Xo) – Z , we need to find M;
The equation becomes:
- M / Xo = R + Z
- M = Xo (R + Z) thereby we have 7.78 (8 + 15) –> 7.78 * 23
- M = 178.94 approximately 180 users
Thus by applying both measurable and derived variables the performance of the system is mathematically obtained, providing a set of quantitative requirements that can be validated by the load test tool during the test-scripting, test-execution and test-reporting phases.
- Computer Capacity Planning: Quantifying Performance Models By Virgilio A.F. Almeida, Lawrence, W. Dowdy, Daniel A. Menasce May 13, 2004
- Capacity Planning for Web Services: Metrics, Models, and Methods by Daniel A. Menasce, Virgilio A. F. Almeida, Prentice Hall, 2002
- “Operational Laws” The Art of Computer Systems Performance Analysis, by Raj Jain Wiley, April 1991
We hope you’ve enjoyed this two part of this series. Please send through any feedback to feedback at practical performance analyst dot com.
Through this two part article we intended to introduce you to the power of Quantitative models for purposes of validating assumptions and outcomes on Performance Testing or Performance Engineering activities.