Using Performance Analytics, Forecasting & Prediction on Agile, DevOps Projects

Introduction – The move to digital over the last decade has brought about significant changes in the way most industries and business interact with their customers. The pace of change in most industries has been relatively high with the evolution of technology driving a lot of that change. Part of the enablers of this Digital paradigm has been movements like the Agile and DevOps movement. With Agile & DevOps practices now being seen as the preferred software development methodologies, we are starting to witness demand for early performance analysis & predictive performance analytics for business critical applications and applications that serve critical business workload. Systems Performance Engineering as a discipline focuses on delivering scalable, reliable and performant solutions that meet the agreed Non Functional Requirements (NFR’s). Customers embracing the Digital paradigm and embarking on some form of Agile or DevOps within their own organizations are now demanding that Systems Performance Engineers start looking at Performance Engineering through Agile and DevOps lens [For more details on Systems Performance Engineering please read through the fundamentals section at Practical Performance Analyst – http://www1.practicalperformanceanalyst.com/fundamentals-of-performance-capacity-management/].

Through the practice of performance modelling using relevant statistical modelling techniques based on historically gathered (empirical) data one can perform complex what-if analysis without needing to subject the system to comprehensive performance tests. We are not arguing against using performance testing since performance testing as an approach to validate system scalability to meet the agreed Non Functional Requirements will always have its place. However using relevant forecasting techniques to perform what-If analysis and provide the business a quick view of what the system performance would look like refining the models along the way has its advantage and can turn performance engineering into a proactive process fitting in well with the Agile and DevOps practices.

Through this paper I intend to focus on the various key aspects that one might consider when building a Predictive Performance Analytics solution using a combination of various Analytical, Regression & Simulation modelling techniques.

What Is Performance Prediction – Before we dive into further details let’s first take a look what Performance Prediction really means and its context within Systems Performance Engineering. We define Performance Prediction as the process which focuses on forecasting aspects of system performance using a combination of different modelling techniques which could include statistical modelling, analytical modelling or simulation modelling techniques. Predictive Performance analysis and the techniques it is based on can only “FORECAST”, potential system perform and provide a view of various what-if scenarios. Predictive Performance Analysis and the various modelling techniques it is based can provide insight into potential system behaviour but cannot ascertain with 100% confidence what might happen.

A performance model built using a combination of different modelling techniques i.e. analytical modelling, statistical modelling or simulation modelling, uses a combination of different building blocks to predict how performance will vary under different scenarios e.g. like varied set of load conditions, change in workload coming into the system or change in underlying compute capacity, etc. If considering an analytical model, the inputs to the model are expressed in mathematical quantities such as number of users, arrival rate, etc. Output is derived as similar mathematical quantities e.g. resulting response time, throughput, resource utilization, etc.

Performance modelling when viewed through the lens of DevOps and Agile can be used to provide visibility into various what-if scenarios without too much investment into building complex models and performance testing the system at production scale. The idea is to obtain a view quickly and then refine the models along the way potentially setting up for a larger performance test when the integrated system is available further down the line. 

Building A Performance Analytics Solution – A robust performance analytics solution using statistical modelling techniques could be built on system performance data collected throughout the software development life cycle. The data once collected can be cleaned and then used for purposes of forecasting using several modelling & prediction techniques to forecast the system performance for various changes in input workload. A good performance analytics solution should facilitate different types of data collection & storage along with the intelligence built through sound modelling & forecasting algorithms to provide realistic forecasts on system performance for various what-if scenarios.

A Predictive model should have the intelligence to prescribe an action to business allowing individuals to act upon the outcomes along with a relevant feedback system. The feedback system is as important as the modelling or forecasting technique uses since it allows for tracking the outcome, quality of the models produced and the error rate thus turning it into a Prescriptive model. For example, a predictive model can predict the peak traffic throughput of the application under test whereas a prescriptive model would go one step ahead to predict and recommend the actions that need to be taken by business. The actions could be possibly to bring down the resident time of specific layer/method or possibly to upgrade a specific hardware resource along the critical path with high service demand to meet the performance SLAs with clear data points about the expected performance improvements.

Let’s visualize the data required to build a predictive performance analytics solution. I tend to look at it as peeling of an onion. As you peel the onion you start to go deeper and look at various different aspects of system behaviour whose performance metrics would help build a strong performance model and hence a smarter predictive performance analytics solution. Some of the critical data layers to be considered include –

  • Performance modelling results & actual performance test results from controlled test environment
  • Network performance simulation results & device side performance metrics (for mobile applications)
  • Test versus Production environment capacity differences
  • Production infrastructure monitoring statistics, website end user traffic patterns
  • Web (browser) performance statistics.

When building your Performance Analytics solution, to start with, production environment monitoring statistics and website user traffic statistics data layers are essential data inputs required for purposes of performing any sensible forecasting. Both production monitoring statistics and website user statistics allow for use of various statistical modelling approach for purposes of forecasting using historical data.

data_layers_in_performance_analytics_solution

Data layers required for building a Performance Analytics Solution

There are several open source & commercial tools generally used to perform the testing & analysis at each of the below data layers. The key challenge lies in building the intelligence to parse the results produced by variety of tools & provision a tool agnostic reporting structure that can interpret the results captured by various tools across the different layers.

The Performance analytics solution with the ability to collect the data across the various layers mentioned above is now ready to be integrated with a performance forecasting solutions (Analytical, Statistical and Simulation modelling tools) to add the intelligence required to perform performance prediction analysis. In case of instances where there is large historical data available, application of statistical modelling techniques (regression modelling techniques, etc.) would provide better results, else combination of analytical, statistical techniques would yield more accurate results.

The Predictive performance analytics solution uses several additional data layers to prescribe recommendations to next steps based on predicted results. 

journey_towards_building_performance_analytics_solution

Journey Towards Building A Performance Analytics Solution

Performance Modelling and Forecasting Techniques – In this section let’s quickly touch up the various Performance Modelling and Forecasting options that a Performance Engineer has available at his or her disposal. Performance Modelling & Forecasting offers a set of techniques that can help IT or business understand how a given system might perform under different workload conditions. Performance Modelling and Forecasting also allows for various what-if analysis to be run and predictions to be made at the very early stage of a project. Modelling techniques could be broadly classified as –

  • Analytical (Queueing Network) modelling,
  • Statistical (Regression) Modelling,
  • Simulation Modelling.

Generally, analytical models are easy to use & comparatively less expensive than simulation modelling to setup. However to use either analytical or statistical modelling techniques, it’s necessary to measure performance under a specific range of load conditions. The input data is then fed into the various model parameters and assumptions made about the steady state behavior of the system allowing for what-if analysis and predictions to be performed.  

Performance Modelling and Forecasting Techniques / Analytical Modelling – Analytical modelling frequently includes the use of Queueing Theory for purposes of modelling and forecasting. Queuing Theory is the mathematical study of waiting lines or queues. It has been frequently used in various fields like telecommunications, networks, traffic management, etc. For an abstract system shown in the below figure, jobs arrive from the input source, each job brings a demand for service from the queue. When a job completes its service, it departs the queue.

T, the length of time we observed the system; A, the number of request arrivals we observed; C, the number of request completions we observed.

qn_model

A Simple Queuing System

From these measurements, the below Operational laws can be derived:

  • Utilization Law expressed as Utilization, U = X * S
  • Little’s Law expressed as Number of Jobs, N = X * R
  • Response Time Law expressed as Response Time, R = (N/X) – Z
  • Forced Flow Law expressed as Device k Throughput, Xk = Vk * X
  • Throughput Bound Law expressed as Maximum Throughput, Xmax <= 1 / Dmax

Using these relationships, different type of modelling techniques could be employed to build QN based models. For more details on the fundamentals of Analytical modelling including Queueing Theory please visit the SPE fundamentals section at Practical Performance Analyst (http://www1.practicalperformanceanalyst.com/fundamentals-of-performance-capacity-management/)

Performance Modelling and Forecasting Techniques / Statistical Modelling –  Regression modelling within statistics is the study of relationships among variables used to predict or estimate the value of one variable from known values of other variables related to it. Regression models can be built using a variety of different open and closed source (commercial) tools with quite ease. Some of the more common regression models used for purposes of what-if analysis and forecasting are –

  • Linear regression modelling
  • Exponential regression modelling
  • Logarithmic regression modelling
  • Polynomial regression modelling

Linear Regression Modelling is probably one of the most commonly used one (based on my experience) and is probably also the easiest to get up and running with. Statistical modelling using Linear Regression models tries to find a linear relationship between two variables, for example between users and CPU utilization. The general form of it is the straight line equation, Y = mx + c, where c represents y-intercept of the line & m represents the slope are known as regression coefficients.

Estimating the regression coefficients can be done through various techniques of which least squares method is popularly used.

Time Series Forecasting Models – A Time series is a sequence of observations on a variable measured over successive periods of time. The measurements may be taken at any other regular interval of time. Time Series Forecasting models do not allow for the complex what-if analysis that one might perform using the regression modelling techniques but nevertheless it allows for a quick view of where things are headed by understanding trends in the data for a given system. Three popular techniques within Time Series Forecasting models are –

Moving Average- The moving average method uses the average of the most recent k data values in the time series as the forecast for the next period. In this method, every time a new observation becomes available for the time series, the oldest observation in the equation is replaced and a new average is computed. As a result, the average will change as new observations become available.

Weighted Moving Average – This method is similar to moving average, the difference being it involves selecting a different weight for each time series data value and then computing a weighted average of the most recent k values as the forecast.                

Exponential Smoothing – Exponential smoothing is a special case of weighted moving average which uses a weighted average of past time series values but only one weight is assigned for the most recent observation. It uses a “smoothing constant” to determine how much weight to assign to the actual values.

Performance Modelling and Forecasting Techniques / Simulation Modelling – A Simulation Model is a mathematical model that calculates the impact of uncertain inputs and decisions without the cost and time investment involved in building it. It is used to understand under what load conditions, the system could fail and what loads it can withstand. Simulation modelling while not very complex to setup requires complex modelling tools which are mostly in the commercial space making Simulation Modelling a very in-frequently used modelling approach from a Performance Engineers standpoint.

Within Simulation Modelling, DES or Discrete Event Simulation technique happens to be the most popular simulation modelling technique. It models a system whose state may change only at discrete point in time. Open source happens to offer a DES modelling solution called SIMPY which could be used to build simple DES models for purposes of forecasting. 

Conclusion – Agile projects require Performance Engineers to provide the project and business a view of what the system performance would look like depending on the design decisions being made and the various coding patterns being deployed. Modelling and Forecasting techniques like the ones suggested in this article could provide both the project and business a quick view of the various scenarios depending on the change in the input workload or platform infrastructure (compute, storage etc.). Systems Performance Engineering can easily employ use of various modelling techniques in an Agile fashion to provide business and IT the answers they need with a certain level of accuracy and build on the models over a period of time refining the quality of the forecasts as the project moves along and more empirical data, integrated system characteristics are made available to the Systems Performance Engineers.

 From a Performance modelling and forecasting standpoint, it’s the business objective including typical what-if scenarios that needs to be evaluated by the Predictive Performance Analytics solution. It’s precisely the type of answers required that decides the type of data layers to be considered for building such solution and the nature of the modelling approach you might need to consider for purposes of Performance Modelling and Forecasting. Performance Prediction should not be considered as an alternative or replacement for a good end to end Performance testing. It should rather be considered as a smart approach to get a sense of direction early on in the project (early life cycle phases) or to get a feel of system performance for changes in input workload including design changes that might have large impacts on the overall system costs, performance.

References

  1. The Operational Analysis of Queueing Network Models by Peter J.Denning & Jeffrey P.Buzen
  2. Operational Laws & Mean Value Analysis by Raj Jain
  3. Cengage Learning Chapter 15 on Time Series Analysis and Forecasting
  4. Simulation, Modeling & Analysis by Law and Kelton

Author – This article was authored by Ramya Ramalinga Moorthy. Ramya Ramalinga Moorthy (LinkedIn) is a Performance Test Architect with over 13 years of experience on Performance Testing & Systems Performance EngineeRamyaring. She has great passion for learning & experimenting new things. In her current role, Ramya works closely with clients around the world providing them consulting services focused on Performance Engineering, Non Functional Requirements Modelling, Performance Modelling & Capacity Planning. Ramya is passionate about training & mentoring budding Systems Performance Engineers. Her key areas of interest includes Workload Modelling, Performance Modelling & Capacity planning. She has authored & presented several papers related to Performance Engineering in International conferences. And not the least, proud mother of two little princesses. Please additional details on the consulting services Ramya provides please see – EliteSouls.  Alternatively you can reach her at info@elitesouls.in.

Editor – This article was edited by Trevor Warren. Trevor Warren (LinkedIn) loves hacking open source, open hardware, designing innovative solutions and building trevor_warrencommunities. Trevor is inquisitive by nature, loves asking questions and some times does get into trouble for doing so. He’s passionate about certain things in life and building solutions that have the ability to impact people’s lives in a positive manner is one of them. He believes that he can change the world and is doing the little he can to change it in his own little ways. When not hacking open source, building new products, writing content for Practical Performance Analyst, dreaming up new concepts or building castles in the air, you can catch-him bird spotting (watching planes fly above his house). Practical Performance Analyst as an Open Body Of Knowledge on Systems Performance Engineering (SPE) built, maintained by Trevor with the support of his army of volunteer elves (PPA Volunteers). You can reach trevor at –  trevor at practical performance analyst dot com. The views expressed on this web site are his own and not that of his employer.

 

Related Posts

  • Monitoring, Log File Analysis & Visualization – The Open Source WayMonitoring, Log File Analysis & Visualization – The Open Source Way What Is Zabbix + ELK In A Box - Those of you who work on designing, building and supporting applications at some point might have felt the need to see what's happening across your environment, understand the nature of errors your applications are throwing or obtain a view of the […]
  • Modelling System Performance Using Fundamental Laws – Littles LawModelling System Performance Using Fundamental Laws – Littles Law In the first article of this series we reviewed some of the most basic Performance Modelling techniques using fundamental performance equations. The first part of this series covered a Performance Modelling technique we call "Response Time Theory" and leveraged easy to use performance […]
  • Use of Quantitative Models For Performance Testing & Engineering – Part IIUse of Quantitative Models For Performance Testing & Engineering – Part II This article is the second in a two part series by Author Ramesh Iyer on the 101 around Use of Quantitative Models for Performance Testing & Engineering. If you haven't read the first part we encourage you to check it out at the following link - Use of Quantitative Models for […]
  • Use of Quantitative Models For Performance Testing & EngineeringUse of Quantitative Models For Performance Testing & Engineering Introduction : It is usually a misconception that performance-testing activities are limited to using a load-testing tool, scripting of  business scenarios, execution of the test, analysis and finally submission of test results. Many are unaware of the importance of basic Quantitative […]