Dr. Leonid Grinshpan (LinkedIn) is currently Technical Director at Oracle with a focus on Enterprise applications capacity planning, load testing, modelling, performance analysis, and tuning. Leonid has a few decades of experience on two complementing areas: computer science and information technology (IT) engineering. He holds a Ph.D in computer science is is also an author of a book on mathematical modelling.
In this series of articles Dr. Leonid Grinshpan (LinkedIn) presents an approach that Performance Architects or Performance Engineers should consider when attacking Performance Engineering related activities across the SDLC. Dr. Leonid Grinshpan (LinkedIn) has decades of experience building and delivering applications that perform. Through this series he sums up some of the learning that might prevent you from making the same mistakes when addressing Systems Performance across the SDLC.
When we have to fix a car engine, the first thing we do is learning how the engine works. But when we plan to tune an application, knowledge of its functions and internals sometimes is frequently considered an unnecessary burden. If acquainting oneself with the application functionality is not included into “must to do” list of project activities, than with 100% certainty the project will be ruined by what we term as the, “Application unawareness wreaking ball”.
To avoid a fiasco, here’s a list of what we think a Performance Engineer should know about the application before they go down the path of diagnostics, optimization and tuning.
Application functionality – Here’s a list of questions you might consider asking:
- What was the application built to accomplish?
- What is the functionality that the application delivers?
- What tasks does it help automate?
- What are the user’s categories if any (for example, planners, analysts, reviewers, C-level executives, etc.)?
- How do the different user types use the application?
- Does application process the batch jobs?
- Does it do it concurrently with the users’ interactive requests?
Knowledge of application functionality and usage enables the Performance Engineer or Performance Architect to identify realistic workload that would have to be generated by load testing tools.
Application components and their deployment – Today’s enterprise applications are no longer the large monolith beasts but rather comprised of a number of software components hosted on different pieces of infrastructure hosted internally and externally i.e. cloud. The most widespread of application components are Web servers, Application servers, and Databases. Depending on functionality, an application can include reporting, consolidation, integration, printing, data transformation, and many other components processing users’ requests. The components have tuning parameters that makes them adaptable to varying service demands and workload types. The examples of tuning parameters for an Enterprise Java Application are:
- JVM or Java Virtual Machine minimum heap size
- JVM maximum heap sizes
- JVM number of software threads supporting concurrent processing by a server
- JVM number of database connections, etc.
Understanding an application’s component composition as well as each components tuning parameter aids identification of bottlenecks. It not only speeds up the process of diagnostics and but also that of optimization.
Knowledge of Transactions – The users communicate with an application by initiating the transactions that require allocation of the system resources when they are processed. The intensity of the user transactions at any given time depends on the number of users actively interacting with a system and the weight of those transactions in terms of compute requirement. It also is a derivative of the pace each user submits one transaction after another.
In order to generate a test workload that closely emulates real business workload we have to identify the following transaction characteristics:
- List of transactions (user generated and operational)
- For each transaction, its intensity expressed in the average number of times a transaction was initiated per one hour by single user or single device
- For each transaction, the number of users or devices requesting it concurrently
Understanding of the transactional workload is essential to working out what elements of the system need focus and tuning of what components of the system will provide the most benefit.
Transaction profiles – Transaction profile is a measure of single transaction demand for system resources. A transaction triggers a multitude of processing activities through code executed by the various application components. Each component allocates its resources for transaction processing for a particular time interval. In general, each component has the following assets to be allocated:
- CPU time (data processing)
- I/O time (data transfer)
- Network time (data transfer)
- Software connections to the servers and services (for example, Web server connections, database connections)
- OS or JVM threads
- Storage space
- Memory space
Active resources implement transaction processing and data transfer. Passive resources provide access to active resources. A consumption of an active resource is measured in the time interval it was serving a transaction. A metric for a passive resource usage depends on passive resource type: for software connections and threads it is a number of connections / threads; for memory and storage it is a size of allocated memory. A transaction profile is a set of numbers (vector) specifying quantity of each resource consumed by transaction during its processing by hardware components.
Knowledge of transaction profile is important to identify a suitable approach for deployment of performance monitors. Performance monitors ensure that they record meaningful performance counters capable to identify shortage of system resources under load.
Application processes – For operating system (OS), an application represents a set of processes working under OS control and receiving from OS resources needed to satisfy demand from user transactions. We have to know the processes of our application in order to monitor system resources allocated to each process. Windows Task Manager provides basic information on each process behaviour:
Windows performance monitor delivers broad range of performance counters for each process. This is an example of performance counter readings for FrameworkService process:
In UNIX universe depending on the UNIX flavor the process performance counters might look like below:
Monitoring application and infrastructure performance counters helps identify potential breach in SLA’s. However to identify process level performance issues more detailed application level instrumentation is required.
Application logs and error files – Application logs and error files are the gold mines – they keep valuable information on executed transactions including, but not limited to, the times transaction was processed by different application components. On some occasions such information let quickly identify a component that is causing a bottleneck.
Information in log and error files helps to check if a workload manufactured for load tests is up to the test requirements.
Dr. Leonid Grinshpan (LinkedIn) is currently Technical Director at Oracle with a focus on Enterprise applications capacity planning, load testing, modelling, performance analysis, and tuning. Leonid has a few decades of experience on two complementing areas: computer science and information technology (IT) engineering. He holds a Ph.D in computer science is is also an author of a book on mathematical modelling. Leonid has worked on over 200 capacity planning and performance tuning projects for Fortune 500 customers over the past eleven years. Leonid is also the recipient of the highest award in the USSR for excellence in IT engineering – Award of the Counsel of Ministers of the USSR for design and implementation of an open CAD system for microprocessor based products. He tested out his entrepreneurial skills by co-founding the first Belarusian-American Joint Venture Software Security Belarus (acquired in 1997 by USA company Rainbow Technologies).