Proactive Performance Management Through Agile Performance Test Practices

Just in case you haven’t heard, Waterfall is out and Agile is in. For organizations that thrive on innovation, successful agile development and continuous deployment processes are paramount to reducing go to market time, fast tracking product enhancements and quickly resolving defects.

Executed successfully, with the right team in place, Agile practices should result in higher functional product quality. Operating in small, focused teams that work well-defined sprints with clearly groomed stories is ideal for early QA involvement, parallel test planning and execution.

But how do you manage non-functional performance quality in an Agile model? The reality is that traditional performance engineering, and testing, is often best performed over longer periods of time; workload characterizations, capacity planning, script development, test user creation, test data development, multi-day soak tests and more… are not always easily adaptable into 2-week, or shorter, sprints. And the high-velocity of development change often cause continuous, and sometimes large, ripples that disrupt a team’s ability to keep up with these activities; anyone ever had a data model change break their test dataset?

Before joining AppDynamics I faced this exact scenario as the Lead Performance Engineer for PayPal’s Java Middleware team. PayPal was undergoing an Agile transformation and our small team of historically matrix aligned, specialty engineers, was challenged to adapt.

 At AppDynamics, our Application Intelligence Platform helps today’s software-defined businesses proactively monitor, manage and optimize the most complex software environments. All in real time, and all in production. With cloud and on-premise deployment flexibility, AppDynamics works and partners with many of the world’s most innovative companies. Customers include Citrix, DIRECTV, Edmunds, Expedia, Fox News, John Deere, OpenTable,, Sephora, StubHub and Union Pacific Railroad.   AppDynamics premier event for today’s software-defined business AppSphere 2014, takes place November 3-5 in Las Vegas.

 For more information and to try instant APM nirvana, visit

Here are my best practices and lessons learned, sometimes the hard way, of how to adapt performance-engineering practices into an agile development model:

  • Fully integrate yourself into the Sprint team, immediately –  My first big success at PayPal was the day I had my desk moved to sit in the middle of the Dev team. I joined the water cooler talk, attended every standup, shot nerf missiles across the room, wrote and groomed stories as a core part of the scrum team. Performance awareness, practices, and results organically increased because it was a well represented function within the team as opposed to an after thought farmed out to a remote organization.
  • Build multiple performance and stress test scenarios with distinct goals and execution schedules. Plan for longer soak and stress tests as part of the release process, but have one or more per-sprint, and even nightly, performance tests that can be continually executed to proactively measure performance, and identify defects as they are introduced. Consider it your mission to quantify the performance impact of a code change.
  • Extend your Continuous Integration (CI) pipelines to include performance testing – At PayPal, I built custom integrations between Jenkins and JMeter to automate test execution and report generation. Our pipelines triggered automated nightly regressions on development branches and within a well-understood platform where QA and development could parameterize workload, kick-off a performance test and interpret a test report. Unless you like working 18-hour days, I can’t overstate the importance of building integrations into tools that are already or easily adopted by the broader team. If you’re using Jenkins, you might take a look at the Jenkins Performance Plugin.
  • Define Key Performance Indicators (KPIs) –  In an Agile model you should expect smaller scoped tests, executed at a higher frequency. It’s critical to have a set of KPIs the group understands, and buys into, so you can quickly look at a test and interpret if a) things look good, or b) something funky happened and additional investigation is needed. Some organizations have clearly defined non-functional criteria, or SLAs, and many don’t. Be Agile with your KPIs, and refine them over time. Here are some of the KPIs we commonly evaluated:
    • Percentile Response Time – 90th, 95th, 99th – Summary and Per-Transaction
    • Throughput – Summary and Per-Transaction
    • Garbage Collector (GC) Performance – % non-paused time, number of collections (major and minor), and collection times.
    • Heap Utilization – Young Generation and Tenured Space
    • Resource Pools – Connection Pools and Thread Pools
  • Invest in best of breed tooling – With higher velocity code change and release schedules, it’s essential to have deep visibility into your performance environment. Embrace tooling, but consider these factors impacted by Agile development: Can your toolset automatically, and continuously discover, map and diagnose failures in a distributed system without asking you to configure what methods should be monitored? In an Agile team the code base is constantly shifting. If you have to configure method-level monitoring, you’ll spend significant time maintaining tooling, rather than solving problems.

Can the solution be enabled out of the box under heavy loads? If the overhead of your tooling degrades performance under high loads, it’s ineffective in a performance environment. Don’t let your performance monitoring become your performance problem.
When a vendor recommends you reduce monitoring coverage to support load testing, consider:

  • The effectiveness of a tool which won’t provide 100% visibility, and
  • How much time will be spent consistently reconfiguring monitoring for optimal overhead.

Performance testing within an Agile organization challenges us as engineers to adapt to a high velocity of change. Applying best practices gives us the opportunity to work as part of the development team to proactively identify and diagnose performance defects as code changes are introduced. Because the fastest way to resolve a defect in production is to fix it before it gets there.

Steve Sturtevant is a software and performance engineer with close to 15-years of experience in the industry. He has held product manager roles in the APM space, and led performance engineering for PayPal’s middleware platforms. Currently Steve works on the Sales Engineering team at AppDynamics.

This article was first published at AppDynamics and has been reproduced with prior permission at Practical Performance Analyst.

Related Posts

  • Does Devops really need a friendDoes Devops really need a friend Adoption APM when Performance Matters As enterprises embrace the DevOps philosophy, and the coalescence of the Development and Operations continues, I foresee the conditions ripening to foster innovative methods of making application performance better and code deployments smoother. […]
  • DevOps Hiring By Dave Zwieback – Free Ebook by OreillyDevOps Hiring By Dave Zwieback – Free Ebook by Oreilly If your organization has embraced DevOps, you need people whose nonlinear career paths and wide-ranging interests will help you remove dysfunctional silos. But your efforts to hire DevOps practitioners aren't working. How do you unearth these DevOps creatures? Think like one. In this […]
  • Application Aware–Infrastructure Performance Management – Virtualization PracticeApplication Aware–Infrastructure Performance Management – Virtualization Practice For years, Gartner has insisted that if an APM tool does not cover each of its “Five Dimensions of APM,” one of which is deep code analysis, then it is not an APM tool. Gartner has therefore defined APM to be relevant only to custom-developed applications. Well, it has finally woken up […]
  • Driving a Simple Performance BaselineDriving a Simple Performance Baseline Adopting an Application Performance Management (APM) strategy will help you manage the quality of the customer experience. The challenge is that APM has evolved into a mosaic of monitoring tools, analytic engines, and event processors that provide many solutions to different problem […]