Diagnostics Shootout 2013 – Setting the Stage

As Performance Engineers we are always curious about what goes in within applications, what makes them tick, what makes them perform the way they do and what are the reasons why they perform the way they do.

Application Diagnostics is an activity aimed at gaining additional insight into the behavior of the application or application container with the objective of identifying any potential hotspots that might exist. Application Diagnostics is an activity that is performed to identify application, configuration, or code bottlenecks with the objective of tuning them so as to ensure that Application Business Transaction Response times are within acceptable bounds. As part of Application Diagnostics a Performance Engineer uses a set of tools to profile the application code by instrumenting the relevant modules across the application. Once instrumented the method level execution timings are captured and reflected along with all the other transactional activities within the application container.

Application Diagnostics tools are typically used across the following phases on a program:

  • Single User Performance Engineering – All through the Development Phase to profile Single User Transactions and tune them to the extent possible.
  • Performance Test – All through the Performance Test Phase. Used to identify Configuration, Code or Infrastructure bottlenecks.
  • Post Go Live / Production – Setup to sample in-frequently. Used to trouble shoot application performance in production

The lot of Diagnostics tools available off the shelf are very sophisticated in what they have to offer. However, some of the mainstream Diagnostics tools a few years ago were really clunky, had poor User Interfaces and required Rocket Scientists to install, configure and instrument. Thankfully things have got a lot easier over the last few years. A lot of emphasis has been on quick setup, ease of use, an intuitive user interface with the ability to capture and report only the relevant metrics. Let’s take a quick look at what is desired from a good Diagnostics tool:

So let’s take a quick look at the characteristics of a good Application Diagnostics solution:

  • Should be easy to install, setup and configure
  • Should be light weight with regards to performance overheads i.e. Utilization or Response times post instrumentation shouldn’t go beyond 3-5%
  • Dashboards should provide relevant performance data by summarizing the relevant information across the various tiers
  • Should provide a Business Transaction view of Performance across the various application components and tiers
  • Should have the ability to trace Business Transaction Performance across the various tiers
  • Should integrate along with other infrastructure monitoring tools or provide the ability to co-relate application performance with infrastructure performance
  • Should be easy and cost effective to maintain
  • Should provide the ability to perform detailed profiling required during the test / development phase and also turn level of tracing down to sampling for use in production environments
  • Should tie application performance with end user transactional performance and infrastructure performance providing the user a single view of application performance across the various application components and tiers

Like most things in Performance Engineering the APM (Application Performance Management) space completely lacks standards, with vendors providing a bunch of different solutions that use different naming conventions, that make products difficult to compare. To obtain a holistic view of application performance during SVT the customer is expected to invest in half a dozen different tools and buy a bunch of licenses most of which are server or CPU based. Having invested in the licenses for these half dozen or so components, the customer is also expected to invest another tidy sum on integrating the solution together to make it work. The end result is a hot-potch of tools that are too complex to use and provide so much information that the Performance Engineering team tends to overlook most of what is being provided to them using the fancy dashboards and charts.

The move to a cloud based SaaS model is happening gradually and personally I am pleasantly surprised to see the development in this space. However, we aren’t there yet and no organization at this point can completely survive off APM SaaS based solutions due to various reasons i.e. current costs of running SaaS based APM solutions for the whole enterprise, lack of support for all applications in an enterprise stack, integration with Service Management applications, etc. SaaS based APM definitely has it’s place in todays enterprise and it’s a matter of time before the SaaS based APM vendors have developed support and integration for all of the major enterprise stacks you would encounter in any large organization. We’ll delve into the benefits and challenges of APM based cloud solutions sometime in the near future.

As part of this series we will look at some of the most commonly used Application Diagnostics solutions on the market. The approach we’ve taken is:

  • Implement the tool on a sample J2EE application
  • Use to tool to instrument and profile the application performance
  • Assess the ease of implementation, setup and use
  • Assess the importance of the information provided and the ability to provide relevant information
  • Assess the overhead of implementing each of these solutions on the sample J2EE application

As part of this series we will be looking at the following tools:

We would have liked to review some of the other enterprise Application Diagnostics tools out there in the market i.e. CA Intrascope, Compuware DynaTrace, Quest Foglight, BMC Application Diagnostics, etc. tools but none of these vendors offer evaluation copies of their tools as a result we are unable to include them as part of this shootout. With regards to the format, each of the forthcoming articles will include a Video Based Tutorial (VBT) that will include setup, configuration, instrumentation and use of the tool from a Performance Engineers standpoint. Each of these tools is complex and has tons of features to offer. For purposes of this shootout we’ve limited ourselves to evaluation them on the points mentioned above. The objective of the shootout isn’t to rate tools or suggest that one is better than the other. It’s rather to assess the current state of the market, take a look at what the different vendors have to offer, review the different tools  from an end users perspective and provide honest feedback to help other Performance Engineers out there make an informed judgement on the pros and cons of using each of them.

This series will include the following articles –

  • Diagnostics Shootout 2013, Part 1 – AppDynamics Lite
  • Diagnostics Shootout 2013, Part 2 – New Relic SaaS
  • Diagnostics Shootout 2013 Part 3 – Visual VM
  • Diagnostics Shootout 2013 Part 4 – YourKit
  • Diagnostics Shootout 2013 Part 5 – HP Diagnostics

If you would like to see other tools covered, or would like other dimensions covered in future shootout episodes, or have ideas for different shootout episodes please write to us and send us your thoughts, inputs and comments.

As always do Send us an email with your input, comments, feedback and suggestions. If you think you’ve got the the talent, are keen on sharing your knowledge / experiences and are keen to help us grow the community here at Practical Performance Analyst please reach out to us Over email.

Related Posts

  • Diagnostics Shootout 2013 – AppDynamics LiteDiagnostics Shootout 2013 – AppDynamics Lite Welcome to the Diagnostics Shootout 2013 at Practical Performance Analyst. Application Diagnostics is an activity aimed at gaining insight into the behaviour of the application or application container (J2EE or .Net) with the objective of identifying any potential hotspots. One […]
  • Diagnostics Shootout 2013 – New RelicDiagnostics Shootout 2013 – New Relic Welcome to the Diagnostics Shootout 2013 at Practical Performance Analyst. In this episode we’ll take a look at New Relic and what it has to offer to the Practical Performance Analyst. New Relic unlike most other Application Diagnostics tools is in class of its own and I say this with […]
  • Diagnostics Shootout 2013 – AppNeta TraceViewDiagnostics Shootout 2013 – AppNeta TraceView Welcome to the Diagnostics Shootout 2013 at Practical Performance Analyst. There’s been a slight change with regards to selection of tools for the Diagnostics shootout here at Practical Performance Analyst. We’ve decided to include AppNeta Traceview in our Diagnostics Shootout for […]
  • Integrated Dashboards, Correlated metrics and Monitoring UtopiaIntegrated Dashboards, Correlated metrics and Monitoring Utopia How it all started - It all started with a conversation a day ago with a colleague about the need to monitor system performance for a complex multi-tiered financial application. The colleague wanted to know what could be done to help the customer manage his application performance […]