What Do I Do, My System Is Running Like A Dog?

A long time ago, I presented on a topic “Measuring Response Times of Code on Oracle Systems.” The presentation was at a CMG (www.cmg.org) conference and paper for this presentation was a subset of “For Developers: Making Friends with the Oracle Database.” In the presentation, I spent a few minutes talking about why to measure response times in Oracle, and then I spent a lot of minutes talking about how. As usual, I focused heavily on the importance of measuring response times of individual business tasks executed by individual end users.

At the end of the talk, a group of people came to the podium to ask questions (always a good sign). The first question was the question that a lot of people ask. It was:

My whole system is slow. That’s all my users will tell me. So then, how do I begin to do what you’re describing?

Here’s the answer:

Ask your users to show you what they’re doing. Just go look at it.

The results of this simple advice are routinely spectacular. Just go look at it: I’m surprised whenever someone doesn’t think of doing that, but I shouldn’t be. That’s because I didn’t do it either, for the longest time. I had to learn to. And that’s the story I want to tell you here.

In the early 1990s, I was a consultant with Oracle Corporation visiting clients with performance problems at a pace of more than 30 per year. Back then, I did Oracle performance work the old fashioned way: I checked everything I knew how to check, and then I fixed everything I knew how to fix. All billable by the hour. (Note: When I was doing it this way, I had not yet been taught by Dave Ensor, who changed me forever.)

On weeks when I was lucky, I’d be finished checking and fixing by sometime Wednesday, leaving a couple of days to find out what people thought of my work. If I were lucky again (that’s two “lucky”s now), everyone would be thrilled with the results. I’d get my hug (so to speak), and I’d catch my flight.

But I wasn’t always lucky. Some weeks, I wouldn’t find anything suspicious in my checking and fixing. Some weeks, I’d find plenty, but still not everyone would be thrilled with the work. Having people be less than thrilled with my work caused pain for me, which motivated me to figure out how to take more control of my consulting engagements, to drive luck out of the equation.

The most important thing I figured out was that; People knew before I came on-site how they were going to measure on Thursday whether they liked the results of my work and…

They were willing to tell me on Monday. All I had to do was be honest, like this:

On the day I’m done working here, I’d imagine you’re going to want to run something that will demonstrate whether I accomplished what you were hoping for while I was here. Would you mind telling me about that now? Maybe even showing me?

I could ask that on Monday, and people were glad to tell me. I’d watch the things run and record how long they ran, and then I’d know how to prioritize my time on site. I’d record how long they ran so at the end of my engagement, I’d be able to show very clearly what improvements I had made.

Sometimes, there would be thirty different things that people would expect to measure on Thursday. If I might not have time to fix them all, then I needed to make sure that I knew the priority of the things I was being asked to fix.

That one step alone—knowing on Monday that prioritized list of what tasks needed to be fast by Thursday—drastically reduced my reliance on luck as a success factor in my job at these sites. Knowing that list on Monday is just like when your teacher in school tells you exactly what’s going to be on your next test. It allows you to focus your attention on exactly what you need to do to optimize your reward for the week. (Note to fellow education enthusiasts: Please don’t interpret this paragraph as my advocating the idea that GPA should be a student’s sole—or even dominant—optimization constraint.)

So, what I learned is that the very first step of any good performance optimization method is necessarily this:

1. Identify the task that’s the most important to you – When I say “task,” think “program” or “click” or “batch job” if you want to. What I mean is “a useful unit of work that makes sense to the business.” …Something that a business user would show you if you just went and watched her work for a few minutes.

Then comes step two:

2. Measure its response time (R). In detail – Why is response time so important? Because that’s what’s important to the person who’ll be watching it run on Thursday, assessing whether she thinks you’ve done a good job or not. That person’s going to click and then wait. Happiness will be inversely proportional to how long the wait is. That’s it. That’s what “performance” means at 99% of sites I’ve ever visited.

Measuring response time is vital. You must be able to measure response time if you’re going to nail that test on Thursday.

The key is to understand that the term response time doesn’t even have a definition except in the context of a task. You can’t measure response time if you don’t first decide what task you’re going to measure. In other words, you cannot do step 2 before you do step 1. With Oracle, for example, you can collect ASH data (if you’re licensed to use it) or even trace data for a whole bunch of Oracle processes, but you won’t have a single response time until you define which tasks buried within that data are the ones you want to extract and pay attention to.

You get that by visiting a user and watching what she does.

There are lots of excuses for not watching your users. Like these…

“I don’t know my users.” I know. But you should. You’d do your job better if you did. And your users would, too.

“My users aren’t here.” I know. They’re on the web. They’re in Chicago and Singapore and Istanbul, buying plane tickets or baseball caps or stock shares. But if you can’t watch at least a simulation of the things those users actually do with the system you help manage, then I can’t imagine how you would possibly succeed at providing good performance to them.

“I’m supposed to be able to manage performance with my dashboard.” I know. I was supposed to have a hover car by the year 2000.

The longer you stay mired in excuses like these, the longer it’s going to be before you can get the benefit of my point here. Your users are running something, and whatever that is that they’re running is your version of my Thursday test. You can check and fix all you want, but unless you get lucky and fix the exact tooth that’s hurting, your efforts aren’t going to be perceived as “helpful.” Checking and fixing everything you can think of is far less efficient and effective than targeting exactly what your user needs you to target.

Lots of performance analysts (DBAs, developers, architects, sysadmins, and so on) assume that when someone says, “The whole system is slow,” it means there must be a single parameter somewhere in the bowels of the system that needs adjustment, and if you can just make that adjustment, everything is going to be ok. It might mean that, but in my experience, the overwhelming majority of cases are not that way. (Pages 25–29 of Optimizing Oracle Performance has more information about this.)

The great thing about measuring response time is that no matter what the problem is, you’ll see it. If the program you’re watching is poorly written, you’ll see it. If some other program is hogging too much of a resource that your program needs, you’ll see it. If you have a bad disk controller, you’ll see it. If some parameter needs adjusting, you’ll see it.

Realize that when a business user says “system,” she doesn’t mean what you would mean if you said “system.” She means that the thing she runs is slow. Look at that thing. Maybe there are seventeen of them. And sure, maybe all seventeen suffer from the same root cause. If that’s the case, then fantastic, because fixing the first problem will magically fix the other sixteen, too. If it’s not, then fantastic anyway, because now all of them are on your prioritized list of tasks to optimize, and you’ll probably surprise yourself how quickly you’ll be able to pick them all off when you focus on one task at a time.


Cary Millsap (LinkedIn) is an entrepreneur, teacher, software technology advisor, software developer, writer, and Oracle Cary_Millsapsoftware performance specialist. His technical work is quoted in many Oracle books, in Wikipedia, in blogs all over the world, and in numerous conference presentations each month. He has presented at hundreds of public and private events around the world, and his blog is read by thousands of people each month. He is published in journals including Communications of the ACM. He wrote the book Optimizing Oracle Performance (O’Reilly 2003), for which he and co-author Jeff Holt were named Oracle Magazine’s 2004 Authors of the Year and The Method R Guide to Mastering Oracle Trace Data . He is a performance specialist at Enkitec LP, a firm that specializes in Oracle engineered systems performance. He is the owner and CEO of Method R Corporation, a company that provides software tools and education services to firms including Fortune 100 companies.

Related Posts

  • Building Systems That Perform : Looking For A Needle In A HaystackBuilding Systems That Perform : Looking For A Needle In A Haystack Dr. Leonid Grinshpan (LinkedIn) is currently Technical Director at Oracle with a focus on Enterprise applications capacity planning, load testing, modelling, performance analysis, and tuning. Leonid has a few decades of experience on two complementing areas: computer science and […]
  • Building Systems That Perform : Application Awareness Building Systems That Perform : Application Awareness Dr. Leonid Grinshpan (LinkedIn) is currently Technical Director at Oracle with a focus on Enterprise applications capacity planning, load testing, modelling, performance analysis, and tuning. Leonid has a few decades of experience on two complementing areas: computer science and […]
  • Building Systems That Perform : Workload Characterization – Doing It RightBuilding Systems That Perform : Workload Characterization – Doing It Right Dr. Leonid Grinshpan (LinkedIn) is currently Technical Director at Oracle with a focus on Enterprise applications capacity planning, load testing, modelling, performance analysis, and tuning. Leonid has a few decades of experience on two complementing areas: computer science and […]
  • A Step by Step Guide to High Performing Websites and Web Applications "Optimizing Digital Experience - A Step by Step Guide to High Performing Websites and Applications " is your practical learning guide for how to deliver great digital experiences to your end users. Get the essential information you need to reduce user wait times for better audience […]