Load Tests: Failures To Avoid and Opportunities to Seize

In no particular order, here are a few final things that have worked well for me and some lessons I’ve learned while load testing.  Examine each one. Think about them in the context of your organization, your personality, your duties, and the daily challenges you face.  Use what works and discard the rest.

Load Testing Is A Team Sport – Load testing requires the help and cooperation of many people. This is usually extra work done at very inconvenient times. Those people can be ordered to cooperate, but they give you so much more if they understand what’s going on and get teamtreated with a little respect. Here are a few good things to do:

  • Ask for their help.
  • Give plenty of advance notice.
  • Explain what you are doing with the test and inquire about any unforeseen difficulties the test might create, or opportunities the test might offer if a small change is made.
  • Bring food.
  • Share your results after the test.
  • Say ”Thank You” and mean it.
  • Pay it forward by always being helpful to others.

Reality v/s The Lab – It’s vastly easier to test the application in its native environment than it is to recreate and test a copy of it in a lab on rented equipment. If you have to go into the lab, expect many delays as you discover just how fussy a computer program can be about the computer it runs on, its surroundings, the placement of files, and network connectivity.

Lab based load test failures I’ve seen more than once – The team showed up with almost everything they needed. Pack carefully for the trip. The team showed up with unfinished or poorly tested software. Don’t plan to fix it in the lab.

The team assumed that key equipment would be there. They assumed the right version of software and hardware would not only be available, but already be installed for them. They assumed an expert would be available at a moment’s notice. Don’t assume, check.lab

The team finally ran the load test successfully after several hardware and configuration changes. Sadly, no one recorded key details of the configuration that finally worked, so the results were meaningless.  Take careful notes and make load tests as self-documenting as possible.

The team did not plan how they would reset the system quickly between tests, so they spent 75% of their lab time resetting for the next run. Plan for multiple runs and do what you can to speed up test resets.

Use Checklists – The smallest forgotten detail can ruin a perfectly good load test. Have a checklist and use it. Checklists work for NASA, for surgeons, for pilots, and for load testers.

Ironically, all checklists are incomplete. When you find something missing from the list, take the time to update the master copy of the list.

Communicate During The Test – A full-power peak load test has the potential to be a disruptive event. Before the test starts, open a voice bridge so that all interested parties can follow what’s going on and report troubles quickly.  Check in with everyone periodically at key points. For example:

  • Before testing starts ask everyone: “Are you ready?”
  • When test is first running at low power: “Do you see that load in your meters?”
  • As the load is ramping up: “Any problems?”

Time is of the Essence – Everything you can do to save yourself time, and everything you can do to speed up and automatehuman_on_start_line your testing, is a good thing to do. This allows you to do more testing in a given amount time and to rapidly diagnose problems.  During the test many people will be impatiently waiting for you.

Spend time before the test preparing your tools to:

  • Start the metering and then the test
  • Check if the test and the metering are running as expected
  • Rapidly capture every possible clue when the test fails
  • Reset things to a known state after a run
  • Do rapid performance analysis of the metering data looking for problems and bottlenecks

Clean Up After Yourself – This will not be your only load test. Make sure you have time in the schedule for cleanup.

If you are in your own internal lab, or some vendor’s lab, leave it nicely picked up when you are done testing. The people who will have to clean up your mess are also the people who will help you set up for the next test.

Someday, you’ll be back in that lab and need their help again.

Sharpen The Sword – After you do a big load test, you should take time to review your notes, logs, and memories looking for things that worked well and things that did not. Improve your tools for next time.  This is also a good time to spend time publicly thanking those who helped you.

Bob Wescott’s (LinkedIn), is semi-retired after a 30 year career in high tech that was mostly focused on computer performance work. Bob has done professional services work in the field of computer performance analysis, including: capacity planning, load testing, simulation modeling, and web performance. He has even written a book on the subject: The Every Computer Performance Book. Bob’s fundamental skill is explaining complex things clearly. He has developed and joyfully taught customer courses at four computer companies and I’ve been a featured speaker at large conferences. Bob’s goal is to be of service, explain things clearly, teach with joy, and lead an honorable life. His goal, at this stage of the game, is to pass on what we’ve learned to the next generation.

Every Computer Performance Book

Price: $19.99

4.4 out of 5 stars (31 customer reviews)

22 used & new available from $15.98

Related Posts

  • Are You Crazy, Baseline Performance Without Testing!!!Are You Crazy, Baseline Performance Without Testing!!! What Is A System Performance Baseline - Before we dive into the challenges of defining a System Performance Baseline, let's look at the definition of Baseline Performance including how it fits within the context of this article. Baselining Performance or the art of Baselining System […]
  • New HowTo Hits The Road – New Skills For The New YearNew HowTo Hits The Road – New Skills For The New Year Our mission at Practical Performance Analyst has always been to establish, build a strong Body Of Knowledge around SPE (Systems Performance Engineering). Practical Performance Analyst aims to deliver a Body Of Knowledge that provides information around SPE (Systems Performance […]
  • Use of Quantitative Models For Performance Testing & Engineering – Part IIUse of Quantitative Models For Performance Testing & Engineering – Part II This article is the second in a two part series by Author Ramesh Iyer on the 101 around Use of Quantitative Models for Performance Testing & Engineering. If you haven't read the first part we encourage you to check it out at the following link - Use of Quantitative Models for […]
  • Proactive Performance Engineering or Proactive Performance Testing – Whats the Sensible Choice ?Proactive Performance Engineering or Proactive Performance Testing – Whats the Sensible Choice ? WOPR22 is scheduled for the 21-23 of May 2014. The host or in the case of WOPR, the Content Owner, Eric Proegler, along with the WOPR Organizers, has a post at the WOPR website calling for invites to submit proposals for WOPR22. WOPR22 will be hosted by Maria Kedemo of Verisure in Malmö, […]