Questions To Ask When Comparing Vendor Solutions

Introduction – Most of us would have been involved in production selection at one time or the other. Product selection could involve helping your organization or business short list and identify a suitable solution to meet an internal business need e.g. SaaS based Customer Relationship Management (CRM) or a Portal solution for your new customer website, etc. Product selection involves bringing in your technical expertise and combining it with your business acumen to help the business make the right choice with regards to suitability, affordability, maintainability, etc.

For a lot of us this entire process is like getting into a new relationship, it all looks very pretty and rosy on the outside but the reality only hits you once you have actually begun using the product and have begun to understand the constraints or challenges of delivering your business process using the solution you have recommended business invest in. Every vendor (and this would include me if i did work for one..:)) would do their best to dress up their bride (product) to look the best, showcasing all the product strengths while wanting to downplay the importance or impact of any limitations that the product might bring to the table. It’s normal human tendency to play-up the good or positive while downplaying the negative or bad.

In this short article we will look at the process of production evaluation from a technology standpoint and will consider various dimensions that you might want to consider when evaluating a product with regards to suitability from scalability, performance and reliability standpoint.

Dimensions to consider – Let’s look at the various dimensions one would consider when reviewing and short listing vendor products for a given business need.  Here’s some of the broad dimensions that you should consider having on your list when reviewing products and short listing vendors.

  • Cost (Capex or Opex in case of a SaaS solution)
  • Performance
  • Security
  • Maintainability
  • Usability
  • Scalability
  • Reliability
  • Vendor track record for support
  • Availability of resources in the market (to help you use the product)
  • Vendors ability to turn around patches/fixes for high severity defects
  • Product roadmap
  • Your internal ecosystem and how the product/vendor fits in

Please note that this list is not meant to be comprehensive and cover ever product selection initiative you might be involved with but rather provide guidance and help you think about the relevant aspects of product selection from a performance, scalablity and reliability standpoint. So please, please use this information responsibly and with care. We do not accept any responsibility for any of the product choices or investments you make.

Does It Scale – In the previous section we looked briefly at the various dimensions that one might consider when short listing products for their organization or business. Let’s now look at what relevant questions one might ask when evaluating the suitability of the solution from a performance standpoint.

  • What is my workload – Understand your workload i.e. mix of key business critical processes (Transactions), jobs, workflows, messaging, etc. that make up the processing needs for the given solution. Every workload is different and every implementation within the same organization might have a different workload. So do your homework well and go armed with a very strong understanding of your workload (For more information on Workload modelling and Workload basics please read up on the SPE 101 section here at Practical Performance Analyst)
  • What are my Non Functional Requirements – Understand your Non Functional Requirements, which ones of those are important to your business and which ones are not. Non Functional Requirements state availability requirements, Web Page performance requirements, Throughput requirements, System Utilization requirements, etc. Armed with a good understanding of your Non Functional Requirements you are able to make a more informed decision on the suitability of the vendor or product for your given business need.
  • What workload did the vendor test the product for – Understand what workload has the vendor used when performance testing or benchmarking their product. Understand the relevance of the workload the vendor used for the benchmark in comparison to your internal workload. It’s unlikely that the workload in both the cases match each other 100% but at-least you have an understanding of the gaps/differences which will influence how much you should be trusting the vendor benchmarks or performance testing results.
  • What are the various performance testing scenarios the vendor used – Ask for details of the benchmark with regards to the different types of performance testing scenarios the vendor has performed. The relevant ones could include – stress tests, soak tests, volume (large data volumes) tests, etc. Find out the nature of performance tests that the vendor performed as part of the benchmark and if those scenarios are comprehensive enough to allow you to make your decisions with regards to product suitability.
  • What is the configuration of the environment the product vendor used – Understand the configuration of the environment i.e. storage, compute infrastructure, network infrastructure, etc. that the vendor used for his benchmark. In a lot of cases we have seen that product benchmarks are conducted on infrastructure configuration that aren’t very relevant to what a customer would typically have within their organization. So be careful and make sure you dig into the details.
  • What version of the product and feature set did the vendor use for the benchmark – Understand the versions of the product, the patch levels and the platform (Windows, Unix, etc.) that the vendor used for the benchmark. The underlying platform will drive performance, scalability of the solution to a certain extent. Obviously this also depends on the design of the vendors solution.
  • Is the vendor able to share results of recent benchmarks along with relevant performance metrics captured – Ask for relevant performance metrics along with the printed benchmarks. It’s very rare (mostly never) for vendors to share details of performance metrics along with the glossy brochures that talk about the benchmarks. Ask the relevant questions, dig into the detail and try to understand the true behavior of the solution for the scenarios the vendor has performance tested the solution for.
  • What is the vendor’s track record in terms of delivering solutions that meet high performance needs – Does the vendor have a track record delivering similar solutions at other clients locally or internationally that meet the same performance criteria as your business.
  • What do your peers using the vendors product say about past experiences with regards to product performance – Peers can be a great source of information, especially since they might have witnessed some of the joy and pain of using the solution in the past. Reach out to your peer network and understand the benefits, challenges and pain points of using the short listed solution.

Conclusion – Selecting a product to meet a given business need among a host of commercial or open source options is definitely an interesting challenge. Like most situations in life you it’s highly likely that you will not have all the information you need but will need to make recommendations. And the vendor will definitely do what they do best i.e. try to sell you their solution while playing up the strengths of the solution and there’s absolutely nothing wrong about it. It’s upto you though to do your homework considering some of the above factors and then work with the partners/vendors on short-listing the relevant solutions that meet your business’s needs.


Trevor Warren (Linked In) focuses on building innovative solutions that have the potential to impact people’s lives in a positive manner. Trevor is inquisitive by nature, loves asking questions and sotrevor_warrenmetimes does get into trouble for doing so. He’s passionate about certain things in life and building solutions that have the ability to impact people’s lives in a positive manner is one of them. He believes that he can change the world and is doing the little he can to change it in his own little ways. When not hacking, building new products, writing content for Practical Performance Analyst, dreaming up new concepts or building castles in the air, you can catch-him bird spotting (watching planes fly above his house). You can reach trevor at –  trevor at practical performance analyst dot com. The views expressed on this web site are his own.

Related Posts

  • Dimensions To Consider When Designing Applications For The CloudDimensions To Consider When Designing Applications For The Cloud Introduction - With the increasing maturity of cloud service offerings and realization by customers of the benefits of moving to cloud based applications/platforms/services it's not a surprise that IT shops around the world have cloud adoption some where on their agenda. In context of […]
  • Let’s not go overboard with our investments in Non Functional Testing!!!!Let’s not go overboard with our investments in Non Functional Testing!!!! Background - When was the last time you were told, “Let’s not go overboard with out investments in a performance validation exercise.!!! On my last project we expended a lot of effort on various Non-Functional initiatives, however the issues we experienced in production were never picked […]
  • A Few Good QuestionsA Few Good Questions Good to be back - G'day folks. It's really been a while since i have had the opportunity to put pen to paper. The last few months have been crazy and I have personally been busy with some really interesting challenges. Some of these challenges have been around scaling some really […]
  • HowTo Create A Performance BudgetHowTo Create A Performance Budget Launching HowTo's - We have just launched a new section here at Practical Performance Analyst called HowTo's. Howto's much like their counterparts (in the Open Source movement) will focus on the specifics around different SPE (Systems Performance Engineering) related tasks. There is a […]