NFR Challenges – Is Real End User Experience King?

Importance of Non Functional Requirements – Non Functional Requirements are to Performance Engineers what Functional Requirements are to Developers on a program. Non Functional Requirements are some of the most important (but not the only) and basic building blocks that go towards building a solid Performance Engineering strategy for your program. Miss out on one of the basic blocks and you’ll have challenges getting everyone on the program to speak the same language, forget achieving your targets. Before we dive deeper into article and look at Non Functional Requirements related challenges it makes sense to talk about the various focus areas for Non Functional Requirements so that all of us are on the same page. For purpose of this article i would like to define focus areas for Non Functional Requirements as follows:

  • Performance
  • Availability
  • Disaster Recovery
  • Reliability
  • Usability
  • Security
  • Maintainability & Operability

In my limited experience over the last decade and a half I’ve seen very few, infact just a handful of programs budget for a full time Performance Engineer all through the duration of the program. And the truth is, one of the reasons we were able to bake Performance Engineering into the program design was because of the fact that we happened to have good relationships with the individuals responsible for putting together the overall program design including the statement of work (SoW). So, don’t hope that you’ll be given a chance to make a difference, even good intentions are enough to bring about change. What one requires to bring about change is the ability to put your foot down, speak your mind and ask for the right thing to be done, albeit in a professional and bureaucratic manner. Don’t be a rude cowboy, you won’t get much traction.

What should my focus be – Systems being designed today are complex interconnected set of applications, partially hosted within the organization datacenters connecting to SaaS based services hosted all over the internet. In a situation like this where you have complex application architectures connecting to service buses that then connect to 3rd party SaaS based application suites hosted within a 3rd party data center the question that generally tends to come up is, “How can we be responsible for defining Non Functional Requirements and delivering End to End Client Experience that include transactions that traverse through infrastructure or environments we have no control over”. This is quite a common occurrence and I will be truly surprised if you haven’t encountered one of these on your own programs.

It’s easy to shy away from taking responsibility for the End to End Transactional response times by drawing a boundary across the system in such a manner that the Non Functional Requirements do not include Response Times targets or throughput targets for the 3rd Party Service provider. I agree that it’s not easy to take responsibility for Performance of any system if you do not have control of the environment of infrastructure over which the transaction traverses.

How can i avoid drawing the wrong system boundaries – In situations like this I would urge you to ask yourself the following questions:

  • What are the entry and exit points for the end customers using the application
  • What really matters to the end customers using the application
  • What are the factors that will influence the End User Experience as seen by the end customer and how many of those variables are currently within your control
  • What are the factors that will influence the End User Experience as seen by the end customer and how many of those variables are currently outside of your control

Drawing system boundaries and making the program take responsibility for End User experience for transactions that flow across boundaries into 3rd party SaaS providers environments is tough. You have to be really sure of what you are trying to achieve, what are the factors within your control, what are the factors out of your control and how you can productively steer the conversation working out the relevant agreements with your client and the 3rd party service provider.

The question i always ask myself is, “What is the end customer going to expect when he/she uses this application”. An answer to that question helps me better picture what should be the nature of system boundaries for the Non Functional Requirements i put in place.

Negotiation is a black art – I don’t claim to be a negotiation superstar and i would generally leave that to the legal and procurement guys within the client organization. However as the Lead Performance Engineer or Lead Performance Architect on the program it is your responsibility to work with the client and ensure that the client has thought through the ramifications of having or not having a agreed set of Service Level Agreements (SLA’s) with the 3rd Party SaaS based service provider. Without strict SLA’s with your SaaS based service provider you will have challenges defining the relevant system boundaries and taking responsibility for Non Functional Requirements across the entire transaction path which is essential for you to deliver an application that meets the document and agreed Non Functional Requirements. Few SaaS based vendors are willing to consider rigid SLA’s especially since they can’t be seen responsible for the internet links that connect your datacenters to theirs over the open public internet. SaaS based vendors might provide SLA’s but when you read into your dotted line you will figure out that they do not hold themselves responsible for performance of their application outside of their data centers, which is understandable and completely reasonable. Make sure you understand the implications of these SLA’s and agreements before you push for any changes.

 Closing notes – Non Functional Requirements are some of the most important building blocks for a solid Performance Engineering strategy. Without a good Performance engineering strategy and clear Non Functional Requirements you will end up in a situation where every team on the program has their own view of what Performance should look like. It’s also not easy taking on responsibility for delivering performance for an application that traverses system boundaries into 3rd party SaaS based providers data centers which are out of your control. It’s a tough one, there is no easy way out and i don’t suggest at any point that any of this is a cake walk. But as the Performance Architect on the program it’s your responsibility to ensure that the End User Experience drives your definition of Non Functional Requirements ensuring you draw the system boundaries accordingly.

As always, send us your thoughts, inputs and comments at trevor at practical performance analyst dot com.


Related Posts

  • HowTo Create A Performance BudgetHowTo Create A Performance Budget Launching HowTo's - We have just launched a new section here at Practical Performance Analyst called HowTo's. Howto's much like their counterparts (in the Open Source movement) will focus on the specifics around different SPE (Systems Performance Engineering) related tasks. There is a […]
  • Questions To Ask When Comparing Vendor SolutionsQuestions To Ask When Comparing Vendor Solutions Introduction - Most of us would have been involved in production selection at one time or the other. Product selection could involve helping your organization or business short list and identify a suitable solution to meet an internal business need e.g. SaaS based Customer Relationship […]
  • A Few Good QuestionsA Few Good Questions Good to be back - G'day folks. It's really been a while since i have had the opportunity to put pen to paper. The last few months have been crazy and I have personally been busy with some really interesting challenges. Some of these challenges have been around scaling some really […]
  • Non-functional Requirements in Architectural Decision MakingNon-functional Requirements in Architectural Decision Making This article first appeared in IEEE Software magazine and was then published by InfoQ at their website. In software engineering, a tight relationship exists between nonfunctional requirements (NFRs) and software architectures (SAs). As early as 1994, Rick Kazman and Len Bass asserted […]
  • Paul Offord

    I think the points raised above are spot on.

    I guess my experiences regarding NFRs are much like those of others here. I see people conducting load tests with little idea of what represents a pass or fail. We investigate problems where we’re told that the system is slow without any definition of precisely what is slow, or how slow is slow. We work on impact projects where the project team wants to know if the migration of a system from one location to another will result in unacceptable response times without any definition of acceptable – no benchmarks.

    The crazy thing is that I find project people getting right down into the nitty gritty of the technologies, and yet without a definition of the NFRs the nitty gritty is all just academic stuff.

    An additional problem is that if you ask for NFRs from a project team you’ll often never get an answer – nobody wants to take the responsibility for defining acceptable performance. My advice; if you don’t have any NFRs for a load test or some other exercise, invent them. You can base it on observations and rough timings. Circulate what you’ve invented and when you get feedback refine the NFRs.

    • tw37

      Agree with everything you said Paul. The challenge we have today is educating our developers and architects on the need for good and sound NFR’s. It’s an easy sell when you are at the design stage and have the ability to influence the structure of a program. It tends to get gradually tougher as you are brought into fix programs that are later in the lifecyle nearing go live.

      I also see value in what you said, come up with your own NFR’s based on experience but the challenge here is most of the time the individuals responsible for fixing/addressing performance have little or no experience and the NFR’s that are designed leave a lot to be desired.
      Glad you liked the post. Please feel free to contribute content. We are always on the lookout for experienced professional writers with experience in performance across the SDLC.

      • Zubair

        It is just the domain/business understanding and common sense required to come up with NFRs, but its truely challenging to get them right in client’s eyes. As I have seen, Paul rightly said, nobody from client’s side wants to take the responsilibty of defining acceptable NFRs. Even if we manage to get them at that stage, how valid is testing/measuring the system’s performance against these NFRs when the design has not taken them into account?

        • tw37

          Hi Zubair,

          Excellent view point. In the last decade that I’ve been associated with development of complex systems I’ve realized that its quite rare that business/application owners do not want to take ownership of NFR’s but more so Technical Architects / Performance Engineers alike lack the capability to articulate Non Functional Requirements clearly upfront.

          Documenting Non Functional Requirements is a process and you’ve got to take the business/application owners/SME’s/program leadership along the journey. Get them to share the data you need and then work with them in evolving the right NFR’s. Just like any design artifacts NFR’s too must get signed before you can move onto the build phase so that NFR’s can be broken down into tiered response time goals for the various application/infrastructure tiers.

          When your stakeholders understand your process, what you are trying to do and how you are trying to get there, they will buy into the process. I haven’t found a single business app owner shirking of responsibility once we’ve gone through the journey together.

          Let me know if you have any more questions.