This series of articles by Dmitry Agranat (LinkedIn) focuses on the basic concepts and gotchas around building high performance applications using Amazon’s AWS platform. Dmitry Agranat explores the foundation principles of AWS while gradually introducing the reader to various important fundamental concepts around AWS/EC2. Dmitry hopes that through this series of articles, you as the Performance Engineer would gain enough of background knowledge on Amazon’s Cloud capability to become an effective contributor when it comes to engaging in AWS related discussions/projects within your organization. These articles essentially are aimed at helping readers build stronger understanding of AWS/EC2 and eventually greater appreciation for what Amazon’s AWS has to offer.
You can read “Engineering High Performance Applications with AWS – Part 1” by clicking here.
Performance Engineering starts with Design – A very large number of applications these days within our organizations are starting to use third party components hosted in the cloud and are also building custom bespoke components that make use of the cloud to scale on demand using infrastructure real estate effectively and efficiently. As a Performance Engineering lead it really doesn’t help to find yourself sitting at an architectural governance meeting (for a system that intends to use AWS/EC2) with you or your team have very little or no clue of what everyone is talking about. Knowing your turf well and having a good understanding of the various AWS/EC2 design principles will help you contribute more effectively and be part of the Architectural decision making process.
Being proactive and working constantly on staying one step ahead of the competition eventually pays off. With that thought at the back of our minds let’s now draw parallels with regards to application development using the cloud and performance engineering across the SDLC (Software Development Life Cycle). Being proactive in this case could be viewed as the Performance Engineering team having a solid grasp of the concepts of the cloud, AWS, EC2, EBS, etc. Being proactive could also be viewed as th
e Performance Engineering team having a good understanding of the various principles of design using the various cloud components to design, build and deliver scalable applications from a Systems Performance Engineering standpoint.
Armed with a understanding of the basic concepts around AWS, EC2, EBS, etc. the Performance Engineering team within the organization now has the ability to support the Architecture, Design, Build teams on selecting the relevant AWS/EC2 building blocks and AWS/EC2 design patterns that enable the program to deliver applications that scale and meet their overall Non Functional Requirements.
Delivering Value is Key to Gaining Visibility – In most organizations Performance Engineering teams do not exist. In the few where Performance Engineering teams exist, unless there is a problem related to performance across any of the business critical systems, the Performance Engineering team is generally invisible. Proactive Performance Management is generally given some lip service since its part of the design/build checklist within the development stream. The Performance Engineering team unfortunately is also not included in important Architectural/Design related meetings at the start of the application Architecture/Design process even though the organization might have burned its fingers numerous times in the past delivering applications that perform poorly. To be effective and deliver value it’s important that the Performance Engineering team work closely with the IT leadership and establish themselves as a team that delivers value by partnering with Architecture, Design, Build, Test and Product support teams across the organization. Proactive Performance Management starts with the PE team being proactive about reaching out and creating the necessary relationships that will help them succeed and deliver perceptible value.
Refreshing The Basics – Having come this far we assume you are by now well acquainted with the importance of SPE (Systems Performance Engineering) across the SDLC (Systems Development Life Cycle) and also understand the basics of Amazons AWS/EC2. If you are not we recommend you do the following:
- Visit the Initiatives/SPE section of Practical Performance Analyst to understand the basics of SPE (Systems Performance Engineering) across the SDLC
- Visit the Amazon AWS Fundamentals links provided in the first part of this article and familiarize yourself with the basics of AWS and EC2 (You can read “Engineering High Performance Applications with AWS – Part 1” by clicking here.)
Let’s now dive in and open a free account (validity of 30 days). You will need to have your credit card details handy since Amazon requires it as part of the registration process. For details on the sign up process and to create your account please visit – Creating an Amazon Web Services Account.
Diving Straight In – Please bear in mind that this free 30 day account isn’t completely free and you could end up with a large bill in the end depending what you do within the evaluation period. We advise you to please read the fine print and understand what is and what isn’t covered as part of the initial evaluation period. From what we’ve been given the understand individuals with new AWS/EC2 accounts get access to entry level Virtual Machine resources for purposes of evaluation. However, bandwidth charges for data transferred (into and out of) the Virtual Machine has an upper cap and you can get billed for crossing those limits. We would encourage you to read between the lines and understand the nature of the Amazon AWS/EC2 evaluation offer. We would also advice that you monitor you daily resource usage bill to make sure you do not breach any of those evaluation limits as set by Amazon. On the other hand if you don’t mind spending a few bucks checking your spend on a daily basis is a good way to ensure that you stay within the targets you’ve set for yourself from a billing perspective.
It’s difficult for a beginner to understand what he/she might end up paying during these initial 30 days. We would encourage you to look at the following link AWS/EC2 Evaluation Offer. If your application use exceeds the evaluation limits set by Amazon at worst you simply pay standard, pay-as-you-go service rates for the resources you’ve ended up using. We recommend that your home minister (partner/wife) and confirm the amount of money you are entitled to spend. A 50$ credit will get you enough of AWS/EC2 compute resources to last the learning period.
Digging Deeper – After you’ve opened an account, navigate through all options and understand the various services that Amazon AWS has to offer. Doesn’t matter if you do not eventually end up using them, obtaining a good understanding of the service will help you position AWS/EC2 for future projects within the organization. We also recommend that you pay special attention to the following Amazon AWS concepts –
- EC2 (Compute capacity)
- S3 (simple storage)
- EBS (block level storage)
- CloudWatch (collect and view metrics).
Play around with these Amazon AWS services to understand basic management operations, architectural design fit, design limitations and most importantly scalability limitations. Focus on picking up the following concepts:
- EC2: Stopping, Starting, and Terminating Instances
- EBS: create/attach/detach storage
- S3: store and retrieve data
- Enable advanced CloudWatch monitoring and go through available metrics for all services
- Play with the granularity and correlate several metrics per graph to understand the visibility that the metrics provide
The best way to understand the features of AWS/EC2/S3/EBS/Cloudwatch is to configure your test applications for failover/scalability/redundancy, deploy them in the cloud and then inject them with realistic workload to see what Amazon AWS can truly do for you and your organization. You might also want to take things to the next level by incorporating scalability, failover and redundancy into your deployment architecture and see the system scale as your scale the workload and pull the plug on various virtual machines. We would encourage you to install a demo application which you can play with on the cloud. Avoid using any internal applications on AWS and if you do please make sure you obtain all the necessary security clearances from your IT/Security teams internally.
Design, Build & Execute Tests using the Cloud – Feel free to download you performance testing tool of choice. We are very familiar with Load Runner and to make things really easy you could also download the trial version of HP Load Runner. A trial version of HP Load Runner can be obtained from the following link LoadRunner Trial Download. Record a few sample test cases and execute a light load with few concurrent users to make sure everything hangs together and works as expected. Check to see that relevant metrics are now visible within CloudWatch monitoring. Don’t try hitting your application too hard at this point. Remember, you will be paying for all that extra traffic and IOPs you put through Amazon. The goal here is to for you to gain the required basic concepts associated with Amazons AWS/EC2/S3/EBS/Cloudwatch to enable you to sit through architectural/design meetings where you will be debating design patterns or architectural decisions for systems that are being designed using the cloud.
Closing Notes – Finally, we recommend that you stop/terminate all of the instances you have created and shut down your account once you are done with the learning process. In the next articles you’ll definitely need a corporate credit card or possibly a larger approval from your home minister (partner/wife). So start the lobbying process right now. Happy hacking and best wishes until next time.
Dmitry Agranat (LinkedIn), is a Performance Engineering Principal with primary focus on bringing a positive impact on overall organizational system performance. His professional career has started in 2005 at Mercury (now HP Software). Dmitry has worked with many internal and external customers to both solve immediate performance problems as well as to educate on building solid organizational performance management methodology. Dmitry believes that Performance Engineering (PE) is still not well understood, agreed upon, unified or can be quantified, thus requires building solid and effective entrepreneurship around Performance domain.