At Practical Performance Analyst we see interviews as a great opportunity to get to know our authors better, understand the work they do, their passions, their experiences and the contributions they’ve made over the years to the field of SPE (Systems Performance Engineering). In this article we talk to Martin Thompson (LinkedIn), the High-Performance & Low-Latency Computing Specialist based out of Brentford, United Kingdom. Martin Thompson has spent over two decades building distributed and high-performance computing systems while working in startup companies through to financial institutions.
Martin speaks regularly at conferences around the world on topics related to High-Performance & Low-Latency Computing. Martin believes in applying Mechanical Sympathy to software development resulting in elegant, high-performance, and reliable solutions. Martin’s specialties include : High-performance distributed Java, C, and C++ systems; low-latency financial exchanges; parallel and concurrent algorithms using lock free techniques; advanced data structures; complex transactions; systems architecture; agile development; and product development strategy.
PPA : Thanks for taking the time to talk to us. The objective of this interview is to explore the work you’ve done over the years and see performance from your perspective. Tell us about your current role as a High Performance Computing specialist and what is it you are currently working on at Real Logic.
Martin : At present I work as an independent consultant helping clients improve performance of their systems. The performance optimization delivered enables new business opportunities or achieves cost savings via efficiency gains which matters a lot to business helping them stay competitive. The performance optimization can range from competing in low-latency trading environments, analysis and transformation on large data sets, through to making websites more responsive to end users.
Currently I have a number of clients. Some want training and coaching to raise the game for their teams. Others outsource key components with exacting performance requirements for whom I would put together and lead development small teams. I like to work on development projects that push the boundaries of what people believe is possible and then exceed expectations. When doing this I like to bring the client team along so they can learn and gain the skills to contribute, eventually they can do similar things themselves.
PPA : What’s the nature of work you tend to do and what are the nature of projects you generally tend to focus on.
Martin : Engagements tend to be one of two major types. Either an immediate performance issue exists that needs urgent addressing, or an organization is looking forward strategically. Strategic views could imply that organization might like some help with how they evolve their current systems with the objective of making their teams deliver more high performing systems. Activities include performance testing, profiling, tuning, development, training, helping with recruitment, and coaching.
Immediate performance issues are often easy to address. It seems our industry has a shortage of people who can demonstrate the scientific discipline to characterize a problem, then design an experiment to prove a hypothesis via experimental evidence which is key to performance profiling. Even skills like interpreting the output of a profiler are not that common. With good discipline and profiling skills, it can take a relatively short time to do a root cause analysis.
I find that the fixing of the immediate performance issue is only the start to an engagement. Clients seem to really like my working practices and want me to continue and help them improve their overall software development life cycle (SDLC). The key to high-performance code is to develop clean, elegant, and appropriate models. For the majority of the time, high performance software development is not about compiler hacks and bit twiddling. It is about fundamental design principles that are key to doing any effective software development.
PPA : When was it that you decided to make a career as a High Performance Computing specialist. What made you make such a decision. Tell us about your journey to a High Performance Computing Specialist.
Martin : Well, I never set out to be a High Performance Computing Specialist. I love the application of technology in ways that enable new business opportunities. Often by taking performance to a new level unlocks a new business opportunity. One of my first commercial projects was as an under graduate at the turn of the 90s that took stock market feeds in EBCDIC, converted them to ASCII, and then processing on PCs. Back then stock market feeds were beginning to approach 1GB of data per day and required mainframes for processing. I realized that you could put multiple 250MB hard drives in a single PC and thus could handle a whole days data without requiring a mainframe. I wrote some libs in C++ to ingest the data and index it for analysis. Queries could then be run on the data to produce extracts that could be input to tools like SPSS and SAS. Now market analysis could be done without requiring access to a mainframe.
This sort of pattern repeated over my career with various forays into the likes of, iterative financial models, motor vehicle modelling and financing, streaming audio data, content management, gaming & gambling, and high frequency trading. At each step I discovered that sufficiently advanced performance unlocked new opportunities. For example, processing a loan application in 12 seconds rather than 25 minutes lets you capture a customer before they leave a showroom; or taking a batch data processing run from 10 hours down to 5 minutes means you can make a mistake, or do what ifs, and run it again in an over night run before it needs published for the next day.
PPA : You’ve worked at various organizations. What in your opinion are some of the most common challenges organizations face when developing systems that need to perform and scale.
Martin : The most fundamental issue I see with software development is two fold. Firstly, as an industry we let complexity get out of control too quickly. We need to fight really hard to tame complexity. For example, we should only be applying abstraction to be more precise and not the leaky type to paper over a poor and complex implementation thus causing confusion and extra indirection. Secondly, the over specialisation in learning frameworks and existing in only one architectural layer makes people myopic and not see the whole picture. We need holistic solutions that are simple, elegant, and only add in libs and frameworks that pay for themselves.
If people are building clean elegant software then performance problems do not tend to exist for all but the most demanding domains. If more exacting performance is required then automated performance testing and monitoring should be an integral part of the SDLC. Having a performance team is an anti-pattern. Just like having a testing team is an anti-pattern. The delivered quality of the software is the responsibility of all developers. It is better having performance specialists integrated with the team who coach the others and raise the standard of the whole team. This approach applies to all quality-of-service measures such as performance, security, usability, etc.
PPA : There seems to be a lot of talk around Web Optimization & Performance these days especially with Velocity conferences throwing their spotlight on it. What avenues should professionals consider if they wanted to broader then perspective on Performance/Scalability rather than focus on the narrow paradigms of Web Optimization & Performance.
Martin : Web performance is crucial in the overall scheme and thus is gets a good level of attention. Steve Souders has done an excellent job of throwing the spotlight of this area. On the server and gateways I’ve been trying to help via the Mechanical Sympathy movement with my blog and the Google Discussion Group. Others who have been doing excellent work include Neil Gunther, Brendan Gregg, Gil Tene, Aleksey Shipilëv, and Kirk Pepperdine, to name a few.
PPA : From what we’ve seen so far there aren’t any good academic offerings around practical aspects of Systems Performance Engineering (SPE). What would you recommend budding professionals consider investing in if they wanted to build a career has a High Performance Computing Specialist.
Martin : Learn the fundamentals of how hardware, operating systems, managed runtimes, and compilers work. Then learn to write clean elegant code that is sympathetic to how the full stack works. Code that is regularly performance tested and profiled. I like to profile everything I do, and as many other systems as I can get my hands on, to give me a feel for relative costs and educate myself about bottlenecks. Then top this off with learning some fundamental theory like Universal Scalability Law, Queuing Theory, and the impact of batching effects on amortized cost.
PPA : You were the CTO and Co-Founder at LMAX responsible for putting together a High Performance Financial Exchange. Could you tell us about some of the Performance challenges involved in such a program of work and how you went about addressing them.
Martin : At LMAX we set out to build the first retail customer focused financial exchange as a spin out from Betfair. Our initial concerns were about throughput and contention which we well understood from the Betfair sports markets. We got surprised by the advent of high-frequency trading (HFT) and the need to be low and predictable latency. This meant we had to go back and address our fundamental design decisions and this give rise to design patterns like the Disruptor. Achieving high throughput is one level of challenge. Trying to achieve high-throughput while keeping latency low and very predictable is a very specialized skill that so few developers have mastered. We found we could not recruit these skills and instead had to grow the internal team in this area. One of my greatest joys from that time was seeing people like Mike Barker grow to become world class, he is now lead maintainer of the Disruptor in addition to doing other great things.
PPA : You are a regular speaker at various national and international conferences. Would you recommend any specific talks on Performance/Scalability for readers interested in the principles of building High Performance Computing systems.
Martin : I’ve tried to do a few talks that would set people off in the right direction.
- Responsive Design: Responding in a timely manner.
- Mythbusting Modern Hardware for Mechanical Sympathy
- Performance Testing Java Applications
Tara Simpson from Instil and I, with the support of Jonas Boner from Typesafe, have been running a small series of conferences that focus on what it takes to make systems responsive, resilient, and elastic. This builds on the Reactive Manifest, with the videos being posted to YouTube.
PPA : Are there any rules or mantras you follow when designing and building systems for High Performance & Scalability. Anything you would like to share with our readers.
Martin : Keep it clean; keep it simple; aim to be elegant. Be ruthless in your desire to drive out complexity. The fastest code is the code you never need to execute.
PPA : Are there any books you would recommend to our readers who might be considering building a career in SPE (Systems Performance Engineering)
Martin : Here’s me recommendations on a couple of books you could sink your teeth into and get started with the basics of SPE (Systems Performance Engineering):
PPA : What advice would you give those interested in building a career on SPE (Systems Performance Engineering).
Martin : Find really good people who work in this area and see if you can get to work with them. There is lots of theory to learn but the most important attributes are to have a curious and very scientific mind, then get lets of real world practice as it takes many hours to master the skills.
We appreciate you taking the time out to answer our questions and truly appreciate the support. Best wishes for the rest of 2015.