Six things I wish we had known about scaling

Looking back at the last few years of building Rapportive and LinkedIn Intro, I realised that there were a number of lessons that we had to learn the hard way. We built some reasonably large data systems, and there are a few things I really wish we had known beforehand.

None of these lessons are particularly obscure – they are all well-documented, if you know where to look. They are the kind of things that made me think “I can’t believe I didn’t know that, I’m so stupid #facepalm” in retrospect. But perhaps I’m not the only one who started out not knowing these things, so I’ll write them down for the benefit of anyone else who finds themself having to scale a system.

The kind of system I’m talking about is the data backend of a consumer web/mobile app with a million users (order of magnitude). At the scale of Google, LinkedIn, Facebook or Twitter (hundreds of millions of users), you’ll have an entirely different set of problems, but you’ll also have a bigger team of experienced developers and operations people around you. The mid-range scale of about a million users is interesting, because it’s quite feasible for a small startup team to get there with some luck and good marketing skills. If that sounds like you, here are a few things to keep in mind.

Please Click Here to read rest of the article.

Related Posts

  • How Disqus Went Realtime With 165K Messages Per Second And Less Than .2 Seconds LatencyHow Disqus Went Realtime With 165K Messages Per Second And Less Than .2 Seconds Latency How do you add realtime functionality to a web scale application? That's what Adam Hitchcock, a Software Engineer at Disqus talks about in an excellent talk: Making DISQUS Realtime. Disqus had to take their commenting system and add realtime capabilities to it. Not something that's easy […]
  • Breaking the Sound Barrier: MongoDB (WiredTiger) on AWSBreaking the Sound Barrier: MongoDB (WiredTiger) on AWS You are reading: Breaking the Sound Barrier: 1.2M inserts/sec with MongoDB WiredTiger on a single AWS instance - Part I of II, by  Dmitry Agranat (LinkedIn).  The second part of this series will be published next week here at Practical Performance Analyst. Introduction - MongoDB is a […]
  • Breaking the Sound Barrier: MongoDB (WiredTiger) on AWS – Part IIBreaking the Sound Barrier: MongoDB (WiredTiger) on AWS – Part II You are reading: Breaking the Sound Barrier: 1.2M inserts/sec with MongoDB WiredTiger on a single AWS instance - Part II of II, by  Dmitry Agranat (LinkedIn).  The first part of this series can be read here. Introduction - MongoDB is a cross-platform document oriented database. […]
  • Engineering High Performance Applications with AWS – Part 2Engineering High Performance Applications with AWS – Part 2 This series of articles by Dmitry Agranat (LinkedIn) focuses on the basic concepts and gotchas around building high performance applications using Amazon’s AWS platform. Dmitry Agranat explores the foundation principles of AWS while gradually introducing the reader to various important […]