Click here to close now.

Welcome!

Security Authors: Elizabeth White, Liz McMillan, Brad Thies, Pat Romanski, Srinivasan Sundara Rajan

Related Topics: SOA & WOA, Java, Linux, Web 2.0, Security, Big Data Journal

SOA & WOA: Article

Twitter’s Story: How Homegrown Load Testing Tools Can Misfire

If Twitter built their own tool to perform their own load test, why did the selfie cause their site to crash?

The selfie that changed the world, or at least Twitter, has been in the news for the past month. On March 2, 2014, the infamous Oscar selfie of Ellen and her celebrity friends broke a record of 2 million retweets before midnight the same night. That record was previously set by President Barack Obama, hugging first lady Michelle Obama after his 2012 re-election.

The selfie caused Twitter to crash for more than 20 minutes, also breaking the record for the longest crash of the social media site. Twitter was infamous for crashing in its early days (anyone remember "Fail Whale?"), so it's no wonder the social media giant worked extra hard to completely prepare their website infrastructure before going public in November 2013. This included building their own load testing tool, Iago, in 2012.

If they built their own tool to perform their own load test, why did the selfie cause their site to crash? The Oscar selfie crash is a perfect example of what companies can easily overlook. Twitter did not test their users properly and their homegrown tool clearly doesn't solve all of their problems... their servers still crash.

What is Iago and why did Twitter decide to make it? And what does that have to do with your decision to use homegrown tools versus vendor load testing tools? Don't worry, we will tell you.

Twitter's Homegrown Load Testing Tool: Iago
Iago was created in June of 2012 by Twitter's internal engineering team. According to Twitter, Iago is a load generator created to help the social media site test services before they encounter production traffic. Chris Aniszczyk, Head of Open Source at Twitter, said, "There are many load generators available in the open source and commercial software worlds, but Iago provided us with capabilities that are uniquely suited for Twitter's environment and the precise degree to which we need to test our services."

Basically their homegrown tool was completely customized for their platform alone - a very attractive aspect of developing your own tool.

The three attributes Twitter focused on in creating Iago were:

  1. High performance: Iago was designed to generate traffic in a precise and predictable way, to minimize variance between test runs and allow comparisons to be made between development iterations.
  2. Multi-protocol: Modeling a system as complex as Twitter can be difficult, but it's made easier by decomposing it into component services. Once decomposed, each piece can be tested in isolation; which requires the load generator to speak each service's protocol. Twitter has in excess of 100 such services, and Iago tests most of them using built-in support for the internal protocols Twitter uses.
  3. Extensible: Iago is designed for engineers. It assumes the person building the system will also be interested in validating performance. As such, the tool is designed from the ground up to be extensible - making it easy to generate new traffic types, over new protocols and with individualized traffic sources.

Why Twitter Couldn't Handle Ellen's Selfie
If we were to do the math, Iago was up and running for nearly two years before the Oscar selfie. What happened to their load testing tool?

There were two main reasons why Twitter crashed. First, the tweet Ellen posted was a picture. On Twitter, a tweet accounts for only 260 bytes of data while a picture on Twitter accounts for 33KB of data, almost 130 times as much as a tweet. Second, Twitter's distributed server system was already at max capacity so the load taken on by the website couldn't be distributed to any nearby servers.

Twitter made one major mistake contributing to the crash back in March: they didn't anticipate and replicate real user activity. Most likely Iago wasn't instructed to generate a load based on a picture being retweeted millions of times, thus Twitter didn't know what to expect when the Oscars rolled around.

Homegrown Tool vs. Vendor Load Testing Solutions
Twitter was looking for a DIY homegrown solution because of their unique platform, and while most load testing tools seek to accomplish the same goal, there are always differences between tools. Here are some of the differences we see between homegrown and vendor-provided load testing tools.

Homegrown

Vendor

1. You can customize it - A homegrown tool, created completely from scratch, means you can build exactly what you think you need. But you don't get the benefit of lots of other people's experiences - so what you think you need may not turn out to be what you actually need.

2. Homegrown tools are free like a puppy is free - We have mentioned this analogy before in another post, and it needs to be brought up again. Homegrown tools aren't really free. You have to pay a team to keep them up and running.

3. What if someone leaves? - A couple of developers are experts on the tool they built, but what happens when they leave? Who is going to be your expert? If the new person isn't properly trained and if the code isn't well documented, then modifications to the code become extremely difficult.

1. Your vendor makes it extensible - A vendor that supports lots of users is constantly adding features and capabilities to support needs you may not have realized you have yet. With extensible APIs and SDKs, you still have the ability to add integrations and the protocol support your application demands.

2. You pay for something that works - With homegrown tools there is no guarantee that it will work 100% of the time. By paying a vendor to use their tool, you have assurances that your load testing tool won't let you down when you need it most.

3. Support Team - Vendors dedicate people to work on any issues and keep you worry-free. Their team of experts is available to make sure all of your questions and concerns are addressed.

Homegrown Tools Are Not for Everyone
Twitter's tool has done well to help them improve the performance of the service, but continued service disruptions show that it isn't perfect. A homegrown tool isn't always the best solution, and most of the time, companies do not have the resources handy to completely build their load testing tool from scratch. If you are interested in learning more about vendor options, give us a call. We're more than happy to help.

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that Dyn, the worldwide leader in Internet Performance, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Dyn is a cloud-based Internet Performance company. Dyn helps companies monitor, control, and optimize online infrastructure for an exceptional end-user experience. Through a world-class network and unrivaled, objective intelligence into Internet conditions, Dyn ensures traffic gets delivered faster, safer, and more reliably than ever.
Containers and microservices have become topics of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming 16th Cloud Expo at the Javits Center in New York June 9-11 will find fresh new content in a new track called PaaS | Containers & Microservices Containers are not being considered for the first time by the cloud community, but a current era of re-consideration has pushed them to the top of the cloud agenda. With the launch of Docker's initial release in March of 2013, interest was revved up several notches. Then late last...
CommVault has announced that top industry technology visionaries have joined its leadership team. The addition of leaders from companies such as Oracle, SAP, Microsoft, Cisco, PwC and EMC signals the continuation of CommVault Next, the company's business transformation for sales, go-to-market strategies, pricing and packaging and technology innovation. The company also announced that it had realigned its structure to create business units to more directly match how customers evaluate, deploy, operate, and purchase technology.
In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect at GE, and Ibrahim Gokcen, who leads GE's advanced IoT analytics, focused on the Internet of Things / Industrial Internet and how to make it operational for business end-users. Learn about the challenges posed by machine and sensor data and how to marry it with enterprise data. They also discussed the tips and tricks to provide the Industrial Internet as an end-user consumable service using Big Data Analytics and Industrial Cloud.
Performance is the intersection of power, agility, control, and choice. If you value performance, and more specifically consistent performance, you need to look beyond simple virtualized compute. Many factors need to be considered to create a truly performant environment. In his General Session at 15th Cloud Expo, Harold Hannon, Sr. Software Architect at SoftLayer, discussed how to take advantage of a multitude of compute options and platform features to make cloud the cornerstone of your online presence.
The explosion of connected devices / sensors is creating an ever-expanding set of new and valuable data. In parallel the emerging capability of Big Data technologies to store, access, analyze, and react to this data is producing changes in business models under the umbrella of the Internet of Things (IoT). In particular within the Insurance industry, IoT appears positioned to enable deep changes by altering relationships between insurers, distributors, and the insured. In his session at @ThingsExpo, Michael Sick, a Senior Manager and Big Data Architect within Ernst and Young's Financial Servi...
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
IoT is still a vague buzzword for many people. In his session at @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, discussed the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. He also discussed how IoT is perceived by investors and how venture capitalist access this space. Other topics discussed were barriers to success, what is new, what is old, and what the future may hold. Mike Kavis is Vice President & Principal Cloud Architect at Cloud Technology Pa...
Even as cloud and managed services grow increasingly central to business strategy and performance, challenges remain. The biggest sticking point for companies seeking to capitalize on the cloud is data security. Keeping data safe is an issue in any computing environment, and it has been a focus since the earliest days of the cloud revolution. Understandably so: a lot can go wrong when you allow valuable information to live outside the firewall. Recent revelations about government snooping, along with a steady stream of well-publicized data breaches, only add to the uncertainty
The explosion of connected devices / sensors is creating an ever-expanding set of new and valuable data. In parallel the emerging capability of Big Data technologies to store, access, analyze, and react to this data is producing changes in business models under the umbrella of the Internet of Things (IoT). In particular within the Insurance industry, IoT appears positioned to enable deep changes by altering relationships between insurers, distributors, and the insured. In his session at @ThingsExpo, Michael Sick, a Senior Manager and Big Data Architect within Ernst and Young's Financial Servi...
PubNub on Monday has announced that it is partnering with IBM to bring its sophisticated real-time data streaming and messaging capabilities to Bluemix, IBM’s cloud development platform. “Today’s app and connected devices require an always-on connection, but building a secure, scalable solution from the ground up is time consuming, resource intensive, and error-prone,” said Todd Greene, CEO of PubNub. “PubNub enables web, mobile and IoT developers building apps on IBM Bluemix to quickly add scalable realtime functionality with minimal effort and cost.”
The Internet of Things (IoT) is rapidly in the process of breaking from its heretofore relatively obscure enterprise applications (such as plant floor control and supply chain management) and going mainstream into the consumer space. More and more creative folks are interconnecting everyday products such as household items, mobile devices, appliances and cars, and unleashing new and imaginative scenarios. We are seeing a lot of excitement around applications in home automation, personal fitness, and in-car entertainment and this excitement will bleed into other areas. On the commercial side, m...
Sensor-enabled things are becoming more commonplace, precursors to a larger and more complex framework that most consider the ultimate promise of the IoT: things connecting, interacting, sharing, storing, and over time perhaps learning and predicting based on habits, behaviors, location, preferences, purchases and more. In his session at @ThingsExpo, Tom Wesselman, Director of Communications Ecosystem Architecture at Plantronics, will examine the still nascent IoT as it is coalescing, including what it is today, what it might ultimately be, the role of wearable tech, and technology gaps stil...
In the consumer IoT, everything is new, and the IT world of bits and bytes holds sway. But industrial and commercial realms encompass operational technology (OT) that has been around for 25 or 50 years. This grittier, pre-IP, more hands-on world has much to gain from Industrial IoT (IIoT) applications and principles. But adding sensors and wireless connectivity won’t work in environments that demand unwavering reliability and performance. In his session at @ThingsExpo, Ron Sege, CEO of Echelon, will discuss how as enterprise IT embraces other IoT-related technology trends, enterprises with i...
When it comes to the Internet of Things, hooking up will get you only so far. If you want customers to commit, you need to go beyond simply connecting products. You need to use the devices themselves to transform how you engage with every customer and how you manage the entire product lifecycle. In his session at @ThingsExpo, Sean Lorenz, Technical Product Manager for Xively at LogMeIn, will show how “product relationship management” can help you leverage your connected devices and the data they generate about customer usage and product performance to deliver extremely compelling and reliabl...
The Internet of Things (IoT) is causing data centers to become radically decentralized and atomized within a new paradigm known as “fog computing.” To support IoT applications, such as connected cars and smart grids, data centers' core functions will be decentralized out to the network's edges and endpoints (aka “fogs”). As this trend takes hold, Big Data analytics platforms will focus on high-volume log analysis (aka “logs”) and rely heavily on cognitive-computing algorithms (aka “cogs”) to make sense of it all.
With several hundred implementations of IoT-enabled solutions in the past 12 months alone, this session will focus on experience over the art of the possible. Many can only imagine the most advanced telematics platform ever deployed, supporting millions of customers, producing tens of thousands events or GBs per trip, and hundreds of TBs per month. With the ability to support a billion sensor events per second, over 30PB of warm data for analytics, and hundreds of PBs for an data analytics archive, in his session at @ThingsExpo, Jim Kaskade, Vice President and General Manager, Big Data & Ana...
One of the biggest impacts of the Internet of Things is and will continue to be on data; specifically data volume, management and usage. Companies are scrambling to adapt to this new and unpredictable data reality with legacy infrastructure that cannot handle the speed and volume of data. In his session at @ThingsExpo, Don DeLoach, CEO and president of Infobright, will discuss how companies need to rethink their data infrastructure to participate in the IoT, including: Data storage: Understanding the kinds of data: structured, unstructured, big/small? Analytics: What kinds and how responsiv...
The Workspace-as-a-Service (WaaS) market will grow to $6.4B by 2018. In his session at 16th Cloud Expo, Seth Bostock, CEO of IndependenceIT, will begin by walking the audience through the evolution of Workspace as-a-Service, where it is now vs. where it going. To look beyond the desktop we must understand exactly what WaaS is, who the users are, and where it is going in the future. IT departments, ISVs and service providers must look to workflow and automation capabilities to adapt to growing demand and the rapidly changing workspace model.
Sensor-enabled things are becoming more commonplace, precursors to a larger and more complex framework that most consider the ultimate promise of the IoT: things connecting, interacting, sharing, storing, and over time perhaps learning and predicting based on habits, behaviors, location, preferences, purchases and more. In his session at @ThingsExpo, Tom Wesselman, Director of Communications Ecosystem Architecture at Plantronics, will examine the still nascent IoT as it is coalescing, including what it is today, what it might ultimately be, the role of wearable tech, and technology gaps stil...