Welcome!

Cloud Security Authors: Liz McMillan, Scott Millis, Elizabeth White, Kevin Jackson, Doron Kolton

Related Topics: Microservices Expo, Java IoT, Linux Containers, Agile Computing, Cloud Security, @BigDataExpo

Microservices Expo: Article

Twitter’s Story: How Homegrown Load Testing Tools Can Misfire

If Twitter built their own tool to perform their own load test, why did the selfie cause their site to crash?

The selfie that changed the world, or at least Twitter, has been in the news for the past month. On March 2, 2014, the infamous Oscar selfie of Ellen and her celebrity friends broke a record of 2 million retweets before midnight the same night. That record was previously set by President Barack Obama, hugging first lady Michelle Obama after his 2012 re-election.

The selfie caused Twitter to crash for more than 20 minutes, also breaking the record for the longest crash of the social media site. Twitter was infamous for crashing in its early days (anyone remember "Fail Whale?"), so it's no wonder the social media giant worked extra hard to completely prepare their website infrastructure before going public in November 2013. This included building their own load testing tool, Iago, in 2012.

If they built their own tool to perform their own load test, why did the selfie cause their site to crash? The Oscar selfie crash is a perfect example of what companies can easily overlook. Twitter did not test their users properly and their homegrown tool clearly doesn't solve all of their problems... their servers still crash.

What is Iago and why did Twitter decide to make it? And what does that have to do with your decision to use homegrown tools versus vendor load testing tools? Don't worry, we will tell you.

Twitter's Homegrown Load Testing Tool: Iago
Iago was created in June of 2012 by Twitter's internal engineering team. According to Twitter, Iago is a load generator created to help the social media site test services before they encounter production traffic. Chris Aniszczyk, Head of Open Source at Twitter, said, "There are many load generators available in the open source and commercial software worlds, but Iago provided us with capabilities that are uniquely suited for Twitter's environment and the precise degree to which we need to test our services."

Basically their homegrown tool was completely customized for their platform alone - a very attractive aspect of developing your own tool.

The three attributes Twitter focused on in creating Iago were:

  1. High performance: Iago was designed to generate traffic in a precise and predictable way, to minimize variance between test runs and allow comparisons to be made between development iterations.
  2. Multi-protocol: Modeling a system as complex as Twitter can be difficult, but it's made easier by decomposing it into component services. Once decomposed, each piece can be tested in isolation; which requires the load generator to speak each service's protocol. Twitter has in excess of 100 such services, and Iago tests most of them using built-in support for the internal protocols Twitter uses.
  3. Extensible: Iago is designed for engineers. It assumes the person building the system will also be interested in validating performance. As such, the tool is designed from the ground up to be extensible - making it easy to generate new traffic types, over new protocols and with individualized traffic sources.

Why Twitter Couldn't Handle Ellen's Selfie
If we were to do the math, Iago was up and running for nearly two years before the Oscar selfie. What happened to their load testing tool?

There were two main reasons why Twitter crashed. First, the tweet Ellen posted was a picture. On Twitter, a tweet accounts for only 260 bytes of data while a picture on Twitter accounts for 33KB of data, almost 130 times as much as a tweet. Second, Twitter's distributed server system was already at max capacity so the load taken on by the website couldn't be distributed to any nearby servers.

Twitter made one major mistake contributing to the crash back in March: they didn't anticipate and replicate real user activity. Most likely Iago wasn't instructed to generate a load based on a picture being retweeted millions of times, thus Twitter didn't know what to expect when the Oscars rolled around.

Homegrown Tool vs. Vendor Load Testing Solutions
Twitter was looking for a DIY homegrown solution because of their unique platform, and while most load testing tools seek to accomplish the same goal, there are always differences between tools. Here are some of the differences we see between homegrown and vendor-provided load testing tools.

Homegrown

Vendor

1. You can customize it - A homegrown tool, created completely from scratch, means you can build exactly what you think you need. But you don't get the benefit of lots of other people's experiences - so what you think you need may not turn out to be what you actually need.

2. Homegrown tools are free like a puppy is free - We have mentioned this analogy before in another post, and it needs to be brought up again. Homegrown tools aren't really free. You have to pay a team to keep them up and running.

3. What if someone leaves? - A couple of developers are experts on the tool they built, but what happens when they leave? Who is going to be your expert? If the new person isn't properly trained and if the code isn't well documented, then modifications to the code become extremely difficult.

1. Your vendor makes it extensible - A vendor that supports lots of users is constantly adding features and capabilities to support needs you may not have realized you have yet. With extensible APIs and SDKs, you still have the ability to add integrations and the protocol support your application demands.

2. You pay for something that works - With homegrown tools there is no guarantee that it will work 100% of the time. By paying a vendor to use their tool, you have assurances that your load testing tool won't let you down when you need it most.

3. Support Team - Vendors dedicate people to work on any issues and keep you worry-free. Their team of experts is available to make sure all of your questions and concerns are addressed.

Homegrown Tools Are Not for Everyone
Twitter's tool has done well to help them improve the performance of the service, but continued service disruptions show that it isn't perfect. A homegrown tool isn't always the best solution, and most of the time, companies do not have the resources handy to completely build their load testing tool from scratch. If you are interested in learning more about vendor options, give us a call. We're more than happy to help.

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
Internet-of-Things discussions can end up either going down the consumer gadget rabbit hole or focused on the sort of data logging that industrial manufacturers have been doing forever. However, in fact, companies today are already using IoT data both to optimize their operational technology and to improve the experience of customer interactions in novel ways. In his session at @ThingsExpo, Gordon Haff, Red Hat Technology Evangelist, will share examples from a wide range of industries – includin...
We're entering the post-smartphone era, where wearable gadgets from watches and fitness bands to glasses and health aids will power the next technological revolution. With mass adoption of wearable devices comes a new data ecosystem that must be protected. Wearables open new pathways that facilitate the tracking, sharing and storing of consumers’ personal health, location and daily activity data. Consumers have some idea of the data these devices capture, but most don’t realize how revealing and...
Unless your company can spend a lot of money on new technology, re-engineering your environment and hiring a comprehensive cybersecurity team, you will most likely move to the cloud or seek external service partnerships. In his session at 18th Cloud Expo, Darren Guccione, CEO of Keeper Security, revealed what you need to know when it comes to encryption in the cloud.
"We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at 20th Cloud Expo, Ed Featherston, director/senior enterprise architect at Collaborative Consulting, will discuss the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
In addition to all the benefits, IoT is also bringing new kind of customer experience challenges - cars that unlock themselves, thermostats turning houses into saunas and baby video monitors broadcasting over the internet. This list can only increase because while IoT services should be intuitive and simple to use, the delivery ecosystem is a myriad of potential problems as IoT explodes complexity. So finding a performance issue is like finding the proverbial needle in the haystack.
According to Forrester Research, every business will become either a digital predator or digital prey by 2020. To avoid demise, organizations must rapidly create new sources of value in their end-to-end customer experiences. True digital predators also must break down information and process silos and extend digital transformation initiatives to empower employees with the digital resources needed to win, serve, and retain customers.
"Once customers get a year into their IoT deployments, they start to realize that they may have been shortsighted in the ways they built out their deployment and the key thing I see a lot of people looking at is - how can I take equipment data, pull it back in an IoT solution and show it in a dashboard," stated Dave McCarthy, Director of Products at Bsquare Corporation, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Fact is, enterprises have significant legacy voice infrastructure that’s costly to replace with pure IP solutions. How can we bring this analog infrastructure into our shiny new cloud applications? There are proven methods to bind both legacy voice applications and traditional PSTN audio into cloud-based applications and services at a carrier scale. Some of the most successful implementations leverage WebRTC, WebSockets, SIP and other open source technologies. In his session at @ThingsExpo, Da...
@GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Onalytica. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like “How is my application doing” but no id...
As data explodes in quantity, importance and from new sources, the need for managing and protecting data residing across physical, virtual, and cloud environments grow with it. Managing data includes protecting it, indexing and classifying it for true, long-term management, compliance and E-Discovery. Commvault can ensure this with a single pane of glass solution – whether in a private cloud, a Service Provider delivered public cloud or a hybrid cloud environment – across the heterogeneous enter...
"IoT is going to be a huge industry with a lot of value for end users, for industries, for consumers, for manufacturers. How can we use cloud to effectively manage IoT applications," stated Ian Khan, Innovation & Marketing Manager at Solgeniakhela, in this SYS-CON.tv interview at @ThingsExpo, held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
"We're a cybersecurity firm that specializes in engineering security solutions both at the software and hardware level. Security cannot be an after-the-fact afterthought, which is what it's become," stated Richard Blech, Chief Executive Officer at Secure Channels, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Information technology is an industry that has always experienced change, and the dramatic change sweeping across the industry today could not be truthfully described as the first time we've seen such widespread change impacting customer investments. However, the rate of the change, and the potential outcomes from today's digital transformation has the distinct potential to separate the industry into two camps: Organizations that see the change coming, embrace it, and successful leverage it; and...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
We are always online. We access our data, our finances, work, and various services on the Internet. But we live in a congested world of information in which the roads were built two decades ago. The quest for better, faster Internet routing has been around for a decade, but nobody solved this problem. We’ve seen band-aid approaches like CDNs that attack a niche's slice of static content part of the Internet, but that’s it. It does not address the dynamic services-based Internet of today. It does...
Internet of @ThingsExpo, taking place June 6-8, 2017 at the Javits Center in New York City, New York, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @ThingsExpo New York Call for Papers is now open.
What happens when the different parts of a vehicle become smarter than the vehicle itself? As we move toward the era of smart everything, hundreds of entities in a vehicle that communicate with each other, the vehicle and external systems create a need for identity orchestration so that all entities work as a conglomerate. Much like an orchestra without a conductor, without the ability to secure, control, and connect the link between a vehicle’s head unit, devices, and systems and to manage the ...