|By Marten Terpstra||
|January 10, 2014 10:00 AM EST||
For as long as I remember networking has struggled with the balance between aggregated and individual traffic flows. Following the abilities of the technology components we use, we have been forced to aggregate, only to be allowed to de-aggregate or skip aggregation when technology caught up or surpassed the needs of today.
The vast majority of networking equipment is driven by specialized hardware. For datacenter switches, speed and port density are driving the requirements and physics and our technology capabilities create trade-offs that ultimately lead to some form of aggregation. Higher speed and more ports are traded off against memory, table space and functionality. These trade-offs will always exist, no matter what we are trying to build. Networking based in servers will have oodles of memory and table space to do very specific things for many many flows, making it extremely flexible, but those same servers cannot touch the packet processing speeds of the specialized packet processing hardware from Broadcom, Intel or Marvell, or the custom ASICs from Cisco, Juniper, or most anyone else.
So like it or not, we will want to do more than our hardware is capable of and as a result, we create aggregation points in the network where we lump a bunch of flows together into an aggregate flow and start making decisions on those. Nothing new, even good ole IP forwarding is doing so on an aggregate set of flows, it only makes decisions for all flows destined to a specific IP address.
Network tunnels are the most obvious examples of aggregation, their purpose is to hide information from intermediate networking equipment. In some cases we hide it to keep our table sizes under control, in some cases we hide it because we do not want the intermediate equipment to be able to see what we are transporting (IPSec, SSL, etc). And while sometimes the intermediate systems can see everything that is there, managing the complexity of that visibility simply becomes too expensive. This is why networks that are entirely managed and controlled per flow do not really exist at any reasonable scale, and probably never will.
For the exact same reason we aggregate, we lose the ability to act on specifics. When our tables are not large enough to track each and every flow, we can only make decisions based on what we have decided to keep in common. When talking about tunnels, the tunnel endpoints put new headers onto the original packets and intermediate systems can only act (with minor exceptions) on the information provided in these new headers. The original detail is still there and often visible to the intermediate system, but the intermediate system does not have the capacity to act on the sheer volume of that detail.
And there is the struggle. If I have more information, I can make better decisions. But when I aggregate because I cannot handle that extra information (due to sheer size or management complexity), my decisions by definition become more coarse and as a result, less accurate. But we want it all. We want the power to make decisions based on the most specific information we can, but want to aggregate for operational simplicity or because our hardware dictates. And this is where we get creative and start to turn what used to be black and white into gray.
There is nothing wrong with attempting to act on specifics for aggregate flows, but in so many cases its done as an afterthought and becomes hard to manage, control or specify. Some of the techniques we use are fairly clean, like taking the DSCP values from a packet and replicating it in the outer header of that same packet in a tunnel. Others are far more obscure like calculating some hash function on a packet header and using it as the UDP source port for the VXLAN encapsulated version of that packet. In even others, the original internals may be completely invisible to intermediate systems. STT for instance re-uses the format of TCP packets for its own purpose, but as a side effect of using it as a streaming-like protocol is that the original packet headers may not actually be in an IP packet on the wire. The STT header provides for a 64 bit Context-ID that can be used to take some bits of information from the original packet, but that STT header only appears in the first of what could be many individual packets that are re-assembled in the receiving NIC. Over the Christmas break I spent some time looking at each of the overlay formats and the tools modern day packet processors give you to act on these headers. I will share some of this in this forum next week.
Ultimately, overlay networks are creating a renewed emphasis on the choices between aggregation and individuality. Designed specifically to allow for more complex and scaled networks that hide a lot of the details from the dedicated network hardware, it comes with the price of less granular decisions by that hardware, which can certainly lead to less than optimal use of the available network.
[Today's fun fact: In the Netherlands, there is a 40% higher chance of homeowner insurance claims on the home owner's birthday. Those are some good parties.]
Data is an unusual currency; it is not restricted by the same transactional limitations as money or people. In fact, the more that you leverage your data across multiple business use cases, the more valuable it becomes to the organization. And the same can be said about the organization’s analytics. In his session at 19th Cloud Expo, Bill Schmarzo, CTO for the Big Data Practice at EMC, will introduce a methodology for capturing, enriching and sharing data (and analytics) across the organizati...
Sep. 24, 2016 09:45 PM EDT Reads: 1,586
DevOps at Cloud Expo, taking place Nov 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long dev...
Sep. 24, 2016 09:30 PM EDT Reads: 3,317
I'm a lonely sensor. I spend all day telling the world how I'm feeling, but none of the other sensors seem to care. I want to be connected. I want to build relationships with other sensors to be more useful for my human. I want my human to understand that when my friends next door are too hot for a while, I'll soon be flaming. And when all my friends go outside without me, I may be left behind. Don't just log my data; use the relationship graph. In his session at @ThingsExpo, Ryan Boyd, Engi...
Sep. 24, 2016 09:15 PM EDT Reads: 1,190
The vision of a connected smart home is becoming reality with the application of integrated wireless technologies in devices and appliances. The use of standardized and TCP/IP networked wireless technologies in line-powered and battery operated sensors and controls has led to the adoption of radios in the 2.4GHz band, including Wi-Fi, BT/BLE and 802.15.4 applied ZigBee and Thread. This is driving the need for robust wireless coexistence for multiple radios to ensure throughput performance and th...
Sep. 24, 2016 08:30 PM EDT Reads: 1,434
The Internet of Things can drive efficiency for airlines and airports. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Sudip Majumder, senior director of development at Oracle, will discuss the technical details of the connected airline baggage and related social media solutions. These IoT applications will enhance travelers' journey experience and drive efficiency for the airlines and the airports. The session will include a working demo and a technical d...
Sep. 24, 2016 08:00 PM EDT Reads: 1,644
SYS-CON Events announced today the Enterprise IoT Bootcamp, being held November 1-2, 2016, in conjunction with 19th Cloud Expo | @ThingsExpo at the Santa Clara Convention Center in Santa Clara, CA. Combined with real-world scenarios and use cases, the Enterprise IoT Bootcamp is not just based on presentations but with hands-on demos and detailed walkthroughs. We will introduce you to a variety of real world use cases prototyped using Arduino, Raspberry Pi, BeagleBone, Spark, and Intel Edison. Y...
Sep. 24, 2016 07:00 PM EDT Reads: 2,778
Fact is, enterprises have significant legacy voice infrastructure that’s costly to replace with pure IP solutions. How can we bring this analog infrastructure into our shiny new cloud applications? There are proven methods to bind both legacy voice applications and traditional PSTN audio into cloud-based applications and services at a carrier scale. Some of the most successful implementations leverage WebRTC, WebSockets, SIP and other open source technologies. In his session at @ThingsExpo, Da...
Sep. 24, 2016 06:45 PM EDT Reads: 1,469
If you’re responsible for an application that depends on the data or functionality of various IoT endpoints – either sensors or devices – your brand reputation depends on the security, reliability, and compliance of its many integrated parts. If your application fails to deliver the expected business results, your customers and partners won't care if that failure stems from the code you developed or from a component that you integrated. What can you do to ensure that the endpoints work as expect...
Sep. 24, 2016 04:30 PM EDT Reads: 1,502
Information technology is an industry that has always experienced change, and the dramatic change sweeping across the industry today could not be truthfully described as the first time we've seen such widespread change impacting customer investments. However, the rate of the change, and the potential outcomes from today's digital transformation has the distinct potential to separate the industry into two camps: Organizations that see the change coming, embrace it, and successful leverage it; and...
Sep. 24, 2016 03:00 PM EDT Reads: 1,023
SYS-CON Events announced today that China Unicom will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. China United Network Communications Group Co. Ltd ("China Unicom") was officially established in 2009 on the basis of the merger of former China Netcom and former China Unicom. China Unicom mainly operates a full range of telecommunications services including mobile broadband (GSM, WCDMA, LTE F...
Sep. 24, 2016 01:30 PM EDT Reads: 1,673
Enterprise IT has been in the era of Hybrid Cloud for some time now. But it seems most conversations about Hybrid are focused on integrating AWS, Microsoft Azure, or Google ECM into existing on-premises systems. Where is all the Private Cloud? What do technology providers need to do to make their offerings more compelling? How should enterprise IT executives and buyers define their focus, needs, and roadmap, and communicate that clearly to the providers?
Sep. 24, 2016 01:00 PM EDT Reads: 1,480
The Transparent Cloud-computing Consortium (abbreviation: T-Cloud Consortium) will conduct research activities into changes in the computing model as a result of collaboration between "device" and "cloud" and the creation of new value and markets through organic data processing High speed and high quality networks, and dramatic improvements in computer processing capabilities, have greatly changed the nature of applications and made the storing and processing of data on the network commonplace.
Sep. 24, 2016 12:00 PM EDT Reads: 750
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
Sep. 24, 2016 12:00 PM EDT Reads: 761
Digital innovation is the next big wave of business transformation based on digital technologies of which IoT and Big Data are key components, For example: Business boundary innovation is a challenge to excavate third-party business value using IoT and BigData, like Nest Business structure innovation may propose re-building business structure from scratch, as Uber does in the taxicab industry The social model innovation is also a big challenge to the new social architecture with the design fr...
Sep. 24, 2016 11:45 AM EDT Reads: 1,026
SYS-CON Events announced today that Pulzze Systems will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Pulzze Systems, Inc. provides infrastructure products for the Internet of Things to enable any connected device and system to carry out matched operations without programming. For more information, visit http://www.pulzzesystems.com.
Sep. 24, 2016 10:45 AM EDT Reads: 1,719
IoT is fundamentally transforming the auto industry, turning the vehicle into a hub for connected services, including safety, infotainment and usage-based insurance. Auto manufacturers – and businesses across all verticals – have built an entire ecosystem around the Connected Car, creating new customer touch points and revenue streams. In his session at @ThingsExpo, Macario Namie, Head of IoT Strategy at Cisco Jasper, will share real-world examples of how IoT transforms the car from a static p...
Sep. 24, 2016 10:30 AM EDT Reads: 1,471
Big Data has been changing the world. IoT fuels the further transformation recently. How are Big Data and IoT related? In his session at @BigDataExpo, Tony Shan, a renowned visionary and thought leader, will explore the interplay of Big Data and IoT. He will anatomize Big Data and IoT separately in terms of what, which, why, where, when, who, how and how much. He will then analyze the relationship between IoT and Big Data, specifically the drilldown of how the 4Vs of Big Data (Volume, Variety,...
Sep. 24, 2016 10:00 AM EDT Reads: 908
The many IoT deployments around the world are busy integrating smart devices and sensors into their enterprise IT infrastructures. Yet all of this technology – and there are an amazing number of choices – is of no use without the software to gather, communicate, and analyze the new data flows. Without software, there is no IT. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the protocols that communicate data and the emerging data analy...
Sep. 24, 2016 09:00 AM EDT Reads: 1,522
Video experiences should be unique and exciting! But that doesn’t mean you need to patch all the pieces yourself. Users demand rich and engaging experiences and new ways to connect with you. But creating robust video applications at scale can be complicated, time-consuming and expensive. In his session at @ThingsExpo, Zohar Babin, Vice President of Platform, Ecosystem and Community at Kaltura, will discuss how VPaaS enables you to move fast, creating scalable video experiences that reach your...
Sep. 24, 2016 09:00 AM EDT Reads: 760
WebRTC adoption has generated a wave of creative uses of communications and collaboration through websites, sales apps, customer care and business applications. As WebRTC has become more mainstream it has evolved to use cases beyond the original peer-to-peer case, which has led to a repeating requirement for interoperability with existing infrastructures. In his session at @ThingsExpo, Graham Holt, Executive Vice President of Daitan Group, will cover implementation examples that have enabled ea...
Sep. 24, 2016 07:00 AM EDT Reads: 1,470