Welcome!

Cloud Security Authors: Mehdi Daoudi, Elizabeth White, John Walsh, Liz McMillan, Kevin Jackson

Related Topics: @DevOpsSummit, Microservices Expo, Linux Containers, Containers Expo Blog, Cloud Security

@DevOpsSummit: Article

An Interview with @JPaulReed | @DevOpsSummit #DevOps #ContinuousDelivery

Intersections: DevOps, Release Engineering, and Security

Intersections: DevOps, Release Engineering, and Security

Derek: Good morning, Paul. There's a lot those pursuing DevOps can learn from Release Engineering practices. I know you've got a lot of experience to share, so let's get started.

J. Paul Reed: Good morning, it's good to be here. My background is release engineering, although these days I am actually called a DevOps consultant. I have about 15 years' experience doing that. That's what my presentation is about: sort of the intersection between DevOps, Rugged DevOps, and release engineering and wanting to explore that with the security and Rugged DevOps communities.

Derek: In your presentation, you touched on the culture between security and DevOps and also release engineering -- that a number of organizations have challenges with and that's the Culture of No. There's a lot of, "Hey, we want to move faster at higher velocity. We have new requirements that we're trying to push out to market, and we have these new practices that we're moving forward with. Can security come and play with the DevOps team?"

J. Paul: I actually put up a tweet that a lot of people liked on one of my slides: "If your answer to every question is ‘no,' do not be surprised when people start pouring effort into ways to not even ask." This idea that if your answer to everything is "no," then that is seen as a bug or a blockage like on the Internet, and organizations will just route around it. I think security found that out in a very visceral, hard way. In release engineering, it's the same thing.

_________________________________

"If your answer to every question is ‘no,' do not be surprised when people start pouring effort into ways to not even ask."

_________________________________

One of the reasons that Git became so popular is because developers didn't have to ask if there were permission to create branches. They created an entire infrastructure and ecosystem around not having to ask. I think that's one of the risks we run, and it's one of the similarities.

One of the interesting things we're finding with DevOps ... because that idea of getting new traction and people do want to move faster ... is we can frame the work that we do in the context of that pipeline. By identifying and optimizing some of the business value that is part of that pipeline, businesses are receptive. Developers are receptive. Different parts of the business are receptive in ways I've almost never seen in my career, and it's great to be a part of that. From a Rugged DevOps or security perspective, I think if we could move that work into the pipeline, not only do we make it visible in terms of the costs and trade-offs, but then also we could possibly do more. It's part of that whole. There are lots of presentations talking about this idea of shift left ... that you can shift that work from your perspective further up into the stream so that you can address it earlier and actually have a chance at fixing the problem.

In talking with Josh Corman and a lot of the Rugged DevOps people, they always talk about how at the end of that process, they would rubber stamp: "Yes, this is secure." Because even if wasn't really secure and it was bad, what were you going to do? As a release engineer, that resonated with me because we felt like that all the time. We were kind of doing a bunch of work at the end, and there was no time to do it right. So a lot of times, it was skimped on.

Derek: When you think about the way traditional security works, how early can we think about Rugged DevOps shifting left?

J. Paul: Yeah, I don't think it's so much about getting everything right at the beginning, per se. I think that the question is how far forward can we shift into that process. I think if you can shift that all the way to the beginning, that is possible. The beginning is where you define your pipeline.

A lot of people define that pipeline as commits, that is developers writing code. Some people will define it actually at the product management stage, so even earlier than that. Or that kind of agile story phase, I think you could certainly integrate it there. This is sort of what I was exploring in my presentation. I open with the slide on what is the intersection of release engineering and Rugged DevOps, and I say I don't actually know. It's a very emergent field.

_________________________________

"There's no shortcuts to production...They put the financial resources and the engineering resources into building the pipeline that moves code quickly through it."

_______________________________

I spend the next few slides talking about sort of the crossover in making that bar. There are a lot of similarities there. I think when you're talking about pushing that stuff forward, it's about the more tools that you can make part of that pipeline, like release engineering tools. So for us, that might be something like: How do we track what developers create as dependencies in the work that they're doing? So how do we make that a little bit easier in time for them to say, "Yeah, I'm using this version of that, and it's integrated here from a release engineering perspective." Then from the security perspective, you can take that information and use it to do different types of security testing or penetration testing. If you can move that earlier in the process, that's what it will do. Then how early you do that really is a function of how good you get at this sort of thing.

I don't think we've seen this with security entirely yet. We're still recognizing the value with release engineering and companies are hitting it out of the park. They just put everything in there and continue delivering pipeline. There's no shortcuts to production. There's no back door to get stuff deployed. They put the financial resources and the engineering resources into building the pipeline that moves code quickly through it. Then once you do that, you can augment that pipeline with more and more features, if you will. One of those might be moving security way forward in that process.

Derek: Are there old ways to do things that just won't work in the new universe and you have to adopt new tools or practices?

J. Paul: I do hear a lot of, "Well, we can't do X because of Y" -- "Y" being one of those cut old ways that you're talking about. One of the things we continually see at conferences is idea of the answer being, "We can't do X because of the old way." In fact, in security, you see this all the time: "I can't do X because of audit compliant stuff." But case study after case study says: If you're willing to rethink the framing on the way you do audit compliance and work with your auditor -- if you're willing to look at the problem slightly differently -- then you can achieve those results. Because we have all this proof, when people say, "Oh, we can't do X because of the old way," my question is, "Are we thinking of the problem in an old frame or in a more traditional framing that is not sewn enough?"

Now that's not to imply the concerns people bring up are invalid. That's the initial question that you had, which was about people. If they have a lot of knowledge, they might be worried, "Well, I can't automate things this way as well as I can test them." I talked in my presentation about how release engineering is undergoing a fundamental shift. I'm very upfront about the fact that if you are a release engineer and you are not building continuous delivery pipeline and involved in the support and service of that continuous delivery pipeline, your job is probably not going to be there in five to 10 years. That's just the way the world works. A lot of people think, "Oh, okay, that's unfortunate or whatever."

_______________________________

"If you are a release engineer and you are not building continuous delivery pipeline and involved in the support and service of that continuous pipeline, your job is probably not going to be there in five to ten years"

_________________________________

I'll give you a QA example that I thought was really innovative.

Organizations spend a bunch of time automating a test and the initial response is, "Well, if you automate all of those tests, what are the QA engineers going to do?" It turns out that because QA engineers are so good at looking at a product and coming up with the requirements, they need a lot of that totally valuable knowledge forwarded into the value stream. They are having those QA engineers doing requirements analysis and working with product management to firm up the actual requirements that go into the continuous delivery pipeline. What was fascinating about it was that it's not that the organization was, "We are going to automate you out of a job and then we're going to fire you, so go automate yourself into a script." People are like, "I'm a person, not a machine." You have that whole conversation, and they end up doing more interesting work.

They put them working on that continuous delivery pipeline in the requirements analysis. It's totally different than what you might expect. It's going to be the same with security and release engineering. For security especially, we're going to see a lot of that work go. There's a set of compliance work you can do in an automated fashion. Once that is automated, I see a lot of discussion about red team, blue team ... kind of wargaming type of thing. And it frees up time to do that and to work as a team in that way. Because you can't automate all those things, or at least today you can't. I think everybody in the security space would agree that it's more interesting work than running around, if you've got a huge project, with a black binder with a bunch of rules.

Derek: One of the concepts that really resonated with you was the software supply chain. How does that concept fit with doing release engineering right and doing Rugged DevOps right or incorporating security into DevOps?

J. Paul: Yeah, the supply chain idea is something that was fascinating the first time I heard it. In fact, it's one of the things that Josh and I spent a bunch of time talking about it when we first met. I think it's a great way to frame a problem. I'm sad that I didn't think of it, actually, and the reason is because release engineers think about that all the time. We've thought that was our role for 20 to 30 years, for as long as release engineering has been around. It's this idea of knowing what the dependencies are, dependency management tracking and trying to make sure that you don't pull in bad dependencies -- whether they are tainted because of the license or containing malicious software. This problem has only gotten worse with open source software, and that's also something that from a supply chain perspective we talk a lot about.

_________________________________

"I told this story about an engineer who was missing a DLL from the build. They just Googled for the DDL and downloaded it, and threw it on all the build machines That was pretty scary."

_________________________________

That was one of the things that I wouldn't think keeps release engineers up at night as much as it keeps security engineers up at night. Where is our software coming from, and what issues may it have in it? That's not something traditionally developers, for whatever reason, seem to think about and that's not to denigrate them. A lot of times they're under deadlines, like we are. They go to the Internet. They grab whatever version of the library. In fact, the one I usually see is the upgraded version because there's some API that they need or something like that. There's a concern there, when you think about it, of where that's coming from. I told this story about an engineer who was a missing DLL from the build. They just Googled for the DLL and downloaded it, and threw it on all the build machines. That was pretty scary.

One of the slides in the presentation I think is really critical is: "If you have one vulnerable library in your product, that is a security problem. If you've got multiple versions of the same library and multiple versions of those are vulnerable, that's a release engineering problem." That's one of the best ways upfront that release engineers can contribute to Rugged DevOps and contribute to the security space in terms of helping to detangle that problem. More interestingly, once you've detangled that problem, you have to figure out how to make it so that that just doesn't turn into spaghetti again.

I've detangled that problem multiple times usually, by the way, not so much in a security context but in a licensing context.

_________________________________

"If you have one vulnerable library in your product, that is a security problem. If you've got multiple versions of the same library and multiple versions of those are vulnerable, that's a release engineering problem."

_________________________________

The way you do that, again, is shifting left. Moving that forward where you have a way that as developers put libraries into the product, new code that isn't written by them because there's a dependency there that was well documented. You can do that audit in kind of a continuous fashion so that maybe an artifact that you build is a list of library conversion. Then from an automated security testing perspective, we can compare that against a list of CD use or known issues.

Derek: I did a lot of research at Sonatype on the software supply chain and one stat boggles my mind. Out of the top 100 components companies were downloading, they downloaded an average of 27 versions of each of those components in a single year. When you think about the complexity and the technical debt, and if there's security debt in that at all ... you only need 100 parts and yet you're using 2,700 parts. Why would you ever want to do that?

J. Paul: One thing I'll point out is that I think the industry's moving in some sense in the wrong direction. What mean by that is you've got your Java you've built in this in to make it really, really, really easy. From the command line, you just pick up libraries from the Internet. Who knows where they came from. Node makes this trivial. In fact, Node was built around npm, the package manager. All of that is online. In fact, it's even worse. One of the things I get called in to help with a lot these days is ... and I kind of giggle at this, just because of the dichotomy ... people were so interested in Git for so long because it was like offline Git, offline commit. It's great, right? You can build offline, and people always use the example of when I'm commuting home on the train, I can commit blah, blah, blah, and that was the big reason for doing it.

Now we've moved with Node and some of the tooling around Java so that our software builds literally require us to talk to the Internet to download packages. There's this big push for offline operations. But it's fine that no download needs 68 billion versions of libraries, and everything is "self-contained." But if you're going to look at a Node package, it's got versions of those things stocked in there. That's a feature, not a bug. Right? In certain platforms ... you see this with RubyGems, when the Ruby Gems site went down, nobody could deploy their web applications. That's a fundamentally broken engineering design in my opinion. Not that it's easy for developers to get that. But that our build processes, our deployment processes, rely on those things. And they rely on us as developers to say, "I want version 1.2.4 of that library, and that 1.2.4. is the same version that you use."

I posted a slide about versioning -- and that's a very release engineering problem. As an example, Open SSL made a mistake in their versioning and instead of bumping the version like they should have, they repackaged binary. I suspect the reason that they did that is because they published all the CVEs with that version number and everybody is like a hawk watching Open SSL. So they couldn't bump the version number easily. Open SSL can't be flexible in their release engineering anymore because they've been so traditionally horrible at it. Right? We've made it really easy to stuff all of those components into our products, but we really don't know what we're stuffing in there.

If you look at it, we end up worrying about a lot of the same things. I think a lot of the nuts to crack, if you will, in the Rugged DevOps community are maybe 50 to 80% release engineering problems. Strengthening that extra feature of security in there, to make that part of it, especially with the supply chain, will work really well.

_________________________________

"A lot of the nuts to crack, if you will, in the Rugged DevOps community are maybe 50 to 80% release engineering problems. Strengthening that extra feature of security in there, to make that part of it, especially with the supply chain, will work really well."

_________________________________

Derek: J. Paul Reed, thank you very much. It was a pleasure talking to you, I really enjoyed the conversation. We'll look forward to seeing you again soon.

J. Paul: Awesome. Thank you.

If you loved this interview and are looking for more great stuff on Rugged DevOps, I invite you to download this awesome research paper from Amy DeMartine at Forrester, "The Seven Habits of Rugged DevOps."

As Amy notes, "DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros' expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps."

More Stories By Derek Weeks

In 2015, Derek Weeks led the largest and most comprehensive analysis of software supply chain practices to date across 160,000 development organizations. He is a huge advocate of applying proven supply chain management principles into DevOps practices to improve efficiencies, reduce costs, and sustain long-lasting competitive advantages.

As a 20+ year veteran of the software industry, he has advised leading businesses on IT performance improvement practices covering continuous delivery, business process management, systems and network operations, service management, capacity planning and storage management. As the VP and DevOps Advocate for Sonatype, he is passionate about changing the way people think about software supply chains and improving public safety through improved software integrity. Follow him here @weekstweets, find me here www.linkedin.com/in/derekeweeks, and read me here http://blog.sonatype.com/author/weeks/.

@ThingsExpo Stories
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things’). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing? IoT is not about the devices, it’s about the data consumed and generated. The devices are tools, mechanisms, conduits. In his session at Internet of Things at Cloud Expo | DXWor...
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution. In his session at @ThingsExpo, Akvelon expert and IoT industry leader Sergey Grebnov provided an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone inn...