Welcome!

Cloud Security Authors: Liz McMillan, Pat Romanski, Zakia Bouachraoui, Elizabeth White, Yeshim Deniz

Related Topics: @DevOpsSummit, Microservices Expo, Linux Containers, Containers Expo Blog, Cloud Security

@DevOpsSummit: Article

An Interview with @JPaulReed | @DevOpsSummit #DevOps #ContinuousDelivery

Intersections: DevOps, Release Engineering, and Security

Intersections: DevOps, Release Engineering, and Security

Derek: Good morning, Paul. There's a lot those pursuing DevOps can learn from Release Engineering practices. I know you've got a lot of experience to share, so let's get started.

J. Paul Reed: Good morning, it's good to be here. My background is release engineering, although these days I am actually called a DevOps consultant. I have about 15 years' experience doing that. That's what my presentation is about: sort of the intersection between DevOps, Rugged DevOps, and release engineering and wanting to explore that with the security and Rugged DevOps communities.

Derek: In your presentation, you touched on the culture between security and DevOps and also release engineering -- that a number of organizations have challenges with and that's the Culture of No. There's a lot of, "Hey, we want to move faster at higher velocity. We have new requirements that we're trying to push out to market, and we have these new practices that we're moving forward with. Can security come and play with the DevOps team?"

J. Paul: I actually put up a tweet that a lot of people liked on one of my slides: "If your answer to every question is ‘no,' do not be surprised when people start pouring effort into ways to not even ask." This idea that if your answer to everything is "no," then that is seen as a bug or a blockage like on the Internet, and organizations will just route around it. I think security found that out in a very visceral, hard way. In release engineering, it's the same thing.

_________________________________

"If your answer to every question is ‘no,' do not be surprised when people start pouring effort into ways to not even ask."

_________________________________

One of the reasons that Git became so popular is because developers didn't have to ask if there were permission to create branches. They created an entire infrastructure and ecosystem around not having to ask. I think that's one of the risks we run, and it's one of the similarities.

One of the interesting things we're finding with DevOps ... because that idea of getting new traction and people do want to move faster ... is we can frame the work that we do in the context of that pipeline. By identifying and optimizing some of the business value that is part of that pipeline, businesses are receptive. Developers are receptive. Different parts of the business are receptive in ways I've almost never seen in my career, and it's great to be a part of that. From a Rugged DevOps or security perspective, I think if we could move that work into the pipeline, not only do we make it visible in terms of the costs and trade-offs, but then also we could possibly do more. It's part of that whole. There are lots of presentations talking about this idea of shift left ... that you can shift that work from your perspective further up into the stream so that you can address it earlier and actually have a chance at fixing the problem.

In talking with Josh Corman and a lot of the Rugged DevOps people, they always talk about how at the end of that process, they would rubber stamp: "Yes, this is secure." Because even if wasn't really secure and it was bad, what were you going to do? As a release engineer, that resonated with me because we felt like that all the time. We were kind of doing a bunch of work at the end, and there was no time to do it right. So a lot of times, it was skimped on.

Derek: When you think about the way traditional security works, how early can we think about Rugged DevOps shifting left?

J. Paul: Yeah, I don't think it's so much about getting everything right at the beginning, per se. I think that the question is how far forward can we shift into that process. I think if you can shift that all the way to the beginning, that is possible. The beginning is where you define your pipeline.

A lot of people define that pipeline as commits, that is developers writing code. Some people will define it actually at the product management stage, so even earlier than that. Or that kind of agile story phase, I think you could certainly integrate it there. This is sort of what I was exploring in my presentation. I open with the slide on what is the intersection of release engineering and Rugged DevOps, and I say I don't actually know. It's a very emergent field.

_________________________________

"There's no shortcuts to production...They put the financial resources and the engineering resources into building the pipeline that moves code quickly through it."

_______________________________

I spend the next few slides talking about sort of the crossover in making that bar. There are a lot of similarities there. I think when you're talking about pushing that stuff forward, it's about the more tools that you can make part of that pipeline, like release engineering tools. So for us, that might be something like: How do we track what developers create as dependencies in the work that they're doing? So how do we make that a little bit easier in time for them to say, "Yeah, I'm using this version of that, and it's integrated here from a release engineering perspective." Then from the security perspective, you can take that information and use it to do different types of security testing or penetration testing. If you can move that earlier in the process, that's what it will do. Then how early you do that really is a function of how good you get at this sort of thing.

I don't think we've seen this with security entirely yet. We're still recognizing the value with release engineering and companies are hitting it out of the park. They just put everything in there and continue delivering pipeline. There's no shortcuts to production. There's no back door to get stuff deployed. They put the financial resources and the engineering resources into building the pipeline that moves code quickly through it. Then once you do that, you can augment that pipeline with more and more features, if you will. One of those might be moving security way forward in that process.

Derek: Are there old ways to do things that just won't work in the new universe and you have to adopt new tools or practices?

J. Paul: I do hear a lot of, "Well, we can't do X because of Y" -- "Y" being one of those cut old ways that you're talking about. One of the things we continually see at conferences is idea of the answer being, "We can't do X because of the old way." In fact, in security, you see this all the time: "I can't do X because of audit compliant stuff." But case study after case study says: If you're willing to rethink the framing on the way you do audit compliance and work with your auditor -- if you're willing to look at the problem slightly differently -- then you can achieve those results. Because we have all this proof, when people say, "Oh, we can't do X because of the old way," my question is, "Are we thinking of the problem in an old frame or in a more traditional framing that is not sewn enough?"

Now that's not to imply the concerns people bring up are invalid. That's the initial question that you had, which was about people. If they have a lot of knowledge, they might be worried, "Well, I can't automate things this way as well as I can test them." I talked in my presentation about how release engineering is undergoing a fundamental shift. I'm very upfront about the fact that if you are a release engineer and you are not building continuous delivery pipeline and involved in the support and service of that continuous delivery pipeline, your job is probably not going to be there in five to 10 years. That's just the way the world works. A lot of people think, "Oh, okay, that's unfortunate or whatever."

_______________________________

"If you are a release engineer and you are not building continuous delivery pipeline and involved in the support and service of that continuous pipeline, your job is probably not going to be there in five to ten years"

_________________________________

I'll give you a QA example that I thought was really innovative.

Organizations spend a bunch of time automating a test and the initial response is, "Well, if you automate all of those tests, what are the QA engineers going to do?" It turns out that because QA engineers are so good at looking at a product and coming up with the requirements, they need a lot of that totally valuable knowledge forwarded into the value stream. They are having those QA engineers doing requirements analysis and working with product management to firm up the actual requirements that go into the continuous delivery pipeline. What was fascinating about it was that it's not that the organization was, "We are going to automate you out of a job and then we're going to fire you, so go automate yourself into a script." People are like, "I'm a person, not a machine." You have that whole conversation, and they end up doing more interesting work.

They put them working on that continuous delivery pipeline in the requirements analysis. It's totally different than what you might expect. It's going to be the same with security and release engineering. For security especially, we're going to see a lot of that work go. There's a set of compliance work you can do in an automated fashion. Once that is automated, I see a lot of discussion about red team, blue team ... kind of wargaming type of thing. And it frees up time to do that and to work as a team in that way. Because you can't automate all those things, or at least today you can't. I think everybody in the security space would agree that it's more interesting work than running around, if you've got a huge project, with a black binder with a bunch of rules.

Derek: One of the concepts that really resonated with you was the software supply chain. How does that concept fit with doing release engineering right and doing Rugged DevOps right or incorporating security into DevOps?

J. Paul: Yeah, the supply chain idea is something that was fascinating the first time I heard it. In fact, it's one of the things that Josh and I spent a bunch of time talking about it when we first met. I think it's a great way to frame a problem. I'm sad that I didn't think of it, actually, and the reason is because release engineers think about that all the time. We've thought that was our role for 20 to 30 years, for as long as release engineering has been around. It's this idea of knowing what the dependencies are, dependency management tracking and trying to make sure that you don't pull in bad dependencies -- whether they are tainted because of the license or containing malicious software. This problem has only gotten worse with open source software, and that's also something that from a supply chain perspective we talk a lot about.

_________________________________

"I told this story about an engineer who was missing a DLL from the build. They just Googled for the DDL and downloaded it, and threw it on all the build machines That was pretty scary."

_________________________________

That was one of the things that I wouldn't think keeps release engineers up at night as much as it keeps security engineers up at night. Where is our software coming from, and what issues may it have in it? That's not something traditionally developers, for whatever reason, seem to think about and that's not to denigrate them. A lot of times they're under deadlines, like we are. They go to the Internet. They grab whatever version of the library. In fact, the one I usually see is the upgraded version because there's some API that they need or something like that. There's a concern there, when you think about it, of where that's coming from. I told this story about an engineer who was a missing DLL from the build. They just Googled for the DLL and downloaded it, and threw it on all the build machines. That was pretty scary.

One of the slides in the presentation I think is really critical is: "If you have one vulnerable library in your product, that is a security problem. If you've got multiple versions of the same library and multiple versions of those are vulnerable, that's a release engineering problem." That's one of the best ways upfront that release engineers can contribute to Rugged DevOps and contribute to the security space in terms of helping to detangle that problem. More interestingly, once you've detangled that problem, you have to figure out how to make it so that that just doesn't turn into spaghetti again.

I've detangled that problem multiple times usually, by the way, not so much in a security context but in a licensing context.

_________________________________

"If you have one vulnerable library in your product, that is a security problem. If you've got multiple versions of the same library and multiple versions of those are vulnerable, that's a release engineering problem."

_________________________________

The way you do that, again, is shifting left. Moving that forward where you have a way that as developers put libraries into the product, new code that isn't written by them because there's a dependency there that was well documented. You can do that audit in kind of a continuous fashion so that maybe an artifact that you build is a list of library conversion. Then from an automated security testing perspective, we can compare that against a list of CD use or known issues.

Derek: I did a lot of research at Sonatype on the software supply chain and one stat boggles my mind. Out of the top 100 components companies were downloading, they downloaded an average of 27 versions of each of those components in a single year. When you think about the complexity and the technical debt, and if there's security debt in that at all ... you only need 100 parts and yet you're using 2,700 parts. Why would you ever want to do that?

J. Paul: One thing I'll point out is that I think the industry's moving in some sense in the wrong direction. What mean by that is you've got your Java you've built in this in to make it really, really, really easy. From the command line, you just pick up libraries from the Internet. Who knows where they came from. Node makes this trivial. In fact, Node was built around npm, the package manager. All of that is online. In fact, it's even worse. One of the things I get called in to help with a lot these days is ... and I kind of giggle at this, just because of the dichotomy ... people were so interested in Git for so long because it was like offline Git, offline commit. It's great, right? You can build offline, and people always use the example of when I'm commuting home on the train, I can commit blah, blah, blah, and that was the big reason for doing it.

Now we've moved with Node and some of the tooling around Java so that our software builds literally require us to talk to the Internet to download packages. There's this big push for offline operations. But it's fine that no download needs 68 billion versions of libraries, and everything is "self-contained." But if you're going to look at a Node package, it's got versions of those things stocked in there. That's a feature, not a bug. Right? In certain platforms ... you see this with RubyGems, when the Ruby Gems site went down, nobody could deploy their web applications. That's a fundamentally broken engineering design in my opinion. Not that it's easy for developers to get that. But that our build processes, our deployment processes, rely on those things. And they rely on us as developers to say, "I want version 1.2.4 of that library, and that 1.2.4. is the same version that you use."

I posted a slide about versioning -- and that's a very release engineering problem. As an example, Open SSL made a mistake in their versioning and instead of bumping the version like they should have, they repackaged binary. I suspect the reason that they did that is because they published all the CVEs with that version number and everybody is like a hawk watching Open SSL. So they couldn't bump the version number easily. Open SSL can't be flexible in their release engineering anymore because they've been so traditionally horrible at it. Right? We've made it really easy to stuff all of those components into our products, but we really don't know what we're stuffing in there.

If you look at it, we end up worrying about a lot of the same things. I think a lot of the nuts to crack, if you will, in the Rugged DevOps community are maybe 50 to 80% release engineering problems. Strengthening that extra feature of security in there, to make that part of it, especially with the supply chain, will work really well.

_________________________________

"A lot of the nuts to crack, if you will, in the Rugged DevOps community are maybe 50 to 80% release engineering problems. Strengthening that extra feature of security in there, to make that part of it, especially with the supply chain, will work really well."

_________________________________

Derek: J. Paul Reed, thank you very much. It was a pleasure talking to you, I really enjoyed the conversation. We'll look forward to seeing you again soon.

J. Paul: Awesome. Thank you.

If you loved this interview and are looking for more great stuff on Rugged DevOps, I invite you to download this awesome research paper from Amy DeMartine at Forrester, "The Seven Habits of Rugged DevOps."

As Amy notes, "DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros' expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps."

More Stories By Derek Weeks

In 2015, Derek Weeks led the largest and most comprehensive analysis of software supply chain practices to date across 160,000 development organizations. He is a huge advocate of applying proven supply chain management principles into DevOps practices to improve efficiencies, reduce costs, and sustain long-lasting competitive advantages.

As a 20+ year veteran of the software industry, he has advised leading businesses on IT performance improvement practices covering continuous delivery, business process management, systems and network operations, service management, capacity planning and storage management. As the VP and DevOps Advocate for Sonatype, he is passionate about changing the way people think about software supply chains and improving public safety through improved software integrity. Follow him here @weekstweets, find me here www.linkedin.com/in/derekeweeks, and read me here http://blog.sonatype.com/author/weeks/.

IoT & Smart Cities Stories
The deluge of IoT sensor data collected from connected devices and the powerful AI required to make that data actionable are giving rise to a hybrid ecosystem in which cloud, on-prem and edge processes become interweaved. Attendees will learn how emerging composable infrastructure solutions deliver the adaptive architecture needed to manage this new data reality. Machine learning algorithms can better anticipate data storms and automate resources to support surges, including fully scalable GPU-c...
Machine learning has taken residence at our cities' cores and now we can finally have "smart cities." Cities are a collection of buildings made to provide the structure and safety necessary for people to function, create and survive. Buildings are a pool of ever-changing performance data from large automated systems such as heating and cooling to the people that live and work within them. Through machine learning, buildings can optimize performance, reduce costs, and improve occupant comfort by ...
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
Predicting the future has never been more challenging - not because of the lack of data but because of the flood of ungoverned and risk laden information. Microsoft states that 2.5 exabytes of data are created every day. Expectations and reliance on data are being pushed to the limits, as demands around hybrid options continue to grow.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
As IoT continues to increase momentum, so does the associated risk. Secure Device Lifecycle Management (DLM) is ranked as one of the most important technology areas of IoT. Driving this trend is the realization that secure support for IoT devices provides companies the ability to deliver high-quality, reliable, secure offerings faster, create new revenue streams, and reduce support costs, all while building a competitive advantage in their markets. In this session, we will use customer use cases...