Welcome!

Cloud Security Authors: Elizabeth White, Xenia von Wedel, James Carlini, Pat Romanski, Rostyslav Demush

Related Topics: @DevOpsSummit, Microservices Expo, Linux Containers, Containers Expo Blog, Cloud Security

@DevOpsSummit: Article

An Interview with @JPaulReed | @DevOpsSummit #DevOps #ContinuousDelivery

Intersections: DevOps, Release Engineering, and Security

Intersections: DevOps, Release Engineering, and Security

Derek: Good morning, Paul. There's a lot those pursuing DevOps can learn from Release Engineering practices. I know you've got a lot of experience to share, so let's get started.

J. Paul Reed: Good morning, it's good to be here. My background is release engineering, although these days I am actually called a DevOps consultant. I have about 15 years' experience doing that. That's what my presentation is about: sort of the intersection between DevOps, Rugged DevOps, and release engineering and wanting to explore that with the security and Rugged DevOps communities.

Derek: In your presentation, you touched on the culture between security and DevOps and also release engineering -- that a number of organizations have challenges with and that's the Culture of No. There's a lot of, "Hey, we want to move faster at higher velocity. We have new requirements that we're trying to push out to market, and we have these new practices that we're moving forward with. Can security come and play with the DevOps team?"

J. Paul: I actually put up a tweet that a lot of people liked on one of my slides: "If your answer to every question is ‘no,' do not be surprised when people start pouring effort into ways to not even ask." This idea that if your answer to everything is "no," then that is seen as a bug or a blockage like on the Internet, and organizations will just route around it. I think security found that out in a very visceral, hard way. In release engineering, it's the same thing.

_________________________________

"If your answer to every question is ‘no,' do not be surprised when people start pouring effort into ways to not even ask."

_________________________________

One of the reasons that Git became so popular is because developers didn't have to ask if there were permission to create branches. They created an entire infrastructure and ecosystem around not having to ask. I think that's one of the risks we run, and it's one of the similarities.

One of the interesting things we're finding with DevOps ... because that idea of getting new traction and people do want to move faster ... is we can frame the work that we do in the context of that pipeline. By identifying and optimizing some of the business value that is part of that pipeline, businesses are receptive. Developers are receptive. Different parts of the business are receptive in ways I've almost never seen in my career, and it's great to be a part of that. From a Rugged DevOps or security perspective, I think if we could move that work into the pipeline, not only do we make it visible in terms of the costs and trade-offs, but then also we could possibly do more. It's part of that whole. There are lots of presentations talking about this idea of shift left ... that you can shift that work from your perspective further up into the stream so that you can address it earlier and actually have a chance at fixing the problem.

In talking with Josh Corman and a lot of the Rugged DevOps people, they always talk about how at the end of that process, they would rubber stamp: "Yes, this is secure." Because even if wasn't really secure and it was bad, what were you going to do? As a release engineer, that resonated with me because we felt like that all the time. We were kind of doing a bunch of work at the end, and there was no time to do it right. So a lot of times, it was skimped on.

Derek: When you think about the way traditional security works, how early can we think about Rugged DevOps shifting left?

J. Paul: Yeah, I don't think it's so much about getting everything right at the beginning, per se. I think that the question is how far forward can we shift into that process. I think if you can shift that all the way to the beginning, that is possible. The beginning is where you define your pipeline.

A lot of people define that pipeline as commits, that is developers writing code. Some people will define it actually at the product management stage, so even earlier than that. Or that kind of agile story phase, I think you could certainly integrate it there. This is sort of what I was exploring in my presentation. I open with the slide on what is the intersection of release engineering and Rugged DevOps, and I say I don't actually know. It's a very emergent field.

_________________________________

"There's no shortcuts to production...They put the financial resources and the engineering resources into building the pipeline that moves code quickly through it."

_______________________________

I spend the next few slides talking about sort of the crossover in making that bar. There are a lot of similarities there. I think when you're talking about pushing that stuff forward, it's about the more tools that you can make part of that pipeline, like release engineering tools. So for us, that might be something like: How do we track what developers create as dependencies in the work that they're doing? So how do we make that a little bit easier in time for them to say, "Yeah, I'm using this version of that, and it's integrated here from a release engineering perspective." Then from the security perspective, you can take that information and use it to do different types of security testing or penetration testing. If you can move that earlier in the process, that's what it will do. Then how early you do that really is a function of how good you get at this sort of thing.

I don't think we've seen this with security entirely yet. We're still recognizing the value with release engineering and companies are hitting it out of the park. They just put everything in there and continue delivering pipeline. There's no shortcuts to production. There's no back door to get stuff deployed. They put the financial resources and the engineering resources into building the pipeline that moves code quickly through it. Then once you do that, you can augment that pipeline with more and more features, if you will. One of those might be moving security way forward in that process.

Derek: Are there old ways to do things that just won't work in the new universe and you have to adopt new tools or practices?

J. Paul: I do hear a lot of, "Well, we can't do X because of Y" -- "Y" being one of those cut old ways that you're talking about. One of the things we continually see at conferences is idea of the answer being, "We can't do X because of the old way." In fact, in security, you see this all the time: "I can't do X because of audit compliant stuff." But case study after case study says: If you're willing to rethink the framing on the way you do audit compliance and work with your auditor -- if you're willing to look at the problem slightly differently -- then you can achieve those results. Because we have all this proof, when people say, "Oh, we can't do X because of the old way," my question is, "Are we thinking of the problem in an old frame or in a more traditional framing that is not sewn enough?"

Now that's not to imply the concerns people bring up are invalid. That's the initial question that you had, which was about people. If they have a lot of knowledge, they might be worried, "Well, I can't automate things this way as well as I can test them." I talked in my presentation about how release engineering is undergoing a fundamental shift. I'm very upfront about the fact that if you are a release engineer and you are not building continuous delivery pipeline and involved in the support and service of that continuous delivery pipeline, your job is probably not going to be there in five to 10 years. That's just the way the world works. A lot of people think, "Oh, okay, that's unfortunate or whatever."

_______________________________

"If you are a release engineer and you are not building continuous delivery pipeline and involved in the support and service of that continuous pipeline, your job is probably not going to be there in five to ten years"

_________________________________

I'll give you a QA example that I thought was really innovative.

Organizations spend a bunch of time automating a test and the initial response is, "Well, if you automate all of those tests, what are the QA engineers going to do?" It turns out that because QA engineers are so good at looking at a product and coming up with the requirements, they need a lot of that totally valuable knowledge forwarded into the value stream. They are having those QA engineers doing requirements analysis and working with product management to firm up the actual requirements that go into the continuous delivery pipeline. What was fascinating about it was that it's not that the organization was, "We are going to automate you out of a job and then we're going to fire you, so go automate yourself into a script." People are like, "I'm a person, not a machine." You have that whole conversation, and they end up doing more interesting work.

They put them working on that continuous delivery pipeline in the requirements analysis. It's totally different than what you might expect. It's going to be the same with security and release engineering. For security especially, we're going to see a lot of that work go. There's a set of compliance work you can do in an automated fashion. Once that is automated, I see a lot of discussion about red team, blue team ... kind of wargaming type of thing. And it frees up time to do that and to work as a team in that way. Because you can't automate all those things, or at least today you can't. I think everybody in the security space would agree that it's more interesting work than running around, if you've got a huge project, with a black binder with a bunch of rules.

Derek: One of the concepts that really resonated with you was the software supply chain. How does that concept fit with doing release engineering right and doing Rugged DevOps right or incorporating security into DevOps?

J. Paul: Yeah, the supply chain idea is something that was fascinating the first time I heard it. In fact, it's one of the things that Josh and I spent a bunch of time talking about it when we first met. I think it's a great way to frame a problem. I'm sad that I didn't think of it, actually, and the reason is because release engineers think about that all the time. We've thought that was our role for 20 to 30 years, for as long as release engineering has been around. It's this idea of knowing what the dependencies are, dependency management tracking and trying to make sure that you don't pull in bad dependencies -- whether they are tainted because of the license or containing malicious software. This problem has only gotten worse with open source software, and that's also something that from a supply chain perspective we talk a lot about.

_________________________________

"I told this story about an engineer who was missing a DLL from the build. They just Googled for the DDL and downloaded it, and threw it on all the build machines That was pretty scary."

_________________________________

That was one of the things that I wouldn't think keeps release engineers up at night as much as it keeps security engineers up at night. Where is our software coming from, and what issues may it have in it? That's not something traditionally developers, for whatever reason, seem to think about and that's not to denigrate them. A lot of times they're under deadlines, like we are. They go to the Internet. They grab whatever version of the library. In fact, the one I usually see is the upgraded version because there's some API that they need or something like that. There's a concern there, when you think about it, of where that's coming from. I told this story about an engineer who was a missing DLL from the build. They just Googled for the DLL and downloaded it, and threw it on all the build machines. That was pretty scary.

One of the slides in the presentation I think is really critical is: "If you have one vulnerable library in your product, that is a security problem. If you've got multiple versions of the same library and multiple versions of those are vulnerable, that's a release engineering problem." That's one of the best ways upfront that release engineers can contribute to Rugged DevOps and contribute to the security space in terms of helping to detangle that problem. More interestingly, once you've detangled that problem, you have to figure out how to make it so that that just doesn't turn into spaghetti again.

I've detangled that problem multiple times usually, by the way, not so much in a security context but in a licensing context.

_________________________________

"If you have one vulnerable library in your product, that is a security problem. If you've got multiple versions of the same library and multiple versions of those are vulnerable, that's a release engineering problem."

_________________________________

The way you do that, again, is shifting left. Moving that forward where you have a way that as developers put libraries into the product, new code that isn't written by them because there's a dependency there that was well documented. You can do that audit in kind of a continuous fashion so that maybe an artifact that you build is a list of library conversion. Then from an automated security testing perspective, we can compare that against a list of CD use or known issues.

Derek: I did a lot of research at Sonatype on the software supply chain and one stat boggles my mind. Out of the top 100 components companies were downloading, they downloaded an average of 27 versions of each of those components in a single year. When you think about the complexity and the technical debt, and if there's security debt in that at all ... you only need 100 parts and yet you're using 2,700 parts. Why would you ever want to do that?

J. Paul: One thing I'll point out is that I think the industry's moving in some sense in the wrong direction. What mean by that is you've got your Java you've built in this in to make it really, really, really easy. From the command line, you just pick up libraries from the Internet. Who knows where they came from. Node makes this trivial. In fact, Node was built around npm, the package manager. All of that is online. In fact, it's even worse. One of the things I get called in to help with a lot these days is ... and I kind of giggle at this, just because of the dichotomy ... people were so interested in Git for so long because it was like offline Git, offline commit. It's great, right? You can build offline, and people always use the example of when I'm commuting home on the train, I can commit blah, blah, blah, and that was the big reason for doing it.

Now we've moved with Node and some of the tooling around Java so that our software builds literally require us to talk to the Internet to download packages. There's this big push for offline operations. But it's fine that no download needs 68 billion versions of libraries, and everything is "self-contained." But if you're going to look at a Node package, it's got versions of those things stocked in there. That's a feature, not a bug. Right? In certain platforms ... you see this with RubyGems, when the Ruby Gems site went down, nobody could deploy their web applications. That's a fundamentally broken engineering design in my opinion. Not that it's easy for developers to get that. But that our build processes, our deployment processes, rely on those things. And they rely on us as developers to say, "I want version 1.2.4 of that library, and that 1.2.4. is the same version that you use."

I posted a slide about versioning -- and that's a very release engineering problem. As an example, Open SSL made a mistake in their versioning and instead of bumping the version like they should have, they repackaged binary. I suspect the reason that they did that is because they published all the CVEs with that version number and everybody is like a hawk watching Open SSL. So they couldn't bump the version number easily. Open SSL can't be flexible in their release engineering anymore because they've been so traditionally horrible at it. Right? We've made it really easy to stuff all of those components into our products, but we really don't know what we're stuffing in there.

If you look at it, we end up worrying about a lot of the same things. I think a lot of the nuts to crack, if you will, in the Rugged DevOps community are maybe 50 to 80% release engineering problems. Strengthening that extra feature of security in there, to make that part of it, especially with the supply chain, will work really well.

_________________________________

"A lot of the nuts to crack, if you will, in the Rugged DevOps community are maybe 50 to 80% release engineering problems. Strengthening that extra feature of security in there, to make that part of it, especially with the supply chain, will work really well."

_________________________________

Derek: J. Paul Reed, thank you very much. It was a pleasure talking to you, I really enjoyed the conversation. We'll look forward to seeing you again soon.

J. Paul: Awesome. Thank you.

If you loved this interview and are looking for more great stuff on Rugged DevOps, I invite you to download this awesome research paper from Amy DeMartine at Forrester, "The Seven Habits of Rugged DevOps."

As Amy notes, "DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros' expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps."

More Stories By Derek Weeks

In 2015, Derek Weeks led the largest and most comprehensive analysis of software supply chain practices to date across 160,000 development organizations. He is a huge advocate of applying proven supply chain management principles into DevOps practices to improve efficiencies, reduce costs, and sustain long-lasting competitive advantages.

As a 20+ year veteran of the software industry, he has advised leading businesses on IT performance improvement practices covering continuous delivery, business process management, systems and network operations, service management, capacity planning and storage management. As the VP and DevOps Advocate for Sonatype, he is passionate about changing the way people think about software supply chains and improving public safety through improved software integrity. Follow him here @weekstweets, find me here www.linkedin.com/in/derekeweeks, and read me here http://blog.sonatype.com/author/weeks/.

@ThingsExpo Stories
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settle...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
BnkToTheFuture.com is the largest online investment platform for investing in FinTech, Bitcoin and Blockchain companies. We believe the future of finance looks very different from the past and we aim to invest and provide trading opportunities for qualifying investors that want to build a portfolio in the sector in compliance with international financial regulations.
Product connectivity goes hand and hand these days with increased use of personal data. New IoT devices are becoming more personalized than ever before. In his session at 22nd Cloud Expo | DXWorld Expo, Nicolas Fierro, CEO of MIMIR Blockchain Solutions, will discuss how in order to protect your data and privacy, IoT applications need to embrace Blockchain technology for a new level of product security never before seen - or needed.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
Imagine if you will, a retail floor so densely packed with sensors that they can pick up the movements of insects scurrying across a store aisle. Or a component of a piece of factory equipment so well-instrumented that its digital twin provides resolution down to the micrometer.
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
We are given a desktop platform with Java 8 or Java 9 installed and seek to find a way to deploy high-performance Java applications that use Java 3D and/or Jogl without having to run an installer. We are subject to the constraint that the applications be signed and deployed so that they can be run in a trusted environment (i.e., outside of the sandbox). Further, we seek to do this in a way that does not depend on bundling a JRE with our applications, as this makes downloads and installations rat...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
DX World EXPO, LLC, a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the time for a truly global DX event, to bring together the leading minds from the technology world in a conversation about Digital Transformation," he said in making the announcement.
In this strange new world where more and more power is drawn from business technology, companies are effectively straddling two paths on the road to innovation and transformation into digital enterprises. The first path is the heritage trail – with “legacy” technology forming the background. Here, extant technologies are transformed by core IT teams to provide more API-driven approaches. Legacy systems can restrict companies that are transitioning into digital enterprises. To truly become a lead...
Digital Transformation (DX) is not a "one-size-fits all" strategy. Each organization needs to develop its own unique, long-term DX plan. It must do so by realizing that we now live in a data-driven age, and that technologies such as Cloud Computing, Big Data, the IoT, Cognitive Computing, and Blockchain are only tools. In her general session at 21st Cloud Expo, Rebecca Wanta explained how the strategy must focus on DX and include a commitment from top management to create great IT jobs, monitor ...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
The IoT Will Grow: In what might be the most obvious prediction of the decade, the IoT will continue to expand next year, with more and more devices coming online every single day. What isn’t so obvious about this prediction: where that growth will occur. The retail, healthcare, and industrial/supply chain industries will likely see the greatest growth. Forrester Research has predicted the IoT will become “the backbone” of customer value as it continues to grow. It is no surprise that retail is ...