EPISODE SUMMARY
Subscribe or listen now: 
Joseph Carson:
Hi, everyone. Welcome back to 401 Access Denied Podcast. I'm here your cohost, Joe Carson, Chief Security Scientist at the Thycotic, based in very late, dark, evening in Estonia, or it's pretty cold outside. So, I'm happy to be inside here talking with you today. And of course, I'm joined with my cohost, Mike G. Do you want to give us a little bit of intro and what the topic is going to be and who we're going to be talking to today?
Mike Gruen:
Yeah. Mike Gruen, VP here at Cybrary and CISO, and today we're going to be talking with Ted. I'll let him introduce himself. Why don't you go ahead and introduce yourself. And we'll just get started.
Ted Harrington:
Yeah. Hey, everybody. Ted Harrington here. I'm the author of a best-selling book called Hackable, How to Do Application Security Right. And I am the executive partner at the security consulting company, Independent Security Evaluators.
Mike Gruen:
Welcome.
Joseph Carson:
Awesome. It's great to be here. And I think that's really important is that the big thing for today's show is really talking about application security and really for organizations and those around the world who really creating applications, delivering applications, and even testing applications is really to make sure that they actually doing it so that we actually think security first, security by design. And that's one of the critical things here to talk about. So, and it's awesome to be joined with Ted here, he was basically the specialist in the field and really interested to see his thoughts and opinions into what's the best way organizations can approach application security. So Ted, you want to give us a bit of understanding of the broader scope of application security and even the book Hackable and what it really approaches and some of the methodologies that it's really educating people on.
Ted Harrington:
Well, software runs the world. And so that's why this whole field needs to exist, securing applications. And the reason that I wrote the book, I guess the scope of it is to think about almost programmatically, how should think about the entire process of securing software. And the book itself covers a number of different areas, but what's interesting... I mean, I wrote the book, that's basically exactly what we advise all of our clients and our security consulting business to do too. And it covers the full spectrum from, how should you think about your mission at all? What's your attitude be as security? How do you work with outside security experts? How do you share information? How do you find vulnerabilities? How do you fix them? How do you deal with change? How do you build security into the development process? How much should you spend? And then ultimately, how do you convert this into business benefit? Which I argue is to gain a competitive advantage.
Joseph Carson:
Well, that's great. So, Mike, you're in the same role.
Mike Gruen:
Yeah. I mean, that's really how I got started into security. I'm at my background software engineer. So application security and bridging that gap between the engineering team and the operations team and the security teams is what led me down this career towards cybersecurity. I'm curious, one of the big things is risk. I assume your book talks about how to try and figure out, when you're trying to figure out how much to spend. Risk is probably the critical thing. How can you maybe talk a little bit about that?
Ted Harrington:
Yeah. I did talk about risk in the book. I mean, you couldn't really write a security book.
Mike Gruen:
Wait, could it mean, in terms of figuring out what's the value? You have to be able to assess. Yeah.
Ted Harrington:
Yeah. So, risk is of course a field unto itself. And so while I did touch on risk, I wouldn't consider this a book about risk. Well, in terms of the field of risk, it's obviously all of security is about managing risk and understanding it. And the whole book is structured around identifying common misconceptions that people have about how to secure software systems and then saying, Hey, everyone thinks it's X, but it's actually not X, it's Y, and then I explained the differences between X and Y and one of the, I guess, common misconceptions that people think about with risks, because it's so many approaches are really not tailored to the system in particular. So for example, most approaches that are largely just vulnerability scanning the output of those tools really isn't customized or calibrated to that particular organization's threat model.
Ted Harrington:
Oh, here's a bunch of critical vulnerabilities, or you might have high severity vulnerabilities, or you might have even low severity vulnerabilities and you look at them and they might not be the right rating based on what the business actually does. And so I talk a lot about how do you think about what the severity is. And then once you can effectively understand the severity, what do you do with that information.
Joseph Carson:
Yeah. That's a great point. It reminds me of being based in Estonia over the years, we've been really looking at this very pragmatically as well. And one of the things I learned was that in Estonia, there was a transition, it was around in 2007 to 2010, where the government was really going through their strategy of about how they're doing security and this of course is the post -2007 nascent state detectives Estonia had. And they really looking at how can they strengthen their actually services and systems and software and applications that they deliver, because we deliver a lot of things through digitalization, whether it be voting or tax or online banking, and a lot of those applications, they want to make sure that they had security by design. And one of the fundamental things was that a lot of organizations, that was a phase when they were going through the software defined network and looking at it delivering software. And what was the software being delivered, but Estonia took it from a very different approach.
Joseph Carson:
They realized that it's not about delivering the software itself. It's about what services that software or these multiple applications providing together. And they looked at it from a service defined network and thinking about... And then what is the cost of that service was unavailable? And it was ultimately where they made this transition to looking at making a risk approach around saying, if this application A is unavailable, it was very hard to justify the tangible cost of that, but if they have a service approach, which has been supported by these multiple applications, they could start looking at it from the actually costs of downtime. If the system was only available for day or an hour, or a week, whatever it might be, they were able to then justify those costs.
Joseph Carson:
And ultimately going to Mike's point about, then they can actually come up with, what did they want to invest with just the risks. And it reminds me, I got a few years ago, I did a penetration test on a power station. And ultimately I did the same thing other security consultants do at that time is, I went back and I provided all of the fear, all the vulnerabilities and all of the risks. And then we presented it to the board and the board said, well, thank you. Great presentation, really scared as... We're frightened, we're afraid to go and look at our emails at home tonight, but you didn't justify the same line as that we understand is they didn't approve the budget that we were looking for.
Joseph Carson:
And they said that the reason for that was that we didn't show the return on investment. We didn't show how we're actually helping the organization reduce the risk. How we're helping employees do their job better. How we're actually providing value into the shareholders and investors and everything else. We didn't do a tangible return on investment for that. And it really made me realize that yes, everything we do, of course security has a risk approach, but we ultimately have to understand about what is all of those combinations together? What is the service ultimately we're delivering? And this ultimately gets into then having understanding about what do you need to invest into? And then the CFO at the time made a major statement to me and it was, I never remember when he did it in his accent, but he said, you have to show when we asked, well, how do we identify that tangible costs?
Joseph Carson:
And he said, it's easy. It's the cost of doing something versus the cost of doing nothing. And that's ultimately where you get that cyber gap is that... And when they looked at it from a business perspective, but they can reduce the risk, they were willing to invest and if it was significant risk, even up to 30% of the costs to reduce that risk even further and sometimes deciding insurance and so forth is an option. And so ultimately it was an interesting approach. So, Ted, have you looked at anywhere around from those risk assessment or even let's say, how software individually or software collectively, regards to applications how they can be worked together because a lot of times what organizations do when they're doing a penetration test, as they focus at application individually, rather than the collective of what it's doing together?
Ted Harrington:
Yeah. I mean, there's a lot of elements to what you just talked about that are so interesting and I'd love to dig into, but the first one that I want to start with is talking about this idea of return on investment, because I actually agree with a lot of what was said there and that the people who own ultimately approve any investment, they need to be understanding the issue in business terms. And when I was writing my book, one of the things that I did was I started looking around our customers and saying, okay, well, they obviously run the spectrum from very, very, very capable organizations in terms of their security maturity, all the way down to like, no, they're just getting started. But they all really shared this common trait, which is they're investing a considerable amount of time, effort, and money with us to do this.
Ted Harrington:
And I had to really unpack why. And the obvious statement across the board is security matters to all of them. It reflects their core values. It's just part of their mission. So that goes without saying, but let's be honest. We live in a capitalist world. That alone isn't sufficient to justify an investment, just the right thing to do. Isn't always the thing that the business is going to do. But the other thing that really became apparent as I was thinking about is every single one of these companies maybe not, maybe I shouldn't make a general statement, but almost all of them were looking at security as something that would help them obtain a competitive advantage. Because if we think about it from the viewpoint of the enterprise buyer, and everyone who's, not everyone, but most people were building software in the business to business space are trying to sell it to an enterprise eventually.
Ted Harrington:
And those enterprise buyers have really high expectations of security, very different from what a consumer cares about. But an enterprise buyers absolutely is expecting the software they buy to be secure. Now, they're not willing to spend more for it, unfortunately, but they definitely demanded as a requirement of the business. So that's one factor to keep in mind, the buyer wants the software that they use to be secure. Most organizations that are commercializing software struggle to actually secure it, and struggle even more to prove it. The ability to communicate it's really difficult. So when you take those two things come together and you realize the buyer wants X, and almost nobody can deliver X. That's an enormous opportunity to differentiate. And so the companies who can deliver X, which is of course, secure your software system and prove it, those are the organizations that can go to their CFO and say, hey, this isn't about this big, scary number that could happen. And we're going to spend a percentage of it to try to reduce...
Ted Harrington:
It's let's invest this money and win more sales. Let's win them faster. Let's earn those enterprise customers we're looking for. And as we've been advocating for that, and that's one of the arguments that I make in the book and teach people exactly how to do that. Now they realize, well, this is a totally different way to think about security, which is that, it's not just removal of a bad thing, it's pursuit of a good thing. And that's really compelling to enterprises and those who approve budgets.
Mike Gruen:
I think it's also similar to, one of the areas where I found I've gotten good traction is that security is no different than any other type of bug. Companies don't generally bug at the idea of a QA team or testing or making sure the features work the way they're supposed to, or don't have unintended consequences because, whether it's a good UX, a good, whatever it is, you want to have a good user experience. And that's where I've found security isn't that dissimilar. And I think there's also a big difference though between companies that are building up applications to sell, as you said, like it gives you that competitive advantage, it has whatever, but then you have plenty of enterprises that are building their own internal applications, anything that's where the struggle is probably the hardest to justify some of this stuff because the pressure to ship and get the thing out to whatever problem they're trying to solve, as opposed to... Because you can't really make... Well, I'm curious, can you make that competitive advantage point when you're talking about an internal application that you're building or whatever?
Joseph Carson:
I mean, for me, I agree. I think it's only important things from an internal security perspective is that they have to look at themselves as actually from let's say, well, there'd be an engineering team and the security team together. They have to look at the combined effort to actually show how they can actually add value. I remember listening to, it was an awesome talk a few years ago. It was about, it's an Italian insurance company that really took... They had a new CISO came on board and really took it from a very different approach about all the things they were doing. The organization from a security perspective, they actually mapped it into actually how they can make it actually part of their business model. So they actually turned it around and said that as we're investing in security and our new applications and awareness training and everything we're doing around culture.
Joseph Carson:
And it was about people technology process and even the services they were building and delivering, they said, they're going to actually identify every time they actually deliver a new service, how to actually integrate the security into it and actually make it something that was sellable. I think that's one that we need to be doing better is that rather than saying, as you're saying, Mike is just a bug, we need to fix it. Was it a deficit or it's refactoring of whatever it might be or changing the design. We always have to think about how does that map to actually innovation making the product better in regards to making it more sellable. Sometimes we end up going and looking at just regulations and standards, people that they ended up going for the ISOs and the PCLs and other types of regulatory things for the checkbox to show that, yeah, we're doing these right things, but ultimately they're failing to actually engage it and map into actually the service that they're delivering if how that makes increased value.
Joseph Carson:
And I think they'll sell it to me world organizations can see and make a change in this going forward is that, by investing in security is actually, it's an business investment into making them more spillable. And that can be a differentiating between many organizations.
Ted Harrington:
Yeah. Definitely. I mean, the word you used is spot on better. That's that is the whole ethos of the entire profession of security is we make things better. It's never done. You're never like, and we're secure. We can now go do other things. It's like, we have to be better today than we were yesterday. We have to be better tomorrow than we are today. And if we're doing those things, then we're doing security right. And Mike, your point was definitely well-founded, which is that the case is different. If you're commercializing the solution, versus you're trying to convince other business units internally to use it. But many of the same elements actually carry over from what we've seen with the large enterprises that are our customers, is that a lot of the same market forces still exist in that two business units in the same enterprise actually still buy things from each other.
Ted Harrington:
They borrow budget, they borrow resources. There's... And in some sense it's even more competitive. Where they're like, Oh, well, we built this thing because our workflow is so special and different from yours. And so the ability to be able to say, Hey, look, we've already thought about security. This already meets all the conditions that the corporate security team requires. This is how it's going to make it easier for you. It all goes, you still have to do a sell job, even when you're commercializing internally. And security is a critical part of that, for sure.
Mike Gruen:
Yeah. Your point about it never ends, I was just talking to a friend this weekend and we were talking about how security and even software development for that matter is like mowing the lawn. Just the grass keeps growing or getting a haircut. As soon as you're done, you're not done. It's already growing again and you have to take care of it, or you can just let it go in terms of weeds and then forest and-
Ted Harrington:
That's a great metaphor. I'm definitely going to borrow that because it really helps you visualize that it's a case, you have to do something or it gets worse.
Mike Gruen:
Right. I'll actually give credit with Steve Jacobs who was a previous guest on the podcast. He and I were talking about it. Yeah.
Ted Harrington:
I love it.
Mike Gruen:
Like Mowing the lawn.
Joseph Carson:
So, Ted for organizations, you might have done this quite a long time. I've been through developing organizations that were very much waterfall approach. And we had released as every one year or two years too, I've seen even here in Estonia, there's a lot of startups who basically go straight to cloud and accelerate and grow very quickly. And they're very agile and dynamic. I mean, is what your methodology approach worked for both of those or do they have to find some different approach for organizations are very traditional. Do they need to change significantly versus those who are the startups too. They tend to more focus of getting an MVP product out the door, and then adding security on later. But ultimately those organizations, both the traditional monolith waterfall approach for the release every one or two years that the security doesn't get updated that frequently. And versus those that lack to prioritize it at the beginning?
Ted Harrington:
What I love about this question is how polarizing it is. Everyone who just listened to you ask that question is going to fall in one of those camps. And they're going to be like, waterfalls are the best, screw agile and agile people are going to be like, screw waterfall. Agile is the best. And then there's someone else being like, but what about... And everyone to me is delivering which is what we do here, because agile is not as agile as you think. In a year, there's going to be another methodology. So, when I was thinking about how to answer that exact question as I was writing these ideas down and again, looking at all of the companies that we're fortunate to serve and how do they deal with it? What are the struggles and how did they overcome them? If you really simplify it and we step back from what we're talking about here and get away from the specifics of how the methodology is actually work, realize the fundamental principles are the same.
Ted Harrington:
Now, how you go about those principles obviously is different agile, waterfall. They're really, really different methodologies, but the fundamentals are the same. You identify a problem, you set the requirements for what you're going to do in order to solve that problem. Then you figure out how you're going to solve it. Then you go build the solution to the problem, and then you go implement and maintain it. So whether you do that on the whole system, or you do that on a user story, those steps are the same. And each of those steps has a security step to it as well. And that's really the big thing that people need to take away from it. It almost doesn't matter what your methodology, I mean, it matters because how you implement it matters. But the point is the same that whatever methodology is being used, there is a security process that integrates into it. And when I ever hear people say like, Oh, well, we use blank methodology. So we do security later. It's like, No, that's incorrect. Security is built into the whole process.
Mike Gruen:
Yeah. I would agree. I think when it comes to those methodologies, really the difference has more to do with the release strategy or the deployment strategy, the frequency, or how... If you are doing quarterly releases or annual releases, how do you deal with something that's needs to be released more, has a more pressing need versus continuous delivery where, as soon as it's done, we get it out the door. Not to say that one's better than the other, but I think there's just, that's where the implications are. As you said, it's not in the, when do we do security or when do we think about security. It's more of the like, okay, if this is how we're delivering our solution, what's the implication for that from a security perspective?
Joseph Carson:
So Ted, based on both those things, I think those are fantastic points that no matter what methodology you're using, that you need to build the security into that entire process in respect of a waterfall or agile. Now, one of the things that, didn't do a lot of it goes into the testing phase of it. Because that's sometimes where the difference between agile and waterfall is around that testing approach that you test, you find bugs, you basically fix the bugs and then you test again, we're in agile, of course, you're doing that continuously as you're building and developing and deploying. Now, one of the things is that for organizations who are building software, I find all the time, they don't really understand security aspect of things. The security testing, or security approaches. I've had organizations come to me are building their startups or billing software and say, I want to do a red team assessment, or want to do a vulnerability assessment, or I want to do a pen test.
Joseph Carson:
And You're going to the mail, like, okay, well, you're going to, you want to do a red teaming on a MVP product that isn't even in the production type of environment where you've got the people in the operators, because red team's really focused around many aspects of when it's actually being delivered. You can't do really do a red team on a product is not actually being actually delivered in production, used by the people who's going to be using it and operating it. So there's a lot of confusion into what security testing is a vulnerability and bug bunty and all of those things between application. And I think it was Joe Vest wrote a really great book around red teaming, that side of things about understanding the methodology. What's your approach when you're talking about it from an application perspective, what applications do you know is for those who don't really understand about the methodologies?
Ted Harrington:
I think that what you just brought up is such a profound problem in security, the idea that people don't always even understand the terms that are being used and why is that an issue? And it's such a big problem. I dedicated an entire chapter in my book to that problem actually. And the way it plays out in real life, I mean, in our business where security consultants and people hire us to test their systems. So almost every day someone asks me, Hey, can you help us with penetration testing? And my answer is always, maybe, what do you want to achieve? That's really the question that matters. What do you want to achieve? Because the practical reality that I've seen in the world that's really troubling is that you have these conditions. Most people when they want to test their system, they typically ask for penetration testing.
Ted Harrington:
Sometimes they ask for red teaming, which is yet another thing that is something different, but they usually ask for penetration testing, but then what they're usually sold is something else which is vulnerability scanning. If you Google right now, penetration testing, 85, maybe 95% of the results you're going to get are scanned based approaches. But what people really need is a third thing is something else altogether, which is vulnerability assessments and the metaphor that I think of, how do we differentiate what these three things are? Is I like to use cars as a metaphor for this one. So penetration testing is like when the automotive maker wants to see how does the vehicle perform in a crash scenario? So they actually literally crashed the car against the wall. And that's penetration testing. You take a real world scenario, you emulate it, but it's really only works for a system that's already been through a lot of testing.
Ted Harrington:
And it tests for like a really specific set of criteria. It's not a holistic approach. It's just like, Hey, we're we want to see in this situation what happens. And then it gives you a binary outcome, the passenger did or did not survive. So that's what penetration testing is like. And that's bad-ass, but what people are mostly sold is vulnerability scanning, which is more like when the check engine light comes on in your car and you go to the oil change and they like stick the little device in and it's like, Oh, here's a code of what the problem is. And here's how we turn off the check engine light. It's very easy and quick and inexpensive to do, which is what vulnerability scanning is also like what it's really different. The crash testing a car and the scan of the diagnostic scan, they're really different.
Ted Harrington:
And yet what people mostly need is they're getting testing because they want to understand the system holistically. They want to understand the severity of issues. They want to fix the issues, and they want to verify that the fix is worked. And so that's really much more like the entire process of automotive safety engineering, which basically says, okay, here's how the seatbelts work with the head restraints with the airbags, for the side impact beans, how does it all work together? And how do we make sure that the passenger is more likely to survive in the event of a collision? And so those three things are really different, really different outcomes, really different investments of time, money, and resources. And my advocacy is instead of, while I wish everyone used the right terms, I think it's at this point, like a lost battle where like, no one's ever going to use the right term to describe secure desk ever.
Ted Harrington:
Instead, let's just talk about the goal. What is the outcome that you want to achieve? And then we work backwards from there to say, okay, well, you need a pen test, or you need a vulnerability scan, or you don't want really assessment, or you need red teaming, or you need bug bounties or whatever.
Mike Gruen:
Yeah. No. I absolutely agree. And that's something that, we get frustrated all the time. I think we did a show, was it a few months ago was on basically buzzword bingo, which is about all the terms that we hear in the industry that are created from basically emphasis of marketing, just trying to re spend something that we've already been dead for many years. So hopefully we'll get away from that. And I agree it should be focused around what is the intention of the goal. And that's really understanding what you're trying to achieve and having some type of scoping discussion and understanding about what is their business, because not many are very different, and don't have the same types of methods or structures. So you can't just take one method and apply it to everyone. You have to understand about what it's going to be used.
Mike Gruen:
Another question I've got for you Ted, is around, a lot of the newer ways of developing applications are using a lot of modular approaches, meaning that they've got a lot of dependencies and a lot of third party components and maybe even shared cloud resources. What's the approach you would suggest for those organizations that have a lot of basically third party resources or plugins or libraries that they're building in and hard to manage those in addition to what they're building themselves?
Ted Harrington:
Yeah. That's definitely an inherent issue with pretty much any software system that's getting built today. There's going to be dependencies. And the likelihood that any organization will be successful in getting that third party to participate in any white box testing is good luck. You're going to call some company and be like, we would like to look at your source code. They're like, yeah, how about shut up? It's just not going to happen. So what organizations need to do is they need to understand where those dependencies lie, they need to account for that and part of their threat model. And then ultimately you defend against it with a defense in depth. And so that's assuming that these shared components might themselves ultimately be the source of a security breach. And then how do you mitigate the likelihood of that compromise impacting your system? And then how do you minimize the impact when that actually were to happen of the compromise of your most valuable assets? So it's a really, really tricky problem. That's obviously a lot more complex than the way I just super, super oversimplified it-
Mike Gruen:
No. It's really just that easy as someone who oversees all, it's just that easy, we are... yeah.
Ted Harrington:
Just do it. I don't understand it but just do it.
Mike Gruen:
Right. Right. We talk about supply chain and we talk about dependency and all that, but the reality is like, you're still just as like, let's say you were to go the opposite extreme. If you were to try to build everything yourself, you're just as likely to create those same problems or those same abilities or whatever. So it's not like there's a good answer. The right answer it's really about dependency management and testing, scanning, being confident in those third parties that you are trusting, try to minimize your footprint. I would say like, don't have a bunch of third party dependencies that all do the same thing, trying unify and use as few as possible. That's our approach. I'm curious what your thoughts are on that.
Ted Harrington:
Yeah. I love the word dependencies, because I think that that single, where I've been struggling with this for a while, thinking of how do we simply describe the problem? And I'm like, is it vendor supply? Is it supply chain management? Is it vendor risk management? And you hit it. I mean, dependencies that's really the issue. And I think a lot of people can visualize the idea of a dependency, like a row of dominoes. If this one domino over here falls, this other domino over here is also going to fall. And this-
Mike Gruen:
Or like, I think of it in terms of like a construction, a building, a high rise or a skyscraper. We're not going to source the steel ourselves, we're going to all the different components that go into it. We're going to trust these various manufacturers. And then, maybe we're not responsible for the whole building. Maybe we're only responsible for the top floor, or the middle floor. And so we're also responsible for all of the guys below us having done their job and all of the people on top of us, but we don't get crushed under their way, stuff like that.
Ted Harrington:
Yeah. I think the construction and building metaphors are really powerful for security because they're so well representative, you have to have a strong foundation and all these pieces work together and different experts do different things. You wouldn't hire the general contractor to do the demolition of the building, because, although they're related, those are different types of expertise. But yeah, thinking through the... I mean, the way that certainly all ethical hackers and malicious hackers as well, think about dependencies is the attacks aren't always straight on. Actually, most of the time, they're not straight on, it's not usually like you go after this fortune 100 enterprise directly, you go after the way that fortune 100 enterprise has its software integrated with some third party, maybe payment process, or whatever.
Ted Harrington:
And those are called stepping stone attacks where you compromise one in order to get to another. And that's definitely one of the big challenges. And we're always looking at how can you... Even if you're not even talking about dependencies, but even within any attack scenario, what we're always looking at is how can you chain exploits? Because oftentimes two things, one might even be, have no significant severity to it at all. And another might be like medium, but then you've combined them and it's like, Oh, well, that's system-wide compromise if those two things happen together.
Mike Gruen:
Yeah. I mean, I think that reminds me, I mean, when you talk about dependencies, especially enterprises that are maybe using things that they've developed and then they're reusing them in other places. Like one of the, I think back to my first job, and I remember sitting around a room and we're talking about, we needed to play a feature, we want it to get done relatively quickly. And everybody's like, well, there's this other component that we could use that does what we want, but not exactly, but we think we could use in this context. And I remember one of the engineers saying like, I can't put my finger on it, but I think this is a terrible idea. But nobody really knew. And they were like, all right, whatever. And so we got it out there.
Mike Gruen:
And then it turned out, I mean, it wasn't a security problem, but it did create a database dead locking issue because these two components, what this component, where it was designed was in a completely different set of libraries and part of the application where we always locked A and then B whereas in the place that was being used, we always liked being in name. So we created this like database set block. It took us six months to find because it was random and whatever, but that's the type of stuff that I think happens from a security perspective that could have just as easily been a vulnerability where the system was designed to handle this stuff and we feel good about it. And then it's like, okay, we're going to use it over here now-
Joseph Carson:
That's so common. I mean, years ago, I remember I wrote, it was a TNG Migrator to ...compute. And I wrote it for one specific client in order to do migration. And it was written a really bad... My development skills are really horrible and I wrote it. It was basically a script and Pearl that actually converted things like watchdogs. And there'll be my calls and this will be chops over to those formats. And ultimately it was dumb for one purpose. And to your point, Mike, what ended up happening was is that, somebody said, Oh, you know what, I'm having another similar thing over here. I can reuse that. And then they take it. And then basically ultimately what I ended up finding about seven, eight years later, after writing this really bad script that did migrations, that it was ultimately created into a product and then use widely, it was not done with security but it was done for one purpose, only thing.
Joseph Carson:
And somebody decided to say, Oh, that could solve a lot of problems, but that's ultimately we get a lot of those challenges and risks is that they end up reusing in so many areas that it should never have been used, while it might've saved time that definitely was not something that should have been productized and delivered to company.
Mike Gruen:
I think another one Ted, I'm curious where you see this, when things are co-deployed where you might have a system that's seen as a low risk, low, whatever we've decided were the implications of this thing being compromised as relatively low, and not really thinking through its neighbors or what other things might be deployed within. And so it does offer that stepping stone of, okay, but we're not going to... The front door is always super loud. We'll just use the side door over here. We know they keep a key in a shed. How do you address that when you're talking to your customers and clients and stuff?
Ted Harrington:
Yeah. By helping them understand the actual attack sequence that might exist. So for example, one of our customers had this combination of issues where the first issue that we found was that this particular application had a flaw called information leakage, which basically meant that any user could identify the user identifiers for all other users. And that's not that big of a deal. And you can't directly exploit it, but you don't want something like that. User IDs should be unpredictable. And in this case, they were very predictable. Database is big of a deal. Right. Yeah. There were definitely sequential. The second problem that we identified was with their authentication mechanism, where our, sorry, their authorization model was broken. And basically the way that the authorization worked was that if you wanted to change the credentials for a user, you needed to supply information.
Ted Harrington:
The information you needed to supply was the user identifier. Now on its own, every user only knows their own user ID. So not a big deal. I mean, it's bad, broken authorization was definitely bad but it wasn't like catastrophic. But now when you take these two ideas and you combine them, you can predict every user identifier. And then all you need is a user identifier to change their credentials. You've just taken over every user, every account in the entire system, including the admin. And it's like, that is system-wide compromise. And so it's slightly different than your question that wasn't two independent systems interacting each other. It was the way that one system was built, but these dependencies and how you could chain exploits was catastrophic. And really the point here for companies to think about is that so much of security today when it comes to software is believes whether they explicitly believe it, or they just have never evaluated this assumption.
Ted Harrington:
But most security today in software believes that you can just do it with a scanner. You can just push a button. The problem is solved, but stories like what I just told, you can only find stuff like that manually. You need to find these issues and you need to be a human who says, well, how do these pieces fit together? And that's really one of the big, big challenges that I'm trying to advocate that companies need to think differently about.
Joseph Carson:
Yeah. They need to think of the, how things work together. Going back to the experience I have with this study, that whole service mapping piece of how does everything work together and what's the ultimate outcome, but that's delivering. Question, one of the big terms, it was the buzzword bingo set of things we've heard of course is DevSecOps. What's your thoughts around that methodology where you're actually moving, where it used to be. If you looked at the last ten, that was all about doing the analysis and assessment based on those different security controls and security risks, the ends application, now basically it's get into these more proactive controls where it's not thinking about, well, let's move it into the development process. Is that something that you think is going to work and we're going to improve things, or it's just it's developers who are not thinking about security or not having to do security. What's your thoughts around that?
Ted Harrington:
I think it's a positive step. Anytime that we're saying, Hey, let's adapt and evolve and let's make security part of that adaptation. That's like in a nutshell, hell yeah. I love the thinking around that. Now, whether or not DevSecOps is forever going to be the solution. I don't know. We'll see. I think it's a cool approach. I think that there is with any change like this, or evolution or state not necessarily change, but with any evolution, there is the risk that some percentage of the population will say, Oh, this new thing solves 100% of my problems. And that's the risk that I see is people thinking like, Oh, the shiny new thing, I don't have to worry about it anymore. And that's definitely not going to be the case, but I do think that it's cool step in the right direction. Word, obviously we talk a lot about this, both internally with some of the things that we're doing, as well as with our customers.
Ted Harrington:
We have a whole series right now of internal talks at our company where we're everyone's just teaching each other all these principles around DevSecOps and ideas on how to improve it. And we're going to eventually publish those and put those out in the world. But yeah, it's a really interesting area-
Joseph Carson:
Because I really-
Mike Gruen:
My fear with DevSecOps it's very similar to my fear with agile or any of these other things where people certainly ascribe a whole bunch of ceremony or other things to it and meaning, and it's misapplied or misunderstood, or you think you're doing it, but you're not really... You ask a bunch of people what DevSecOps means in the security field, you'll get a different answer, and I think for me, it's about a methodology and a mentality of... And it's similar to any other... When I look at my software engineering team, I don't expect all of my engineers to be security experts, just like I don't expect all of my engineers to be backend developers or front end developers or any number of these things. And it's just about having the right team or testing or QA.
Mike Gruen:
And so I think that's where I talk to people all the time and they're like, Oh, yeah, my company is going more DevSecOps. This is my job. And now I'm being asked to do this other thing, and I'm fighting with the engineers all the time. And it's like, well, you guys are clearly, you're not understanding the greater more abstract, what's the problem you're trying to solve with DevSecOps. And you're just jumping on this, like this is how you do it, as opposed to like... And that's where agile started. Agile started as these guys were like, Hey, we have this process. It seems to work for us. This was the foundation of that, do with it what you want.
Mike Gruen:
And then now there's a whole bunch of companies that have taken that and found it whole religions on it of how this is how you do scrum. And this is how you do this when really, it was like, Hey, what agile was about was just shifting things a little bit sooner. What's iterate to success, let's get testing involved, let's get something deployed. And it's more abstract. I think it's hard for companies to glob onto the abstract. It's easier for them to... Yeah, we're going to adopt this very structured thing and think that, that's going to solve their problems. That's my big fear.
Joseph Carson:
It's dynamic. It has to be adaptive. The whole point here is that it has to adapt to the needs of the business, because there's not one peg, if it's everything. And to your point, Mike, I remember a few years ago where an organization switched completely agile, and they were following the storm and the following everything, all the methodology, they went done definitely the religious approach. And it got to the point where you get got you teams doing stand-ups and you had designers saying, I built the tree yesterday, and I built the tree today, and tomorrow I'm going to build another tree. And these were game designers, and game developers. I'm going to be building trees for the next three months and I'll have the same update every single day. So it really has to get into adapting to the really needs and making sure you're not wasting time for the sake of being agile, but really making sure that it's becoming efficient to your point is delivering faster, more efficient.
Joseph Carson:
And let's say, smaller improvements over time, rather than waiting a long time for the one big release. So, one thing I really thought that this whole DevSecOps was what was lost was around the... Proactive controls, because for me, that was really... The big change was about really being proactive and putting it much early in all of the phases. And I think Mike, your point is critical is that, it's not about changing your development team to being security. It's about actually embedding security into the development team and having people who's security minded, not having everyone doing it and doing a buddy system where you have people overseeing and making sure that security practices are being adhered to. Ted, another question from organizations who always get the... When we talked about software and development and insecurity about whether open source versus proprietary side of things. How do you find that in and the companies you're dealing with and what's your advice and findings there? Is there a big difference and benefits to either either or?
Ted Harrington:
One of the common things that I hear people believe, which is a mistaken belief, is that because something is open source, thus it is more secure. And the reason is, Oh, well, more eyeballs are on and more people can look at it and it's visible to everybody. And it's like, yes, those conditions are true, but that doesn't necessarily mean that someone is actively doing security at the level of where someone who's maybe commercializing the software might invest in it. That also doesn't mean that someone who is, has a close door system that they're investing appropriately either. So in my opinion, the nature of the code, whether it's an open source or not actually doesn't have a bearing on the security, but I do believe that the way that it practically plays out is you're more likely to see the proper investment of security if it's not open source, because when it's open source, people feel attached to it because they're contributing to this greater good, but they're not necessarily going to invest appropriately or on the right cadence.
Ted Harrington:
I mean, also who wants to stop someone from putting something malicious in there and all these kinds of things. So I think it's really an independent question, it's good that you're asking it because they really do need to be broken apart.
Joseph Carson:
Yeah. I agree. So, Mike, what's your thoughts? Have you had any?
Mike Gruen:
Yeah, I think Ted nailed it. Neither of those is inherently better or worse. For certain open source projects, there's there are security people attached and it goes through a lot of review and it's a lot harder to do. For others, it's a little loose. And I think the same thing closed proprietary systems, how many people just do security through obscurity and you know what, there's something to be said for security through obscurity, it's obscure, it's hard to know how this is going to work. And so, it's one of the layers of depth. So no, I think you're absolutely right. I do want to say though, just jumping in real quickly back to DevSecOps, because I know we're talking about application security, but one of the things about DevSecOps, I think it's overlooked a lot is that it's also infrastructure code. It's how all of the other side also benefits from software engineering, best practices being applied to how we deploy.
Mike Gruen:
So when I think about it, the benefits aren't just about embedding security people into software application teams, so that the security people can teach application security. It's also set those security people who are also doing operations and those operations people who are doing operations, are also learning the best practices from software engineering. How do we do code reviews? How do we treat infrastructure as code? Everything is virtual these days. And so I think it's more than just a one way thing. It's this bi-directional everybody benefits from it. And if you attack it from that perspective, you get a much better result because then everybody feels like they're contributing and everybody's helping each other as opposed to creating this like one way like, Oh, the security guys are here just to educate the application helpers.
Joseph Carson:
Silos.
Mike Gruen:
Right.
Joseph Carson:
Well then by building up the silos-
Mike Gruen:
But anyway, so I did, I just wanted to bring that up, but I'm happy to jump back on the ... open source.
Joseph Carson:
But expanding on that one. That's a good question, Mike, that you're bringing up, is around... So Ted, one of the things that can my background is all around virtualization and containerization, doing things in a modular approach and doing microservices. Do you think that's a better way going forward, especially when you're doing application security is when you're building everything in smaller modules and that when you do run into a problem or you do run into an issue that as much, you just need to swap out that one modular or fix the one piece. What's your thoughts around virtualization, containerization and those things, is there benefit to security in that way?
Ted Harrington:
There are benefits undeniable. I mean, you just rattled off several of them, for sure. I think the broader answer to that question is, to a certain extent, it depends on what you're trying to achieve with the system, both the business problem that you're trying to solve and then the business of solving that. So those are two things that I would be, I think, mistaken to make a universal statement that said, everybody should do X of this, on this particular specific topic area. And that's one of the things that I've found. That's really tricky about trying to, I guess, educate and advocate around ideas related to applications security, because you can make some universal statements that do apply to everybody. Like you should share information in this way. You should do this kind of testing. You should prove the business case in this way.
Ted Harrington:
You should think about your threat model in this way. But then when you get down into the brass tacks of like, okay, developer, step one, do this. When you start talking about at a language level or an operating system level, that's where it becomes very dependent on the conditions of what problem you're trying to solve and the business model built around it.
Mike Gruen:
Yeah. I mean, on microservices, that's that's one of my big buzzword that actually bothers me a lot. I think that the vast majority of companies that think that they need to use these microservice, there's like, maybe 1% of companies in the world actually really need to go down a microservices route. I think that you, the complications of having that far outweigh some of the benefits you get out of it, I think it's a very interesting, exciting buzzword or whatever, like concept, but then in practice you have to be like a Google or Facebook to have to deal with that level of scale of data and that many engineers and that many different problems. I think Ted brought up a great example or counterpoint in a monolithic service, chances are that authentication problem that you saw that you were talking about earlier where it's like, you only need to know a user ID in order to change the thing.
Mike Gruen:
Chances are, if that was in a monolith, it would be harder to explain because it would probably be all internal calls, all internal functions back in the backend, and there would be all of these things that would sort of prevent that from being exposed to an end user, potentially. Whereas if that's a microservice, there's a good chance that there's all of these now tiny little boxes and it's so much harder to identify how they're working together and like, Oh, because these five things are interacting in this way, that, that's what's causing the security vulnerability. When you look at them each independently, they all look fine. It's not until you see them in full operation that the problem shows itself. And I think that, as nice as it is to be able to have smaller teams and smaller projects and be able to do that from a microservices perspective, there's all these other complications that people don't necessarily take into account that the complexity of that, and of tracking all of these now APIs and messages that are going through the system.
Mike Gruen:
And how do you make sure that all of these boxes are... All those messages are secure and coming from who you think they're coming from. And all of that it just adds a lot of complexity that I don't think enough people take into account.
Joseph Carson:
Absolutely.
Ted Harrington:
Complexity ultimately increases the likelihood of a successful attack. So certainly one of the things that we all advocate for it's simple simplicity, keep it simpler, reduce the tax surface, that isn't always viable because of what the solution needs to do to solve the problem it's setting out to solve. But it's for exactly the reason that you just brought up complexity introduces attack probability.
Joseph Carson:
Right. So Ted, tell us about your book and how people can get obtained a copy of it. And also, let's say, are you thinking about a newer book coming out or an additional later, and is there something... that you're looking to? You're working on secretly in the background and who should be the person that purchaser but I mean, I'm sitting here with my copy and my desk here. It's almost sitting not too far away when I need to go for a reference. But tell me about who should read it and where can they get a copy of the book?
Ted Harrington:
Yeah. So I wrote the book for basically three audiences. The first is what I lumped together as technical leadership, CISO, CTOs, VPs of engineering, whoever it is, that's ultimately at the other end of the points that finger, when someone says you're responsible for the security of this thing. So that's the first audience. The second audience is a software developers, and then the third audience is security professionals. So it really serves all three of those audiences and where, I mean, you can buy it on Amazon if you want, if you want it to learn more about the book itself, or if you need to even get ahold of me, or you need to touch base to ask questions about security testing, and how am I able to help you with that? Everything you can find all that@hackablebook.com is the website for the book. And yeah, as far as another book, it's funny. I only just wrote this one and I get that question every day. And I asked myself that question every day, there might be another, there's probably another book in me, but I haven't committed to one yet.
Mike Gruen:
Well, just horror stories, I think would be good.
Joseph Carson:
Always happy to be your reviewer. That's-
Ted Harrington:
Oh. I might take you up on it.
Joseph Carson:
But it's been awesome chatting with you. It's awesome having you on the show really. Hopefully I'll get to see you at some point this year or later, whenever things come back to somewhat, one of our truffles starts again. It's not been officially over a year or what I've actually been in Estonia without traveling. It's my longest period in Estonia. So I'm looking forward to when the things start to open up a bit more and hopefully catch you at some point this year, at an event and yourself, Mike, as well when I get hopefully to DC. So for your honest Ted, it's been fantastic. Hopefully this has been valuable for the audience. Mike, any final thoughts or words?
Mike Gruen:
No. Definitely enjoyed the conversation. I'll look forward to reading your book. I probably should have read it before this, but I'm bad like that. But yeah, it was definitely a pleasure and I enjoyed every, every moment. Thanks for joining us.
Joseph Carson:
Ted, Is there an audio version yet?
Ted Harrington:
That's in the works. That's a great question. It will be out, I don't have the exact date, but it's in production right now. And I want to say it's going to be somewhere in April.
Mike Gruen:
Okay. I'm looking forward to that.
Joseph Carson:
Are you looking forward to that you're doing the voice or is it?
Ted Harrington:
I opted to hire a professional voice actor, and that was actually a really difficult conversation because I'm listening to all these auditions and I'm like, he's not saying it the exact way that I would say it. And so I asked a bunch of people for advice on that, and it was a good humbling moment because they're like Ted, you're the only person who's ever going to read this book the way that you hear it. So make it so that the person who's going to listen to this is going to be able to consume the ideas you're trying to help them with. And so I was like, okay, well, that is great advice. So I got this amazing voice actor. Who's like super baritone, and you just want to fall asleep, listening to him, talk about like the nutrition label. He's just amazing.
Joseph Carson:
And just keep in mind that most people like may listen to it, to X speed, anyway. So, information and the rest of it don't really matter. We're just trying to consume the information.
Mike Gruen:
I'm more like a 1.25, and then I usually go double.
Joseph Carson:
But absolutely Ted has been awesome having you on the show. And again, great chatting with you and for the audience, I hope this has been enjoyable. Again, application security, as Ted mentioned so far runs the world today. So if you're not doing applicants and security, let's make sure you're thinking about them prioritizing. So again, this is 401 Access Denied, your award winning podcast, and it's a pleasure serving you again and look forward to here sharing our thoughts and ideas every two weeks, Chenin. And if you miss previous episodes, go and subscribe and listen to the previous episodes, I'm pretty sure you all had fun. So thank you. It's been awesome and stay safe and take care.
