Penetration Testing Top Tips with Dave Kennedy

Mike Gruen:
Hi, and welcome back to another episode of the 401 Access Denied podcast. I'm your cohost Mike Gruen, VP of engineering and CISO here at Cybrary. Once again, I'm joined by my cohost Joe Carson from Thycotic.

Mike Gruen:
Joe, do you want to give us an intro and tell us what we're going to be talking about today?

Joseph Carson:
Absolutely. It's a pleasure to have you back and hopefully everyone has been enjoying the recent episodes. But we have a very special episode today, which I'm really excited to have, a really well-recognized and well-respected industry expert. It is fantastic to have such a special guest on the show today. Dave Kennedy, welcome to the episode of 401 Access Denied.

Joseph Carson:
Today, we're going to be really going into a day in the life of a security expert and researcher and really covering a lot of important topics, things that are really important in the industry. Dave welcome to the show. Do you want to give us a bit of background into what you do at TrustedSec and the things basically you've delivered to the industry? It's good to have you.

Dave Kennedy:
Yeah, absolutely. Thanks so much for having me on today. I really look forward to it. I've been in the security industry for over 20 years, everything from a chief security officer all the way to a security researcher. I always like to maintain my technical roots in everything that I do, that's what I enjoy as my hobbies. So even though I'm a CEO of two companies TrustedSec and Binary Defense, I'm always in the weeds developing tools and hacking into customers and doing all the fun stuff that we get to do.

Dave Kennedy:
But it's great because, we have teams now that do all that stuff and they get to learn from a lot of other people and really work back and forth with some just brilliant individuals where we're helping organizations get better with security, we're pushing research forward, we're looking at what other groups and adversaries are doing to ensure that we stay up to date with everything that's happening. It's a continuously moving industry and being part of that is just an awesome thing.

Dave Kennedy:
So yeah, thanks again for having me, I look forward to the discussions today.

Joseph Carson:
Oh, absolutely. I mean, for me, I've also 25 years in the industry and every day I'm learning something new and it's really important. One of the things I find is, surrounding myself with industry peers, where we all come from different backgrounds and different skillsets. But every time I'm talking to them, listening to them, I'm always learning something new and interesting. I think what's really important.

Joseph Carson:
It's an industry where some people forget that there's a large portion, which is that day-to-day administrative stuff, the things you have to do, but you do have the spend... I spend probably a large amount of my time, 30%, I think, into actually continuous learning and continuous understanding. I'm not the greatest developer, but I'm always getting my hands dirty in things like Python scripting. My background in Perl is not the best these days or Cobalt. But there's always that continuous learning even getting looking into-

Dave Kennedy:
I was going to ... but...

Joseph Carson:
Perl was one of my scripting that which is a long time ago. But not the greatest, let's say if it was still being used today, I'd be quite surprised. But yeah, absolutely. Continuous learning is something that I think is very valuable. I spent about 30% probably of my time, just even getting into understanding and pulling things apart.

Joseph Carson:
I mean, Dave, you mentioned, you're still hands-on, how much of the time you're spending in continuous learning and where do you kind of pull that educational information from? Where do you get it from? How do you continue that?

Dave Kennedy:
Yeah. That's a great question. When you're running multiple companies, the biggest thing is, time management is so important, time management for yourself, personally, your family, those types of things. But also, where do you spend most of your time at? For me, some of the biggest areas are, I've put in a great leadership structure, both at TrustedSec and Binary Defense. The company is really run autonomously with a great leadership structure. So, it affords me a lot of time to be able to focus on making sure that we're staying visionary in the company, that we're ahead of the curve when it comes to what our peers are doing and that our teams are top-notch, the type of work that we're doing is important.

Dave Kennedy:
So, I would say that, I'm definitely in the 30% to 40% range of research and understanding and spending time to dive into what we're doing, hopping on engagements and helping out with customer environments. So, for me, it's making sure that I spend my time appropriately to ensure that I can still stay at a top level in my career, because I don't want that to ever actual fear to go away.

Dave Kennedy:
As you mentioned, the industry changes all the time and you can easily fall behind with everything that's going on there. So for me, I worked very closely with our threat intelligence team, understanding what's happening from an adversary perspective, what different groups are doing, which groups are prominent, what are we seeing from a capabilities perspective.

Dave Kennedy:
Obviously in the news, you see the stuff with like JBS and Colonial and all those different ones. What is REvil up to? What do they look like? What are they doing from a capabilities perspective? Is there anything there that we need to add to simulate those types of environments?

Dave Kennedy:
On the TrustedSec side, working with our research and development teams on our weaponization and tooling, we like to do a lot of cool research and cool things that we don't disclose or publish, that we go out and do things in the Wild West with our customers. Those types of things are really a lot of the focus areas. Maintaining internal tools, maintaining internal methodologies, updating how we do things and communicating that out. Those are all important things for me to go and do. I really rely heavily off of my team.

Dave Kennedy:
But also, social media is a great area for researchers. You look at what a lot of the great researchers out there are doing like Durkee on from an Azure perspective. There's just so many great researchers sharing their information out there that, when they start to see, okay, well, hey, this is an area that I'm not familiar with and I want to dive down into it. I have the flexibility to say, "Okay, well, I'm going to spend four hours on this to understand what this is, or spend a couple of days on this to understand where it is, because it's relevant to my interests."

Dave Kennedy:
I think that's the cool thing about the industry is that, we're very keen on collaboration, sharing, building upon one another which you don't see in a lot of other industries and that's really how this industry continues to move forward.

Dave Kennedy:
There's a lot of discussions on offensive security tools and whether or not you should publish them or not. But, at the end of the day, we're all very passionate about what we're doing to try to protect it. There's different ways of how we do that and different beliefs around how you secure organizations and, whether or not you release offensive security tools or not. At the end of the day, collaboration I think is really key. I think the collaboration we've seen between red and blue authentic defense has really made this industry substantially stronger.

Mike Gruen:
Yeah, I agree totally. I think it's a little shortsighted to think about it in terms of like, if we're not all sharing our tools and our knowledge, the other side absolutely is, they're all working together, they're all sharing this information. So, there's no way we can ever compete if we're not also doing that same level of sharing. I think that's an important part that there's very little security that comes through obscurity. There's a place for it, but we're not-

Dave Kennedy:
Going back here, we've all been in this industry for a number of years. You flash back 10, 15 years ago and red teaming or, at that time, early penetration testing, was kind of a mystic art, it's like a mystic dark art, where no one knew what we were doing. We had all these crazy tools and these exploits and all this stuff that we never shared and we would, specifically focus from a red teaming perspective on, well, here's your technical flaws that you have.

Dave Kennedy:
You wouldn't understand the trade craft that went into then, so what happened with organizations and companies is, they would fix and plug these holes, but they didn't look at what happens after initial access is established. Changing our frame of mind that it's not all about creating this castle because we can never have a castle anymore, especially with cloud infrastructure or, work from home and everything else that's going on out there, we have to focus on, what happens if an intruder is successful and what does that look like from an attacker's perspective around privileged escalation, lateral movement, post exploitation scenarios that we see out there?

Dave Kennedy:
That was largely a very, very close kept secret in the security industry during periods of time. Companies didn't progress forward, in fact, they probably got worse down the road. Then when you started to see, well, now we're starting to understand these offensive capabilities, we're starting to understand what this looks like. Now, companies actually have a defensible approach where they can say, "Well, if we invest in monitoring detection and we can boot out an attacker day one versus day six, day 10, day 15, day 20, six months down the road. We have a much better way of handling our risk as an organization and we don't ever want to go back to that."

Dave Kennedy:
The collaboration of sharing tools, being able to simulate what an actual attacker would do, and to also emulate that to, known specific TTPs and stuff like that. That sharing and collaboration is so invaluable, that to me, it continues to progress.

Dave Kennedy:
For now there's responsibility that we have to have in it. We don't want to drop a zero day when people aren't patched. There's discussions around, well, when's the appropriate time to actually release a POC after patch has been... those are all good discussions I think to have. But at the end of the day, we have to have the same types of tools and the same type of collaboration that the organized crime groups, that the nation states do in order for us to have any type of way of handling these types of threats in the future.

Joseph Carson:
Absolutely. For me, I'm always worried about the unknown and I'd rather get things out there as quickly as possible, so you actually know how to deal with the threat, you know how to deal with it. Because once it's out there, then you actually understand what the risks are, how you can mitigate it, how you can harden and how you can actually make it more difficult.

Joseph Carson:
Because one of my things is that I agree that, once the attacker is in the door, I've actually found that, it's actually better to force the attackers to take more risks. If you get them to actually take a more risk and repeating their techniques and processes over and over again, and you provide them more hurdles to go through, what happens is, you forced them to create more noise. I find that the more noise they create in the network, the more chance you have of detecting them and that better chance that you have at basically preventing them from doing something catastrophic to your business.

Joseph Carson:
That's ultimately kind of what we look to do is that, you always assume breached, you always assume that they have access. In a lot of cases I've done in some response, I ended up finding that you'll find another attacker that's on the network at the same time, because you're starting to look, you're starting to actually pull through the logs and you find that, you're not just dealing with one attacking group, you're dealing with multiple. Maybe they have different motives in the environment, but that's a lot of times ... covers that you're dealing with sometimes with multiple groups.

Joseph Carson:
What's your thoughts around those, is that when you're actually getting into those and you starting to do threat hunting, should organizations always assume that they're breached and continually looking?

Dave Kennedy:
Yeah. First of all, a couple of topics there that I want to hit on. If you look at something like Hafnium, for example, where China was actively exploiting, 20,000, 30,000 plus on-premise exchange servers, the cat's out of the bag at that point in time.

Dave Kennedy:
What was interesting to see is that when Hafnium was being exploited, the patch was out for about a week. A proof of concept came on, GitHub removed that proof of concept out from the system. Which, I can tell you that, our friends at Incident Responders, once they had access to that POC, it was substantially easier for them to look for artifacts on those systems and to be able to identify what was occurring.

Dave Kennedy:
The collaboration that we had between the red team and the Incident Response from TrustedSec, you're not going to get that if you're just looking at it from appearance and response perspective. Well, Microsoft release was not sufficient to do a full investigation on an Incident Response side. So, that type of release helps things out. The POC intentionally broken, missing a very key critical part for remote code execution, yet it was still removed and we can't have that. We have to be transparent about what's occurring out there.

Dave Kennedy:
I can understand if it's an O-day and no one's patched, but I also understand, listen, it takes time for these organizations to patch as well and we have to be very mindful of that. We have a responsibility for that. I probably would have waited a little bit longer to release a POC. But at the end of the day, we shouldn't be hindering researchers.

Dave Kennedy:
But on that point, threat hunting is probably one of the largest missing links that I see in most organizations. You look at the cyclical effect of security operation centers and, traditionally SOCKS that just sit there and they have the default alarms that they get in their SIM or their EDR product and they assume that they're fine and good, and it never matures. They rely heavily off of the technology itself versus trying to understand what attack patterns could potentially happen in their environment.

Dave Kennedy:
So, things like assume breaches where you come in and you go through a series of emulation cycles. Or, if you're at a level to handle simulations, you start to look at their environments and say, "Well, here's where you have gaps in your environments where you can't identify specific threats in those specific chains." Your whole goal is, if you look at something like the Mitre ATT&CK framework, is to have really good coverage and effectiveness, of those different types of attacks that can happen throughout different stages of an attack. So that as you'd mentioned, it creates more and more noise so that you're responding to those much more effectively.

Dave Kennedy:
So, for me, threat hunting and going through the cycle of looking at not just north to south, but east of west traffic, looking at process relationships, persistence hooks like registry modifications, PowerShell syntax, scripting languages, living off the land. There's a whole attack surface specifically on windows, Linux, and OSX that we need to be able to identify to look for unusual patterns of behavior and baseline that pattern of behavior and environment, if it's normal, if it's legit and then look for those deviations. That's really where threat hunting comes into place.

Dave Kennedy:
The missing link is, that threat hunting team also has to provide a input mechanism for the security operation center to get better at monitoring detection. So it becomes this circle where your SOCK is improving, your threat hunting teams are assuming that there's a breach and you're going through those cycles of looking through information. They're building detections based off of the threat models and the different adversaries that are in play and all of a sudden, your entire security operation center and your monitoring detection program elevates very fast than it ever has before, because it's not staying in that stagnant state.

Dave Kennedy:
But a lot of companies can't grasp that concept of, well, we installed the tool, we're good. You're not good. No, you're...

Joseph Carson:
Absolutely.

Mike Gruen:
Well, the other thing on that, I mean, having worked on user entity, behavior analytics, but product, one of the things we know is that like machine learning and AI, like AI is just a bunch of statements and automation and machine learning is all well and good. But you can fool these models. If you do things right, those tools will never detect it as an anomaly, because it's not an anomaly anymore and so you have to be active, a human has to get involved at some point, and you can't just rely on these tools and these technologies, which can be fooled, which can be taken advantage of and make you feel like you're secure when really you're not anymore. I think that's an area that's missed by a lot of organizations as well.

Joseph Carson:
Yeah. In addition to that, one of the things Dave, I got into, it reminds me years ago, I dealt with... fortunate enough, we prevented a major breach from happening. What was happening was that, we were actually going to deploy it ourselves, but what prevented it from happening, actually the attackers gain access to a software catalog, basically embedded and executable. We were going to deploy that using basically a software delivery mechanism.

Joseph Carson:
Ultimately what happened was is that, the attacker they knew our basically schedule. They knew basically everything we were going to be doing. They had access to our calendars and project, but what happened was, we basically had a gut feeling, we decided to do unpredictability. We decided to do just this ad hoc and let's double-check, let's run physically file ... the libraries, to make sure we have the right versions and all of a sudden we had a mismatch. Something that was unpredicted, something that was ad hoc, just something we just physically did sparingly without having a plan or having basically a project behind it. We decided let's do this, let's do it right now.

Joseph Carson:
I think for me, even when I get into instance and I look at the logs, there's so much noise out there when you do have an attacker that's in the environment. You'll see a lot of the data in the logs, it's right there in front of it. It's just organizations are not looking sometimes in the right places. Should organizations get into more doing those types of ad hoc? Let's say sporadic unpredictable types of basically, looking at logs and go, "Let's go all of a sudden and look in this location and let's go and look at the logs on certain systems and see is everything clean. Is there any suspicious activity?"

Joseph Carson:
How should organizations really get into, getting outside of the norm, just, let's be relying on those default settings and predictable outcomes?

Dave Kennedy:
I think you hit the nail on the head from the perspective of, you need to understand and know your environment and look for the differential behaviors that occur in your environment that are not normal. When you look at the different techniques that attackers use, I mean, it's not rocket science and majority of them use the same type of methods. You look at REvil, for example, they focused on living off the land and Rundll32 injection, for DLL injection to download their binary. Same thing happened for DarkSide.

Dave Kennedy:
You look at the most latest attack coming from the same group that was specifically behind SolarWinds, the Russian FSB attacks. When you look at the methods, everybody looks at that, oh my gosh, they're using ISO file to get their malware. But if you look at all the stages afterwards that occurred, well, yes, they, might've used kind of a novel attack using an ISO as a way to get around and skirt around detection. But then you look at like, okay, well, they use Rundll32 to import in this executable, this binary, which then calls out and downloads your second stage, which then calls Rundll32 again, which then does registry modifications in Run for registry modifications for persistence. All of those are clear red flags, that something is wrong in your environment. You don't see those happen on a regular basis.

Dave Kennedy:
There are very specific patterns that we can look for in our environment. One of my biggest areas that I love going through, I can spend all day, all night, looking at parent-child process relationships. I get so much value from parent-child process relationships, because you can literally spot an attacker and most are extremely, extremely good in those parent-child process relationships.

Dave Kennedy:
For example, why is Explore ... kicking off, Outlook ... kicking off, Excel ... kicking off. It's like, boom, you have something right there that you need to go and investigate clear as day. It could be a business process, but most likely, probably not. I think it's really important for organizations to one, A focus on visibility. Number one. We need to have access to the data. Installing things like SIS monitor, for example. Event log tracing for Windows is fantastic, from a logging perspective or ways that you can get other logs off of your systems to centralize them, to be able to conduct these types of activities and then getting into a cyclical effect.

Dave Kennedy:
You look at some organizations that have dedicated threat hunters that are doing it every single day, every single minute, every single exercise, most organizations can't do that. But now, let's just say you can do it once a month or once a quarter, go through that data to have that visibility and to look for those abnormal patterns. Then from there, writing detection so that you don't see them again. That is as a way to continuously improve your monitoring attack capabilities in your environment that most organizations just don't go through. It's just carving a day out, carving two days out, carving an hour out, carving a lunch out and just going through that data to figure something out and, getting into a cyclical effect of doing that regularly, I think is really important.

Mike Gruen:
It's interesting that you mentioned that, because it reminds... so my background is a software engineer, one of the things that we enjoy doing, and we recently did at Cybrary was like a hackathon, where the software devs and sales and all parts of the different organizations work together to come up with cool, unique product features and whatever. Hackathons have been sort of proven pretty successful for jump-starting innovation. Taking that same sort of idea and applying it and saying, "Hey, you know what? We're going to carve out two days and we're going to do more of like a real hackathon of actually looking at our systems." Doing that I think is an interesting...

Dave Kennedy:
Hackathons are great. There's a number of companies that we do work with that have regimented hackathons where anybody can compete in either, have simulated environments they've run through. But that type of stuff creates, a lot of creativity. It also from a software development perspective... one of the things I remember I was doing an assessment for an organization and we were completely flying underneath the radar. We had access to a bunch of systems and we were high-fiving each other because this was a really tough customer to get into from a red teaming perspective. We had accessed a database from another server to get some of the data of that database.

Dave Kennedy:
Literally the database administrator had done a hackathon and was super paranoid about unusual connections to their database and was monitoring all the IP addresses coming into the server, literally real-time. He had a window open of the ... destinations. I'm like, "Seriously, someone actually does that." It caught on. It was because of that hackathon that literally spawn his interest of, well, I'm going to look for unusual patterns because I own this infrastructure, and I'm in charge of it, so I'm going to ensure that no one is connecting to it from an unusual perspective and literally snagged us and it blew up the whole thing, so had to start from ground zero again. So, those types of things are huge and getting people into that mindset to understand offensive capabilities is fantastic.

Joseph Carson:
Yeah, absolutely. I think one of the things you mentioned earlier that I think is critical is, you mentioned about the red team, blue team working together. We have this more concept than purple teaming now. For me, I think that's one of the things that organizations really need to embrace, because it's how you basically cross-pollinate. Some of the hackathons, you bring a lot of people from different skills and backgrounds and knowledge together, in order to pair them up into basically creating something great, innovative.

Joseph Carson:
I think that's really where we need to have from my red team and blue team perspective, is we need the defenders to know the techniques and the methods that attackers use. So, basically we need to educate them, because it really helps basically... One of the things I always say is that, the best way you can defend is knowing hacker techniques. You know basically the methods that they use, or you can actually know what to look. Just as you mentioned that, the person looking for those database connections, basically once they knew that that was something to look for, they implemented and been monitoring it consistently. How important is purple teams today? Is this something you see as much more, the industry needs to embrace and develop further in the future?

Dave Kennedy:
I am such a huge advocate of purple teaming. I understand why, I think the industry has gotten caught up on just doing penetration testing and they hear the sexy word, red teaming. I would say 90% of the customers that we perform red teams on, are nowhere near the ability to handle a red team or what a red team engagement produces. There's a big difference between adversary emulation and adversary simulation.

Dave Kennedy:
In adversary emulation, you're taking known tactics, techniques and procedures of attackers and you're working collaboratively to figure out where you have gaps and weaknesses in from your monitoring detection program, that purple teaming aspect, where it's not about being covert, it's about being over it. You're spending time with the blue team and going back and forth and you walk out of there with 20 kick-butt detections that you never had before that are high fidelity, high confidence detections out there that are going to help identify attackers now, that you didn't have before in the past.

Dave Kennedy:
They want to skip that step and say, "Well, I want a red team. I want to simulate this nation state, this sophistication level." Then you completely wreck them and they're like, "What are we supposed to do with this?" I'm like, "Exactly, you're not at a point yet to handle the simulation efforts because your program is not mature enough yet to get to that specific point." So our whole goal should really be, no, listen, if you need penetration tests for validation, for verification, that makes sense. Compliance, et cetera, whatever, but really our efforts should be doing more over testing to understand where do we have weaknesses in that? Not just the initial access phase around, hey, we found a technical exposure to exploit or deficient user or whatever, but assume that there's a breach there and what happens after all of these different phases of an attack to really build your detections out.

Dave Kennedy:
That's really where we need to mature as an industry, because not a lot of companies do it. It just boggles my mind because I'd rather walk out of an engagement with 20 new detections than have one or two technical fixes that I'm going to fix and maybe, some strategy things that I need to fix, but I'm not walking out of there with anything that's really made my security program substantially better.

Mike Gruen:
Yeah. Having, I think a number of years ago made a similar mistake where, we hired a security company and we started working with them. The nice thing was we went into it going, we just know that we need to improve our security and so how what's going to work best? In our head it was yeah, obviously we want you guys to do like a pen test.

Mike Gruen:
They went through the exact same explanation of, "We can guarantee that we're going to get in. Let's not do that, let's actually help you to understand what you really need and let's go through all of this." That was so beneficial and so eye opening and so much better than that conversation of like, "Yeah, we're going to get in." That's just a waste of 20, 30, 10, whatever, thousands of dollars that you're going to get like one or two things out of.

Dave Kennedy:
When you're getting an assessment done, let's just say by a third party or even your internal organization, you're very much locked into a fixed time window. So you have this amount of time to complete. A lot of customers are like, "Well, hey, I want you to get around our EDR. I want you to do this. I want you to do this one." Nope. So we'll spend all this time getting around their products and by the time we get access to a system, and we have remote code execution and we're on a system, we're already 80% through the hours that we've already done.

Dave Kennedy:
Again, your value is substantially lower of what you're getting for those types of engagements. Yes, we proved we can own you, great, that's anybody. Right? But what happens after the fact? Our mindsets really have to shift away from that initial access component. Yes, it's important to do ... management. Yes, it's important to build protective mechanisms, to shut a lot of these things down ahead of time. It's important to have multifactor authentication, network segmentation, et cetera, et cetera, et cetera. But at the same time, what happens when an attacker successful and can we identify those different stages of an attack and boot them out much faster?

Dave Kennedy:
That's really what our mind shift has to be in security in order to have any type of defensive approach. I guarantee you, I don't know anything about Colonials program. But, rumor is they had McAfee, the malware itself shut down McAfee. The first thing they did was shut down McAfee. That was their only way of identifying a threat. Attackers literally had maintained access across their entire environment after that component was eliminated. You can't have that. We have to be focusing on all the different stages after if they get around a preventative mechanisms.

Dave Kennedy:
This isn't McAfee or anything else, I'm just saying you went out the window, once that McAfee was circumvented.

Joseph Carson:
Yeah. Absolutely. In a lot of the cases, what I see and, one of the things I've learned a lot is basically from an adversary techniques, actually going into the incidents and preventing them or stopping the midway through the attack and all the things they've left behind, that's where I learn. I've learned so much from basically attackers leaving a lot of breadcrumbs and scripts and everything behind.

Joseph Carson:
Ultimately what you have to define is similar, the ones you're missing, is that you might have a lot of security on an endpoint, on a device or a server. But if you're giving somebody local administrative rights on that system, all of a sudden they simply run a script and it switches everything off, they can do the changes, modifications, they can change the registration event.

Dave Kennedy:
At least make it a little bit difficult for them.

Joseph Carson:
Exactly. But some people have a missed assumption that since they're running these, but they sometimes assume that a local administrator on a system is limited to just that one system. But basically what it simply allows them to turn it into a staging machine. That, that machine can simply be turned into something that can then be used for enumerating, the entire network for them. Basically you'd be standing and watching and living off the land and learning about what other security defenses you've been placed.

Joseph Carson:
Absolutely, I think you're right, is that sometimes, let's assume that you already have access, let's assume you have gain access to a system and the environment, and let's do the test from there to see if you're triggering the right alarms, that are creating enough noise so that your defenders can actually detect it. I think that's really where we need to stop assuming that there is that castle, that there is that perimeter anymore. We have to assume that the perimeter is gone there in that network. How can we further detect them when they're in that network? How can we make sure we have that visibility?

Dave Kennedy:
What you said right there for some reason is so difficult for organizations to understand, they assume that, hey, we bought this firewall, we bought this specific piece of technology. We're good to go. What they don't recognize is that there's more things after that, that we have to focus on, it doesn't just stop at preventative. When you start going through these emulations where you assume a breach occurs, and you're starting to test, what you can see from an... Again, when you start to go through these assumed breaches, there are preventative things you can put in place to shut them down too, so it's not just detective, it's also, "Hey, is there a preventative mechanism that we're not using?

Dave Kennedy:
For example, why should a regular user ever call regsvr32? There's no reason. This is what the new directory commonly used as a living off the land for remote code execution and downloading. So can we disallow regsvr32 from ever being used from an application control or, allow listing perspective, to say, "Okay, well, we're just going to shut this down from a regular user context." There are preventative things that we can gleam off of this that make it much more difficult for adversaries and for those that we can't do preventative mechanisms, placing in detective controls, allow us to be able to identify them as it's going along, but you have to go through those cycles.

Dave Kennedy:
Again, the industry is like, well, hey, I had a pen test done this year and we did awesome. We're great." Well, okay, that's one specific test. How do you actually fair when there's a compromise in your environment? As you had mentioned, assuming a breach has occurred, testing it from after an initial access and going in after that obvious is insanely valuable and purple teaming literally should be what we're focusing on 90% of the time, not the covert testing that we typically do.

Mike Gruen:
I'm optimistic in terms of the shift, I think part of the reason why it's been so hard, it's not that the organization itself can't understand it or grasp it, it's more that the costs haven't been as clear. The risk and how much it's going to cost me if this were to happen and so on and so forth. Those costs just continue to go up and up and up and so I think more and more organizations are being forced to...

Dave Kennedy:
Look at the newsroom right now with Colonial Pipeline and JBS, these aren't sophisticated actors, this isn't like nation state Russia coming in from a supply chain perspective for remote code execution and tucking back doors in and pushing updates out. These are relatively less sophisticated attackers using common techniques that we all are very well aware of.

Dave Kennedy:
But yet the shock of that and what executive... I mean, I can't tell you how many board calls I've had with different companies like, what are we doing for ransomware? Hey, by the way, we should have been talking about this 10 years ago, not now. But second, here's the things that we need to start focusing on.

Dave Kennedy:
I think you're right. I think that conversation of, well, maybe we aren't protected, maybe we don't have the right controls in place, maybe we do need to look at this, I think that's starting to happen now. I think the businesses are really starting to understand the destructive nature of what's actually occurring here. We've been talking about it for years, like critical infrastructure attacks we've been talking about for since I started in ... 20 plus years ago.

Mike Gruen:
Since Ukraine.

Dave Kennedy:
We've crying wolf forever, it just finally happened. Right?

Mike Gruen:
Right.

Dave Kennedy:
I remember I was on panel over at Rapid7 and I was down there with H. D. Moore and H. D. was wrong by the way, I'd like call it out there. He's a good friend. But he was right. No, he was just too fast to think about it. He said, "I believe we'll see a major attack on our oil and gas, that could be potential loss of life or major shortages or things like that." This was probably six years ago. He said, "I think it's going to happen within the next year." He was right. He was right on what was going to happen, just timing was a little bit off.

Dave Kennedy:
I think that's what we're seeing now is that, it's real, it's happening, it's happening to larger and larger organizations. You look at Universal Health systems that shut down their ERs. You look at Acer Computers, they couldn't manufacture computers. You look at all of these different companies now that are pretty large organizations are being completely shut down. I think, companies are like, "Okay, we're on notice now and the government is not going to protect us. We have to do something, so what is that?"

Dave Kennedy:
I think that conversation is finally starting to occur.

Mike Gruen:
Absolutely.

Joseph Carson:
I mean, I agree. It's a couple of years too late. I mean, we've already had one in Ukraine NotPeyta. We had the Ukrainian energy sector shutdown. So these are things we've known for several years.

Dave Kennedy:
But I think that's where-

Joseph Carson:
But it's always been a bit further away. It's been ... other companies since they've been impacted by this.

Dave Kennedy:
But we can't get our meats, that's a major problem.

Joseph Carson:
Yeah. Right.

Mike Gruen:
Right.

Dave Kennedy:
If I'm filling garbage bags with gasoline, then-

Dave Kennedy:
I don't know if it was real or not, but it was like two guys with AR-15s, with plastic bags, filling plastic bags and I'm guarding these plastic bags of the AR-15s, I'm like, "Man, this stuff is getting real." I mean, like this ... is getting real.

Mike Gruen:
Yeah. I saw a video of a guy filling the back of his pickup truck and I was like-

Dave Kennedy:
This is crazy. It's crazy. Right?

Mike Gruen:
... "Are you going to drive around with gasoline just sloshing around back there?"

Joseph Carson:
Fear makes people do some things.

Mike Gruen:
It wasn't that bad. It really was not that bad. We had like week of disruption, which I'm not saying it was a good thing, but it wasn't like we were looking at months of outages or things like that. But you see how much chaos this creates by one organization, 85-plus percent critical infrastructure owned by private sector, can cause a disruption from a cyber perspective. I think it's finally clicking to people that, hey, it's not just our iPhone that can get hacked, it's not just our computer and our pictures that gets stolen, it's like literally how we live our daily lives. The complete disruption to our food, to our energy, to our water, I mean, all this stuff now is possible. We've known it's possible in the secured industry, but now executives are like, "Whoa, we've neglected security forever. We actually need to do something here now."

Dave Kennedy:
Right. Or maybe not neglected, but certainly not invested as much as we should have. I mean, I-

Joseph Carson:
I would say that most organizations have just done the bare minimum to even call it security. That's the challenge.

Dave Kennedy:
Right. I mean, I think there is a lot of box checking. There is a lot of, "Oh, well we want to get this contract. What do we have to do? Oh, well we have to pay these guys some amount of money so we can check this box." There's no question about that.

Dave Kennedy:
I do want to get back to the purple team and learning and stuff.

Joseph Carson:
Yeah. I want to kind of get around the security research side of things and the purples teaming side. Is that how important is security research in the industry today and the assuring side of things? I'd like to get your thoughts. Because sometimes I find that there's a negative perception against the security research industry, especially from a media perspective that creates a negative.

Joseph Carson:
I consider myself a hacker, but the problem is that, that's such a bad persona. Basically a recognition in the media industry that they see as the hacker is such a bad person, but majority are good citizens and security research is so vital. A lot of security researchers are sometimes putting even there basically personal lives on the line to do research because there always is that gray line between the legal aspect of things as well.

Joseph Carson:
How important do you think it is in the recognition that some people shouldn't deserve to get for at least bringing things to the surface?

Dave Kennedy:
Yeah. Security researchers for me, there's Kevin Durant, I mean, who always says, "You're the real MVP, if you're in the basketball." Hopefully people have seen that I made reference here, but security researchers are the real MVPs.

Dave Kennedy:
It's important to understand that security research drives this industry forward, whether that researching and attack and understanding how that attack works. It's not just offensive security researchers, it's researchers that are reverse engineering malware and figuring out how they work. The first one to get a sample and then share that out with the community so that we can all kind of break it open so that everybody else is protected, all the way to coming up with a new way of identifying attacks or coming up with a new attack vector itself.

Dave Kennedy:
Security research is literally, you're talking about a building, it's the foundation for the building, without security research we don't have any insight into security products or products in general, how well they fare against attackers or adversaries that could be potentially exploited, as well as what other adversaries are doing. So for me, it's a vital component of everything that we do day-to-day. At TrustedSec and at Binary Defense research is paramount.

Dave Kennedy:
One thing I'll say is, a few years ago, this was about four years ago. One of our folks on our team Justin Elze, who runs our research team as well as our red team and Carlos Perez who runs our actual research team, but Carlos reports through Justin. Justin said, "Listen, it's getting increasingly harder for some of our more sophisticated customers for us to break in and it's also very vital for us to have our own tooling weaponization."

Dave Kennedy:
So, we've invested very heavily in our own research division over at TrustedSec, about four and a half years ago, and so we have a fully dedicated research team that all they do is look at weaponization tooling, detections, purple teaming aspects, things to that effect so that we are continuously spawning innovation from ourselves to make our own teams better. It's been really successful.

Dave Kennedy:
I think you really need to have that today. Whether it's public research or your own private research team, if you can do it, to really keep up with what's happening out there. I really believe that the vilification of some of the security researchers that we've seen is really a myopic view of the grand scheme of things, of how the whole security industry works together.

Dave Kennedy:
Now, again, I want to say that there's responsibility that we have to have in the security research side, dropping zero days, time of the past. That was the cool thing to do back in the Millwork days and everything else. But that times have changed, from that perspective. We have to focus on responsible disclosure, to impact... Do no harm, right?

Joseph Carson:
Do no harm.

Dave Kennedy:
Yup. I think that's an important case to remember here is that, security researchers deserve the credit for the time and effort. I mean, some of these exploits, it takes months and months and months of research to go and do, and it's just these brilliant minds doing incredible things. I mean, I remember when I first wrote my first zero day, and it took me... and this is by the way, it was way easier back then than it is today with all the preventative mechanisms that you have and going through them.

Dave Kennedy:
When you finally take like a bug, for example, a crash and you hijack that crash and you understand how the logic works and you redirect execution flow and you're able to get remote code execution onto a system, it's the most amazing feeling ever. But man, it took months to do something like that and time and effort. So the security researcher should definitely get the credit that they deserve and really driving the industry forward. At the same time, obviously be responsible with it. But we should recognize that and we should continuously promote that, because it's important for the industry to continue to move it forward or else our adversaries are not going to slow down.

Dave Kennedy:
I mean, it's not like, "Hey, Russia and China, Iran, North Korea, we're just going to stop hacking tomorrow." Not going to happen.

Joseph Carson:
You're actually building teams to do specifically just that. So, we have to make sure that our resources are getting skilled and on top of things quickly as well.

Joseph Carson:
Mike, did you want to add anything more than that?

Mike Gruen:
Yeah, a little bit. We were talking a little bit about learning and how the information sharing goes. I think one of the things that I learned over my career is going to certain conferences and things like that has been a great way. Even as a software dev, going to some of these has what has led to me being in more security oriented technologist, if you will, depending on who you ask.

Mike Gruen:
I'm curious what your thoughts are on that? I think one of the nice, one of the pluses that came out of COVID is, how many of these conferences that were totally, when I'm saying in real life have now have a virtual aspect. I think it really opens the door. I'm just curious what your thoughts are on how to get more people to these conferences and conferences and more conferences?

Dave Kennedy:
Yeah. Conference are really important. For one, the relationship aspect is key. That's probably number one for me. You get to meet peers in your industry or somebody that is like-minded to you that you can learn from. That's really important. When I was first coming into the industry, it was a very different industry. There wasn't a lot of technical information out there, there wasn't documents, there wasn't training courses., there wasn't college programs for cybersecurity.

Dave Kennedy:
I got lucky because I got part of a group that was called Remote Exploits at the time in IRC and Remote Exploit eventually actually turned into Offensive Security. So the guys that made BackTrack and ... and Kali and became very good friends with Muts, Mati. I was able to learn from that whole group and that whole team and become kind of one of their other core members. I was in the BackTrack development team. I helped with Exploit DB.

Dave Kennedy:
I learned a ton from people that were figuring things out that hadn't been done before and that was really instrumental for me, having that type of peer group that we can learn of off one another and then, really skyrocketed my career. I think, connections, networking, especially that you get these events either with in-person or remote, are really important. But also, you find a topic that you like, and you can take a security researchers topic, and you can expand on that. You can take that concept and use that somewhere else to make it even better. That's the cool part about this industry.

Dave Kennedy:
I think conferences really provide a lot of great valuable from a talk perspective, as well as, the networking aspects. Obviously, I started at a conference called DerbyCon with a few of my good buddies. The biggest thing with DerbyCon was that community feel, where if you're brand new to the industry or you're seasoned, everybody was on the same level and we're all working together to try to help one another.

Dave Kennedy:
You look at that and you have the same type of feel at BSides. I'm a huge fan of Wild West Hackin Fest with John Strand's group. I think it was a really big one. I always go to ... I love it there, it's just like a middle of nowhere, you feel like you're in the old Wild West. Kernelcon.

Joseph Carson:
Yeah. One that I recently went digitally was Kernelcon, which was fantastic. Kernelcon was awesome. Watching Joe Grand running around trying to find burners to do some ... was quite impressive.

Dave Kennedy:
Joe was amazing. One of my heroes in the security industry and a friend. Kernelcon was great. I was there in person two years ago, prior pandemic. A phenomenal run conference as well. I think what you see is a lot of these smaller tight knit conferences for me are a lot better than the large sized conferences. I feel like I get lost in the big conferences. I feel overwhelmed at the big conferences, I can't network, I can't socialize, I don't know where to go, there's just a lot of stuff out there. These smaller sized conferences, it's definitely much more intimate and you get to hang out with people, meet new people.

Dave Kennedy:
That's the thing. A lot of problem, I think we have with DerbyCon, is that DerbyCon started off of as a small tight knit conference and it just blew up to epic proportions where like, "Hey, this isn't what we signed up for. This is too much. We're just running this for fun. We don't want 5,000 people at a conference." So we bowed out on that.

Dave Kennedy:
But it's one of those things where you look at the tight-knit conferences, like what BSides is, and a lot of the ones that spun around there and you just find so much value from those types of conferences. I love going to smaller size conferences. In fact, when I get asked to go to smaller size conferences, I prefer those over the big ones, because I find it just to be so much more valuable for me.

Mike Gruen:
Yeah. I think one of the downsides to the big conferences it's like anything, whenever you're talking about research, when a big organization is sponsoring research or a big organization is sponsoring a talk, there's this other mode of which is to sell you on something. Whereas when you go to the more intimate, the smaller, the people who are just doing the research, it's more pure-

Joseph Carson:
It's educational.

Mike Gruen:
You're getting to get those more raw, consumable, applicable information and they're not trying to sell you on a solution to the problem, they're just showing you the cool tools. I, 100% agree with the pick a topic, that you're interested in and it's going to be applicable to other things.

Dave Kennedy:
Oh yeah, they're great.

Mike Gruen:
I learned so much from their car hacking guys that applied to what we were doing with this-

Mike Gruen:
I mean, we were building a totally unrelated system, but it had a message bus. The nice thing is we could actually throw real technology at ours, because we weren't limited by the budget of what a car can do and the timing and all the rest of it. So, seeing what they're able to do, and then being like, aha, this is what we need to do in our system to protect against those types of attacks, it's just so interesting.

Mike Gruen:
I definitely encourage people, even if you're not really interested in security research or becoming that type of security person, if you're involved in technology, if you're a software developer, you get so much benefit out of attending these things. I think, look for opportunities to, to learn what the other side is doing, because it just pays benefits.

Mike Gruen:
I guess that gets back to that whole purple team, red team, blue team, and how we can work together as an organization to just improve security. Or not as an organization, but as a industry to improve security across the board.

Dave Kennedy:
Look at the old school days of DEF CON at Alexis Park. I think that type of model works extremely successful. Because I remember, I was in the military at the time and they flew me out to go to DEF CON and I had just turned 21 which was a mistake to go to Vegas when you're just 21. I literally turned 21 two days before I got to Vegas.

Dave Kennedy:
But the best part about Alexis Park is, I remember seeing some of my early heroes in the security industry, like the Schmooze group, for example, or CDC. I remember seeing Bruce Potter up on stage and they had this UPS box that they would physically break into a location. It looked like a UPS, but it was actually sending all the data back and, doing that in the middle of attacks and all sort of stuff and we're giving them remote access. They did it for fun. It was just like, we're going to do this because it's awesome and it sounds cool.

Dave Kennedy:
That's what you get from those intimate type things-

Dave Kennedy:
... where someone's like, "Hey, this is exciting for me and this is awesome. I want to share this with my peers and my friends in an intimate type of environment, where I'm not at this big stage, where I got to have ATM spitting out cash and stuff like that." It's a different type of perspective and it's that community. We talk about, you heard the concept of tribes, the tribe of hacker books, the tribes you have. There's a lot of tribes inside of the information security industry. The community aspect of tribes to me is the most fascinating and probably the most collaborative and sharing that continuously drives industry forward. Those security researchers, is a great example, once you share that information with tribes that are similar to themselves.

Dave Kennedy:
So, it's really that fundamental piece where, we're all working together for one common goal to make the world a better place. That is a noble cause. It's one that I think is really good. I think those small conferences really promote that. Give it out to the conference organizers. I mean, it's so much stress running those and expenses and hopefully you get sponsors to be able to do it. There's companies like yourself. Obviously I've seen you folks sponsor a number of conferences out there before. I've seen your name all over the place. So, kudos to you for the fostering of that community.

Dave Kennedy:
But, we have to continuously foster that because the industry is getting very large, but we still have to have that intimate setting that it needs to be important.

Joseph Carson:
Yeah. We have to try to get new talent-

Mike Gruen:
Moreover and we need to continue to grow it.

Joseph Carson:
We have to get new talent and people that come in, because when people look at the industry it's quite scurry. That's one thing I wanted to touch on Dave, is I've seen in the past year, I've seen it kind of growing as definitely around the mental health side of things. Because we were very dedicated and passionate into what we do.

Joseph Carson:
I've spent countless hours and days just getting into resource and getting stuck into things and sometimes you forget to even step out and, see the daylight. I've seen it, probably be more in the past year, I guess, because of the pandemic and everyone working remotely. We're losing the intimacy from the conferences because we're not able to meet up with your friends and your peers to do that sharing.

Joseph Carson:
I've seen a lot of increase in social media in the past year where people are struggling. People are struggling by not getting that social aspect, by not being able to get into the community. Sometimes what do you recommend in the industry? How do people get help if they are struggling? What's the way and some indications of people to reach out to even us all?

Dave Kennedy:
That's one my biggest fear is for any of my employees at TrustedSec and Binary Defense. It's what we call burnout. Burnout is a real thing in our industry. Because, as you mentioned, we are very passionate about what we do, we always want to continue to seek help. We forget that we need to take time for ourselves and that our personal lives need to be more than just cybersecurity.

Dave Kennedy:
Now, there's a difference. For me, I made cybersecurity my hobby. I play around hacking and stuff like that for fun, but there's a time where you need to decompress, you need to give your mind a break, you need to go out for a walk, you need to do fitness, you need to spend time with your family. All those things are extremely important to prevent burnout. You also need to take time off, and that's the thing.

Dave Kennedy:
We force PTO to our employees as much as possible. I probably send a once a month, monthly reminders saying, "Listen, please, please, please, please take time off. I don't care if there's a pandemic, I know there's a pandemic out, just take time off and hang out at your house, do what you need to do for yourself." Because the time that you need to decompress and to unwind is so paramount to your overall stability, both as a employee, but also just as a person." That's the most important thing is that we're all people here.

Dave Kennedy:
There's a term called imposter syndrome and things like that, where people feel like, "Well, hey, I'm kind of riding off the back of others or, I'll never be to a certain level of a certain person." That's not the case. We're all here to share our perspectives of our life experience. Our experiences are unique to us, so you shouldn't worry about being an imposter, your experiences are you. It doesn't mean that you are a expert in car hacking, that's somebody else, you haven't spent your entire career focusing in car hacking, but hey, you spent your entire career working in this area and I would consider you an expert in this area. Do you know everything? No, I don't know everything. I learn something new every single day.

Dave Kennedy:
So, we have to recognize that there's no way that we can know everything. We have to take time for ourselves. We have to recognize when we're burning on. That's the biggest challenge I see most people run through, is recognizing the symptoms of burnout.

Joseph Carson:
Absolutely.

Mike Gruen:
Yeah. I mean, I think the imposter syndrome is one. So my dev team, a few, probably about a month or two ago, we were all just talking, we have a nice range of experience and years in the industry. I think one of the most universal themes, especially amongst the more senior people was not just that they suffered from imposter syndrome, but the fact that they were able to recognize it and turn it into a positive. That's for me as well. I know what I know and I know what I don't know. Sometimes I feel a little bit I'm an imposter or whatever, but I let that drive me and I let-

Joseph Carson:
Me too.

Mike Gruen:
... that like helped me to understand, that's why I have people like Jonathan on my team who can really keep me up to date and he makes me look good and stuff like that. I think it's an important thing to recognize that, yes, just accept it and learn how to deal with it in a positive way-

Mike Gruen:
... and turn it into a positive.

Joseph Carson:
Yeah. I think for me, Dave, one of the things is that, the most valuable thing in this entire world is our time. If we don't take time for ourselves, that's the most valuable asset is the time that we have in this earth.

Dave Kennedy:
We're not getting that back.

Joseph Carson:
If we don't take that time, you can never get it back. It's a one way road, it's a one way street that you can reverse it. If you get your time machine to work, I'll be the first person to-

Dave Kennedy:
Unless you have a DeLorean that you happen to be working on and-

Mike Gruen:
Let me tell you, I've been trying to get that done soon, but... yeah.

Joseph Carson:
It will only make you go back in time, or on the future, but it won't change your age. But I think that's a great point to end on, and I think it's really important is that, absolutely, take care of yourself as you're the number one important person and always make sure to take time. If you do find that you're burning out and you don't know how to deal with it, definitely reach out to somebody. There's a lot of people on social media, whether it be myself, Dave and others, that will basically be there to point you in the right direction or just provide you a set of ears to listen to.

Joseph Carson:
So Dave, any final thoughts for the audience that you'd recommend?

Dave Kennedy:
Yeah. One thing I wanted to share to close out with is, I sent an email just last Friday, May 28th, 2021 to my entire company about burnout. I do this once a quarter just to remind everybody that it's not a sign of weakness, that it's a sign of strength to recognize burnout. I just wanted to read it really quick if you're okay with that. I sent this out to my entire company.

Joseph Carson:
Absolutely.

Mike Gruen:
Please.

Dave Kennedy:
I said, "Hey all, I hope everybody's ready for a nice long and relaxing weekend here soon. I wanted to send out a reminder to everybody that burnout is a real thing. Recognizing that if you're burned out, it isn't a sign of weakness, it's a strength. We do some amazing work here on our customers and our performance levels are always at the highest capacity, that requires us to continuously be racking our brains and going above and beyond in every aspect.

Dave Kennedy:
I like to compare us to top athletes for various sports. Athletes are required to physically put their bodies through continuous strain and stress on an ongoing basis. However, they focus very heavily on ensuring their bodies have a chance to recover and take the time needed for that. Very similarly here, we're flexing our brain muscles on a regular basis and performing on athlete level. Your brain needs the time to unwind, to be there for family and friends, and to recover from the daily stress as we put our minds through.

Dave Kennedy:
It's important to take PTO and time off for yourself and for your family, it's equally important to not overwork yourself and recognizing when you are. If you are burning out or feeling stressed, please let us know. Again, we don't view this as a sign of weakness, but a sign strength and I kind of go on after that.

Dave Kennedy:
"But I just want to say, as much as I love working with every single one of you, at the end of the day, our lives are far more than work, being there for family, being there for yourselves and doing things that are hobbies and interests is what your life is about. We'll never look at someone taking time off or reduction of workload or falling behind as a weakness here."

Dave Kennedy:
That was kind of the message I sent out to the entire team just to say, "Listen, if you're struggling, just let us know. We can't help you if we don't know." That's, one thing I think important for managers, especially in InfoSec, is to recognize that we put our folks through a lot of stuff, that we require them to do on a day-to-day basis, and they're so compassionate about doing that work and they know that it's for the greater good, so they get wrapped up in those day to day aspects. So, we have to recognize when burnout is there and really help our folks on as it goes along.

Dave Kennedy:
But, to close on that, mental health is important. Industry-wide, I'm actually really optimistic on where we're at and where we're heading at from an industry perspective. I think yeah, we're at a spot now that we never have been in the security industry. I think, we're going to have definitely challenges, we're going to have organizations to continue to get breached. But at the same time, I think we have the right mindset of defensible strategies.

Joseph Carson:
Awesome. Absolutely. I think that was amazing. I think that's not just a message to your employees now, but everyone who's listening, whatever company or whatever you're doing, is really take that message and make it actionable. As I mentioned, time is the most important asset we have in this earth, so use it wisely.

Joseph Carson:
Dave, it's been awesome having on the show, it's been an honor. Mike, great having you leading the cohosting and the show today. So it's always awesome, it's fantastic. For the audience out there, definitely tune in 401 Access Denied. We're here to provide you, as much education as valid as possible, and hopefully you get some lessons and some great ideas to really make the world a safer place.

Joseph Carson:
Again, thanks for listening. It's been awesome. Take care. See you next time.

Mike Gruen:
Thanks everybody, we appreciate it.