EPISODE SUMMARY
Subscribe or listen now: 
Joseph Carson:
Hello, everyone. Welcome back to another episode of 401 Access Denied. I'm your host for today's podcast, I'm Joseph Carson, chief security scientist and advisory CISO at Delinea. And I'm really excited, welcome back to another awesome, amazing guest. So, this guest has been on previously and has been one of our top-rated podcasts to date. So, it's great to welcome back Dustin, EvilMog, welcome to the show. Can you tell us, for the audience, a little bit about who you are, what you do and some of the things that you love to break and have fun with?
Dustin Heywood:
Yeah, not a problem. So, hi, I'm EvilMog, otherwise known as Dustin Heywood. I am now the chief architect of X-Force and senior technical staff member at IBM X-Force, so that covers red and IR. I'm also known for my exploits on Team Hashcat where I'm known as EvilMog, we've won Crack Me If You Can for a number of years and come in second for others. I collect black badges as a habit. Yeah, I'm black badge to DEF CON, DerbyCon, Rest in Peace as well as a few others like Chicago's THOTCON, et cetera. And I do a lot of password security research as well as Windows Active Directory and now, lately, root telnet security research.
Joseph Carson:
Awesome. Is the Church of Wi-Fi making a comeback this year?
Dustin Heywood:
We are absolutely making a comeback this year. I'm actually performing at Render Man's wedding the weekend before DEF CON in the full, ancient Church of WiFi tradition, so that should be unique.
Joseph Carson:
So, for anyone who's going to a Black Hat in DEF CON, that's definitely going to be an entertaining show this year for, what? DEF CON 30, I guess it's going to be a big one for everyone.
Dustin Heywood:
Yeah, I'm also running a contest at DEF CON this year again because I also run the DEF CON MUD. So, for those who don't know what the MUD is, this is going back into the '90s. MUDs were text based games that were the precursors to all modern, massively multiplayer online RPGs. This is the text based versions of it, we're bringing back a new and far more evil one again and, because it's a sanctioned DEF CON contest, I usually give away at DEF CON ticket to the next year.
Joseph Carson:
Awesome, awesome. So, how many black badges you got now? Is it two to three, three?
Dustin Heywood:
Four.
Joseph Carson:
Four? Okay.
Dustin Heywood:
So, my first one was to SkyDogCon, which was a little conference out of Tennessee, it no longer exists. This actually started the black badge curse. My second one was the DerbyCon because I got awarded DerbyCon 8, DerbyCon cancels DerbyCon 9. I won my DEF CON black badge at Hacker Jeopardy the year before the pandemic hit.
Joseph Carson:
I was there.
Dustin Heywood:
Yeah, you were there for that.
Joseph Carson:
I was there for that one, absolutely.
Dustin Heywood:
Yeah, so we won that one then DEF CON ends next year because of the pandemic so it got canceled. So, the joke is, every time I get a black badge, the conference gets canceled.
Joseph Carson:
It ends.
Dustin Heywood:
Yeah, then I bought one for the Chicago's THOTCON at one of their auctions so I can help them deal with some of the pandemic expenses. And then we won one at Hacker Jeopardy and so I gave the one I bought over to one of our members and kept the one that I earned. So, that'd be four and a half, five.
Joseph Carson:
That's pretty impressive. For many, getting one is a lifetime achievement. So, round of applause, that's definitely amazing achievement. Can you tell the others, how did you get into password cracking and cracking passwords in the first place? What was the thing that brought you into that journey?
Dustin Heywood:
You know what? That was entirely accidental. So, what happened was, I used to be a network engineer. Well, I can't say engineer, I was a network technician over in Afghanistan hooking up comm towers as a civilian contractor. I came home and then I got a job, someone took a chance on me as a security analyst so I was working at a government owned bank and we had to go start doing Active Directory password audits because people were reusing passwords. And so, we started doing this on a weekly basis and then I'm like, "I need to get some more power."
Dustin Heywood:
So, Hashcat was just in the pre-alpha stages, you needed a beta key to use it back in 2010, 2011. So, I got involved in the IRC channels, et cetera, and we got invited to go help out with Crack Me If You Can because I was good at analyzing patterns. And then, eventually, we started building a large GPU farm at the last company and we got involved in everything from drown to a number of academic papers and then we just kept bringing larger and larger amounts of hardware and that's actually how X-Force stole me, they needed a password guy and I got poached and it's been bliss ever since.
Joseph Carson:
That's awesome. So, many of us in this field, we tend to fall into the areas mostly because we focus on a specific area that we enjoyed and we just basically got embraced into it and, all of a sudden, that becomes, definitely, the expertise area and it's a lot of fun as well. So, question, when you get into this, what's some of the things, the shocking things that you see in the state of passwords today? What's the challenges? What mistakes do we commonly find? How attackers been successful? What's, let's say, the top five mistakes that people make?
Dustin Heywood:
I just want to say the biggest one is reused passwords. So, this has been the same problem since the '70s. The password, really, is just a shared secret, there's nothing special about it, it's just something that human can type in. The thing is, we as humans are incredibly intelligent people but our long term memory and even our short term memory, sometimes, can be a little shoddy, especially if you're trying to remember a long sequence of strings, I'm not going to remember 400 different passwords. The problem is, in order to stay secure, I need 400 different passwords on every single site. Yeah, so you have some options, what do we do with this?
Dustin Heywood:
We can write them down, we can use password managers or we can reuse the same password between environments. The biggest thing with passwords though, and I'm going to hate to bag on Microsoft but I'm going to for this moment and I apologize sincerely. Windows passwords, predominantly in a desktop and server environment, are based off of a protocol called NTLM. NTLM is basically MD4 which was, quote, broken back in the '80s. It is the single fastest password to crack on GPUs but that's not the problem with it.
Dustin Heywood:
The problem is, in Windows, password hashes like NTLM, not v2 or v1 or Kerberos or whatever else, but straight NTLM are password equivalent. So, even if you have 128-character password, the second somebody like a storage admin leaves or a Windows AD admin leaves or there's a breach or there was a pen test, that secret becomes known and that secret can be reused as if it was the password so they have to get rotated. Now, obviously, NIST special instruction state we shouldn't be rotating passwords which is great for things like your password manager. If you didn't have a really good, strong, complex password with multi-factor authentication and your password manager, that's perfect guidance. But people say we shouldn't be rotating passwords because at NIST, most end users or even CIS admins, like myself, get lazy, use a variant of the same password. The only good, secure password, to be honest, is one that's randomly generated and looks like line noise.
Dustin Heywood:
The other issue we've got is length. Now, because of this, say your brute forcing a password, I can go randomly guessing an eight-character password that a human generated thanks to the three or four rule. The three or four rule which is, basically, you have to select the digit, an uppercase, lowercase or a special or a three of those four actually allows us to reduce the search space a fair bit, especially on an eight-character password in Windows. A standard GPU or a 3080 or a bank of 16 GTX 1080s can brute force an eight-character password in less than eight hours. So, at this point, the eight-character password is effectively dead.
Dustin Heywood:
A 12-character password will take you, brute forced, let's call it 34,000 years, assuming it's random, lead generated and looks like line noise. Roll your face across the keyboard a little bit and you should be okay. The problem with this whole process though is most users will select something like summer 2021 or 2022 now, sorry. Yeah, we're spring 2022, it fits within the range. These passwords are easily crackable and those things will rotate. So, the only truly, properly, good password will be one that's randomly generated. If you have a random 12-character password with special characters and everything else, the sun will go out before I randomly guess it unless I'm really lucky and, because I haven't won the lotto and retired yet, good luck.
Dustin Heywood:
So, that's the real big issues we're seeing with passwords. But let's say we advance upwards to modern Ubuntu, just as the example, or most modern Linux. They use salted SHA512 crypt which is massively iterated. Trying to brute force one of those will take you a million years at 12 characters. So, that's why I'm saying 12 characters now is roughly about the correct length for a password as long as it's randomly generated. If it's not randomly generated...
Joseph Carson:
We're leaving it to humans to decide on what the password is. We do things that make us remember and that's the thing. When we're looking for something to remember, we will use something that's easy to remember. And one of the things as well is that, when you're looking to create your rules and asks and trying to get an idea of how to reduce that rule set or that word list down, what I ended up doing is looking at that person's history of password choices because you can find it. Most of the people on this earth have already been the victim of, basically, password disclosures either being in the hash format or being in the clear text.
Joseph Carson:
So, having already had your previous password decisions and choices exposed means that an attacker can simply take that as the base and are creating variations of that. And fairly common, people will take that same path, just like you said, spring, summer, winter 2022, add an exclamation to the end of it and you're done and we know that that's the trend and the habit. So, absolutely, humans should not be creating passwords, they should be creating passphrases and that passphrase plus a multifactor should be what's protecting the password manager and the password manager is doing all the randomly generated credentials for everything else. And that means that the difficulty of an attacker being successful significantly reduces.
Dustin Heywood:
Oh, absolutely. And I think what companies should be doing, and this is also aimed at some consumers, but companies especially should be issuing employees smart devices that are out of band from your typical Windows. And the reason for that is your smart device has things like device attestation keys, you can find out if it's been rooted, there is secure elements that you can store your password managers in. A mobile device is far more hardened or can be far more hardened, not in all cases, but in most cases can be far more hardened than your typical PC. So, say, a company pays $800 plus the 40 bucks a month for all their employees and then they provide them with a password manager, that's a huge step up in security for what I'd argue would be a cheap investment compared to the cost of a breach.
Joseph Carson:
Yup, absolutely. And this one thing as well, I think organizations should look beyond just their employees and really look into your suppliers and to your contractors. And one thing I remember years ago, one of the organizations that I was contracting, consulting for and it really changed my mindset when they realized that security starts with your social network around you. So, why not even get the employees' families using it, reward them. That ends up becoming a reward, becomes loyalty. They see that you're not just taking care of the company but you're actually extending security to the social sphere, to their family so that their family and kids can even extend to using password managers and reduce the threats that it is because attackers will actually target them first as stepping stones to get into your organization so why not extend your perimeter to the social sphere around the organization. Your supplier, your contractor, partners, your customers and everybody.
Joseph Carson:
I think businesses now that's actually going to implement logon systems to their websites or to their applications, password manager should be something they should be looking to integrate into as part of that default implementation. And moving away, I hate the term security by design because security by design doesn't necessarily mean it's on unused. I want to get to security, one of the CISOs, I remember having a roundtable last year, they said the term, "It needs to be security by default," and that's what we should actually strive to get to. And I think that's one of the important things is, when we're looking at implementing these, it should be on by default.
Dustin Heywood:
Well, it should be more than just on by default, right? It should be on and usable by default.
Joseph Carson:
Yes, absolutely.
Dustin Heywood:
Interesting. Everything we implement a security control, if it slows things down and that's friction, people are going to bypass it.
Joseph Carson:
Yeah, we're going to write it down.
Dustin Heywood:
Yeah, they're going to write it down. They need to understand that we are here to enable the business. The brakes on a race car aren't there to make it go slower, they're there to make you go faster. If you make things easy, people will comply. If you make things hard, they're going to work around it just like people stick webcams pointing at the multi-factor authentication token so they can go sign in if they forget it back at the bar or at the house or whatever.
Joseph Carson:
They take photos of their passwords so they can remember. Absolutely good, right. When I looked back in a lot of the breaches that I, basically, was investigating and looking at what you find in the desktop, clear text files, passwords sitting in them. One thing, even the browsers, everyone goes to their browser and the browser loves to ask you for your credentials to save them. But again, by default, that browser has security turned off, you go to the password tab in the browser and all the passwords are there, basically, for partaking.
Dustin Heywood:
Yeah, and there's maybe cache modules to go extract that straight out of your browser, that's one thing red teams love to do this on versus the password manager is far more secure. And I'm glad you brought up the whole companies should give password managers out to employees, Big Blue does that for every single employee. We have an option for our entire family to be covered by an enterprise password manager that is separate from our work password manager so we can wipe the work one and they transfer the personal one over even if they separate from the company so it's a huge win.
Joseph Carson:
Yup, I think making it portable, making it something because you want security to continue with that person's journey for the rest of their life. You don't want to, all of a sudden, take security away from the person, they become vulnerable so it should be something we should embrace. And for me, that's your organization's giving back to society as well, is bringing security up. What's some of your favorite tools? There's lots of password cracking tools out there, there's lots of tools. Some of my favorites are Cool or Hashcat, John the Ripper. What's some of your favorite tools that you enjoy?
Dustin Heywood:
Wordsmith 2 by Sanjiv Kawa, who's now actually an X-Force Red employee, is one of my favorite tools. What it does is you go through and it says, "Pull me down the word list from all of this particular region or here's the geographic specific ones for, say, Estonia, Calgary, here's the one for Chicago," and then to use that with, say, targeted rules. Oh, it is so deadly.
Joseph Carson:
That's actually... that's cool. Because one of the things I used to see in Estonia was a lot of the typical brute force or cracking would not work because of the character set that Estonia only uses. And if you're able to take that and customize it down and use that as your base word list for commonly used credentials from that region, that definitely significant reduces your potential word list down and, also, your success possibility. That's pretty cool. I'll have to check it out. I've still been using Pipal and Cool and others that create the word list but I haven't checked it out yet, so it's definitely one of the things I'll be looking in my free time next week.
Dustin Heywood:
Yeah, and it's an older tool. Last release was about 2019 or so but it's still been highly effective. Other than that, there's a couple of standard word list rules I'm still ... I've got a couple of rules and hashcats default like generated two dot rule that was all done by machine, it's still effective. Biggest thing though is word list and attack management. The rest of the tools like MDX Find, John the Ripper, Hashcat, et cetera, they're all fairly effective. My favorite tool is one I wrote myself because, obviously, I'm biased. There's a tool-
Joseph Carson:
And you're allowed to be.
Dustin Heywood:
I'm allowed to be, yeah. There's a tool called the NTLMv1 multi tool and it's based on research from Moxie Marlinspike back in the day whereby NTLM version one effectively takes an NTLM hash and then uses it as the key for generating your NTLMv1. So, what we can do is you can reverse that using a known ciphertext or a known plaintext attack using DES mode 14,000, it'll output here's the hashcat specific format for reversing that to an NTLM or it'll also put the crack.sh capabilities so you use their FPGAs. If NTLM version one is turned on or MS-CHAPv2, those are basically instantly reversible now these days with some magical hashcat shenanigans. So, those protocols should probably be deprecated sometime soon. Even if the password is 128 characters, it'll reverse it to an NTLM that can just be passed.
Joseph Carson:
Okay. So, it all likely makes it weaker in the result.
Dustin Heywood:
Oh, yeah. The nice part is the attack path looks something like this. Say you run PetitPotam or print spooler against the domain controller, you get an NTLM version one response back from the computers machine hash, you then turn that into an NTLM hash, sign yourself a Kerberos Silver Ticket, DCSync, you pop the DC. Total runtime, 15 minutes.
Joseph Carson:
So, for a warning, for anyone who's actually using that technique today, it's probably a warning that you probably should revisit it and consider changing it. Question, for different types of password cracking, from doing the traditional type of NTLM or network NTLM to SSH keys and also to, for example, different hashes, whether it being documents or browser database, key databases, what different techniques do you use? Do you use the same technique for all of them or do you try to augment your technique based on the type of hash that you're trying to crack?
Dustin Heywood:
Well, obviously, you're going to have to change your technique depending on the hash. I split my techniques into three major categories. So, there's really slow hashes such as your Bcrypt, those that are using basic word lists, very few modification rules. If you're only doing a thousand hashes per second on a GTX 1080, yeah, you're not really getting much out of those, we do a lot of CPU type stuff on those in order to get things to go faster. The IBM Power Systems are great for those, yeah, not to advertise or anything like that. But yeah, if you got that much horsepower on, say, a Power10, oh, it's gorgeous. But for a BDM hash, a Kerberos MOD 13 100 or the TGS type 18s, those I'm using Best 64 with smaller word lists or smaller word lists, or with larger word lists rather and then smaller word lists with a bunch of advanced rules targeting specific information such as Wordsmith 2, Cool, et cetera.
Dustin Heywood:
With faster hashes, I call it the excrement against the vertical air displacement device method which is basically just throw whatever you can at it randomly and it's all about diffusion effectively. I'm trying to spray as much of that password space as possible. So, I'll do things like use Attack MOD A1 with the left side and the right side, using expander and cut B, I will use Prints Processor and a technique known as purple rain where, basically, you take a word list like rock you, you pipe it through shuffle it, you pipe it through Prints Processor 16 or Prints Processor and you pipe that into hashcat and then you give it 100,000 generated rules and just sit there and see what happens. You turn on debug mode, capture what works and then you use those and get another attack if you start this grind. That'll get you at 50, 60% faster than using most generated word lists. Also pipe in long collections of previously cracked passwords, CrackStation, human only, Hashmob, hashes.org, those kinds of word lists or I'll just go randomly generate things.
Dustin Heywood:
I've got a set of rules that are designed to crush a GPU, it's about a gig of rules. You would never want to use this normally unless you're running out of stuff or you're trying to break into a new cluster but it's really good at finding faults in your build especially when you're using, say, times one PC Express extenders as opposed to the full bus, et cetera. So, there's techniques like that, they get really aggressive. I'll brute force update characters now on NTLM because I can. I never thought I'd be saying this, 10 years ago, that was unheard of.
Joseph Carson:
So, also, what techniques are gone? Old techniques such as rainbow tables used to be used.
Dustin Heywood:
Rainbow tables are still used but they're not used in everything. Rainbow tables are very algorithm specific. If I see LAN MAN, I'm still pulling out rainbow tables, although hashcat is now faster for those kind of attacks. There's a rainbow table for desk key cracking that's what you use in the NTLM version one or version two NTLM. So, we still use those on large banks of FPGAs but, otherwise, rainbow tables have gone the way of the dinosaur. Word lists are king especially with proper intelligent password cracking rules. Straight up brute force isn't used as much as most people would think. So, those tables that people say, "Here, it's time to brute force an eight character password," take those with a grain of salt, people love to put them up and rant about them every year. But a large number of those are based off of here's 440 v100s in a cluster, they'll cost you $10,000 an hour on cloud. Yeah, people are spending that kind of money if they've stolen a bunch of money but you're more concerned with your average hobbyist with 16 GTX 3080s sitting in their basement that used to be a Bitcoin mining rig.
Joseph Carson:
Now they've been repurposed for password cracking.
Dustin Heywood:
Exactly. Speaking of, your Bitcoin mining rigs or your cryptocurrency rigs do not make decent hash crackers but hash crackers do make decent mining rigs. And the reason for that is mining rigs, they take certain shortcuts that were great for blockchain but the way hashcat stresses the GPU, you will set your house on fire if you're using some of those techniques. We've seen fires caused because of shoddy components, the little ribbered extension cables, stuff like that tend to cause fires. And then also, hashcat, when you put four rigs on a house on, let's say, 110 volt, you're going to exceed your amperage rating in the house.
Joseph Carson:
So, for anyone looking to specializing in passwords and cracking, becoming an expert into pen testing and helping organizations find the best way to protect, what type of hardware do they need to really get themselves either get started or at least look to get into it professionally? What's the hardware baseline somebody would want?
Dustin Heywood:
The baseline I would consider as normal is any of your modern gaming GPUs. I run an old GTX 1080 still at the house because I haven't upgraded my gear but mostly because I use cloud. A lot of people using the GTX 3060s, 3070s, 3080s, you can do a lot of damage with just one of those cards. The key is make sure you understand the power. One of those systems, fully specced out is a good draw of about 1,200 watts. And then, by time you count in the conversion losses, et cetera, if you're using, say, a bronze base supply, you're looking at about, let's call it nine, 10 amps from the wall. And your average house, let's say it's a 15 amp circuit but there's also efficiency losses going out to the panel, so you can easily hit 15 amps and start breaking breakers if you go much beyond that range if you have other stuff on that circuit. So, make sure you're running from your panel to your password cracking rig is appropriate for your job and then make sure you got appropriate cooling, but beyond that, you should be pretty much good to go. The nice part about hashcat is you can detune it a little bit to use less power. Just don't say, "Hey, look, I" ... If your cooling is not sufficient and your power is not sufficient, don't complain about your hash rates. Split them up or start using some clouds-
Joseph Carson:
Optimize it.
Dustin Heywood:
I know the cloud is expensive ... Yeah.
Joseph Carson:
Yeah, do some optimization and find out what your rig is capable of and make sure you don't exceed the limits. Set some limits. Find out what your capabilities are from the hardware, understand them and make sure you don't exceed them because-
Dustin Heywood:
Yeah. And airflow is life is the other thing. You need adequate airflow. A lot of people do cracking on these little Intel NUCs or these little small board computers, mini ITXs, those work absolutely fine, just don't stick it in your home theater cabinet with no airflow. A hash cracking job at full load will probably push you about 82 Celsius at the card and internal case temps can get really, really high. You don't really want it to exceed much beyond 85, 90-ish. I've seen some cards hit 110, like the older AMDs, and that's just a recipe for, oh, my god.
Joseph Carson:
So, is liquid cooling any effective better than airflow? What's the-
Dustin Heywood:
It is. Liquid cooling, if you have the case room, is more effective. That being said, in a corporate world, liquid cooling adds points of failure and when you take one card, it's okay. But when you times that by 32 cards, you need to move to enterprise liquid cooling solutions such as engineered fluids, immersion cooling, those techniques unless you have a really good hands and feet arrangement and you can get people to come in to swap out video cards, it's not going to be a nice solution so that's why a lot of enterprises go air cool. IBM uses liquid cooling on our mainframes and it's just gorgeous but it's an engineered solution.
Joseph Carson:
Okay, it's very, very specific for that system itself. So, a quick question, when we met the last time and we had a lot of discussions around the future side of things, where things are going, I have to put my hands up to one thing. We've had a lot of discussions here, we talked about biometrics getting effective at, I will say, biometrics replace usernames, not passwords, but they have better secure attributes. There's one thing I remember and we talked a lot about password lists. And I think I was always referring to is that I don't see passwords going away and password lists doesn't really change the password. I do have to correct myself in that area because I did have a very ... It was a really interesting discussion with another counterpart who has been working and moving more to password lists. He mentioned that it's not that passwords are not going away, it's that secrets are not going away, there's going to be always a secret and if we could just come into the definition or what we are referring to, what is the password because I always referred to a password as a deviation of a secret itself.
Joseph Carson:
But yes, passwords are changing in how we interact with them but, at the end, there's some type of secret exchange that's happening in the background. So, I just have to clarify, I was always saying that was a password or a key, but just to get into the right terminology, there is some type of secret and that secret is either dynamic or static, depending on the type of algorithm or system that's been used. So, that person did correct me and it was actually really great because it was a very in depth conversation. So, what's your views on where's the future going with password use?
Dustin Heywood:
I strongly agree with this, to be honest. Let's look at SSH, for example, we evolved from the password to the SSH public key and now there's new techniques such as SSH certificates which are completely different from X. 509. So, what we're seeing now is companies are tying in a short lived certificate generator that's tied into, say, OpenID connected SAML based authentication, multi factor authentication on the back end based on everything from your IP, your telemetry information, all the way to your MFA answers, to your geolocation, all these things get combined into a risk and then make real time risk decisions. If you pass all the risk decisions, so it's a risk based authentication, they issue a short lived certificate that's valid from logging on to SSH. Now there's things with Windows where they're tying in to Windows Hello where they can issue you a short lived certificate to login to Windows and then the machine certificate is randomly generated in the back end.
Dustin Heywood:
So, this is the essence of when people talk about password lists, what they're really talking about is shifting to more machine managed secrets with risk based authentication of the end user and I'm all for it. This is turning into beautiful things, things like password managers, short lived certificates, short lived authentication, QR code based authentication by taking your phone and scanning a QR code, signing in out of band. These are all the things that were actually forecast back in the '70s with things like Trusted Computing and hardware devices and trusted path, it's just we've evolved the technology to be actually end user friendly now.
Joseph Carson:
And usable. That's the great thing. Because I will say that it's moving ... I always say it was moving passwords into the background, it's really moving secrets into the background. And as you're pointing out, is that not that that becomes much more possible to make context based security decisions based on many different factors that actually removes that decision making as solely on the user, solely on the person that's interacting. So, we're making it less interactive, the experience is much improved but there's a lot more sophistication and complexity in the background that has to be also done correctly. I've seen bad implementations of it where it still becomes a static password that never gets changed and, all of a sudden, that's what's been unlocked. And if you capture that in the network, you can basically expose that and abuse it.
Joseph Carson:
So, to your point, what you described, for me, is one of the ideal scenarios if it's basically short lived certificates, it's dynamically created based on the security context at that time with other additional security factors, multi factor, how long was the last time you signed into? What location you're signing in from? Is this machine known? Does it meet the security requirements in order to access this system? Is the system you're accessing highly sensitive and, therefore, maybe require different types of additional access workflows like I need my colleague to prove this for me to access this system.
Dustin Heywood:
Yeah, the whole four eyes process as opposed to two. And I'm glad you brought up the whole static secrets piece as well. I hate to bag on NIST again, let me get back into this. Is I got into a debate on Twitter the other day about how I greatly dislike this special instruction and the reason for that isn't that I don't believe ... End users shouldn't have to rotate their certificates to their password manager. However, secrets in general do need to rotate. Here's the thing, breaches happen, datas get out there, secrets are only secret as long as nobody else knows them.
Joseph Carson:
Exactly.
Dustin Heywood:
And, in order to keep that, they have to be frequently rotated but that process shouldn't be user driven, it should be fully automatic. The user doesn't need to know, "Hey, I've got 400 passwords and they're all rotating, here's my password list," they should be notified when the change fails.
Joseph Carson:
Yes.
Dustin Heywood:
Same deal in the corporate world. A service account password should not live for more than, I want to say, a month. Right now the current standard is a year for most places or never, they never change. How many companies changed their KRBTGT hash in the last 10 years? Right?
Joseph Carson:
Unless they have to rebuild their active directory.
Dustin Heywood:
Or unless they had a pen test that it showed up in the pen test format. And, even then, half of them ignore changing it. So, like this.
Joseph Carson:
Or they only do it once and forget that they need to do it twice, actually, to properly rotate it.
Dustin Heywood:
Yeah. So, public service announcement, if you have an active directory domain, which is most of you, change your KRBTGT hash once every six months. This way, you're guaranteed to have it rotated twice, at least, within a year and you won't break out your Kerberos tickets.
Joseph Carson:
Yup. And I think a few years ago when this product had recommendation to you don't have to rotate your passwords if you're using multi factor authentication. I've got into arguments online about that with others as well, we've had disagreements. I always say it's all about the risk that you're willing to accept. What's the risk you're willing to accept? If you have multi factor authentication, and we have to remember, not all multi factor authentication is the same. So-
Dustin Heywood:
And this bypasses to it, right?
Joseph Carson:
There's just lots of backdoors. And were people saving their backup keys or their MFA? As a photo on their phone.
Dustin Heywood:
So, let's just say a popular enterprise password manager, so I'm not advertising for other folks, happens to give you the option to store your authentication secrets right in the Password Manager. So, you go to your Password Manager, you click up and, look, here's my MFA ready to rock and it's like, okay. The whole point of the MFA is to have my password stored on my cell phone separate from my password manager or wherever else they are. So, I get the convenience factor and I'm all here for it but, at the same token, I just cringe a little bit.
Joseph Carson:
Derisk is the most important thing. You don't want to keep all your eggs in one basket. Therefore, if whatever it is that's sensitive in your password manager and you're using multi factor authentication, sometimes use another method of storing that. For me, I use multiple password managers for that specific purpose. I've got online and offline, I've got, basically, isolated and segregated systems. So, to make sure, at least you know that, if I ever need to, I can go to it but I know that if one is compromised, it doesn't impact the other. And that's what...
Dustin Heywood:
Well, and here's the other thing, how many people also backup their authentication secrets via printing out a rescue kit? What happens, say, I have a house fire, as an example, and I lose my phone, I lose everything else and I can't get into any of my systems. Maybe I managed to get out with my wallet tops. Trying to get those reset, next to impossible, especially in this day and age. So, make sure you go and you print off your backup options, store it in a bank that's double guarded by guys with guns so people can't get in and steal it or find something that works for you.
Joseph Carson:
That's your digital life, if you lose access to it ... I've had people come to me and say they've had account takeovers or their social media account, which is critical to the business, is now, all of a sudden, stolen and they no longer have access to it. Or the website domain that they were managing, basically, the people basically took it over and they have no longer access to their entire website and the user database is all in there and even the access to the banks and so forth. So, you end up in a situation where you get these stolen accounts that you need to have ways to regain access to it, you need to have ways to be able to get back into those accounts. And basically, if you don't have those systems or you haven't implemented the restoration, that becomes really devastating for all of you. I've seen people lose their business as a result of not having that backup, those keys.
Dustin Heywood:
Well, here's the other thing. Say, in the untimely event, I take part in lots of extreme sports and, say, I die, how are people going to get into things like my financials, stocks, et cetera. I keep an emergency rescue kit with my will and it gets updated routinely. Why? Because I want to go bequeath my massive iTunes collection out to my family, as an example, I spent a lot of money on that.
Joseph Carson:
Yup, absolutely. It's the same when you get into, not only your music store, but also your movies, you could get into, as you mentioned, your stock options, other types of things you maintain. I think one thing that I had a good experience recently where I was actually a bit worried about at the beginning was actually migrating to a new mobile phone. I remember in the past, that was always, especially for someone like myself. Like you, when we talk about the number of accounts that we have to manage because we have different personas and different accounts and systems and we sometimes go a little bit crazy. I think I've got well over 500 different accounts that I have to manage with different passwords and credentials and different multi factors for different things. And it came to getting a new phone and I cringed at the idea of having to reprovision and move things across but I actually found that, today, a lot of the systems have actually made it much easier. There's a few things that I did have to get reset to move across but the experience was a lot much easier than what it was maybe three, four years ago.
Dustin Heywood:
Yeah. My authenticator app got transferred over automatically. All the secrets and everything just came on over and I was shocked.
Joseph Carson:
Yeah, even that itself. I think one of the things in the past that I was ... I think for many organizations, especially when you're talking about security and that they're moving between security solutions. I remember one organization saying that it wasn't the security solution that they're worried about, it was the time, basically, when they had to uninstall it and the systems were not being protected. That was the fear. So, some years ago in the past, when you wanted to move MFA across to another phone, you sometimes had to disable it, basically, and then re-enable it on the new phone. And in that timeframe, that's when basically you're most exposed, is when you have to go through those processes with disabling security in order to migrate it or transfer it or reprovision the new system.
Dustin Heywood:
Oh, yeah. Migrating security solutions is the worst, right? Remember when RSA got breached back in 2012 or so? We all had to go swap out of SecurID fobs on to new ones, we did improved since then but it was just a nightmare. People had to go changed into new hardware tokens and imagine issuing notes, say, 1,400 tokens across your workspace.
Joseph Carson:
It's the same. In Estonia, we've had the same experience. Fortunately, through the years, with somebody who has been quite ... The first time we had the issue was back in 2008 was we were heavily dependent on time stamping protocol within the browser. And in 2008, Google decided to end of life Chrome support for time stamping and that meant that our ID cards that we use for literally everything in Estonia, from voting to banking to prescriptions to loyalty cards to even getting on the tram, meant that, in 2008, there was something like, I think, 50,000 cards that no longer worked with a Chrome browser and that actually created a big issue.
Joseph Carson:
And it ended up meaning that, in order to find ways in order to resolve that, it introduced this piece of software that you had installed on your computer that will actually regenerate the keys and apply new protocol to the card. So, it was great, that's all that small issue for those 50,000 cards because, of course, we've got 1.3 million citizens who is using the system in the background. Fortunately enough, it was limited to only a set amount of people. And then fast forward to, I think, it was 2017, 2018, when Estonia then decided that they were moving from their old legacy smartcard systems to, I think, it was a jimalta based one.
Joseph Carson:
And about a year after those new cards were issued, what happened was they find that there was actually a flaw in the key module that we're using to generate the keys in those cards. And it ended up meaning, I think, it was 800,000 at that time that had been since issued, that meant that they were now vulnerable. And, if they didn't have that system that they sold back in 2008 to fix the Chrome browser issue, that was the same system that, fortunately enough, allowed them to regenerate the keys on to people's computers. That actually saved them from having to go and re-issue all of those cards. So, sometimes, these systems are not always perfect, you do have bumps along the road but it's how you're able to resolve the issue and how you make it usable for everybody. And how that experience is is what, basically, is the decision between people continuing to adopt it and embrace it versus those who decide to change path and look for something else. So, I think it's really important. We will always have these bumps in the road, we'll always have vulnerabilities in encryption, we'll always have challenges in migrating and we'll always have systems and applications end of lifing some type of protocol and we just have to find ways on how to make sure that we continue maintaining that standard of security as we move forward.
Dustin Heywood:
Well, I'm glad you bring that up. Here's the thing. We focus a lot on how do we prevent the breach or what do we do, yeah, let's call it left of boom? But what we do right of boom is just as important. How you respond is absolutely critical. We talked about some places, when people get breached, hey, it's a resume generating event. But yeah, here's the thing, if I was a CISO, I'd be talking about, "Hey, these people just went through the most expensive training money can buy, why am I going to fire them when I can use this knowledge to make this place more secure?" Same deal with how do you recover from a password breach. If you just rotate your passwords once and call it a day, have you really fully recovered? Unless you get into, hey, we've changed our processes and the way we operate to make sure these would get rotated again routinely. There's no such thing as a failed pen test, for example. The only failed pen test is one where you don't learn.
Joseph Carson:
Yeah, you don't change from the lessons of experiences that you find and discover and that's really-
Dustin Heywood:
Exactly.
Joseph Carson:
I think cyber insurance is important in this regards, I think cyber insurance does help organizations, at least, provide that financial support when they do become a victim or have an incident. But that should not be the dependent, only thing that you do part of a breach. That helps you support it financially but you have to go and you have to find and discover how that initial foothold happened, how they got in, how they elevated credentials and you have to go...
Dustin Heywood:
And that's the thing. And it shouldn't be your company doing it, right?
Joseph Carson:
Right.
Dustin Heywood:
You should typically have an IRA company on retainer and the reason for that is lawyers and insurance companies love to see an independent third party do some of this. There's evidence preservation, there's a number of things that you can spoil real quick if you're doing this yourself.
Joseph Carson:
In many organizations, one of my things that I've been ... My motto was, I've been doing it the last year and a half, is that there's the difference between having an incident response plan, which is one thing. That means, okay, it went through and you got your checklist and it's sitting on the SharePoint but there's a big difference between being incident response ready and that's completely different. The last thing I want to be sitting in an incident response meeting is trying to agree on what time format we're going to be using for gathering images. What waste of time is that? It should have already been planned beforehand. Next thing, okay, where are we going to store these images?
Joseph Carson:
Well, we don't have enough space because, all of a sudden, you need terabytes of data to store all of these images of victims machines and the last thing you want to be doing is now doing a same day delivery at Amazon and hoping they get there in time. That's the difference between having, one of them is having a plan, which sometimes gets encrypted with the ransomware on the SharePoint and if you don't have an offline copy, but the next thing is having readiness. And that's why I always say it's important to have retainers with incident response experts who do this day in day out and who have the knowledge about how to make quick decisions and actually can quickly move along to where you guys should get proper containment very quickly. Because-
Dustin Heywood:
And there's also things like out of band communication, right? What happens if your VoIP system has been encrypted by the ransomware? How are you contacting people over compromised systems? Do you have backup solutions in place? Have you practiced these processes? Are they reasonably fresh or are you reaching through some locked file cabinet in Zurich somewhere trying to go find it and some people are on the international flight to go pick the thing up?
Joseph Carson:
Absolutely. I was even saying, even the accounts is what gets into the access that incident responders have on the systems as well as the forensics evidence gathering to look at the audit logs. The last thing you want to have is them using the same accounts which are compromised by the attackers and, all of a sudden, now you're contaminating evidence. You're contaminating the evidence process as well which potentially puts you into a situation later where you cannot use it in a legal frame as well. So, it's really important make sure that, with the responder, you've got accounts set up specifically for that purpose that have no previously contamination.
Dustin Heywood:
Or a process to create them rapidly because who knows those accounts could already been compromised.
Joseph Carson:
Absolutely. To create them and deploy them and give them access to the system they need to, absolutely. So, I think it's always crucial in incident response, especially for organizations, especially when it comes to credential compromise, how quickly you can rotate those credentials, how you can make sure that you can actually determine where they actually laterally move to a new environment, what did they have access to? Did they expand to cloud environments as well? That's a question I've got for yourself as well is that now, a lot of times it's been basically isolated to a lot of on premise, what do you see in the cloud side of things?
Dustin Heywood:
I've seen cloud really-
Joseph Carson:
I was talking to Carlos Polop a few weeks ago and he was talking about his PurplePanda, which is now doing, basically, privilege elevation, escalation scripts for cloud environments. So, what's happening in the cloud side?
Dustin Heywood:
Cloud really is no different from on prem. I mean, it is different, I don't want to say it's not but a lot of the same processes still work. Create new accounts, isolate the systems, copy things over. The nice part about cloud is you can do cloud to cloud account transfers. So, what we're doing in IR, for example, is we have a dedicated cloud account just for capturing those IR images and we do an inside, let's use Amazon as an example, an inside Amazon to Amazon transfer so you're not paying massive data egress fees. Let's say you're transferring from Amazon to IBM Cloud, there we're paying data egress fees up the nose. Same deal with Azures or whatever cloud providers are all out there.
Dustin Heywood:
So, really, as long as your incident response provider has experience in cloud and a cloud plan, you're largely good to go. If you're large enough that you can do your own internal IR, then obviously, people are already building up contingencies, ready to rock. I've seen everything as weird as sending an Amazon snowball or whatever their big truck is out to the system, plugging it into the data center and driving it out to do the forensics or driving out massive piles of trucks and getting the data shipped to them. So, it just adds a little more logistics but moving data around is the global economy, so it's really no different even though it is different.
Joseph Carson:
Oh, absolutely agree. So, I want to get one final question in from you and this is going back, this is the big question. Is passwords going to die? Are they going away? Or we should refer to it, is secrets going away?
Dustin Heywood:
No, I will see the secret evolving, it will never truly go away. And the reason for this is not all systems can be connected to the internet, not all systems should. Let's look at SCADA and OT as an example, those systems I'm not going to connect to a giant enterprise password manager, we're going to go manually rotate those secrets. You need to authenticate those things while you're out at some power station or whatever else, secrets aren't going away. We still need some way of exchanging data. How we access them, how they get managed evolve but they were a effective tool for the last, let's call it 50, 60 years or even longer, however long computers have been running, they're going to continue to be there, it's just how we access them, how we manage them will evolve.
Joseph Carson:
Yeah, and the experience, I think. From the user experience, it's definitely evolving but I agree with you. I think a lot of those systems, SCADA controls their lifespan, seven to 20, 30 years. We're not going to be changing satellite stuff frequently so I don't see a lot of those critical systems changing very fast, but yes for the human experience.
Dustin Heywood:
I think I'll retire before they go away is what will happen.
Joseph Carson:
That's you and me both, so there. But Dustin, EvilMog, it's been awesome having you on the show and I really always enjoy talking about password cracking and techniques, where it's evolving, what's been changing. Any final thoughts or any words of wisdom for the audience?
Dustin Heywood:
If you don't have a password manager, please go out and get one, they've improved so much in usability over the last couple of years. I'm still trying to get people to go use them, that's going to be my mantra for the next five, six years. But get a password manager, get unique passwords between systems, backup your secrets somewhere safe in case there's an end to your digital life or some other major event, but just please get a password manager and make sure that your passwords are unique between systems.
Joseph Carson:
Absolutely. EvilMog, Dustin, it has been awesome having you on and it's been a pleasure. It's great to have you back on again and I'm pretty sure that this will not be, you'll hopefully be on again in the future. We'll definitely be talking.
Dustin Heywood:
Yeah, I don't think this will be the last.
Joseph Carson:
It will not be. So, for the audience, again, you've heard the latest updates and where passwords have been coming from and where they're going to and, hopefully, this has been educational, enjoyable for you. Stay safe out there. Again, tune in. This is the 401 Access Denied podcast every two weeks bringing you thought leadership, expertise, knowledge and information sharing on the trends in cybersecurity industry. So, stay safe, take care and thank you. Dustin, you've been awesome.
Dustin Heywood:
Thank you.
