EPISODE SUMMARY
Subscribe or listen now: 
Joseph Carson:
Hello everyone. Welcome back to another episode of 401 Access Denied. I'm your host for the episode today. I'm Joe Carson, chief security scientist and advisory CISO at Delinea. It's a pleasure to be here with you, and I've got an exciting episode for you. And it's one that we haven't really read into in previous episodes or actually to date. And I'm really excited about this topic, because for me, it's a passion of mine. And I've got two awesome guests on the show today, which we're going to go and introduce them. So first of all, Beau, I'll pass over to you. Do you want to give us a little about yourself, what you do and where you're from and your backgrounds and what passions do you have?
Beau Woods:
Thank you. I'm really excited to be here. I'm Beau Woods. I wear a variety of hats. Been in the InfoSec cyber security cyber policy world for 15 to 20 years now professionally, before that, I like to say I was a security amateur. I made a lot of mistakes on my own and other people's computers. Got into this line of work because I was just drawn into it. Like a lot of us, we were powered by curiosity to just get engaged, get involved, pull something apart, figure out how it works, put it back together, maybe you got some extra pieces lying around or whatever. I've done a variety of things over my career, including a lot of consulting work. I still run a small consulting company.
Beau Woods:
Done a lot with DEFCON. You can see the DEFCON flag in the background here. And a lot of the villages, Aerospace Village, Biohacking Village, Hack The Sea and ICS Village, as well as got involved a few years ago a lot more in public policy. And as we were talking about before we kicked it off, worked for the Atlantic Council, which is a global public policy think tank for a few years, then got pulled in by the FDA to help develop new pathways to market for medical devices, which is some really, really interesting work. And now in addition to many of the other hats that I'm wearing, I'm advising CISA with some of the security of critical infrastructure in the US and internationally.
Joseph Carson:
Awesome. And we probably crossed paths at some point in the past, because when you mentioned a lot of those things, we talked about the Atlantic Council and also in the maritime side, I spent a lot of time doing the autonomous shipping side and hacking in the ships. So absolutely, so we probably at some point cross paths. And also another guest on the show, which is fantastic to have as well is Paulino. Paulino, welcome to the show. Give us a little bit about yourself and your passions and what you do.
Paulino Calderon:
Thanks Joseph for the invitation. Well, I'm Paulino Calderon. I'm a Mexican, I've been in the InfoSec industry for about 15 years, I think. Right now I'm working in WebSec which is a consulting company. We do a bunch of AppSec and network security testing. And I guess the same as Beau, I started playing with routers back a long time ago. One of my business partners was really into router hacking. He started router... dotcom back in the day. So I guess that was when I started getting into IoT. At some point I crossed paths with Beau at some DEFCON I think after-party. And we ended up co-authoring this book, which led me to work with people.
Paulino Calderon:
It's a small world, as you were mentioning. I ended up working with people from a different state in Mexico. They're called Electronic Cats. Since we were developing new tools for some radio IoT protocols, they had some, I guess, USB dongles that were brand new and we were testing them and using them in our book. And we started communicating with them to debug certain things. And that ended up becoming another project that I started with them called the CatSniffer, which is a sniffer for radio IoT protocols. So I guess I'm in between also between AppSec, network security. I'm also a big fan of open source software. So I've been contributing to Nmap since 2011. So I guess some of you might have heard my name because of Nmap, a lot of my contributions are there. I've been developing Nmap script in engine scripts for a while now and a bunch of different things. I guess I run a local OWASP chapter, an official OWASP project as well. And between work, hobbies, everything, I also own a restaurant. So I made the jump actually owning a cafeteria, which I'm still deciding if it was a good idea or not, but I love it.
Joseph Carson:
Absolutely. That's one thing is curiosity is what drives all of us. That's for sure. And we really do... I agree. One of the things I find difficult sometimes is separating work from my hobby. And ultimately what I find is that actually my hobby turned into my job, which is always a fun thing to do for sure. And one of the things that brought us together absolutely is the fantastic book that you put together, that you coauthored, which is fantastic. And it's one of the really the... It's something that's been missing in the industry for a long time is really getting into the IoT risks and IoT hacking and hardware hacking. And the threats is out there. I've been following the likes of Joe Grand for many years, and I think we all have, into a lot of his early work in the early 2000s.
Joseph Carson:
And it's an area in the industry, which is really probably lacking a lot of the... let's say end-to-end understanding about how everything works because it is very diverse. There are so many components to it. Just give me a little bit about it, where did the idea come from the book, who came up with the idea and I load it up, how much effort was put into putting it together because I've done a few books. Mine have been very thin as you can see. So it takes months to do those, but putting a book together that you did, I think it's amazing. And it's definitely needed in the industry. Who wants to start on that? Beau?
Beau Woods:
I think it was Fotios Chantzis who brought the idea to us. He was like, "Hey, we're doing a lot more in IoT security." And at the time he was working at the Mayo Clinic. So initially we were going to focus on medical devices. But I think very, very smartly No Starch Press, who's a great publisher if you ever get a chance to write with them, they're really good, they were like, "That's a little bit too niche. Why don't you broaden it out to more IoT?" We were like, "Oh yeah, that makes a lot of sense." So it became an IoT book. But over the course of doing it, it took us probably two years to get it all written and published, many different iterations, many versions. We had some ideas that we jettisoned. We had some things that... Since we sat with the material for so long working on it, we had new things that we were able to pull in.
Beau Woods:
And we have a whole chapter on methodologies. So it's not just, here's a very narrow thing, go hack on this stuff. It's like, what's the philosophy behind this? We've also got a lot of words dedicated to thinking through some of the consequences of this. We had guest writers come in like Dr. Marie Moe who's very well known. She is both a security researcher and worked with Norwegian CERT and she's also a pacemaker patient. And she talked about some of the consequences of hacking IoT devices because ultimately at the end of the day, we don't want just to empower people and equip people to be able to do the technical bits and bites, with the internet of things broadly, including medical devices, power plants, maritime equipment, aerospace, everything is very, very different. And if you don't have a great understanding of what happens after the hack, it could set up some really dangerous situations. We wanted to avoid that. We wanted to make sure that we're empowering people to understand all of the context and the consequences of what they're going to do so that they can be way better at it and do things that are more meaningful, more important rather than just finding the next little bug. Find a bug in something that's really critical and then you can change the world with that.
Joseph Carson:
Absolutely.
Paulino Calderon:
And especially with technology that dies really fast. We wanted to put the holistic approach, teaching people how will you approach this thing? How could you create new tools that are not out there yet? Because that's what the majority of the time you will be doing. You'll be breaching the gap and coming up with new things and new techniques. And also at the beginning of the book, we had very good content that couldn't get published because of the nature of it. So we ended up having to look for more generic and I guess, applicable examples, which ended up becoming a good thing because we focus on getting things that are easy to replicate and that are cheap and accessible for everyone.
Joseph Carson:
Absolutely.
Beau Woods:
In fact, because we recognize that devices change so quickly, Paulino do you want to talk a little bit about the software stack that you guys put together?
Paulino Calderon:
Which one?
Beau Woods:
The IoTGoat and some of the other things that we did in order to have evergreen examples that people could work on?
Paulino Calderon:
Exactly. One of the projects that came out of the book while we were writing, it was OWASP IoTGoat, which I'm sure most of you are familiar with the older version or the more, I guess, known version web code. So it was the same idea. It was vulnerable in purpose IoT PAM. So it was based on OpenWRT, it became an official OWASP project. We got funding for a student to work on it full-time during Google Summer of Code, I think last year, right before the pandemic started. And it became an OWASP official project since then.
Joseph Carson:
Absolutely. And actually, hopefully that some point in time, because I know that OWASP does has their IoT top 10, I think it goes back to 2018 or so, hopefully it'll be updated soon. So definitely it's an area that's... I know they updated the web application one in the last couple of months. So hopefully the IoT one will also get that same attention. I'd like to also get into... Because for me, I think over the years I've been dabbling into the hardware hacking and IoT side of things. I've been involved in projects, I've been involved in things like where we talked about autonomous shipping. I remember in the early days of the issues was not technical issues, it was legal issues in autonomous shipping. One of the challenges was in international waters, they couldn't actually have vessels that didn't have anybody on the vessel.
Joseph Carson:
It was actually international maritime law meant that you actually had to have a person on the boat. So what they ended up doing was having those types of projects in local waters, the likes of Finland in let's say barges and so forth and ice breakers and other things. So they actually focusing on the projects. So one thing I like to... It's also a very focused area where it can be expensive to get into because when you get into hardware hacking, the equipment, if you think about some of the things, even digital multimeters and the telescopes, and I think of the array of things like Bus Pirates and blasters and great Flight One and stuff. What would you recommend people start to get into it? Even you can get into not just about the hardware debugging side of things and extracting firmware, but you also get into there's the software side. There's also the wireless side of things, the radio frequencies, and RFIDs, it gets into whole extensive different things. Where we do recommend people get started? What's the essential issue to have in their labs?
Beau Woods:
I think one of the things that is cool, but also daunting about IoT is there's so much stuff that can be IoT. On a lot of these devices, you've got a web stack. So if you're familiar with web application security or mobile application security a lot of them have apps. So that could be the gateway that gets you into this. And in fact, when we were doing for the Biohacking Village, Fotios and I... I met Fotios through the Biohacking Village. I started the device lab there and we started up a capture the flag. And I think it might be the first, but now it's not the only, but the first capture the flag for medical devices. And we wanted to have a steady ramp because IoT is intimidating for a lot of people.
Beau Woods:
But what we wanted to show them is that if you have skills here, you can translate them here. If you don't have skills over here, then maybe you team up with somebody in the capture the flag, and you learn those skills and you teach them the web skills and you learn the RFID skills, or what have you. And so I think for a lot of people, IoT can be really accessible, some corner of IoT that you can start working on and learn some of the other things. But I know Paulino's got a really extensive lab and does a ton of stuff, he's way smarter on some of the pieces that I'm not, which is great. Paulino, why don't you talk about your lab a little bit? I've seen some pictures of it but...
Paulino Calderon:
I think I was going to say the book got brought together like that. Everyone brought their own expertise in their own area and that's how it came one piece. But same as me, I suppose that I started with the basic multimeter, oscilloscope, some board to communicate with the traditional protocols. And then as I mentioned before, recently I started going more into radio IoT hacking. So we had this new board and we've been working on this board for a while. I like this project a lot because it's an open source hardware. So all these schematics are posted and the software as well, but I don't know. I would always recommend to get one of those. It's a CatSniffer. I guess the most common IoT protocols are supported there now, if you're planning to play with radio IoT hacking, you should definitely get one. And we're planning to do something as the Bus Pirate does. How the Bus Pirate, you have macros pre-program to attack certain protocols or known attacks to replicate? We're trying to do the same on the CatSniffer for radio IoT.
Joseph Carson:
Oh, interesting.
Paulino Calderon:
My answer will be just depending on what area you're planning to get to, just familiarize yourself with what technologies are there and take it from there.
Joseph Carson:
Because that's actually, I think where I started a lot is in the radio side. I get into quite a few years ago into SDR listening to radio signals, amateur broad, was it radio bands, get into the things of listening to the ISS space station and listening to local pilots and then getting the ACARS and ADIS and stuff coming back in. So that's really where I started and moved from there into other areas. So for me, it's absolutely when you're talking about being able to with the radio side of things it's always interesting. Because for me, it was in maritime signals, it was all about the navigation ships and vessels and communications and so forth. So that was always interesting and also a very cheap way to get in as well, because a lot of those SDRs now were very affordable were previously could be very expensive.
Joseph Carson:
So definitely an area... For people, when you're going through the book and there are a lot of devices that you recommend in the book, what would be the first area in the book that you recommend people to go through? Let's say is one of the test scenarios. I know you mentioned a lot about the... there's a couple of scenarios with the Blue Pill or the Black Pill that's been used into being able to read and write firmware. What areas are you recommend people getting started, or just getting some older vulnerable hardware and peeling it apart since some older webcams or readers, or we do recommend people getting started when they're going through the book in regards to doing some of the tests?
Paulino Calderon:
I would recommend that they try to get the hardware that we put there because most of it is open source and they can get it for a fairly low price that is accessible for anywhere in the world. But what I would recommend is start playing around with what you have around, the technologies or the things or appliances that you have at work and at home. And that way you don't need to end up like me with boxes full of devices that you never use and you don't have space anywhere to put them.
Joseph Carson:
Absolutely. I think we all have a lot of old mobile phones around our homes that definitely are just sitting gathering dust. So you can definitely pull those apart or start checking and reading the firmware from them as well. Absolutely. That's a good starting point, is use the things that surround you that might be broken, be curious, pull it apart, open it up. It's definitely one of the things I went through is the FCC website where you can actually go and start looking at a lot of the schematics and looking at what things, the radio frequencies that are published there because they do have the post the radio frequencies, whether being things like doorbells or whether it being remote controls or wifi signals and so forth. So definitely looking at a lot of those areas. So definitely, Beau, what are you thinking around that side of things? Where would you recommend people get started as well?
Beau Woods:
Start with whatever you're excited about. For me anyway, if I'm excited about something, I'm naturally going to put more time into it, I'm going to learn it better. And then like you said, anything you've got just laying around can also be a good starting point. But also I want to point out that different people might have different starting points and it can be okay. I've had a number of people who they're not technical at all. And they're like, "Oh, I bought the book and it was great." I'm like no disrespect, but you don't have a technical background, technical knowledge. Like, "But some of the examples upfront and some of the other things in there, I just perused through it. And some of it was really accessible to me and some of it was really super dense, but I enjoyed reading it anyway." And we did try to create pathways to entry for different types of people. So the first couple of chapters are really non-technical.
Joseph Carson:
Very strategy-focused and risk-focused.
Beau Woods:
That's right. And the last chapter too is, okay, so you've got these things in your enterprise, how do you defend against potential attacks? And so we made it that way intentionally so that it would have a broader accessibility so that people could get it, be less intimidated. And again, have that entry point where they could start learning like, "Oh, I really want to know how you do, pivoting from one device you get access to into the rest of the network." And so like, there's some bread crumbs there that can lead people to jump into whatever section they're most curious about and then keep the rest of the book on the shelf as just a reference.
Joseph Carson:
Absolutely. I showed you my version, which has got full of yellow Post-it Notes, which is something that I've calmly done. I think one of the first things I end up getting more into the hardware diagnosis was one, as I mentioned, I got into the radio side of things, I got into RFIDs and I had a couple of Proxmarks and communication support. And one of the things I end up doing was years ago when I did accidentally, I plugged it into the wrong version machine. And I did a flash of my Proxmark and like many people out there, we bricked it and I'm just like, what do I do? And then you go and you get yourself a Bus Pirate or something similar, and you start trying to flash the firmware through JTAG.
Joseph Carson:
And that was probably one of the first ones that really started getting into was actually by myself, breaking something and then going and trying to find as much information as possible. And that was for... For me at the time, there was no one guide. There was no somebody had something about here's how you flash the firmware on it. Somebody had something about, well, if you run into this or when you're flushing it, this is the chip of the pin that you need to short circuit in order to actually get it to reset. It was things all over the place in order to try and go through it. And that was for me as breaking my own things was one of the reasons why... Of course, you don't want to go out and buy new ones or replacements, so the best thing is to repair it.
Joseph Carson:
And I think that's really a lot of things is that curiosity about things in repairing them. So for me, that was one of the areas I started was breaking a Proxmark and trying to get it to work with the low cost as possible. I was just curious into, is that... One of the things that at the time when I was going through that, I realized what I wasn't really good at was soldiering skills. I've set off too many fire alarms and smoke alarms over the past couple of years, which probably more than what I'd like to mention, but is that something... You mentioned about going to other people who have knowledge in those areas. Do you recommend people trying to explore skills and soldering and power? I don't know how many times you've short-circuited and got electric shocks over times as well. So safety is always first. So any thoughts around people who's looking to enhance those skills? Because it is very physically focused. It's very hands-on rather than what people have done, more at the soft side of things, any thoughts around people who are looking to get into the soldiering side of things and repairing and electricity?
Beau Woods:
I'd say it's a skill that I encourage people to get familiar with. Even if you're not going to use it a ton because when you really need to use it and you don't have it, then you're going to get overambitious and you're going to end up breaking something. Or in some cases breaking it more than it already is broken. One of my first hardware hacking was way back in the day with the original Xbox that came out. And folks like Bunny Wang figured out that you could mod the Xbox and do other things with it, like turn it into a media server, which was pretty cool. And then send stuff out to your TV. And to do that you had to get a mod ship for it. And I remember I bought a mod... I had the Xbox, which was, I don't even remember how much money that was. It was more than I wanted to spend, but it was a fun project. And then I had to get the mod ship special ordered from someplace else. And then it just came with like all of these pins. And I was like, "Oh no, what do I do now?"
Beau Woods:
And so, fortunately, I had a friend of mine who is an electrical engineer and he had some of the equipment around and taught me how to do some basic soldering. And after I screwed up two or three of the pins, he was like, "Let me just help you with the rest of that." And so he ended up doing all of the soldering to get the chip installed, but it's one of those skills that again, I think it's really handy to have and to develop so that you'll have it later when you need it, but I'm still not very good. If I sit down at hardware hacking village, or someplace like that at a hacker conference, it's one of the things that's still daunting and intimidating to me. But there are some cool kits out there too that you can get.
Joseph Carson:
Patience. Patience. That's one thing you're in ball pins. I was watching, I think it was Jills who was doing some ball pin stuff a few weeks ago. And just watching as he's posting his images, I'm just like some people have really lots of patience to really focus on that area and so on. Paulino, I'll just like to get some of your thoughts as well on that area. You like to be, was it agreeing on the electricity side of things? I think I've seen... I've been sitting in the dark. I always have a flashlight nearby nowadays because a lot of cases I blow the fuses. So just thoughts around what your...
Paulino Calderon:
For me was a new thing as well. I have bad eyesight and also my hands are quite big. So it's like that part by itself it's hard. So I go to practice a little bit when I start working with Electronic Cats. And also I think back in the day, I remember one of my friends took one of Joe Grant's courses and that's exactly what I remember the first times I started breaking routers or in mind and just playing exactly more and more hands-on approach into that.
Joseph Carson:
Absolutely. I was registered for Joe Grant's course and it was the first one that was canceled when the pandemic started, unfortunately. So that was a sad thing. Hopefully at some point in time, when things get back to normal that I'll be able to get into Joe's was it a course because it is very interesting just watching it...
Paulino Calderon:
I always hear from people that it's fantastic.
Joseph Carson:
Because when you get into, Beau, you mentioned around DEFCON and DEFCON is definitely when I go around the villages, I hang out sometimes at the IoT, there's also the Packet Village, there's, of course, the Biohacking Villages now and the medical devices and there's the Voting Village. Now it's really got into where it's almost everywhere. When you go to DEFCON, pretty much a lot of the villages are really focused on the hardware side of things. What other events throughout the year, and I think there's a couple of hardware, there are hardware.io events. So I think that's another major event that focused around...
Joseph Carson:
And I did see one of my favorite events last year was Kernelcon, which I so much... I think for me, it was one of the most enjoyable events just to sit back as a... Normally, when I'm going to events I'm speaking, but that was one event where I just sit back and I'm just going to watch and learn. And I was watching, it was Joe Grand running around looking for basically a blow torch or a heater or something. It was really entertaining. What other events do you recommend people? As you mentioned, Beau, you mentioned about that looking for someone who can help you. This is a very much a... You need people to help you. There's a lot of... This isn't something that I recommend that you go and only do by yourself. You do need people who have specific expertise and skills to help guide you, to make sure one is safety and you don't electrocute yourself because there are volts flowing through a lot of the devices. What other events do you recommend people go to try and network and connect with people to you to learn more?
Beau Woods:
You mentioned one of them, Hardware IO. The folks that run that are great and they've expanded it. I remember the first Hardware IO that they had in the Netherlands. I went there and brought some stuff, some car hacking stuff and some people who did car hacking and got them introduced. And it was really cool to see that community start to build and grow. And now they've got events all over the world. I think they've got... I think they just did one in Berlin. They do some in the US, obviously the Netherlands they've still got that, but that's a great place to go get training on hardware hacking and it's exclusively dedicated to hardware stuff. So even if you just go and hang out, you'll learn a ton. I learned a ton. Obviously too some of the DEFCON trainings and other things, some of the villages you mentioned are really cool, but also make our hacker spaces in your own community, if you've got one around, where you might be able to just drop in on a Wednesday or whatever, when they're doing an open session and go learn there or some other similar types of things. There's a big community around hardware hacking and growing too, depending on where you live in the world.
Paulino Calderon:
Online, you can also find help online by just asking people that you know if they are working on something similar to yourself. Most of the time they'll be willing to help you.
Beau Woods:
That's true. And we've thought about starting up a Discord channel or something to work with people. I don't know if other people would be interested, but maybe one of the things we can do with this is, okay, we've got the first joiner, But there's got to be some communities, some other communities to tap into. Paulino, we should probably get more involved and check it out and see if there's already a Discord that we can join or how we want to do that.
Joseph Carson:
Absolutely. And the one probably hardware that is the closest to get to is on with Christian Iceman. He runs the RFID channel, which is mainly focused on radio, but it's focused on of course the Proxmarks, other things and they focus on the radio and RFID signals, but absolutely having a much more broad scope hardware one that of course, you can separate into different channels. That would be a great idea because I haven't seen one myself. I've seen ones that have focused in certain areas, but not as you said when you talked about, when you're doing the book that with No Starch saying you need something a bit broader and absolutely for me a niche area that you focus into specific industries would exclude a lot of people's interests. But when you get into a much broader perspective, it gets a lot of people's interest in there. So having a community and Discord is a great way. And I thought the way Kernelcon did their event last year virtually, they did a combination of, I think it was Twitch and Discord was a great way and it was a really exciting event. So definitely one that I thought was really well executed.
Beau Woods:
And I guess to that point, we do have a channel for the book it in the IoT Village Discord. So if people want to check that out, you can go there, we can get you the link to that.
Joseph Carson:
We'll make sure all the links that you send and share, we'll get them into the show notes as well so the audience will not have to go searching for them.
Beau Woods:
It makes me wonder now if... I don't know if Hardware Hacking Village has a Discord, but maybe people could go check that out if they do.
Joseph Carson:
Absolutely. For organ, one of the things that of course, a lot of the, in the past, the hardware side was mainly either focused at consumer side and also most consumers were adopting a lot of IT and then there was also dedicated businesses. And I remember one of the things I worked on, it was actually one of the earliest things I worked on was basically was in the medical side of things where I was responsible for the ambulance service. And one of the first things I did was, we had to buy a bunch of new ambulances. And at the time there was a major... We were under SLA, which was that if the ambulances didn't get to the emergency or to the victim and back to the emergency room within a certain amount of time, we actually had legal requirements in order to make sure that we were able to get to any victims in 21 minutes.
Joseph Carson:
So one of the things at the time, and this is back in, I think it was 1999, 2000, we ended up connecting the, was it defibrillators and EPGs within the ambulances? We took an old Nokia 3110 with a data cable, and we were sending all of those through fax into the emergency room. And so the doctors basically in the emergency room would have this fax print out of all the health readings from the patients as they were on route to the emergency room. And for me, when I go back think of some of those early things, those were some of the things that we were trying to you use technology to put all these pieces together to save lives. And it becomes very critical. A lot of these things. And over the times when people ask me in the security industry about what's my biggest fear and I see that our lives are becoming more dependent on a lot of technology.
Joseph Carson:
You mentioned about pacemakers that are connected, maybe communicating through people's mobile phones through the Bluetooth that's near field proximity and so forth. And what risks do we face with a lot of... As IoT is becoming... My interpretation was always IoT... I think sometimes we use IoT as this very broad term and I remember one of my mentors saying it's basically network-connected devices, we talk about IoT, it's literally the internet, a device is connected.
Paulino Calderon:
I think Beau had a great definition for IoT. What was that, Beau?
Beau Woods:
I don't even remember what it was, but it was like something kinetic can happen from a computer connected thing, which is again, it's still a very broad definition, but it gets into that. One of the things I didn't really touch on in my intro is I've been pretty heavily involved in something called I am the cavalry over the past several years, which the problem statement for I Am The Cavalry is our dependence on connected tech has grown faster than our ability to secure it in areas impacting human life public safety. And with that initiative, what we're really trying to do is not to say we need to take computers out of everything. It's that the trust we place in these things should be commensurate with their trustworthiness. So if you have trust in something that's untrustworthy, you can work on one or the other side of that or both sides, but you can either depend on it less or make it more dependable. And so what we want to do is make it more dependable, because like you said, if you can get cardiac telemetry to a doctor before the patient gets there, then they can have the right types of technologies, people and processes waiting to save that life. And so...
Joseph Carson:
That was exactly the... That was the main goal was having readiness in the emergency room. So when people arrived, you had the right people that are waiting.
Beau Woods:
That's right. And you'll be happy to know, or maybe happy, maybe dismay, but now having cellular connectivity on defibrillators in an ambulance is standard practice for that reason. And so we can use this technology to greatly improve lives, to save lives and to make things better if we do it right. But one of the things that I really fear is not the insecurity of the technology, it's the lack of trust, the distrust. So when a hedge fund started short selling a medical device maker and they brought their trained hackers on TV to say, "Look how bad this is. Everybody could die." You had a lot of like little old ladies and little old grandpas going into the doctor saying, "Cut this thing out of me. I don't trust it."
Beau Woods:
I talked to some nurses from the veterans affairs hospital and they said that they had patients, military patients who would come in and they would say, "I'm not going to get a pacemaker, even though I know it's going to save my life, because I'm afraid that somebody might hack it and kill me." And so as part of the reason why we wrote the book the way we did, to talk to those consequences so that people are more conscious of them and don't make similar mistakes that can lead to people turning away from the best capabilities that we have to bring to bear on any particular problem. And I think that there's... I've also done a lot of work in other areas so if you're... Sounds like you're really into medical hacking, one of the things I do is I'm on the board of a organization called the CyberMed Summit. And we did the first-ever clinical simulations for hacked medical devices.
Beau Woods:
So just like pilots do flight simulators so that the first time they're landing with a 30 knot crosswind into fog, they've done it in a simulator and they know how to do it, they know how it feels, doctors do the same thing. And so we had some brilliant physicians who are also DEFCON speaker and hackers themselves who created these scenarios and ran a bunch of doctors through them in order to train on that. And in order to get an understanding of what would actually happen. And it turns out that in healthcare, they've got a lot of redundancies, a lot of backups. Very good reasons. So in running through some of those scenarios, the doctors through individual heroics were able to bring the patients back to life after a hacking event. But there was a big gap there, which is that the doctors themselves never questioned the medical devices. They just thought that it would either be working perfectly or not working at all. And so there are some things that we can take from this to improve the way that things like medical devices work, to improve the way that things like autonomous vehicles work, maritime systems, aerospace, planes...
Joseph Carson:
Industrials. Controls...
Beau Woods:
Yeah, industrial. And so I'm in this field. One of the reasons why I'm in this field is, well, it's cool, it's fun, it's exciting, it's challenging. But also we can really do something better if we're consciously thinking, how do we improve it, not just how do we break it. How can we build something that is better, that impacts human lives in a positive way, rather than just the technical bits and bites of it?
Joseph Carson:
Absolutely, well said. I think we're all looking to make the world a safer place. And one of the things I remember, I attended, it was one of the last digital, not the last year's Digital Summit, which was more about government geopolitical side of things, the previous year was on artificial intelligence. For me, which is basically just automation. It's doing things in an automated way. But then we realized when we're talking about it was about, we must do it with responsibility and accountability, it's about doing things for the right reasons. And not just because you can, but because you're going to make a difference. That's why I mentioned the story about the ambulances and the defibrillators and the emergency room. And that was all about basically saving lives. My SLA at the time was if my system wasn't running for 21 minutes, people died. Those were my metrics. That's what I was basically measuring each year. And every week, I had to make sure the system was running. And as you mentioned...
Paulino Calderon:
That is one of the worst SLAs ever.
Joseph Carson:
It's the worst SLA you can have. I remember it was... And the worst scenario, Paulino, one of the worst things was then I was doing a Y2K. I was going to run with the floppy discs, trying to protect against Y2K at the time. And one of the things that we were doing at the time was switching over our power. Our power was directly connected to the energy, the phases. And we were switching to a UPS for redundancy, just in case Y2K took out the electricity. And during that switch over to redundancy, I remember we had switched on the mainframes at the time, which was an old IBM office of servers, and we switched them off and we brought them back online. We're standing looking at this dumb terminal, old McDonnell Douglas dumb terminal screen, and with a little green flashing light.
Joseph Carson:
And I was freaking out because we passed the point of no return. The clock had already passed the point where we can actually revert and we're going, is the system running or not? We had no idea. And we ended up having to get... Because we had no... We were sitting in front of this server mainframe and no visibility of what was happening. So we had to get somebody from a remote ambulance service to connect back in through dial-up line in order to see if actually anything was running on the server, because it was basically the emergency call line, and that's how people find where people's addresses were. So if you wanted an ambulance to go to an address, that was how they find it in their system. And ultimately ended up being a serial cable was broken and I've ever since always have an actual serial cable in my body, unfortunately after that event. But that was my worry.
Joseph Carson:
It was always about... And that's where we always should approach things when we talk about IoT and as you're mentioning, Beau, we approach it with health and safety as a priority. And I think that's where we historically have probably not in our industry have been really focusing on the right metrics. We always focusing on vulnerabilities and patches but we should really be focused on the risks of the devices, what's the real true impact, and what's the likeliness of that possibility happening. And I that's really liked at the beginning of the book, it really brought that as the first thing was about what is the risks? What impact do these devices potentially have? How do you do threat modeling? How do all these things connect together? What are the dependencies in each of these areas?
Joseph Carson:
And I thought the book got really fantastic coverage in that some of the methodologies at the beginning, really highlighted the importance that organizations that we should approach this. And also of sometimes as well, it should not be just an IT problem or a security team problem. This is a business approach. And that sometimes when things go wrong, they point to the IT team, "You need to fix it." Or security team, "This is vulnerable. You need to patch it." But don't realize it's a business response. It's an entire end to end and we all need to work together.
Beau Woods:
Well said.
Joseph Carson:
So other things, what do we need to do... You mentioned about, I Am The Calvary to really highlight these, how do we move forward? This is one of the things that I'm always worried that we're a little bit behind. We're definitely playing catch-up in this field and area. We're coming up with a lot of great standards. In the EU they're coming out with... I don't get labels and devices will be changing things very drastically. We did see the UK introducing a new law that said no default credentials on the system which is another great step forward. What things do we need to be doing in order to have more trustworthy devices?
Beau Woods:
I think you mentioned the UK code of practice for IoT security. I think that's a great start it's within the last few years, it's become a EU standard. So the European standards group called ETSI, E-T-S-I, I always forget what it stands for, has now issued a couple of standards based on that are really well thought out. And for manufacturers, you can start building towards that. For hackers, you can start looking at those things and looking for those in devices and then framing it against that to have a more powerful way to talk through that with manufacturer or with policy makers. Singapore, Australia, I think a couple of other countries have started adopting that as well. I know that in, I think it was in Norway and in the UK and in Germany where the consumer product safety groups have raised awareness about security issues with IoT devices and those countries have actually banned those specific devices. So I remember there was one where it was a Cayla doll. And if you talk to Ken Munro at Pen Test Partners, he's always got one of these that he brings out, but you could hack the back end of it and pull down all the data and I think that...
Beau Woods:
One of the things for researchers especially is if you know about these types of issues, don't just tell the company they say go away and then drop it on Twitter, but there's other avenues. In the US, you can contact US-CERT, ICS-CERT, CERT CC. You can talk to other organizations who can help with that disclosure. In different countries they have different groups that you can go talk to. But if overall our goal is to save and improve lives with these things that we should be careful and take all due care to avoid incidentally accidentally harming it. Because at the end of the day, it's not about you, it's not about the company, it's not about the tech, it's about the impact to people.
Paulino Calderon:
And I do see some challenges there. For example, you've been talking a lot about policies in different countries, but for Latin America, we're not really pushing anything new in regards to that. And I see a bunch of security problems at least. Some of them have already been solved in different technologies. So part of the problem I guess, is how we are reusing the existing and knowledge and the challenge of what companies have. If there are smaller companies, how can they afford a full on assessment of external security, maybe you don't have that knowledge. So I think the risk is or part of the risks might come from different countries. So whoever is producing those devices need to start taking security more seriously. And as you mentioned, you see other, for example, Mexico's so close to the US and still we don't have anything in regards to that. We're about to come up with the first cybersecurity laws and we're ages behind or years behind, even though we're so close. And at the end of the day, we consume almost the same technology. Most of the same vendors are here.
Joseph Carson:
Absolutely. Paulino, you bring up a very important point, which is supply chain, is that when you get into a lot of this, is that a lot of the hardware and components of IoT devices is sourced through the supply chains, sometimes through the lowest component and bidder. And we had a discussion, we had CJ and Katie on a while back in one of the episodes and we were covering responsible display programs. In IoT, that's a difficult thing because sometimes it means replacing the hardware. Is there a responsible disclosure for vulnerable hardware? And is that something that we need to be also treating with a little bit more caution because it replacing... One thing is updating the firmware and a lot of these things that even the manufacturers, these hardware probably has even two year warranty, and then they stop delivering updates. Unlike software where you might have a longer stretch. What's your thoughts around responsible disclosure for IoT and also the vendor's responsibility for keeping it secure over the longer term?
Beau Woods:
It can get really, really tricky when you think about coordinated disclosure on IoT devices. And I know that there are some security researchers who have looked at the... they found a bug, they looked at the ecosystem of all the things that affected and they said, sometimes security through obscurity is all you've got. And when that's the case, maybe you can't disclose. Now, as we all know, repeat discoveries happen. And some of those were discovered again and disclosed in a very, I'll say, limited coordinated fashion. So you talk to the single vendor who makes it, but not all of their customers. And so like at that point, you're just in a race with the adversaries, which is not a place that anybody wants to be in.
Beau Woods:
And it gets really, really difficult because if you do have to replace the device, this is common in healthcare, I'm sure it's common in maritime, it's common in a lot of places. This medical device is still saving people's lives. If we have to replace all of them that are in the hospital, it might cost $8 million. How many treatments will that get for people who can't otherwise afford it? How many doctors will that afford? How many nurses will that afford? How many other things could we buy with that? And so you're in this situation where there's no easy answers. There's only really, really hard trade offs. And part of the equation is not knowable. Someone may hack it and cause something bad to happen. On the other hand, if you take those devices out of circulation, you absolutely know that patient safety and patient care will go down. So in a lot of cases on the operator side, it's an easy discussion. It's an easy choice for them.
Beau Woods:
You take a guaranteed harm against a potential harm, and they're going to favor the avoid the guaranteed harm every time. For security researchers though, it's really tough. And I've talked to a number of people who just wrestle with this. They know things that could probably kill folks, and they don't know whether to tell and who to tell about it. So A, we've got to do a lot better job shortening the time cycle between when the vulnerability is created by the developer and when it is ultimately fixed across the entire ecosystem. Not just the products, but when the patches and updates are applied. Second, building more defensible architectures so that we can withstand the occasional time when you can't update. Third, having a little bit better, more responsive ecosystem among security researchers and companies to be able to understand how, when, where to apply some of these different approaches to coordinate a vulnerability disclosure. And in 50 years, we'll have it all worked out, but we're going through a bumpy period and it'll get worse before it gets better, I think without some concerted effort. So if you're a researcher, think about those types of things whenever you find vulnerability and something that could be in the supply chain for something that can't be replaced.
Joseph Carson:
Think about the impact, Paulino, any thoughts around vulnerability disclosure, yourself, any...
Paulino Calderon:
I've gotten mixed responses from the vendors. I disclosed a few of them to... And I guess I've gotten better responses from, I guess, more reputable vendors. For example, I'm thinking one case that we actually didn't get a response back. And it's one of the examples in the book, the smart water bottle that we mentioned in the book. It was the top seller back two or three years ago. And when we disclosed the vulnerability to them, we try at least five or six, seven times all the details there in the email, we didn't get a response back. And part of the reason we chose that water bottle is because we knew it was harmless to disclose. You could just spoof that they're drinking too much water. No one cares. But at the end of the day, that ended up becoming a more critical thing when we discovered that your location was being disclosed without your consent every 10 minutes by itself. So it is a tricky situation that you mentioned. I find that you'll get mixed responses. Most of the time you'll have to hold on the information and just keep the secret, I suppose. It's not like you can patch it yourself. As Beau mentioned, you are in a tough spot and you need to think about the risks and consequences.
Joseph Carson:
Especially for software vulnerability. What we do is we wait for enough patches available and that's typically within usually that 90 days or 108 days, depending on the criticality there, but hardware it's a little bit more challenging. So it's been awesome having you on the show. And one of the things definitely recommends for the audience, definitely do read the Practical IoT Hacking. It's a fantastic book. It's definitely something that you will learn a lot. For me, I learned loads of new things and new skills. For the audience that's really in the hacking side of things, you really want to expand into IoT side of things. Beau, Paulino, anything you'd like to lead the audience into something that they should do next beyond reading the book?
Beau Woods:
I just really encourage everybody to get involved with something that they really care about and to take on IoT, it's challenging, but it's a lot of fun and you can make a real difference.
Paulino Calderon:
I also feel like this field is very new, so a bunch of new technologies that are being brought into a bunch of products. So there's a huge opportunity for you to write new tools, software and hardware as well. So I encourage you to play around with that. You'll definitely have fun.
Joseph Carson:
Absolutely. I think for many in the industry that this is a really great area to expand your skills into, and it's definitely in the future there's going to be a lot of jobs and a lot of exciting opportunities for people to work in this field. So Beau, Paulino, it's has been fantastic having you on the show. Really, hopefully look forward to catching up with you at some point in an event. Whether being at DEFCON in the near future or another event, I really look forward to catching up. Again, many thanks. For the audience, again, subscribe. Every two weeks we have the episodes, make sure you stay up to date. Go and look at some of the older episodes. And it's been fantastic. Beau, Paulino, Thank you for being on the show. You've been awesome guests.
Beau Woods:
Thank you, bye-bye.
Paulino Calderon:
Thanks.
