EPISODE SUMMARY
Subscribe or listen now: 
Intro: You're listening to the 401 Access Denied podcast. I'm Mike Gruen, VP of Engineering and CISO at Cybrary. Please join me and my cohost Joseph Carson, Chief Security Scientist at Thycotic as we discuss the latest news and attempt to make cyber security accessible, usable and fun. Be sure to check back every two weeks for new episodes.
Joseph Carson: Hello everyone. Welcome back to another 401 Access Denied podcast. I am your host Joseph Carson and I've got a very, very exciting discussion. We're really looking forward to, hopefully, going into some fun topics and some interesting things … that you'll probably start to see more in a future scenario. And interesting to see where this conversation's going to go today. I'm joined again with my cohost Mike. Want to give a Mike Gruen hello?
Mike Gruen: Yep. VP of Engineering and CISO here at Cybrary. Very excited about today's topic. We have Josh Lospinoso once again joining us. Always exciting to talk to him. Today the topic is OT, specifically around transportation security. And I'll let Josh introduce himself. Welcome back.
Josh Lospinoso: Thanks. Thanks for having me. I guess the last episode didn't go too poorly and I'm glad you invited me back. I'm Josh Lospinoso. CEO of Shift5. We're an OT cybersecurity company that defends planes, trains and tanks from cyber-attacks.
Mike Gruen: So you know nothing about this topic.
Josh Lospinoso: I don't. I'm coming in blind here, so I'm hoping to learn a lot from you guys and...
Joseph Carson:
Really interesting because we're talking about transportation. For a person who travels a lot, and you do get to see a lot when traveling around the world and different countries. But being based here in Estonia, I got involved in quite a lot of new innovative areas and I'm really interested to get your insights as well, Josh, today on some of these topics. When I got involved in specifically in OT and transportation, it was around probably just a little bit over 10 years ago, probably around 12 years ago now. And it was in the maritime industry. And I was working on projects. We just did a project in the mining industry where they actually did, basically with large mining trucks, those super basically massive dump trucks that they have. What they'd done was they basically, not made them autonomous, but what they did was they had driver-assisted. So they were no longer having the drivers in their trucks, they were actually having them remotely operate them.
And the whole goal around the initiatives here was to remove people from harmful, hazardous environments. That was ultimately the initial goal, was to take people out of places where their life could be in danger by falling rocks or explosions going off or there are trucks not being able to see people on the ground. So they really focused on the people-side of things and health and safety. And then again, in the maritime industry, the same project. It was around making ships autonomous. What we end up getting to, we did get to autonomous methods and techniques in there, but initially, it all became remote assisted again. It all became captains of ships were no longer sitting on the bridge of a ship. They'd be sitting in a remote office location with a simulator, a proper bridge. It was set up to look exactly like the bridge of a ship. And they would sit behind and actually remotely operate the ship.
Now, there's a lot of challenges. The things that we did find into when we actually went down this path was all about the safety systems. Safety became the major, major concern. If you take, for example, anything where it involves people, anything where it involves people getting on public transportation. There was a car ferry in Finland and ultimately what happened was this car ferry was to be automated, completely automated. Take passengers and vehicles from one port to the next. And ultimately the major challenge we came across was, again, the safety systems. That if somebody fell off the ferry into the water, what machine or what technology could actually throw the life vest to that person? And ultimately what resulted again, is that yes, they become more efficient done in the actually autonomous operations, but it ultimately came down that they still needed people to be involved in the health and safety side of things. So that was always the driver that I've been finding a lot of these projects, it was around taking people out of hazardous environments.
Josh Lospinoso:
I think that is one absolutely appropriate reason why OT systems have seen so many... the integration of technology onto these systems to replace analog systems. There are a lot of other ones. As early as the 1980s we saw OT systems starting to bring electronics for controls and sensors. Bosch is one of the pioneers, actually, in this space. They invented controller area network, CAN, I think it was for Mercedes-Benz vehicles. I think for two reasons they went down this path. One is, well, electronics were just a lot cheaper than analog systems for sensing and actuating things. They're oftentimes more reliable and you can manufacture them to higher tolerances. And the second thing is that they're better functionality and they're cheaper. And so given those two very strong economic drivers, all of a sudden now you've got all the incentive in the world for all of the OEMs that manufacture this stuff to be moving towards electronic systems to operate these things.
And I think it's really helpful to start this discussion off as you did with the safety discussion because I don't want to give people the impression that OT systems aren't safe, that they're not engineered for safety in the physical world. They very much are. Engineers that design these systems think a lot about how these systems are supposed to be robust and reliable in the physical world, like in all kinds of operating scenarios. But where people like me come in is we say, that's great, but what if you have a whiting adversary who's compromising the electronic components of that system that can take on so many different forms? But what if all of a sudden you introduce EVE into the scenario and now you've got an adversary who's trying to subvert this system on you, how does your system perform now?
We do this analysis on IT systems all the time. We've got unit tests and reliability integration tests to make sure that our software does what we say it's supposed to do. And then we also have, as you very well know, we have penetration tests to tests our systems for robustness against whiting adversaries. We don't have that ladder scenario in the vast majority of OT systems, and I think this is going to be a big frontier in cyber security over the next…
Joseph Carson:
Absolutely. Because what you're saying is one of the things, the big difference, I mean we talk about IoT and OT, is the big difference is those are typically very, very focused to specific tasks. They have one function; open a valve, close the valve, open a valve, close the valve. Read the temperature of some of the tools, I get why you're a correlator. They have very, very specific tasks. And what we're used to in typical IT systems is they're multifunctional, we can reprogram to repurpose. And therefore, we had to think of the bigger picture, but what's all these possibilities? How can it be abused? How can you really take advantage by taking one protocol, modifying it, and then being able to take advantage of that? But in OT, they basically test in these very, very specific limited scenarios, and they don't step back and say, well, if somebody has access to this and changes the parameters or change the configuration, how can it be abused, how can it be manipulated? And that's…
Mike Gruen: I think it also…
Mike Gruen:
On that though, I think it's very, very similar in software engineering to the difference between a unit test. You have your function. Your function, it has inputs, it has outputs, you write all the unit tests. You're like, this is great. You can even test some things under breaking, but then that's why integration and system and end-to-end tests are so important is because it's not how does thing operate in and of itself, it's how does it operate within the larger system. And it's once you get all those interplays between these very small autonomic things, that's when you get the problems is when you start bringing them all together, and then they start working in ways that maybe you hadn't anticipated.
And one thing I wanted to say on Josh's first point though, I think... and yours, Joe, as well, which is I think the systems really did come from this notion of we want to try and create a safer world. Again, I just can't emphasize enough what Josh was saying about these systems that were designed for safety. That's where they started, right, and so now we're just talking about what happens when people start to try to abuse them or take advantage of them. There's also plenty of opportunities, I think, where there are just unintended consequences. You plug in a device, now cars have USB, things can interact, and it might just be an unintended consequence.
Josh Lospinoso:
They have telemetry solutions connected to the internet. The access problem has sort of gone away, it's not theoretical anymore. We're so many ways onto these systems, we have to stop ignoring OT systems as technology. I couldn't agree more with you, G. And just to emphasize what we're all saying, just to make it crystal clear, I think for people that maybe haven't thought about the IT/OT distinction, the way I think about it is IT systems are there to help make business decisions. They're there to store data and help business people operate their business. OT systems are there to do business. They're there for the actual operation of your business. And oftentimes that means manipulating the physical world. And when you have a cyber compromise on the IT system, it can be bad, don't get me wrong. We've seen examples where people get CryptoLockered at hospitals and it has real impacts on people's lives. You can have losses in the millions and billions of dollars as a result of credit card fraud and all these sorts of things. But if you compromise an aircraft or a train, it is very easy for people to lose their lives.
The seriousness of this is just the order of magnitudes higher than it is on IT systems. I feel like Chicken Little sometimes screaming about what a problem this is. The metros that you ride to and from work every day are operated by computers. And those computers are not hardened against cyber-attacks. It's insane. It makes me very, very, very nervous. And it doesn't take a lot of imagination, to Joe's point, to imagine, well say you either plug the rogue device onto the database of the train, you compromise a cell modem that's attached to that bus. You compromise the wayside infrastructure, you compromise the laptop that gets plugged into that train to do maintenance. Any component of this or supply chain attacks, and no all of a sudden you broadcast a series of messages on that bus and the train can accelerate indefinitely into the blocks at the end of the... You know what I mean? It doesn't take a lot of imagination to see...
Joseph Carson: In addition to that, one of the things... So for example, when we did the car ferry, the autonomous side of things, that was a very interesting project. And the reason why we did the car ferry first was that we didn't have to deal with international maritime law because international maritime law specifically has a clause with no unmanned vessels.
Josh Lospinoso: I didn't know that.
Joseph Carson:
You have to have someone on the vessel, so that's why they did it in local waters to start off with. So we looked around a lot of these different scenarios. And ultimately even the problem was, and I think this is crucial when we talk about, and Josh going back to your point, is about IT is really there to help people make decisions and to interact. There's a lot of human interaction. When you get into OT, the interaction between the humans is to observe monitoring of the data and to make sure that those systems are performing to what you're actually intentionally... perform or intend to do. Whether it's producing electricity, whether it's basically logistics and shipping calculations, and so forth.
And the things that I found was is that when you mix, let's say the OT and IT together, that's what creates the big risk when you actually converge those both together. Just like if you take, for example, when we did the shipping lane, is that everything was perfectly fine if you remove all other vessels from the water. And we have this little person who's on his day cruise in his little boat, and is coming across, and isn't actually, let's say, putting up his AI system tracking to let other vessels know that he's in the water. That's where the problems occur. It's that even in autonomous vehicles. Here in Estonia, we've been doing a lot of testing, and when we did the testing a few years ago during the EU presidency, what we found was we actually had to have a dedicated autonomous lane because when you mix autonomous vehicles with other drivers in a road, that's where accidents happen. So we get into it's this convergence, which is actually increasing the risk. And we really had to step back and think about do we continue down that path of convergence, or do we need to do that continuous separation of OT and IT to make sure that we don't have that convergence.
Where people walking into manufacturing floors, just as you mentioned like in Bosch or car manufacturers, the minute they step across that line where this actually OT environment, when you put humans in that, that's where accidents happen. That's where the safety problems evolve. And that's my concern is, is that what I've seen in my recent experience is that if you buy an engine for, let's say a … or a ship or something, if you buy the engine or even if you get an autonomous vehicle today, so if you go buy a Tesla, you own the physical hardware. You own that device, but the data that's being generated continues to be owned by the manufacturer. And that's where the requirement is, is to keep those devices internet-connected. Meaning that that data, again, be transferred to the actual manufacturer. And this is where, for my concern, a lot of the problems start to arise, is that convergence of OT and IT. And therefore, those OT systems were not designed, basically, in a penetration or security design. They were designed for safety.
Josh Lospinoso:
Exactly. Human nature is so persistent across all of these different evolutions, and it reminds me, I read about this because I wasn't alive in the seventies, but it reminds me of when I read about how academics invented the Internet. And so you've got ethernet and TCP/IP were invented for these... they're amazing, but they're invented for redundancy across links going down. Again, reliability and robustness, not for cybersecurity. And so if you've got a network of trusted people TCP/IP and ethernet make a lot of sense. But what has happened is since we opened this up to the broader population, we're having to cope with all of the problems that are associated with the fundamental insecurity that's built into the protocols that underpin modern society. And we're doing, I think, an admirable job. I trust banking transactions online now. PKI's got a lot of problems, but by and large, it does raise the bar quite a bit for cyber attacks.
I think operating systems today are much, much, much more secure than they used to be, and they do a pretty good job of mitigating. And then if you're doing what you're supposed to be doing on an enterprise IT network, I can say from years of experience as an attacker, it is hard for us to gain access. Someone has to mess up, basically, for the most part. And unless I'm very careful, I'm going to get caught. If I'm doing a bunch of crazy stuff on the network, I'm going to get caught pretty quickly. And so we've gotten to a place where I'm not happy with it, but I have an acceptable level of irritation about the level of IT cyber security. I am terrified on the OT side because we've had this parallel evolution of all these technologies that have evolved for the purposes of serving some sort of economic purpose without a corollary growth in the cyber security products to patch what's going on on the side of them.
Putting my money where my mouth is, this is what Shift5 is trying to do is we are making an intrusion detection and intrusion prevention system for the databases that are on OT systems. Not ethernet because there are a lot of ethernet IDSes. We do IDS on the serial protocols that are on these systems. And so it's an obvious thing when you think about it. You're like, why are we doing intrusion detection on OT systems? There's this entire... As you said, there's data getting generated, gigabytes of data a day getting generated per system. Why aren't we monitoring that stuff and bringing that into our environment for the SOC so that people have a good understanding of what's going on there?
I think there will be an ecosystem of cyber security companies attacking different parts of this problem. You've got the endpoint security problem of is the firmware hardened on the electronic control units so that they're hard to compromise? Are we signing firmware that goes across these databases to upload onto the controllers? That sort of thing. Maybe firewalls for each of the electronic control units. There's a whole ecosystem, I think, of products that are going to arise around trying to retrofit security onto these systems that are not going anywhere. And then there's also OEMs trying to build more secure systems by first principle. There's a lot of activity here, but it's really early days.
Mike Gruen: But I think the other thing with that is that OT, especially transportation systems, tend to be around a lot longer. When you think about IT systems and how the lifespan on most of those is a couple of years, whatever it is. Cars, planes, trains.
Josh Lospinoso: Decades.
Mike Gruen: Exactly. Looking at the D.C. metro system.
Josh Lospinoso: Decades.
Mike Gruen: Decades. Decades. And so....
Josh Lospinoso: They're really expensive assets. The stuff that we're looking at isn't really automotive, it's multimillion-dollar assets. The earthmovers that Joe was talking about or an aircraft is millions and millions. A military weapon system like an Apache attack helicopter or an F-35. Oh my God, those are hugely expensive assets that are not going anywhere for a very, very long time because you have to amortize that cost over years of use.
Joseph Carson: Because I remember a few years ago I got the opportunity to see a satellite decommissioning. And that was always interesting. I always remember it, the person it was that we were observing, and a colleague of mine, a kind of peer said to me, he's like, "Right now, as they push this button that was designed 30 years ago, we hope it works."
Josh Lospinoso: Everyone's retired, so if it doesn't work, there are no ramifications for anybody's careers.
Joseph Carson: So the person designed it is probably retired for many years.
Josh Lospinoso: That's right.
Joseph Carson: And it was interesting. He's like, "Because right now we have to push this, and then what it's going to do, it's going to take the fuel out of the satellite, and then it's going to slowly move it to this parking lot, which then is …” I'm just going, and the satellite was meant to be, in circulation, was meant to be for 20 years and this is now in its 25th year. And actually prior to that, it was actually already in development for five years. So this single button that they're sitting there …
Josh Lospinoso: Using parts that were 10 years old when they started design because... So it's basically made out of vacuum tubes.
Joseph Carson: Exactly. And that's the perspective, as Mike and yourself are saying, is that's the perspective. Is that a lot of these are designed to be around for a very, very long time. And even I see, when I get into ships and I go into the bridge of a ship, and you're looking at things that show Windows 2000. We're not even talking XP, we're talking Windows 2000. You might even hear the Windows '95 chime once in a while if you stay around long enough.
Josh Lospinoso: This is where NT, new technology, actually met new technology because it was the next generation of the Windows Operating System. These things are operating in the real world. You want to talk about bad cybersecurity. You compromise a user mode program, you own the box. There's no kernel user mode distinction.
Joseph Carson: There's no separation.
Josh Lospinoso: It's crazy. It's just like... For sure. I read this article that Her Majesty's submarines were running, I think it was Windows Me or something insane. I was just like, I can't... There might be a nuclear payload on this thing. This is just nuts.
Joseph Carson: Even the new submarines that they're actually... Was it in the UK? They were designing the latest carriers and submarines and they were still running XP as well. I'm just going to... It's like it's because it's proven.
Josh Lospinoso: That's right.
Joseph Carson:
It's something that they're used to. It's very costly to then try and transition some of those things over to work… So they're using what they know. And this is the difference between the OT and IT, is that we are continually, let's say, even in IT, the life cycle's expanded slightly. We have to be real. It used to be two to three years you get new hardware. You'd change a third of your equipment every year. And you'd just keep that rotation till... and then we started getting a bit more cost-effective and we started getting new processors and chips that would last a bit longer and they were advancing quicker than the operating systems could take advantage of them. Things when we moved from 32-bit to 64-bit and so forth. And then hard drive space started accelerating as well.
So we've expanded in the IT to being somewhere, maybe five years, you could push it to seven if you really wanted to. But when we get into OT, we're talking 20 plus in some cases. To your point, is that it's too costly to replace them. So do we need to get to a point where it's more modular, that these things are not designed as massive, let's say, one component? That they can be modular, you can replace it much easier. Just like we get into sensors. Just like sensor light bulb breaks, you take it out, you put another one in, you're good to go. Do we get into a much more modular system for OT systems?
Josh Lospinoso:
Yeah, I think it's a great observation and we're getting there. One of the things that struck me in the rail industry, for example, is a lot of OEMs that manufacture components to make up a train, they will use communication protocols that are standardized. There's one, the Society for Automotive Engineering, SAE, put a standard out called J1939. It's this 1,000-page document that in exquisite details says, hey if you're going to put a CAN message together that says what the engine RPM for the main motor is, this is the format for it.
And for economic reasons, it's just made a ton of sense for that industry to start to congeal around that standard. And it permits the sorts of things that you're alluding to, which is well, if I want to add a new sensor or replace the braking system or whatever, obviously there are the physical components you have to figure out, but from a controls perspective or a sensors perspective, if the sensor complies with J1939, it means you can swap it out. It's much like you think of a motherboard and you've got, oh, this talks PCI, cool. I can plug that in. And so I think it's definitely getting there for economic reasons.
So as G is fond of saying, we do agile software development, but agile development for physical systems in the world is a really bad idea. You can't agilely develop a bridge and you probably shouldn't do it with an aircraft. MVP of an aircraft is a terrifying prospect. And so all these things nest together into what you guys were saying because when you design an aircraft, you're like, we are going to hit this out of the... We have to get this right the first time. And for us to do that, that means we're going to put components that are super reliable that we know and we've tested the crap out of, into this system. And then we're just going to keep it the way it is because when we start changing stuff, Boeing 737 MAX 8, we end up with all kinds of really difficult problems that can kill people. And it's fundamentally different from upgrading your data center to solid-state drives.
Joseph Carson: My concern is when we get into OT is that more of it becomes software. As the more you get the vulnerabilities.
Josh Lospinoso: Dude, this is where we live. That's it.
Joseph Carson: …software adds to it.
Mike Gruen:
To your point, Joe, earlier, about them being connected. On the one hand, having it not connected is somewhat good, I guess, in ways, but having them be connected to the network means you can do things like what Tesla did, which was you can actually push out a patch and upgrade and update systems that are on the vehicle. And so I think you have this push and pull there as well. The fact is if you have physical access to a thing, it's much easier to compromise. That's always been true. But then internet connecting it just means you can do it from that much further away. But I think, in addition to modular, being able to actually patch and update these systems is also a critical part. I'm curious, Josh, if you're seeing that as well?
Josh Lospinoso:
Oh, a 1,000%. Again, for economic reasons. I didn't know a ton... I've always done a lot of systems programming. I got recently into a lot more embedded device type stuff and it's just blown me away how... Basically, I looked at the stuff 15 years, and if you wanted to design a tiny computer, you're making a PCB and you're designing circuits, you're doing all this very low-level stuff that's fundamentally electrical engineering and there's some logic to it. You can get a taste of this if you ever have any sort of masochistic tendencies. You can learn Verilog and you can program an FPGA to simulate some of this really low-level stuff.
These days, microcontrollers have gotten so cheap that now the difference between programming a computer and making embedded hardware, there's very little difference these days. You can go on Adafruit or SparkFun or some of these vendors and you can buy dev boards where you can run Python on a microcontroller and interact with hardware analog signals, digital signals. They've lifted it up to the point where you could write Python on a … It's insane. And the reason is that microcontrollers have gotten so cheap and so sizeway and power-efficient that now when people are manufacturing new hardware, they're like, we could go through the pain of designing circuits to do this operation, but actually, I'm just going to stick this four dollar microcontroller, this PIC microcontroller or whatever, onto this system and write C. And it's just going to do all the IO and calculations and stuff for me. And oh, by the way, that means I can update the firmware which it's just an operating system. You compile a new operating system, you send it to the firmware.
So we've seen this... basically now, there are, when I say "tiny computers," I mean tiny computers that are running these real-time operating systems on all of these devices. And it's amazing and terrifying at the same time because now you've got a network of computers, a network of computers. It's like think of systems that were running on ethernet, which in the seventies would probably have less computational power than... definitely have less computational power than an ARM System-on-Chip that you might install nowadays. That's what you have that's operating a train, is a network of computers communicating over CAN and serial protocols, and they're running teeny little real-time operating systems. So all the things you're saying, it's already there. These things have happened for economic reasons because it's just so much cheaper, more efficient and you're able to do a lot more from an engineering perspective with very little capital investment.
Joseph Carson: Absolutely.
Mike Gruen: Are they planning on doing anything with the protocol itself in terms of security, because if all it is is messages, right, and the protocol sort of... Do the people who, the bodies in charge of those protocols, are they doing anything to try and look at it from a security perspective and signing messages and the assurance side of it?
Josh Lospinoso: Two comments here. One is academics are hell-bent on fixing these systems and redesigning them from first principles with security properties. Where have we seen this before? But the industries are like dude, you think we're going to replace CAN at this point? It's here to say, man. Ethernet, here to stay. It's not going anywhere. Ethernet's got a lot of problems.
Josh Lospinoso: Totally. You can ARP spoof an ethernet network and basically own all... Man in the Middle of all the traffic on a collision domain with a very simple technique. The protocol's broken. From a security perspective, it's broken. We build security on top of it because we've got all these legacy systems that aren't going to go anywhere. And to Joe's point, OT assets are even stickier than IT assets. Call me a skeptic, but I do not think we're redesigning the systems. It's just not going to happen.
Joseph Carson: And to your point as well, I got involved a few years ago, looking at a lot of … does and the stuff he does with embedded boards. And I got into Bus Pirates and connecting with … and …
Josh Lospinoso: I got one in my closet.
Joseph Carson: You're really getting into low-level things. And one of the things that you're missing is these embedded chips and the boards, that things like the Pis and the Ardunios and stuff, and just getting them and just build them to your specification as to what you need. What happens is that adding security is sometimes actually limiting what you can actually use them for.
Josh Lospinoso: Always. It always does.
Joseph Carson: Adding encryption caused a lot of problems. Even when I was working in the maritime, adding encryption to the VSAT communication uses up about 20, 25% of your bandwidth. And over basically an L band or a K band communication, you're not going to do it.
Josh Lospinoso: Unacceptable. You're not going to do it.
Joseph Carson: Because I remember even in the oil rigs, landing a helicopter, you had to stop all patching, all communications because the most important thing was that helicopter to land safely.
Josh Lospinoso: That's right.
Joseph Carson: And that's where you get into, is when there's a helicopter landing on a platform, that if you were doing any types of maintenance or system updates or even an IT or OT, everything was put on hold until that helicopter was on the deck. And this is what you get into, and when you add security, security uses that bandwidth, uses up processor power, and some of these devices don't have enough to actually do both.
Josh Lospinoso:
I couldn't agree more. And so, in fact, this is one of the key points that we make is something as simple as software like attestation. You sign software, that is not a complicated operation for a modern CPU to do. You're doing some software signature verification checks, it's like an unnoticeable blip on your CPU usage. Most microcontrollers just basically can't handle that computation, period. It's a bunch of big numbers that you have to multiply together in checks and they can't do it. They cannot do it. And so you're left with basically if I want to take firmware updates on this chip, I can either upgrade it, which means it's going to cost more to manufacture, it's going to take up way more power, and I'm essentially adding a lot of cost for this cybersecurity feature that no one seems to really care about. Not going to do it. Or just say the industry standard here is we don't do firmware verification and now you've got a huge problem. Now you've got a huge problem.
And so the way I see this, and of course this is totally talking my book here because this is what Shift5 does is we put a security appliance on the bus that will handle all of that computation for you. And since it's a single collision domain, we can see all the traffic that's going across this bus. We can do all the software verification on our device and that can leave the little microcontrollers to do their job.
Joseph Carson: So that reminds me, you reminded me of a scenario that happened about, it must be five years ago now, where when I was doing a penetration test. This was on a ship management company, so the company manages the insurance and the logistics and the staff and who goes on the vessel. And one of the things is that they had an incident. During the penetration test, we found out that their light bulbs were a major vulnerability to their actually internal networks.
Josh Lospinoso: Of course. Of course, it's the light bulbs.
Joseph Carson:
It's the light bulbs directly connected to their wifi. Now it was the guest network, which was good, it wasn't their production or corporate network, but it was still connected to a network where you could actually use that as a, let's say, an entry point, and then start elevating from there. Because once you get on the network, it's only a few more steps before you're in further.
Now with that, one of the discoveries was, and this ultimately getting reported to the vendor, the manufacturers of the bulbs, and ultimately resulted in what was referred to then was the smart hubs. In that rather than connecting these devices directly to the network, just like all vehicles should not be connected to the public internet, you should actually be connected to some type of vehicle segmentation or aggregate or correlator. And therefore, what they started being able to do was know the access points, know the data that was going through, know the firmware versions that were out there.
Once you start correlating these into, let's say gateways so that those edge devices are not truly on the edge of the internet, they're actually on the edge of a gateway that actually controls those entry points. That became really where you're starting to... Even in power stations, they did the same thing. Rather than the substation being connected through, so they're pumping internet back in, which you have in much of Ukraine, which is mind-boggling, but you'll actually them go through these proxies that you can control. It's almost like putting a fence around the things you can't secure.
Josh Lospinoso:
Absolutely. Because it's the world I live in of OEM operational technology, I've been talking a lot about these autonomic OT assets that are common in transportation and defense and things. But the broader OT term includes things like IoT devices and ICS and SCADA, which is something we haven't talked... You've been talking quite a bit about it, Joe, but I think we could discuss some of that.
So I ICS and SCADA cybersecurity, you're seeing a lot of really great activity in because we've seen incidents. There are a bunch of companies like the Clarities, the Nozomis, the Dragoses of the world.
Josh Lospinoso:
Exactly. Really smart people working on an incredibly difficult problem and making, I think, really good headway in that space. And I think, from my perspective, ICS and SCADA is really the bellwether for OEM operational technology because, of what you're articulating, which is just by its nature, ICS and SCADA has evolved so that a lot of this resides on TCP/IP networks. Look at Modbus and PROFIBUS, you've got these huge machine interfaces that operators are using, they're communicating with the programmable logic controllers over ethernet. This isn't a familiar environment for cyber attackers, which is ethernet and sure, I have to write a Wireshark to sector to understand what Modbus looks like and get a better understanding of how does this control and what are functions, that sort of thing.
But it's a familiar enough environment and these things are connected to networks that I'm familiar with compromising, whether that's through a light bulb or it's through a phishing attack. And so we're seeing a lot of cyber intrusions on the ICS and SCADA space because historically has been a very bad separation between these two things. And that's been both on a network segmentation perspective, and also just on the kinds of technologies that you find embedded in ICS and SCADA systems, which just tend to be ethernet for whatever reason.
Joseph Carson: And going to your point that you were actually mentioning earlier, is when you talked about the train scenario, a ransomware scenario is really getting to what I do find is that the big differences in IT, we're talking typically data and some kinetic, some types of interactions. But when we're talking about OT, it's very kinetic.
Josh Lospinoso: Exactly.
Joseph Carson: It's very interactive. It has physical world interactions. Whether it is pushing a train forward on a track, or whether it is controlling the vehicle. As I mentioned, here in Estonia we had the World Rally championships recently. And one of the things that they actually did and showed off was actually we had an autonomous bus do the actually WRC rally track.
Josh Lospinoso: That's amazing. That's amazing.
Joseph Carson:
Which is impressive. Just going and taking it and it went and followed the track. And that just moved to one of my next topics is that one thing that I find is the energy, the energy was always a bigger problem. Is that those companies move, for example, historically legacy OT would have been using energy that was oil-related, petrol, gas. Now as they moved into more, let's say green energy, what I've found is those devices tend to be less stable. They are lighter, they're more efficient, but they require a lot more human intervention to keep them going. We did the shipping side. One of the things was we moved people away from the vessel. They're LNG-powered ships, so they're saving huge amounts of weight on the vessels themselves. But the minute you move people away is then the maintenance and the reliability of those engines are much less than it is from things like diesel. So we get into this where do we get into, what's the ideal scenario for savings and security, and making sure they're efficient and also from an investment perspective. There are always those challenges as well.
Josh Lospinoso:
For sure, and I'm fascinated to see where these things go. I think, generally speaking, corporations are run by smart people that are constantly monitoring their P&L. Am I operating a profitable enterprise here? And I think the theme with a lot of this conversation, I think so much of the adoption of OT security and procedures around that and how humans fill into that overall ecosystem is driven by how profitably you can run your business. And so to your point, I think once it becomes economical, for whatever reason, whether that's government regulations making it so or because we just end up with green tech that's so much cheaper than oil and gas or some combination of the two, we're going to continue to see the roll-out of a lot of really smart power generation facilities. And I think it's an open question. How are corporations going to figure out, well, we've got this new tech, how many people do we need to help operate this thing? What's the most efficient... I think it's called the labor returns to capital or something. What's the right ratio there? I don't know that I have the right answer, but I think we'll certainly, in general, tend to see fewer people involved in the operations and maintenance of OT. I think that's a pretty safe bet.
Joseph Carson: A lot of increase in the use of drones in this area as well. Looking at oil pipe and gas lines, looking at, let's say, the hull of a ship. So that you don't have people going and observing, let's say, after mining does some basically, let's say, explosions. And then rather than sending people down to see what happened, maybe just fly drones down and not have actually people going…
Josh Lospinoso:
How much sense does that make? On the entrepreneurial side, I get so excited about these old storied industries that have really... In the past 150 years, these industries have made our lives fundamentally indistinguishable from where human existence has been for 100,000 years. The whole reason we are talking over electrons halfway around the world is underpinned by all of these old industries that have made our modern lives possible. And I get so excited as an entrepreneur, about looking at these old industries and figuring out how can we give them the next leap forward and take advantage of all this amazing technological progress we've made.
I get less excited about TikTok or the next dating app. I don't know how you feel about Peter Thiel, he's a very polarizing figure, but this is a big thing that he goes on about. Which is just there's so much incrementalism of building the next Uber for whatever and we've lost sight of there are these huge problems that we want to solve as a human race. If we either create new industries or we go back to these really old industries that have been resistant to technological progress, and as entrepreneurs, figure out how do we go to those industries and make them better, faster, safer. I get really excited by that. I think there's, not to torch the analogy, but there's a goldmine of possibilities there.
Joseph Carson: I remember I was involved in the Shipping 2030 project, which was all about the future of shipping. I was heavily involved in things like energy shipping, renewables, and the mining industry and autonomous vehicles here in Estonia. So I got heavily involved in those. But seeing the innovation was so exciting.
Josh Lospinoso: It's so exciting.
Joseph Carson: And seeing … it was, for me, I get more excited about what a drone capability has versus what the next influencer on a social media app has the ability of... Because ultimately what they're doing is just selling their time. What we're doing is making change in people's lives. That's the difference-
Josh Lospinoso: I totally agree.
Joseph Carson: ... is you're making a difference to society, not just a difference of entertaining, and that's the biggest difference.
Josh Lospinoso:
I get it, but it makes me so sad that some of our smartest peers are working at big tech companies figuring out how to get people to click on ads. That just makes me very... Not to get super philosophical or anything, but I think that's where the market is failing us, honestly, is I think there's just a ton of money that's going into what I view as just not really helpful to society. Another one is in finance. I think for up to a certain point, things like hedge funds that provide liquidity to the market, there's some utility, but we've gotten to the point of diminishing returns. And some of our best and brightest systems programmers are figuring out how to reprice options within microseconds rather than milliseconds. And it's like, dude, come on. You should be figuring out how to put people on Mars. What are you doing, making billionaires richer?
Mike Gruen:
That's funny because that's basically what my college professor, the dean of our department said when I came back in '96 from a summer internship working on web form software. He turned to me and my colleague, my peer, and said, "You two are a couple of brightest kids in my class and this, this is what you chose to do with it." And it really did drive my career. I still remember that. I still remember the look in his eyes and just the total disdain.
Josh Lospinoso: Let's be clear, web forms tends to do that to people.
Mike Gruen: Well, this is '96, '97 so we were the first ones. We were out there.
Josh Lospinoso: It was terrible just back then too.
Mike Gruen: No, totally terrible. I will say, on the plus side, we did create a university. We were helping Davis University, so we were helping to educate people, but no, it was terrible. And so that drove a lot of my decisions throughout the rest of my career in terms of trying to go to, whether it was contracting in NIH or whatever. And I think to your point, Josh, I think there are various ways to give incentives to people to move into these other spaces, but money always talks, right.
Josh Lospinoso: It always talks.
Josh Lospinoso: And fortunately though, I think it just takes a couple of people leading the way. As I said, there's gold in those hills.
Mike Gruen: Absolutely.
Josh Lospinoso: These old industries, there's so much operational efficiency to be gained by smart people solving important problems. The revenues in mining, for example, are astounding. And so if you can make operations three percent more efficient, you're talking about real money. And so I just think it takes people talking about this stuff and bringing it to public consciousness. We don't need to be talking about the next JavaScript framework on podcasts. Let's talk about what are drones doing to the mining industry, how can we make planes more secure from cyber-attack? The more that technical people are excited about these old and crufty industries that people are maybe not on the face of them super excited about, the more we're going to encourage young people to get into those spaces.
Joseph Carson: Turning ships into massive 3D printers. Giant 3D printer on the ocean, that's what we should be doing.
Joseph Carson: And then… it gathers the minerals and actually prints whatever it needs to print on its way to the destination.
Josh Lospinoso: This is how we get Skynet. Exactly. You want to talk about an AI, now you've got self-propagating species.
Joseph Carson: And we're holding back the AI for another conversation. Maybe for the audience that's a good segue into we will talk about, I call it cyber buzzwords, buzzword bingo. But we will talk about that. We will get on that topic soon.
Josh Lospinoso: Joe, I've got opinions.
Mike Gruen: So we'll have to have you back.
Josh Lospinoso: I'm inviting myself back.
Joseph Carson:
That's going to be an interesting conversation. But at that point, I think we'll wrap it up now. I think it's been very interesting. I think we've got, as you were saying, a quick summary is that at the end of the day it's all about safety, it's all about making our lives better. It's about making sure safety is the number one priority when it comes to OT. There is the risk as they basically become more connected and these systems are around for a long time, that they do become vulnerable. And it's a question of how we actually maintain somewhat of a segmentation or keeping them in some type of gap that they shouldn't be directly connected. So therefore we can measure the firmware updates, measure their uses, measure the data so they actually became safer in regards to what the potential risks are from cyber threats. So, Josh, I'll pass it over to you for final thoughts, or do you see we're going in the right direction, or we're hitting bumps in the road? What are your thoughts on the path that we're taking at the moment?
Josh Lospinoso:
I'm hopeful. I'm very hopeful. I think that we've got a situation where there are all these OT assets out there that aren't going anywhere. We, as a cybersecurity industry, need to figure out... learn the lessons from the past, how we secured IT systems that in the seventies weren't going anywhere, and how do we make these things more secure so people don't lose their lives. Because the stakes couldn't be higher, they really couldn't be. I think there's an ecosystem that's going to evolve around this problem. We've got to figure out how to secure the old stuff, and then also let's figure out how, as cybersecurity professionals, we take a deep breath, let the nihilism leave us and say how do we make future systems more secure? And then there's this challenge of, in these industries, people are very motivated by what are the profits and what's the effect on the bottom line and how do we make a compelling case to people that cyber security is really important. And then oh, by the way, are there ways that we can design systems that both give you cybersecurity and make your operations more efficient? I think I'm hopeful. I'm an optimist, but I'm hopeful we're moving in the right direction.
Joseph Carson: So would you get on a plane that has no pilot or a train that doesn't have a driver? Where are your boundaries at the moment?
Josh Lospinoso: I will say we do, I drive a semi-autonomous car and I will say... Let's put it this way, I supervise the algorithm manually, but I'm hopeful that we're heading in the right direction. Because I'll tell you, I've been in the car with some awful human drivers and there's a lot of room for improvement.
Joseph Carson: The point is that I'm not too worried about being on a plane that doesn't have a pilot, I'm just worried about the other pilots in the sky.
Josh Lospinoso: Exactly. And maybe to the point of AI, now we'll get to talk about where's the game theory of if I overwrite my car's firmware so that it takes advantage of the risk-averse nature of other cars on the road, can I zip through on the highway? The future's going to be weird.
Joseph Carson: It's going to be interesting. And Mike, any thoughts, any final comments? Would you get on a plane with no pilot?
Mike Gruen: With no pilot, no. But I mean right now most planes are mostly flown by computers. I think the one thing we didn't touch on too much though is that one of the things that's very tough to deal with is when you have humans and computers and the human can override what the computer wants to do. Because that's where safety can really breakdown and there can be a fight. My risk tolerance might be higher in this particular driving situation. I might jab on the gas and cut hard to the left, whereas the AI is like, no, stop.
Josh Lospinoso: Or you're in a Boeing 737 MAX 8 and actually the pilot's right and it's the controllers that are wrong.
Mike Gruen: Right. But no, I think my line... I'm optimistic with Josh, but I don't think that I'm ready for fully autonomous anything yet. But hopefully, we'll get there. And really happy that people like Josh and his company are in the space and that companies and people are seeing the opportunity, both not just from a hey, let's make the world a better place, but also from the financial aspect. Because I think that will be what really drives the innovation and the security and the rest of it.
Joseph Carson: Absolutely. And these industries are core to making the world a better place. Society, it's what connects people, what moves people around. And it's ultimately what eventually they get into … the likes of Star Trek and space travel and the real things that we vision. I think these are the things that if we put the right people in place with these industries, and really talented, we will create an amazing, let's say, visual future that we all want to be part of.
So that point, many thanks, Josh for being on the show again. It's awesome, and I'm really looking forward to this buzzword bingo one. So artificial intelligence.
Josh Lospinoso: Anytime guys. I love getting on the podcast with you. So anytime you want to, we'll follow up.
Joseph Carson:
Sounds great. And Mike, again, many thanks for joining me. It's awesome to get into these fun conversations. So for the audience, hopefully, it's been valuable. Hopefully, you learned a lot. Hopefully, Skynet is not so close, but hopefully, we'll be able to, let's say, embrace technology with, I will say, responsibility. That's what world we want to be, is embrace it, but with responsibility. So again, tune in every two weeks for the 401 Access Denied podcast. And thanks for listening. I look forward to having you on future shows. Awesome. Thank you.
Outro:
Learn how your team can get a free trial of Cybrary for Business by going to www.cybrary.it/business. This podcast is also brought to you by Thycotic, the leader in Privileged Access Management. To learn more, visit www.thycotic.com.
