EPISODE SUMMARY
Subscribe or listen now: 
Joseph Carson:
Hello, everyone. Welcome back to another episode of 401 Access Denied. Really excited to have another exciting couple of guests with me today. My name is Joseph Carson, chief security scientist and advisory CISO at Thycotic and based in Tallinn, Estonia. So, really excited to have another guest who's returning today, and it's always great when we have fantastic guests come back on the show. Jon, do you want to give us an intro and let new guests know who you are and a little bit about your background, please?
Jon Ramsey:
Yeah, sure. Last time was so much fun I was really excited to come back and have another conversation. Jon Ramsey, I was formerly the chief technology officer for Secureworks, a managed security service provider for 21 years. Prior to that, I was at the Computer Emergency Response Team at Carnegie Mellon University where I also did my masters work at CMU in software engineering. And then prior to that, at Siemens Corporate Research.
Joseph Carson:
Awesome. Welcome back to the show. It's a pleasure to have you back again. And we're here with a new guest in the episode, so Juan, welcome to the show. Hopefully, this is going to be an interesting and fun conversation for today, so can you give us a little bit about who you are and your background, please.
Juan Espinosa:
Sure, Joe. Thank you very much for having me in the show. Really looking forward to share some of my thoughts about cybersecurity in OT with you guys, and thank you for having me. So, Juan Espinosa, I'm a Parsons employee specialized in cybersecurity for OT systems. I've been with Parsons eight years doing a lot of work for the intelligence community protecting critical networks. Think about air conditioning systems for data centers, generators, and things like that.
Juan Espinosa:
Before that, I spent 15 years of my life building American embassies, so I traveled all over the world building the infrastructures. That was fun and I have a growing family and I decided to change and do something that required less travel, so I'm here in the Maryland area and learning every day about cyber. It's one of those things that you never become a true expert, you really learn every day. So, looking forward for this conversation, thank you very much.
Joseph Carson:
Absolutely. Welcome to the show, and I couldn't agree more with you. Cybersecurity there is no one person who knows everything. It's always a continuous learning and I've been in the industry for close to 30 years and I'm still learning. I specialize in certain areas, but I can never be an expert in everything. That's why I surround myself with amazing talented people that when I have a question, I go to them for it.
Joseph Carson:
So, for the audience, for today's show, we're going to be going into operational technology, OT. I know we've had episodes before, which some of the well and greatest performing episodes were all on operational technology. We've talked about with sciences and data, and we talked about production lines, we talked about attacks on oil rigs, and so forth. So, we've had a lot of interesting discussions, but I think it's been quite a while since we've had that OT discussion. So, I thought it was really important to bring back some extreme intelligent guests to come in and help provide us an update into where we are today.
Joseph Carson:
One of the things is that with OT, it's always advancing. It's always evolving, especially in the cybersecurity, and we've seen a lot of major incidents in the past year. So, I'd like to get... One if you can provide us current state of the union type of thing into what is operational technology so people can get an understanding what it means?
Juan Espinosa:
Sure, absolutely. So, think about the internet of things, right? A common term. Think about the convenience that you have when you forgot to close the garage door in your house and then you have the ability to close it from remote from your cell phone. Think about a security camera in your Ring bell in your door and then you can check if your kids arrived safe home from your office computer.
Juan Espinosa:
So, that convenience, that connectivity relies on a holistic network of IT and OT devices. So, think about that example in the example of the garage door. You have an OT device, which is the classic... Think about the engine, the motor that is activating the door to open. Think about the example of the camera. You have the actual camera, OT devices that are talking with a special communication protocol, but then the ability of those devices to be controlled from remote is basically leveraging the internet and an IT network behind it.
Juan Espinosa:
So, in that example, you're using the convenience of the system. However, security was never considered in the development. Security is a new thought and now, all of a sudden you have a network that is all connected and is full of entry points and plenty of vulnerabilities to exploit. Now, think about industrial installations. They were not any different. They were developed with convenience in mind, with the facility to save money, so automation was great and that was the rush of a lot of companies to automate, connect things so you will save money.
Juan Espinosa:
Think about having data and control come into your centralized station. That will save you money because you don't have to send technicians all over the country in a truck with a notepad taking the measurement of a power meter. All that is automated now. So, that convenience and that simplicity and efficiency was captured with a all-connected network. In the internet of things now so much as in industrial control systems, you have the same issue. Everything's connected, IT/OT you cannot tell the difference, so convenience came with a price. The price of cybersecurity and that is a role now, trying to protect something that was not designed with security in mind.
Joseph Carson:
Absolutely. Jon, anything that you want to add to that?
Jon Ramsey:
Yeah, I think it's a fascinating time the convergence of IT and OT and the ramifications of that convergence, both positive and negative. I mean take the power industry, for example, it's a lot cheaper to shed power into the ground instead of store it in a battery. It's cheaper to reproduce the power than to have to store it. And so, because the production and consumption side are not technically driving demand across one another, you have this problem where power companies are shedding some percentage of their power in the ground or they're trying to store it by pumping water up a hill and turning it into potential energy as an example.
Jon Ramsey:
And so, here you bring in IT, and now you can directly connect the production and the consumption together in the power industry and be able to now not have to generate or consume as many natural resources because you have more real-time decisions being made. And so, there's definitely upside, but as Juan said, that comes with incremental risk and incremental risk. The risk to me, in this case, is really when we think about security from a cyber perspective, we tend to think about confidentiality, integrity, and availability.
Jon Ramsey:
In the OT side, when a OT system doesn't have confidentiality, integrity, or availability, then what happens? Does it become a life safety issue? That's really the ramification of the influx of IT technology into systems is now it's become clearly a safety issue.
Joseph Carson:
Absolutely. When I think about it and when I've been involved in many different projects over the years and really interesting and exciting ones. I've been involved in autonomous shipping projects or really taking large, massive vessels and actually making them autonomous with live projects we've seen in Finland where we've had tugboats and we've had car ferries being completely automated regards to being able to do their jobs.
Joseph Carson:
I've also been involved into the mineral industry as well where we've had mining trucks completely. Now, not fully autonomous. They're mostly remotely operated, but again, the same technology, the same ability in order to make people be able to operate those trucks. I think really where I've seen a lot of these advancements was focused around the safety side of things. It was all about taking people out of high-risk places and putting them in safe places to operate those technologies.
Joseph Carson:
And one of the first that I ever got involved into actually goes back into the early 2000s when I got involved... I used to be responsible for an ambulance service. When I was responsible for ambulances, you'd basically have to provide ambulances to different areas and then those ambulances pick up the patients or the victims and they would transport them to the emergency room, and then basically, once they arrive, the doctors would basically perform emergency services.
Joseph Carson:
But I saw there was a major gap between the ambulances whether en route to the hospitals and basically the data that the paramedics were collecting could have been actually vital if it actually had already had visibility of the surgeons and the emergency services before they got there.
Jon Ramsey:
Yeah, brilliant.
Joseph Carson:
It was back in 2000 and we had these new defibrillators and EPGs that actually had data cables that you can actually... But the data cables, the purpose of those was to connect them directly to things like printers and fax machines in the emergency rooms. So, what I saw was an opportunity then was I took old Nokia telephones that had data cables, which were sitting on edge computing, connected it to the defibrillators, and we literally made basically the defibrillators act through that Nokia phone back into the emergency room that actually provided the live data of the patient 15 minutes before they even arrived at the hospital so that doctors could look at that data and already start analyzing and prepping the rooms before they arrived.
Joseph Carson:
And that was kind of... So, when I look at back then, but going to your point that security was not the focus. It wasn't the primary focus. The primary focus was to make it work. To make sure we're able to take data for one location and make it available to another person who makes those critical decisions and makes the prep. So, when I think about all of these areas of OT, and that was probably my very first introduction where you're taking technology that was not necessarily internet-connected, but it started having ability to connect it through data services and mobile phones and everything else. Of course, now they naturally have it embedded and available.
Joseph Carson:
So, that was kind of my first introductions, and it really got me thinking about... And later when I look at from 2000s, that was in early 2000s when we started connecting ambulances, and we think about today. To your point, Jon, it's so exciting the technology and the convergence, but at the same time as I'm excited about seeing all of this evolution, I'm also scared. I also have a scare because when I get into...
Joseph Carson:
So, just to give an example, I recently wrote a very in-depth blog, which it hasn't been released yet, but it'll be released very soon. I'm sitting here surrounded by devices that I've pulled apart and I've been connecting things to ... and I'm also sitting with my logic analyzers as well where I plug into everything and try to analyze the data. But the blog was really getting into where we look at these devices, whether it be IoT connected devices that are just providing, let's say temperature data or lighting information or traffic flow. All of them, they're designed for basically efficiency and cost, and security is not an element in any of them. It's tried to be patched later. It's tried to be added on after it's already been designed and in production.
Joseph Carson:
So, looking at that one, I'd like to get some examples of what you've seen where IoT itself and OT has really been lacking in security areas. Do you have some examples that you'd like to share with the audience where you've seen that security has been, let's say sacrificed in many cases just for the efficiency of productivity?
Juan Espinosa:
Absolutely, not only because of the sacrifice on productivity but also because... or pure obsolescence. Think about elevators in every building. That's the classic example that we always find. They're designed for a useful life for 30 years. So, picture yourself going through an assessment on cybersecurity and taking a look of an elevator that is 20 years old. So, you walk with the building owner in charge of the maintenance, and he says, "Oh, the elevator's working great. I have 10 more years of useful life. For me, it's great. I don't have any comments whatsoever with the functionality of the elevator."
Juan Espinosa:
And then you go and talk to the cybersecurity guy in charge of the cybersecurity of the building and then who will check the elevator and most likely, it will be running on an antiquated operating system. Think about XP. So, the cyber guy will say, "Hey, no way. XP is not supported anymore by Windows, I cannot deploy any security patches to it, it really represents a liability from the point of view of cybersecurity." But not only that, in addition to that, the elevator needs to be connected to the fire suppression system of the building because, in the case of an emergency, in fire, those things need to be connected and monitored from remote.
Juan Espinosa:
So, the cybersecurity guys is saying your elevator is a liability not only for the elevator itself, it's a liability for the whole building when it comes to cybersecurity. So, what do you do? Are you going to change the whole elevator when you still have 10 more years of useful life? Probably not. So, then you have the classic IT/OT dilemma, who's right and who's wrong? And that's a classic, right? Those two employees of that company, they work in different departments, they have different priorities, and they have different standards when it comes to cybersecurity.
Juan Espinosa:
So, in those cases, we sometimes recommend simple solutions that are not high-tech at all. We just simply say, "Hey, let's put a lock in the controller of the elevator and then develop a process to who has access to it that way you will satisfy, to a point, the cybersecurity concern, and you can still use the elevator for another 10 years." So, that's the challenge. There's different priorities. There's different groups that see the same issue in a different way. One is functionality, and the other one is IT cybersecurity.
Jon Ramsey:
Yeah, I think it's fascinating that just the cultural difference there is you have an IT person look at a thermostat on the wall and go, "I know there's something broken about it. We're going to have to update it." And an OT person looks at it and goes, "It's not broken, don't fix it. The temperature in here is perfect." The two approaches really culturally conflict with one another and there has to be some dialog and shared language between the IT and OT person to be able to understand exactly what... They're both valid, to understand exactly where that's coming from. The IT world, back to confidentiality, integrity, and availability, we tend to think more about confidentiality and integrity. And in the OT world, availability is king.
Joseph Carson:
Yeah, safety and availability.
Jon Ramsey:
Availability is the thing. It has to be up all the time. And so, if it's up and it's working, it's doing its job, why introduce the risk of changing and breaking it and making it unavailable now. And so, it's an interesting balance.
Joseph Carson:
Yeah, I think the dilemma has been in the last 10 to almost 15 years the dilemma is starting to increase because I think, to both your points is absolutely, I remember just for the audience, and I've probably mentioned this before, is that IT is much faster moving. We tend to recycle things every three to five years. People get new laptops, upgrade their operating systems, get new mobile phones. We patch systems almost on a monthly basis or every patch Tuesday.
Joseph Carson:
So, things in IT, they may not appear to be very fast. Three years waiting for a new phone could be extremely lifetime for many people, but in the IT, that's fast. But we look at OT and I look at the complete opposite side of the spectrum. I remember years ago getting the opportunity to go watch the satellite decommissioning process, and the person was mentioning that, "This button was designed 30 years ago, and this is the button that sets off the decommissioning process." And they hope that the design of 30 years ago, and it might even be longer because by the time they implement it and the satellite's been... I think the satellite was originally planned for 20 years and had now been orbiting for 25 and now they're going, "We hope all of this works."
Joseph Carson:
That process would set off the... To make sure that the satellite would move to what was called the decommissioning orbit where rubbish floats around, and that it had to be exactly empty in fuel by the time it got to that point. But this was all designed 30 years ago, and you think about in OT, things are meant to last for a long time. We look at, even I mentioned, mineral trucks, and we look at maritime ships, we look at airplanes. Airplanes we see planes flying that's 20, 30 years old. We see vessels even older than that.
Joseph Carson:
So, in that area, it means that technology it's always about cost production safety. The elements of the focus and priorities are very, very different from that in IT. So, that kind of gets into that basically, I think that when we have different opinions or we get two people from the different views is that their priorities and viewpoints are very, very different.
Joseph Carson:
So, that becomes a major challenge, but I think your point about the elevator scenario is that all of a sudden somebody decides, "Well, let's connect something else. Let's put a camera in that elevator and connect it to the same machine." All of a sudden, now it's not just maybe isolated, that elevator now becomes internet-connected. Same in the vessel side of things is that we might have a SCADA control system that's providing diagnostics data back to a command-and-control center, but now we need to have that diagnostics going to the vendor because the vendor owns the data.
Joseph Carson:
So, we're starting to see contractual difference as well because if you buy a ship's engine today or even a power station engine, you might own the physical device, but you don't own the data. Contracts are starting to change, especially in OT. Even with vehicles and cars, we're starting to see you might own the physical hardware, but you don't own the data that has been generated.
Joseph Carson:
What that means is you have to provide connectivity back to the manufacturer, back to the production. They might also provide services to provide services and maintenance, so you actually have to provide that data and access back to the company who's providing that. That's where we start seeing these devices which were meant to have long life cycles are now getting connected to the internet to provide that access.
Joseph Carson:
I think that's where that conflict between the IT person saying, "Now it's connected, now it's exposed. Now the risk is much greater." We have to find a balance between how do we reduce that risk? So, I'm just interested in any thoughts around those different viewpoints and those scenarios because, of course, we're starting to see more connectivity.
Joseph Carson:
I think we're seeing a lot more on the consumer side, but definitely in industry, in manufacturing, that's starting to accelerate very, very quickly, and I don't think right now we're prepared yet, from a security perspective to reduce that risk. I think that security is being added afterwards especially after incidents happen. So, just interested maybe, Juan, you have any thoughts around that.
Juan Espinosa:
Yeah, you brought a very good point, which is in OT, you highly depend on vendors because sometimes the providers of the automation and think about Honeywell, Allen Bradley, Siemens. Those guys not only use the data like you said, but they own a proprietary communication system. That's what they're selling you. They'll send you their proprietary technology.
Juan Espinosa:
You don't own that. So, because you don't own it, it's very difficult for you to provide the security associated with that particular device. You rely on the Siemens of the world to come to your facility to deploy security patches, to generate upgrades, even for maintenance, right? And it's not one or two. I mean, in OT you rely on hundreds of vendors in a single facility.
Juan Espinosa:
So, you cannot ignore that fact that you're constantly having companies coming and connecting to your network. So, you're not air-gapped. Air-gapped in OT just that concept doesn't exist because you rely on those vendors. You don't have a choice. So, instead of ignoring that fact. Instead of living in a world in which you feel that you're air-gapped like in an IT system that you might easily develop a series of firewalls, some protective boundaries, and things like that, OT is not like that. You depend and you rely on those vendors.
Juan Espinosa:
So, the only choice is for you to acknowledge that fact and develop your security around integrating those vendors into your processes. Think about it, and you mentioned the internet. In theory, a critical network shouldn't be connected to the internet and then you see cases in which it happens. But let's assume for one second that you're not connected to the internet, that your vendor actually physically comes to your facility with their laptops, and they want to deploy maintenance and patches to their assets.
Juan Espinosa:
You need to integrate that processing to your own security protocols. Maybe you have a mechanism in which you make sure that that laptop that they're connected is scanned for viruses. It's a laptop that has control for only that piece of equipment and so forth. You develop your processes around the fact that you have people connecting. Maybe you close every unused port in your facility. You only let those guys use a single port and you make sure that the rest of the ports in your endpoints are closed.
Juan Espinosa:
Maybe you develop training for those guys. You make sure that they understand that they cannot connect personal devices when they're doing that exercise. Maybe you are sure that they don't have any email or instant messaging in the computer that they're connecting. I mean, there's things that you can do about it if you think about this issue holistically. You need to work with the vendors and understand that an OT network is very different from an IT network and your operations need to understand that.
Juan Espinosa:
The challenge though is typically, security policies are issued by the IT department. So, then you're trying to impose security, and the OT guys just struggle trying to understand those requirements. So, your policy needs to include that acknowledgment that you depend on vendors, that you're different, and then that's the only way to secure it.
Joseph Carson:
Just one other thought here before, Jon, just while I'm on this point then I'll move over to you is that when I think about it, sometimes we've taken this approach in the past where segmentation. I've always taken an approach that we have production servers, you've got production desktops and clients and maybe mobile devices. You have BYOD, you have UAT, you have backup.
Joseph Carson:
I was always in the kind of methodology segmentation that they should ever be on the same LAN, and they always have strong access controls between all of them. I kind of get in that same thought process that both IT and OT need to have that same, we need to have the segmentation. We need to have strong... Because what it gets to is that, to your point, it's hard to put security into that OT environment, into the actually processes into the systems because they are proprietary. They might be legacy. They might be old.
Joseph Carson:
Does that mean that we have to think about putting security around it and minimizing that way? To your point, the ports, how people access because when you have people coming in, that to me is already... That's no longer air-gapped. When you have people go, that's basically no longer an air-gapped system. Jon, maybe you have some thoughts along that that you want to follow up on from Juan's point before.
Jon Ramsey:
No, maybe you want to jump into some models or-
Juan Espinosa:
Yeah, you're right, absolutely right. It is a challenge trying to bolt on security into an OT system. So, there is a concept that is being implemented nowadays, which is the concept of defense in depth. It's a series of layers that the final goal is to protect the endpoints. Your most important assets and that layered approach is exactly what you're saying, so you have a combination of methods.
Juan Espinosa:
One method is clearly standard operating procedures. Don't connect USB ports. Don't use personal email in these networks and so forth, so there's a little bit of SOP involved with the operation of that network. But there's other concepts that are a little bit more technical, to your point of separation. So, you should try to design your networks in a way that the operational function, the automation is separated from the IT, and believe it or not, that's not the case nowadays.
Juan Espinosa:
It's a lot cheaper to exercise control and data acquisition from the IT devices because it's a lot cheaper instead of trying to design your own proprietary protocol. So, nowadays technologies are highly reliable on the network piece of it. Think about the IT piece of the ICS or OT network, and that's not good for security because if the network fails, your function fails. So, you need to find a way to separate those two things and keep the function separated from the network. That's another concept.
Juan Espinosa:
There are other concepts that had to do with classic IT separation of networks. So, a combination of firewalls, maybe data diodes, there's a lot of good technology out there. I think the challenge is that it's not totally integrated. There's individual vendors that sell individual tools for certain purposes, but there's still a clear silo between the IT and OT groups. So, you don't have a network that is all protected with a holistic approach. It's being patched if you will.
Juan Espinosa:
So, maybe there's sections of the network that are newer, very well isolated, firewalled, good SOPs and then all of a sudden, you have, like you said, a brand-new system. Maybe security cameras that are included in that particular facility and they wanted to connect that as well. So, all of a sudden, you have a new system that has not been designed that way, and then you almost need to start from zero in developing the firewalls, developing the SOPs for the vendors that take care of that section of the network.
Juan Espinosa:
So, it's a dynamic process. It never ends. It's like a cycle. You constantly need to be reacting and not even thinking about vulnerabilities because they change also every day. It's super dynamic. Every day there's new vulnerabilities. It's not like you patch once, you design your network and then you can go. No, I mean you constantly need to be coming back to that zero point in which you baseline again and then you keep tracking for changes.
Joseph Carson:
Because if you want to update something, it's production line. You're not updating one component. You have to stop everything in the chain in order to replace one thing. I remember when they were doing some maintenance in the power station, and they had to ship the entire engine off just to change little components that had no impact to the engine at all. It's just that that whole thing is a system. It's a system in its entirety. Not just one vendor's machine or one SCADA controller, one PLC, or one cable. Basically, that is seen as an entire system end-to-end. So, any maintenance on it, it's the system that's impacted, not just the one vendor component.
Juan Espinosa:
Absolutely.
Jon Ramsey:
I think that there is an opportunity that exists that if we're going to get in front of it in the long game. This is a long game thing because these systems are around forever, and there are a lot of systems that are already employed. As Juan has taught me so much about this space, they're in the operate and maintain phase of their life cycle. But there's a life cycle in front of that, which is the design and build life cycle and I think that as we are in the long game, I think now that we know so much more about how to secure technology, in the end as an IT practitioner, I like at it I go, "Well, OT, really? It's got hardware, it's got software, it's got data. How different is it really?" It's different. It's really different.
Joseph Carson:
Very different.
Jon Ramsey:
And so, what's important is in the design and build stage, how do we build it or design by security. Design it in in the beginning that enables options, from a security perspective, to do things when we're operating and maintaining it because we know we're going to need to do things. So, how do we make sure as a design and an architecture that we build a system that we can operate and maintain? And when we make changes, the changes don't introduce more risk than they mitigate.
Jon Ramsey:
And so, I think there's a lot of work that needs to be done and I think there are companies like Parsons, for example, that have a philosophy around safety and building safety and security and design in in the early stages. So that when they also operate and maintain those networks, they can do it in a way that has more rigor and is more natural than sort of ad hoc, bolted on kind of approaches.
Joseph Carson:
Just to give you a... I completely... I can give you a great example into how all of this plays out. I really love the OWASP top 10 for IoT. That was a great... Just showing people what they need to be thinking about, especially getting into that security by design, security upfront. Years ago, doing some maritime engagements, specifically maritime management company that when doing the penetration tests, we found that they'd actually deployed in their office space smart light bulbs.
Joseph Carson:
And so, the smart light bulbs, basically, what we were able to do was basically identify that it was using weak protocols, and then basically being able to connect. We actually had a Raspberry Pi pretending it was a light bulb. All of a sudden, they had hundreds of light bulbs deployed and never recognized this additional light bulb that appeared in the network, and then being able to use that to gather information, laterally move, and then move across the network.
Joseph Carson:
Ultimately what it got down to, and we reported the vulnerability back to the vendor and the vendor came back and said, "Right. Okay, here's a patch you need to update the light bulbs." When we looked at the process for that, the process for that was to unplug the light bulb. It was a micro USB. You had to plug it in, connect to the laptop, flash the firmware, take it out, replug the light bulb back in again. The company that had deployed hundreds of those things, it just became inefficient way of doing patch updates. That you had to physically go and take them all out, patch them, do redeployment again.
Joseph Carson:
That's where we started seeing... And also, we started seeing it at the time was the smart hubs. I think this was really a great starting point where organizations and vendors who provide these devices started thinking about patch updates. Started thinking about secure connectivity, started thinking about better protocols. So, this was kind of where we started seeing that introduction where now, actually, the hub becomes the direct communication. The light bulb's not connected directly to the Wi-Fi network. It's connected to the hub and the hub to the network. So, therefore it can able to manage an inventory and it can also automatically deploy the patches.
Joseph Carson:
So, those things, unfortunately, it's always after. It's after you find the vulnerability. So, a question is how can we get ahead of this? How can we get the vendors to really start thinking about it before when they get into the design process? Because when I get in those scenarios, it's always too late and to change it is time-consuming. That vendor had to... They got rid of the light bulbs. They just could not because it meant in between that time they were vulnerable and to replace all the light bulbs and flash the firmware was very costly. It was more costly than the light bulbs to do that process. So, just interested in how can we get manufacturers to think about this beforehand?
Juan Espinosa:
There's a lot of strategies that I think you can take to minimize that risk and I like a lot the concept of security by design. So, it is in that stage in the design that you have the most opportunities to do things right because then when you do them later, it's going to be very difficult or super expensive. So, I think the vendors are in a race for cheap connected, automated, and I don't think you can fight that battle because everybody is trying to offer you the simplest, more efficient, cheaper way to connect.
Juan Espinosa:
So, I don't think that the battle that we need to fight as cybersecurity professionals is with the vendors. I think it's with our own designs. So, there is a concept especially when it comes to OT, and Jon mentioned this really well, which is the concept of availability of your network or your system. It needs to be available 24/7 for critical operations. Think about nuclear facilities, think about Department of Defense, think about critical networks.
Juan Espinosa:
You need to build resiliency into your design from the beginning. What do I mean by that? You need a system that is resilient to the attack. Not only you try to avoid the attack, but you need to be resilient in the case that an attack actually happens. So, you need to have, for instance, redundancy in your networks. Special components that are critical need to be designed with certain level of redundancy. An example is the redundancy in the supply of power for instance. If you use commercial power, you need to have the ability to get into the automatic transfer and then have the generators in your facility provide you with that power, right? So, that's a simple concept of resiliency.
Juan Espinosa:
However, some of the assets in your infrastructure are not designed with that. So, one of the things that we look for in our assessments is we take a look of say POCs and we look behind the box and then you see that they have a single source of power. So, they have one single way to get power into that device. Even if you have the generator outside ready to give you power, that device is only getting the commercial power. So, it's a simple solution, just buy the devices with dual sources of power and then you connect one to the commercial power and the other one to the generators and UPSs.
Juan Espinosa:
So, that's an opportunity that you have only in the design phase because when you have thousands of POCs deployed in an OT installation and then you're making an assessment, it's very difficult to retrofit all that infrastructure when it's already operational. I mean, the discovery and assessment of the vulnerability might be easy, but the opportunity is passed.
Juan Espinosa:
So, that's a big call to say, "Hey, let's design these systems from the beginning." It wasn't expensive to buy the box with redundant power. It probably would have been, what? 5% more expensive, probably less. But it was never thought at the beginning, let's put it that way. Security was never thought. It was just, "Let's automate, let's move quickly, let's make it efficient, let's integrate."
Juan Espinosa:
Are we redundant? Are we resilient? Do we have dual sources of power and strategies that are not new? This has been in the market for many years. It just need to be included into the design. Only the afterthought of security is just not efficient anymore.
Joseph Carson:
And Jon, just kind of get your expertise. One thing we've seen a lot of major attacks and incidents in the past year on OT infrastructure and your background in CERT and incident response, is it very different in a OT environment than it is in a traditional IT? Is there differences there? Maybe there's some...
Jon Ramsey:
From an incident-
Joseph Carson:
Yeah, from incident response.
Jon Ramsey:
Yeah, I think from an incident response perspective, IT and OT is very, very different. Generally, in the OT side of things, you have one button to push, and you disconnect and shut everything down. The incidents that sort of happen historically and in recent history, most of the incidents have specifically still been contained to the IT side or the corporate side but have touched things that were closer to the OT side.
Jon Ramsey:
And so, usually, as a means to mitigate any kind of further exposure, you tend to find proactive take the OT systems or disconnect the OT systems for a period of time. That's mostly on the industrial side. Now, ransomware, and on the healthcare side, has still also largely been in IT-based systems but has had an impact on standard of care and healthcare. And so, time is not on your side when you have a standard of care issue. Time isn't on your side when you're in incident anyway, but when there's life on the line, you tend to have to think about the problem a little bit different.
Jon Ramsey:
For me, whenever I'm an incident commander and walk into incident IT or OT, I always ask the question is what is the priority here? Is the priority to get the systems back online, which would mean you have to get to eviction day as soon as possible, or is the priority to catch the infiltrator? If the priority is to catch the infiltrator, you might want to leave them in the environment for a while and continue to-
Joseph Carson:
Watch and learn.
Jon Ramsey:
And watch and learn. So, understanding the priority is really, really critical, and there's a different calculus on the OT side than there is on the IT side.
Joseph Carson:
Yeah, it makes me think of some of the incidents we've seen in the past year where we've really started seeing that kinetic and human impact. We're seeing hospitals being victims of ransomware and you've got patients that's en route to that hospital and then have to be rerouted and time is critical. You're seeing patients dying because of those reroutings.
Joseph Carson:
So, absolutely, it becomes for me it's always life and safety is always my top two priorities and that's where you make those decisions. I get that a lot when I think about... I've had some incidents in the ambulance service where you have those situations. The system, my SLA in the ambulance service was 23 minutes. For the rest of my life, I'll never forget my SLA for the ambulance service because after 23 minutes from that call into the emergency room, if the ambulance didn't get there in 22 minutes, the probability and likeliness of that person dying was very high, and that was the SLA was a life and death. So, we knew that the minute the call came in I had to do everything in my power in order to make sure that the ambulance got to that victim within 23 minutes because that was the difference between life and death.
Joseph Carson:
Yeah, I mean I've had times where we were doing... It was back in the good old Y2K days where we were updating systems from Y2K and moving systems over to new phases and generators and we had a scenario where basically the systems were offline, and we had passed that 23 minutes. That was probably the worst night of my life is when we knew that those systems that you had to get them back up as quickly as possible because it was a life-or-death scenario and that's why it was a priority.
Jon Ramsey:
For sure, for sure.
Juan Espinosa:
The other thing, Joe, the other thing to incident response that is different is that IT, typically, the higher priority is confidentiality, so thinking about the data. I mean if your network was compromised and you're getting asked for money for ransomware to recover the data because maybe you're concerned that they have financial data for your corporation or maybe you have PPI information that is relevant for you. So, you want to recover the data. You're paying for the data back. So, when the data is back, the data is back immediately. You have the data back in your servers.
Juan Espinosa:
In the OT though, when they interrupt the service, those systems were not designed to be rebooted like that. So, they might give you power back, they may give you the control back, it might take you weeks to recover. You need to reimage all the computers again. So, think about Colonial Pipeline. It took them a while to recover. Even when they recover the control, it might still take weeks to take the operation back to the normal state.
Juan Espinosa:
So, the impact is completely different and if you think about national security, the risk is immense. You need to protect all those critical systems 24/7 and you're not concerned about the data so much. The data that basically you're collecting through those networks is maybe temperature data, valve pressure, things like that. But if you lose control, recovering the control might take some time and that'd be a significant impact.
Joseph Carson:
Absolutely.
Jon Ramsey:
Yeah, certainly on the-
Joseph Carson:
You also have damage as well. You can also have physical damage to the systems too that might need to be replaced.
Juan Espinosa:
Absolutely.
Jon Ramsey:
I mean, and the IT side absolutely. "Do we have to disclose?" Is one of the first questions from a response perspective and there isn't that same calculus on the OT side.
Joseph Carson:
Yeah because-
Jon Ramsey:
I think one other important thing to think about is... And we have this problem... Two quick thoughts. We have this problem on the IT side as well, which is I don't think anyone understands the overall systemic risk like what the dependencies of the piece parts are and what a failure in one piece means to the implication of another piece. I mean, we saw that in the financial issues where no one knew that credit default swaps were going to cause a cascading set of failures. And so, understanding now that the OT systems and IT systems have dependencies across one another and understanding the systemic risk I think is critically important.
Jon Ramsey:
The other thing about your earlier question, getting vendors to be able to design security in or produce systems that are secure. In the end, it has to come down to some kind of business dollar motivation. So, I studied software engineering and learned that you can mathematically prove the completeness of a program. The problem is the cost to prove the completeness of a program is expensive. It takes a long time to do and a lot of computational cycles and not a lot of people can do it.
Jon Ramsey:
So, there has to be a business justification for a company who, if they wanted to build a mathematically provable secure system, that there has to be some upside for taking that time. And so, really, to me, ultimately, what that will come down to is it will have to be market-driven in some way and the market will have to demand or have companies compete on their systems from a security perspective and to some degree even be willing to pay more to get a system that's designed securely.
Jon Ramsey:
There might be long-term economics there to the consumers of the system where maintenance is cheaper, but the cost upfront is a little bit higher. And so, it has to come down to the economics at some point.
Joseph Carson:
It can also be compliance and regulation. It can also impede it as well that certain organizations and industries have no option but to do it. So, we're seeing, of course, executive orders coming out from the US government administration that really is pushing in that direction that you will have to do... You can no longer ignore sec, especially in OT.
Joseph Carson:
My concern is that a lot of these OT companies used to be much more isolated vendor supplies. Now they're dependent much more on IT side of the technology as well, such as DNS, which we've seen major outages recently as well. You can have those cascading impacts where you might have one company who's really heavily reliant. Even though they might be a small vendor, but they might be heavily reliant on the entire system and process.
Joseph Carson:
So, I think there's a lot of work to be done. I'm excited seeing this evolution because it's definitely going to benefit society in entirety. It makes us a more connected, it makes our lives much more benefited from all of these things that have been automated. But at the same time, I do get worried at the security risks, especially dealing with a lot of them firsthand and responding to them like yourselves. We get to see sometimes in the background what's happening and some of the lack of security implementation or thought process.
Joseph Carson:
Just kind of getting into summaries and final thoughts, where do you see this going in the future? What do you think the next steps would be? To your point, Jon, does it take that economic factor or compliance? Who should be initiating this discussion going forward? Who should be accountable and responsible for making this happen?
Juan Espinosa:
So, Joe, that's kind of the million-dollar question because right now the reality it is completely siloed. You have the OT professionals in charge of running the critical networks and sometimes even owning them. So, they are the ones sometimes paying for their own infrastructure, and then you have the IT cybersecurity savvy personnel defining on the sign-in policies. And then policies apply across the board to any networks in those organizations.
Juan Espinosa:
So, that's the challenge, still very siloed, so I think unless there's an acknowledgment that the solution is a holistic approach, it's an IT/OT integrated solution, the other services are still going to enjoy the benefit that they have today because the network is all connected. So, they are completely at an advantage when they try to attack a network that has multiple vulnerabilities available for them. And the group defending the networks are completely divided in their priorities.
Juan Espinosa:
So, there needs to be a holistic approach. I think that there's a lot of better understanding right now what needs to be done. There's great technologies available in the industry that you can use to protect your networks. So, it's not necessarily a lack of technology. It's more a lack of integration. I think an integration between OT and IT professional, they need to define standards that are compatible and useful. Policies need to be thinking about beyond compliance in my opinion. Compliance is just checking the box if you will, and sometimes that doesn't mean that you're secure. Think about going to the doctor on a yearly basis for your physical. That's a compliance check the box. It doesn't mean you're going to be healthy.
Joseph Carson:
It doesn't make you healthy.
Juan Espinosa:
Yeah, it doesn't make you healthy per se. You need to eat well, you need to exercise, you need to do all the things. So, that's the difference between compliance and security. So, it's almost like a cultural shift, but I see it coming. I see these podcasts, I see a lot of technologies coming, those things that are improving, we just need to keep pushing on the integration of IT/OT in my opinion.
Joseph Carson:
Absolutely. I agree to your point, compliance, and regulations, it's a measurement in time. It doesn't necessarily mean you're better or worse. Jon, you were saying?
Jon Ramsey:
Yeah. No, and I think from a corporate responsibility perspective there's lots of sort of... There's standard of care, there's right to exist, other types of models used in different areas that I think we need to apply in this space. I think a corporation or a business who has the responsibility of operating and maintaining and designing and building and procuring these kinds of OT/IT capabilities need to put together, in my opinion, some mechanism where you can bridge the divide between safety and availability and confidentiality and availability.
Jon Ramsey:
So, you need an IT person, you need an OT person, you need a procurement person who make sure and drive that risk conversation when procuring things, and I think you need a risk person. A risk person who can think about this represent this in a way that's a systemic risk to the organization and both of the environments coming together. And also help facilitate the conversation of how to mitigate the risk between both environments and make that conversation and the results of that conversation effectively a board-level conversation maybe at the audit level.
Jon Ramsey:
But certainly, this needs to be talked about because there's going to be money needed one way or another and there's going to be talent needed and it's not going to fix itself. And so, getting proactive by putting a group together that can show some corporate governance over this is where I think it needs to start.
Joseph Carson:
Absolutely. Yeah, we need to turn on the silos. I think just like the government's put together the joint task force for ransomware, I think we really need some type of task force that's for OT and IoT.
Jon Ramsey:
That's brilliant.
Joseph Carson:
I think that's really where we need to start really considering. The great thing is we're having these discussions. We're bringing them up to the surface, and I think that's where a lot of it's happened over the last few years, and we've seen major incidences occur and therefore it's got a lot of focus. But I think we really need to start bringing down those silos. We need to have joint cooperation, and to both your points, we definitely need IT and OT and risk and compliance and the board all working together to find what's the priorities and goals and what's the way forward?
Joseph Carson:
So, definitely valuable points, and hopefully, anyone who's... We do have a lot of people listen to these podcasts, so hopefully, one of those listening will help maybe trigger some of those initiatives because I think we need to go beyond executive orders and really get into really joint collaboration, joint working together, think tanks that's really looking at this from a future perspective because my concern is that we'll always wait for the next incident to happen and then we tick that as the move forward or the step forward. We want to be more proactive here and I think that's what our discussions are all about.
Joseph Carson:
So, both Jon, Juan, it's been a pleasure having you on the show. For the audience, I really hope this has been exciting and interesting. I think this is a space that we might have more discussions on in the near future because this is definitely one of the biggest areas in security where we're starting to see more impact as we start seeing there's convergence, more devices connected. So, any final thoughts? Any final words, Jon, Juan? Anything you would like to sum up?
Juan Espinosa:
Yeah, I think that the future holds a lot of good things. I think that universities are listening. I think there's a wave of new cybersecurity professionals. So, when I said at the beginning that security was an afterthought, maybe not anymore. Maybe the new generations already are aware of the risks. You don't need to convince company owners anymore that they have a risk, so I'm optimistic of the future. There's a wave of new education, new resources that none of us really had before 20 years ago. So, I think it will be much better, and I must say unpredictable in the future.
Joseph Carson:
Absolutely well said. Jon?
Jon Ramsey:
Nothing to add.
Joseph Carson:
Nothing to add.
Jon Ramsey:
Well said, Juan. Nothing to add.
Joseph Carson:
Absolutely. I doubt we can add any more to that. I think that was a solid ending point, but absolutely, pleasure having you both on the show. Hopefully, we'll be able to welcome you back again in the near future. For the audience, this is 401 Access Denied, one of your leading podcasts, award-winning podcast that really helps bring very, very educational, valuable topics to you so you can actually stay up to date, stay in the know, and get educated as much as we possibly can.
Joseph Carson:
Tune in every two weeks, make sure you subscribe so you can get continuously updated, and look forward to seeing our guests on future shows. It's been a pleasure. Thank you and take care.
