OT Security: Scientific Sensors

Mike Gruen:

You're listening to the 401 Access Denied Podcast. I'm Mike Gruen, VP of Engineering and CISO at Cybrary. Please join me and my co-host Joseph Carson, chief security scientist at Thycotic as we discuss the latest news and attempt to make cyber security accessible, usable and fun. Be sure to check back every two weeks for new episodes. Hey, welcome back to another episode of 401 Access Denied. Today we'll be talking to Steve Jacobs about OT, operational technology and scientific sensors. I'm your co-host, Mike Gruen, VP of Engineering and CISO here at Cybrary and once again joined by Joe from Thycotic.

Joseph Carson:

Hi everyone, Joe Carson here again and welcome back to another episode of Access Denied. I'm your chief security scientist from Thycotic and excited to get introduced with Steve. Steve, you want to give us a bit of introduction about who you are and what you do.

Steve Jacobs:

Hi.  My name is Steve Jacobs. I work on a large scale ecological science program called Neon. We have a website at neonscience org. We collect ecological instrumentation and observational data as well as remote sensing data at 81 field sites across the continent and United States, Puerto Rico, Alaska and Hawaii. That culminates about five billion sensor readings a day. We collect about 400 to 600 terabytes of remote sensing data a year as well.

Joseph Carson:

So not a lot of data?

Steve Jacobs:

No. Not much. When I say ecological sensors, everything from meteorological sensors like air temperature, humidity, and barometric pressure to things like soil water content, salinity, pH stream flow, we have these Eddy flux measurements that are really crazy that measure the soil breathing with the ecosystem. We get things from once every 10 seconds to 40 times a second, depending on the type of sensor. All kinds of crazy stuff.

Joseph Carson:

Well, what’s the use cases? How will hat data be used often? Is it for agricultural reasons? Is it for weather? Is it for transportation, traffic, logistics? What's the main use cases there?

Steve Jacobs:

Neon was one of the Grand Challenge, things that were formed by NSF, the idea is that we just collect ecological data distributed to folks during research. We're prohibited from doing ecological forecasting or anything like that. The idea is to use that information to validate or build models for climate change. That's why there's such a wide variety of information that we record. I didn't talk much about the observational stuff, but they use things like track mosquitoes, send them off to labs for DNA pathology analysis, and all kinds of really crazy stuff, trying to provide one place where you can get data from multiple different areas of ecology, and then use that as the basis for science.

Joseph Carson:

Okay. And do you do users at all your own sensors or do you actually bring in third parties, because I know here, I'm based in Tallinn, Estonia and there is a company that's based here, which is planet Io. They did a similar type of thing where they created a platform for basically anyone and for example, in maritime and marine so that they can actually take their own sensors, and then upload it into their, let's say, algorithms for basically statistic goals and analysis. Is it all your own sensors or do you actually bring in third parties and others and correlate it together?

Steve Jacobs:

I think it depends on how you define sensor. We have our own data logging platform. We don't use Campbell loggers and things like that, not that they wouldn't work, they would work fine, it's just that we have 10,000 of them. And they don't auto configure and they don't report back and that sort of stuff. For the most part, the probes and the things that are used to do the sensing are not things that we design. We use a steering committee that figures out what the most appropriate use is and what the scientific protocol is that needs to be done and tolerances and things like that.

Some things like most of our temperature measurements are done through the use of Platinum resistive thermometers. And they provide a resistance measurement that's calibrated and then those calibration coefficients are used to turn it into temperature, but we don't manufacture the PRT, we just make the cable harnesses and things that attach it. Other things like we have this... I don’t know if it was ring down cavity spectrometer from a company called Picarro, and that thing does all its own stuff. But we interface with the protocol it provides to pull the data back and there's no data logger involved just talks over Ethernet, set the device itself.

Just really depends on what it is. We're not going to go make a ring down capital spectrometer, I assure you.

Joseph Carson:

You don't have any third parties that are putting... Like all those sensors, you guys basically own that stuff, right? You're not-

Steve Jacobs:

No.

Joseph Carson:

...getting... Interesting.

Steve Jacobs:

There's talk of because we have these sites, our networking team has it pretty rough and they're basically an ISP for customers in the worst locations. It's not like we put one of these things in the middle of the city, where people can go. There's one in the middle Yellowstone, there’s one on the northern slope of Alaska up in Kenai, where there's only one satellite provider that can hit it, and it's a two-meter dish. When it snows, we don't have connectivity for six weeks. Because we have these locations, and there's connectivity available, there's talk of using it as an assignable asset for other programs that want to do science at these locations.

Either us just providing pure transit, or providing some sort of interface back into our system for collection or distribution, it just depends. Hasn't really been sorted out yet. We have our first co-located experiment coming online or being worked out right now, but nothing's been formalized.

Joseph Carson:

From the sensors side of things, what types of connectivity, because I think really, when we get into, we're looking from a security perspective, and we look at all these devices, one thing I've been used to in my past work is, when I'm working on... I worked in a lot of maritime, so we had a lot of, let's say, buoys that were actually pulling back sea data. We had a lot of shipping containers that would tell you the temperature and the moisture in the container so that you know if food was going off or cold or whatever.  We can look from a shipping perspective as well and also from a mining perspective is mostly for coordinates and data about maintenance and performance of the machines and trucks themselves.

Seeing a lot of different kind of data coming back and for me one thing is all about when you talk about different sensors and edge devices and different areas where the... Basically just picking up one piece of data and then communicating it. What's the medium that you're using or seeing? Is it over things like Bluetooth? Is it over Wi-Fi? Is it using, let's say, mobile carriers like 4G or 5G? What type of standards are you keeping to them? What are you seeing as being used?

Steve Jacobs:

For the most part, the design that was built for Neon predates me, but there was a desire for providing power to a lot of these types of sensors like that PRT needs voltage. The data loggers themselves are all built to run off POE. The whole entire site is basically POE powered and then the data loggers do DC to DC conversion for different voltages. 24 volt, five volt, 3.3 volt depending on what type of things are attached. That's most of the instrumentation.

There are some things like you mentioned buoys, we get a buoy, I forget the name of the manufacturer, but they have their own design that has pluggable modules for doing different water quality measurements and things and that has a wireless radio that comes back to one of the data loggers we have on the shore. I believe that is LoRa or something similar to that, it's not Bluetooth, but it's some industrial protocol that they use basically a long a serial port to pull the data back.

Joseph Carson:

Okay. From the long cable.

Steve Jacobs:

…there isn’t enough cable.

Joseph Carson:

Right. Radio.

Steve Jacobs:

Yeah, right. The problems they have with that instrumentation tend to be high flow events and things that tend to be more frequent now than they used to be. Stuff washes away, or they need to do something to moar it. Then these things tend to be in national parks and places where you can't just dig and put a big concrete buoy somewhere, so you have to get agreement about what can be done.

Joseph Carson:

All of this data has been generated and the frequency is quite impressive, because the frequency of the communications I was working was much less frequent, because of power consumption. The more frequently you communicate, the more power you used and these are very power sensitive devices because they're meant to be very low power consumption. I think that's one of the things I'm seeing the advantages of at least 5G coming is 5G definitely has powered advantages.

Steve Jacobs:

It doesn't help me though, 5G is not where these things are.

Joseph Carson:

Right. What type of security things are put in place? Is there any security at all for those devices? And when we're talking about a lot of these, if you're talking about serial cable or we're talking about Bluetooth or wireless, a lot of those I tend to find in the environments I've worked in, is that they tend to have no security or no encryption, because just the loss in bandwidth is significant when you put in place. Then what happens is it means you need a much more powerful device in the end to be able to manage that. Again, that means more power consumption. What types of security is put in place in those devices that you're using?

Steve Jacobs:

It's basically what you're saying in terms of folks weren't thinking about security when this stuff was engineered, to the point where a lot of the instrumentation is just sitting out there, you can plug into it. It's not even in a box. They were very concerned about environmentals for this stuff to make sure that it could work up in negative 40 in Alaska, in Arizona, and that sort of places. Magnesium or aluminum casings, and very high temperature ratings and that sort of stuff. I think part of it just comes from a maturity of understanding what people mean by security.

If you went to someone at my program, five years ago, and you said, "What are you doing for security?" They'd say, "What do you mean? Who cares if people steal the data, that's public data. Our job is to distribute this data to everybody, so if someone comes and gets it, what's the big deal?" But if you think about it in terms of the CIA triad, we don't have a high confidentiality requirement. The data that we retrieve, it's going to be publicly available anyway, there's some quality measurement and aggregations that are done for time periods, and that sort of stuff.

Some of them has some more complicated calculations that have to be done for derived data, but there's not really much of a confidentiality requirement. But when you look at integrity, and availability, those are pretty important. I think availability has been fairly well handled in the design, that we don't care so much if the data is late, but we do care that we have to be able to process it if it comes late, which presents us with a different design and different concerns than a typical like IoT type thing. My thermostat in my house, reports back the temperature in my house to ecobee, but if it doesn't do it for three days, and the fourth day, it updates ecobee doesn't care about the last three days. Not really.

Joseph Carson:

So time is important?

Steve Jacobs:

Right. We care about keeping all that data.  There's buffering at every level of the design, data, loggers buffer, in case they get disconnected from this computer, we have running each side, those things are designed to buffer for at least 30 days, and then that sends the data back. All the availability pieces are done pretty well. Integrity, though, I think is something that's just not really adequately thought about, and it's a difficult problem for sensor networks like this, because I think fundamentally, it all boils back to provisioning issues. You can handle the cryptography overhead and that sort of stuff, it's not really that bad.

Especially when you start talking about using something like Protocol Buffers or Avro or some means of bulking and batching and compressing data for something like Kafka or something like that. There's ways to get the bandwidth down. But if you're going to do any keying or signature, outside of just a CRC or hash, you have to have some way of providing a route of trust in manufacturing, that you can leverage for a bootstrapping and that's not really a solved problem. Anywhere. Ignore the sensor space, for instance. It's very similar to the messaging space.

If you think about you sending messages to other people and Apple, they see their phones with the route of trust and there's a way they can drive keying from it, and they use that for iMessage. But if you're another messaging provider on top of that, that doesn't work, so you need to either leverage those mechanisms to store secrets or end up driving trust back to a phone number or something for bootstrapping. For us, we would have to inject some key material or public key or enrollment key or however you want to look at it, in order to chain that whole process off and it's a complicated question about how best to do that.

Mike Gruen:

And I think it also comes into what's... Data integrity is the important part and then it's also a question of what's the risk or what's the... I imagine, at the individual sensor level, an attack against a specific one sensor isn't messed up. You guys are collecting so much data. Maybe I'm wrong, but what would be the scale of the attack or what would be the... I think you have to think it through from that perspective. What would be the point... How much would they actually have to compromise in order to have any significant impact? I imagine most of the algorithms that are being used on the data coming back from the sensors are imperfect.

Steve Jacobs:

Sure.

Mike Gruen:

You probably already have things built into throw out like this is an outlier. How many would you actually have to compromise? How much data would you actually have to compromise in order to compromise the system?

Steve Jacobs:

I think you're hinting at it here, it comes down to threat modeling, what's the actual threat. I think for us as a program, ransomware is a far higher threat than somebody modifying sensor data coming back into the observatory. The motivation is just not there.

Mike Gruen:

If I'm Exxon Mobil, and I don't want people to believe in climate change, sorry.

Steve Jacobs:

That would be difficult. It would be difficult for you to figure out exactly what you would need to modify, because we don't do forecasting on that data. It's not really well defined as a science yet. We're just collecting information. You would have to have the foresight to know what models would be built off of this data to then know what you would want to modify to get that desired outcome.

Mike Gruen:

I would just replay the data from 10 years ago, so it looks like everything-

Mike Gruen:

…I know.

Joseph Carson:

Going back to Mike's point as well is data poisoning. It's from an integrity perspective, that's one of the biggest risks just that... And Mike's going to, is that, how much data poison do you need to do for the rest of the data to become invalidated? Or that the accuracy of that data becomes seriously impacted? You're looking at, if you only need... It's like getting into, let's say, threat feeds. In security, we use a lot of our risks based on the threat feeds coming in, and how much of that threat feed do you need to poison in order for you get too much false positives that creates distrust in the result itself?

Mike, we discussed the same around the election side of things, how much distrust you do need to put in the system in order for people to lose the trust in the system itself?

Steve Jacobs:

Right. I think that, to some extent, when you talk about information security, a well secured system is also a well running system in many regards and you can use cryptography to improve, remove errors from your system. If there's no authentication required at all, for a sensor to communicate, how do you know that it's even in the right place? That it's the right type of sensor?

Joseph Carson:

On the same?

Steve Jacobs:

Yeah. The same thing. Not having any mechanism to do that leads to errors and those errors lead to decreased availability. There's still a need for those sorts of mechanisms just to ensure the quality of what's being collected. The question is, where's the line?  Where's it appropriate? In my case, this is a program funded by the NSF, I have to justify every piece of the budget, and there's always tons of work to be done. Is there money available to go build these new mechanisms or do we want to concentrate this other problem that we've got? That's pressing. Its triage? Right?

Mike Gruen:

Right. That's what security is. It's always that threat modeling. It's always that trade off. If I had unlimited resources and unlimited money and unlimited time, obviously, I would secure things well beyond whatever. But I have to look at what's the data that I have? What's the risk if this data were-

Steve Jacobs:

… build-in?

Mike Gruen:

Right. Exactly. Whether it's corrupted data or public... In your case, you have the advantage of it's all publicly accessible data anyway, so that makes-

Joseph Carson:

… much easier.

Joseph Carson:

The two biggest risks are availability, is there a ransom or threat? Is there is a true risk of not making that data available, even if it's publicly available, if the service is not there, then that's a risk. Then integrity creates distrust in the accuracy of the information. Since it has already gone public, then it's about encryption, confidentiality, as you said, doesn't make sense. Now, I get into one of the things when I'm always looking at these edge devices and sensors, and they're communicating in. And it reminds me of one that I worked on this...

I'm getting old these days. Is now six years ago. It's getting longer and it was when I first interacted, I was doing a penetration task quite a number of years ago, and it was in a ship management company. Ultimately, what happened was they had light bulbs that were actually deployed, and simply those light bulbs were connected to the internet and connected to a Wi-Fi access point, but they were directly connected. They were not segmented; they were not having any dedicated gateway or correlation point. It meant that as long as they can access any access point, therefore, basically, you can intercept the communication, you could...

Ultimately, what we were able to do was also make a Raspberry Pi, pretend to be a light bulb, because we were able to actually simulate the communication. From, let's say a skill side of things, do you look at correlators and gateway for these devices or how is it that they get correlated together? Because for me, I think in these areas, the biggest risks is not the device itself, it's the aggregation point of where they actually come together and if to look at security measures, that's what really needs the emphasize, because even the light bulb itself, the vendor, when we notified the vendor of the risks and the vulnerabilities, they ended up coming and creating what was called as a Smart Hub.

These then devices were not connected directly to the Wi-Fi access points, they connected to the Smart Hub and the Smart Hub that was connected to it. And that provided the protocols and additional controls, because then those aggregators and correlators became more powerful. They had more power and more energy and more connectivity. What do you see is the risk in those areas? Do you have similarities?

Steve Jacobs:

Well, I can talk about what we do and part of this is just because we had to design the system to survive communications failures for long periods of time. But we actually have a dedicated computer each site that is basically a full on Linux box that we can do this with and it coordinates all the collection activities and buffering, and then everything that gets sent from the site gets sent from that system back in. Similar to what you were talking about the gateway, just a much higher powered version of that. We're currently working on our next generation design for that that is tremendously more secure than what we're doing previously. Has much better software update mechanism and that sort of stuff.

Mike Gruen:

You don't have to drive out to the site to update it.

Steve Jacobs:

Well, we don't do that today.

Mike Gruen:

No, I know, I’m just kidding.

Steve Jacobs:

… A fleet of machines and a lot of this connectivity. You can't do something like run an Ansible playbook and have a hit every box an expected way. You have to... Then introducing things like maintenance windows where you can actually do it completely.

Mike Gruen:

Right.

Steve Jacobs:

But I think the challenge that I see is really on the communications protocols that are used for these sorts of devices. If you look at Collab or MGTT, the messaging format is fine, but ultimately, if you want to authenticate a device, when it comes to one of these systems, the choices are not particularly great. Built in, there's usually username and password and if you're lucky, those passwords get hashed on the back end. In a good way, most of them hash-

Mike Gruen:

But the password's a password anyway.

Steve Jacobs:

There's no scalable way to do username and passwords from millions of devices.

Mike Gruen:

Right. Obviously.

Steve Jacobs:

That's just not that's not useful. Then they usually have some support for MTLS, which you can build a scalable MTLS implementation, but it's a ton of work and if you don't do things like handle revocation through some robust infrastructure, then you're in the same boat you were before, you can't move things around, and they expire and renewal has to be handled, and all that kind of stuff.

Then if you look at newer systems, and this would probably work, assuming that you have an online system, they're all using a bearer token type setup, where they go to an OIDC type service, and then there's a bearer token, but you still have this provisioning process that you have to get through where, how does the device get the bearer token and renew the bearer token. The authentication problem for devices that you ship out and don't touch, then come back in is a tough one.

A long time ago, I used to work, doing some government security stuff, and in the way that they would think about things for voice would have a lower standard than something that was automated, because with voice, I've known Mike a long time, I'd recognize Mike's voice. So if he's confirming to me the authentication is working, I know it's not being intercepted and changed because I can hear Mike's voice and I know what he sounds like, but machine to machine communication won’t have any of that, so there's a higher standard of care trust.

Joseph Carson:

Some of that care trust is having. I remember getting into a lot of discussions, because one of the main areas I worked in as a deputy access management and privileged access, that's one of my main fields. One of the things is that it's about, as you've mentioned, even when you talk about the Apple side, it's about how you verified a root of trust and how do you bring people into that trust?  How do you provision new machines? Because provisioning is always from an... That's probably the biggest challenge when you're bringing new devices in.

Does this device know that other device? Are they introducing you to something that we're already familiar with? Has it got similarities? And I think, yes, it reminds me this one, I go back to even one of the old Windows licensing mechanisms. That's what sometimes we always go back to, is that old licensing mechanism where basically, the license is built off five hardware components that was on the device. And if three out of those five components changed, then the license was invalidated and that ultimately gets into that same, the fingerprinting side, you take it off the serial number, the hard disk, how much memory is installed, the CPU, and you basically combine those altogether, you get an ultimate hash.

Then if that hash becomes too distant from the original source, the three components could change, then the license becomes invalidated. And ultimately, the same thing-

Steve Jacobs:

Right?

Joseph Carson:

Yeah. It's ultimately the same kind of mechanism, we need to make sure that one is we have self-provisioning capabilities, and also having the ability to have trust parameters, that allows these devices to self-provision and self-gain access, as long as they meet these set controls that verifies where they're from.  It could be the same from a mechanism. I remember, even what was it?  TPMs was the same problem. They had to come into central locations, you had to put the keys on it, and then in these depots, before it goes to its then destination, because provisioning was always the biggest problem.

That's probably if you're looking to do solve an authentication, identity challenge, then the biggest problem we end up coming down is provisioning enabling and do you have this Depot system that has to go to before it goes out to the site.

Steve Jacobs:

And the way Microsoft and other people get around this thing is with an activation key that can only be used a certain number of times or some sort of business logic rule attached to the activation that has been derived and maybe some sort of internal hash for the value so that you can detect typos easily or something. All that stuff. I think that works fine from a consumer facing application, but not so well, for some more industrial type applications. It just gets difficult.

Then manufacturing wise, we have a manufacturing for where I'm at, where they do cable harnesses and that sort of stuff, but how am I going to get them to do a controlled key load in a way that doesn't leak it accidentally, and they can barely get some eeproms program for IDs off of some barcodes. Doing that sort of thing securely would be a huge training barrier.

Mike Gruen:

Back to the other point, probably not the cost, just isn't worth the squeeze. And it's funny, you're talking about your system and we've talked about OT in other contexts, OT security. I think one of the common threads is how much is over the serial protocols and the lack of security there and basically, what you have is not that dissimilar from a giant car, but unfortunately, your wheels and your steering wheel and your sunroof and everything are just-

Steve Jacobs:

We don't have … in this application, really.

Mike Gruen:

Well, that's true. You don't have the CAN bus, but in terms of that serial that's, again, at the heart of it is that... And I think when we talked about that it was, well, what can we put on top? Because we're not going to change those protocols. We're not going to change...

Mike Gruen:

Yeah.

Joseph Carson:

We're going to change TCP and Ethernet. Those are not going to change.

Steve Jacobs:

But the industrial equipment manufacturers try to do that all the time. There's EtherCAT, right? It's not just us talking about that and when you start talking about a car or factory equipment, there's a life safety aspect to it that I don't have to worry about that I've got but once you start talking life safety now it's the folks that design these industrial protocols, they weren't dumb, they were thinking about that stuff. They were thinking about well the robot arm needs an update every 20 milliseconds and if it doesn't get one, it'll fail into life safety mode and move the robot away and stop moving. They weren't thinking about somebody manipulating that, they were just thinking about if the equipment fails, what do I do?

Joseph Carson:

I had a big conversation when I worked in a project a number of years ago, which was an autonomous shipping project as well and this was a similar scenario, it was about... Not so much about life and safety at the time and it wasn't about encryption and confidentiality, it was all about availability. When you put ships into the middle of the ocean, it was about how well those metal components perform without human intervention. How long can they go? It was a big discussion we had over things like LNG, most ships today are running gas, and therefore basically lighter fuel, therefore, they get longer distances, from a weight perspective, but with an LNG engine, they're heavily human needs to maintain them.

They need more maintenance. People they can actually have in the vessels to maintain those versus things like diesel engines where it's more heavy fuel, and therefore, diesel engines will run for hours and days without human intervention. It's always that balance by tradeoffs. And getting into this big discussion when we were looking at from a shipping side and we had some other issues around connectivity and bandwidth. When you have a storm or a satellite goes out of direction. These were heavily dependent on things like GPS coordinates, DPS, ViaSat links and you've got, of course, multiple systems to make sure you have high availability of communication lines.

But ultimately what they end up getting into as well, and there was a patent, eventually done, which was using a relay system was, to make sure that if you lose connection back to your main, let's say path, that you could relay it through other paths, other vessels in the ocean. If you lose connection with the satellite, what you could do is relay it to another ship that has the connection. It's the innovations that you're doing out of that looks at these higher connectivity advantages, and especially those places that have poor communication as you said, if it was in Alaska, or in the ocean, we're bound with is, is it a premium where do you see that improves this availability?

Steve Jacobs:

Well, we don't do anything like that for any mesh topology or anything like that. Not that it's something that we couldn't do, but there's not really much of an advantage for the way that our sites get deployed in doing that. There's not anything nearby in some of these places at all. And if there is something nearby, we're looking at leveraging it for connectivity, more than anything else, just to get the cost down. That satellite provider in Alaska, there's actually not a bandwidth problem, when it's online, there's plenty of bandwidth, it's just very expensive, because they're the only provider that can hit that-

Joseph Carson:

Think monopoly.

Steve Jacobs:

Yeah and that's your only choice. You can't go to ViaSat or something and get a cheaper connection. We do that at other places... I'd have to talk to our networking guys, it's really only about 10 sites we have that have those really tight constraints and some of them are really difficult because there's an absolute cap. We can't send more than a certain amount of data a month period, there is no option for buying more bandwidth, it's just not available. Then there's another set that are all cellular that, they're a little more expensive, but there's not really a cap and then we have hardware and fiber to a lot of sites, because we end up having to run hard line power.

You have to run power for 20 miles to get somewhere and might as well run fiber while you're doing it. It really just depends on the site, but those mesh topologies are always interesting to me, because they're always talked about in the context of providing greater availability. But as I'm sure you know, the more complexity you add, the more risk you have to availability.  You want to make a high availability web service, well, that's probably going to take a lot of testing, your deployments are going to take longer, and there's going to be more complexity in that solution and it would be for such a simple setup. And really have to ask how much does downtime cost me? Is it worth it?

It comes down to economics, just like everything else, right?

Joseph Carson:

Yeah.

Steve Jacobs:

How much does it cost me? Is it worth the extra complexity to enable it?  What would my cost be if this didn't work? And is it worth the exercise. I think in this case, that ship solution you're talking about is worth it. It's totally worth it.

Joseph Carson:

Because if you lose connection to ship, it takes two miles for them to stop our team. It's not a lot of time you want to wait for that connection to reestablish again.

Steve Jacobs:

Absolutely.

Joseph Carson:

Those things-

Steve Jacobs:

…as much as probably connection and a ViaSat connection and it's probably failure modes that go to lower bandwidth when you hit Iridium. We're really looking to using Iridium next, actually, for one of our sites.

Joseph Carson:

Okay. A lot of the providers are using K-band that L-band, types of connections and-

Steve Jacobs:

…to Alaska.

Joseph Carson:

Yeah. It's an interesting because even how much speed you think it would be the speed that you get through those is so slow and I think that we even look back and I think, probably, sometimes slow connections is good.  We look back even at ransomware cases with NotPetya must be probably really happy that their servers, I don’t forget, had a really slow connection, because it wasn't possible to get infected on time, because of that problem with and ultimately end up saving them. Been able to use that domain controller to restore the rest of the environment.

They were quite lucky and fortunate. Sometimes, low bandwidth connection can save you in the end, because they become very difficult to impact as well, from an attack perspective.

Steve Jacobs:

Right. Another challenge we have on the bandwidth site is just our data hosting and distribution. The instrumentation stuff that we're talking about here, it doesn't actually take up that much space. When I talk to folks that are trying to understand it, they ask, "How much sensor data do we get a day?" And I give them that number, five billion readings a day. No, okay, but how many megabytes? How many megabytes is that? Well, no one asks ViaSat, how many megabytes the transactions they conduct a day take up. That's not the figure that is crazy. It's the individual measurements that have to be processed and handled.

It's not the size. On our remote sensing platform, we fly this plane that's got a hyperspectral imager, a camera, high resolution LiDAR return, we fly each site once a year, and produce a bunch of data. Like they look at foilage coverage from hyperspectral imagery and that kind of stuff. But an individual image from that camera, it's 100-megapixel camera. One image is 28 megabytes or something, compressed. It's one piece of data in any other context and that one, the size is relevant, but the problem I have is I have people that want to come in and download four years of that remote sensing data.

Next week, and they'll tell me, they just want to get it and it makes it very difficult pricing cloud hosting or bandwidth and that sort of stuff, just to try to accommodate the use case for the data and we expect that use case to increase over time.

Joseph Carson:

Do you also have to compete with bandwidth from other services as well? Because one thing that I find quite often working, it was the oil rigs, whenever a helicopter was landing because of the safety aspect, any other communication we were doing, sensor readings, we had to switch it off until that helicopter landed. I also had major problems in vessels where all of a sudden, we were doing some engine diagnostics over the ViaSat link, and we're looking at this data, and then all of a sudden, boom, everything disappears and connectivity is lost. We're, going what was happening?

And we end up finding out that it was the captain decided to make a Skype call to the family and use up all the bandwidth. Do you end up competing for bandwidth you have dedicated or shared links?

Steve Jacobs:

The problem we have tends to occur at sites where there's no cell coverage and we have field technicians that go out there to do work. And this hyperspectral... Not the hyperspectral stuff, the ring down cavity spectrometers, that I was talking about, like a lot of other industrial equipment, they have like a Windows 7 computer attached to them, that manages the system and it's online and it has to be online because the manufacturer supports it through TeamViewer.

Joseph Carson:

Okay.

Steve Jacobs:

That's great. Right? People were going on Facebook on these systems, because it was the only computer out there that was hooked up. They'd get on Facebook, or watch a video on Facebook or something, and then the whole site goes offline.

Joseph Carson:

That's the experience I've seen.  I've seen similar-

Steve Jacobs:

…we run into is, someone does something in this very limited bandwidth and they go, "Look, this works."

Joseph Carson:

I think there's the benefits and the negatives is most of the negatives is likely it's their own self infliction that we end up causing,

Steve Jacobs:

What we've done is we put a wireless access point at the environmental hub that they can get on and it's prevented from eating all the bandwidth that they want to do something and they had a laptop so they can hook it up. If you try to fight that to keep people out, I found that people will find a way in.

Mike Gruen:

Right. I think that's another theme that we found when talking to anybody in security is well, you have to come up... People will find a way around your draconian policies. It's better just to figure out a way to make it so that they can do what they want to do in a secure way that doesn't impact and doesn't do this. You're just better off than trying to fight it. Because otherwise you’re just going to be standing on the sideline, yelling and screaming, "What are you doing?" And the game just continues.

Joseph Carson:

You might find the user worse.

Steve Jacobs:

It's like an architect complaining that the office would be beautiful if people just didn't come in and mess up their desks. Same problem, it's if we just didn’t have these damn users on the system, everything would be secure. They wouldn't click all these email links. Well, I know. But that's their job. That's all they-

Mike Gruen:

And I think that also gets probably back to one of your bigger challenges the ransomware attack. Now these people are using these Windows 7 machines that are out there for a very specific purpose, and that purpose isn't social media.

Steve Jacobs:

Also the internet, because that's how … supports it. Okay.

Mike Gruen:

Right.

Steve Jacobs:

And I can upgrade them to Windows 10 for $10,000 apiece,

Mike Gruen:

Wow.

Steve Jacobs:

And we have 100 and some of them.

Joseph Carson:

And you'd be still vulnerable, but not as-

Mike Gruen:

Right. You'll just be-

Steve Jacobs:

Well, we have another system that is also Windows 7, that is completely unpatched. It's Windows 7 from the first release of Windows 7. And it's needed for scientific process and it's found an uncommon problem to have which is just not on the network. Don't put it on the network, you have to put mitigations in place.  We have someone on my team that used to work for Gates Manufacturing and all of their manufacturing line equipment was like Windows NT. Until very recently, and they have to keep that running because it's billions of dollars of revenue attached to that stuff. But you have to mitigate the risk of running it. You can't just plug it in and go. None of that happens.

Mike Gruen:

Right. That’s one of the other challenges, right? Is as these systems that were never designed to be online or put into context where maybe they can be, or even in air gapped network, there's still problems with that? There's still the potential for, even if it's not on the system, there's still the potential of getting-

Steve Jacobs:

The first problem is most people don't know what that actually means. They say air gapped and they go, "There's a firewall?" That's what people think. They go, it's just firewall off the internet. It's air gapped.

Mike Gruen:

Right.

Steve Jacobs:

It's not what-

Joseph Carson:

It's not even when-

Steve Jacobs:

Government spaces, it's a little different. DOD type stuff, but commercially, what's the temptation to go, "I just need to download some stuff to this thing real quick."?

Joseph Carson:

If it's got a keyboard, it's got interface ports and it's connected to a network it's not very good.

Mike Gruen:

Right. My definition is if someone has physical access to it, it's already-

Joseph Carson:

Does it have a USB port?

Mike Gruen:

Exactly.

Joseph Carson:

It has USB port. I've seen this a lot in my experience in the maritime industries, you've got active systems running and bridges the ships and people plug their phones in to charge their phones into navigational equipment.

Steve Jacobs:

Yes.

Joseph Carson:

And ends up infecting them systems.

Steve Jacobs:

Unless you… box… all the ports shut.

Mike Gruen:

On the plus side, Steve, Windows NT doesn't support USB devices, you have to have a PS2 keyboard.

Steve Jacobs:

Well actually that was really brilliant, the older the operating system, you go back, the less problem is to actually impact today because there's not many people who run and there wasn't many exploits at the time as well that were actually talented to do.

Joseph Carson:

I still have a bunch of OS2 running.

Steve Jacobs:

It'll be fun.

Mike Gruen:

I did work at the national library of medicine for a while and it was interesting talking about, not from a security perspective, but the notion of these digital archives and what does it mean to preserve something and the fact that you had to preserve. If you had a digital file, do you constantly upgrade it to the latest standard in order so that you can read it or do you keep these old computers around that are still able to read those older things and that same thing. There are things in the world that are so hard to talk to you these days that even though they're completely insecure, it's just so hard. It's security through obscurity essentially, because there's not many of them and it's not worth the effort.

Joseph Carson:

The lowest common denominator format for us right now is CSV for a lot of things.

Mike Gruen:

I think that's universal. I think there's a reason why every job I've ever worked at I've written a... This is the first time I've ever had a job where I haven't written at least one CSV parser, usually it's two or three.

Steve Jacobs:

There's a lot of benefit not being CSV for a lot of these data sets. Column or formats have a lot of benefits and we use them internally pretty heavily, but in terms of what we publish, we don't currently expose that stuff and we've got a lot of debate going on internally about whether it makes sense to dynamically convert from the columnar format to various formats people might request or what, and then even in the proprietary accepted formats like that remote sensing stuff like the hyperspectral imagery and that sort of stuff that changes too. We're using geo tests for most of that imagery, but now everyone's moving towards cloud optimized geo tests.

Do we want to make that shift or not? It's constantly upgrading with the data format and I think at some point you have to sign up for that, but then do you keep the old one around too? Is there money to keep an extra couple of petabytes around?

Mike Gruen:

Right.

Steve Jacobs:

It's not exactly free.

Joseph Carson:

Just convert on the fly, man. Just convert on the fly.

Steve Jacobs:

That particular one is pretty interesting. the only difference between GeoTIFF and cloud optimized, GeoTIFF do the exact same format. They're both GeoTIFFs. The cloud optimized GeoTIFF is restructured to allow a range of quests. You can actually view parts of the image over HTTP, as opposed to having to get the whole file.

Joseph Carson:

Having to get the whole file and look at it and then-

Steve Jacobs:

Which also makes it bigger.

Mike Gruen:

Interesting.

Steve Jacobs:

Because the compression is not as good.

Mike Gruen:

Right. You can't do as much compression.

Joseph Carson:

Which again, comes down to the bandwidth issue.

Steve Jacobs:

Well, actually in this case it's data storage.

Mike Gruen:

Right. Storage. Plus, storage is cheap, we all know that.

Joseph Carson:

It's always the case with every question you get into that I've always done infrastructure design architecture, it comes down to storage and speed and you always have to decide which one do you want. And I think that's actually one of the things I've seen here in the study, that was one of the reasons why they're moving to 5G much faster is that they don't need to move the data as much. The data can stay on the edge devices and what they're not doing is they're using that higher bandwidth-

Joseph Carson:

Yeah. They ask the questions that they want, rather than saying, let's bring everything centrally. Let's leave the data out there and we'll ask it when we need it.

Steve Jacobs:

What's that project? It was a former Facebook project that got spun out open source that provides like a SQL query engine across your cluster of machines and you can get dynamic responses back. I can't remember the name of it, but it's similar to that. You run the query and the query gets distributed across the whole environment and then you can get your average CPU consumption off the whole thing. You don't have to store it. It just pulls it back from the machine itself.

Joseph Carson:

That's ultimately what I find as well is that, I’ve done a lot of things like immersive VR types of things as well, would that became an advantage for now what you're doing is, is that you've got these machines-

Joseph Carson:

Yeah. You've got a device which is actually sending it faster and quicker to the central storage over 5 G and therefore you have less battery consumption, so I don't need to walk them around with a heater. I can actually just have a very lightweight sensor that will actually send it right quickly cross.  It's going back to the old mainframe scenario. Basically, just all you’re doing is putting characters in and just getting processed centrally. There's a lot of benefits.

We are doing that pendulum swing again, back and forward because of improved connectivity and being able to query, as you mentioned about the algorithms that you can create out to those devices and not need to bring all the data centrally as well.

Steve Jacobs:

Right. Yeah. That's what we're looking at with this next gen design for data loggers. We're looking at is these new SSCs, they're much faster and so the amount of time you spend doing processing is so much smaller than the power consumption, even though the theoretical power consumption could be so much higher, your processing needs haven’t increased, so you're actually using it so much less that the power consumption is so reasonable.

Joseph Carson:

It also gets into benefits against the reduces the risk against things like ransomware and denial of service attacks because what happens is, is that you're still keeping your data decentralized and what happens is that becomes much more difficult to target from both an availability and an integrity perspective, because then you have the target, all of those sensors and all those devices. It's one of those practices that is only done. We have discussed, Mike, several times about but if somebody is messing with what they do is they don't centralize all the data repositories they keep them decentralized.

Then what they have is a message sharing algorithm that allows you to ask the questions that goes out in real time and gets the data from all the boxes as you need it. There's benefits of doing that. Of course there's also the side effects as well, but it does mean that you have to have good bandwidth connectivity.

Steve Jacobs:

The challenges for us are things like those buoys. They require field calibration for the most part. Every two weeks someone has to go out with a little pool kit and make sure that the measurement for the science they're trying to conduct, they need a certain amount of accuracy, so they have to do that on a regular basis. That information gets logged back to the pipeline that does the calibration and conversion, but it's always delayed, because if someone goes and does that calibration in the field right now.

It'll be at least a few hours before there's somewhere where they can actually enter in the data somewhere as a result and the buoy doesn't have any interface for logging it directly into the buoy because in the model you're talking about, that's what you'd do. You'd log in into the buoy. It would do its own calibration based on what was going on and you wouldn't actually have this downstream system that you'd have to worry about. Now you've got, 10, 20,000 of these systems all doing their own independent piece, and there's not one place to attack or manipulate.

Joseph Carson:

Correct.

Steve Jacobs:

But then you have to maintain 20,000 systems consistently and that's a different problem.

Joseph Carson:

Then that becomes the identity and access piece becomes very critical, important. The access controls for all of those.

Steve Jacobs:

There's a project you might be interested in called Waggle. Apparently waggle is the dance bees do to say where the honey is or the flowers are to go get something. These folks had an interesting idea, which is an edge compute type approach. At least the original system they deployed, they have a camera attached to an embedded  SPC type system and they want to count bees. They use a machine vision application that counts bees to go by on video. It's probably not a hundred percent accurate, but it's more accurate than not counting any bees, because that's the other choice.

They’re just sending back the bee count, not the video feed back across some crazy link and retaining it and everything else. That's an interesting way to approach it is to say, "Well, the science we can do with that data is better than no science and we can improve the algorithms over time and get a better idea what's going on."

Joseph Carson:

Yeah. It's similar to the contact tracing that Google and Apple have done. The same, same concept, decentralized edge contact tracing, or basically they're just creating unique identifiers, sharing them with the devices in proximity and simply uploading that to a central database for tracking and alerting. Anything you do on the edge, I always find, it's better long-term. Ultimately the less steady at the centralized.

Mike Gruen:

Yeah.  The other thing I think; Steve you're talking to is also the idea that like usually getting 80% accurate is that's relatively easy. It's that last 20%, that's always really, really hard and usually 80% accurate is actually really good. If you look at the math of how much harder it is to get more accurate and what does that better accuracy result in, it's so diminishing returns that these types of systems, especially when talking about scientific systems that are massively distributed, so you have lots and lots of nodes. The accuracy... Yeah.

Steve Jacobs:

I'm going to use some military terminology here, but I'm going to talk about my program. I would divide data needs for my program into two classes, their strategic needs, and then there are tactical needs. Strategic needs are the goal of the program. It's a 30-year data repository of ecological information. Someone needs to be able to come in and near 30 and asked for a 30-year history of soil types at one of our sites, whatever it is they're trying to research. It's a very strategic resource. It's for long-term planning, it's needed over a long period of time and an edge type approach like you've described.

I don't see any way to make that work, to meet that strategic need. However, there's another need internally, which is a tactical need. And by the way, that strategic need, if it takes me three to four weeks to provide data where there's a 30-year tail on the data, no one's really concerned. They want to make sure the data is of high quality, that it's got all the things apply that it needs to have. And if it takes longer to assure that that's more important than the speed at which you can have that data. On the strategic side-

Mike Gruen:

You mean the tactical side?

Steve Jacobs:

Sorry. The tactical side, thank you. From a tactical side, I need to run the observatory. If someone unplugs something, and that means that I'm getting zero measurements back from a sensor, and they're only at a site every two weeks or every month, I don't want to wait a month to fix it. I want to know immediately, as soon as that happens, there's a problem and that they can go fix it. I want to shift those problems left, just like we always talk about for engineering. I want to have, as soon as the problem occurs, I have a very latency sensitive need to know there's a problem, or there’s a change in implementation.

If someone has calibration coefficients for one of these thermometers that starts showing it’s, 90 C outside, and that can't be possible, I don't want to publish data with an incorrect calibration. On the strategic side, I don't want that. I want to make sure I can get it out, but on the tactical side, I want it as soon as possible, so I can avoid publishing it the other way and really on these latency sensitive needs, I think that's where the edge strategy really shines. Is, it gives me the ability to make decisions as far out on the edge as I can, so I can correct problems as fast as possible, preserves that shift left mentality.

And if I have the ability to query across the whole system, as a consistent state, it gives me that big picture of what's going on. I can do status reporting and that sort of stuff. That's really where that use case shines.

Joseph Carson:

Absolutely. And the comparison I make is when we were doing this vision for the future in regards to transportation. What you're saying is literally the difference between a car and this chip. It's literally that difference is car is online, it’s very well connected, and you can have your strategic data flow immediately, on demand. You've got good connectivity; you've got good access to it.

Steve Jacobs:

And that can actually be a negative-

Steve Jacobs:

...you have spoofing that they're going to hit you, and that sort of thing.

Joseph Carson:

Yeah. Then you've got the maritime side, whereas you've got pro connectively, so you want to keep it out there as long as possible, but you want to have the maintainability. Get it when you need to. I think the comparison, it means that you have to strategically decide for certain locations, which one to do, which one is the more positive side for each of those locations, depending on those aspects. For me, I see it from a different perspective, from my experience of the past looking at maritime shipping, and looking into transportation, those side of things.

That's where I've been more familiar with. And that we're in the agriculture side of things, the sensory for a scientific side, the only times I've ever been is looking at it from, what I've seen, drone maintenance. Going and checking. Using drones to go out and check the things as expected. That's where I've seen it being used, but it's really insightful. I've learned a lot from you today and I really appreciate it and I'm sure the audience have also been finding it interesting, because many cases when we're talking about security roles, talking about privacy and confidentiality.

But I think this really highlights that not everything is about encryption, not everything is about confidentiality and that other types of programs and other types of technology does have an importance that availability and integrity, the accuracy and veracity of that data becomes really critical.  I've learned a lot today and a really pleasure talking with you and actually getting introduced. I think my takeaways here for the audience is really is that not everything needs to be a secret and encrypted. It really comes down to some areas of security is really about integrity and availability.

It's important to make sure that you, depending on what service you're providing, that when you take the CIA triad, you really understand how it applies to your service. Mike, any thoughts, any key takeaways that you've gathered from today?

Mike Gruen:

Yeah. I think a lot of that same stuff. It's just always nice to see those same security trade-offs and balances available, all of that. It's always applicable. It's just different trade-offs, different things in these different environments. I always enjoy talking to Steve. We've been good friends for a long, long time. It's always fun to talk. Steve, any final thoughts or anything you wanted to part with?

Steve Jacobs:

I think it sums it up pretty well and I think it's also interesting to note that in an environment where confidentiality is less important, the other aspects that, it's like an area graph that has grown on the other ones that you don't get away from it. You have to spend more time on those other areas it seems.

Joseph Carson:

Absolutely. Awesome.  Steve, great to get introduced to you. I'm pretty sure the audience have really had a very enjoyable conversation from the discussion today and I'm sure like me, they've learned a lot. I t's great having you on the show, look forward to seeing you on future shows and episodes to discuss more details. Again, this wraps up another 401 Access Denied. It's been fantastic and enjoyable for me. Again for the audience, don't forget to tune in every two weeks for our latest episodes, stay up to date. Send us what questions you might have for us, if there's topics you'd like to hear, about, all of this.

Feel free to connect with myself and Mike G on social media, we'll be happy to connect and get your feedback into input for future episodes. Mike, awesome chatting with you as usual. Steve, great to connect with you and for the audience, stay safe and secure. Thank you.