Many devices on modern networks aren't what their labels claim. This episode, Rob King, Director of Applied Security Research at runZero, explores white-labeled surveillance and IoT hardware, why some vendors are banned by governments, and how hidden risks can spread across enterprises. Discovery, device fingerprinting, and protocol analysis reveal what's really connected — and why knowing your true inventory is now essential for security, compliance, and trust.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: In October 2016, a large botnet brought down Dyne, an internet provider that supported Netflix and other large streaming services. Botnets were nothing new. They've been used to target companies and then extort fees to get the attacks to stop.

Basically, a denial of service attack works by sending a website or server requests, then not following up with typical responses so that the server is opening more and more requests, such that it handle any new requests from users legitimately trying to access the site.

Obviously one computer can't send all those requests, so distributed denial of service attacks are used where compromised computers all over the world target a specific server to flood it with requests. And yes, content delivery networks can shift the address of the server to mitigate the attacks.

That's not what's important here. Here the DDoS attack came from ordinary street and home surveillance cameras. The Mirai botnet used a vulnerability in a white label chip that was sold in white labeled camera. White label means that someone produces the camera and then others buy it and slack their name on it.

This is a story about white label devices that are in your organization, and why it's important to identify them before it's too late.

I'm Robert Vamosi

This is Error Code.

[MUSIC]

KING: So hi there, everybody. My name is Rob King. I am the director of applied security research at run zero. And run zero is the, in my personal opinion, the world's greatest exposure management platform, vulnerability discovery platform, vulnerability management platform and asset discovery, we really do, and I've been doing this a long time, we really do the best asset discovery in the world. We really do the best exposure management of the world. It's super neat. Free to try. Run zero.com/try so check it out. Cool.

VAMOSI: If I haven't mentioned it before, SecTor is one of my favorite security conferences. It's held in Toronto, and it's been under the radar for years, yet it attracts some of the best presentations. I always look forward to it. Ironically, Rob and even RunZero are American. His talk on white label devices at SecTor was really good. So perhaps we should start with Rob's definition of what a white label device is.

KING: sure, so it might be a surprise to some people, but most of the time, in some for some types of technology, when you buy it. The brand that's on the label is not who made it. The example I always like to give is essentially every microwave oven sold in North America, regardless of the brand, General, electric Sears, Whirlpool insignia, whatever. They're all made in the same factory in Guangdong. So there's they just come out of the factory. A retailer puts their label on it, puts a fancy brand on it, and then sells it. And that's really true with things like security cameras. Security cameras are extremely commoditized. So for example, if you look at a company like Lorex, or even Honeywell, their cameras are actually made by Dahua. So dah Hua is a Chinese company, and I'm sure I'm saying it wrong, it's dah Hua. There we go, still saying it wrong, but less wrong. And they they make cameras for tons of different vendors, Honeywell, Lorax, activecam, Amcrest, ikagami, hiltron, KV vision, all these different cameras. When you buy them, you're really buying a dot block camera.

VAMOSI: So when I when I hear you talk about white label surveillance cameras, I'm immediately going back to the Mirai botnet and the fact that a vulnerability and a white label camera propagated through a whole bunch of systems and allowed a botnet to take over. That's kind of an extreme example, but something like that could happen with these devices as well, certainly.

KING: And there was one, and it wasn't one of the ones I just named, but we did have an interesting example of one of these cameras that recommended that you download a root signing certificate from the camera and install it across your organization so that your traffic to the camera was encrypted and secure, right? And never mind, you could also use that certificate to spoof any website in the world. So little bit of a potential backdoor there, right?

VAMOSI: But the thrust of your your talk was more on the legality of having certain devices in your system, because there are certain devices that are now banned by various federal and state and provincial governments. Could you talk a little bit about that?

KING: Exactly. So DAW was one of the big ones named also hic vision, which some people say, Hi K vision, I think it's hick vision, but either way, I've heard it both ways. But those devices are banned by the United States federal government in any situation where it could be considered a secure environment. So observing federal buildings or installed in, you know, organizations, installed in locations where federal data is processed, that sort of thing. And then a lot of state governments also have that ban. And in fact, a lot of state and local governments in the United States will base their own technology lists on the federal government's list. So it tends to be banned just about everywhere. And then in Canada, same sort of thing, there's a federal ban across all Canada for certain vendors. And then also several individual provinces, Quebec, especially, have banned certain devices and certain vendors for security reasons, for national security.

VAMOSI: Okay, so I want to dive into this whole idea of how you discover these devices, as Rob was saying, if I buy a microwave and it says Amana or something on it, and it's not deep down inside. It's actually mass produced somewhere else in the world. It's white labeled. Same with a network device. Let's talk a little bit about the discovery and how you would go about finding this out.

KING: So we there's lots of little telltale signs. And I love, you know, I got into this industry because I just love dissecting protocols and fingerprinting devices. And this is not a joke. I really love doing this. I would do it even if I wasn't getting paid. I'm very lucky that I get paid to do it. So there's lots of little bits and bobs on these devices that, if you know what to look for, you can, you can kind of tell. I mean, sometimes they'll just straight up tell you, like, if you go to, you know, if you open up one of these cameras or devices or whatever, in your web browser, it'll say somewhere in the corner, like, oh, it's tick vision, right? Or oh, it's Dahua. But most of the time, they're not quite that, that obvious, right? And in fact, what you'll often see with these things is that they're almost, unusually, almost, almost pathologically generic, like you can go to their web interface and there's literally no mention of a brand, there's no copyright message, there's nothing. It's just, you know, this very generic username, password, login prompt. But that generic, this generic city is is still common across all of them. So you can say, oh, you know, I, I can, it's really fun at parties. I can look at a random camera web interface and be like, Oh, actually, that's a big you know, that's like, said fun at parties. But you can also look at little individual protocol works, which I think are fun. So for example, a lot of times these devices will register a name with DNS. So you're if you have an environment where you're getting an IP address or whatever via DHCP, and you can do dynamic DNS, it'll say, oh, yeah, this the host name is, pick vision, dash, 1732, or whatever. So it's pretty telling that, Oh, I thought that was a Honeywell camera. No, it's actually. And then you also have, you know, some obvious stuff as well, like MAC addresses. So every device in the world, almost, you know, there's a million people in the audience saying, Actually, okay, there's a MAC addresses. Are the hardware addresses of these devices, right? They're supposed to be unique across all devices, and they're maintained. That uniqueness is maintained by manufacturers asking for blocks of MAC addresses from the management authority, which is the I triple E. And so you can just go and look at the MAC address and go look up the vendor, which is based on the prefix, and it'll say, oh, it's actually, you know, the Hua technology code limited, right? Which is always fun. So there's some really obvious things, and then there's also some more, some more, more subtle things.

VAMOSI: And among the more subtle things, you talked about how even the documentation is generic,

KING: yes, yes, which is, which is always fun. At one point, I had a slide where I had a side side by side comparison of the documentation from Dahua for that one of their network video recorders, and then one from Mountain View. And it's very much like, you know, Hey, can I copy your homework? Okay, but don't make it obvious. And so there's, you know, the blue is a slightly lighter shade of blue, and the model numbers are slightly different, but it's the exact same, you know, the exact same page, the exact same documentation, more or less just the logos change, which is always interesting, and then sometimes the documentation is also aggressively generic. Where they don't even mention the name of the product, they're just like, Thank you for buying my network video recorder. And it's like, which one did I buy? I don't know, though, right? Always, always useful there.

VAMOSI: So yeah, in your presentation, you talked about the login screens, and how, you know, just looking at the login screens, you might not be able to tell that it's a generic one, but sometimes it's blatantly obvious, because there's no branding on it, or, to your point, copyright, etc, etc. And then right, and, okay, no, you, continue, sorry,

KING: oh, I was just but then sometimes they will put the logo there, right? They'll say, Oh, it's a, you know, an Intel Ross or, or big vision, of course, you know, that's one of the obvious ones. But then even the some of the subtle stuff in if you go look at the source for the login page, you say, view source, which, for those of us of a certain age, that's how we learned how to write HTML. Was viewing source. It almost felt like and by breaking the law, if I view the source of this web page back in 1996 but you can go and look, and it'll just be like, you know, pick vision.js, or something like that. And so it becomes obvious there as well, if you just even peek a little bit behind the curtain. And then also looking at just the HTTP headers. So you know, when you talk to these systems, the system will send back HTTP headers saying, you know, my server is whatever. And the server for hick vision cameras is often a web server called hick vision web server. So regardless of what kind of camera you've got, if it's running pic vision web server, you can be relatively assured that it is a pic vision camera or device.

VAMOSI: So right now, we're talking about a specific example. But if you're running, you know, critical infrastructure, if you're running a utility or factory or whatever you have 1000s of these devices. So how do you begin that discovery process to find out what you have on your network?

KING: I would recommend that you get run zero.

VAMOSI: Yes, okay,

KING: it is truly it is truly quite useful. You can search for, you know, all these different First off, we'll do the fingerprint for you, and you can say, hey, please show me any dotwa cameras or hit vision cameras. Here's the, you know, we even have a built in query for different vendors and devices banned under various regulatory regimes. So they'll just pop up on your dashboard and say, Hey, by the way, here's all these, you know, big vision devices. But if, in a bout of peak, you just decide you can't possibly use run zero, but you should, but you just want to be recalcitrant, I would recommend starting to look through your inventory using the discovery protocols for these these devices themselves, a lot of them would have, will have proprietary protocols and open source, you know, standardized protocols that they use for discovery. And what's interesting is, all of you know, all of the Dahua devices will respond to the Dahua Information Protocol dhip, even if they are not branded Dahua, right? So you can download the Dahua DH IP app on your phone and run it, and suddenly your cameras that aren't labeled Dahua will start responding to to this, you know, these, these probes, which is like, oh, okay, maybe those are Dahua kind of things, which is fun.

VAMOSI: So explain to me a little bit about this discovery protocol, the proprietary ones. I hadn't really heard much about that.

KING: Yeah, so there's been with the advent of IoT and everything's a computer now, your refrigerator, your microwave, your washing machine, there's been this huge push to make them configurable and standardized on the network using protocols that anybody can talk to. You know they want to be able to be configured remotely and talk to the Internet. In fact, some of them will even do nice things like punch holes in your firewall, which we had a nice discussion about that on our blog the other day. But these discovery protocols are designed because these are consumer These are pieces of consumer technology, and they want consumers to use them. So they want to make it easy to show up on your phone and look how high tech they are. And they don't want to, you know, long gone are the days where they expect you to, you know, laboriously type in an IP address on the printer's front panel, or, you know, flip some dip switches or something like that, so, or just have a static IP address that is always there that just conflicts with everything. So now they've got these discovery and configuration protocols, and a lot of them, so a lot of them, for discovery are fairly standard, like multicast DNS is almost universally supported. UPnP, quite popular. But for configuration, the protocols tend to be a lot more proprietary, and these proprietary protocols will also give a lot more detailed information in some cases. So like I was saying with Dawa, they have the hip, and then hick vision has their own called, I forgot what hick visions protocol is called. It is called sadp, which is the something something Discovery Protocol. But these protocols provide, you know, a mechanism for discovery of these devices, and often, a way of configuring them that would just wouldn't be available with something like mDNS, you know, and these protocols are one of my, one of my joys, one of the things I really enjoy doing is reverse engineering these protocol that I can, you know, send out pro packets myself without relying on a third party tool. And it's amazing what you find when you do that.

VAMOSI: So just to abstract it, to make it friendly for everybody, you buy, say, an LG washing machine, and you download the LG app and it says, Oh, now, now you can configure your wash load on your phone, but you also have an LG refrigerator. Would you like to configure that as well? Etc, etc. So that is an example of LGs Discovery Protocol going out there and fingering your network and making sure, like, all the devices that they have control over are pulled into the app for your convenience,

KING: exactly, exactly. And yes, interesting use of the word control there, but yes, absolutely right.

VAMOSI: And then I just imagining, though, like, are these, like the Dawa and so forth? Are they for consumers, and you're using them in a network, and, sorry, an enterprise or industrial environment, are these actually industrial discovery protocols?

KING: So a lot of these cameras are what you would get for home security, which there's been just an absolute explosion of home cameras recently. I remember, you know, thinking it was so weird that my friends were like, Oh, I've got cameras on every corner of my house. And now that's that's almost common. Spoiler alert, if you see the movie weapons, that's a major plot point. But a lot of these devices are also industrial and prosumer and designed for outdoor use, heavy weather use, secure use, monitoring businesses and gates and in fact, a lot of them even integrate with things like door controllers and gate controllers and people counters, which is a whole class of technology that will actually use facial recognition and AI and say, Hey, this is, you know, this person came in, this person, the same person left. Two people came in, but one, only one left going out, you know? And that's a whole other fascinating piece of technology that a lot of these devices integrate with them.

VAMOSI: So a moment ago, you laughed when I used the word control. Are you thinking nefarious purposes, or is it just something as simple as, like, Oh, my washing machine needs a software update and, Oh, it got pushed out automatically. I don't have to think about it.

KING: I mean, it's definitely, that's part of it, the nice part, but the reason why a lot of these devices are banned in government networks is simply because the powers that be are worried about how much control these companies and these vendors can have in known secure or supposed to be secure environments. We don't, you know, I'm not going to say that we have any evidence, because I don't think we do. But theoretically, speaking, you know something, something nefarious could happen with sending out a software update that sure it improves the code, but also it adds a little bit of surveillance tech that maybe you don't know about. So it's, I don't want to say that. It's I don't want to, I don't want to impugn anybody. But yes, there's, there's a little bit of nefarious potential there, I suppose.

VAMOSI: So it's a bit of you're assuming good intent by having the vendor push out the code for you and update the device for you, you don't have to run around on Saturday and do patch Saturday of all your electronic devices.

KING: I'm assuming good intent is not necessarily the right word. I'm assuming consumer convenience, right? And also, by the way, you should patch all your IoT devices. When was the last time you checked to make sure your printer was running the latest firmware? Maybe not ever. Yeah.

VAMOSI: Well, that is the IoT problem. It's like, the stuff that businesses were kind of good at consumers are really bad at, and often they don't really think of the two things together. It's like, yeah, I can turn on my dishwasher remotely with my app, but it's like, on the same token, what software is it running and how what was the last time it was updated? You know, nobody thinks in those terms on their own has to be right, pushed down from somewhere.

KING: So what was it the I'm flying 35,000 feet, but thanks to the wonders of technology, I'm annoying my family by making my car horn deep in the garage, like exactly that's it's great.

VAMOSI: So going back to the white labeling in businesses and so forth, having these prohibited devices, are there consequences for businesses that are unaware and haven't scoped to see whether they have these devices.

KING: So, you know, I am not a lawyer, so don't, don't take anything, I say, as legal advice. But these devices, yeah, you it can put contracts with federal business and or government public business entities in jeopardy. You can face fines. You could face, you know, up to including, I guess, contract cancelation if you can't prove that the environments that you're using are free of these vendors. And we even had a famous example I like to give run zero in, I can't remember where it was, but it insisted that there was, you know, we were absolutely sure that there was, in fact, a band vendor at this site. And everyone was like, No, there's not. We checked. We looked everywhere. We looked everywhere there is a demand vendor on this network. And the guy was like, you know, if we don't find this soon, if you either you've got a bug or something else is going on, anyway, in the vendor's pocket was a phone from event from a band vendor that had connected to the Wi Fi. So, you know, the call was coming from inside the house, literally.

VAMOSI: Yeah. So this is then, just part of a regular security assessment that would be performed. It's one of the things that would be scoped into that assignment.

KING: Yeah, indeed, and it's generally considered part of the due diligence before you enter one of these contracts, right, right?

VAMOSI: And so you attest that your network is free of these devices, and the contract goes forward, hopefully, hopefully. Yeah, I noticed also that there was no grandfathering in of these devices. It's pretty, pretty strict. It's like, if you have them anywhere, they must be removed at any cost.

KING: For a lot of them, yes, and in at least one of the provincial governments in Canada, they did give a, you know, no going forward. But the ones that were there were okay, but that was the exception to the all the other ones I read about were, were definitely, you know, nope, you know, these must be removed by this date or else, right?

VAMOSI: And that that is a problem with this whole IoT conundrum, because there's, you know, I've seen estimates, billions of devices out there. Yeah, the problem is out there in the world, and now you've suddenly got to rein these devices in. And oftentimes they're really dumb and they don't have a lot of memory and they can't be queried or whatever. But you know, so be it that's that's where your business comes in, finding good asset discovery system, right? Exactly. So you guys don't do just devices, right? You're also doing all sorts of network discovery, right?

KING: Yes, it is. I take it personally. If there's a device on your network or a service on your network that I can't fingerprint, and I really do like, I get mad. I'm like, no, no, no, no. I've never, you know, we've never seen this before, but by the end of today, I'm going to tell you exactly what it is or die trying, right? So it's a lot of fun. So we can detect any any piece of software that's listening on a network socket. We will try to identify that software, any device that's on the network, including stuff that speaks relatively esoteric protocols or even tries to kind of hide itself. We tried really hard to detect it. We also will detect devices that are acting as network bridges. So, you know, there's a famous example of the aquarium pump in the casino where their network was completely locked down and everything was secure. But then they brought in a smart aquarium pump and connected it to the Wi Fi, and it was bridging the public, the public network, with the secure network. And attackers were actually able to ransomware a casino by going through the aquarium, home and run zero can say, Oh, by the way, did you know this device is acting as a router? You should probably look at that.

VAMOSI: It's funny. For years people in IoT have cited this example to me, yet other than a brief mention in DarkTrace's global threat report in 2017 — which is now offline — there is very little documentation that this event even happened. And I haven't found anyone who can provide details about the attack other than an attacker used the online aquarium system as a way into an otherwise secure casino. I am not saying it didn't happen; I'm just saying for the last ten years you'd think there would be more detail about this event. From credible sources. And there is not.

KING: Although it is incredible, like, Why do smart light bulbs act as routers. They really shouldn't be doing that, but they right. And then one that our illustrious founder, HD Moore has brought up a few times, that I think is really interesting is even people, you know, in the industry, they don't realize that oftentimes certain bits of tooling, like, if you're running, you know, a large number of containers, or, I'm sorry, any number of containers on your on your laptop, or, you know, you're running some virtual machine software or something. Oftentimes, those configurations will turn your laptop into a router because they want to be able to route to those machines, but they don't say only route on this one local interface, because they want to be able to talk to be able to talk to the rest of the world or whatever, and so you're suddenly your router becomes a bridge. Yeah, I'm sorry, your laptop becomes a router and a bridge between potentially many networks.

VAMOSI: I hadn't considered that. That makes total sense, because you want your VM to talk to the internet. Yes. Okay, so for those of us at home that are just playing around with this, you mentioned in your talk Wireshark, I have something on my phone called thing, and it allows me to fingerprint networks, and I love going to like public spaces and just seeing who's connecting and what's connecting and all of that. How is that? I mean? Is that a good entry into this space?

KING: I will tell you that you it is the most fun you can have at a coffee shop to open up Wireshark. And I'd recommend anybody do it, even if you're not really interested just to see how the sausage is made. It's the amount you'll learn just by kind of watching it flow by is amazing. Years and years ago, I, for several years, I taught a class on network forensics. And you know, we would have people come into the class and they never even heard of Wireshark. Some of them didn't even understand. And I don't want to make this sound like you should just, you know, nobody's bored understanding this. But they came to this class with no foreknowledge of even the concept of, like, packet oriented communication, like, you know, breaking things up into packets and then sending them across the wire or the ether or whatever. And by the end, they were like, This is great, you know, I can see exactly how it works, and I can, I'm going to go home and I'm going to run Wireshark on everything. And I was like, yes, yes, I have succeeded lots of fun.

VAMOSI: So neither of us are lawyers, but so the Look, But Don't Touch rule, you're looking at these packets. What can you glean from that, legally or successfully?

KING: So, in a lot of networks, the packets are just there anyway. It's not like you're doing anything wrong. You know, your computer is receiving them kind of, sort of anyway. And a lot of cases and they're just dropping so just looking at opening up Wireshark, there's, again, I'm not a lawyer, but really, there should be no harm we have gotten as an industry. We have gotten better about trying to make things encrypted by default, zero trust by default, and all that. But there's still an immense amount of traffic out there that is unencrypted, and you can just see information flying across the wire. And with the advent of all this, these IoT devices who want to be discovered, right? They want to be easy to connect to. There's just a regular pulse of discovery traffic going out saying, you know, oh, by the way, I'm here in DNS advertisement. Hey, you know, look at me. I'm, I'm here to be, to be connected to and printed with. Or, you know, whatever this device is. So even if we've started encrypting most traffic, the IoT wave has made discovery so much more popular, so much easier. And then also, if you get into these sort of, I don't want to say like retail, but you know, if you know, if you go to like a place of business, oftentimes, you know, the coffee shop doesn't have a dedicated IT staff, right? And so a cash register is probably not going to be going to be set up once and never updated again, you know, or you have a contractor that only comes out once a, you know, once a quarter or something. And so these devices often have some very interesting bits and bobs running on them. You can even see things flying across like you may not. Hopefully it's encrypted, but you still see things like, you know, the cash register is very talkative now somebody, somebody is running a job or something, which I think is, is fun, and then also just host names like, oh, I now know that the guy sitting over there must be named John because there's a device on the network named John's iPhone. A little bit of PII just sitting there for the picking.

VAMOSI: Yeah, that's what I have done, is just like, Oh, you've named all your devices. How convenient. I know who you are. Awesome, exactly. So you also mentioned in your talk that SS seven is gone away, and now it's SIP its voice over IP. It's how are devices taking advantage of that? We just talked about Wi Fi, but now we're talking about, like, other forms of communication.

KING: Yep, and yeah, SS seven, man, that's Wow. Thank you for Thanks for the memories I have. I've totally never done anything illegal with the phone network ever so and that's my story, and I'm sticking to it anyway. Yes, so sip, which is the session initiative, Session Initiation Protocol, is what's used to control voice over ID calls in most situations. And I remember, this was in the early 2000s wired or some other technical application said, oh, you know, by the by the 2030s you know, 50% of phone calls will be going over an IP network. And I was like, that's never gonna happen. That's impossible. And of course, you know now, everything is voiced over IP. And so SIP is interesting in that it will, if you're, if you're talking to a SIP endpoint, it will tell you quite a bit about the device. Can tell you quite a bit about the device that it's running on. So a lot of these, these, you know, potentially banned vendors, will have interesting, you know, sip headers like, oh, it's, you know, the hospital server, or whatever. So there's yet another way of potentially identifying, identifying these devices.

VAMOSI: But what are they using those for? If they've got a web server, if they've got other things on there, what is the SIP server doing for them.

KING: So a lot of them are also intercoms. So if you want to talk to somebody over there at the gate, that's often set up using sip and then a related protocol, sort of related protocol, RTSP, the real time streaming protocol is what's often used for shipping video data and audio data back from a camera or capture device to a recorder. Okay, so lots of RTSP, going on, right? If we're talking about cameras, that makes total sense. If we're talking about washing machines. Made me not so if you see your washing machine running, RTSP, yes, I would start. I would start, I would start asking questions.

VAMOSI: One of the cool aspects of digital fingerprinting is that you can find regional variations. For example, the same device except the firmware has differences because of local regulations or requirements. That allows someone like Rob to look for the odiities– the needles in the haystacks. That one device that doesn't belong.

KING: So yeah, so one of the things that we did was scan the entire internet, at least the entire ITV four space, which, again, if you had told me in 1997 you're going to be scanning the entire Internet to look for things. I would have laughed in your face, but no, you know, it's something that I get to do for fun now, and I get paid for.

VAMOSI: So I want to stop you there. I've talked to people that have done this before. When you're saying you span, you scan the entire IP before you're talking about all the open ports, or you were looking at a particular port.

KING: For this case, we're looking at specific pieces of information, SNP, Telnet, banners, things like that. And yes, people still do some devices do run telnet in 2025 that's that's always fun.

VAMOSI: And how long did that take for, say, Telnet, or any of the protocols that you were looking at, or open ports that you were

KING: looking at, excuse me, so to be polite, you don't want to scan because you don't want to upset a, whoever's hosting a scan and B, you don't want to be rude. And, you know, just hammer a bunch of IP spaces. You also want to try to, you know, kind of spread the spread the spread the love a little bit. And don't, you know, don't do things sequentially, because if you hit a bunch of ideas in the same in sequence, you might bump into you might, you know, exhaust a router, a small routers, a state table or something. You know, be nice. But you can scan the entire internet, picking your speed in 48 hours, if you want to be super slow, which is, again, just mind boggling.

VAMOSI: Yeah, agreed. I'm sorry. I interrupted. There you were. You were saying that you scanned the internet, right? So one

KING: of the things that I wanted to learn about with these, these white label devices, is not, or it's, it's not necessarily white labeled in this case, but different regional versions of the same product. So a lot of vendors will have a specific version of their firmware for a specific market. One of my favorite stories there was, there was a tool, way back in the day, called HP open view, and it was a network management discovery piece of software. It was super cool. And in the background of the main sort of interface was a world map. And the world map just had all the countries colored in different shapes of gray. And they had to have a special release of this software specifically for the People's Republic of China, because the island of Taiwan was a slightly different shade of gray in the worldwide release, and you cannot possibly imply what that implies. So special version of software there different countries, different polities will have different regulatory requirements, including the strength of encryption, or theoretically speaking, whether there's a backdoor enabled or not. And so what I wanted to do was see if we could form some sort of clustering of here's interesting versions of software in different parts of the world. This seems to be this version, or this telltale sign seems to be highly correlated with this particular region or location. And so what we found was it's pretty interesting that you can, you can see that there's a strong correlation for some devices. For example, in Cisco IOS, there are 51 and this is just what's publicly visible on the internet. Obviously, there are 51 distinct versions of Cisco IOS visible in Canada on the public Internet, whereas in the Russian Federation, there's 282 distinct versions. But even though there's 282 distinct versions, Russia still has one version that makes up the plurality, whereas it's much more fragmented in Canada, right? But what this can tell you is, if you see a version, you see a telltale sign of firmware that's extremely common in one location and then only pops up once or twice somewhere else in the world. What does that mean? And what that usually means is you potentially bought something off eBay and had it shipped to you or somebody moved from, you know, Indonesia to the United States, right? One of the things we found that was interesting is there's a very, very common version of my protect router OS that seems to be handed out by some ISP in Indonesia. And so they're just everywhere in Indonesia. The specific version of microtik, there's only one instance of that version in all of Canada. Now, microtik is not super popular in Canada to begin with, but there's, you know, there's several 1000 that you can see on the public Internet, but only one instance of this thing that's extremely popular in Indonesia. So what does that mean? It means either somebody moved from Indonesia to Canada and took their router with them, or somebody in Canada ordered a router off of Ebay or somewhere else, and it was shipped from Indonesia. And you know, you might say, oh, that's just an interesting bit of trivia. And you know, maybe it is, but because of these regulatory requirements, because of the different update cycles that different regions get, that router might not get updates. That router might have a configuration option that isn't available or doesn't have something that you need for regulatory compliance in your country, certain encryption algorithms, certain protocol protocol support, default settings, things like that. And so it's just something to be mindful of that, okay, this piece of hardware is or this piece of software is different, and that's, you know, things that are different on your network are always things that should be investigated. Wow.

VAMOSI: So it could even be like geo fenced or something where it doesn't get updates, if it's outside of Indonesia,

KING: potentially, yeah. And we have one of the things we have in run zero that I always one of my favorite features, and it's such a it's such a small thing in terms of the UI, right? But every asset has a an outlier score. So if you're looking at your network and run zero, and you can say, Oh, here's all, here's all my assets. And if you reach a certain number, and there's a certain bit of mathematical chicanery, you know, there's this large enough sample size to make this chicanery possible, assets will get an outlier score that say, Okay, this outlier is or this asset is extremely different on this one dimension from everything else. Why is that the case? And sometimes it's obvious. It's like everything in your network is running this version of Windows, but this one is running not that version of Windows. So does that mean you missed an update, or something like that? But sometimes it's more subtle, like the default TCP window size on this system seems different from every other system. That seems weird. Why is your TCP stat acting odd? You should really look into that.

-30-