Friday, October 21, 2016

IoT Can Never Be Fixed

This title is a bit click baity, but it's true, not for the reason you think. Keep reading to see why.

If you've ever been involved in keeping a software product updated, I mean from the development side of things, you know it's not a simple task. It's nearly impossible really. The biggest problem is that even after you've tested it to death and gone out of your way to ensure the update is as small as possible, things break. Something always breaks.

If you're using a typical computer, when something breaks, you sit down in front of it, type away on the keyboard, and you fix the problem. More often than not you just roll back the update and things go back to the way they used to be.

IoT is a totally different story. If you install an update and something goes wrong, you now have a very expensive paperweight. It's usually very difficult to fix IoT devices if something goes wrong, many of them are installed in less than ideal places and some may even be dangerous to get near the device.

This is why very few things do automatic updates. If you have automatic updates configured, things can just stop working one day. You'll probably have no idea it's coming, one day you wake up and your camera is bricked. Of course it's just as likely things won't break until it's something super important, we all know how Murphy's Law works out.

This doesn't even take into account the problems of secured updates, vendors going out of business, hardware going end of life, and devices that fail to update for some reason or other.

The law of truly large numbers

Let's assume there are 2 million of a given device out there. Let's assume there are automatic updates enabled. If we can guess 10% won't get updates for some reason or other. That means there will be around 200,000 vulnerable devices that miss the first round of updates. That's one product. With IoT the law of truly large numbers kicks in. Crazy things will happen because of this.

The law of truly large numbers tells us that if you have a large enough sample set, every crazy thing that can happen, will happen. Because of this law, the IoT can never be secured.

Now, all this considered, that's no reason to lose hope. It just means we have take this into consideration. We don't build systems that can handle a large number of crazy events. Once we take this into account we can start to design a system that's robust against these problems. The way we develop these systems and products will need a fundamental change. The way we do things today doesn't work in a large number situation. It's not a matter of maybe fixing this, it has to be fixed, and someone will fix it, the rewards will be substantial.

Comment on Twitter

Monday, October 17, 2016

Can I interest you in talking about our savior, its name is Security!

I had a discussion last week with some fellow security folks about how we can discuss security with normal people. If you pay attention to what's going on, you know the security people and the non security people don't really communicate well. We eventually made our way to comparing what we do to the door to door religious groups. They're rarely seen in a positive light, are usually annoying, and only seem to show up when it's most inconvenient. This got me thinking, we probably have more in common there than we want to admit, but there are also some lessons for us.

Firstly, nobody wants to talk to either group. The reasons are basically the same. People are already mostly happy with whatever choices they've made and don't need someone showing up to mess with their plans. Do you enjoy being told you're wrong? Even if you are wrong, you don't want someone telling you this. At best you want to figure it out yourself but in reality you don't care and will keep doing whatever you want. It's part of being an irrational human. I'm right, you're wrong, everything else is just pointless details.

Let's assume you are certain that the message you have is really important. If you're not telling people something useful, you're wasting their time. It doesn't matter how important a message is, the audience has to want to hear it. Nobody likes having their time wasted. In this crazy election season, how often are you willing to not just hang up your phone when a pollster calls? You know it's just a big waste of time.

Most importantly though, you can't act pretentious. If you think you're better than whoever you're talking to, even if you're trying hard not to show it, they'll know. Humans are amazing at understanding what another person is thinking by how they act. It's how we managed to survive this long. Our monkey brains are really good at handling social interactions without us even knowing. How often do you talk to someone who is acting superior to you, and all you want to do is stop talking to them.

Now what?

It's really easy to point all this stuff out, most of us probably know this already. So what can we start doing different? In the same context of door to door selling, it's far more powerful if someone comes to you. If they come to you, they want to learn and understand. So while there isn't anything overly new and exciting, the thing that's best for us to remember today is just be available. If you're approachable, you will be approached, and when they do, make sure you don't drive your audience away. If someone wants to talk to you about security, let them. And be kind, understanding, and sympathetic.

Monday, October 10, 2016

Only trust food delivered by a zebra

If you're a security person you're probably used to normal people not listening to you. Sometimes we know why they don't listen, but often the users get blamed for being stupid or stubborn or something else to justify their behavior. After having a conversation the other day it was noted that some of our advice could be compared to telling someone they should only trust food that has been delivered to them by a zebra.

It's meant to sound silly, because it is silly.

If you tell someone they should only trust food delivered by a zebra, they might nod and agree, some will tell you that's silly, but fundamentally nobody is going to listen. They won't listen because that advice is completely impractical. If you give impractical advice, your advice gets ignored. This gets tricky though because what I call impractical advice you may not. Things can get complicated especially when a difficult topic is being discussed. It's even harder when you have a lot of people who are self proclaimed experts but in reality don't know very much.

This is basically the story of security though. We give advice that we think is practical, normal people hear advice that makes no sense, makes their life worse, and is probably something they don't even want to do. They have two choices. Tell you they think your advice is bad, or just nod and agree while doing whatever they want. The latter is much less work. If someone tells you the advice you just gave them is bad, you're not going to think about how to give better advice, you're going to spend a lot of time convincing them why you're right and they're wrong. Smart people don't argue, they just nod and agree.

The solution to this problem is very simple to explain, but will be very hard to do. It's not uncommon for me to talk about listening as a very important thing we should being doing more of. If listening was easy, or solved as many things as I claim it would, we wouldn't have any more problems. While it is super important we listen to those we must help, it's only a small part of what we have to do. We must learn to be tactical first. You can't listen to people who won't talk to you. And if you show up demand zebra food, nobody will ever tell you anything useful. You get branded as a kook and that pretty much ends everything.

Stop demanding zebra food.

Comment on Twitter

Monday, October 3, 2016

Impossible is impossible!

Sometimes when you plan for a security event, it would be expected that the thing you're doing will be making some outcome (something bad probably) impossible. The goal of the security group is to keep the bad guys out, or keep the data in, or keep the servers patched, or find all the security bugs in the code. One way to look at this is security is often in the business of preventing things from happening, such as making data exfiltration impossible. I'm here to tell you it's impossible to make something impossible.

As you think about that statement for a bit, let me explain what's happening here, and how we're going to tie this back to security, business needs, and some common sense. We've all heard of the 80/20 rule, one of the forms is that the last 20% of the features are 80% of the cost. It's a bit more nuanced than that if you really think about it. If your goal is impossible it would be more accurate to say 1% of the features are 2000% of the cost. What's really being described here is a curve that looks like this
You can't make it to 100%, no matter how much you spend. This of course means there's no point in trying, but more importantly you have to realize you can't get to 100%. If you're smart you'll put your feature set somewhere around 80%, anything above that is probably a waste of money. If you're really clever there is some sort of best place to be investing resources, that's where you really want to be. 80% is probably a solid first pass though, and it's an easy number to remember.

The important thing to remember is that 100% is impossible. The curve never reaches 100%. Ever.

The thinking behind this came about while I was discussing DRM with someone. No matter what sort of DRM gets built, someone will break it. DRM is built by a person which means, by definition, a smarter person can break it. It can't be 100%, in some cases it's not even 80%. But when a lot of people or groups think about DRM, the goal is to make acquiring the movie or music or whatever 100% impossible. They even go so far as to play the cat and mouse game constantly. Every time a researcher manages to break the DRM, they fix it, the researcher breaks it, they fix it, continue this forever.

Here's the question about the above graph though. Where is the break even point? Every project has a point of diminishing returns. A lot of security projects forget that if the cost of what you're doing is greater than the cost of the thing you're trying to protect, you're wasting resources. Never forget that there is such a thing as negative value. Doing things that don't matter often create negative value.

This is easiest to explain in the context of ransomware. If you're spending $2000 to protect yourself from a ransomware invasion that will cost $300, that's a bad investment. As crime inc. continues to evolve I imagine they will keep a lot of this in mind, if they can keep their damage low, there won't be a ton of incentive for security spending, which helps them grow their business. That's a topic for another day though.

The summary of all this is that perfect security doesn't exist. It might never exist (never say never though). You have to accept good enough security. And more often than not, good enough is close enough to perfect that it gets the job done.

Comment on Twitter

Monday, September 26, 2016

Who left all this fire everywhere?

If you're paying attention, you saw the news about Yahoo's breach. Five hundred million accounts. That's a whole lot of data if you think about it.  But here's the thing. If you're a security person, are you surprised by this? If you are, you've not been paying attention.

It's pretty well accepted that there are two types of large infrastructures. Those who know they've been hacked, and those who don't yet know they've been hacked. Any group as large as Yahoo probably has more attackers inside their infrastructure than anyone really wants to think about. This is certainly true of every single large infrastructure and cloud provider and consumer out there. Think about that for a little bit. If you're part of a large infrastructure, you have threat actors inside your network right now, probably more than you think.

There are two really important things to think about.

Firstly, if you have any sort of important data, and it's not well protected, odds are very high that it's left your network. Remember that not every hack gets leaked in public, sometimes you'll never find out. On that note, if anyone has any data on what percentage of compromises leaked I'd love to know.

The most important thing is around how we need to build infrastructure with a security mindset. This is a place public cloud actually has an advantage. If you have a deployment in a public cloud, you're naturally going to be less trusting of the machines than you would be if they were in racks you can see. Neither is really any safer, it's just you trust one less which will result in a more secured infrastructure. Gone are the days where having a nice firewall is all the security you need.

Now every architect should assume whatever they're doing has bad actors on the network and in the machines. If you keep this in mind, it really changes how you do things. Storing lots of sensitive data in the same place isn't wise. Break things apart when you can. Make sure data is encrypted as much as possible. Plan for failure, have you done an exercise where you assume the worst then decide what you do next? This is the new reality we have to exist in. It'll take time to catch up of course, but there's not really a choice. This is one of those change or die situations. Nobody can afford to ignore the problems around leaking sensitive data for much longer. The times, they are a changin.

Leave your comments on Twitter: @joshbressers

Tuesday, September 20, 2016

Is dialup still an option?

TL;DR - No.

Here's why.

I was talking with my Open Source Security Podcast co-host Kurt Seifried about what it would be like to access the modern Internet using dialup. So I decided to give this a try. My first thought was to find a modem, but after looking into this, it isn't really an option anymore.

The setup

  • No Modem
  • Fedora 24 VM
  • Firefox as packaged with Fedora 24
  • Use the firewall via wondershaper to control the network speed
  • "App Telemetry" firefox plugin to time the site load time

I know it's not perfect, but it's probably close enough to get a feel for what's going on. I understand this doesn't exactly recreate a modem experience with details like compression, latency, and someone picking up the phone during a download. There was nothing worse than having that 1 megabyte download at 95% when someone decided they needed to make a phone call. Call waiting was also a terrible plague.

If you're too young to understand any of this, be thankful. Anyone who looks at this time with nostalgia is pretty clearly delusional.

I started testing at a 1024 Kb connection and halved my way down to 56 (instead of 64). This seemed like a nice way to get a feel for how these sites react as your speed shifts down.


I picked the most popular english language sites listed on the Alexa top 100. I added becuase I like them, and my kids had me add twitch. My home Internet connection is 50 Mb down, 5 Mb up. As you can see, in general all these sites load in less than 5 seconds. The numbers represent the site being fully loaded. Most web browsers seem to show something pretty quickly, even if the page is still loading. For the purpose of this test, our numbers are how long it takes a site to fully load. I also show 4 samples because as you'll see later on, some of these sites took a really really long time to load, so four was as much suffering as I could endure. Perhaps someday I'll do this again with extra automation so I don't have to be so involved.

1024 Kb/s

Things really started to go downhill at this point. Anyone who claims a 1 megabit connection is broadband has probably never tried to use such a connection. In general though most of the sites were usable from a very narrow definition ofh the word.

512 Kb/s

You're going to want to start paying attention to Amazon, something really clever is going to happen, it's sort of noticeable in this graph. Also of note is how consistent is. While not the fastest site, it will remain extremely consistent through the entire test.

256 Kb/s

Here is where you can really see what Amazon is doing. They clearly have some sort of client side magic happening to ensure an acceptable response. For the rest of my testing I saw this behavior. A slow first load, then things were much much faster. Waiting for sites to load at this speed was really painful, it's only going to get worse from here. 15 seconds doesn't sound horrible, but it really is a long time to wait.

128 Kb

Things are not good at 128 Kb/s. Wikipedia looks empty, it was still loading at the same speed as our fist test. I imagine my lack of an ad enhanced experience with them helps keeps it so speedy.

56 Kb

Here is the real data you're waiting for. This is where I set the speed to 56K down, 48K up, which is the ideal speed of a 56K modem. I doubt most of us got that speed very often.

As you can probably see, Twitch takes an extremely long time to load. This should surprise nobody as it's a site that streams video, by definition it's expected you have a fast connection. Here is the graph again with Twitch removed.
The Yahoo column is empty because I couldn't get Yahoo to load. It timed out every single time I tried. Wikipedia looks empty, but it still loaded at 0.3 seconds. After thinking about this it does make sense. There are Wikipedia users who are on dialup in some countries. They have to keep it lean. Amazon still has a slow first load, then nice and speedy (for some definition of speedy) after that. I tried to load a youtube video to see if it would work. After about 10 minutes of nothing happening I gave up.

Typical tasks

I also tried to perform a few tasks I would consider "expected" by someone using the Internet.

For example from the time I typed in until I could read a mail message took about 600 seconds I did let every page load completely before clicking or typing on it. Once I had it loaded, and the AJAX interface timed out then told me to switch to HTML mode, it was mostly usable. It was only about 30 seconds to load a message (including images) and 0.2 seconds to return to the inbox.

Logging into Facebook took about 200 seconds. It was basically unusable once it loaded though. Nothing new loaded, it loads quite a few images though, so this makes sense. These things aren't exactly "web optimized" anymore. If you know someone on dialup, don't expect them to be using Facebook. took 800 seconds. Reddit's front page was 750 seconds. Google News was only 33 seconds. The newspaper is probably a better choice if you have dialup.

I finally tried to run a "yum update" in Fedora to see if updating the system was something you could leave running overnight. It's not. After about 4 hours of just downloading repo metadata I gave up. There was no way you can plausibly update a system over dialup. If you're on dialup, the timeouts will probably keep you from getting pwnt better than updates will.

Another problem you hit with a modern system like this is it tries to download things automatically in the background. More than once I had to kill some background tasks that basically ruined my connection. Most system designers today assume everyone has a nice Internet connection so they can do whatever they want in the background. That's clearly a problem when you're running at a speed this slow.


Is the Internet usable on Dialup in 2016? No. You can't even pretend it's maybe usable. It pretty much would suck rocks to use the Internet on dialup today. I'm sure there are some people doing it. I feel bad for them. It's clear we've hit a place where broadband is expected, and honestly, you need fast broadband, even 1 Megabit isn't enough anymore if you want a decent experience. The definition of broadband in the US is now 25Mb down 3Mb up. Anyone who disagrees with that should spend a day at 56K.

I know this wasn't the most scientific study ever done, I would welcome something more rigorous. If you have any questions or ideas hit me up on Twitter: @joshbressers

Sunday, September 18, 2016

Why do we do security?

I had a discussion last week that ended with this question. "Why do we do security". There wasn't a great answer to this question. I guess I sort of knew this already, but it seems like something too obvious to not have an answer. Even as I think about it I can't come up with a simple answer. It's probably part of the problems you see in infosec.

The purpose of security isn't just to be "secure", it's to manage risk in some meaningful way. In the real world this is usually pretty easy for us to understand. You have physical things, you want to keep them from getting broken, stolen, lost, pick something. It usually makes some sort of sense.

It would be really easy to use banks as my example here, after all they have a lot of something everyone wants, so instead let's use cattle, that will be more fun. Cows are worth quite a lot of money actually. Anyone who owns cows knows you need to protect them in some way. In some environments you want to keep your cows inside a pen, in others you let them roam free. If they roam free the people living near the cows need to protect themselves actually (barbed wire wasn't invented to keep cows in, it was used to keep them out). This is something we can understand. Some environments are very low risk, you can let your cattle roam where they want. Some are high risk, so you keep them in a pen. I eagerly await the cow related mails this will produce because of my gross over-simplification of what is actually a very complex and nuanced problem.

So now we have the question about what are you protecting? If you're a security person, what are you really trying to protect? You can't protect everything, there's no point in protecting everything. If you try to protect everything you actually end up protecting nothing. You need to protect the things you have that are not only high value, but also have a high risk of being attacked/stolen. That priceless statue in the pond outside that weighs four tons is high value, but nobody is stealing it.

Maybe this is why it's hard to get security taken seriously sometimes. If you don't know what you're protecting, you can't explain why you're important. The result is generally the security guy storming out screaming "you'll be sorry". They probably won't. If we can't easily identify what our risk is and why we care about it, we can't possibly justify what we do.

There are a lot of frameworks that can help us understand how we should be protecting our security assets, but they don't really do a great job of helping identify what those assets really are. I don't think this a bad thing, I think this is just part of maturing the industry. We all have finite budgets, if we protect things that don't need protecting we are literally throwing money away. So this begs the question what should we be protecting?

I'm not sure we can easily answer this today. It's harder than it sounds. We could say we need to protect the things that if were lost tomorrow would prevent the business from functioning. That's not wrong, but electricity and water fall into that category. If you tried to have an "electricity security program" at most organizations you'll be looking for a new job at the end of the day. We could say that customer data is the most important asset, which it might be, but what are you protecting it from? Is it enough to have a good backup? Do you need a fail-over data center? Will an IDS help protect the data? Do we want to protect the integrity or is our primary fear exfiltration? Things can get out of hand pretty quickly.

I suspect there may be some value to these questions in the world of accounting. Accountants spend much time determining assets and values. I've not yet looked into this, but I think my next project will be starting to understand how assets are dealt with by the business. Everything from determining value, to understanding loss. There is science here already, it would be silly for us to try to invent our own.

Leave your comments on Twitter: @joshbressers