Tuesday, September 20, 2016

Is dialup still an option?

TL;DR - No.

Here's why.

I was talking with my Open Source Security Podcast co-host Kurt Seifried about what it would be like to access the modern Internet using dialup. So I decided to give this a try. My first thought was to find a modem, but after looking into this, it isn't really an option anymore.

The setup


  • No Modem
  • Fedora 24 VM
  • Firefox as packaged with Fedora 24
  • Use the firewall via wondershaper to control the network speed
  • "App Telemetry" firefox plugin to time the site load time

I know it's not perfect, but it's probably close enough to get a feel for what's going on. I understand this doesn't exactly recreate a modem experience with details like compression, latency, and someone picking up the phone during a download. There was nothing worse than having that 1 megabyte download at 95% when someone decided they needed to make a phone call. Call waiting was also a terrible plague.

If you're too young to understand any of this, be thankful. Anyone who looks at this time with nostalgia is pretty clearly delusional.

I started testing at a 1024 Kb connection and halved my way down to 56 (instead of 64). This seemed like a nice way to get a feel for how these sites react as your speed shifts down.

Baseline

I picked the most popular english language sites listed on the Alexa top 100. I added lwn.net becuase I like them, and my kids had me add twitch. My home Internet connection is 50 Mb down, 5 Mb up. As you can see, in general all these sites load in less than 5 seconds. The numbers represent the site being fully loaded. Most web browsers seem to show something pretty quickly, even if the page is still loading. For the purpose of this test, our numbers are how long it takes a site to fully load. I also show 4 samples because as you'll see later on, some of these sites took a really really long time to load, so four was as much suffering as I could endure. Perhaps someday I'll do this again with extra automation so I don't have to be so involved.

1024 Kb/s

Things really started to go downhill at this point. Anyone who claims a 1 megabit connection is broadband has probably never tried to use such a connection. In general though most of the sites were usable from a very narrow definition ofh the word.

512 Kb/s


You're going to want to start paying attention to Amazon, something really clever is going to happen, it's sort of noticeable in this graph. Also of note is how consistent bing.com is. While not the fastest site, it will remain extremely consistent through the entire test.

256 Kb/s

Here is where you can really see what Amazon is doing. They clearly have some sort of client side magic happening to ensure an acceptable response. For the rest of my testing I saw this behavior. A slow first load, then things were much much faster. Waiting for sites to load at this speed was really painful, it's only going to get worse from here. 15 seconds doesn't sound horrible, but it really is a long time to wait.

128 Kb

Things are not good at 128 Kb/s. Wikipedia looks empty, it was still loading at the same speed as our fist test. I imagine my lack of an ad enhanced experience with them helps keeps it so speedy.

56 Kb

Here is the real data you're waiting for. This is where I set the speed to 56K down, 48K up, which is the ideal speed of a 56K modem. I doubt most of us got that speed very often.

As you can probably see, Twitch takes an extremely long time to load. This should surprise nobody as it's a site that streams video, by definition it's expected you have a fast connection. Here is the graph again with Twitch removed.
The Yahoo column is empty because I couldn't get Yahoo to load. It timed out every single time I tried. Wikipedia looks empty, but it still loaded at 0.3 seconds. After thinking about this it does make sense. There are Wikipedia users who are on dialup in some countries. They have to keep it lean. Amazon still has a slow first load, then nice and speedy (for some definition of speedy) after that. I tried to load a youtube video to see if it would work. After about 10 minutes of nothing happening I gave up.

Typical tasks

I also tried to perform a few tasks I would consider "expected" by someone using the Internet.

For example from the time I typed in gmail.com until I could read a mail message took about 600 seconds I did let every page load completely before clicking or typing on it. Once I had it loaded, and the AJAX interface timed out then told me to switch to HTML mode, it was mostly usable. It was only about 30 seconds to load a message (including images) and 0.2 seconds to return to the inbox.

Logging into Facebook took about 200 seconds. It was basically unusable once it loaded though. Nothing new loaded, it loads quite a few images though, so this makes sense. These things aren't exactly "web optimized" anymore. If you know someone on dialup, don't expect them to be using Facebook.

cnn.com took 800 seconds. Reddit's front page was 750 seconds. Google News was only 33 seconds. The newspaper is probably a better choice if you have dialup.

I finally tried to run a "yum update" in Fedora to see if updating the system was something you could leave running overnight. It's not. After about 4 hours of just downloading repo metadata I gave up. There was no way you can plausibly update a system over dialup. If you're on dialup, the timeouts will probably keep you from getting pwnt better than updates will.

Another problem you hit with a modern system like this is it tries to download things automatically in the background. More than once I had to kill some background tasks that basically ruined my connection. Most system designers today assume everyone has a nice Internet connection so they can do whatever they want in the background. That's clearly a problem when you're running at a speed this slow.

Conclusion

Is the Internet usable on Dialup in 2016? No. You can't even pretend it's maybe usable. It pretty much would suck rocks to use the Internet on dialup today. I'm sure there are some people doing it. I feel bad for them. It's clear we've hit a place where broadband is expected, and honestly, you need fast broadband, even 1 Megabit isn't enough anymore if you want a decent experience. The definition of broadband in the US is now 25Mb down 3Mb up. Anyone who disagrees with that should spend a day at 56K.

I know this wasn't the most scientific study ever done, I would welcome something more rigorous. If you have any questions or ideas hit me up on Twitter: @joshbressers

Sunday, September 18, 2016

Why do we do security?

I had a discussion last week that ended with this question. "Why do we do security". There wasn't a great answer to this question. I guess I sort of knew this already, but it seems like something too obvious to not have an answer. Even as I think about it I can't come up with a simple answer. It's probably part of the problems you see in infosec.

The purpose of security isn't just to be "secure", it's to manage risk in some meaningful way. In the real world this is usually pretty easy for us to understand. You have physical things, you want to keep them from getting broken, stolen, lost, pick something. It usually makes some sort of sense.

It would be really easy to use banks as my example here, after all they have a lot of something everyone wants, so instead let's use cattle, that will be more fun. Cows are worth quite a lot of money actually. Anyone who owns cows knows you need to protect them in some way. In some environments you want to keep your cows inside a pen, in others you let them roam free. If they roam free the people living near the cows need to protect themselves actually (barbed wire wasn't invented to keep cows in, it was used to keep them out). This is something we can understand. Some environments are very low risk, you can let your cattle roam where they want. Some are high risk, so you keep them in a pen. I eagerly await the cow related mails this will produce because of my gross over-simplification of what is actually a very complex and nuanced problem.

So now we have the question about what are you protecting? If you're a security person, what are you really trying to protect? You can't protect everything, there's no point in protecting everything. If you try to protect everything you actually end up protecting nothing. You need to protect the things you have that are not only high value, but also have a high risk of being attacked/stolen. That priceless statue in the pond outside that weighs four tons is high value, but nobody is stealing it.

Maybe this is why it's hard to get security taken seriously sometimes. If you don't know what you're protecting, you can't explain why you're important. The result is generally the security guy storming out screaming "you'll be sorry". They probably won't. If we can't easily identify what our risk is and why we care about it, we can't possibly justify what we do.

There are a lot of frameworks that can help us understand how we should be protecting our security assets, but they don't really do a great job of helping identify what those assets really are. I don't think this a bad thing, I think this is just part of maturing the industry. We all have finite budgets, if we protect things that don't need protecting we are literally throwing money away. So this begs the question what should we be protecting?

I'm not sure we can easily answer this today. It's harder than it sounds. We could say we need to protect the things that if were lost tomorrow would prevent the business from functioning. That's not wrong, but electricity and water fall into that category. If you tried to have an "electricity security program" at most organizations you'll be looking for a new job at the end of the day. We could say that customer data is the most important asset, which it might be, but what are you protecting it from? Is it enough to have a good backup? Do you need a fail-over data center? Will an IDS help protect the data? Do we want to protect the integrity or is our primary fear exfiltration? Things can get out of hand pretty quickly.

I suspect there may be some value to these questions in the world of accounting. Accountants spend much time determining assets and values. I've not yet looked into this, but I think my next project will be starting to understand how assets are dealt with by the business. Everything from determining value, to understanding loss. There is science here already, it would be silly for us to try to invent our own.

Leave your comments on Twitter: @joshbressers

Monday, September 12, 2016

On Experts

Are you an expert? Do you know an expert? Do you want to be an expert?

This came up for me the other day while having a discussion with a self proclaimed expert. I'm not going to claim I'm an expert at anything, but if you tell me all about how good you are, I'm not going to take it at face value. I'm going to demand some proof. "Trust me" isn't proof.

There are a rather large number of people who think they are experts, some think they're experts at everything. Nobody is an expert at everything. People who claim to have done everything should be looked at with great suspicion. Everyone can be an expert at something though.

One of the challenges we always face is trying to figure out who is actually an expert, and who only thinks they are an expert? There are plenty of people who sound very impressive, but if they have to deal with an actual expert, things fall apart pretty quick. They can get you into trouble if you're expecting expert advice. Especially in areas like security, bad advice can be worse than no advice.

The simple answer is to look at their public contributions. If you have someone who has ZERO public contributions, that's not an expert in anything. Even if you're working for a secretive organization, you're going to leave a footprint somewhere. No footprint means you should seriously question a person's expertise. Becoming an expert leaves a long crazy trail behind whoever gets there. In the new and exciting world of open source and social media there is no excuse for not being able to to show off your work (unless you don't have anything to show off of course).

If you think you're an expert, or you want to be an expert, start doing things in the open. Write code (if you don't have a github account, go get one). Write blog posts, answer questions, go to meetups. There are so many opportunities it's not even funny. Just because you think you're smart doesn't mean you are, go out and prove it.

Tuesday, September 6, 2016

You can't weigh risk if you don't know what you don't know

There is an old saying we've all heard at some point. It's often attributed to Donald Rumsfeld.

There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know
If any of us have ever been in a planning meeting, a variant of this has no doubt come up at some point. It came up for me last week, and every time I hear it I think about all things we don't know we don't know. If you're not familiar with the concept, it works a bit like this. I know I don't know to drive a boat. But because I know I don't know this, I could learn. If you know you lack certain knowledge, you could find a way to learn it. If you don't know what you don't know, there is nothing you can do about it. The future is often an unknown unknown. There is nothing we can do about the future in many instances, you just have to wait until it becomes a known, and hope it won't be anything too horrible. There can also be blindness when you think you know something, but you really don't. This is when people tend to stop listening to the actual experts because they think they are an expert.

This ties back into conversations about risk and how we deal with it.

If there is something you don't know you don't know, by definition you can't weight the possible risk with whatever it is you are (or aren't) doing. A great example here is trying to understand your infrastructure. If you don't know what you have, you don't know which machines are patches, and you're not sure who is running what software, you have a lot of unknowns. It's probably safe to say at some future date there will be a grand explosion when everything start to fall apart. It's also probably safe to say if you have infrastructure like this, you don't understand the pile of dynamite you're sitting on.

Measuring risk can be like trying to take a picture of an invisible man. Do you know where your risk is? Do you know what it should look like? How big is it? Is it wearing a hat? There are so many things to keep track of when we try to understand risk. There are more people afraid of planes than cars, but flying is magnitudes safer. Humans are really bad at risk. We think we understand something (or think it's a known or known unknown). Often we're actually mistaken.

How do we deal with the unknown unknowns in a context like this? We could talk about being agile or quick or adaptive, whatever you want. But at the end of the day what's going to save you is your experts. Understand them, know where you are strong and weak. Someday the unknowns become knows, usually with a violent explosion. To some of your experts these risks are known, you may just have to listen.

It's also important to have multiple experts. If you only have one, they could believe they're smarter than they are. This is where things can get tricky. How can we decide who is actually an expert and who thinks they're an expert? This is a whole long complex topic by itself which I'll write about someday.

Anyway, on the topic of risk and unknowns. There will always be unknown unknowns. Even if you have the smartest experts in the world, it's going to happen. Just make sure your unknown unknowns are worth it. There's nothing worse than not knowing something you should.

Monday, August 29, 2016

How do we explain email to an "expert"?

This has been a pretty wild week, more wild than usual I think we can all agree. The topic I found the most interesting wasn't about one of the countless 0day flaws, it was a story from Slate titled: In Praise of the Private Email Server

The TL;DR says running your own email server is a great idea. Almost everyone came out proclaiming it a terrible idea. I agree it's a terrible idea, but this also got me thinking. How do you explain this to someone who doesn't really understand what's going on?

There are three primary groups of people.

1) People who know they know nothing
2) People who think they're experts
3) People who are actually experts

If I had to guess, most of #3 knows running your own email server is pretty dangerous. #1 probably is happy to let someone else do it. #2 is a dangerous group, probably the largest, and the group who most needs to understand what's going on.

These ideas apply to a lot of areas, feel free to substitute the term "security" "cloud" "doughnuts" or "farming" for email. You'll figure it out with a little work.

So anyway.

A long time ago, if you wanted email you basically had to belong to an organization that ran an email server. Something like a university or maybe a huge company. Getting a machine on the Internet was a pretty big deal. Hosting email was even bigger. I could say "by definition this meant if you were running a machine on the Internet you were an expert", but I suspect that wasn't true, we just like to remember the past as being more awesome than it was.

Today anyone can spin up a machine in a few seconds. It's pretty cool but it also means literally anyone can run an email server. If you run a server for you and a few other people, it's unlikely anything terrible will happen. You'll probably get pwnt someday, you might notice, but the world won't end. How do we convince this group that just because you can, doesn't mean you should? The short answer is you can't. I actually wrote about this a little bit last year.

So if we can't convince them what do we do? We get them to learn. If you've ever heard of the Dunning Kruger effect (I talk about it constantly), you understand the problem is generally a lack of knowledge.


You can't convince experts of anything, especially experts that aren't really experts. What we can do though is encourage them to learn. If we have someone we know is on the peak of that curve, if they learn just a little bit more, they're going to fall back to earth.

So I can say running your email server is a terrible idea. I can say it all day and most people don't care what I think. So here's my challenge. If you run your own email server, start reading email related RFCs, learn about things like spam, blacklisting, greylisting, SPF. Read about SMTPS, learn how certificates work. Learn how to mange keys, learn about securing your clients with multi factor auth. Read about how to keep the mail secure while on disk. There are literally more topics than one could read in a lifetime. If you're an expert, and you don't know what one of those things are, go learn it. Learn them all. Then you'll understand there are no experts.

Let me know how wrong I am: @joshbressers

Sunday, August 21, 2016

The cost of mentoring, or why we need heroes

Earlier this week I had a chat with David A. Wheeler about mentoring. The conversation was fascinating and covered many things, but the topic of mentoring really got me thinking. David pointed out that nobody will mentor if they're not getting paid. My first thought was that it can't be true! But upon reflection, I'm pretty sure it is.

I can't think of anyone I mentored where a paycheck wasn't involved. There are people in the community I've given advice to, sometimes for an extended period of time, but I would hesitate to claim I was a mentor. Now I think just equating this to a paycheck would be incorrect and inaccurate. There are plenty of mentors in other organizations that aren't necessarily getting a paycheck, but I would say they're getting paid in some sense of the word. If you're working with at risk youth for example, you may not get paid money, but you do have satisfaction in knowing you're making a difference in someone's life. If you mentor kids as part of a sports team, you're doing it because you're getting value out of the relationship. If you're not getting value, you're going to quit.

So this brings me to the idea of mentoring in the community.

The whole conversation started because of some talk of mentoring on Twitter, but now I suspect this isn't something that would work quite like we think. The basic idea would be you have new young people who are looking for someone to help them cut their teeth. Some of these relationships could work out, but probably only when you're talking about a really gifted new person and a very patient mentor. If you've ever helped the new person, you know how terribly annoying they become, especially when they start to peak on the Dunning-Kruger graph. If I don't have a great reason to stick around, I'm almost certainly going to bail out of that. So the question really is can a mentoring program like this work? Will it ever be possible to have a collection of community mentors helping a collection of new people?

Let's assume the answer is no. I think the current evidence somewhat backs this up. There aren't a lot of young people getting into things like security and open source in general. We all like to think we got where we are through brilliance and hard work, but we all probably had someone who helped us out. I can't speak for everyone, but I also had some security heroes back in the day. Groups like the l0pht, Cult of the Dead Cow, Legion of Doom, 2600, mitnick, as well as a handful of local people. Who are the new heroes?

Do it for the heroes!

We may never have security heroes like we did. It's become a proper industry. I don't think many mature industries have new and exciting heroes. We know who Chuck Yeager is, I bet nobody could name 5 test pilots anymore. That's OK though. You know what happens when there is a solid body of knowledge that needs to be moved from the old to the young? You go to a university. That's right, our future rests with the universities.

Of course it's really easy to say this is the future, making this happen will be a whole different story. I don't have any idea where we start, I imagine people like David Wheeler have ideas. All I do know is that if nothing changes, we're not going to like what happens.

Also, if you're part of an open source project, get your badge

If you have thoughts or ideas, let me know: @joshbressers

Monday, August 15, 2016

Can't Trust This!

Last week saw a really interesting bug in TCP come to light. CVE-2016-5696 describes an issue in the way Linux deals with challenge ACKs defined in RFC 5961. The issue itself is really clever and interesting. It's not exactly new but given the research was presented at USENIX, it suddenly got more attention from the press.

The researchers showed themselves injecting data into a standard http connection, which is easy to understand and terrifying to most people. Generally speaking we operate in a world where TCP connections are mostly trustworthy. It's not true if you have a "man in the middle", but with this bug you don't need a MiTM if you're using a public network, which is horrifying.

The real story isn't the flaw though, the flaw is great research and quite clever, but it just highlights something many of us have known for a very long time. You shouldn't trust the network.

Not so long ago the general thinking was that the public internet wasn't very trustworthy, but it all worked well enough that things worked. TLS (SSL back then) was created to ensure some level of trust between two endpoints and everything seemed well enough. Most traffic still passed over the network unencrypted though. There were always grumblings about coffee shop attack or nation state style man in the middle, but practically speaking nobody really took these attacks seriously.

The world is different now though. There is no more network perimeter. It's well accepted that you can't trust the things inside your network any more than you can trust the things outside your network. Attacks like this are going to keep happening. The network continues to get more complex, which means the number of security problems increases. IPv6 will solve the problem of running out of IP addresses while adding a ton of new security problems in the process. Just wait for the research to start taking a hard look at IPv6.

The joke is "there is no cloud, just someone else's computer", there's also no network, it's someone else's network. It's someone else's network you can't trust. You know you can't trust your own network because it's grown to a point it's probably self aware. Now you expect to trust the network of a cloud provider that is doing things a few thousand times more complex than you are? You know all the cloud infrastructures are held together with tape and string too, their networks aren't magic, they just have really really good paint.

So what's the point of all this rambling about how we can't trust any networks? The point is you can't trust the network. No matter what you're told, no matter what's going on. You need to worry about what's happening on the network. You also need to think about the machines, but that's a story for another day. The right way to deal with your data is to ask yourself the question "what happens if someone can see this data on the wire?" Not all data is super important, some you don't have to protect. There is some data you have that must be protected at all times. That's the stuff you need to figure out how to best do something like endpoint network encryption. If everyone asked this question at least once during development and deployment it would solve a lot of problems I suspect.