Web dev in DC http://ross.karchner.com
586 stories
·
15 followers

Are voting-machine modems truly divorced from the Internet?

1 Comment and 2 Shares

(This article is written jointly with my colleague Kyle Jamieson, who specializes in wireless networks.)

[See also: The myth of the hacker-proof voting machine]

The ES&S model DS200 optical-scan voting machine has a cell-phone modem that it uses to upload election-night results from the voting machine to the “county central” canvassing computer.  We know it’s a bad idea to connect voting machines (and canvassing computers) to the Internet, because this allows their vulnerabilities to be exploited by hackers anywhere in the world.  (In fact, a judge in New Jersey ruled in 2009 that the state must not connect its voting machines and canvassing computers to the internet, for that very reason.)  So the question is, does DS200’s cell-phone modem, in effect, connect the voting machine to the Internet?

The vendor (ES&S) and the counties that bought the machine say, “no, it’s an analog modem.”  That’s not true; it appears to be a Multitech MTSMC-C2-N3-R.1 (Verizon C2 series modem), a fairly complex digital device.  But maybe what they mean is “it’s just a phone call, not really the Internet.”  So let’s review how phone calls work:

The voting machine calls the county-central computer using its cell-phone modem to the nearest tower; this connects through Verizon’s “Autonomous System” (AS), part of the packet-switched Internet, to a cell tower (or land-line station) near the canvassing computer.

Verizon attempts to control access to the routers internal to its own AS, using firewall rules on the border routers.  Each border router runs (probably) millions of lines of software; as such it is subject to bugs and vulnerabilities.  If a hacker finds one of these vulnerabilities, he can modify messages as they transit the AS network:

Do border routers actually have vulnerabilities in practice?  Of course they do!  US-CERT has highlighted this as an issue of importance.  It would surprising if the Russian mafia or the FBI were not equipped to exploit such vulnerabilities.

Even easier than hacking through router bugs is just setting up an imposter cell-phone “tower” near the voting machine; one commonly used brand of these, used by many police departments, is called “Stingray.”

I’ve labelled the hacker as “MitM” for “man-in-the-middle.”  He is well positioned to alter vote totals as they are uploaded.  Of course, he will do better to put his Stingray near the county-central canvassing computer, so he can hack all the voting machines in the county, not just one near his Stingray:

So, in summary: phone calls are not unconnected to the Internet; the hacking of phone calls is easy (police departments with Stingray devices do it all the time); and even between the cell-towers (or land-line stations), your calls go over parts of the Internet.  If your state laws, or a court with jurisdiction, say not to connect your voting machines to the Internet, then you probably shouldn’t use telephone modems either.

Read the whole story
rosskarchner
19 hours ago
reply
in Fairfax County VA, at least, what matters are the vote reports printed by the DS200’s at the end of the night. I don’t think the dial-out feature is used at all
DC-ish
Share this story
Delete

On Django's longevity

2 Shares

Django is about to be a teenager.”

I don’t remember exactly who said that, but it came up in a discussion of potential followup events to 2015’s “Django Birthday”, held in Lawrence, Kansas, to celebrate the tenth anniversary of Django’s initial public release. There might still be some parties to throw — maybe a Django sweet sixteen — but regardless of what the future holds, the summer of 2018 will mark thirteen years since ...

Read full entry

Read the whole story
rosskarchner
1 day ago
reply
DC-ish
acdha
1 day ago
reply
Washington, DC
Share this story
Delete

The Technology Transformation Services launches new job site

1 Share

Today, we’re happy to announce the first open position on the new hiring page for the Technology Transformation Services (TTS). JoinTTS will be the new home for all TTS job postings, including open 18F jobs.

Besides 18F, TTS is also home to an acquisitions team, the TTS Front Office, an operations group, the Presidential Innovation Fellows program, and the Office of Products and Platforms, which operates all kinds of helpful services like USA.gov and FedRAMP. The new JoinTTS page has more details about all these offices, timelines for what to expect when applying to TTS, and information about the interview and security clearance process.

The TTS Talent Team is preparing additional open positions, but today we’re looking for four product managers to work at 18F.

As a Product Manager at 18F, you’ll lead cross-functional teams to deliver user-centered products using agile methodologies and modern software development practices while building capacity for product innovation in government. When you build products with a partner agency, you’ll coach them on modern product development practices so they’re set up for success in the long term.

If that sounds like you, head over to JoinTTS to learn more and apply. We’re accepting applications only until Monday, February 26 at 11:59 p.m. ET, so hurry! Make sure you don’t miss an open job post by following 18F on Twitter and subscribing to our newsletter.

Read the whole story
rosskarchner
2 days ago
reply
DC-ish
Share this story
Delete

The Problem With Algorithms

1 Share

Algorithms

Facebook's algorithms are in the news a lot these days, and with good reason. But this isn't about Facebook. I've been thinking about algorithms in general lately. This has been brought on by my wife's insulin pump, the Medtronic 670G. It's the first closed loop pump approved by the FDA. That means it can monitor your blood sugar levels (actually an approximation as it's not testing your blood) and turn your insulin supply on or off as needed to avoid dangerous lows or highs. If you run it on automatic mode it will micro bolus insulin to try to keep you glucose level between 120-180.

That sounds great in theory. However, the "decision" to shut down or reestablish insulin delivery is driven by an algorithm that we have no legal right to know the details on. This thing could literally kill my wife, and we have no real idea how it actually works. We have only a general idea based on observation of the conditions that cause it to do something with insulin delivery. Compounding the concern is that we have documented that the blood glucose monitor accuracy varies with the rate of change of blood glucose readings. If your glucose levels are rising or falling rapidly the CGM readings can lag an actual blood sugar test by 30% or more. So we have a black box system making decisions with less than accurate data in some cases.

O take another, more speculative example. Imagine an autonomous vehicle doing 55 mph in heavy traffic. A person jumps into the path of the car, braking won't stop the vehicle in time, and swerving right or left will cause an accident and potentially endanger additional innocent people. What does the vehicle do? Now imagine that the developer of the algorithm had certain beliefs about the value of lives based on skin color, and programmed skin color recognition into the danger avoidance algorithm, so that the vehicle might react differently based on the skin color of the potential victim.

If you think this is too far-fetched to be worthy of conversation you haven't been paying attention.

Algorithms are going to continue to take on more and more of the decisions traditionally made by people. What rights do we have to know the underlying logic of these algorithms? It's already in play with things like credit scores, which can pretty much ruin your life. It's only a matter of time before an otherwise innocuous seeming algorithm kills somebody, if it hasn't already happened.

Today we have no rights. I think we should we should have a lot of rights to these logic powering these algorithms.

Read the whole story
rosskarchner
5 days ago
reply
DC-ish
Share this story
Delete

Seven years at Docker

2 Shares

TL,DR: I have left Docker Inc. to take a sabbatical and recover from depression and burnout. I plan to dedicate the next six months to family, friends, meditation, music, and generally speaking, enjoy life to recharge for whatever will come next.

This text is an adaptation of the message that I sent last week to my coworkers to announce my departure. I’m now sharing it with a wider audience, because mental health is serious stuff, and I wish we all felt more comfortable talking about it. I also wanted to share with my friends, the Docker community, the container ecosystem, and beyond, some thoughts about what has been for me an incredible journey.

February 6th was my last day at Docker. Seven years and one day earlier, I boarded a big bird of metal that would take Sam Alba, Sébastien Pahl, and me from Paris to San Francisco, and we joined the dotCloud office on Third Street. I couldn’t imagine What Would Happen Next.

Five persons around a table in a conference room

The dotCloud office at Founder’s Den, early 2011. (Credit: SFGate)

From dotCloud to Docker

In 2011, our tiny startup was fearlessly competing with Heroku, which had just been acquired by Salesforce for $250M. We were the first PaaS to support so many languages and databases, thanks to the extensive use of this obscure kun-tay-nerr technology. You could count our engineering team on one hand, and all of us were both on-call and doing customer support. We had weekly contests about who would solve the most support cases.

In 2012, I gave my first “real” talk at a “real” conference. It was about the other cool piece of tech at dotCloud: our ZeroRPC library. (One of the Xooglers who joined us back then even told us, “I wish we had something that simple and straightforward at Google!”) I’m grateful for the incredible work that my peers had put into this project, as it enabled me to speak at PyCON, and encouraged me to try and speak at more conferences.

In 2013, you know what happened: Solomon Hykes presented Docker at the same PyCON conference (one year later), and over the following months, the whole dotCloud engineering team shifted to Docker. Meanwhile, I gave my first container talk at the SCALE conference in Los Angeles; and after that talk, I was invited to present Docker in Beijing, and then in Moscow. These were incredible opportunities, both personally (I forged some long-lasting friendships during these trips) and professionally: thanks to our combined efforts, we were able to issue joint statements with Baidu and Yandex, announcing that they were now using Docker!

From SRE manager to evangelist

In 2014, I gave an average of two talks per week; but most importantly, I spoke at LinuxCon, OSCON, and LISA. I would have been satisfied with my career if it had given me the opportunity to attend these conferences; but now I was speaking there (and would be, multiple times). Again, this wouldn’t have been possible without the fantastic work done by the Docker core team. Being a developer advocate or an evangelist is generally hard; but it’s markedly easier when your product is as helpful and as approachable as Docker. That year, I also turned down an invitation to speak at AWS re:invent because they didn’t have a code of conduct back then. (They eventually added one; probably not by my sole request, but I like to think that it contributed!)

In 2015, [HEAVY SPEAKING INTENSIFIES]. I enabled our partners in Europe by training about a hundred customers and other trainers in a couple of weeks, and gave my first keynotes in Paris and São Paulo. For the first time, I found the courage to speak on stage about sexism and harassment in open source communities, and the reactions I got made me realize that these problems were far worse and more prevalent than I had thought. I was on stage 7 times at LinuxCon that year, and I still don’t know if that deserves an entry in the wall of fame or shame. I finally spoke at re:invent, and it was ridiculous. During that whole time, I was helped and empowered by the whole Docker team to give my best: engineering was always here for me if I had a tricky last-minute technical question; and I could also rely on everyone else in the company for logistics and overall support. That made a huge difference.

From busy to burned out

In 2016, in addition to my regular talks, I delivered an increasing number of orchestration workshops. Unfortunately, that’s also when I found my limits. I should have been kinder with myself; but I didn’t realize it until it was too late: my mental state deteriorated until I was diagnosed with depression in October. Fortunately, by that time, the company had many fantastic speakers among its ranks; and the Docker Captains program had taken off — so there was no negative impact when I shifted my focus.

I started antidepressants and therapy. Results were not encouraging at first; but after switching medication twice and finally being referred to a psychatrist, my symptoms became easier to manage. I started having more energy, so I used it to take care of myself and do things that would make me happy. Cooking fine meals. Reading. Learning the cello. Dating. Building cool stuff with Raspberry Pis. Eventually, things got better.

In 2017, I continued to deliver workshops, and I helped to shape DockerCon’s Black Belt track. It’s hard to find words to describe how much joy and satisfaction I drew from this opportunity. In Austin, the Black Belt track is the track that got the highest ratings and attendance. I also improved the diversity of that track: in Copenhagen, the majority of the talks featured a speaker from a traditionally underrepresented background. Reaching out to these outstanding speakers, helping them when necessary, sometimes coaching them, has been one of the most rewarding steps of my career; and there again, I would never have been able to do it without the full support of our team.

Group picture with the Black Belt Track speaker at DockerCon 2017 in Austin.

Black Belt track speakers from DockerCon 2017 in Austin.

In the summer of 2017, while participating in a study about mental health, expatriation, and remote teams in the tech industry, I took the Maslach Burnout Inventory. The MBI is a test to assess burnout factors. I was in the red zone. Alas, neither my GP nor my psychiatrist knew much about burnout, and I felt on my own. Out of sheer coincidence, I ended up talking to a doctor who was more knowledgeable on that topic. I will write more about this in the future; but long story short, I in September, I decided that I needed to take a break in 2018.

Before taking that break, I focused my energy on Docker’s Kubernetes strategy. One week after we announced support for Kubernetes plans in Copenhagen, I was delivering a Kubernetes workshop at ContainerCon; and I delivered that workshop 3 times internally at Docker (which gave me the perfect opportunity to visit our Raleigh office and hang out with the wonderful folks there!). The materials are available on kube.container.training, by the way.

The last two months of 2017 were a grueling struggle to figure out what would be the best way for me to take that break. I wanted to take at least 6 months off, which is more than the 12 weeks allowed by the FMLA. (The FMLA allows employees to take up to 12 weeks of unpaid leave.) Docker doesn’t have a sabbatical program, and didn’t want to create one. My doctors didn’t want to fill out the paperwork that would have allowed me to take a medical leave of absence. Switching doctors wouldn’t help because filling that kind of paperwork for mental health reasons requires to be seen over a longer period of time; and I didn’t want to wait 3 or even 6 more months — to perhaps be denied my leave anyway. So my only solution was to quit. This would have been a financially difficult proposition, but I was able to sell a large chunk of my equity in Docker in 2017, meaning that I have a comfortable safety net for now.

From startup to sabbatical

In 2018, I’m going to take a lot of time for myself. I’m learning Rust. I’m writing a tiny Ableton clone to connect a grid controller (like the Monome or the LaunchPad) to a Raspberry Pi to play live music. I’m going to do a Vipassana meditation retreat. I hope to mentor folks who weren’t as lucky and privileged as I was, and be a better ally. The first step was to quit Docker, and that was the most difficult one; but the road ahead looks great.

A lot of people have asked me if I would be joining Heptio / Microsoft / some other company, and some folks asked if I’d be open to some consulting gigs. First of all, while I would be humbled and honored to be deemed fit to work with teams like Heptio’s or some of the Azure folks, I don’t plan on going back to full-time employment until at least September. As for consulting, sure! You can contact me here.

From me to you

One last thing — all the achievements that I listed above are not mine alone. I assume that you mostly saw my happy, productive, engaging side during all these years; but one person in particular also had to deal with me when I was heavily depressed, exhausted, struggling to perform the simplest tasks, and much less interesting to be around. My partner since 2014 supported me unconditionally all that time, and helped me walk through some of my darkest moments. I owe her more than words can tell.

I also owe to a very long list of coworkers, friends, and everything in between. If we’ve worked together or collaborated in any way; if you’ve been a supportive ear or even just a smile during these years — I want you to know that these successes are also yours. I hope that our paths will cross again and that the future holds many opportunities to help each other.

Peace,

jpetazzo out 🎤💨🤚

Read the whole story
acdha
1 day ago
reply
Washington, DC
rosskarchner
5 days ago
reply
DC-ish
Share this story
Delete

Monitoring and Observability

1 Share

Ah, observability, the new buzzword of the day. Monitoring vendors aplenty are using the word, to basically mean “better monitoring!” You know, #monitoringlove not #monitoringsucks. Because monitoring doesn’t help with debugging and doesn’t have app instrumentation right?

Well, I have to say “bah” to that.  So here’s the thing.  I’m an electrical engineer by education, and I spent a lot of time working at National Instruments, an engineering test and measurement company.  You may be surprised to know these terms have actual definitions that don’t require Twitter arguments to discover.

Monitoring is an activity you perform. It’s simply observing the state of a system over a period of time.

Why do we monitor? For three reasons, in general.

  • Problem Detection – you know, alerting, or seeing issues on dashboards.
  • Problem Resolution – root cause and troubleshooting.
  • Continuous Improvement – capacity planning, financial planning, trending, performance engineering, reporting.

How do we monitor?  Well, that’s called instrumentation. You can instrument your systems and get CPU and stuff, you can use synthetic probes, you can use JavaScript bugs to get end user monitoring, you can emit metrics from applications, you can introspect services and apps via whatever parts are exposed (from JMX to nginx stats to sysdig traces), you can take network traces… (Some folks are similarly trying to redefine “instrumentation” to just mean application instrumentation, which is lame, and in defiance of the fact that application performance management tools that do app instrumentation have existed for decades.)

You can instrument metrics or events; metrics have certain sampling frequency and resolution…

So what is observability?  This isn’t a new term. It comes from system control theory. You know, the stuff that makes your A/C system and electrical plants and your car work.

Observability is a measure of how well the internal states of a system can be inferred from knowledge of its external outputs.

Observability is a property of a system. You can monitor a system using various instrumentation, but if the system doesn’t externalize its state well enough that you can figure out what’s actually going on in there, then you’re stuck.

So is observability hippy bullcrap?  No, of course not. In a DevOps world, it’s very important that the apps and systems concentrate on making themselves both observable and controllable (I leave it to the reader to research controllability, unless I get agitated enough to post about that too). Do you make yourself “easy to monitor”?

Externalizing custom metrics contributes to observability (you know, like with dropwizard metrics).  So does good logging.  So does proper architecture!  Take a system that sticks all kinds of messages into one message queue rather than using separate queues for separate types – the latter is more observable; you can more readily see how many of what is flowing through.  (It’s more controllable too, as you can shut off one queue or another.)

Making your system observable is therefore important, so that if you monitor it with appropriate instrumentation, you understand the state of the system and can make short or long term plans to change it.

While a monitoring tool can definitely contribute to this via its innovation in instrumentation, analysis, and visualization, in large part observability is a battle won or lost before you start sticking tools on top of the system. It’s very important to take it into account when designing and implementing services. No tool is going to “give you” observability and that’s the usual silver bullet fallacy heard from someone who wants to sell you something.

I’m not saying every vendor is using the term wrongly (in fact I just came across this New Relic post that is very well done), but I have to say I am less than impressed when common engineering terms are so widely misused and misunderstood widely in our industry.

Would you like to know more?  Peco and I are working on a new lynda.com course on monitoring and observability!  There’ll be real engineering, a broad canvas of the different kinds of monitoring instrumentation, tips on implementation and use… We’ve both been using and/or building monitoring tools for decades now so we hope to have some useful info for you.





Read the whole story
rosskarchner
7 days ago
reply
DC-ish
Share this story
Delete
Next Page of Stories