27 January 2015

Geeks vs. Normal People

Geeks react very differently to normal, sane people in some very important ways.  Use this handy guide to help you distinguish the two categories...

It's your first day on a new job, and you have no idea what to expect.  You sit down at your desk, and the computer screen looks like this:

"This is going to be awesome!  Let's get some real work done."

Normal Person:
*goes bug-eyed* ... "I think it's broken."

The latest fruity gadget was just announced at WWDC:

Normal Person:
"That looks nifty.  When can I start using it?"

"Great- another way that tens of thousands of people can have personal info stolen."

A [senior] relative offers to give you this laptop from the basement:


"Wow that's awesome!  You're serious- I can have it?  I wonder if I can put OS/2 on it..."

Normal Person:
"Yeah, um... no thanks, I'm good."

You suggest that you and your buddy watch a movie:

Normal Person:
"Sure!  I saw this great preview on Netflix for one..."

"Okay, let me fire up XBMC and you can browse the collection.  If there's nothing in there you like we can always torrent something."

The summary: geeks are special people, with special needs.  If you decide to befriend or fall in love with one, you had better a) be one yourself, or b) mentally prepare for the incessant stream of odd behavior.  Don't ever assume that anything will be normal.

24 January 2015

The Five People You'll Meet in Planning Committees

It's a scenario that many of us have faced at some point in our professional careers: you're in a meeting with coworkers or clients, and some seemingly-reasonable proposal is casually made by a relatively new coworker.  The firestorm that erupts would be enough to stun into silence even witnesses of the 1980 Mount St. Helens eruption.

When confronted with the reality of maintaining day-to-day operations in the face of a constantly evolving field, people react in different ways.  In an IT organization, recognizing these attitudes in co-workers and clients can mean the difference between success and failure of a project (and sometimes between life and death of your professional career).  Here are some of the types that I've encountered so far:

"The Wave Rider"

This individual is in love with the bleeding edge.  Whatever the latest and greatest gadgets, tools, tricks, and fads are, these folks are convinced that your organization must upgrade- regardless of whether they make sense, or are feasible in any real-world scenario.  You can recognize these individuals by their constant references to sites like Engadget, Mac Rumors, and Tech Crunch.
  • Pro: these individuals will be the prime mover for most improvements that are made
  • Con: sometimes those aforementioned movements happen to be off cliff edges
  • Nemesis: The Unretired

"The Greener Grasser"
Nearly always unhappy with the way things are currently being done, this is the certified professional complainer.  It doesn't matter how easy, simple, effective, or resilient the current site is; it doesn't matter how many dollars the replacement will cost- the status quo must change.  These wonderful creatures can be spotted a mile away by their near-incessant complaining about every aspect of your organization.
  • Pro: when these individuals stop complaining, you know you've got a truly winning product
  • Con: your annual earplug budget to deal with these folks will typically exceed any raises you get
  • Nemesis: The Tape Wrapper

"The Toll Troll"
Extortion is the name of the game that this ruthless cut-throat plays.  Any time an institutional purchase or policy change needs to be made, they somehow always manage to come out on top.  Need to deploy a new customer portal?  Be prepared to fund five new positions in HR.  Need to replace some aging network infrastructure?  You're going to have to remodel an entire floor first.  These folks can be spotted by their presence at meetings in which you'd think they have no vested interest, during which they talk of how inconvenient all of the changes will be to their departments / divisions.
  • Pro: once given the proper incentives, these folks will become your strongest allies
  • Con: over budget, every time
  • Nemesis: everybody else in the meeting, if united

"The Unretired"
Just as the undead are stuck between two natural states, so too is this individual.  They have given up all interest in anything but working through the day-to-day grind, and resent any changes in routine, which they deem utterly superfluous and counterproductive.  They can be picked out of a crowd quite handily by the the immensely annoyed face that they present whenever large-scale changes are discussed.
  • Pro: a veritable fountain of views both pragmatic and skeptical
  • Con: rarely, if ever, say anything but 'no' to new ideas
  • Nemesis: The Wave Rider

"The Tape Wrapper"
To say that this person has a singular obsession with rules would be like saying that Mozart was fond of music.  No project, undertaking, or visit to the restroom would be complete without a thorough recitation of institutional policy.  These odd characters can be spotted by sounding eerily cheerful when starting sentences with the words, "but our policy says that..."
  • Pro: can be relied on for a fair, unbiased, balanced, and considered opinion
  • Con: throw your project schedule out of the window
  • Nemesis: The Greener Grasser

As always, I'd like to know what you think.  Do you know which type you are?  Can you think of any additional types that should be on the list?

20 January 2015

An Afternoon Stroll with DTrace

So, for those of you who haven't heard, DTrace is a tool that's been around since late 2003, but has recently started gaining popularity.  It's an end-to-end instrumentation framework for system debugging and performance analysis, which allows developers to gain a complete picture of the environment in which an application is being deployed, from the kernel all the way down to individual library function calls.

To utilize it, you start with an OS kernel that has DTrace support.  At the moment, that means Solaris, FreeBSD, NetBSD, and Mac OS X, with an experimental Linux port well underway.

Then, you get your compiled / interpreted languages and dependent libraries with DTrace support.  It's available for just about every language used in client apps and server-side scripting today, with the merciful exception of that awful "dot" stuff.  There are also quite a few full application suites with end-to-end support as well, notably Firefox.

Once you're working with a complete toolset, you use the dtrace command to load scripts (written in the 'D' programming language, hence the name) that describe the things you want to know and, optionally, what action(s) to take based on that information.

For example, you might ask DTrace, "which of my application's threads take the longest CPU time to execute, and within those threads, which syscall in particular consumes the most time, and what's blocking that syscall that it takes so long?"  Such information is immensely useful- it can tell you about e.g. deadlocks between otherwise-independent threads in your application that want disk access at the same time, or resource conflicts between your application itself and other tools on which it depends.  Such information is virtually impossible to resolve with conventional debugging and tracing tools.

So, just how usable is DTrace?  Since I'm an avid FreeBSD user myself, and Erlang is my poison of choice for application development, I decided to give that combo a try.

First off, you need to have a FreeBSD kernel with DTrace support.  This has been the default in GENERIC kernels on Tier 1 platforms since 7.1, iirc.  If you're on a custom kernel, instructions can be found here.  It's worth noting that on -CURRENT, as usual, your mileage will almost certainly vary (and in odd and confusing ways).  Once you've got an appropriately-configured kernel, you still need to actually load the DTrace modules.

Second, you need an application and/or language suite with DTrace support.  Erlang staight from ports has this capability, but it's not enabled in the default build (so ditto binary pkgs).  Grab an up-to-date ports tree, cd /usr/ports/lang/erlang; sudo make config and turn on DTrace.  Build and install the port with the usual process.  If you're going with a language suite here, then you also need an application to run with it.  I just used my own (very immature, pre-alpha) EMiLE project as a crash-test dummy.

Finally, you need some probes to actually run.  There are plenty of demo scripts in
/usr/share/dtrace for 10.0 and later.  If you're running something earlier, you can get them with the sysutils/DTraceToolkit port.  I also found some demo goodies just for Erlang in this presentation.

Once you've got all of those things, you fire up your application, and run dtrace(1) with the appropriate arguments to run your command directly, or to point to your script file.  The results vary by your target, but are generally pretty amazing.  I discovered several resource concurrency issues and corrected them in about fifteen minutes thanks to DTrace, whereas I probably would have otherwise spent weeks hunting them down (if I found them at all).

The summary: apart from the headaches of using a new scripting language, if you're an application developer and you're not using DTrace, you're probably missing out on the 'deep view' of what your code is actually doing not just within its own confines, but to the hosting system at large.  It's definitely a tool that I'll be using going forward.

16 January 2015

Vexing the Snoops

There are a ton of interesting things in the recent congressional report on the NSA's rampant spying on the communications of the American public.  Now, like most normal folks, I don't have much of anything to hide, unless you count my obsession with cute cat photos or the time that I was really drunk and somehow thought that it would be a good idea to order workout-boosting supplements online from a sketchy "all-natural" vendor.  It took me two hours on the phone to cancel that junk...

But, I do believe that I have a basic right to privacy, and that exercising such a right at my discretion does not in any way form a legal justification that I have something to hide.  I have the right to have a private conversation with my friends without even having to wonder if some employee of the federal government will be reading that conversation at a later time.

To that end, I dug through the congressional report with the goal of finding out which privacy and anonymity technologies frustrate the NSA.  The short (and unpleasant answer): there is no single security technology available to typical consumers today that, alone, reliably protects against privacy intrusion.

There is hope, however.  Several technologies are mentioned in the report as being reasonably effective if deployed properly.  Furthermore, there are some basic strategies to ensure that your personal ramblings aren't the laughing-stock of the NSA water cooler.

Do Not:

  • Have any expectation of privacy when using major online services, like Google, Facebook, AIM, etc.  Just because you see that little padlock in your web browser doesn't mean your communications are safe.  In fact, the congressional report suggests that TLS alone, even with the latest protocols, is pretty much a joke, because the NSA has gained access to the data at the servers themselves.
  • Attempt to secure all communication.  If your communication at work or school, or between yourself and your doctor, is already covered by FOIA, FERPA, or HIPAA, then there's not much point in going security-crazy, as all of those already have legal access mechanisms built in.
  • Conduct any illegitimate commerce or business using ultra-secure communication mechanisms.  Uncle Sam gets really angry when he doesn't get his tax dollars, and that's likely to make you a priority target.  Oh, and don't be a terrorist- that'll do it too.  Once your sketchy commerce has been cracked open, which it almost inevitably will be, viz. Silk Road, your really embarrassing personal details are likely to follow suit.  In court.
  • Reveal extensive personal details about your life, identity, occupation, family, etc. to unknown persons, no matter how tempting it might be.
  • Take intelligent, informed steps to protect your personal, private, communication from prying eyes (more details on this to follow)
  • Feel free and unfettered to discuss whatever you want to discuss, even highly taboo or controversial subjects, provided that you take basic precautions.  The free and open exchange of ideas is the cornerstone of a free and open society, and fear of who might be listening should never hinder you in this regard.
  • Develop a close and trusted network of confidantes, with whom you trade secrets for secrets, dialog for dialog, idea for idea.

Now, the hot question of the day- how can one truly secure online communications in 2015?  Guess what- it is possible, but it's neither convenient nor fool-proof.  Convenience and security are mutually exclusive, and each time you augment one, you diminish the other.  That's the way it works- get over it.  To secure your online communications, the trick is simple: use layers.  In fact, layer like you're stepping out of a tent into an Antarctic Winter to go p*ss beside a frozen waterfall.

Examples of some good things to layer:
  • OTR (Off-the-Record) messaging: OTR can sit on top of virtually any other instant messaging protocol, including XMPP (Facebook, Google Talk, et al.), AIM, MSN, Yahoo!, and many more.  When used properly, it provides two-way trust relationships with your buddy, session encryption of messages, and perfect forward secrecy (which is really important in the age of supercomputers).
  • Tor (The Onion Router network): Tor is an anonymizing "hidden network" within the public Internet that, again, when used properly, allows two parties to communicate with reasonable security.  It's often abused to access public websites which store all sorts of unique data on you in other ways, but when your communication stays within the Tor network it's a reasonably effective tool, when layered with other mechanisms.
  • PGP (Pretty Good Privacy): it's a decades-old encryption technique utilizing public-key cryptography  that the recent report surprisingly suggests can still stymie snoops in some scenarios.  Like your house keys, however, you have to keep your PGP private keys secure, or all bets are off!
  • FDE (full-disk encryption) and/or FBE (file-based encryption): once an attacker has physical access to your computing devices, like your laptop, desktop, or mobile phone, all bets are off unless you're using comprehensive encryption.  Think that Windows or OS X login password alone protects you?  Think again- I can get past those in less than three minutes in most cases, and that's just as a casual effort.  If it's that easy for me, imagine how quickly an NSA snoop or a malicious hacker could do it, unless you pair a strong password with strong encryption... and even that's not a complete guarantee (hence the layering)
Using any two of those together is probably sufficient for most private communication- preferably combining a forward secrecy technology like OTR or Tor with a physical access protection like FileVault 2, Bitlocker, or eCryptFS.  If you want an extra layer of security, as a buffer, pick three.  Here are some examples, ranging from easiest to most difficult (note that all of these require your friends to have similar capability):

Easy: Turn on Bitlocker or FileVault 2, with strong passwords, and use OTR for your IM conversations with friends.  Make sure they have disk encryption enabled as well!

using Tor and Privoxy, create a free online e-mail account (make sure to set it up through Tor and only access it that way as well), and encrypt all the messages you send and receive with trustworthy PGP software, encrypting your private keys with strong passwords.

Difficult: Turn on FDE or FBE, set up your own XMPP micro-server with federation over Tor, and communicate with your friends' micro-servers via Onion hidden services, using OTR on your IM client(s).

So, to summarize, there's no such thing as guaranteed privacy.  Having said that, avoiding "high-risk activities," and using some of the privacy tricks mentioned above, can help ensure that your private communications stay... well... private!

15 January 2015

Protecting Sensitive Data in Erlang with Function Callbacks

Function callbacks are one of those programming tricks that, once discovered, seem difficult for one to live without.

The concept of a callback is simple: you create a function, "package" it in some way, and pass it as an argument to another function, to be executed wholesale at a later time.  Here's what the process looks like in Erlang:

Why would one do this?  The main reason in most languages is to be able to build a function's logic in stages, particularly when such stages cross different compilation units, with the final stage being the callback.  There are other reasons, though.  Erlang, in particular, has one added benefit: when callback functions are implemented as anonymous functions, then assigned to a variable, it becomes much more difficult for the callback function to be executed unintentionally.  An Erlang module can't export variables, only regular functions, so putting an anonymous function in a variable means that only other functions launched with the callback as an argument will ever know that it exists.

My personal favorite usage of callbacks in Erlang is to protect functions which should only be available to authenticated users.  It's also handy for building GUIs with e.g. wxWidgets (and Erlang's wx module).  I'd be interested to hear what other uses people have found for function callbacks; suffice it to say, though, that I'm hooked on them.

13 January 2015

Mind the Gap: why end users are constantly frustrated with tech support

It's a simple assertion- IT is not what it was a decade ago.  Like any profession or field of expertise, it's constantly evolving and changing.  In this particular case, it's growing, and rapidly.  According to the US Census Bureau's report on computer and Internet use, the number of US households with a computer has jumped by more than ten percent in the last decade, even as the population grew by twenty million people.  Of course, those census numbers aren't including today's smartphones in the study, many of which are more powerful than the typical desktop or laptop computers of 2003 anyway.  As if that's not enough, six of the thirty job fields projected to grow the most in 2016 are explicitly IT related.

But we have a big problem.  When IT was a reasonably compact field, it was straightforward for an individual to specialize in one particular aspect, but still to be reasonably well-versed in other areas as well.  One could be a network engineer in day-to-day duties, but still be a competent server or desktop hardware repair technician in a pinch.  As the market diversifies, that kind of cross-disciplinary knowledge is becoming less and less practical.  It's not that those in tech fields are getting lazy or complacent (though that does happen), but rather that there's just so much that it's impossible to keep up with it all.

Why is this a problem?  Consider another one of the premier and expansive professional fields with many parallels to IT: medicine.  In medical practice, you have individuals with basic, across-the-board knowledge (general practitioners and registered nurses).  If you have a problem that a "generalist" can't resolve, you go to a "specialist," e.g. an orthodontist, a podiatrist, etc.  That approach entails horrors in its own right, but let's assume for a moment that the model is sane.  What about IT generalists- who are they, and what can they do?  Think about it for a moment.  I'll wait...

Okay, so there's an answer, but it's not a pretty one.  Our IT generalists these days are the folks who staff help desks and call centers.  These individuals are required to have wide-reaching, general diagnostic knowledge across the whole IT spectrum.  They're expected to be able to diagnose and correct some of the most common issues, and are also expected to know when to pass the buck on to specialists.  Without these generalists, the specialists would be in an awful mess.  But the generalists are often some of the lowest-paid positions in IT, have some of the highest turnover rates and, to make matters worse, due to those other factors have very controlled and regulated interaction with those they're assisting.  That's particularly troubling when one considers that, like those in the medical field, those in IT are often treated with disdain, rudeness, and downright disrespect by those whom they serve.

Imagine for a moment that your family doctor(s) typically had less than five years of experience and rotated out every six months due to poor pay, bad hours, and/or moving on to more lucrative 'specialist' careers.  Now imagine that those temporary doctors only consulted with you by telephone, or for in-person appointments less than fifteen minutes.  If you're alarmed, that's good- it means we're on the right track here.  That's where your tech support is right now, and if you've ever been ready to lose your cool while trying to get tech help, that's the reason why.

The only solution immediately apparent to me, though there are almost certainly others, is to make the "IT generalist" a real, bona fide, reasonably-compensated profession with degrees, training, internships, and the rest to go with it.  It seems to me that until a few institutions grasp this basic concept, "big IT" is heading towards a cliff edge in a car with no brakes.

12 January 2015

fun() -> functional end.

There's something special about having obscure, dust-covered tidbits of knowledge.  Like a collection of rare books (though much easier to move) or an exotic flower garden (though requiring less water), collections of the mind tend to bring much joy and diversion.  The same principle, it seems, must apply to programming languages.  After years of working with more traditional languages like C / C++ and Fortran, and then delving into the exotic but almost useless ones like Forth and m68k assembly, practical circumstances finally forced me to learn a programming language that is slightly less confined to particular niches (when did C get that way?)... perhaps even a bit, dare I say, bourgeois.  Going too mainstream, of course, would be unthinkable, so I settled on Erlang.  That was more than six months ago now, and I haven't looked back.  The project, by the way, was an laptop encryption status reporting tool for Linux, and a corresponding web application for viewing the information.  There are plenty of them out there for Windows and Mac OS X, but none for our favorite FOSS platform.

What's so special about Erlang, you might ask?  Well, here are the highlights...

  • It relies heavily on the functional paradigm (more on that later)
  • There's built-in concurrency and message passing - no extra libraries needed
  • Typically, Erlang is compiled to byte-code, but can also be interpreted
  • It runs in a VM (comparing this to Java would be a disservice, let's avoid that)
  • The code of an module can be changed while the module is running, provided that a few basic rules are observed.
  • It's notoriously difficult to break and/or compromise, again provided that a few basic rules are observed
What is it particularly good at?  It's great for moving data from point 'A' to point 'B' based on intelligent, reliable, and rich decision-making.  It was originally created in the mid-80s to power telephone network switches.  Today, it shines as an integral component in products like ejabberd, WhatsApp, Facebook Messenger, Call of Duty, and Chef, to name just a few.

What is it particularly bad at?  Erlang is flat-out terrible at advanced manipulation of binary data.  Things like image and video processing, emulators, heuristic file scanning are out of its purview.

So, it's useful and has some big names associated with it, but is unknown to the general populace.  Perfect!  But where should one start?  The Erlang web site is fairly helpful, and will get you going in terms of getting the programming environment up and working (if you're on just about any GNU/Linux distro, it's probably already available for download from the normal software repo).  But that doesn't address what to do with the language.  If you like the modular, discrete, "learn it one small step at a time" approach, then exercism is for you.  In my case, I prefer the "sink or swim" approach, so after using the online version of Learn You Some Erlang for great good! as a primer, I started writing a simple demo point-of-sale system.  I wanted menus, tickets, and receipts.  And that's when the proverbial sh*t hit the fan...

Part of Erlang's magic is called referential transparency.  This means that any given expression (an action taken) could be replaced with its value (what it 'returns') without changing the way that the program works.  When referential transparency is maintained, Erlang is deterministic.  This is what allows hot code swapping to work, and also contributes to the system's message passing power.

...but if you're coming from imperative programming, it's your worst nightmare!

Take a simple 'for loop' in C:
for ( x = 0; x < a; x++ ) { printf( "Doing something...\n", x ); }
The concept is simple: you want to do something a variable number of times.  But because the value of x changes as the loop runs, and x could be used before or after the loop (try setting it as, e.g. a void and watch the hilarity!) this breaks referential transparency.  For this reason, in Erlang, 'variables' (which aren't actually variable) cannot have their value changed once defined, in any particular context.

Instead, to accomplish this same goal, you have to back up on your chain of reasoning, and ask what it is that you're trying to accomplish.  To just do the exact same thing as above, preserving referential transparency, in Erlang you might instead say:
myMessage( a ) -> io:write( string:copies( "Doing something...\n", a ) ).
In this example, one would simply call myMessage (a function) with the number of times a as the first argument.  This preserves referential transparency, i.e. any instance of myMessage( N ) could be replaced with the output value (which in this case would be the exit value of the io:write function, the atom ok).

Most probably, however, if you're using a 'for loop' in C, you're trying to do something with an array.  If you back up far enough on your chain of reasoning in Erlang, most such actions can be very quickly represented as list comprehensions, which (as the name suggests) build one list by processing one or more others in some way.  Say you want to convert a string, which is just syntactic sugar for a list and works the same way in Erlang, to uppercase... but remove all spaces.
[ string:to_upper( char ) || char <- "just a simple test", char /= 32 ]
In this case, you (the programmer) don't need to know how long the list is, or walk through it one item at a time- just tell Erlang how to process it, and let it go!

Now we've covered referential transparency, the pros and cons of Erlang, and how to go about getting started in it.   Soon I'll talk about the message passing and concurrency parts, and eventually get into anonymous function callbacks, one of my favorite Erlang tricks.

The Obligatory "Inaugural Post"

What brought this calamity on, you ask?  It's simple: learning.  As a seasoned IT professional with more than a decade of experience, I frequently encounter situations which require unorthodox solutions, highlight social issues, or are just downright comical.  Most of the time, all three categories apply.  Since it seems a shame to let all of these things go to waste, and I'm not sadistic enough to actually make people listen to these stories in person, I offer this collection for you to peruse at your own leisure.  Enjoy!