28 February 2016

Hands-on Review: HardenedBSD

When it comes to operating system security in the traditional sense, FreeBSD has a strong track record (to say the least).  Having said that, there are several twenty-first-century exploit mitigation technologies that are woefully absent.  Chief among these, of course, is support for Address Space Layout Randomization (ASLR), which helps to reduce the risk of a whole host of vulnerabilities such as buffer overflow exploits and return-to-libc attacks.  There have also been various and sundry attempts to take on FreeBSD jails over the years, with varying degrees of success.

Though I can and do trust a well-configured FreeBSD system over other FOSS alternatives- for its ground-up, only-what-you-need approaches as much as any other reason- I'm also not about to turn my nose up at attempts to improve the situation.

Enter HardenedBSD, a fork of the FreeBSD code base that ports various PaX enhancements (such as as ASLR and W^X pages), and provides a few FreeBSD-specific improvements for items such as jails and procfs.  It's been called "grsecurity for FreeBSD" on more than one occasion.  The HardenedBSD projects seems ultimately to be interested in merging these protections into "vanilla" FreeBSD once they've matured and been well-tested by the community.

To that end, I decided to try HBSD on a mature and well-established FreeBSD workstation.  That was roughly two months ago now.  While I can say categorically that there are a few parts of HBSD that are "rough around the edges," on the whole it seems to be a very well-maintained and valuable contribution to the *BSD ecosystem.  Here's the synopsis of my conversation process (feel free to skip past the indented parts if you just want the closing summary):

First off, I decided to do an in-place conversion from "vanilla" FreeBSD 11.0-CURRENT to HardenedBSD 11.0-CURRENT.  This process is not documented anywhere, and your own mileage may vary.  Having said that, it went fairly well from my perspective. 
Since I was on ZFS, I began by setting a mountpoint of "none" on the filesystem containing my system source code, creating a new one with a mountpoint of /usr/src, and pulling the HardenedBSD source down to that.  ZFS deduplication left a total disk usage increase of a few megabytes, once all was said and done for the source code transfer.
The next step was to copy over my custom kernel config, merge in the HardenedBSD-specific items (I did enable ptrace protection), then build and boot into the HardenedBSD kernel.  This went precisely as expected, at least at first.  The early boot messages reflected the enabled security protections, and the system came up to multi-user.  Then came the first snag: the "non-hardened" PAM module for Kerberos didn't work, and failed all attempts.  Since I had the console and all vtys marked as insecure, I had to reboot single-user to log in. 
Naturally, this seemed like a good time to build world, as it seemed obvious that there must be a compatibility issue at work.  This proved to be the case, as a normal ZFS snapshot / make buildworld / mergemaster -p / make installworld / mergemaster followed by another reboot had me logging in with Kerberos again.  Miraculously, all of my out-of-tree kernel modules (namely the nvidia driver and pefs) apparently worked fine without being rebuilt, though I did rebuild them anyway for good measure. 
The final conversion step was to rebuild any ports with changed Makefile options, which I figured out quickly with a diff between the official FreeBSD ports tree and that maintained by HardenedBSD.  For the most part, these consisted of adding PIE support into Chromium, Firefox, and a few others.  HardenedBSD does have a binary package repo that works well, but I usually prefer to build from source anyway.  Sucker for punishment, I guess... 
X.org and SLiM worked out of the box, I logged in to MATE just fine, and tried to fire up the recently-rebuilt Firefox.  Nothing happened that I could see.  Here came the first real "gotcha."  HardenedBSD relies heavily on an in-house utility called secadm to control some of its more advanced functionality.  Though some broad configuration can be done with the usual sysctl mechanisms, secadm is used for e.g. exempting certain programs (ahem, Chromium and Firefox, again) from W^X protection. 
For whatever reason, I didn't get secadm with my installworld process.  Turns out that it's a port, including a kernel module and a userland tool.  Once I installed that, I tried to apply the exemptions recommended on the HBSD wiki for Firefox, Chrome, and Java.  Whoops again- the documentation on their wiki was out of date at the time (it's since been fixed) and didn't reflect a syntax change in their secadm tool's configuration file syntax.  I found an updated example here, fired it up, and voila!  I was off and running.

Two months later, and I've barely noticed the difference between HardenedBSD and stock FreeBSD.  As far as I can tell, that's a good thing.  There are a few extra steps needed in some places, however, and a few quirks of which to be aware:
  • Any file that is currently the subject of a secadm rule cannot be deleted or modified.  The rule must be temporarily removed to allow changes to the file.  It makes sense from a security perspective, but this led me to clearing out my secadm rule set when doing portupgrade runs, and then re-installing the rules using the provided rc script afterwards.
  • Care must be taken in jails, and in particular when using any of the non-W^X safe programs like the web browsers or Java in jails.  secadm is jail-aware, and any per-program exemptions from system protections are not carried from the host in to the jail.  These must be activated within the jail by having a secadm config file and enabling the appropriate rc script.
  • Linux binary compatibility works only if some or all of the PaX protections are disabled per-program or per-jail.  I haven't yet been able to find a reliable pattern.  If you have a small number of seldom-used Linux binaries, you'll probably be fine brute-forcing the secadm rules that work for them (as I've done), but if you rely heavily on many Linux binaries or full Linux jails in FreeBSD, you might want to hold off on HardenedBSD for now.
  • PEFS and HardenedBSD don't like eachother at the moment.  Filesystems will mount and key addition works fine, and the filesystem will behave fine for most normal usage, but e.g. attempts to execute scripts (bash, Python, Ruby, you name it) from within a PEFS mountpoint using "./" syntax will often trigger a kernel panic.  You can get around this by explicitly calling the interpreter first, e.g. "bash ./my-script" though that's a tough habit to remember.  Normal binaries work fine though.  I need to delve in to this, and file a patch or at least a PR with the HBSD folks soon (or perhaps the PEFS folks, as the case may be).
That's really it.  Though there are definitely a few small kinks to work out, I think that the HBSD team has done a remarkable job.  For going completely off-the-map on the in-place conversion, I had a remarkably smooth experience, and my own two months without serious problems suggest the HBSD has a bright future ahead.  You can, of course, always download the pre-built versions and go through the normal installation from scratch, but where's the fun in that?

Hands-on Review: FreeBSD 11

It's often fun to think of myself as some bold pioneer for running the bleeding-edge, fresh-from-Subversion, hours-old version of an operating system.  Terms like "kernel panic" and "data loss" (even an occasional "kill your puppy") abound in the tutorials and instructions for individuals who run FreeBSD's most current development code, designated appropriately enough as -CURRENT.

In reality, however, in my 3+ years of indulging this habit on some of my workstations, it's very rarely been a problem.  Sure, there was that little foobar in early 2015 where the RNG had been badly broken for months and I had to regenerate all of my private keys.  That's really the only major issue that I can recall, though, from multiple years of running the latest development branch of code.  Not too shabby, methinks.

That being said, why would I do this to myself at all?  Well, there is a certain level of excitement to being able to try the latest features and improvements before they roll off of the presses.  While some might consider running -CURRENT on a production workstation to be risky, I find it much safer than other adrenaline-rush options.  Base jumping, for instance.

So, what's cooking for the next major release of FreeBSD?  You can see a relatively complete list here, but the following are some of my personal highlights from the upcoming 11.0, scheduled for release in mid-2016:
  • Bhyve1 now supports loading specially-built firmware binaries, incl. a UEFI implementation.
  • There's now a FreeBSD port of Docker.
    • No, that's not a joke.  Yes, it's amazing.  
    • It integrates tightly with ZFS and jails, and can run either native FreeBSD docker images or those based on CentOS 7 or earlier, using... (wait for it)...
  • 64-bit Linux binary compatibility
  • Netmap is now supported by libpcap.  This allows e.g. user-land network traffic analysis by existing tools on par with that of ntop's PF_RING driver for Linux, including a zero-copy mode!
  • If you're a Xen fan, early support is now available for running FreeBSD as dom0.
    • Update: this is available in 10-STABLE, and looks slated to be included with 10.3 when that version ships.
  • KMS / DRM2 now includes early support for Intel's Haswell-generation integrated graphics
  • Support for 64-bit mode on ARM CPUs that support it is coming along nicely, with the first target hardware being the Cavium ThunderX server.
  • Initial reference implementation for running CloudABI binaries.
    • This is a truly cross-platform binary interface that will run as-is, no recompilation necessary, on any supporting *nix platforms.  Several BSDs offer support at the moment, and patches for the Linux kernel are in early testing stages.
  • The bundled version of xz has been updated, supporting multi-threaded compression out of the box.
  • Clang and libcompiler_rt continue to make significant progress on non-x86 platforms.2
  • Both IPX and AppleTalk have finally been yanked.3
  • Some networking performance improvements (a.k.a. SR-IOV) for FreeBSD on Amazon EC2
  • Capsicum capabilities are moving out to protect more utilities, including ping.
Those are just a few of the more exciting developments to me, but suffice it to say that users of 10.x and earlier versions have a lot about which to be excited this year.  If you're feeling adventurous, you could check it out now.

I might be biased, but I suspect that 11.0 will be one of the most game-changing releases in recent memory, on par with or even surpassing the legendary 7.0 release of 2008.

Coming soon: an early review of HardenedBSD

Footnotes
1 - FreeBSD's native hypervisor
2 - Sorry, GCC folks.  I like the compiler and tool chain better.  I'm a convert.  Hate me if you like.
3 - Ancient layer 3 networking protocols with virtually no presence in the modern world