A Minimal Raspberry Pi 3 GPS Time Server

A Raspberry Pi can make an outstanding NTP time server for your network. Below, I document the bare minimum needed to get a Raspberry Pi 3 Model B with the Adafruit Ultimate GPS Hat working as an NTP server.  You can do this, even buying all the parts new, for under $100 USD.

What makes this project interesting is that you end up with potentially a very compact time server – a Pi with a nicely stacked board on top. Just bring network and power to the box, put the box near a window, and it’s otherwise self-contained. For an even simpler deployment, in a corporate environment with PoE switches, use a PoE to USB cable like this one (disclaimer: this is an Amazon affiliate link, so I get a tiny amount of money if you buy something here): PoE to USB Splitter – then all you need is a network cable (no wall wart is required) which supplies connectivity and power.  Note that the PoE (Power over Ethernet) adapter requires that you have ethernet switches that support PoE.  Your business might have these, but you probably don’t have them at home (they cost a lot more money).

The only thing I’m missing is a good case – if anyone knows of a good Raspberry Pi 3 case that fits well with the Adafruit hat installed, I’m very interested — please comment!

Why do you want to do this?  There are several reasons:

  • You have a need for “correct” time on your servers (to do log correlation or for regulatory requirements)
  • You have an asymmetric internet connection.  There is no way for internet time servers to be more accurate than 10ms or so if you have this type of connection – virtually any DSL or cable internet connection.  On these connections, you have a faster download than upload.  That means that that it takes several ms (milliseconds) less time to travel from the internet to your machines than it takes to travel from your machines to the internet.  Because NTP measures round trip delay and divides it by two, assuming equal delay on both paths, your time will be off in such cases. Whether it’s off enough to bother you is a different question!
  • Related to the above, it is very unusual for internet backbone paths to be the same for both directions of your connection.  In fact, this is more common than not. This causes issues with NTP.
  • During network congestion (whether it’s a peering point, a congested NTP server, or your internet connection), you will see additional delay added, typically more in one direction than another.  Right now, NIST’s Maryland servers (time-a.nist.gov, time-b.nist.gov, time-c.nist.gov) have about 70ms of extra delay due to congestion at their end – this makes the time about 35ms off (70 divided by 2) for anyone synchronizing to their servers.
  • You need accurate time on a network not connected to the internet and/or during internet outages
  • You don’t want to be dependent upon volunteer server operators for the correct time
  • You might just like knowing your time is more correct than your friend’s!

There are lots of tutorials on the building NTP servers from Pis, but this one is a bit different because:

  • It’s based on the Pi 3 Model B instead of the 2 or older.  There are some serial oddities on the Pi 3, so you do need to do minor modifications.
  • Most tutorials connect to gpsd via shared memory to fetch the time of day while using the kernel PPS module to get the seconds to start at the right instant.  This means you can have time that is several hundred milliseconds off reality upon startup of NTP.  This can be mitigated slightly by fudging the stratum of the GPS and having internet time services also configured, but it can’t be eliminated completely.
  • I don’t generally manage computers from the console, I use network-based access to do so.  Most tutorials assume it’s convenient to haul a monitor out to the Pi.  This tutorial never requires you to connect a monitor (but you can if you want).
  • This tutorial uses versions of ntpd and gpsd I compiled from more recent sources than come with the stable branch of Raspbian.  The ntp code also has a patch I’ve made to allow it to see both the GPS and PPS devices from gpsd easily.

So, here’s the procedure: Continue reading

Secrets to Good Home Wifi

You open up your laptop, enter your password, and then start your browser. Maybe you click on a movie.

What happens next? For far too many homes, what happens next is your browser starts playing and then rebuffers the movie. Maybe you get kicked off your wifi completely. The ISP gets blamed – and may be at fault, but probably not for the reason you are yelling at them. The problem is most likely not your internet connection. It’s your wireless network. These tips are for experienced IT people who are comfortable reading IT system manuals, but who don’t have strong expertise in wireless 802.11 networking. Wifi is, sadly, not a “plug-and-play” technology – certainly anyone can learn about it and become an expert, but sometimes it’s better to hire an expert. If you need your Wifi to work like your home’s electricity, and always be there when you need it, sometimes you’ll need to hire experts in the field. But I’m going to assume that you are more of a do-it-yourself type of person who is comfortable learning more about their technology, and are able to read and understand your network devices’ manuals. With that in mind, how do we fix this?
Continue reading

Fixing OSX’s Broken External Display Support

I have two external monitors on my iMac – both connected via mini-DP to DP cables. I didn’t want to spend the money Apple charges for displays, so these are third-party displays with specs very similar to Apple’s displays.

But…the text on them looks like crap. It’s jagged, blocky, and just generally ugly – why? The iMac built-in monitor displays text beautifully.

It turns out that if I go to About This Mac, then to System Report, then to Graphics/Displays, I can see the problem: OSX thinks my two external monitors are televisions. This has color space issues and causes anti-aliasing of text to be disabled. In other words, the display noticeably looks like crap.

I suspect this is a way for Apple to discourage use of third party hardware – I’d like to believe it isn’t, but since my displays have a native resolution unlike that of any TV, it would be simple for Apple to differentiate between TV and monitor. Even better, they could provide an easy interface to override this guess. Besides, Windows and Linux look beautiful on these monitors without any tweaking, so unless anyone thinks Apple lacks the brilliant minds of Microsoft, this isn’t an impossible problem to solve – particularly because Apple has had many years to fix it.

Back to the topic, how do you fix it? Fortunately, others have figured out how to deal with this problem – rather than repeating the solution here, just go to the I Reckon blog and read Force RGB mode in Mac OS X to fix the picture quality of an external monitor .

Making PAR::Packer Work with RPM

I needed to distribute some RPMs with a PAR::Packer created (using pp) executable.

If you don’t know what PAR::Packer is, it is the incredibly awesome Perl 5 utility that allows you to create an executable that contains your Perl interpreter, your code, and any libraries you need.  It’s basically one of the ways you have to escape dependency hell with Perl – you can use all the wonderful CPAN modules without fear. You can use the new features of recent versions of Perl (such as subroutine signatures and enhanced performance), without worrying about what version of Perl is installed on someone else’s machine. You don’t need to worry about missing core modules in Fedora and other platforms (Fedora is missing autodie – an essential module for lazy programmers).  So building an archived executable is a great way to distribute Perl apps to people not comfortable with the Perl toolchain.

You can think of it like a Perl compiler (it’s not, but the distinction isn’t terribly important for this context).

It’s a somewhat finicky beast, and can take some effort to get it to package things the way you need them, but it’s a standard trick of the trade when I can’t just use perlbrew to build what I want on a machine. It certainly is a lot quicker to install a single executable than to compile and test hundreds of CPAN modules!

So, it seems natural to combine this with RPM. This way, users can install my stuff just by doing a simple rpm --install <file>.rpm – that’s awesome.  Well, it would be, but building an RPM with a PAR::Packer built executable doesn’t actually work, at least not on Red Hat Enterprise Linux 5 (RHEL5) or RHEL6.

Let me explain….

Take this script, hello.pl:

#!/usr/bin/env perl

use v5.22; # Use perl 5.22's features
use strict;
use warnings;

use autodie; # This causes issues on Fedora out of the box

    say reverse (split //, "!dlroW olleH");

If I run it on most distributions, line 3 will give me an error (the OS Perl isn’t new enough). On Red Hat/Fedora installs, if I changed line 3 to support older perl interpreters, the inclusion of autodie in line 7 will error out, unless that module has already been installed.

For instance:

perldemo:demo$ ./hello.pl
Perl v5.22.0 required--this is only v5.18.2, stopped at hello.pl line 3.
BEGIN failed--compilation aborted at hello.pl line 3.

Of course it works fine when I use my local perl interpreter (a perlbrew-built 5.22.1):

perldemo:demo$ perl hello.pl
Hello World!

So, I can create a packed executable, which runs great (at least on other systems with the same system libraries – the same limitations a an executable you dynamically link and build using C):

perldemo:demo$ pp -o hello hello.pl
perldemo:demo$ ./hello
Hello World!

Perfect! Note that I ran the executable hello, not hello.pl. The hello file is a self-contained executable that doesn’t depend on the system perl.

So what happens when I put this in an RPM package?  I won’t go through the steps of building an RPM, but will let you see what error you will get when you try running the executable:

perldemo:demo$ ./hello
Usage: ./hello [ -Alib.par ] [ -Idir ] [ -Mmodule ] [ src.par ] [ program.pl ]
./hello [ -B|-b ] [-Ooutfile] src.par

What the heck? It’s expecting a PAR file to be passed to it – the PAR, or Perl ARchive, of course contains the perl interpreter, your code, and your modules – for some reason the file can’t find it.

You can duplicate this without RPM by using strip on the output of pp on Linux:

perldemo:demo$ pp -o hello hello.pl
perldemo:demo$ ./hello
Hello World!
perldemo:demo$ strip ./hello
perldemo:demo$ ./hello
Usage: ./hello [ -Alib.par ] [ -Idir ] [ -Mmodule ] [ src.par ] [ program.pl ]
./hello [ -B|-b ] [-Ooutfile] src.par

What is going on? The strip removes debugging information and the like, but generally isn’t expected to change how your program runs – but in this case it seems to be removing a lot more. Running this same strip on OSX (Apple) is enlightening:

perldemo:demo$ strip hello
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: /Users/jmaslak/pp/hello
perldemo:demo$ ./hello
Hello World!

So it appears Macs won’t strip the archive out of the extraction executable produced by pp – but Linux has no problem happily stripping it.

It turns out that standard configurations of rpmbuild on at least RHEL5 and RHEL6 automatically strip all binaries in the binary package (that’s the “standard” package you would install).

So how do you disable this evil? It turns out it’s not easy. On RHEL6, you add this near the top of your SPEC file (the SPEC file rpmbuild uses to create the RPM):

%global __os_install_post %{nil}

Now, rpmbuild won’t strip the file. But this doesn’t help RHEL5. So I added another line, that did what I want (assuming I don’t need a debug package, which I don’t in this case):

%define debug_package %{nil}
%global __os_install_post %{nil}

Sure enough, now my code does exactly what I want, even when I install the PAR executable via RPM:

perldemo:demo$ ./hello
Hello World!


A Fix for Perl SSL on MacOS X 10.11

When trying to install some Perl modules on MacOS X, using a perlbrew-built perl, I was getting some weird linking errors with various OpenSSL headers.  Here’s an example from Net::SSLeay:

laptop$ cpan install Net::SSLeay
Configuring M/MI/MIKEM/Net-SSLeay-1.72.tar.gz with Makefile.PL
CPAN::Reporter not installed.  No reports will be sent.
*** Found OpenSSL-0.9.8z installed in /usr
*** Be sure to use the same compiler and options to compile your OpenSSL, perl,
    and Net::SSLeay. Mixing and matching compilers is not supported.
cc -c   -fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -O3   -DVERSION=\"1.72\" -DXS_VERSION=\"1.72\"  "-I/Users/jmaslak/perl5/perlbrew/perls/perl-5.22.1/lib/5.22.1/darwin-2level/CORE"   SSLeay.c
SSLeay.xs:163:10: fatal error: 'openssl/err.h' file not found
#include <openssl/err.h>
1 error generated.
make: *** [SSLeay.o] Error 1
  /usr/bin/make -- NOT OK

I highlighted the interesting parts – I run homebrew on my Mac to manage development tools and the like. Sure enough, it had installed openssl already:

laptop$ brew install openssl
Warning: openssl-1.0.2e already installed

Hmmm, version 1.0.2e, which doesn’t look like the 0.9.8z that Net::SSLeay found.

A bit of Googling and scratching my head and I found the magic incantation:

laptop$ brew link openssl --force

Once that was done, I could successfully install Net::SSLeay and other SSL Perl modules.  I’m guessing that the links broke sometime during an OS upgrade. Hopefully this post will save you a bit of time!

Perl is Good for Nothing

I love Perl – and the perl interpreter always impresses me.  Today, I decided to try a few languages to see how they compare.

How well does the language do nothing?  I decided to test this out on my fairly speedy Macbook Pro.  All tests were executed multiple times and the best result was used for this post, to account for various caching speedups.

First, C:

do-nothing$ touch nothing.c
do-nothing$ time clang nothing.c
Undefined symbols for architecture x86_64:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

real	0m0.041s
user	0m0.017s
sys	0m0.017s

So, C can spit out an error that will probably not make sense to people new to the language (it’s missing a main function, although usually you’ll define main without a leading underscore – for reasons I won’t get into here).  But it was fairly quick – about 40ms (I repeated this several times to account for caching).

Next I tried Java (using the SunOracle implementation):

do-nothing$ touch nothing.java
do-nothing$ time javac nothing.java

real 0m0.759s
user 0m1.323s
sys 0m0.105s

Java doesn’t throw any errors, but it takes over 750ms to compile nothing (in a somewhat satisfyingly mathematically pure way, it literally produces nothing – no output files are created). When I ran javac with the -verbose option (how a Unix workstation company would think long options with a single hypen are okay is beyond me, but I digress), it spits out some timing information. It takes 23ms to parse nothing and roughly 290ms to do the compilation. I can only assume the other 450ms or so are going to compiler startup overhead.

How about Ruby?

do-nothing$ touch nothing.rb
do-nothing$ time ruby nothing.rb

real 0m0.073s
user 0m0.053s
sys 0m0.010s

It takes 73ms to do nothing, but it does properly do nothing.

How about Python (v2)?

do-nothing:t$ touch nothing.py
do-nothing:t$ time python nothing.py

real 0m0.018s
user 0m0.009s
sys 0m0.007s

Python does nothing pretty darn well – 18ms!

Now, my language of choice, Perl 5:

do-nothing$ touch nothing.pl
do-nothing$ time perl nothing.pl

real 0m0.006s
user 0m0.002s
sys 0m0.003s

Brilliant – it does nothing very quickly compared to other languages – 6ms.

That said, my old version (Christmas) of the Rokudo-based Perl 6 takes roughly 250ms – not all that good. I’m not sure how the newer versions do. It’s certainly a powerful new language (you should think of Perl 5 and Perl 6 as distinct language – both are being actively developed with new features, optimizations, bug fixes, etc, added to both continually, with no plans to discontinue development on either).

So, It think, in conclusion:

  • C isn’t good for nothing
  • Java can’t do nothing quickly
  • Perl 6 can do nothing, but not too quickly
  • Ruby seems okay for nothing, while Python 2 is pretty darn good at nothing
  • Perl 5 is good for nothing!

(Yes, this post is 90% jest – startup time of the tools is important, but is almost always a dumb reason to pick a language)

Securing Against OpenSSH’s Less-Then-Perfect Defaults

There’s a general security principle: Lock everything up, and then unlock only what needs to be unlocked.

However, this is in contrast to OpenSSH‘s defaults for cryptography, which are, “Open most things up so that old clients can connect.”

Combination Lock

Photo by Self

For those who don’t know, OpenSSH is the most widely used SSH (Secure Shell) server – a tool used by sysadmins (and others) to access remote computers. It’s also the foundation of most Git (a version control system for software developers) security systems. It secures up some of the most sensitive parts of our systems: our intellectual property and “root” access.

Now, none of SSH’s choices are truly horrible, but the defaults certainly aren’t the strongest they could be, either. SSH defaults to allowing several less secure algorithms, along with some very strong algorithms. It does this for backwards compatibility.

However, backwards compatibility is not relevant to SSH in many environments.  Do you really need to support older, and somewhat less secure, cryptography algorithms because some admin might be using an ancient version of Putty?  Perhaps it’s time to get that person to upgrade!

So, here’s some quick notes – you can find more elsewhere – on OpenSSH. You might need to loosen some of these settings in your environment, but at least they’ll start fairly secure. I’ll put up another post on Apache at a later date.

For SSH:

Note you may have to alter paths or other items in the below snippets. Also note that I assume you have reasonable defaults from recent versions of OpenSSH – I.E. no version 1 support, no rsh fallback, etc.

In your sshd_config file (typically in /etc/ssh/sshd_config):

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
# HostKey /etc/ssh/ssh_host_dsa_key
# HostKey /etc/ssh/ssh_host_ecdsa_key
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com

What does this do? First, we comment out the DSA and ECDSA keys. DSA is on the cusp of being insecure (max key length is 1024 bits) while ECDSA uses NIST-supplied curves, which some believe have been backdoored by the NSA.

Next, I enabled two key exchange algorithms. Notably, I didn’t enable the SHA1 versions of the DH algorithms, or the ECDH algorithms with NIST curves (which some believe are backdoored by the NSA). Why use a known-to-be potentially less secure algorithm when there are algorithms that are secure?

For encryption, I picked a subset of ciphers I believe to be secure and re-ordered them more appropriate to my needs (but, again, your needs may differ). The big deal was that I didn’t want 3DES, CAST128, or RC4 (arcfour) in the list – these all have problems and shouldn’t be used if you don’t really need them.

For MACs (this verifies you are talking to the person you think you’re talking to, and someone isn’t changing the ciphertext), I removed all MD5 MACs and any MAC less than 128 bits. MD5 is known to have issues, so why use it? And longer MACs are good in this context.

I also like to use PasswordAuthentication no, assuming I’m in an environment where I can force the use of SSH keys or other non-password authentication choices. Passwords can be guessed, and you can bet you’ll see thousands of attempts to guess them, should your firewall allow people on the internet to try. But of course you need to set this up in advance.

Finally, there are two other things that need to be done – creating a stronger RSA host key and selecting non-standard Diffie Hellman parameters (this makes your system secure against Logjam).

The standard RSA key, at least on my Ubuntu system, was a 2048 bit key. That’s not a bad length, but I wanted something more secure. To do this:

ssh-keygen -b 4096 -t rsa -f /etc/ssh/ssh_host_rsa_key

This generates a 4096 bit RSA key. Note that users that had the old key stored in their .ssh/known_hosts file will have to remove the old key and add the new one (and they are validating signatures, right? As an alternative you might store the fingerprints in a DNSSEC-secured DNS zone).

To generate the non-standard DH paraemters (note this takes several hours on my machine):

ssh-keygen -b 2048 -G moduli.candidates -M 127
ssh-keygen -T moduli.2048 -f moduli.candidates
ssh-keygen -b 4096 -G moduli.candidates -M 127
ssh-keygen -T moduli.4096 -f moduli.candidates
cat moduli.2048 moduli.4096 >/etc/ssh/moduli

Of course you should read and understand what these do – not just take my word for it!

You’ll need to restart sshd after doing this (on Ubuntu, service ssh restart). Make sure you have a way of recovering access if you lock yourself out over SSH!

Finally, the changes you made to your sshd_config, particularly the algorithm choices, can be added to your ssh_config so your clients don’t try to use less secure algorithms. In the “Host *” section, add:

    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com

Of course these changes may prevent older clients from connecting to servers, or your client from connecting to older servers. If that’s the case, you need to rework the defaults (for the new client, old server problem, you can use a host-specific override in your ssh_config).

Finally, generate a longer user key pairs if you use SSH key-based authentication (you do this as the user you are generating the key for):

ssh-keygen -b 4096

The default key length is 2048, which is good, but 4096 is a bit better.

Will this keep the NSA out? Probably not, but at least you’re using better-than-default practices now.