Monthly Archives: May 2014

Updating OpenSSL for Apache without breaking your base OpenSSL install on FreeBSD 9.2-RELEASE

After two very frustrating days, I have finally been able to perform what used to be a fairly routine thing for me, which is creating a FAMP server using FreeBSD, Apache 2.2.x, MySQL 5.x.x and PHP 5.5.x.

Several issues arose out of my attempts.

My first attempt was installing Apache 2.4.9, MySQL 5.6.17 and PHP 5.5.12 on FreeBSD 10.0-RELEASE-p3 (AMD64). This was my first time installing anything greater than MySQL 5.1.x. ‘./configure’ has been replaced with ‘cmake .’; other than that, all things ordinary. This was also my first time installing Apache 2.4.x and there are several new steps involved. The first is that you now have to manually install Apache’s portable runtime library and associated utilities. Piece of cake. Grabbed those from the APR website and installed without issue. Apache installed. But now the trouble. I was able to ‘./configure’ and ‘make’ PHP without issue, but during the ‘make install’ I ran into an issue.

# make install
Installing PHP SAPI module: apache2handler
/usr/local/httpd/build/instdso.sh SH_LIBTOOL='/usr/local/apr/build-1/libtool' libphp5.la /usr/local/httpd/modules
/usr/local/apr/build-1/libtool --mode=install cp libphp5.la /usr/local/httpd/modules/
libtool: install: cp .libs/libphp5.lai /usr/local/httpd/modules/libphp5.la
libtool: install: cp .libs/libphp5.a /usr/local/httpd/modules/libphp5.a
libtool: install: chmod 644 /usr/local/httpd/modules/libphp5.a
libtool: install: ranlib /usr/local/httpd/modules/libphp5.a
libtool: install: warning: remember to run `libtool --finish /usr/local/src/php-5.5.5/libs'
Warning! dlname not found in /usr/local/httpd/modules/libphp5.la.
Assuming installing a .so rather than a libtool archive.
chmod 755 /usr/local/httpd/modules/libphp5.so
chmod: /usr/local/httpd/modules/libphp5.so: No such file or directory
apxs:Error: Command failed with rc=65536

Keep in mind, this was a brand new, entirely fresh FreeBSD 10.0-RELEASE install with ‘freebsd-update’ run bringing it to -p3. Nothing else had been done on this box except the MySQL and Apache installs.

I found it very strange that in that small bunch of code, two separate libtools were referenced. Foolishly, I didn’t check my working server to compare. In any case, ‘/usr/local/apr/build-1/libtool’ and the libtool included with the PHP source were both referenced. For good measure, I even installed libtool using ‘pkg install’. Nothing worked and libphp5.so failed to be created. Not being a software developer, I went to Google looking for the answer but came up short. A few suggestions were to use the .bz2 version instead of the .tgz version, use ports and one person even pointed the finger at the AMD64 build of FreeBSD. I briefly attempted to install a known working version of libphp5.so from another one of my servers but that just created more issues. I moved on.

I reloaded the OS from scratch and tried again. Same result. I’m a sucker for punishment.

I then reinstalled the OS using FreeBSD 9.2-RELEASE. As before, I ran ‘freebsd-update’ first thing and brought it up to -6p. I compiled MySQL using the new steps I had learned earlier. Same with Apache, except I used 2.2.27 this time to simplify things; go with what you know. Then the true test, PHP 5.5.12, which also installed no problem. Added a few times to httpd.conf and started the server. Problem. I telneted to port 80 to see what version or OpenSSL I was using and it came back as 0.9.8y. This version isn’t vulnerable to the heartbleed vulnerability, but it does lack TLSv1.1 and TLSv1.2 support, which I need, if only to give my clients a greater sense of security. So I search Google for “freebsd upgrade openssl” and a few articles come up, including one I had recently visited. I went back, performed the needful using ports and OpenSSL was upgraded. Sort of. The base OpenSSL install has its binaries in /usr/bin, libraries in /usr/lib and config file in /etc. The port install puts the binaries in /usr/local/bin, the libraries in /usr/local/lib and the config file in /usr/local/etc/ssl. So I have two completely autonomous OpenSSL installs at this point. Typing ‘which openssl’ shows /usr/bin/openssl which is version 0.9.8y. ‘/usr/local/bin/openssl version’ shows 1.0.1g. Again, 0.9.8y is not vulnerable to heartbleed, but it is insufficient for my needs. After further reading, a few people suggested to remove the old binary and symlink /usr/local/bin/openssl to /usr/bin/openssl and also symlink the configuration files. This seemed to work, so I rebuilt Apache again thinking that version 1.0.1g would show up. But it didn’t. Knowing that this was a new install and that I would probably blow it away once I found a working formula, I started deleting things, symlinking things, moving things, etc. I reminded myself of my father-in-law who while looking to free up space on his internal HDD, he deleted his windows system files with the rational of “I’ve never used those, I can delete them.” A mess I made. Server trashed. Blank slate.

Again, I started with FreeBSD 9.2-RELEASE. I immediately updated using freebsd-update to -p6. I then downloaded and installed OpenSSL 1.0.1g from the OpenSSL website. I then confirmed that my base and secondary install were as expected:

# /usr/bin/openssl version         
OpenSSL 0.9.8y 5 Feb 2013

Good.

# /usr/local/ssl/bin/openssl version
OpenSSL 1.0.1g 7 Apr 2014

Great.

When it came to compiling Apache, I had to compile mod_ssl as a static module. In the process, a few of the core modules were also made static, so I had a mishmash of static and dynamic. I had to comment out a few of the LoadModule statements in the httpd.conf, but in the end, everything works as it should.

[root@server01 ~]# telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
HTTP/1.1 / GET

HTTP/1.1 501 Method Not Implemented
Date: Wed, 28 May 2014 23:07:36 GMT
Server: Apache/2.2.27 (Unix) DAV/2 PHP/5.5.12 mod_ssl/2.2.27 OpenSSL/1.0.1g
Allow: GET,HEAD,POST,OPTIONS,TRACE
Content-Length: 220
Connection: close
Content-Type: text/html; charset=iso-8859-1

Success! I now have a server running exactly the way I want and I have no compromised any of the base OpenSSL files, or anything that was built against them. While I think this was a fairly good way to accomplish this task, there are probably even better ways to do it. However, my Googling efforts did not lead me to those.

Another other method, which some will say is more elegant, would have been to install from pkg or ports. However, I disagree. Since I compiled my first web server twenty years ago to present day, I have always been an advocate of compiling critical services from source. If all you ever do is type ‘pkg install apache22’ how do you learn to troubleshoot when something goes wrong? How do you know how to fine tune your installation when you need to compile in sockets, or curl, or other libraries and modules? I prefer infinite control.

Juniper SRX data center firewall default security policy

Being a routing and switching guy (mostly service provider stuff), I don’t deal with firewalls very often, because I am far from a security expert and I would be doing a disservice to my clients. However, this week, I was tasked with getting two brand new Juniper SRX1400s setup and updated I learned three things during this process: 1) unlike the branch-series SRXs, the data-center series comes with very little configuration and no security configuration whatsoever; 2) the default policy if no policy is to deny, and; 3) there’s a difference between inbound host packets and transit packets, and at the very least, the inbound host configuration must be set before you can do anything to the firewall. Example below:

Show

zones {
    security-zone untrust {
        interfaces {
            xe-0/0/7.0 {
                host-inbound-traffic {
                    system-services {
                        all;
                    }
                }
            }
        }
    }
}

Display set

set security zones security-zone untrust interfaces xe-0/0/7.0 host-inbound-traffic system-services all

A couple caveats:

First, if this is the only policy you add, you’ll probably find people trying to brute force any services you have running on the firewall within minutes, such as SSHD.

Second, the statement must be applied to a logical interface and not a physical interface (xe-0/0/7.0 vs. xe-0/0/7) in order for it to work.

Forcing users to use HTTPS the simple and secure way

Very simple Apache configuration using mod_rewrite.

<VirtualHost 1.2.3.4:80 [2001:4800::2]:80>
DocumentRoot /home/someuser/websites/whatever.com/html
ServerName whatever.com
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
</VirtualHost>

Do not forget to create the SSL entry as well:

<VirtualHost 1.2.3.4:443 [2001:4800::2]:443>
DocumentRoot /home/someuser/websites/whatever.com/html
ServerName whatever.com
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
SSLEngine on
SSLProtocol -ALL +TLSv1 +TLSv1.1 +TLSv1.2
SSLHonorCipherOrder On
SSLCipherSuite "YOURPREFEREDSUITE"
SSLCompression off
SSLCertificateFile /home/someuser/ssl/whatever.com.crt
SSLCertificateKeyFile /home/someuser/ssl/whatever.com.key
SSLCertificateChainFile /home/someuser/ssl/gd_bundle.crt
</VirtualHost>

Rackspace FreeBSD VPS and their lack of swap

Update #2 – 2015-05-19 – Three months after my first update I noticed that Rackspace has removed their FreeBSD VPS offering. Not surprising and probably for the best. Two clients that I recommend Rackspace to, that are also using FreeBSD VPS, are experiencing the same issues, so I know I am not alone. I have moved my mission-critical services to a Leaseweb bare-metal server. I’ve been with them for five months now. 100% uptime and no issues whatsoever. There’s twice been a 30ms jump in latency from where I am monitoring the server from; perfectly acceptable performance, if not amazing by server-farm standards. I am considering moving to Packet as I have just been told they intend to support FreeBSD very soon. They also offer a BYoIP (bring your own IP) service, which is extremely attractive as we have an IPv4 /23 and an IPv6 /48.

Update #1 – 2015-02-19 – After what seems like one of my servers hanging hundreds of times, and in reality is probably at least 60, I have moved off of Rackspace’s FreeBSD VPS for all my mission critical sites. Rackspace dev ops seem very uninterested in putting any effort into their FreeBSD product. Additionally, few of their Linux ops can support it. After all the swap nonsense and being affected by “noisy neighbors”, I’ve moved onto a bare-metal server. So far, zero downtime. A decade ago, I envied Rackspace’s product and support. Now, they are a far cry from Softlayer, Packet.net and many others who have left them in the dust. Their now lackluster support and lack of innovation could see them completely fall off the map. It’s quite sad to see how far the mighty have fallen.

Last October, one of the best VPS and bare-metal server providers on the planet, Rackspace, quickly fell from grace with many of its VPS customers. They made a decision — a poor one at that — to eliminate swap from their default Linux and FreeBSD VPS builds. They claim that this was in an effort to reduce disk IOPS, but in reality, they’ve achieved nothing but instability. Here’s their KB article:

http://www.rackspace.com/knowledge_center/article/changes-to-swap-on-cloud-servers

Notice at the bottom they’ve provided a link for how Linux users can add a swap file. Also notice that there’s no mention of how to add a swap file using FreeBSD. I’ll get to that in a second. First, let’s break down what they’ve actually accomplished:

They’ve not actually reduced IOPS at all, because nearly everyone running a VPS will have manually created a swap file, because if you don’t, it’s possible that this can happen:

Apr 20 08:20:37 monitor kernel: pid 10861 (xenstore), uid 0, was killed: out of swap space
Apr 20 08:20:37 monitor kernel: pid 172 (nova-agent), uid 0, was killed: out of swap space
Apr 20 08:20:38 monitor kernel: pid 778 (sshd), uid 0, was killed: out of swap space
Apr 20 08:20:38 monitor kernel: pid 965 (devd), uid 0, was killed: out of swap space
Apr 20 08:20:38 monitor kernel: pid 2345 (sendmail), uid 0, was killed: out of swap space

Yes, that’s right, whilst compiling MySQL, and apparently running out of memory on a box that supposedly has 2GB of RAM, the server killed several critical processes.

Because you cannot have custom partitioning on their FreeBSD VPS, they force you to choose from one of many of the already capacity-deprived file systems in which to create the swap file.

Filesystem                   Size    Used   Avail Capacity  Mounted on
zroot                        2.4G    205M    2.1G     9%    /
devfs                        1.0k    1.0k      0B   100%    /dev
zroot/tmp                    2.1G    424k    2.1G     0%    /tmp
zroot/usr                    5.5G    3.3G    2.1G    61%    /usr
zroot/usr/home               3.8G    1.6G    2.1G    43%    /usr/home
zroot/usr/ports              3.1G    996M    2.1G    31%    /usr/ports
zroot/usr/ports/distfiles    2.2G     41M    2.1G     2%    /usr/ports/distfiles
zroot/usr/src                2.4G    276M    2.1G    11%    /usr/src
zroot/var                    2.2G    3.4M    2.1G     0%    /var
zroot/var/crash              2.1G    148k    2.1G     0%    /var/crash
zroot/var/db                 2.3G    185M    2.1G     8%    /var/db
zroot/var/db/pkg             2.2G    6.0M    2.1G     0%    /var/db/pkg
zroot/var/empty              2.1G    144k    2.1G     0%    /var/empty
zroot/var/log                2.1G    1.7M    2.1G     0%    /var/log
zroot/var/mail               2.1G    1.4M    2.1G     0%    /var/mail
zroot/var/run                2.1G    296k    2.1G     0%    /var/run
zroot/var/tmp                2.1G    1.7M    2.1G     0%    /var/tmp

Which file system are you likely to throw a 2GB swap file on? This partitioning at its absolute worst. I’m using a grand total of about 200MB of the nearly 18GB of space allocated to all the many /var file systems.

But, I digress. Here’s how to add a swap file to your VPS:

Requirements

You’ll need the memory disk driver install, which comes by default with the generic kernel. Rackspace at least had the foresight to keep this in their kernel.

device     md

Next, find a good place for your swap file. This will likely be determined by your partitioning. I’m going to create a 1024 MB swap file in /usr/swap and name is swap0. On my server the /usr file system is the largest partition on the server, so it makes the most sense. Next, I am doing to create the actual swap file:

dd if=/dev/zero of=/usr/swap/swap0 bs=1024k count=1024

Adjust the count to adjust your swap file size. 1 count is equal to 1 MB.

Next, change the swap file permissions:

chmod 0600 /usr/swap/swap0

Remember to add this new swap file to your /etc/rc.conf file:

swapfile="/usr/swap/swap0"

To enable the new swap without rebooting, use the following commands:

mdconfig -a -t vnode -f /usr/swap/swap0
swapon /dev/md0

When you enter the mdconfig command, it will list the device name for your new swap file. Mine was md1. Yours may be different.

Adding a hard drive to FreeBSD with a larger than 512-byte sector size

Introduction

I recently tried to add another USB HDD drive to my FreeBSD system and forgot the steps I had taken previously to add a USB drive with a larger than 512-byte sector size and larger than 2TB capacity.

The sysinstall utility uses fdisk to try and add disks. This is the same utility that was present when I first started using FreeBSD way back in 1996 (2.1.5-RELEASE). Things have changed dramatically since. The first Gigabyte-sized hard drives had just hit the market a year or two prior. If you were lucky enough to own a massive 3.2GB HDD, you were the envy of all your friends. 56.6k was already pretty bitchin’ too. So, needless to say, as technology has evolved, the built-in FreeBSD fdisk tool has become antiquated. However, there is a replacement, and it’s so much better. Its name is gpart.

Forgetting that I had used gpart previously (I won’t forget again, since I am documenting this for my own benefit as well), I tried the sysinstall method first. This failed for a couple reasons. The first is that fdisk assumes the 512-byte sector size and I had a 4096-byte sector size. Because of this, the geometry was showing my 3TB drive as a 350GB drive. Because it never partitioned properly, the labeling tool didn’t work either. After reading a few mailing lists and blog posts about failed fdisk attempts, I remembered that I used gpart the last time around as well. Here’s step-by-step directions on how I was able to very quickly and easily get my USD HDD partitioned, labeled and mounted.

Assumptions

  • You’re installing a brand new hard drive that has no data on it, or
  • You understand that any data currently on the hard drive will be erased using this method
  • You’re using the entire disk as a single partition
  • You understand that I am not liable for any data you may lose following these directions. 🙂

Finding your HDD

I cannot give exact detailed instructions here as your drive name will depend on several factors: is your hard drive internal hard drive or an external portable USB hard drive; how many drives are currently installed; and perhaps even by which USB port you have plugged your drive into.

In my case, it was an external USB HDD. To easily find it when I plugged it in, I ran the following command:

tail -f /var/log/messages

The following was appended to the file and onto my screen:

May  3 16:50:32 gateway kernel: da2 at umass-sim2 bus 2 scbus2 target 0 lun 0
May  3 16:50:32 gateway kernel: da2:  Fixed Direct Access SCSI-6 device 
May  3 16:50:32 gateway kernel: da2: 40.000MB/s transfers
May  3 16:50:32 gateway kernel: da2: 2861556MB (732558336 4096 byte sectors: 255H 63S/T 45599C)

Easy peasy. My HDD was da2.

Learning more about your HDD

To find out what size your sectors are, the diskinfo utility is great:

diskinfo -c da2

You should see something like this:

da2
        4096            # sectorsize
        3000558944256   # mediasize in bytes (2.7T)
        732558336       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        45599           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        574343344530383939393633        # Disk ident.

Checking contents

Once your drive is installed and you’ve located it, it’s time to see what’s on the drive. Since most USB drives come from the store formatted to work with one operating system or another, it’s likely not entirely blank, but it needs to be for gpart to work. To see what’s on your drive, use the following command:

gpart show da2

In my case, I can see the partition that fdisk failed to create properly:

[root@gateway ~]# gpart show da2
=>       63  732558273  da2  MBR  (2.7T)
         63  732547872    1  freebsd  [active]  (2.7T)
  732547935      10401       - free -  (40M)

Deleting existing slices and partitions

Above we saw a single slice with the index of 1. To delete that slice, use the following command:

gpart delete -i 1 da2

The -i flag is used to specify which index to use. This was the result of running that command:

[root@gateway ~]# gpart delete -i 1 da2
da2s1 deleted

[root@gateway ~]# gpart show da2
=>       63  732558273  da2  MBR  (2.7T)
         63  732558273       - free -  (2.7T)

Now it’s time to destroy:

gpart destroy da2

The output looks as such:

[root@gateway ~]# gpart destroy da2
da2 destroyed

Now that we’ve scorched the earth, it’s time to rebuild.

Create and add

First, we need to create a new partition scheme. In this case, I’m using GPT.

gpart create -s GPT da2

The resulting output should look like this:

[root@gateway ~]# gpart create -s GPT da2
da2 created

Next we need to add the partition:

gpart add -t freebsd-ufs da2

If all goes well, you should see something similar to this:

[root@gateway ~]# gpart add -t freebsd-ufs da2
da2p1 added

At this point, we’re done with gpart. Time to revel in its simplicity and give a mental shout out to the fantastic developers that create such wonderful tools, with little to no fanfare. Thank you, wonderful and kind developers.

The “F” word

Now we must talk about a word that strikes fear into the heart of geriatrics everywhere. That’s right, we must talk about format. If you’re technical, you know there’s not much to it. If you’re not, the word format probably evokes feelings of anxiety and have you reaching for Xanax. I mean, seriously, who doesn’t know a grandparent, senior, etc, who hasn’t accidentally formatted their hard drive instead of a floppy? Anyway, luckily, it’s pretty foolproof using newfs, and it’s used like so:

newfs -U /dev/da2p1

The output is long, so it has been truncated:

[root@gateway ~]# newfs -U /dev/da2p1 
/dev/da2p1: 2861556.0MB (5860466600 sectors) block size 16384, fragment size 4096
        using 8492 cylinder groups of 336.98MB, 21567 blks, 21568 inodes.
        with soft updates
super-block backups (for fsck -b #) at:
 160, 690304, 1380448, 2070592, 2760736, 3450880, 4141024, 4831168, 5521312,
 6211456, 6901600, 7591744, 8281888, 8972032, 9662176, 10352320, 11042464,
 11732608, 12422752, 13112896, 13803040, 14493184, 15183328, 15873472,
 16563616, 17253760, 17943904, 18634048, 19324192, 20014336, 20704480,
[...]
 5846209984, 5846900128, 5847590272, 5848280416, 5848970560, 5849660704,
 5850350848, 5851040992, 5851731136, 5852421280, 5853111424, 5853801568,
 5854491712, 5855181856, 5855872000, 5856562144, 5857252288, 5857942432,
 5858632576, 5859322720, 5860012864
[root@gateway ~]# 

That’s it. Simple right?

Mounting

Time to mount your drive. Nothing special about this. Just as straightforward as any other drive:

mount -t ufs /dev/da2p1 /storage/usbdisk2

Time to verify:

[root@gateway ~]# df -h
Filesystem                Size    Used   Avail Capacity  Mounted on
/dev/ad0s1a               3.9G    383M    3.2G    11%    /
devfs                     1.0k    1.0k      0B   100%    /dev
/dev/ad0s1e               347G    158G    161G    50%    /usr
/dev/ad0s1d               7.8G    813M    6.3G    11%    /var
10.0.253.11:/nfs/Media    1.8T    1.6T    237G    87%    /storage/netdisk1
10.0.253.13:/nfs/Media    1.8T    1.6T    204G    89%    /storage/netdisk3
10.0.253.14:/nfs/Media    1.8T    1.6T    207G    89%    /storage/netdisk4
/dev/da0p1                3.6T    2.8T    476G    86%    /storage/usbdisk3
devfs                     1.0k    1.0k      0B   100%    /var/named/dev
/dev/da1s1d               1.3T    1.3T   -108G   109%    /storage/usbdisk1
/dev/da2p1                2.7T    8.0k    2.5T     0%    /storage/usbdisk2

Adding your /etc/fstab entry

Your fstab will likely be different, but here’s what I added:

/dev/da2p1   /storage/usbdisk2   ufs   rw,noauto   0   2

Errors

Here are a couple errors I ran into while preparing this post and becoming familiar with gpart a second time:

fdisk: could not detect sector size

Resolution: Determine your sector size with diskinfo and use gpart if your sector size is larger than 512 bytes.

gpart: Device busy

Resolution: I ran into this error when trying to gpart destroy my drive while it still had slices on it. gpart delete your slices and then try your gpart destroy again.

Conclusion

Congratulations! If you followed this to the end, you very likely have a new, quite large, HDD installed and ready for data. For me, it’s time to offload some data from usbdisk1 to usbdisk2, as apparently, usbdisk1 is 108GB over capacity!