Category Archives: freebsd

Making files immutable in FreeBSD

Occasionally, it’s necessary or desirable to make files immutable. For instance, your /etc/resolv.conf file might be being overwritten by your provider’s DHCP settings and you wish to use DNS servers other than those that are being set by your provider.

To determine if your /etc/resolv.conf file is immutable or not, use the following command:

ls -lo /etc/resolv.conf

The result will look like this if the file is mutable:

-rw-r--r--  1 root  wheel  - 38 Mar 25 13:58 /etc/resolv.conf

Or, the result will look like this is the file is immutable (notice the schg):

-rw-r--r--  1 root  wheel  schg 38 Mar 25 13:58 /etc/resolv.conf

To add the immutable flag to a file, use the following command:

chflags schg /path/to/your/file

To remove the immutable flag to a file, use the following command:

chflags noschg /path/to/your/file

Caveat: chflags does not work inside a jail. All flags must be set from the host server.

Using Let’s Encrypt on FreeBSD

Basic commands for installing and using Let’s Encrypt on FreeBSD

Installing Certbot

Install certbot using packages for Python 2.7:

pkg install py27-certbot

Install certbot using packages for Python 3.7:

pkg install py37-certbot

Before Using Let’s Encrypt for the First Time

Create your /usr/local/etc/letsencrypt/letsencrypt.ini file

rsa-key-size = 4096
server = https://acme-v02.api.letsencrypt.org/directory
email = email@domain.tld
text = True
agree-tos = True
renew-by-default = True
authenticator = standalone

Now, you must register your account

certbot register

Creating a Certificate

The standalone server is the easiest way to authenticate, but often requires you to stop your web server to do so. If this is an option for you, then I’d recommend doing this as it only takes a few seconds as long as you’re properly prepared. Otherwise, use the webroot method which is likely your only option in a production environment. I use webroot, as per the configuration file above.

Create your certificate as follows:

certbot certonly -c /usr/local/etc/letsencrypt/letsencrypt.ini -d domain.tld -d www.domain.tld

Time to Renew

certbot manages your domains and you are not required to renew each individually. Instead, you can simply issue the following command:

certbot renew

Deleting a Domain

In order to delete a certificate, you must know the certificate name, which is usually the domain name. But, to verify, run the following command:

certbot certificates

The results should resemble the following:

certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: domain.tld
    Domains: domain.tld www.domain.tld
    Expiry Date: 2019-06-16 18:12:08+00:00 (VALID: 89 days)
    Certificate Path: /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem
    Private Key Path: /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem

To delete that certificate, you would type the following:

certbot delete --cert-name domain.tld

Default username for FreeBSD AWS AMIs

For most Linux flavours and *NIX variants, the username you will connect with is root. However, FreeBSD AMIs are different. Ignore what the EC2 instance dashboard tells you when looking for instructions on connecting, which will look something like this:

ssh -i "your-aws-key.pem" root@ec2-54-219-0-69.us-west-1.compute.amazonaws.com

Instead, you will want to change the username to ec2-user, as such:

ssh -i "your-aws-key.pem" ec2-user@ec2-54-219-0-69.us-west-1.compute.amazonaws.com

SSH to the ec2-user and then you can su to a passwordless root account.

Make sure you set a root password. It’s also a good idea to get rid of the ec2-user account, but create your own account and setup your SSH keys first.

Granting Cacti access to the MySQL timezone database on FreeBSD

Cacti, as of version 1.0.1, requires access to the MySQL time zone database. You will see the following error at initial setup or during the upgrade process if you are required to perform this:

ERROR: Your Cacti database login account does not have access to the MySQL TimeZone database. Please provide the Cacti database account “select” access to the “time_zone_name” table in the “mysql” database, and populate MySQL’s TimeZone information before proceeding.

To accomplish this, do the following:

/usr/local/bin/mysql -u root -p mysql < /usr/local/share/mysql/mysql_test_data_timezone.sql 

Once that is done, login to MySQL and run the following:

GRANT SELECT ON mysql. time_zone_name TO 'cacti'@'localhost' IDENTIFIED BY 'password';
flush privileges;

That's all.

Installing Python package manager pip on FreeBSD

UPDATE: You can now install pip using pkg.

[cmp@server ~]# pkg install py27-pip
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        py27-pip: 9.0.1

Number of packages to be installed: 1

The process will require 13 MiB more space.
2 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching py27-pip-9.0.1.txz: 100%    2 MiB 858.8kB/s    00:03    
Checking integrity... done (0 conflicting)
[1/1] Installing py27-pip-9.0.1...
Extracting py27-pip-9.0.1: 100%

Success!

Original post
Somewhat strangely, pip is not installed when you install Python using FreeBSD’s pkg. However, there’s a very simple fix.

[cmp@server ~]# python -m ensurepip
Ignoring indexes: https://pypi.python.org/simple
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages
Collecting pip
Installing collected packages: pip
Successfully installed pip-7.1.2

[cmp@server ~]# pip
pip     pip2    pip2.7  

Setting your timezone in FreeBSD

To set your desired timezone you need to copy your timezone file from /usr/share/zoneinfo directory to /etc/localtime file. Just change directories:

# cd /usr/share/zoneinfo

A directory listing reveals many regions and zones to choose from:

# ls   
Africa          Arctic          Australia       EET             Etc             HST             MST             Pacific         WET
America         Asia            CET             EST             Europe          Indian          MST7MDT         SystemV         posixrules
Antarctica      Atlantic        CST6CDT         EST5EDT         Factory         MET             PST8PDT         UTC             zone.tab

Listing the America directory with recursion looks like this:

# ls -R America/
Adak            Belem           Chicago         Edmonton        Guyana          Lima            Metlakatla      Noronha         Rankin_Inlet    St_Barthelemy   Tortola
Anchorage       Belize          Chihuahua       Eirunepe        Halifax         Los_Angeles     Mexico_City     North_Dakota    Recife          St_Johns        Vancouver
Anguilla        Blanc-Sablon    Costa_Rica      El_Salvador     Havana          Lower_Princes   Miquelon        Ojinaga         Regina          St_Kitts        Whitehorse
Antigua         Boa_Vista       Creston         Fortaleza       Hermosillo      Maceio          Moncton         Panama          Resolute        St_Lucia        Winnipeg
Araguaina       Bogota          Cuiaba          Glace_Bay       Indiana         Managua         Monterrey       Pangnirtung     Rio_Branco      St_Thomas       Yakutat
Argentina       Boise           Curacao         Godthab         Inuvik          Manaus          Montevideo      Paramaribo      Santa_Isabel    St_Vincent      Yellowknife
Aruba           Cambridge_Bay   Danmarkshavn    Goose_Bay       Iqaluit         Marigot         Montreal        Phoenix         Santarem        Swift_Current
Asuncion        Campo_Grande    Dawson          Grand_Turk      Jamaica         Martinique      Montserrat      Port-au-Prince  Santiago        Tegucigalpa
Atikokan        Cancun          Dawson_Creek    Grenada         Juneau          Matamoros       Nassau          Port_of_Spain   Santo_Domingo   Thule
Bahia           Caracas         Denver          Guadeloupe      Kentucky        Mazatlan        New_York        Porto_Velho     Sao_Paulo       Thunder_Bay
Bahia_Banderas  Cayenne         Detroit         Guatemala       Kralendijk      Menominee       Nipigon         Puerto_Rico     Scoresbysund    Tijuana
Barbados        Cayman          Dominica        Guayaquil       La_Paz          Merida          Nome            Rainy_River     Sitka           Toronto

America/Argentina:
Buenos_Aires    Cordoba         La_Rioja        Rio_Gallegos    San_Juan        Tucuman
Catamarca       Jujuy           Mendoza         Salta           San_Luis        Ushuaia

America/Indiana:
Indianapolis    Knox            Marengo         Petersburg      Tell_City       Vevay           Vincennes       Winamac

America/Kentucky:
Louisville      Monticello

America/North_Dakota:
Beulah          Center          New_Salem

I want America/Los_Angeles so I would type the following:

# cp America/Los_Angeles /etc/localtime

Verify your work with date:

# date
Fri Nov 13 16:12:30 PST 2015

Success.

Prevent screen(1) from resizing your SecureCRT windows

Have you ever noticed that when you start or reattach screen it resizes every SecureCRT tab you have open to 80 columns?

There’s an extremely easy fix. First, find out which term you are using. This must be done outside of screen or your term is simply screen.

[cmp@server ~]$ echo $TERM
vt100

Then add the following line to your ~/.screenrc file, adjusting for your term, as needed:

termcapinfo vt100* 'is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l'

For xterm, you’d replace vt100 with xterm.

Your terminal will now be the width of your SecureCRT window.

Fixing Postfix permissions on FreeBSD

I recently ran into issues with Postfix not being able to create files in the mail queue because of a permissions problem. Postfix was installed via pkg so it’s a bit perplexing that the permissions were not set as part of the install process. This was inside a jail, but I don’t see how that should make a difference. Here’s the error I was seeing in /var/log/maillog:

Oct  2 15:36:25 monitor postfix/postdrop[20795]: warning: mail_queue_enter: create file maildrop/615300.20795: Permission denied
Oct  2 15:36:29 monitor postfix/postdrop[20823]: warning: mail_queue_enter: create file maildrop/695303.20823: Permission denied

Here’s how I fixed it:

/usr/sbin/service postfix stop
/usr/bin/killall -9 postdrop
/usr/bin/chgrp -R maildrop /var/spool/postfix/public
/usr/bin/chgrp -R maildrop /var/spool/postfix/maildrop/
/usr/local/sbin/postfix set-permissions 
/usr/local/sbin/postfix check
/usr/sbin/service postfix start

Updating OpenSSL for Apache without breaking your base OpenSSL install on FreeBSD 9.2-RELEASE

After two very frustrating days, I have finally been able to perform what used to be a fairly routine thing for me, which is creating a FAMP server using FreeBSD, Apache 2.2.x, MySQL 5.x.x and PHP 5.5.x.

Several issues arose out of my attempts.

My first attempt was installing Apache 2.4.9, MySQL 5.6.17 and PHP 5.5.12 on FreeBSD 10.0-RELEASE-p3 (AMD64). This was my first time installing anything greater than MySQL 5.1.x. ‘./configure’ has been replaced with ‘cmake .’; other than that, all things ordinary. This was also my first time installing Apache 2.4.x and there are several new steps involved. The first is that you now have to manually install Apache’s portable runtime library and associated utilities. Piece of cake. Grabbed those from the APR website and installed without issue. Apache installed. But now the trouble. I was able to ‘./configure’ and ‘make’ PHP without issue, but during the ‘make install’ I ran into an issue.

# make install
Installing PHP SAPI module: apache2handler
/usr/local/httpd/build/instdso.sh SH_LIBTOOL='/usr/local/apr/build-1/libtool' libphp5.la /usr/local/httpd/modules
/usr/local/apr/build-1/libtool --mode=install cp libphp5.la /usr/local/httpd/modules/
libtool: install: cp .libs/libphp5.lai /usr/local/httpd/modules/libphp5.la
libtool: install: cp .libs/libphp5.a /usr/local/httpd/modules/libphp5.a
libtool: install: chmod 644 /usr/local/httpd/modules/libphp5.a
libtool: install: ranlib /usr/local/httpd/modules/libphp5.a
libtool: install: warning: remember to run `libtool --finish /usr/local/src/php-5.5.5/libs'
Warning! dlname not found in /usr/local/httpd/modules/libphp5.la.
Assuming installing a .so rather than a libtool archive.
chmod 755 /usr/local/httpd/modules/libphp5.so
chmod: /usr/local/httpd/modules/libphp5.so: No such file or directory
apxs:Error: Command failed with rc=65536

Keep in mind, this was a brand new, entirely fresh FreeBSD 10.0-RELEASE install with ‘freebsd-update’ run bringing it to -p3. Nothing else had been done on this box except the MySQL and Apache installs.

I found it very strange that in that small bunch of code, two separate libtools were referenced. Foolishly, I didn’t check my working server to compare. In any case, ‘/usr/local/apr/build-1/libtool’ and the libtool included with the PHP source were both referenced. For good measure, I even installed libtool using ‘pkg install’. Nothing worked and libphp5.so failed to be created. Not being a software developer, I went to Google looking for the answer but came up short. A few suggestions were to use the .bz2 version instead of the .tgz version, use ports and one person even pointed the finger at the AMD64 build of FreeBSD. I briefly attempted to install a known working version of libphp5.so from another one of my servers but that just created more issues. I moved on.

I reloaded the OS from scratch and tried again. Same result. I’m a sucker for punishment.

I then reinstalled the OS using FreeBSD 9.2-RELEASE. As before, I ran ‘freebsd-update’ first thing and brought it up to -6p. I compiled MySQL using the new steps I had learned earlier. Same with Apache, except I used 2.2.27 this time to simplify things; go with what you know. Then the true test, PHP 5.5.12, which also installed no problem. Added a few times to httpd.conf and started the server. Problem. I telneted to port 80 to see what version or OpenSSL I was using and it came back as 0.9.8y. This version isn’t vulnerable to the heartbleed vulnerability, but it does lack TLSv1.1 and TLSv1.2 support, which I need, if only to give my clients a greater sense of security. So I search Google for “freebsd upgrade openssl” and a few articles come up, including one I had recently visited. I went back, performed the needful using ports and OpenSSL was upgraded. Sort of. The base OpenSSL install has its binaries in /usr/bin, libraries in /usr/lib and config file in /etc. The port install puts the binaries in /usr/local/bin, the libraries in /usr/local/lib and the config file in /usr/local/etc/ssl. So I have two completely autonomous OpenSSL installs at this point. Typing ‘which openssl’ shows /usr/bin/openssl which is version 0.9.8y. ‘/usr/local/bin/openssl version’ shows 1.0.1g. Again, 0.9.8y is not vulnerable to heartbleed, but it is insufficient for my needs. After further reading, a few people suggested to remove the old binary and symlink /usr/local/bin/openssl to /usr/bin/openssl and also symlink the configuration files. This seemed to work, so I rebuilt Apache again thinking that version 1.0.1g would show up. But it didn’t. Knowing that this was a new install and that I would probably blow it away once I found a working formula, I started deleting things, symlinking things, moving things, etc. I reminded myself of my father-in-law who while looking to free up space on his internal HDD, he deleted his windows system files with the rational of “I’ve never used those, I can delete them.” A mess I made. Server trashed. Blank slate.

Again, I started with FreeBSD 9.2-RELEASE. I immediately updated using freebsd-update to -p6. I then downloaded and installed OpenSSL 1.0.1g from the OpenSSL website. I then confirmed that my base and secondary install were as expected:

# /usr/bin/openssl version         
OpenSSL 0.9.8y 5 Feb 2013

Good.

# /usr/local/ssl/bin/openssl version
OpenSSL 1.0.1g 7 Apr 2014

Great.

When it came to compiling Apache, I had to compile mod_ssl as a static module. In the process, a few of the core modules were also made static, so I had a mishmash of static and dynamic. I had to comment out a few of the LoadModule statements in the httpd.conf, but in the end, everything works as it should.

[root@server01 ~]# telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
HTTP/1.1 / GET

HTTP/1.1 501 Method Not Implemented
Date: Wed, 28 May 2014 23:07:36 GMT
Server: Apache/2.2.27 (Unix) DAV/2 PHP/5.5.12 mod_ssl/2.2.27 OpenSSL/1.0.1g
Allow: GET,HEAD,POST,OPTIONS,TRACE
Content-Length: 220
Connection: close
Content-Type: text/html; charset=iso-8859-1

Success! I now have a server running exactly the way I want and I have no compromised any of the base OpenSSL files, or anything that was built against them. While I think this was a fairly good way to accomplish this task, there are probably even better ways to do it. However, my Googling efforts did not lead me to those.

Another other method, which some will say is more elegant, would have been to install from pkg or ports. However, I disagree. Since I compiled my first web server twenty years ago to present day, I have always been an advocate of compiling critical services from source. If all you ever do is type ‘pkg install apache22’ how do you learn to troubleshoot when something goes wrong? How do you know how to fine tune your installation when you need to compile in sockets, or curl, or other libraries and modules? I prefer infinite control.

Rackspace FreeBSD VPS and their lack of swap

Update #2 – 2015-05-19 – Three months after my first update I noticed that Rackspace has removed their FreeBSD VPS offering. Not surprising and probably for the best. Two clients that I recommend Rackspace to, that are also using FreeBSD VPS, are experiencing the same issues, so I know I am not alone. I have moved my mission-critical services to a Leaseweb bare-metal server. I’ve been with them for five months now. 100% uptime and no issues whatsoever. There’s twice been a 30ms jump in latency from where I am monitoring the server from; perfectly acceptable performance, if not amazing by server-farm standards. I am considering moving to Packet as I have just been told they intend to support FreeBSD very soon. They also offer a BYoIP (bring your own IP) service, which is extremely attractive as we have an IPv4 /23 and an IPv6 /48.

Update #1 – 2015-02-19 – After what seems like one of my servers hanging hundreds of times, and in reality is probably at least 60, I have moved off of Rackspace’s FreeBSD VPS for all my mission critical sites. Rackspace dev ops seem very uninterested in putting any effort into their FreeBSD product. Additionally, few of their Linux ops can support it. After all the swap nonsense and being affected by “noisy neighbors”, I’ve moved onto a bare-metal server. So far, zero downtime. A decade ago, I envied Rackspace’s product and support. Now, they are a far cry from Softlayer, Packet.net and many others who have left them in the dust. Their now lackluster support and lack of innovation could see them completely fall off the map. It’s quite sad to see how far the mighty have fallen.

Last October, one of the best VPS and bare-metal server providers on the planet, Rackspace, quickly fell from grace with many of its VPS customers. They made a decision — a poor one at that — to eliminate swap from their default Linux and FreeBSD VPS builds. They claim that this was in an effort to reduce disk IOPS, but in reality, they’ve achieved nothing but instability. Here’s their KB article:

http://www.rackspace.com/knowledge_center/article/changes-to-swap-on-cloud-servers

Notice at the bottom they’ve provided a link for how Linux users can add a swap file. Also notice that there’s no mention of how to add a swap file using FreeBSD. I’ll get to that in a second. First, let’s break down what they’ve actually accomplished:

They’ve not actually reduced IOPS at all, because nearly everyone running a VPS will have manually created a swap file, because if you don’t, it’s possible that this can happen:

Apr 20 08:20:37 monitor kernel: pid 10861 (xenstore), uid 0, was killed: out of swap space
Apr 20 08:20:37 monitor kernel: pid 172 (nova-agent), uid 0, was killed: out of swap space
Apr 20 08:20:38 monitor kernel: pid 778 (sshd), uid 0, was killed: out of swap space
Apr 20 08:20:38 monitor kernel: pid 965 (devd), uid 0, was killed: out of swap space
Apr 20 08:20:38 monitor kernel: pid 2345 (sendmail), uid 0, was killed: out of swap space

Yes, that’s right, whilst compiling MySQL, and apparently running out of memory on a box that supposedly has 2GB of RAM, the server killed several critical processes.

Because you cannot have custom partitioning on their FreeBSD VPS, they force you to choose from one of many of the already capacity-deprived file systems in which to create the swap file.

Filesystem                   Size    Used   Avail Capacity  Mounted on
zroot                        2.4G    205M    2.1G     9%    /
devfs                        1.0k    1.0k      0B   100%    /dev
zroot/tmp                    2.1G    424k    2.1G     0%    /tmp
zroot/usr                    5.5G    3.3G    2.1G    61%    /usr
zroot/usr/home               3.8G    1.6G    2.1G    43%    /usr/home
zroot/usr/ports              3.1G    996M    2.1G    31%    /usr/ports
zroot/usr/ports/distfiles    2.2G     41M    2.1G     2%    /usr/ports/distfiles
zroot/usr/src                2.4G    276M    2.1G    11%    /usr/src
zroot/var                    2.2G    3.4M    2.1G     0%    /var
zroot/var/crash              2.1G    148k    2.1G     0%    /var/crash
zroot/var/db                 2.3G    185M    2.1G     8%    /var/db
zroot/var/db/pkg             2.2G    6.0M    2.1G     0%    /var/db/pkg
zroot/var/empty              2.1G    144k    2.1G     0%    /var/empty
zroot/var/log                2.1G    1.7M    2.1G     0%    /var/log
zroot/var/mail               2.1G    1.4M    2.1G     0%    /var/mail
zroot/var/run                2.1G    296k    2.1G     0%    /var/run
zroot/var/tmp                2.1G    1.7M    2.1G     0%    /var/tmp

Which file system are you likely to throw a 2GB swap file on? This partitioning at its absolute worst. I’m using a grand total of about 200MB of the nearly 18GB of space allocated to all the many /var file systems.

But, I digress. Here’s how to add a swap file to your VPS:

Requirements

You’ll need the memory disk driver install, which comes by default with the generic kernel. Rackspace at least had the foresight to keep this in their kernel.

device     md

Next, find a good place for your swap file. This will likely be determined by your partitioning. I’m going to create a 1024 MB swap file in /usr/swap and name is swap0. On my server the /usr file system is the largest partition on the server, so it makes the most sense. Next, I am doing to create the actual swap file:

dd if=/dev/zero of=/usr/swap/swap0 bs=1024k count=1024

Adjust the count to adjust your swap file size. 1 count is equal to 1 MB.

Next, change the swap file permissions:

chmod 0600 /usr/swap/swap0

Remember to add this new swap file to your /etc/rc.conf file:

swapfile="/usr/swap/swap0"

To enable the new swap without rebooting, use the following commands:

mdconfig -a -t vnode -f /usr/swap/swap0
swapon /dev/md0

When you enter the mdconfig command, it will list the device name for your new swap file. Mine was md1. Yours may be different.