Creating and Managing A Jailed Virtual Host in FreeBSD

It is possible to spawn a completely jailed second (or many!) operating system within a modern FreeBSD install. Doing so can be tricky, but here I will document the method that I have found works for me. The first most important resource about jails, is the man page, and many of the examples that you will see are basically straight following of the man page. Once you are done, each jail will operate as if its a complete independant operating system.

Scope Of This Document

This Howto article is intended to be a practical example, and I will start my host system with FreeBSD 6.2-RELEASE. The Install will be of the "minimal" variety, and for the first part of this document, we will not update the system with buildworld. After a jail is created, we will then update the host, and then update the jail. This will demonstrate a practical example of how to build, and then maintain a jail thru critical security releases.

System Preparation

Before we begin this part of the article, I will assume that we have already built our FreeBSD 6.2-RELEASE system. The example laid out in Installing FreeBSD 6.2 might be a good place to start, but only do the first half of the article (do not proceed to step 3, "Rebuild the World, and Recompile the Kernel") As we begin, there are a few tasks you must take care of on your host system to prepare it to run jails.

1) You will need the full sources tree in /usr/src/. Using 'csup' is the best way to do this. While you are doing this, you may as well use csup to get the latest ports tree too.
2) Specify another IP address as an alias, that your jailed operating system will use.

ifconfig_fxp0="inet 192.168.1.180  netmask 255.255.255.128"
ifconfig_fxp0_alias0="inet 192.168.1.181  netmask 255.255.255.128"

As you can see, my main ip is 192.168.1.180, and my alias ip is 192.168.1.181. We will need to specify this alias ip later when we start the jail.
3) You should then cull back all the services on the host system to use only the hosts main IP address. Most of these changes would happen in your /etc/rc.conf file, but some happen in the config files of the daemon. A good example is sshd. In order for it to listed only on one IP address, we must edit /etc/ssh/sshd_config. Look for the line:

#ListenAddress 0.0.0.0

Uncomment that, and change the zeros to your host's main IP address. Another daemon off the top of my head that would use its own config file for this type of configuration, would be Apache. However, remember that I recommend the slimmest configuration possible on your host, so not installing Apache or any other network applications would be advised. After all, the point of jails is to install virtual systems that all our network daemons can then be installed into. My host system does use inetd, so I add this to my /etc/rc.conf to keep inetd only on the hosts IP addrress:

inetd_enable="YES"
inetd_flags="-wW -a 192.168.1.180"

On my systems, I use snmpd. However, snmpd will not be able to be used correctly against jailed instances, but only against the host. So, to keep snmpd from operating on all IP addresses available (which is the default behavior), we need to add this line to /usr/local/share/snmp/snmpd.conf:

agentaddress 192.168.1.180:161

Realistically, you need to evaluate your host system's needs, and personalize the "slimming down" of your host to what you need. The

Building a Jailed Virtual Host

The man page specifies a little script that can get you started. From the man page:

D=/here/is/the/jail
cd /usr/src
mkdir -p $D
make world DESTDIR=$D
make distribution DESTDIR=$D

What we have here, is that we will specify 'D' is the path we will put our jailed system in, and that's really the hardest part. I named my script 'mkjail'. For 'D', I specified /usr/jails/[hostname] (in my case, my hostname was TEMPLATE, so I had /usr/jails/template). For my first jail, I always create one that I can use as a template to just spawn others, without going through the whole scripted 'make world' and 'make distribution' again.

I then ran the script with 'sh mkjail'. I think it probably took about 2 hours for my system.

Copy the /etc/resolv.conf into your jail. Without this, your jail will not be able to find its way out to the internet.

cp /etc/resolv.conf /usr/jails/template/etc/

Also, create the jail's /etc/make.conf, so that later when we install a port, our jail will use a local workdir, instead of writing into our hosts workdir (we don't like conflicts!). Add this line to the jail's /etc/make.conf:

WRKDIRPREFIX=/tmp

Also, we need to make a ports directory so that we can nullfs mount the hosts ports to it later.

mkdir /usr/jails/template/usr/ports

After that, we are ready to archive our jail template, so we can reuse it later.

cd /usr/jail
tar zcvf jail-template.tar.gz /usr/jails/template

At this point, our jail template directories can be renamed to be used for our example. I am going to name my jail ANTARES.

mv /usr/jails/template /usr/jails/antares

Final step is mounting a proc and devfs file system to the jail, and we can start an interactive shell on it.

mount_devfs devfs /usr/jails/antares/dev
mount_procfs procfs /usr/jails/antares/proc
jail /usr/jails/antares/ antares 192.168.1.181 /bin/sh

This will drop us to a basic shell. Before we run our jail for real, there are a few things to take care of in the jail. First, create a "dummy" fstab to keep the startup scripts from complaining.

touch /etc/fstab

Then, run sysinstall, go to 'configure', and use the timezone tool to set your timezone. Finally, set the root password with the 'passwd' command. After that, were ready to try it out. Hit a ctrl-d to exit the shell. Now, lets start it up so that it exec the startup scripts.

jail /usr/jails/antares/ antares 192.168.1.181 /bin/sh /etc/rc

You should see what appears to be a normal start up, ending at a root prompt. The first thing I alawys need, is my favorite editor, nano.

pkg_add -r nano

Then lets also add a couple entries to the jail's /etc/rc.conf.

network_interfaces=""
rpcbind_enable="NO"
sshd_enable="YES"
syslogd_flags="-ss"

Eliminate 'adjkerntz -a' from the jail's /etc/crontab. We don't need this in a jail (man jail), the CMOS clock is already managed by the host. The entire line to comment out or remove looks like this:

1,31   0-5     *       *       *       root    adjkerntz -a

Now you can ctrl-d and exit back out of that shell. Now its time to add some information to the host's /etc/rc.conf, so that our jail will automatically start with the host. Add this to the host's /etc/rc.conf:

jail_enable="YES"
jail_interface="fxp0"
jail_devfs_enable="YES"
jail_procfs_enable="YES"
jail_list="antares"
jail_antares_rootdir="/usr/jails/antares"
jail_antares_hostname="antares.example.com"
jail_antares_ip="192.168.1.181"

After this is added, reboot the host. When it comes backup, log in as root, and use the 'jls' command to see the running jail.

[root@acrux /usr/jails]# jls
JID  IP Address      Hostname                      Path
1  192.168.1.181  antares.example.com             /usr/jails/antares

Looks good, now try to ssh into your new jailed instance of FreeBSD. If it works, then throw your arms in the air, and yell "SUCCESS!!"

Installing Ports and Updating the World

If you have read my article Managing Multiple FreeBSD Systems, then you might know that I only like to keep one copy of the source tree and ports tree on my network. This saves bandwidth at both freebsd.org's end, as well as my own internet connection. But on top of that, 'buildworld' only needs to be run one time, and can then be distributed to other machines. This theory also includes jailed instances. From earlier in this article, our host system pulled down both the operating system sources and ports trees. Let's nullfs mount them to our running jail, so that our jail can take advantage of these resources.

mount_nullfs /usr/ports/ /usr/jails/antares/usr/ports/
mount_nullfs /usr/src/ /usr/jails/antares/usr/src/
mount_nullfs /usr/obj/ /usr/jails/antares/usr/obj/

In your jail, you should now see the same contents in /usr/src, /usr/obj, and /usr/ports as your host system. We already edited our jail's /etc/make.conf, so when we install ports, we won't step on our host's toes. The jail is now ready to build any ports you intend to run.

But ports are not all our system will need. At this point, we still need to buildworld and update both our host and our jail. So, drop out of our ssh session, and go back to our host. Change into the hosts sources directory.

cd /usr/src

Edit a config file file to match your system, and save it as an ALLCAPS name to /usr/src/sys/i386/conf (if you dont have an i386 machine, remember to save the config file to the appropriate architecture's folder tree). As you may have noticed above, my host's name is ACRUX, and this is the name of my config file. After the file is saved, its time to build the world and the kernel.

make buildworld
make buildkernel KERNCONF=ACRUX

Those steps will probably eat up an hour or 2, as they do on my system. After they are done, follow the steps laid out in either the FreeBSD handbook, or my daring way (at your OWN RISK!!) in my article The Method I Use When I Buildworld On My Personal Systems. After your kernel and world are installed on your host system, reboot.Once the host system is backup, log in as root, and now we can update the jail. If you ssh into your jail, you will notice that the new kernel is already installed, as the jail always runs from a copy of the current running kernel of the host.

cd /usr/src
make installworld DESTDIR=/usr/jails/antares

Remount our nullfs resources to the jail, so that we can perform our last step for updating the world.

mount_nullfs /usr/ports/ /usr/jails/antares/usr/ports/
mount_nullfs /usr/src/ /usr/jails/antares/usr/src/
mount_nullfs /usr/obj/ /usr/jails/antares/usr/obj/

Finally its time to ssh into the jail, and apply the final part of the update. Log in to your jail as root, and:

cd /usr/src
mergemaster

Once this is completed, the jail can be restarted from within a shell on the host, like this:

/etc/rc.d/jail restart antares

And now our end product should be both a host and a jail that are fully up to date.

Written by Sharaz, our resident FreeBSD guru.