Learning Notes on FreeBSD Jails


I have heard about jails many times since my early days of FreeBSD life but it was only the last year I began to use it in production.

This article is a sort of personal notebook where I summarize what I learned about jails. It would be frequently updated as I learn more.

Assumptions

Creating a Template

When you start using jails, the first thing to do is creating a template for future jails.

  1. Create ZFS datasets for jails and templates.
    Here I’m going to create a FreeBSD 11.2 template in /vm/tmpl/11.2.

    sudo zfs create -o mountpoint=/vm zroot/vm
    sudo zfs create zroot/vm/tmpl
    sudo zfs create zroot/vm/tmpl/11.2
    
  2. Download a base install set tarball for FreeBSD 11.2 release.

    cd ~/tmp
    fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/base.txz
    
  3. Extract the base tarball in the template directory.

    sudo tar -xJvpf base.txz -C /vm/tmpl/11.2
    
  4. Copy timezone configuration from the host to the template.
    Usually /etc/resolv.conf is also copied from the host, but after some experiments I decided to copy it by using /etc/jail.conf’s exec.prestart and delete it by exec.poststop.

    sudo cp /etc/localtime /vm/tmpl/11.2/etc/
    
  5. Write a minimum /etc/rc.conf for the template.
    Here I disable cron in jails but it might be better to leave it enabled and fine-tune its configurations.
    I need more research on this.

    sudo vi /vm/tmpl/11.2/etc/rc.conf
    
    cron_enable="NO"
    sendmail_enable="NO"
    sendmail_submit_enable="NO"
    sendmail_outbound_enable="NO"
    sendmail_msp_queue_enable="NO"
    syslogd_flags="-ss"
    
  6. Edit system crontab in the template to disable adjkern.
    Obviously this is not necessary because I disabled cron in the previous step but I do this as a precaution anyway.

    sudo vi /vm/tmpl/11.2/etc/crontab
    
    #1,31 0-5 * * * root adjkerntz -a
    
  7. Create directories to handle ports in jails.
    The first one is to read-only mount host’s ports tree and others are working spaces.
    Also create /etc/make.conf to use the directories for building ports in jails.

    sudo mkdir /vm/tmpl/11.2/usr/ports
    sudo mkdir -p /vm/tmpl/11.2/var/ports/{distfiles,packages}
    sudo vi /vm/tmpl/11.2/etc/make.conf
    
    WRKDIRPREFIX = /var/ports
    DISTDIR = /var/ports/distfiles
    PACKAGES = /var/ports/packages
    
  8. Apply system updates to the template.

    sudo freebsd-update -b /vm/tmpl/11.2 fetch install
    
  9. Take a snapshot of the template dataset.
    Strictly speaking, a template is a snapshot not a dataset. The snapshot can be cloned or sent/received to generate new datasets for production jails.

    sudo zfs snapshot zroot/vm/tmpl/11.2@p3
    

Creating Jails from the Template

Once the template is ready, jails can be created by cloning or sending/receiving its snapshot.
Usually, I use clone for quick testing while I prefer zfs send/receive for production jails.

By ZFS Clone

Create four jails h1, h2, h3 and h4 by cloning the 11.2 template’s p3 snapshot.

for jail in h1 h2 h3 h4; do
    sudo zfs clone zroot/vm/tmpl/11.2@p3 zroot/vm/$jail
done

By ZFS Send/Receive

Create four jails h1, h2, h3 and h4 by sending/receiving the 11.2 template’s p3 snapshot.

for jail in h1 h2 h3 h4; do
    sudo sh -c "zfs send zroot/vm/tmpl/11.2@p3 | zfs receive zroot/vm/$jail"
done

Configuring Jails

System Startup Configuration

Do not forget to create a cloned loopback interface “lo1” on which jail’s addresses are configured.

PF is used for NAT and port forwarding.

[/etc/rc.conf]

cloned_interfaces="lo1"
pf_enable="YES"
pflog_enable="YES"
jail_enable="YES"
jail_list="h1 h2 h3 h4"

System Jail Configuration

[/etc/jail.conf]

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = "/vm/$name";
exec.consolelog = "/var/log/jail_${name}_console.log";
exec.prestart = "cp /etc/resolv.conf $path/etc";
exec.poststop = "rm $path/etc/resolv.conf";

# nullfs mount
h1 {
        ip4.addr = "lo1|127.1.1.1/32";
        ip6.addr = "lo1|fd00:1:1:1::1/64";
        allow.chflags;
        allow.raw_sockets;
        mount.fstab = "/vm/${name}.fstab";
}

# WINE (sysvsem)
h2 {
        ip4.addr = "lo1|127.1.1.2/32";
        ip6.addr = "lo1|fd00:1:1:1::2/64";
        allow.chflags;
        allow.raw_sockets;
        sysvsem = "new";
}

# PostgreSQL (sysvsem, sysvshm)
h3 {
        ip4.addr = "lo1|127.1.1.3/32";
        ip6.addr = "lo1|fd00:1:1:1::3/64";
        allow.chflags;
        allow.raw_sockets;
        sysvsem = "new";
        sysvshm = "new";
}

# Java (fdescfs, procfs)
h4 {
        ip4.addr = "lo1|127.1.1.4/32";
        ip6.addr = "lo1|fd00:1:1:1::4/64";
        allow.chflags;
        allow.raw_sockets;
        mount.fdescfs;
        mount.procfs;
}

Per-Jail fstab

If you want some directories on the host to be visible in jails, create a fstab containing nullfs mount entries for each jail and specify it in /etc/jail.conf’s mount.fstab parameter.

[/vm/h1.fstab]

/usr/ports /vm/h1/usr/ports nullfs ro 0 0

Or you can also use mount parameter to specify a single line of fstab entry directly in /etc/jail.conf.

[/etc/jail.conf]

...
mount = "/usr/ports /vm/${name}/usr/ports nullfs ro 0 0";
...

NAT Configuration

PF configuration should be written in the strict order of categories.

For IPv6 NAT, I explicitly specify a external (global) address instead of the interface name because an IPv6 interface usually has multiple addresses and it seems the interface name could be resolved to its link-local address.

[/etc/pf.conf]

# MACROS/TABLES
XIF = "em0"
JAILNET_V4 = "127.1.1.0/24"
JAILNET_V6 = "fd00:1:1:1::0/64"
EXT_V6ADDR = "2001:db8::1"

# OPTIONS (set skip, etc.)
# NORMALIZATION (scrub)
# QUEUEING

# TRANSLATION
## NAT
nat on $XIF inet from $JAILNET_V4 to any -> ($XIF)
nat on $XIF inet6 from $JAILNET_V6 to any -> $EXT_V6ADDR

## REDIRECT (Port Forwarding)
rdr pass log on $XIF inet proto tcp to ($XIF) port 8080 -> 127.1.1.4
rdr pass log on $XIF inet6 proto tcp to $EXT_V6ADDR port 8080 -> fd00:1:1:1::4

# FILTERING (Pass/Block)

Hosts Table

[/etc/hosts]

127.1.1.1 h1
127.1.1.2 h2
127.1.1.3 h3
127.1.1.4 h4
fd00:1:1:1::1 h1
fd00:1:1:1::2 h2
fd00:1:1:1::3 h3
fd00:1:1:1::4 h4

Managing Jails

Start all jails.

sudo service jail start

Start specific jail(s).

sudo service jail start h1

Login to a jail.

sudo jexec h1

Run a command on a jail.

sudo jexec h1 ifconfig

List running jails.

jls
jls -v
jls -s

NOTE: At first I was confused by ip4=disable output of jls -s. But later I looked at /usr/src/usr.sbin/jls/jls.c and learned that it’s correct because ip4’s value is ‘disable’ unless explicitly set otherwise. It’s just regarded as ‘new’ when ip4.addr is set.

Stop specific jail(s).

sudo service jail stop h1

Stop all jails.

sudo service jail stop

Manage binary packages on a jail.

sudo pkg -j h1 update
sudo pkg -j h1 upgrade
sudp pkg -j h1 install git-lite vim-console

Update package repositories on multiple jails using shell’s for loop.

for jail in $(jls name); do
  sudo pkg -j $jail update
done

Apply system updates to a jail.

sudo freebsd-update -b /vm/h1 fetch install

Upgrade system on a jail.

sudo freebsd-update -b /vm/h1 -r 11.3-RELEASE upgrade

When you have already upgraded the host, run the following commands in a jail.

setenv UNAME_r `freebsd-version`
freebsd-update -r 11.3-RELEASE upgrade
/usr/sbin/freebsd-update install
(No reboot required for a jail)
unsetenv UNAME_r
/usr/sbin/freebsd-update install

VNET Jails

Sometimes I feel that jails networking is not always intuitive.

In my current understanding, a standard (non-VNET) jail shares its IP address with the host.
If the jail doesn’t listen to a specific TCP or UDP port on the address, packets destined to the port/address are processed by the host.
A jail’s address is usually an IP alias (secondary address) on the host but it’s not limited to that. Actually a jail can use the host’s primary IP address unless the host and the jail doesn’t use the same ports.
I think this behavior is confusing because I expect that the host and jails can be viewed as spearate systems.

It’s also hard to understand at first that jail’s 127.0.0.1 is mapped to an address deligated to the jail, inter-jail communications go through lo0 even if jails’ addresses are assigned to lo1 and so on.

In this regard, VNET jails look simpler to me.
So I gave it a try despite the fact that they require custom kernel with VIMAGE feature enabled (I completely got used to using binary updates with GENERIC kernel and almost forgot the time when I always rebuilt kernel to slim it down).

GOOD NEWS! (2018-12-14)
VIMAGE is enabled by default on FreeBSD 12.0.
If you are using the version, you don’t have to recompile your kernel.

Installing and Updating FreeBSD Source

From Distribution Tarball

Download the tarball.

cd ~/tmp
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/src.txz

Extract the tarball.

sudo tar -xJvpf src.txz -C /

From Subversion

Check out the source.

sudo svnlite checkout https://svn.freebsd.org/base/releng/11.2 /usr/src

Update the source.

sudo svnlite update /usr/src

Building VIMAGE-enabled Kernel

The Simplest Way

Build kernel using a configuration file which comes with the FreeBSD source tree.
This config file contains some netgraph modules handy for VNET jails.

sudo cp /usr/src/share/examples/jails/VIMAGE /usr/src/sys/amd64/conf
cd /usr/src
sudo make KERNCONF=VIMAGE kernel
sudo shutdown -r now

Minimum Configuration

To add only VIMAGE to GENERIC, create a kernel configuration file with the following content.
[/usr/src/sys/amd64/conf/VIMAGE]

include GENERIC
ident VIMAGE
options VIMAGE

Then build.

cd /usr/src
sudo make KERNCONF=VIMAGE kernel
sudo shutdown -r now

Preparing Helper Script

I made a quick-and-dirty modification to the jng script in the FreeBSD source tree and named it vnet. This small script is used to setup netgraph-based virtual switch/interfaces for the host and jails.

mkdir ~/src
cd ~/src
git clone https://github.com/genneko/freebsd-vimage-jails.git jails
sudo ln -s ~/src/jails/vnet /usr/sbin/

/etc/jail.conf for VNET Jails

Depending on the network configuration, slightly different configuration should be added to /etc/jail.conf for each VNET jail.

Bridged Configuration

                      |
                      |
                 +----o----+
                 | gateway |
                 +----o----+
                      | $gwv4/$plen
                      |
+--------+------------+------------------+
         |
     em0 | (ng_ether)
  (=$net)|
+--------|-------------------------------+
|        |                               |
|        | lower +-------------+         |
|        +-------+    em0br    |         |
|        +-------+ (ng_bridge) |         |
|        | upper +---------+---+         |
|        |                 |             |
|        |          em0_b1 | $ipv4/$plen |
|    em0 |     (ng_eiface) |             |
|    +---o---+         +---o---+         |
|    |  Host |         |Jail b1|         |
|    +-------+         +-------+         |
+----------------------------------------+

For a bridge configuration, $net is the name of a physical ethernet interface (ng_ether) on the host.
$gwv4 is the IP address of a default gateway for the jail and the host while $ipv4 is the IP address of the jail’s virtual interface (ng_eiface).
$plen is a subnet mask length for the bridged network.
The jail’s virtual interface and the host’s physical interface are connected by a ng_bridge named ‘${net}br’.
In this configuration, the jail uses the same router (gateway) as the host to go out.

b1 {
        $net = "em0";
        $gwv4 = "192.168.1.1";
        $ipv4 = "192.168.1.11";
        $plen = 24;

        vnet;
        vnet.interface = "${net}_$name";
        exec.prestart += "vnet add $net ${net}_$name";
        exec.start    += "ifconfig ${net}_$name $ipv4/$plen";
        exec.start    += "route add default $gwv4";
        exec.poststop += "vnet delete $net ${net}_$name";
}

Routed Configuration

                      |
                      |
                 +----o----+
                 | gateway |
                 +----o----+
                      |
                      |
+--------+------------+------------------+
         |
Physical |
     I/F |
+--------|-------------------------------+
|    +---o---+                           |
|    |  Host | (Internal gateway)        |
|    +---o---+                           |
|    vi0 | $gwv4/$plen                   |
|(ng_eiface)                             |
|        |       +-------------+         |
|        +-------+    vi0br    |         |
|                | (ng_bridge) |         |
|                +---------+---+         |
|                          |             |
|                   vi0_v1 | $ipv4/$plen |
|              (ng_eiface) |             |
|                      +---o---+         |
|                      |Jail v1|         |
|                      +-------+         |
+----------------------------------------+

For a routed configuration, $net is the name of an internal virtual ethernet interface (ng_eiface) on the host and could be any name you like. This interface is created by the vnet script if it doesn’t exist.
$gwv4 is the IP address of this internal interface while $ipv4 is the IP address of the jail’s virtual interface (another ng_eiface).
$plen is a subnet mask length for the virtual network.
The jail and the host’s virtual interface are connected by a ng_bridge named ‘${net}br’.
In this configuration, the jail use the host as a gateway to the outside world.

v1 {
        $net = "vi0";
        $gwv4 = "172.31.0.1";
        $ipv4 = "172.31.0.11";
        $plen = 24;

        vnet;
        vnet.interface = "${net}_$name";
        exec.prestart += "vnet add -4 $gwv4/$plen $net ${net}_$name";
        exec.start += "ifconfig ${net}_$name $ipv4/$plen";
        exec.start += "route add default $gwv4";
        exec.poststop += "vnet delete $net ${net}_$name";
}

For a routed configuration, IP packet forwarding should be enabled to allow the jail to go out to the internet.
IPv4/IPv6 packet forwarding can be controlled by sysctl variables net.inet.ip.forwarding and net.inet6.ip6.forwarding.
Packet forwarding can be enabled dynamically by running the following commands.

sysctl net.inet.ip.forwarding=1
sysctl net.inet6.ip6.forwarding=1

To make those configurations permanent, they can be configured in /etc/sysctl.conf but the following line in /etc/rc.conf looks nicer to me.

[/etc/rc.conf]

gateway_enable="YES"
ipv6_gateway_enable="YES"

Separate Network Configuration

This is an example configuration for five VNET jails in a network separated from the host. I used this configuration to experiment network software such as WireGuard VPN (see this article).

In this configuration, one jail acts as a central router which crudely emulates a public network and is to be used as a monitoring post running tcpdump, two are routers on private sites and remaining two are hosts on the private sites.

                                 Site 1

                      vri1_vpnr1      vri1_vpnh1
       [Router(vpnr1)]o ------ (vri1br) ------ o[Host (vpnh1)]
     vi1_vpnr1 o      192.168.1.1   192.168.1.11
   172.31.1.11 |
               |
            (vi1br)
               |
   172.31.1.1  |
        vi1_r1 o
          [Router(r1)]
        vi2_r1 o
   172.31.2.1  |
               |
            (vi2br)
               |
   172.31.2.11 |
               |
     vi2_vpnr2 o      192.168.2.1   192.168.2.11
       [Router(vpnr2)]o ------ (vri2br) ------ o[Host (vpnh2)]
                      vri2_vpnr2      vri2_vpnh2

                                 Site 2

To use tcpdump on the central router (r1), define the following devfs ruleset which is specified by devfs_ruleset parameter in /etc/jail.conf.

[/etc/devfs.rules]

[devfsrules_bpfjail=10]
add include $devfsrules_jail
add path 'bpf*' unhide

[devfsrules_tunjail=11]
add include $devfsrules_jail
add path 'tun*' unhide

The whole jail.conf looks like this.

[/etc/jail.conf]

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = "/vm/$name";
exec.consolelog = "/var/log/jail_${name}_console.log";

allow.chflags;
allow.raw_sockets;
mount = "/var/cache/pkg /vm/${name}/mnt nullfs ro 0 0";

r1 {
        $net1 = "vi1";
        $ipv41 = "172.31.1.1";

        $net2 = "vi2";
        $ipv42 = "172.31.2.1";

        $plen = 24;

        vnet;
        vnet.interface = ${net1}_$name, ${net2}_$name;
        exec.prestart += "vnet add -b $net1 ${net1}_$name";
        exec.prestart += "vnet add -b $net2 ${net2}_$name";
        exec.start += "ifconfig ${net1}_$name $ipv41/$plen";
        exec.start += "ifconfig ${net2}_$name $ipv42/$plen";
        exec.start += "sysctl net.inet.ip.forwarding=1";
        exec.poststop += "vnet delete $net1 ${net1}_$name";
        exec.poststop += "vnet delete $net2 ${net2}_$name";

        devfs_ruleset = 10;
}

vpnr1 {
        $net1 = "vi1";
        $ipv41 = "172.31.1.11";

        $net2 = "vri1";
        $ipv42 = "192.168.1.1";

        $gwv4 = "172.31.1.1";
        $plen = 24;

        vnet;
        vnet.interface = ${net1}_$name, ${net2}_$name;
        exec.prestart += "vnet add -b $net1 ${net1}_$name";
        exec.prestart += "vnet add -b $net2 ${net2}_$name";
        exec.start += "ifconfig ${net1}_$name $ipv41/$plen";
        exec.start += "ifconfig ${net2}_$name $ipv42/$plen";
        exec.start += "route add default $gwv4";
        exec.start += "sysctl net.inet.ip.forwarding=1";
        exec.poststop += "vnet delete $net1 ${net1}_$name";
        exec.poststop += "vnet delete $net2 ${net2}_$name";

        devfs_ruleset = 11;
}

vpnr2 {
        $net1 = "vi2";
        $ipv41 = "172.31.2.11";

        $net2 = "vri2";
        $ipv42 = "192.168.2.1";

        $gwv4 = "172.31.2.1";
        $plen = 24;

        vnet;
        vnet.interface = ${net1}_$name, ${net2}_$name;
        exec.prestart += "vnet add -b $net1 ${net1}_$name";
        exec.prestart += "vnet add -b $net2 ${net2}_$name";
        exec.start += "ifconfig ${net1}_$name $ipv41/$plen";
        exec.start += "ifconfig ${net2}_$name $ipv42/$plen";
        exec.start += "route add default $gwv4";
        exec.start += "sysctl net.inet.ip.forwarding=1";
        exec.poststop += "vnet delete $net1 ${net1}_$name";
        exec.poststop += "vnet delete $net2 ${net2}_$name";

        devfs_ruleset = 11;
}

vpnh1 {
        $net = "vri1";
        $ipv4 = "192.168.1.11";

        $gwv4 = "192.168.1.1";
        $plen = 24;

        vnet;
        vnet.interface = "${net}_$name";
        exec.prestart += "vnet add -b ${net} ${net}_$name";
        exec.start += "ifconfig ${net}_$name $ipv4/$plen";
        exec.start += "route add default $gwv4";
        exec.poststop += "vnet delete $net ${net}_$name";
}

vpnh2 {
        $net = "vri2";
        $ipv4 = "192.168.2.11";

        $gwv4 = "192.168.2.1";
        $plen = 24;

        vnet;
        vnet.interface = "${net}_$name";
        exec.prestart += "vnet add -b ${net} ${net}_$name";
        exec.start += "ifconfig ${net}_$name $ipv4/$plen";
        exec.start += "route add default $gwv4";
        exec.poststop += "vnet delete $net ${net}_$name";
}
See Network Configuration

The helper script can list networks (roughly equals to ng_bridges), jails, interfaces (ng_eiface) and their addresses.

$ sudo vnet list -r
vi1
  r1 vi1_r1 172.31.1.1/24
  vpnr1 vi1_vpnr1 172.31.1.11/24
vi2
  r1 vi2_r1 172.31.2.1/24
  vpnr2 vi2_vpnr2 172.31.2.11/24
vri1
  vpnh1 vri1_vpnh1 192.168.1.11/24
  vpnr1 vri1_vpnr1 192.168.1.1/24
vri2
  vpnh2 vri2_vpnh2 192.168.2.11/24
  vpnr2 vri2_vpnr2 192.168.2.1/24
pkg without Network Connection

Because the jails in this configuration cannot access outside network, pkg install cannot be used. Instead you can use pkg add to install the package files downloaded with pkg fetch.

I use the following procedures.

  1. Download package files on the host.
    With -d option, pkg fetch downloads the specified package plus its dependencies.
    Downloaded files are stored in the host’s /var/cache/pkg.

    pkg fetch -d wireguard
    
  2. Make the downloaded package files available to jails.
    Although you can simply copy the files to jails’ filesystem but nullfs mount /var/cache/pkg is much better and easier.
    The following line in /etc/jail.conf achieves this. The host’s /var/cache/pkg is mounted on the jails’ /mnt directory.

    mount = "/var/cache/pkg /vm/${name}/mnt nullfs ro 0 0";
    
  3. Start the jails and install the package from the host.

    pkg -j vpnr1 add /mnt/wireguard-0.0.20181218.txz
    [vpnr1] Installing wireguard-0.0.20181218...
    [vpnr1] `-- Installing bash-4.4.23_1...
    [vpnr1] |   `-- Installing gettext-runtime-0.19.8.1_2...
    [vpnr1] |   | `-- Installing indexinfo-0.3.1...
    [vpnr1] |   | `-- Extracting indexinfo-0.3.1: 100%
    [vpnr1] |   `-- Extracting gettext-runtime-0.19.8.1_2: 100%
    [vpnr1] `-- Extracting bash-4.4.23_1: 100%
    [vpnr1] `-- Installing wireguard-go-0.0.20181222...
    [vpnr1] `-- Extracting wireguard-go-0.0.20181222: 100%
    [vpnr1] Extracting wireguard-0.0.20181218: 100%
    

References

Revision History