Migrating a live system from GEOM mirror to ZFS mirror


I had been wanting to migrate a FreeBSD system on a GEOM mirror (gmirror) to a ZFS mirror.

After several rehearsals on a VM, I have finally achieved that goal with the following steps.

  1. Load ZFS-releated kernel modules and set a tunable to use 4K sector drives.

    sudo kldload zfs
    sudo sysctl vfs.zfs.min_auto_ashift=12
    
  2. Remove one (da0) of the two disks (da0, da1) which make up the gmirror (gm0).

    sudo gmirror remove gm0 da0
    
  3. Re-create partitions on the disk removed from the gmirror (da0). Label them appropriately.

    sudo gpart destroy -F da0
    sudo gpart create -s gpt da0
    sudo gpart add -a 4k -t freebsd-boot -s 512k -l zfsboot0 da0
    sudo gpart add -a 1m -t freebsd-zfs -l zfssys0 da0
    sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
    

    GEOM mirrored disks usually use MBR partitioning because both gmirror and GPT(another partitioning scheme) store metadata at the end of the disk.

  4. Create a ZFS pool (zroot) with a partition (gpt/zfssys0) on da0.

    sudo zpool create -O compression=lz4 -O atime=off -o altroot=/mnt -m none zroot gpt/zfssys0
    
  5. Export and re-import the pool and mount it on a temporary location (/mnt).

    sudo zpool export zroot
    sudo zpool import -o altroot=/mnt zroot
    
  6. Create datasets in a Boot Environment compliant layout.

    sudo zfs create -o mountpoint=none zroot/ROOT
    sudo zfs create -o mountpoint=/ zroot/ROOT/default
    sudo zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
    sudo zfs create -o mountpoint=/usr -o canmount=off zroot/usr
    sudo zfs create -o canmount=off zroot/usr/local
    sudo zfs create -o mountpoint=/home zroot/home
    sudo zfs create -o setuid=off zroot/usr/ports
    sudo zfs create zroot/usr/src
    sudo zfs create zroot/usr/obj
    sudo zfs create -o mountpoint=/var -o canmount=off zroot/var
    sudo zfs create -o exec=off -o setuid=off zroot/var/audit
    sudo zfs create -o exec=off -o setuid=off zroot/var/crash
    sudo zfs create -o exec=off -o setuid=off zroot/var/log
    sudo zfs create -o atime=on zroot/var/mail
    sudo zfs create -o setuid=off zroot/var/tmp
    sudo zfs create -o mountpoint=/data zroot/data
    
  7. Set bootfs property and fix permissions on temporay directories.

    sudo zpool set bootfs=zroot/ROOT/default zroot
    sudo chmod 1777 /mnt/tmp
    sudo chmod 1777 /mnt/var/tmp
    
  8. Copy files from the gmirror to the new ZFS pool.

    sudo sh -c "find -x / | cpio -pmd /mnt"
    sudo sh -c "find -x /var | cpio -pmd /mnt"
    sudo sh -c "find -x /usr | cpio -pmd /mnt"
    sudo sh -c "find -x /home | cpio -pmd /mnt"
    sudo sh -c "find -x /data | cpio -pmd /mnt
    
  9. Modify configuration files on the pool.

    [/mnt/boot/loader.conf]

    # geom_mirror_load="YES"
    kern.geom.label.disk_ident.enable="0"
    kern.geom.label.gptid.enable="0"
    zfs_load="YES"
    

    [/mnt/etc/sysctl.conf]

    vfs.zfs.min_auto_ashift=12
    

    [/mnt/etc/rc.conf]

    zfs_enable="YES"
    

    [/mnt/etc/fstab]

    # comment out all mount entries for gmirror (gm0).
    
  10. Reboot into ZFS.

    sudo shutdown -r now
    
  11. After reboot, destroy the remaining gmirror (gm0) which is no longer used.

    sudo gmirror load
    sudo gmirror destroy gm0
    
  12. Re-create partitions on da1 which had remained in the gmirror.

    sudo gpart destroy -F da1
    sudo gpart create -s gpt da1
    sudo gpart add -a 4k -t freebsd-boot -s 512k -l zfsboot0 da1
    sudo gpart add -a 1m -t freebsd-zfs -l zfssys0 da1
    sudo gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
    
  13. Add a partition (gpt/zfssys1) on da1 to the ZFS pool (zroot) to form a mirror again. Resilvering starts.

    NOTE: To add mirrored disks, use zpool attach instead of zpool add. The latter expands a pool by adding the specified vdev as another stripe.

    sudo zpool attach zroot gpt/zfssys0 gpt/zfssys1
    
  14. When resilvering is done, you’ve got a brand new ZFS mirror pool!

References