🐝 bhyve: Virtual Machines Built into FreeBSD
Create a virtual machine with nothing but the base system: a ZFS volume for a disk, a private network, a serial console, and a FreeBSD guest booted from the install disc
What bhyve is
bhyve (pronounced "beehive") is FreeBSD's hypervisor: the part of the system that runs whole virtual machines. Unlike a jail, which shares the host's kernel, a bhyve guest is a complete machine with its own kernel, so it can run a different operating system. It is part of the base system, so there is nothing to install: a kernel module (vmm) and three tools, bhyve (runs the machine), bhyveload (loads a FreeBSD kernel into it) and bhyvectl (controls and inspects it).
If you have read the Proxmox pages on this site, think of bhyve as the engine in the same role as KVM and QEMU, only built in, small and driven from the command line.
Two honest notes about this page:
- bhyve needs hardware support for virtualization: Intel VT-x with EPT, or AMD-V with nested paging.
- Our test machine is itself a virtual machine, so this is a nested setup: a bhyve guest inside a QEMU virtual machine on a laptop. It works, and everything below is real output, but it is slower than on a real server. Real servers run bhyve directly on the hardware.
Step 1: is the CPU ready, and load the module
root@practice:~ # grep 'AMD Features2' /var/run/dmesg.boot AMD Features2=0x8003f7<LAHF,CMP,SVM,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,PCXC> root@practice:~ # kldload vmm root@practice:~ # kldstat | grep vmm 7 1 0xffffffff83200000 340438 vmm.ko
- The boot messages list the processor's features. On AMD, SVM (Secure Virtual Machine) in the AMD Features2 line is the one we need. (On Intel the matching flag is VMX, shown in Features2: from the documentation, not run here.) Our virtual machine passes it through from the laptop, which is what nested virtualization means.
- kldload vmm loads the virtual machine monitor module. It printed nothing, which means success, and kldstat shows vmm.ko.
- To load it at every boot, we ran sysrc kld_list+=vmm. The setting in /etc/rc.conf is kld_list="vmm".
root@practice:~ # sysrc kld_list+=vmm kld_list: -> vmm root@practice:~ # grep -n 'vmm' /etc/rc.conf 9:kld_list="vmm"
Step 2: a disk, from ZFS
A guest needs a disk. The best disk for a bhyve guest on FreeBSD is a ZFS volume, a "zvol": a block device carved out of the pool. You get everything from the ZFS page for free: snapshots before risky changes, clones, and zfs send for backups.
root@practice:~ # zfs create -p -V 4G -o volmode=dev zroot/vm/guest0 root@practice:~ # ls -l /dev/zvol/zroot/vm/guest0 crw-r----- 1 root operator 0x6b Oct 6 01:32 /dev/zvol/zroot/vm/guest0 root@practice:~ # zfs list -r -t volume zroot/vm NAME USED AVAIL REFER MOUNTPOINT zroot/vm/guest0 4.06G 11.1G 56K -
zfs create -V 4G -o volmode=dev zroot/vm/guest0 made a 4 GB volume (the -p created the parent zroot/vm). volmode=dev makes it appear as a device file, /dev/zvol/zroot/vm/guest0, which is what bhyve opens. Note USED 4.06G: the space is reserved up front. (A "sparse" volume that grows as needed is made with -s: from the documentation, not run here.)
A note from a later test run on a fresh machine: with -o volmode=dev the device file sometimes does not appear at all (there was no /dev/zvol folder, and bhyve then stops with Could not open backing file). Creating the volume with the default mode, zfs create -V 4G zroot/vm/guest0, gave the device file every time, and it works for bhyve just the same. If you want volmode=dev, check with ls -l /dev/zvol/zroot/vm/ before you start the guest. The vm-bhyve page describes the same problem.
Step 3: a private network
The guest needs a network card. bhyve connects it to a tap interface (the virtual end of a cable), and a bridge acts as the virtual switch. We built a small private network that is not connected to anything else, which is the safest way to start:
root@practice:~ # sysctl net.link.tap.up_on_open=1 net.link.tap.up_on_open: 0 -> 1 root@practice:~ # ifconfig tap0 create root@practice:~ # ifconfig bridge0 create root@practice:~ # ifconfig bridge0 addm tap0 root@practice:~ # ifconfig bridge0 inet 10.77.0.1/24 up root@practice:~ # ifconfig bridge0 bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=10<VLAN_HWTAGGING> ether 58:9c:fc:10:bd:50 inet 10.77.0.1 netmask 0xffffff00 broadcast 10.77.0.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 bridge flags=0<> member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> port 4 priority 128 path cost 2000000 vlan protocol 802.1q groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED>
- net.link.tap.up_on_open=1 makes a tap interface come up when the guest opens it.
- ifconfig tap0 create and ifconfig bridge0 create make the two virtual devices, and addm tap0 makes the tap a member of the bridge.
- The bridge itself got the address 10.77.0.1/24, so the host will be on the same little network as the guest. The listing shows member: tap0.
To give a guest real network access, you add the physical card as a bridge member too (addm vtnet0 in our case). Do that carefully on a remote machine, since the card's address then moves to the bridge: from the documentation, not run here.
Step 4: the install disc must be a file
We want to boot a FreeBSD guest from the same installation disc that this machine came from. A first try pointed the loader at the disc device:
root@practice:~ # bhyveload -c stdio -m 768M -d /dev/cd0 guest0 Consoles: userboot FreeBSD/amd64 User boot lua, Revision 3.0 ERROR: cannot open /boot/lua/loader.lua: device not configured. Type '?' for a list of commands, 'help' for more detailed help. OK
The loader could not read /boot/lua/loader.lua: device not configured. bhyveload wants a regular file (a disk image or an ISO file), not the raw CD device. (On a terminal, bhyveload then waits at the OK prompt for more commands; we ended it with timeout 5 in front.) So we copied the disc into a file:
root@practice:~ # time dd if=/dev/cd0 of=/var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso bs=1m
4201+1 records in
4201+1 records out
4405243904 bytes transferred in 16.114265 secs (273375413 bytes/sec)
16.11 real 0.04 user 2.72 sys
root@practice:~ # ls -lh /var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso
-rw-r--r-- 1 root wheel 4.1G Oct 6 01:37 /var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso
16 seconds for 4.1 GB.
Step 5: load the kernel and start the machine
In two steps. First bhyveload loads the FreeBSD kernel from the ISO into the guest's memory. Then bhyve runs the machine:
root@practice:~ # bhyveload -c stdio -m 768M -d /var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso guest0 Consoles: userboot FreeBSD/amd64 User boot lua, Revision 3.0 [... the FreeBSD boot menu and its logo are left out ...] Loading kernel... Loading configured modules... root@practice:~ # bhyve -c 1 -m 768M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/dev/zvol/zroot/vm/guest0 -s 4:0,ahci-cd,/var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso -l com1,stdio guest0 [... most of the guest kernel messages are left out ...] FreeBSD 15.0-RELEASE releng/15.0-n280995-7aedc8de6446 GENERIC amd64 real memory = 805306368 (768 MB) avail memory = 742187008 (707 MB) vtnet0: <VirtIO Networking Adapter> on virtio_pci0 vtnet0: Ethernet address: 00:a0:98:d1:c4:c3 vtnet0: netmap queues/slots: TX 1/1024, RX 1/512 uart0: console (115200,n,8,1) Trying to mount root from cd9660:/dev/iso9660/15_0_RELEASE_AMD64_DVD [ro]... cd0: <BHYVE BHYVE DVD-ROM 001> Removable CD-ROM SCSI device cd0: Serial Number BHYVE-3F04-FA29-E8DC cd0: 600.000MB/s transfers (SATA 3.x, UDMA6, ATAPI 12bytes, PIO 8192bytes) [... boot continues ...] Console type [vt100]:
The long bhyve command has a lot in it, but each part is simple:
- -c 1 and -m 768M: one virtual CPU and 768 MB of memory.
- -H (let the host have the processor back when the guest is idle), -A (give the guest the ACPI tables it expects) and -P (make the guest exit to the host when it runs a PAUSE instruction, which spinning code uses, so that it cannot hog a core).
- -s SLOT,DEVICE: put a device in a PCI slot. 0:0,hostbridge and 1:0,lpc are the basics (the lpc bridge also provides the serial port), 2:0,virtio-net,tap0 is the network card connected to our tap, 3:0,virtio-blk,/dev/zvol/zroot/vm/guest0 is the disk, our ZFS volume, and 4:0,ahci-cd,FILE is a DVD drive holding the ISO.
- -l com1,stdio: the guest's serial port is this terminal. There is no screen and no window: the console is text in your SSH session.
- guest0: the machine's name, which bhyveload used too.
And there it is, the boot of a second FreeBSD 15.0-RELEASE: 768 MB of memory, a VirtIO Networking Adapter (vtnet0), the DVD drive (cd0), and finally the installer asking for the terminal type. (We left out most of the kernel messages. The boot took a minute or two in this nested setup.) We pressed Enter for vt100, and at the installer's welcome dialog chose Shell (the S key) to get a command prompt instead of installing.
Inside the guest
# uname -a FreeBSD 15.0-RELEASE FreeBSD 15.0-RELEASE releng/15.0-n280995-7aedc8de6446 GENERIC amd64 # sysctl kern.vm_guest hw.ncpu hw.physmem kern.vm_guest: bhyve hw.ncpu: 1 hw.physmem: 769253376 # geom disk list | grep -E "^Geom name|Mediasize" Geom name: vtbd0 Mediasize: 4294967296 (4.0G) Geom name: cd0 Mediasize: 4405243904 (4.1G) # dmesg | grep -E "^vtnet0|^vtbd0" | head -4 vtnet0: <VirtIO Networking Adapter> on virtio_pci0 vtnet0: Ethernet address: 00:a0:98:d1:c4:c3 vtnet0: netmap queues/slots: TX 1/1024, RX 1/512
- kern.vm_guest: bhyve: the guest knows it is a virtual machine, and which hypervisor runs it. (The program uname shows an empty hostname because this is the install disc's live shell.)
- One CPU and 769 MB, exactly what we gave it.
- geom disk list shows two disks: vtbd0, 4.0G, which is our ZFS volume as the guest sees it (a VirtIO block device), and cd0, the DVD.
- The dmesg lines confirm the VirtIO network card.
And from the host's side:
root@practice:~ # ls /dev/vmm guest0 root@practice:~ # bhyvectl --vm=guest0 --get-stats | head -4 vcpu0 stats: number of times rendezvous pending at exit 0 number of vm exits due to exceptions 0 number of ticks vcpu was idle 533
Every running guest has an entry in /dev/vmm, so ls /dev/vmm lists the virtual machines on the machine. bhyvectl --vm=guest0 --get-stats prints counters for the guest (vcpu0 stats, how many times the guest left to the hypervisor, how long it was idle). bhyvectl can do much more, such as pausing or inspecting memory: from the documentation, not run here.
The guest's network, and an old friend
Give the guest an address on our private network and ping the host:
# ifconfig vtnet0 inet 10.77.0.2/24 up # ping -c 2 -t 4 10.77.0.1 PING 10.77.0.1 (10.77.0.1): 56 data bytes --- 10.77.0.1 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss
100 percent packet loss. Why? The host is running the firewall from the pf page, and its default rule is block return in all, which applies to every interface, including the new bridge0. Packets from the guest arrive on bridge0 and the firewall refuses them. The cure is a one-line change, to the line near the top of /etc/pf.conf:
root@practice:~ # pfctl -sr | head -2
block return in all
block drop in log quick proto tcp from <badguys> to any port = http
root@practice:~ # sed -i '' 's/set skip on lo0/set skip on { lo0 bridge0 }/' /etc/pf.conf
root@practice:~ # pfctl -nf /etc/pf.conf
root@practice:~ # pfctl -f /etc/pf.conf
set skip on { lo0 bridge0 } tells pf to leave the bridge alone, as it already does for the loopback. We checked the file with pfctl -nf and reloaded it with pfctl -f. Try again:
# ping -c 2 -t 4 10.77.0.1 PING 10.77.0.1 (10.77.0.1): 56 data bytes 64 bytes from 10.77.0.1: icmp_seq=0 ttl=64 time=0.798 ms 64 bytes from 10.77.0.1: icmp_seq=1 ttl=64 time=1.490 ms --- 10.77.0.1 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.798/1.144/1.490/0.346 ms root@practice:~ # ping -c 2 -t 4 10.77.0.2 PING 10.77.0.2 (10.77.0.2): 56 data bytes 64 bytes from 10.77.0.2: icmp_seq=0 ttl=64 time=1.800 ms 64 bytes from 10.77.0.2: icmp_seq=1 ttl=64 time=0.998 ms --- 10.77.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.998/1.399/1.800/0.401 ms
Both directions work, in a millisecond or two: the guest reaches the host, and the host (the root@practice prompt at the bottom) reaches the guest. Remember this one: a default-deny firewall also blocks the traffic of your virtual machines until you tell it not to.
The disk really is the ZFS volume
A last proof that the guest's disk and the host's volume are the same thing:
# echo "hello from the guest" | dd of=/dev/vtbd0 conv=sync 2>&1 1+0 records in 1+0 records out 512 bytes transferred in 0.042310 secs (12101 bytes/sec) root@practice:~ # head -c 22 /dev/zvol/zroot/vm/guest0; echo hello from the guest
The guest wrote one 512-byte block to /dev/vtbd0, and the host read the very same bytes straight from /dev/zvol/zroot/vm/guest0: hello from the guest. (Do not write to a disk that holds an installed system, of course. This one was empty.)
Shutting down
# halt -p [... the guest shuts down and bhyve exits ...] root@practice:~ # echo "exit status: $?" exit status: 1 root@practice:~ # ls /dev/vmm guest0 root@practice:~ # bhyvectl --vm=guest0 --destroy root@practice:~ # ls /dev/vmm ls: /dev/vmm: No such file or directory
halt -p in the guest powers it off, and bhyve exits on the host. Its exit status was 1, which in bhyve means "the guest powered off" (a restart is 0, a halt is 2, and 3 means the guest crashed hard: from the documentation, not run here). The process is gone, but the machine still exists: /dev/vmm still lists guest0 (a test run on a fresh machine showed this; an earlier run had found it already gone). bhyvectl --vm=guest0 --destroy removes it, and afterwards /dev/vmm is empty. Destroy a guest you have finished with, or the next start with the same name will find the old one.
That last command is for the other situation: a guest that did not exit cleanly. Earlier, while we were getting this to work, we killed some half-started bhyve processes, and the next attempt failed with Device busy, because the old machine still held its name. The way out was bhyvectl --vm=guest0 --destroy after the stuck processes were gone. In general, power off the guest from inside when you can, and use kill -9 only as a last resort.
What we did not do
All of these are from the documentation, not run here:
- Install onto the volume: run the installer (or bsdinstall with a script) in the guest, then boot the installed system with bhyveload -d /dev/zvol/zroot/vm/guest0 guest0 and the same bhyve line without the DVD.
- Non-FreeBSD guests: Linux, Windows and others boot with UEFI firmware (the edk2-bhyve package) instead of bhyveload: -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd.
- A graphical console over VNC: -s 5:0,fbuf,tcp=127.0.0.1:5900 plus a keyboard and mouse device.
- Managers: most people drive bhyve with a helper such as vm-bhyve or CBSD, which turn the long command line into vm create, vm install and vm start, and take care of the tap and bridge devices. The vm-bhyve page tests it for real. Learn the plain commands first and the helpers will make sense.
- Autostart, persistent bridges (cloned_interfaces in /etc/rc.conf), CPU pinning, PCI passthrough and snapshots of running guests.
bhyve, jails and the rest
- Jails share the host's kernel: very light, FreeBSD only, best for services.
- bhyve gives each guest its own kernel: any operating system, strong separation, a bit heavier.
- Proxmox/KVM (see the Proxmox pages on this site) is the same idea with a web interface, clusters and many extra features, on Linux.
- Both bhyve and jails love ZFS, and both make it easy to build, test and throw away machines.
⚠️ Do not write to disks you do not mean to. A bhyve guest sees whatever device you give it, and a typo in the -s line (a real disk instead of a volume) hands it your data. Use zvols or image files, and double-check the path.
He heard it can run several guests<br>
without a single cell of overhead.
🧪 Try it yourself
On a FreeBSD machine with a CPU that supports virtualization (a physical machine, or a VM with nested virtualization on):
1. Check the CPU for SVM or VMX, load vmm and confirm with kldstat. 2. Make a ZFS volume of 4 GB for a guest and look at its device file. 3. Create a tap, a bridge, put the tap on the bridge and give the bridge an address. 4. Copy the installation ISO to a file. Boot it with bhyveload and bhyve, and choose the live shell. 5. Find the guest's CPU count, memory and disks. What does kern.vm_guest say? 6. Give the guest an address on the bridge's network and ping the host. If it fails, what might a firewall be doing? 7. Write a line into the guest's disk and read it from the host. 8. Shut the guest down and check that /dev/vmm is empty.
The cheat sheet
- Ready? grep 'AMD Features2' /var/run/dmesg.boot (look for SVM), kldload vmm, sysrc kld_list+=vmm.
- Disk: zfs create -V 4G -o volmode=dev zroot/vm/NAME gives /dev/zvol/zroot/vm/NAME.
- Network: ifconfig tap0 create, ifconfig bridge0 create, ifconfig bridge0 addm tap0, sysctl net.link.tap.up_on_open=1.
- Boot: bhyveload -c stdio -m 768M -d FILE NAME, then bhyve -c 1 -m 768M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,DISK -s 4:0,ahci-cd,ISO -l com1,stdio NAME.
- Look: ls /dev/vmm, bhyvectl --vm=NAME --get-stats.
- Stop: halt -p in the guest (bhyve exits with status 1). bhyvectl --vm=NAME --destroy for a stuck guest.
- Firewall: set skip on { lo0 bridge0 } in /etc/pf.conf.
← The Ports Tree: Building FreeBSD Software from Source · ↑ FreeBSD Corner · poudriere: Build Your Own Packages, Properly →