🐝 vm-bhyve: Virtual Machines with Short Commands
The manager that turns the long bhyve command line into vm create, vm install and vm start, with a tested walkthrough: install, boot, network, snapshot and clone
Why a manager?
On the bhyve page we started a virtual machine with one very long command: slots, devices, a tap, a bridge, a disk path, a console. It teaches you how bhyve works, and it is miserable to repeat. vm-bhyve is the helper most people use on top of it. It is a set of shell scripts (the package puts them in /usr/local/lib/vm-bhyve) that keeps each virtual machine's settings in a small file, creates the tap devices, bridges and ZFS volumes for you, and gives you a single command, vm, with sub-commands: vm create, vm install, vm start, vm stop, vm snapshot, vm clone and so on.
Everything below was run for real on FreeBSD 15.0-RELEASE, in the same nested lab as the bhyve page: a virtual machine (2 processors, 2 GB) running guests inside it. It is slower than real hardware, and it works.
Step 1: install and initialise
root@practice:~ # pkg install -y vm-bhyve 2>&1 | grep -E "vm-bhyve|Number" vm-bhyve: 1.7.5 [FreeBSD-ports] Number of packages to be installed: 1 [1/1] Fetching vm-bhyve-1.7.5: ......... done [1/1] Installing vm-bhyve-1.7.5... [1/1] Extracting vm-bhyve-1.7.5: .......... done
(A note from our own mistakes: the first pkg install said No packages available to install matching 'vm-bhyve'. On the poudriere page we had switched the official repositories off on this machine, so pkg could only see our own small repository. We removed our repository file and ran pkg update -f, and it was found.) Two lines in /etc/rc.conf switch the manager on:
root@practice:~ # sysrc vm_enable=YES vm_enable: -> YES root@practice:~ # sysrc vm_dir="zfs:zroot/vm" vm_dir: -> zfs:zroot/vm root@practice:~ # vm init root@practice:~ # ls -a /zroot/vm . .. .config .img .iso .templates root@practice:~ # ls /usr/local/share/examples/vm-bhyve config.sample freebsd-zvol.conf legacy-arch.conf legacy-ubuntu.conf linux-zvol.conf openbsd.conf default.conf freepbx.conf legacy-debian.conf linux-grub-zvol.conf linux.conf resflash.conf dragonfly.conf legacy-alpine.conf legacy-gentoo.conf linux-grub.conf netbsd.conf windows.conf root@practice:~ # cp /usr/local/share/examples/vm-bhyve/* /zroot/vm/.templates/ root@practice:~ # ls /zroot/vm/.templates config.sample freebsd-zvol.conf legacy-arch.conf legacy-ubuntu.conf linux-zvol.conf openbsd.conf default.conf freepbx.conf legacy-debian.conf linux-grub-zvol.conf linux.conf resflash.conf dragonfly.conf legacy-alpine.conf legacy-gentoo.conf linux-grub.conf netbsd.conf windows.conf
- vm_enable="YES" starts it at boot (and starts the virtual machines you list for autostart, see the end of the page).
- vm_dir="zfs:zroot/vm" says: keep everything in this ZFS dataset. (A plain folder works too: vm_dir="/vm".) Each guest becomes a child dataset, so snapshots, clones and backups come from ZFS.
- vm init creates the folder structure: .config, .img (disk images), .iso (install media) and .templates.
- The package ships templates as examples: freebsd-zvol, linux, windows, openbsd and more. A template is just a starting settings file for a type of guest. We copied them all into .templates.
Step 2: a virtual switch
root@practice:~ # vm switch create -a 10.88.0.1/24 private
root@practice:~ # vm switch list
NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS
private standard vm-private 10.88.0.1/24 no - - -
root@practice:~ # ifconfig vm-private | grep -E "^vm-private|inet |member"
vm-private: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
inet 10.88.0.1 netmask 0xffffff00 broadcast 10.88.0.255
root@practice:~ # grep 'skip' /etc/pf.conf
set skip on { lo0 bridge0 }
root@practice:~ # sed -i '' 's/set skip on { lo0 bridge0 }/set skip on { lo0 bridge0 vm-private }/' /etc/pf.conf
root@practice:~ # pfctl -nf /etc/pf.conf
root@practice:~ # pfctl -f /etc/pf.conf
root@practice:~ # grep 'skip' /etc/pf.conf
set skip on { lo0 bridge0 vm-private }
vm switch create -a 10.88.0.1/24 private made a virtual switch named private. Behind it vm-bhyve built a bridge, vm-private, with the address 10.88.0.1/24 for the host. Every guest connected to this switch will be on that network, and nothing else: it is a private network inside the machine. (A switch that is joined to a real network card, so that guests appear on your LAN, is made with vm switch add: from the documentation, not run here.)
Then the firewall: the host runs pf from the pf page, whose default rule blocks every interface. We added the new bridge to the interfaces that pf skips, as we did for bridge0 on the bhyve page, and the problem never appeared. Any bridge that carries your guests' traffic needs this.
Step 3: install media
root@practice:~ # time dd if=/dev/cd0 of=/var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso bs=1m
4201+1 records in
4201+1 records out
4405243904 bytes transferred in 8.345128 secs (527882148 bytes/sec)
8.34 real 0.05 user 1.62 sys
root@practice:~ # vm iso /var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso
/zroot/vm/.iso/FreeBSD-15.0-RELEASE-dvd1.iso 1% of 4201 MB [... progress shown while it copies ...]
root@practice:~ # vm iso
DATASTORE FILENAME
default FreeBSD-15.0-RELEASE-dvd1.iso
We copied the installation disc into a file (8 seconds), because, as on the bhyve page, the loader needs a regular file. vm iso FILE imports it into the .iso folder, and plain vm iso lists the available images. (Given a web address, vm iso URL downloads an image for you: from the documentation, not run here.)
Step 4: create a guest
root@practice:~ # vm create -t freebsd-zvol -s 4G -c 1 -m 768M guest1 root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - No Stopped root@practice:~ # cat /usr/local/share/examples/vm-bhyve/freebsd-zvol.conf loader="bhyveload" cpu=1 memory=256M network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="disk0" disk0_dev="sparse-zvol" root@practice:~ # cat /zroot/vm/guest1/guest1.conf loader="bhyveload" cpu="1" memory="768M" network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="disk0" disk0_dev="sparse-zvol" uuid="e7fc4fa2-c18c-11f1-ae02-589cfc10bd50" network0_mac="58:9c:fc:03:60:8b" root@practice:~ # sysrc -f /zroot/vm/guest1/guest1.conf network0_switch="private" network0_switch: public -> private root@practice:~ # grep network0 /zroot/vm/guest1/guest1.conf network0_type="virtio-net" network0_switch="private" network0_mac="58:9c:fc:03:60:8b" root@practice:~ # zfs list -r -o name,used,volsize zroot/vm NAME USED VOLSIZE zroot/vm 3.59G - zroot/vm/guest1 156K - zroot/vm/guest1/disk0 56K 4G root@practice:~ # ls /zroot/vm/guest1 guest1.conf vm-bhyve.log
One command made the guest: vm create -t freebsd-zvol -s 4G -c 1 -m 768M guest1 (template, 4 GB disk, 1 CPU, 768 MB of memory, name). vm list shows it, Stopped. Now read the guest's settings file, which is the whole configuration of the machine:
- loader="bhyveload" loads a FreeBSD kernel, as before.
- cpu and memory: what we asked for (the template's own default was 256M).
- network0_type="virtio-net" and network0_switch: the network card and which switch it is plugged into. The template says public, a switch that does not exist here, so we changed it to private with sysrc -f (the same tool you use for rc.conf, pointed at this file). network0_mac is a fixed address generated for the guest, so it keeps the same one every time.
- disk0_type="virtio-blk", disk0_dev="sparse-zvol": the disk is a sparse ZFS volume. In the zfs list at the end of the listing it is 4G in size and uses 56K, because nothing is written to it yet.
- vm configure guest1 opens this file in your editor (from the documentation, not run here).
A small trap: the missing device file
Our first attempt to start the guest failed, with no message on screen. The manager's log said why, or at least how:
root@practice:~ # tail -4 /zroot/vm/guest1/vm-bhyve.log 2026-10-06T13:51:04+00:00: [bhyve iso device: -s 3:0,ahci-cd,/zroot/vm/.iso/FreeBSD-15.0-RELEASE-dvd1.iso,ro,bootindex=1] 2026-10-06T13:51:04+00:00: starting bhyve (run 1) 2026-10-06T13:51:04+00:00: bhyve exited with status 4 2026-10-06T13:51:04+00:00: destroying network device tap0 2026-10-06T13:51:04+00:00: stopped root@practice:~ # [the bhyve command from the log, run by hand to see its error message] bhyve: Could not open backing file: /dev/zvol/zroot/vm/guest1/disk0: No such file or directory Could not open backing file: No such file or directory Device emulation initialization error: No such file or directory root@practice:~ # ls /dev/zvol ls: /dev/zvol: No such file or directory
bhyve exited with status 4 is all that vm-bhyve knows. Running the same bhyve command by hand showed the real message: the disk's device file, /dev/zvol/zroot/vm/guest1/disk0, did not exist, and there was no /dev/zvol folder at all. The volume was there and healthy. The device file for it was not.
We tried to find out when this happens:
root@practice:~ # zfs create -s -V 8M -o volmode=dev zroot/vm/t3 root@practice:~ # ls /dev/zvol/zroot/vm/ guest1 root@practice:~ # zfs create -V 8M -o volmode=dev zroot/vm/t4 root@practice:~ # ls /dev/zvol/zroot/vm/ guest1 t4
A sparse volume (-s) created with volmode=dev got no device file, while the same volume without -s did. vm-bhyve's template uses exactly the sparse kind. When it happened, setting the volume's volmode again (zfs set volmode=geom and then back to dev) made the device file appear, and the guest started. We could not make it happen every time: later runs, with brand-new guests, started without any trick. In a later test on a fresh machine it came back (the first guest after vm init had no device file, and vm install stopped at once), and the same zfs set volmode trick fixed it. So treat it as a quirk to know about, not as something you will certainly meet. If a guest fails with exited with status 4 and a "backing file" message, check for the device file. A way around it is a non-sparse volume: change disk0_dev to zvol in the guest's file (from the documentation, not run here), at the cost of reserving the whole 4 GB.
Step 5: install the guest
root@practice:~ # vm install guest1 FreeBSD-15.0-RELEASE-dvd1.iso Starting guest1 * found guest in /zroot/vm/guest1 * booting... root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - No Bootloader (11541) root@practice:~ # vm console guest1 [... the guest boots from the ISO: the same boot messages as on the bhyve page ...] Console type [vt100]: [... in the guest console: the installer shell, the unattended settings typed in with a heredoc, then bsdinstall script (it took 68 seconds in this nested setup), and halt -p ...] root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - No Stopped
- vm install guest1 FreeBSD-15.0-RELEASE-dvd1.iso starts the guest with the install disc attached. vm list shows the state Bootloader while the kernel loads.
- vm console guest1 connects your terminal to the guest's serial console (through a null-modem device that vm-bhyve set up, /dev/nmdm-guest1.1B, which you can see in vm info below; it uses the cu program: from the documentation, not run here). This is where the installer lives. You leave the console with ~ followed by Ctrl-D (from the documentation, not run here).
- We chose the installer's Shell and typed an unattended install script, as we did on the FreeBSD overview page for the lab machine: ZFS on vtbd0, no swap (the disk is only 4 GB), the host name guest1, a fixed address 10.88.0.2/24 with the host as the gateway, and the SSH server on. The installation took 68 seconds (this nested lab is slow; real hardware does it in much less). Then halt -p powered the guest off, and vm list shows Stopped.
The ordinary way is to click through the installer's screens on the same console. We only scripted it so that the page could be tested from start to finish.
Step 6: boot it from its own disk
root@practice:~ # vm start guest1 Starting guest1 * found guest in /zroot/vm/guest1 * booting... root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - No Bootloader (12175) root@practice:~ # vm switch list NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS private standard vm-private 10.88.0.1/24 no - - - root@practice:~ # ping -c 2 -t 3 10.88.0.2 PING 10.88.0.2 (10.88.0.2): 56 data bytes 64 bytes from 10.88.0.2: icmp_seq=0 ttl=64 time=2.722 ms 64 bytes from 10.88.0.2: icmp_seq=1 ttl=64 time=0.987 ms --- 10.88.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.987/1.854/2.722/0.867 ms root@practice:~ # ssh root@10.88.0.2 'hostname; uname -srm; zpool list -H -o name,size,alloc; df -h / | tail -1' (root@10.88.0.2) Password for root@guest1: guest1 FreeBSD 15.0-RELEASE amd64 zroot 3.75G 694M zroot/ROOT/default 3.6G 686M 2.9G 19% / root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - No Running (12247) root@practice:~ # vm switch list NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS private standard vm-private 10.88.0.1/24 no - - -
vm start guest1 boots the guest from its disk, without the install disc (the disc was only attached by vm install). The state goes through Bootloader to Running. About a minute later it answers ping on 10.88.0.2 in a millisecond or two, and we can log in with SSH from the host: the guest says it is guest1, FreeBSD 15.0-RELEASE amd64, with its own ZFS pool, zroot, of 3.75 GB with 694 MB used, and a root file system that is 19 percent full. (SSH asked for a password this time. For real guests you will use keys.) That is a complete, separate FreeBSD machine, created, installed and booted with a handful of short commands.
Looking at a guest
root@practice:~ # vm info guest1 | head -24
------------------------
Virtual Machine: guest1
------------------------
state: running (12247)
datastore: default
loader: bhyveload
uuid: 89e8fb3f-c190-11f1-ae02-589cfc10bd50
cpu: 1
memory: 768M
memory-resident: 187420672 (178.738M)
console-ports
com1: /dev/nmdm-guest1.1B
network-interface
number: 0
emulation: virtio-net
virtual-switch: private
fixed-mac-address: 58:9c:fc:00:4a:aa
fixed-device: -
active-device: tap0
desc: vmnet/guest1/0/private
mtu: 1500
bridge: vm-private
vm info guest1 prints everything about a running guest: its state and process number, the CPU and memory, memory-resident: how much of its 768 MB it is actually using on the host (178 MB), the console device, and the network card: virtio-net, plugged into the switch private through the bridge vm-private, using the tap device tap0. The other everyday commands:
- vm list: all guests and their states.
- vm stop NAME: a graceful shutdown (it sends the guest an ACPI "power button" press, and the guest shuts itself down). vm poweroff NAME: the hard way, like pulling the plug.
- vm console NAME, vm configure NAME, vm destroy NAME.
- vm restart and vm reset: from the documentation, not run here.
Snapshots and clones, the ZFS way
root@practice:~ # vm snapshot guest1@clean /usr/local/sbin/vm: WARNING: guest1 appears to be running on practice (locked) /usr/local/sbin/vm: ERROR: guest1 must be powered off first (use -f to override) root@practice:~ # vm stop guest1 Sending ACPI shutdown to guest1 root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - No Stopped
We tried to take a snapshot of the running guest, and vm refused: guest1 must be powered off first (use -f to override). A snapshot of a running machine's disk is like a snapshot of a computer whose power was pulled: it will usually be fine, but the manager makes you decide on purpose with -f. The clean way is to stop the guest first. vm stop sent the shutdown request, and after a few seconds the state was Stopped.
root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - Yes [1] Stopped root@practice:~ # vm snapshot guest1@clean root@practice:~ # zfs list -r -t snapshot -o name,used zroot/vm/guest1 NAME USED zroot/vm/guest1@clean 0B zroot/vm/guest1/disk0@clean 0B root@practice:~ # vm clone guest1@clean guest2 root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - Yes [1] Stopped guest2 default bhyveload 1 768M - No Stopped root@practice:~ # zfs list -r -o name,used,refer,origin zroot/vm/guest2 NAME USED REFER ORIGIN zroot/vm/guest2 8K 112K zroot/vm/guest1@clean zroot/vm/guest2/disk0 8K 481M zroot/vm/guest1/disk0@clean
- vm snapshot guest1@clean took ZFS snapshots of the guest's dataset and of its disk, in an instant, using no space.
- vm clone guest1@clean guest2 made a second virtual machine, guest2, from the snapshot. Look at zfs list: guest2 takes 8K, and its disk refers to 481M of data, all of it shared with guest1 (the ORIGIN column shows the snapshot it came from). A complete copy of the machine in a moment, at no cost in space.
root@practice:~ # vm start guest2 Starting guest2 * found guest in /zroot/vm/guest2 * booting... root@practice:~ # ping -c 2 -t 3 10.88.0.2 PING 10.88.0.2 (10.88.0.2): 56 data bytes 64 bytes from 10.88.0.2: icmp_seq=0 ttl=64 time=2.037 ms 64 bytes from 10.88.0.2: icmp_seq=1 ttl=64 time=0.894 ms --- 10.88.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.894/1.465/2.037/0.571 ms root@practice:~ # ssh root@10.88.0.2 'hostname; uname -sr' (root@10.88.0.2) Password for root@guest1: guest1 FreeBSD 15.0-RELEASE root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - Yes [1] Stopped guest2 default bhyveload 1 768M - No Running (14283) root@practice:~ # zfs list -r -o name,used,refer zroot/vm/guest1 zroot/vm/guest2 NAME USED REFER zroot/vm/guest1 481M 112K zroot/vm/guest1/disk0 481M 481M zroot/vm/guest2 2.65M 120K zroot/vm/guest2/disk0 2.56M 482M
We started guest2, and it booted like any other guest. Notice what a clone inherits: it has the same host name (guest1, as the SSH login shows) and the same fixed address 10.88.0.2. Never run a clone and its original at the same time without changing those, or you will have two machines with the same IP address on one network. (We started only the clone: the original was stopped.) The sizes in the last zfs list: the original disk 481M, the clone's disk 2.56M: only what has changed since the snapshot. Clones are perfect for testing a risky change on a copy of a real machine.
root@practice:~ # vm stop guest2 Sending ACPI shutdown to guest2 root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - Yes [1] Stopped guest2 default bhyveload 1 768M - No Stopped root@practice:~ # vm destroy -f guest2 root@practice:~ # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE guest1 default bhyveload 1 768M - Yes [1] Stopped root@practice:~ # zfs list -r -o name,used zroot/vm NAME USED zroot/vm 4.06G zroot/vm/guest1 481M zroot/vm/guest1/disk0 481M
vm stop guest2 and vm destroy -f guest2 removed the clone completely, and zfs list shows only guest1 left.
Autostart
root@practice:~ # sysrc vm_list="guest1" vm_list: -> guest1 root@practice:~ # sysrc vm_delay=5 vm_delay: -> 5 root@practice:~ # grep "^vm_" /etc/rc.conf vm_enable="YES" vm_dir="zfs:zroot/vm" vm_list="guest1" vm_delay="5"
vm_list="guest1" in /etc/rc.conf makes the guests in the list start when the host boots, and vm_delay="5" waits five seconds between them. (After setting it, vm list shows Yes [1] in the AUTO column: guest number 1 in the start order. You can see it in the listing before the snapshot.) The guests are also shut down gracefully when the host shuts down (from the documentation, not run here).
What we did not do
All from the documentation, not run here:
- Linux and Windows guests: templates linux, linux-grub and windows. They boot with UEFI firmware (the bhyve-firmware package) or the grub2-bhyve loader, instead of bhyveload.
- A graphical console over VNC: one line in the guest's file (graphics="yes"), and the VNC column in vm list shows the port.
- Cloud images: vm img downloads and uses ready-made images, and vm create -i creates a guest from one.
- Other switch types: switches with a VLAN, switches joined to a real network card, and a NAT switch (vm switch nat) which gives guests internet access through the host.
- Limits and moving guests: CPU and memory limits with rctl, and migrating a guest to another host (vm migrate).
Which tool for which job
- Plain bhyve: to learn, and for one-off tests.
- vm-bhyve: a good fit for a handful of guests on a server you run by hand, because it is small, it is just text files and ZFS, and nothing is hidden from you.
- CBSD and similar: heavier managers for bigger setups (from the documentation, not run here).
- Proxmox/KVM (see the Proxmox pages): a web interface, clusters and live migration, on Linux.
⚠️ A clone is a twin. It has the same host name, the same fixed IP address and the same SSH keys as the original. Change them before you put a clone on a network next to the original.
Too many clones,<br>
and nobody knew which one was the original.
🧪 Try it yourself
On a FreeBSD machine with ZFS and CPU virtualization (a nested VM works, slowly):
1. Install vm-bhyve, set vm_enable and vm_dir in /etc/rc.conf, run vm init and copy the example templates. 2. Make a switch with an address, and (if you use pf) let pf skip its bridge. 3. Import an install ISO with vm iso. 4. Create a guest from the freebsd-zvol template, set its switch, and read its settings file. 5. Install it with vm install and vm console (use the installer's screens, or an unattended script). 6. Start it from its disk, ping it and log in with SSH. 7. Stop it, take a snapshot, clone it, start the clone and look at the sizes with zfs list. Whatever you do, check the clone's host name and address. 8. Destroy the clone, and set up the original for autostart.
The cheat sheet
- Setup: pkg install vm-bhyve, sysrc vm_enable=YES, sysrc vm_dir="zfs:zroot/vm", vm init, templates from /usr/local/share/examples/vm-bhyve.
- Network: vm switch create -a 10.88.0.1/24 NAME, vm switch list. Let pf skip the bridge vm-NAME.
- Media: vm iso FILE-or-URL, vm iso.
- Guests: vm create -t TEMPLATE -s 4G -c 1 -m 768M NAME, vm list, vm info NAME, vm configure NAME.
- Install: vm install NAME ISO, vm console NAME (leave with ~ then Ctrl-D).
- Run: vm start NAME, vm stop NAME (graceful), vm poweroff NAME (hard), vm destroy NAME.
- ZFS tricks: vm snapshot NAME@SNAP (power off first), vm clone NAME@SNAP NEW.
- Autostart: sysrc vm_list="NAME ...", sysrc vm_delay=5.
- If it will not start: tail /zroot/vm/NAME/vm-bhyve.log, and check the disk's device file in /dev/zvol.
← poudriere: Build Your Own Packages, Properly · ↑ FreeBSD Corner · ipfw: The Other FreeBSD Firewall →