📁 File Sharing with NFS: One Server, Many Machines

Share a ZFS dataset from one FreeBSD machine and mount it on another, with the user-ID surprise, the access-list trap that let the wrong network in, what a client does when the server disappears, and a mount that survives a reboot

Video: File Sharing with NFS (FreeBSD video)
Prefer to watch? This short video runs this page's main commands on a real FreeBSD 15.0 machine, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.

What NFS is

NFS (Network File System) lets one machine export a directory, and other machines mount it as if it were a local disk. The client sees ordinary files. The server keeps the real data (and its snapshots, its backups, its quotas). It is the oldest and simplest way to share files between Unix machines, and it is in the base system: nothing to install.

We use the same two real FreeBSD 15.0 virtual machines as on the backups and WireGuard pages. prod (192.168.99.1) is the server and backup (192.168.99.2) is the client. We use NFS version 4, the modern one: it needs one TCP port (2049) instead of the several that NFS version 3 uses, which is much easier to put behind a firewall.

First, what is there at the start: nothing is enabled and nothing listens, but the programs are installed.

root@prod:~ # sysrc -a | grep -E "nfs|rpcbind|mountd"; cat /etc/exports 2>&1; service -e | grep -E "nfs|mountd|rpcbind"; zfs list -o name,mountpoint,sharenfs zroot/data
cat: /etc/exports: No such file or directory
NAME        MOUNTPOINT   SHARENFS
zroot/data  /zroot/data  off
root@prod:~ # ls /usr/bin/showmount /usr/sbin/mountd /usr/sbin/nfsd /sbin/mount_nfs; sockstat -4l | grep -E ":(111|2049)" ; echo none-listening
/sbin/mount_nfs		/usr/bin/showmount	/usr/sbin/mountd	/usr/sbin/nfsd
none-listening

(The first command prints nothing for nfs, mountd or rpcbind: none is switched on. There is no /etc/exports. The second command lists the programs: nfsd, mountd, showmount and mount_nfs.)

The data to share

On the server, a dataset of its own for the share (a ZFS dataset is the right unit: you can snapshot it, give it a quota and share it on its own), and two users who will use it:

root@prod:~ # zfs create zroot/share
root@prod:~ # pw useradd alice -u 1001 -m -s /bin/sh; pw useradd bob -u 1002 -m -s /bin/sh; id alice; id bob
pw: uid `1001' has already been allocated
id: alice: no such user
uid=1002(bob) gid=1002(bob) groups=1002(bob)
root@prod:~ # mkdir /zroot/share/team; chown alice:wheel /zroot/share/team; echo "Team notes, written on the server" > /zroot/share/team/notes.txt; chown alice /zroot/share/team/notes.txt; ls -l /zroot/share/team
chown: alice: illegal user name
chown: alice: illegal user name
total 1
-rw-r--r--  1 root wheel 34 Oct  6 19:06 notes.txt

The output has a mishap worth keeping. pw useradd alice -u 1001 said uid 1001 has already been allocated (it belongs to the bsd user who logs in to this machine), so alice was not created, id alice found nothing, and the two chown commands failed with "illegal user name". The files were left owned by root. This matters for NFS: users are numbers, and you choose them on purpose. We picked 1101 for alice:

root@prod:~ # pw useradd alice -u 1101 -m -s /bin/sh; id alice
uid=1101(alice) gid=1101(alice) groups=1101(alice)
root@prod:~ # chown alice:wheel /zroot/share/team; chown alice /zroot/share/team/notes.txt; ls -ln /zroot/share/team; ls -ld /zroot/share/team
total 1
-rw-r--r--  1 1101 0 34 Oct  6 19:06 notes.txt
drwxr-xr-x  2 alice wheel 3 Oct  6 19:06 /zroot/share/team

ls -ln shows the numbers (1101 and group 0), and ls -ld shows the directory with the names. NFS will send the numbers across the network, and that is the first thing to understand.

The server: three switches and a file

root@prod:~ # sysrc nfs_server_enable=YES nfsv4_server_enable=YES nfsuserd_enable=YES mountd_enable=YES
nfs_server_enable: NO -> YES
nfsv4_server_enable: NO -> YES
nfsuserd_enable: NO -> YES
mountd_enable: NO -> YES
root@prod:~ # printf 'V4: /zroot\n/zroot/share -network 192.168.99.0/24\n' > /etc/exports; cat /etc/exports
V4: /zroot
/zroot/share -network 192.168.99.0/24
root@prod:~ # service nfsd start 2>&1; service mountd status; service nfsuserd status; service nfsd status
NFS on reserved port only=YES
Starting rpcbind.
Starting mountd.
Starting nfsd.
mountd is running as pid 67203.
nfsuserd is not running.
nfsd is running as pid 67430 67643.
root@prod:~ # sockstat -4l | grep -E ":(111|2049)"; showmount -e localhost
root       nfsd       67430  7 tcp4  *:2049                *:*                  
root       rpcbind    57841 11 udp4  *:111                 *:*                  
root       rpcbind    57841 13 tcp4  *:111                 *:*                  
??         ??            ?? ?? udp4  *:2049                *:*                  
Exports list on localhost:
/zroot/share                       192.168.99.0

The client: the first mount

root@backup:~ # mkdir -p /mnt/share; mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share; echo exit=$?
exit=0
root@backup:~ # mount | grep share; ls -l /mnt/share/team; cat /mnt/share/team/notes.txt
192.168.99.1:/share on /mnt/share (nfs, nfsv4acls, named attributes)
total 1
-rw-r--r--  1 root wheel 34 Oct  6 19:06 notes.txt
Team notes, written on the server

One command: mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share. Look at the path: /share, not /zroot/share. With version 4 the path starts at the V4: root, which we set to /zroot. The files of the server are there, readable. (The owner of notes.txt is root: because of the mishap above.)

Who owns what: users are numbers

With the owner fixed on the server (alice, 1101), here is what the client says. On the client there is no alice yet, and we start the name helper there too:

root@prod:~ # service nfsuserd start 2>&1; service nfsuserd status
Starting nfsuserd.
nfsuserd is running as pid 93746 93857 93976 94052 94283.
root@backup:~ # sysrc nfsuserd_enable=YES; service nfsuserd start 2>&1; mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share; ls -l /mnt/share/team
nfsuserd_enable: NO -> YES
Starting nfsuserd.
total 1
-rw-r--r--  1 nobody wheel 34 Oct  6 19:06 notes.txt
root@backup:~ # id alice 2>&1
id: alice: no such user

The file is owned by nobody: the client got the user from the server, does not know it, and shows nobody (that is the name mapping, the job of nfsuserd, on both sides). Now we make alice on the client with the same number, 1101, and write as alice:

root@backup:~ # pw useradd alice -u 1101 -m -s /bin/sh; ls -l /mnt/share/team
total 1
-rw-r--r--  1 nobody wheel 34 Oct  6 19:06 notes.txt
root@backup:~ # su alice -c "echo written by alice on the client >> /mnt/share/team/notes.txt; echo ok"
ok
root@prod:~ # cat /zroot/share/team/notes.txt; ls -l /zroot/share/team
Team notes, written on the server
written by alice on the client
total 1
-rw-r--r--  1 alice wheel 65 Oct  6 19:06 notes.txt
root@backup:~ # echo as root >> /mnt/share/team/rootfile 2>&1; echo exit=$?
-su: cannot create /mnt/share/team/rootfile: Permission denied
exit=2

And after a remount, the owner is shown correctly:

root@backup:~ # ls -l /mnt/share/team
total 1
-rw-r--r--  1 alice wheel 65 Oct  6 19:06 notes.txt
root@backup:~ # umount /mnt/share; mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share; ls -l /mnt/share/team
total 1
-rw-r--r--  1 alice wheel 65 Oct  6 19:06 notes.txt

The access-list trap

We want only the network 192.168.99.0/24 to use the share. We tested what the list really does, by changing it to a network that the client is not in (10.99.0.0/16) and mounting again:

root@prod:~ # printf 'V4: /zroot\n/zroot/share -network 10.99.0.0/16\n' > /etc/exports; service mountd reload; showmount -e localhost
Exports list on localhost:
/zroot/share                       10.99.0.0 
root@backup:~ # umount /mnt/share; mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share 2>&1; echo exit=$?
exit=0
root@prod:~ # printf 'V4: /zroot\n/zroot/share -network 192.168.99.0/24\n' > /etc/exports; service mountd reload; showmount -e localhost
Exports list on localhost:
/zroot/share                       192.168.99.0 
root@backup:~ # mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share 2>&1; echo exit=$?; ls /mnt/share
exit=0
team

Read the middle: the export list says 10.99.0.0, and the client at 192.168.99.2 mounted it anyway: exit=0. For NFS version 4, the list on the V4: line decides who may use version 4 at all, and it had no network, so everybody was let in. The restriction on the share line alone did not stop it. (We then put the right network back.) The correct way is to put the network on both lines:

root@prod:~ # printf 'V4: /zroot -network 10.99.0.0/16\n/zroot/share -network 10.99.0.0/16\n' > /etc/exports; service mountd reload; showmount -e localhost; cat /etc/exports
Exports list on localhost:
/zroot/share                       10.99.0.0 
V4: /zroot -network 10.99.0.0/16
/zroot/share -network 10.99.0.0/16
root@backup:~ # umount /mnt/share; mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share 2>&1; echo exit=$?
umount: /mnt/share: statfs: Permission denied
mount_nfs: nmount: /mnt/share: Permission denied
exit=1
root@prod:~ # printf 'V4: /zroot -network 192.168.99.0/24\n/zroot/share -network 192.168.99.0/24\n' > /etc/exports; service mountd reload; cat /etc/exports
V4: /zroot -network 192.168.99.0/24
/zroot/share -network 192.168.99.0/24
root@backup:~ # mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share 2>&1; echo exit=$?; ls /mnt/share
exit=0
team

Now with -network 10.99.0.0/16 on both lines, the mount from the wrong network gave Permission denied. (The first umount line says statfs: Permission denied too: it is the mount we already had, now refused to even answer.) And with the right network back on both lines, the mount works.

Lesson: after you write an exports file, test from a machine that should be refused. Do not trust that the list says what you meant.

Let ZFS do the exports

On a ZFS server there is a nicer way: a property on the dataset. The V4: line stays in /etc/exports, and the share itself moves to the dataset:

root@prod:~ # printf 'V4: /zroot -network 192.168.99.0/24\n' > /etc/exports; cat /etc/exports
V4: /zroot -network 192.168.99.0/24
root@prod:~ # zfs set sharenfs="-network 192.168.99.0/24" zroot/share; zfs get -H -o property,value sharenfs zroot/share
sharenfs	-network 192.168.99.0/24
root@prod:~ # cat /etc/zfs/exports; service mountd reload; showmount -e localhost
# !!! DO NOT EDIT THIS FILE MANUALLY !!!

/zroot/share	-network 192.168.99.0/24 
Exports list on localhost:
/zroot/share                       192.168.99.0 
root@backup:~ # umount /mnt/share; mount -t nfs -o nfsv4 192.168.99.1:/share /mnt/share 2>&1; echo exit=$?; ls /mnt/share/team
exit=0
notes.txt

What the mount looks like

root@backup:~ # nfsstat -m | cut -c1-175
192.168.99.1:/share on /mnt/share
nfsv4,minorversion=2,tcp,resvport,nconnect=1,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65
root@prod:~ # sysrc -a | grep -E "^(nfs|mountd|rpcbind)"; sockstat -4l | grep -E ":(111|2049)"
mountd_enable: YES
nfs_server_enable: YES
nfsuserd_enable: YES
nfsv4_server_enable: YES
root       nfsd       93335  7 tcp4  *:2049                *:*                  
root       rpcbind    90802 11 udp4  *:111                 *:*                  
root       rpcbind    90802 13 tcp4  *:111                 *:*                  
??         ??            ?? ?? udp4  *:2049                *:*                  
root@prod:~ # ls /var/db/mountdtab /var/db/mountd.lease 2>&1 | head -2; nfsstat -e -c 2>&1 | head -2; echo; showmount -a localhost 2>&1
ls: /var/db/mountd.lease: No such file or directory
/var/db/mountdtab
Client Info:
RPC Counts:

All mount points on localhost:

nfsstat -m shows the real options of the mount. The ones that matter: nfsv4, tcp, hard, sec=sys, and the attribute cache times (acregmin=5, acregmax=60): the client may show you a directory listing that is up to a minute old. That explains the old owner above, and why two clients can see each other's changes a little late.

The last commands show what is switched on: the four settings in rc.conf and the listening ports.

When the server goes away

hard is the default, and it means: if the server does not answer, the client waits, and keeps waiting. We took the server's nfsd down and started a write on the client in the background:

root@backup:~ # rm -f /root/w.done; (su alice -c "echo late write > /mnt/share/team/late.txt"; echo finished > /root/w.done) > /dev/null 2>&1 & echo ok
ok
root@backup:~ # sleep 2; cat /root/w.done; ls /mnt/share/team
finished
late.txt	notes.txt
[1]   Done                    (su alice -c echo late write > /mnt/share/team/late.txt; echo finished >/root/w.done)
root@prod:~ # service nfsd stop 2>&1 | tail -1; sleep 1
Waiting for PIDS: 84035.
root@backup:~ # rm -f /root/w.done; (su alice -c "echo second write > /mnt/share/team/second.txt"; echo finished > /root/w.done) > /dev/null 2>&1 & sleep 20; ls /root/w.done 2>&1; ps -axo pid,stat,wchan,command | grep "[s]econd write"
ls: /root/w.done: No such file or directory
55410    I wait     su alice -c echo second write > /mnt/share/team/second.txt
55540    D rpccon   su -c echo second write > /mnt/share/team/second.txt (sh)
root@backup:~ # dmesg | tail -2 | cut -c1-150
wg0: link state changed to UP
newnfs: server '192.168.99.1' error: fileid changed. fsid 70df524c:cdee01de: expected fileid 0x22, got 0x2. (BROKEN NFS SERVER OR MIDDLEWARE)
root@prod:~ # service nfsd start 2>&1 | tail -1
Starting nfsd.
root@backup:~ # sleep 20; cat /root/w.done; ls /mnt/share/team
finished
late.txt	notes.txt	second.txt
[1]   Done                    (su alice -c echo second write > /mnt/share/team/second.txt; echo finished >/root/w.done)
root@prod:~ # ls /zroot/share/team
late.txt	notes.txt	second.txt

The price: a server that is gone for good can freeze everything that touches the mount, including the shell of an administrator who types ls in the wrong directory. Two things help: do not leave your own working directory inside an NFS mount, and look at the options soft (the client gives up and returns an error: this risks half-written files, so only for read-only data) and intr-style forced unmounting (umount -f). Both from the documentation, not run here.

A mount that survives a reboot

On the client, a line in /etc/fstab and two settings:

root@backup:~ # umount /mnt/share; sysrc nfs_client_enable=YES
nfs_client_enable: NO -> YES
root@backup:~ # echo "192.168.99.1:/share /mnt/share nfs rw,nfsv4,late 0 0" >> /etc/fstab; tail -2 /etc/fstab
fdesc /dev/fd fdescfs rw 0 0
192.168.99.1:/share /mnt/share nfs rw,nfsv4,late 0 0
root@backup:~ # mount /mnt/share; mount | grep share; df -h /mnt/share
192.168.99.1:/share on /mnt/share (nfs, nfsv4acls, named attributes)
192.168.99.1:/share on /mnt/share (nfs, nfsv4acls, named attributes)
Filesystem             Size    Used   Avail Capacity  Mounted on
192.168.99.1:/share    5.3G     96K    5.3G     0%    /mnt/share

The real test, as always, is a reboot: both machines, server first:

root@prod:~ # uptime | cut -c1-30; service nfsd status; showmount -e localhost
 7:12PM  up 2 mins, 2 users, l
nfsd is running as pid 93335 93676.
Exports list on localhost:
/zroot/share                       192.168.99.0 
root@backup:~ # uptime | cut -c1-30; mount | grep share; ls /mnt/share/team; grep -E "nfs" /etc/rc.conf
 7:12PM  up 1 min, 1 user, loa
192.168.99.1:/share on /mnt/share (nfs, nfsv4acls, named attributes)
late.txt	notes.txt	second.txt
nfsuserd_enable="YES"
nfs_client_enable="YES"

The server has nfsd running and the export is there. The client came up one minute after, with the share mounted and the files visible, including the ones written by the experiments. Nothing was typed after the boot.

If the server is not up when the client boots, the client's boot waits for the mount and may be slow: for important machines, put the share on late and test what happens when the server is off.

Security: what we did and did not do

What we did not do

The cheat sheet

← WireGuard on FreeBSD: A Private Tunnel Between Two Machines · ↑ FreeBSD Corner · The FreeBSD Corner Cheat Sheet: Every Command in One Place →

© 2026 TechCommune. All rights reserved.
"NFS is a network cable for your file system. It works until someone unplugs it, and then it waits patiently."