Recover a Deleted File with /proc (while a process still has it open)
ls -l /proc/PID/fd | grep deleted
You deleted a file you needed. If a running process still has that file open, Linux has NOT erased the data yet, and you can
copy it straight back out through the /proc directory. The classic example is a log file: you run "rm /var/log/syslog" while the
logging service (rsyslog) is running. The name disappears, but rsyslog keeps writing into the same file, and you can rescue
everything it holds. This guide explains why that works, gives you the exact steps for the syslog case, and covers what to do
when it does NOT work. Read the whole "Read this first" section before you touch anything.
Read this first: what you can and cannot recover
- This works ONLY if some running process still has the file open. That is what keeps the data alive
- "As long as the server has not restarted" is a shorthand for that rule. A reboot kills every process, and restarting or
stopping the service that holds the file has the same effect. The moment the last process closes the file, the data is gone
- So: do NOT restart, stop, reload or kill the service that holds the file until you have copied the data out
- If NO process has the file open, /proc cannot help. You need backups, snapshots, or a filesystem-level undelete tool (see the
last sections), and every write to the disk makes those harder. Stop writing to that filesystem
- This is a Linux technique (it needs the /proc filesystem). It works on ext4, XFS, btrfs and most others, because the data is
never touched by the delete. It does not depend on the filesystem's undelete support
- You copy what the file holds NOW. Data that was already overwritten or truncated before you started is not recoverable this way
- Work on a copy: send the rescued data to a DIFFERENT filesystem or folder with free space. Never write it back over the only good copy
Why it works (the short version)
On Linux a file name is just a link to the file's data (an inode). "rm" removes the LINK. The data is freed only when both are true:
1. No links (names) point to it any more, and
2. No process has it open.
When you delete a file that a process has open, condition 1 is met but condition 2 is not, so the data stays on disk and the process
keeps reading and writing it happily. Space is not freed either, which is why "df" can show a full disk while "du" cannot find the files.
The kernel exposes every open file of every process under /proc. For process number PID, the folder /proc/PID/fd/ holds one entry per
open file (a "file descriptor", numbered 0, 1, 2, 3 ...). Each entry is a special link you can open and read like the original file,
EVEN AFTER the name is gone. ls shows such a file as: 4 -> /var/log/syslog (deleted)
Quick recovery in five steps (the syslog example)
Scenario: someone ran sudo rm /var/log/syslog on an Ubuntu or Debian server. rsyslogd is still running and still has it open.
1. Find the process that holds the deleted file
sudo lsof +L1 | grep syslog
or without lsof:
sudo ls -l /proc/$(pidof rsyslogd)/fd | grep deleted
Example output:
l-wx------ 1 root root 64 Sep 26 10:15 7 -> /var/log/syslog (deleted)
Note two numbers: the PID (from pidof or lsof) and the file descriptor (here: 7)
2. Look at it before you copy (optional but reassuring)
sudo ls -lL /proc/PID/fd/7 # -L follows the link, so you see the real file size
sudo head -n 3 /proc/PID/fd/7 # the first lines
sudo tail -n 3 /proc/PID/fd/7 # the newest lines
(Some files are opened write-only by the program. Reading them through /proc still works for root, because it opens the file again from scratch.)
3. Copy it somewhere SAFE (a different folder, ideally a different disk or filesystem)
sudo mkdir -p /root/recovered
sudo cp /proc/PID/fd/7 /root/recovered/syslog.recovered
Check that it is complete and readable:
sudo ls -lh /root/recovered/syslog.recovered
sudo tail -n 5 /root/recovered/syslog.recovered
Do not restart anything yet. The data is safe now, but the process still has the good copy too
4. Put the file back where it belongs, with the correct owner and permissions
sudo cp /root/recovered/syslog.recovered /var/log/syslog
sudo chown syslog:adm /var/log/syslog
sudo chmod 640 /var/log/syslog
Catch up on lines written while you were working (this appends only the bytes added after your first copy):
size=$(sudo stat -c %s /var/log/syslog)
sudo tail -c +$((size + 1)) /proc/PID/fd/7 | sudo tee -a /var/log/syslog >/dev/null
5. Make the service start using the NEW file
The process is still writing to the deleted copy. Tell it to reopen its log file. For rsyslog this is a HUP signal:
sudo systemctl kill -s HUP rsyslog.service
(or: sudo kill -HUP $(pidof rsyslogd) or sudo systemctl restart rsyslog because the new file is in place)
Check that new lines appear in the recovered file:
logger "recovery test"; sudo tail -n 3 /var/log/syslog
Then confirm no deleted files remain: sudo lsof +L1 | grep syslog
On RHEL, Fedora and other SELinux systems also restore the security label: sudo restorecon -v /var/log/messages
That is the whole idea. The rest of this guide is the detail, the tools, the other cases, and what to do when it does not work.
A safe practice lab (try it before you need it)
Do this on a test machine so the steps are familiar when it matters.
1. Make a file and keep it open with a running process:
echo "my important data" > /tmp/demo.txt
tail -f /tmp/demo.txt & # a background process that holds the file open
PID=$!
2. Delete the file:
rm /tmp/demo.txt
cat /tmp/demo.txt # cat: /tmp/demo.txt: No such file or directory
3. Look for it in /proc:
ls -l /proc/$PID/fd # look for: 3 -> /tmp/demo.txt (deleted)
4. Copy it out, using the descriptor number you saw (3 here):
cp /proc/$PID/fd/3 /tmp/demo-recovered.txt
cat /tmp/demo-recovered.txt # my important data
5. Clean up:
kill $PID; rm /tmp/demo-recovered.txt
Run the same steps while ALSO appending to the file from another terminal (echo more >> /tmp/demo.txt before you delete it) to see that everything written is kept.
Finding deleted-but-open files: the toolbox
lsof +L1 # every open file with a link count below 1, meaning deleted but still open (needs sudo to see all processes)
sudo lsof +L1 | grep -i syslog # narrow it down
sudo lsof | grep '(deleted)' # older style: slow on busy servers, but simple
sudo lsof -nP -a -p PID +L1 # deleted files held by one process
sudo find /proc/*/fd -ls 2>/dev/null | grep '(deleted)' # no lsof needed
sudo ls -l /proc/*/fd 2>/dev/null | grep '(deleted)' # same idea with ls (shows the proc/PID/fd path above each group)
sudo fuser -v /var/log/syslog # who has it open? (fails for deleted names: the name no longer exists)
sudo ls -l /proc/PID/fd # everything one process has open
sudo readlink /proc/PID/fd/7 # where does descriptor 7 point?
sudo cat /proc/PID/fdinfo/7 # details: pos (current position), flags (read/write, append), mnt_id, and inode
pidof rsyslogd pgrep -a rsyslog # find the PID by name systemctl show -p MainPID rsyslog # the main PID of a systemd service
ps -o pid,user,cmd -p PID # who is it, and what is it?
What the output means:
"7 -> /var/log/syslog (deleted)" The name is gone. The (deleted) suffix is added by the kernel
File type letters in lsof (TYPE=REG) A regular file. NLINK 0 means no names remain
FD column The number (7) and the mode: r = read, w = write, u = read and write
SIZE/OFF Size on disk (or your current offset). Compare with ls -lL /proc/PID/fd/N for the real size
You need root (sudo) to look inside processes owned by other users. The copy is created by YOU, so it is owned by the user who ran cp.
Recovering a file that is still being written to (logs, databases, downloads)
- Logs: the copy is a snapshot of what exists now. Copy again later to get more. The "catch-up" trick in step 4 appends only the new bytes
- Every open of /proc/PID/fd/N starts a fresh read from the beginning, independent of the process's own position
- Very large logs: check free space first (df -h /root/recovered) because the copy needs as much room as the file (ls -lL /proc/PID/fd/N shows the size)
- Use a different filesystem if you can, so you do not fill the disk holding the good copy
- Compress while copying to save space: sudo cat /proc/PID/fd/7 | gzip > /root/recovered/syslog.gz
- Verify a copy: compare sizes and checksums (the file may keep growing, so compare a fixed number of bytes):
sudo head -c 1000000 /proc/PID/fd/7 | sha256sum sudo head -c 1000000 /root/recovered/syslog.recovered | sha256sum
- Databases and other files that programs change constantly: a live copy can be inconsistent. Use the database's own backup tool, or stop writes in a controlled way,
AFTER you have secured a copy
Other things you can recover the same way
A deleted program that is still running (a binary you removed or replaced during an upgrade):
sudo ls -l /proc/PID/exe # shows "... -> /usr/bin/myapp (deleted)"
sudo cp /proc/PID/exe /root/recovered/myapp # copy the executable back out
sudo chmod +x /root/recovered/myapp
Deleted shared libraries and files that a program has memory-mapped:
sudo ls -l /proc/PID/map_files/ # root only. Each entry is a mapped file, including deleted ones
sudo cp /proc/PID/map_files/7f12a0000000-7f12a0021000 /root/recovered/libfoo.so # use the address range you see
sudo grep deleted /proc/PID/maps # a quick list of deleted mapped files
A file that a program opened and then unlinked on purpose (temporary files): often gone by design, but the data is available while the process runs
A deleted directory: the directory itself cannot be recovered this way, but any FILE inside it that a process holds open can be
The current directory of a process: sudo ls -l /proc/PID/cwd (if the folder was deleted, it shows "(deleted)")
The process's environment and command line (helpful for finding what to restart): sudo tr '\0' '\n' < /proc/PID/environ sudo tr '\0' ' ' < /proc/PID/cmdline; echo
Bonus: the "disk is full but I deleted the big log" problem
The same effect that lets you recover a file also causes this classic mystery: you delete a huge log to free space, but "df" still shows the disk full,
because a process still has the file open. Find it:
sudo lsof +L1 # look for large SIZE values
Free the space WITHOUT restarting the service by truncating through /proc (this destroys the data, so make sure you do not need it):
sudo truncate -s 0 /proc/PID/fd/7 # or: : | sudo tee /proc/PID/fd/7 >/dev/null or: sudo sh -c ': > /proc/PID/fd/7'
Or restart or reload the service (HUP, or the program's own "reopen logs" signal), which closes the deleted file and frees the space.
Better: never delete a live log with rm. Empty it in place or use logrotate: sudo truncate -s 0 /var/log/syslog
Then the name, owner and permissions stay, and the service keeps writing to the same file.
Getting the service to reopen its logs (per service)
rsyslog sudo systemctl kill -s HUP rsyslog.service (or: sudo kill -HUP $(pidof rsyslogd))
nginx sudo nginx -s reopen or sudo kill -USR1 $(cat /run/nginx.pid)
Apache sudo systemctl reload apache2 (httpd) a graceful reload reopens the logs
MySQL/MariaDB mysqladmin flush-logs or FLUSH LOGS;
PostgreSQL SELECT pg_rotate_logfile(); or sudo systemctl reload postgresql
sshd, cron Log through syslog/journald, so reopening the syslog file is enough
systemd-journald Does not use /var/log/syslog. Its files are under /var/log/journal/. Check them with journalctl --disk-usage and journalctl --verify
If a program has no reopen signal, restarting it is the only way to switch to the new file. Do that only AFTER the copy is safe.
If no process has the file open (the harder cases)
/proc cannot help here. The data is unlinked and unreferenced, so the filesystem may reuse its space at any moment. Act fast, and stop the damage:
1. STOP WRITING to that filesystem immediately. Stop services that write to it, and avoid installing or saving anything on it
2. Check your safety nets FIRST. They are far more reliable than undelete tools:
Backups (rsync, restic, borg, Bacula, cloud backups) restore from the latest one
Snapshots: LVM (lvs), ZFS (zfs list -t snapshot; look in .zfs/snapshot), btrfs (btrfs subvolume list), cloud disk snapshots, VM snapshots
Version control: git checkout / git restore, git reflog
A desktop trash folder: ~/.local/share/Trash/files/
Logs and other data can often be regenerated (syslog entries may also live on a central log server or in the journal: journalctl)
3. If the file matters and there is no backup, take a full disk image of the partition and work on the IMAGE. Better still, unmount the filesystem or remount it read-only:
sudo mount -o remount,ro /mountpoint (fails if files are open for writing. Unmounting or shutting down the machine may be needed)
sudo dd if=/dev/sdXN of=/mnt/external/disk.img bs=4M status=progress # /mnt/external must be a DIFFERENT disk with enough space
(ddrescue is better for a failing disk)
4. Try an undelete tool on the image or on the unmounted partition (never install the tool onto the affected disk):
ext3/ext4 extundelete (works on ext3, unreliable on ext4 with journaling), ext4magic, debugfs, and photorec (carves files by their content)
Any type photorec and testdisk (from the testdisk package): they recover many file types by signature, but names and folders are usually lost
XFS There is no dependable undelete: rely on backups, or xfs_db for experts
btrfs btrfs restore (reads data from an unmountable filesystem), snapshots
NTFS/FAT ntfsundelete, photorec, testdisk
Success is never guaranteed, and SSDs with TRIM often erase the data almost immediately.
5. For an important cloud server: snapshot the volume NOW, then experiment on a copy of the snapshot
6. If the data is critical, stop and call a data recovery specialist. Each extra write reduces the chance of getting it back
Prevention: so this never ruins your day
- Never "rm" a live log. Use truncate -s 0 file or logrotate instead, and let services reopen their logs
- Take backups you have tested, and use snapshots (LVM, ZFS, btrfs, cloud) before risky changes
- Safer deleting: alias rm='rm -i' or the trash-cli package: trash-put file (undo with trash-restore)
- Use rm -I (prompts once for bulk deletes) and think before using wildcards. Run ls pattern first, then rm pattern
- Type the path, not a glob, when deleting in /var/log, /etc or a database folder
- Keep important config in git (etckeeper tracks /etc automatically)
- Mount user data with snapshots and off-server backups. Ship logs to a central server (rsyslog forwarding, journald remote, Loki, ELK)
- Use chattr +i file on a few critical files to make them undeletable (remove with chattr -i). Do not use it on logs that must change
- Rehearse this recovery in a lab (see the practice section) so the steps are familiar under pressure
- Restrict who has sudo, and review dangerous commands before running them (see the AI-Assisted Shell Scripting guide)
Common problems and questions
"No such file or directory" when copying /proc/PID/fd/N The process closed it, exited, or you have the wrong PID or descriptor number. Run the ls again
The (deleted) entry is gone from the listing The process restarted or closed the file. The data is now gone from this route
"Permission denied" on /proc/PID/fd Use sudo. You can only inspect your own processes without root. Some hardened systems restrict /proc (hidepid mount option)
The copy is 0 bytes The file was empty, or was truncated before you copied. Check ls -lL /proc/PID/fd/N and lsof +L1 for its size first
The copy is smaller than I expected The program truncated or rotated it earlier, or the data was never written because of buffering. Some programs keep parts in memory
Nothing shows up in lsof +L1 No process holds it. See "If no process has the file open"
I see many PIDs holding the file Pick the one that writes it (mode w or u). Any of them gives the same data when they share the same open file
The service restarted itself (systemd Restart=on-failure) Copy first, before anything else. If it already restarted, the deleted file is gone
The file is on a network share (NFS) Behavior differs. NFS uses "silly rename" files named .nfsXXXX in the same folder. Look for them: ls -la /path/.nfs*
Container (Docker or Kubernetes) Find the process from the HOST. The path is under the host's /proc/HOST_PID/root/... Use docker top or ps to get the host PID
Copied the file back but the service still writes elsewhere It holds the OLD deleted file open. Send the reopen signal (HUP or the app's own) or restart the service
Wrong owner or permissions after restoring Fix with chown and chmod (see step 4), and restorecon on SELinux systems
I already restarted the service Then the deleted data is gone from memory. Go to the backups, snapshots and undelete sections
The disk is still full after deleting A process holds the deleted file. Run lsof +L1, then truncate through /proc or restart the service (see the Bonus section)
A directory disappeared, not a file /proc only rescues open files. Restore the directory from backup or snapshot, then rescue any open files inside it as above
Is this legal or safe to do? Yes, on a system you administer. It only reads data the OS already holds. As always, handle recovered files with the same care as the original (they may contain private data)
Command summary
lsof +L1 Find deleted files that are still open
ls -l /proc/PID/fd | grep deleted Find them through /proc
cp /proc/PID/fd/N /safe/place/file Recover the file (copy it out)
cat /proc/PID/fdinfo/N Position, flags and details of an open file
ls -l /proc/PID/exe Recover a deleted running program: cp /proc/PID/exe /safe/place/prog
truncate -s 0 /proc/PID/fd/N Free the space held by a deleted file, destroying its data
systemctl kill -s HUP rsyslog.service Tell rsyslog to reopen its log file
tail -c +$((size + 1)) /proc/PID/fd/N Copy only the bytes after the first "size" bytes (catch-up)
Tips
The rule: /proc can save a deleted file only while some running process still has it open. Do not restart or stop that process until the data is copied.
Copy first, tidy later: get the data onto a different disk or folder, verify it, and only then restore names, owners and permissions.
Use lsof +L1 to find deleted-but-open files. It is also the answer to "disk full but nothing big in du".
After restoring a log, tell the service to reopen its log file (HUP or its reload command), or it keeps writing to the deleted copy.
Never rm a live log file. Truncate it in place or use logrotate.
If no process has the file open, stop writing to the disk, look for backups and snapshots, and only then try an undelete tool on an image.
Make backups and snapshots before you need them, and practice this recovery once in a lab.
Handle recovered data as carefully as the original. Logs and files can contain passwords and personal information.
