The case

An Oracle 11.2.0.4 database on RHEL 7 (running inside a VM) had its datafiles deleted with rm -rf while the instance was still up. The customer wants to know whether recovery to the moment of the deletion is realistic.

Environment

  • Database: Oracle 11.2.0.4
  • OS: RHEL 7
  • Platform: Virtual machine

What was available

  • RMAN full backup taken on 14 Nov 2020
  • No other RMAN backups
  • Archived redo logs from 17 Nov 2020 through 5 Dec 2020 (the day of the deletion)
  • Missing archived redo logs for 15 Nov and 16 Nov 2020 — a two-day gap right after the only full backup

The question: can data be recovered up to the point of the rm -rf on 5 Dec 2020?

The single most important question first: is the instance still running?

On Linux, when a process has a file open and someone runs rm on it, the directory entry disappears but the inode and the underlying disk blocks are not freed as long as a process holds the file descriptor open. Oracle keeps every datafile open for the entire life of the instance via the database writer (dbw0), checkpoint (ckpt), and other background processes. That means there is a free, deterministic recovery path that beats every other option below — but only if the instance has not been restarted yet.

Path 0 — recover from /proc/<pid>/fd while Oracle is still up

Do not shut down the database. Do not bounce the host. From a root shell:

# Identify the database writer process
$ ps -ef | grep -i ora_dbw

# List its open file descriptors
$ ls -l /proc/<dbw_pid>/fd | grep -i .dbf

# Each deleted datafile shows up as:
# lrwx------ 1 oracle oracle 64 Dec  5 14:00 257 -> /u01/oradata/PROD/users01.dbf (deleted)

# Copy each open descriptor back to a new location
$ cp /proc/<dbw_pid>/fd/257 /backup/recovered/users01.dbf
$ cp /proc/<dbw_pid>/fd/258 /backup/recovered/system01.dbf
# ... repeat for every fd that points at a (deleted) .dbf, .ctl or redo file

Each cp pulls the live file out via the open inode. After every datafile, controlfile, and online redo log has been copied to safe storage, you can shutdown abort, move the recovered files into the original paths, and start the database normally. This restores the database to the exact moment of the deletion — no data loss, no archive log gap to worry about.

If even one critical background process has died or the instance has been restarted, this path is gone. The blocks become eligible for reuse the moment the last open descriptor closes, and on a busy filesystem they get overwritten quickly.

If the instance is already down — preserve the disk first

Stop work. The longer the OS runs after a rm, the more freed blocks get reallocated by other writes. Before running any tool:

  • Take a VM-level snapshot of the disk now. Snapshot first, investigate later.
  • If snapshots are not possible, make a raw block-level image of the volume holding the datafiles (dd to a different disk).
  • Stop everything that writes to the volume — log shippers, backup agents, cron jobs, monitoring agents.
  • Mount the volume read-only when running recovery scans.

Path 1 — block-level scan with PRMSCAN / DBRECOVER

If the instance is gone, the next-best path is to scan the raw volume for surviving Oracle data blocks and stitch them back into datafile images. PRMSCAN (the block-fragmentation scanner shipped with DBRECOVER for Oracle) walks the volume, identifies blocks by Oracle's on-disk header signature, groups them by file id and tablespace, and reassembles the original datafiles even when the filesystem metadata is gone.

Reference walkthrough: https://youtu.be/skH9nJOvIkQ

Notes:

  • Recovery quality depends on how much of the volume has been overwritten since the delete. Snapshot or image first.
  • The reassembled datafiles are usually good enough to load into DBRECOVER for Oracle in dictionary mode and export rows directly. They may not always be clean enough for Oracle itself to mount, especially if some blocks were partially reused.
  • UNDO and TEMP datafiles are not worth recovering — DBRECOVER does not need them.

Path 2 — restore from the 14 Nov RMAN backup

This is the "clean Oracle" path, with a known caveat: the missing archive logs for 15 and 16 Nov create a gap. Oracle cannot apply redo across a gap in the normal way; you can either accept losing 15–16 Nov changes and recover anyway, or accept that the database opens at the end of 14 Nov and lose everything after that.

2a. Restore + roll forward as far as possible

RMAN> STARTUP NOMOUNT;
RMAN> RESTORE CONTROLFILE FROM '/backup/14Nov/o1_mf_s_*.bkp';
RMAN> ALTER DATABASE MOUNT;
RMAN> RESTORE DATABASE;
RMAN> CATALOG START WITH '/archive/'; -- register 17 Nov - 5 Dec logs
RMAN> RECOVER DATABASE UNTIL SEQUENCE <first_log_after_14Nov> THREAD 1;
RMAN> ALTER DATABASE OPEN RESETLOGS;

This stops at the end of the available archive logs from 14 Nov before the gap. You then have a consistent open database, but the changes from 15 Nov onward are lost from the redo stream.

2b. Force-skip the gap, apply 17 Nov – 5 Dec

If the application can tolerate the side effects of a non-consecutive recovery, you can force RMAN to skip past the missing logs and apply the surviving 17 Nov – 5 Dec stream. The result is a logically inconsistent database (the SCN window between 14 Nov backup and 17 Nov first archive is unrecovered), and Oracle support will not bless it. It is suitable as a data extraction target: open the database long enough to expdp what you need, then throw it away.

This is what the original answer above offered as option 2 — useful only when the data after 17 Nov matters more than transactional consistency. We can drive the force-skip RMAN sequence remotely.

Path 3 — DBRECOVER for Oracle on the recovered datafile images

If Path 0 produced full-fidelity datafiles, you do not even need DBRECOVER — just put them back. If Path 1 produced reassembled datafiles that Oracle won't mount, or if you want to extract data without going through a force-open, load them into DBRECOVER for Oracle in dictionary mode (SYSTEM01.DBF first, then user tablespaces with their TS# / RFILE# from FILE$). DBRECOVER browses tables, previews rows, and exports to flat files or directly into a fresh Oracle instance via the data bridge — without needing controlfile, redo, or UNDO.

Download: DBRECOVER for Oracle (latest).

Recommended order of attempts

  1. Path 0 if the instance is still running. Free, fast, lossless. Stop reading and copy from /proc/<pid>/fd right now.
  2. Snapshot the disk.
  3. Path 1 (PRMSCAN) on the snapshot to recover datafile images, then load with DBRECOVER (Path 3) to extract rows.
  4. In parallel, prepare Path 2 (RMAN restore from 14 Nov + force-apply 17 Nov – 5 Dec logs) on a sandbox host. If Path 1 yields better results, ignore Path 2; if Path 1 misses critical tables, merge what survives from each.

Prevention for next time

  • Keep daily incremental + weekly full RMAN backups, never just one monthly full.
  • Ship archive logs to a second host or to object storage every few minutes — not to the same volume that held the datafiles.
  • Never give shared OS accounts rm rights on the oradata mountpoints. Use a wrapper or sudo policy that refuses recursive deletion under $ORACLE_BASE/oradata.
  • Set db_recovery_file_dest on a separate filesystem and verify the FRA is included in OS-level backups.
  • For VMs, schedule regular consistent VM snapshots — they are the single best safety net for accidental deletes.

How we can help

  1. Live triage. If the instance is still up, we can join via screen share within minutes and walk through the /proc/<pid>/fd recovery before any background process exits.
  2. PRMSCAN block recovery. If the instance is down, we run PRMSCAN on a snapshot or raw image of the volume and reassemble datafile images.
  3. RMAN-driven restore with forced gap skip. When you must take the 14 Nov backup forward despite the missing logs, we drive the force-skip sequence remotely and extract data via Data Pump.

Send the database version, the alert log around the deletion time, and a snapshot of ls -l /proc/<dbw_pid>/fd if the instance is still running, to [email protected]; we will recommend the path that fits your situation.

DBRECOVER Recovery Options

For Oracle incidents, start with the DBRECOVER for Oracle trial to verify table visibility, row previews, and export readiness on copied datafiles. For MySQL and InnoDB incidents, DBRECOVER for MySQL is free software and can inspect.ibd files, ibdata1, and database directories locally.

When the case is urgent, preserve the original files first, work from copies, and contact paid emergency support with the database version, platform, error messages, file list, and recovery objective.

Archive ParnassusData Blog Migration Archive