Ask a team whether they have backups and almost all of them say yes. Ask when someone last restored one and the answer changes shape: a pause, then either a specific date or a slightly uncomfortable "well, they're running."
That gap is the whole subject. A backup is a claim about the future. A tested restore is evidence. Until someone has taken a backup file and turned it back into a working system, you have a process that produces files nobody has opened.
What actually goes wrong
Restores fail in specific, boring, repeatable ways. These are worth knowing in advance, because each of them looks fine right up until the moment you need the data.
The retention window is shorter than you think. Many managed database plans keep seven days. That is fine for "I dropped the wrong table this morning" and useless for "a bad migration three weeks ago has been quietly corrupting records ever since." The second scenario is the one that takes a business down, and it is the one short retention does not cover.
The backup covers less than the system. Your database is backed up. Are the uploaded files? The object storage bucket holding customer documents is frequently outside the database backup entirely, which means a restore gives you rows pointing at files that are gone. This is the single most common surprise in a first drill.
The restore takes longer than anyone assumed. A logical dump of a large database restores at a rate that has very little to do with how fast it was to create. Teams who have never timed it tend to guess minutes and discover hours.
The schema has moved on. A backup from before a migration restores fine and then the application will not start, because the code expects columns that do not exist yet in that snapshot. Restoring old data into current code is its own small project.
Nobody knows the procedure. The person who set up the backups has the steps in their head, or in a terminal history, or nowhere. At the point you need a restore, that person is on a plane.
The drill
Set aside two hours. You want the whole thing to feel unhurried, because the point is to discover problems, not to get a tick.
- Pick a restore point deliberately. Not the most recent backup. Take one from a few days ago, because that is closer to the real scenario where a problem went unnoticed for a while.
- Restore into a fresh environment. A new database instance, a scratch project, a local container. Never over the top of anything live.
- Start a timer when you begin and stop it when the application serves a page from restored data. Not when the restore command finishes. The number you want is time to working system, which includes reconnecting the app, running any migrations, and waiting for indexes to rebuild.
- Point a copy of the application at it and use the product. Sign in. Open a record. Download a file. Run a report. The failures that matter appear here, not in the restore log.
- Write down what broke and what you had to do by hand. That list is the first draft of your runbook.
The file download step deserves emphasis. If your product stores documents, images or exports outside the database, a database-only restore produces an application that looks healthy and is full of broken links. You want to find that on a Tuesday afternoon, not during an incident.
What to record afterwards
A drill that produces no document was a nice exercise. Write down four things, in whatever you already use:
- The date and who ran it.
- Time to working system, measured not estimated.
- Data loss window: how much time sat between the backup you used and the moment of simulated failure. This is your real recovery point, and it is usually larger than the backup interval because backups finish some time after they start.
- What needed manual intervention. Every item here is either a runbook step or a thing to automate.
Those first two numbers are what an RTO and RPO should be built from. A recovery objective written without a drill behind it is a guess in formal clothing, and an auditor asking about recovery time is really asking whether you have ever measured it.
What "good" looks like for a small team
You do not need a hot standby and a runbook with change control. For a team of five running a normal SaaS product, defensible looks like:
- Automated daily backups with retention long enough to cover a problem nobody noticed for a fortnight. Thirty days is a common, comfortable answer.
- Object storage covered as well as the database, either through the provider's own versioning or a separate copy.
- One tested restore on record, with timings, within the last quarter.
- A short written procedure that somebody other than its author could follow.
That last point is the cheapest and most often skipped. A page of numbered steps, stored somewhere that does not require the production system to be up to read it, converts a restore from an act of individual memory into something the team can do.
What to back up beyond the database
The database is the part everyone remembers. A restore that produces a working product usually needs three more things, and each is missed often enough to be worth naming.
- Uploaded files. Object storage is a separate system with separate retention. Many providers offer versioning, which protects against deletion and overwriting but is not the same as a backup you can restore to a point in time. Check which you have.
- Configuration. Environment variables, DNS records, queue definitions, scheduled jobs, webhook endpoints. These are rarely backed up because they are not files, and they are what turns a restored database into a running service. A document listing them, kept current, is usually enough.
- The things outside your infrastructure entirely. If a third party holds state you depend on, a subscription record, a customer list, a set of templates, your backup does not cover it. Knowing which vendor holds what is the same exercise as building a vendor inventory.
A drill that restores the database and stops short of these will pass while leaving the real gap in place. Restoring the database and then failing to download a customer document is the finding you want.
Immutability, briefly
Ransomware changes the shape of this question. If an attacker gets credentials that can reach your backups, backups that can be deleted or overwritten are part of the blast radius rather than the recovery from it.
Object lock, versioning with a retention policy, or simply a copy held in an account with separate credentials all address this. Whether it is proportionate depends on what you hold and who would want it, and it is a reasonable thing to decide against deliberately. Deciding against it on purpose, and writing down why, is a different posture from never having considered it.
The honest version
Most teams reading this have backups running and have never restored one. That is an extremely common position and not a shameful one. It is also a position where you do not actually know whether your data is recoverable, and you will find out at the worst possible moment.
Two hours, once a quarter, converts an assumption into a fact. It is one of the few items on a readiness checklist where the effort is small, the finding is binary, and the thing you are protecting is the entire business.