The 2 AM Success: Why Your Backup Is a Promise, Not a Backup

Your backup ran at 2 AM. It returned "Success." You have never opened it.

Now something is on fire and you need that data from March 15th. Not March 14th, not the latest snapshot. March 15th, because that's the last day the database was clean before the bad rows started landing, and the log in front of you says the March 15th job finished in 14 minutes with zero errors.

So you open the restore console. And the next four hours decide whether you get that data at all.

This is the part of backup culture nobody talks about honestly. We treat a clean job log the way sailors once treated a calm sea. Evidence of the journey, taken as proof of the destination. The job ran. The job succeeded. Therefore the data is recoverable. That "therefore" is the whole gap this article is about, and it only shows up at the one moment you can least afford a failure.

What "Success" actually tells you

A backup job that completes successfully has proven a narrow set of things:

1. The source was readable at 2 AM.
2. The destination was writable at 2 AM.
3. The write path did not error out.

That's it. It tells you nothing about whether the bytes you wrote are the bytes you need, whether the full set of files made it, whether the permissions travel with the data, whether the database is logically consistent at that instant, or whether a VM will actually boot when you bring it back. Write success and restore success are different engineering properties, and the backup industry has spent twenty years blurring them.

The numbers are uncomfortable, so here they are, and I'll be honest about where they come from. Sophos' State of Ransomware 2024 found that in 94% of incidents, cybercriminals attempted to compromise the organization's backups during the attack, and that in 57% of cases the compromise succeeded. An October 2023 At-Bay report on ransomware claims found the other side of the ledger: 31% of businesses that were hit could not restore data from their backups, even though 92% said they had backup coverage in place, and organizations that failed to restore were three times more likely to pay the ransom, while a successful restore cut the severity of the claim by 41%. An Avast survey published for World Backup Day 2017 adds that 60% of backups are incomplete and roughly half of restore attempts fail; caveat, that's survey data, not peer-reviewed science. None of these numbers prove that your backup is broken. They prove that the failure mode is common enough, and quiet enough, that you cannot assume you are clean.

The incidents behind those numbers are well documented, and it is worth being honest about what they show. None of them is a clean "the job said OK but the restore failed" case, which is itself a telling fact. What you find instead is the backup targeted as a deliberate step. REvil's operators, working out of the Sodinokibi campaign, built a routine around destroying backups before the encryption ever started: enumerate the repositories, purge the older generations, delete the catalog, kill the backup services, all documented as a named technique in threat-intel writeups. The Kaseya VSA attack in July 2021 pushed that from a tactic to a supply chain: one RMM vulnerability (CVE-2021-30116) let the payload run with high privileges across roughly fifty MSPs and up to 1,500 downline businesses, and in a lot of those environments the backup repository was gone before the ransom note was. NHS in 2017 shows the failure from the other end: 81 of 236 hospitals hit, shadow copies wiped, a recovery posture that had quietly decayed for over a year on unpatched endpoints. And in Düsseldorf in September 2020, a hospital's ransomware disruption contributed to the first reported patient death tied to an attack, a patient diverted mid-transfer who died in transit. I am not relitigating ransomware, and I am not claiming any of this is proof of an unverified backup. I am listing them because every one of these incidents contains a restore step that was impossible or degraded, and in every one the backup infrastructure had been assumed to be the safety net it was not. The restore point is where these stories end. It is where this article starts.

There is a second, quieter problem that no ransomware note will ever mention, and it is the one a homelab operator is more likely to meet: the backup that was never attacked, never lost, and still won't restore. Because it was written by a different version of the tool than the one you have now. Because the snapshot captured a database mid-transaction. Because the share lost its ACLs on the way back. Because the VM's second disk was marked "no backup" and fstab now points at a device that doesn't exist. These failures have no attacker, no incident response, no ransom note. Just a log that said Success and a restore that doesn't.

The 3-2-1 rule, and the two digits it was missing

The 3-2-1 backup rule came from photography, not IT. Peter Krogh popularized it in his 2009 book The DAM Book for managing digital photos: three copies, on two different media, with one offsite. It was a good model for its era, an era of tape and hard-drive failure, where the risk was physical. Lose the drive, restore from the copy.

For a long time, that was the whole risk model for most of us. Then two things changed. Attackers learned that a backup is just another datastore, and they started hunting it. And storage got cheap enough that we could afford to be a little more careful than the rule required.

So the industry grew the rule to 3-2-1-1-0: three copies, two media, one offsite, plus one immutable or offline copy, and zero unverified backups. I want to be careful about the origin of that extension. Datto, Veeam, and AvePoint all trace their version of it to different places, and there is no single primary source that predates the others. It looks less like an invention and more like an industry settling on the same conclusion at roughly the same time, because ransomware made the old rule provably insufficient. Different vendors also define the "0" slightly differently: Veeam frames it as zero recovery errors, AvePoint frames it as verification after every job. Those are subtly different emphases, and both are harder than they sound.

Here is where a homelab typically falls short, and I say this with my own rack in mind. The "1" is usually the easy digit to fake. A separate bucket in the same account is an "immutable copy" in the way a lockbox is "armored." The "0" is the digit nobody has, because it requires doing a restore, and a restore takes time, a staging target, and the willingness to be embarrassed by the result. Most of us, myself included at various points, have a 3-2-1 setup that is real, and a 1-0 that exists only in the vendor's marketing page. The rule was never the hard part. The drill was.

What "immutable" actually means at the storage layer

If your one-offline-copy is an S3 bucket with Object Lock, you probably believe it is WORM: write once, read many, un-deletable during the retention window. That belief is mostly true, and the mostly is where people get burned.

Object Lock works at the API level on versioned buckets. In compliance mode, nobody can shorten the retention, not even the root account. In governance mode, anyone holding the s3:BypassGovernanceRetention permission can release the lock early. If your "immutable" copy is in governance mode, its immutability is only as strong as your IAM discipline, and IAM discipline degrades the way all credentials do.

The failure modes that matter at restore time:

Delete markers are not WORM-protected. This one catches most people off guard. An attacker who can put objects in the bucket can place a delete marker on a version. The marker is not protected by the retention lock the way the object is. The locked object still physically exists, but it is hidden from the latest-version view, which is what most backup tools read. There are documented cases, in Veeam's forums, of direct-to-object backups where attackers planted delete markers across millions of objects and the backup tool simply could not see what it needed to restore. The data was there, the lock was intact, and the restore failed anyway. If your restore tool only ever asks "what is the current version?", a delete marker defeats you without touching a single locked byte.

Same account, same blast radius. Object Lock does not protect you from an attacker who has IAM access to your account. If the backup bucket and production live in the same cloud account, and that account's credentials drift, get shared, or get phished, an attacker can change lifecycle rules, rotate keys, or in governance mode release retention outright. Immutability inside a compromised account is a property of the account's control plane, not of the bytes.

Retention expires. A compliance-mode object with a retention date of 2025-03-15 is locked until that date. After it passes, the object is simply unlocked, no longer protected by Object Lock, and deletable by anyone with normal delete permission. Nobody deletes it automatically. It just stops being special. If your "immutable copy" retention is shorter than your longest plausible incident window, you have not bought immutability. You have bought a countdown.

KMS key loss makes it functionally gone. If the objects are encrypted with a customer-managed KMS key and that key is deleted, the objects are technically immutable and physically present and completely unreadable. You have a backup that passes every existence check and fails every restore. I don't know how many people would catch that until the fire.

You can't turn it on later. Object Lock must be enabled when the bucket is created. It cannot be retrofitted onto a pre-existing bucket. So the "immutable" copy you added to your existing setup last month, on a bucket that had data in it before you switched the flag on, may not be immutable in the way you think. Worth checking the bucket's creation metadata, if you've ever done it. I have. It is a two-minute check that saves a lot of self-confidence.

None of this means S3 Object Lock is bad. It is a real protection against a real threat, against the naive delete. It is not a protection against an attacker who has the account, and it is not a guarantee that what you wrote at 2 AM is what comes back in April.

The restore drill, done properly

This is the section that is actually the point. A verification restore is not a restore. It is a test with a pass/fail criterion, and the criterion has to be "the application works," not "the file exists."

Isolated target, always. Never restore into production. The staging environment is not bureaucracy; it is the difference between "found the corruption on Tuesday" and "wrote corrupted data over production on Friday at 6 PM when the deadline was real." Restore to a clean, network-isolated target that has no access to live systems.

Checksum validation where you can. If your tool computes checksums or content hashes, compare the restore against the recorded source hashes. Borg does this properly: borg check --verify-data runs full cryptographic verification over CRCs and HMACs. Restic has restic check --read-data, with the caveat that it checks all snapshots in the repository, not one, and that a metadata-only restic check does not validate blob contents. There are forum threads, documented in the project's own tracker, of pack files failing with "ciphertext verification failed" while the metadata check reported the repository healthy. The metadata layer says the map is fine. The map is fine. The territory is not.

File count and tree comparison. A restored directory tree with the right number of top-level folders is not the same as one with the right number of files at every depth. Diff the tree, not just the root. This is the check that catches the NAS share that came back with 40% of its contents and a permission set that no longer matches the source.

Application-level verification. This is where a real drill and a fake drill differ:

• Databases: pg_verifybackup for a PostgreSQL backup you have restored. DBCC CHECKDB for a restored SQL Server instance. For SQLite, open the database and run an integrity check. A file that restored cleanly and fails its own integrity check is not restored. It is a corpse with a pulse.
• VMs: boot them. Not "the restore finished," the VM boots to a functional login. I have restored VMs that came back with missing disks, because a second disk had been excluded from the backup and the guest's fstab referenced a device that no longer existed. The result was a rescue shell, not a desktop. There are solved threads in the Proxmox forum documenting exactly this, and a second thread where restore failed outright because the target storage's content type didn't match the source's. The VM's config migrated fine. The storage didn't.
• Services: after restore, hit the endpoints. HTTP 200, not HTTP 500. A service that starts and returns errors has been restored about as well as a car with all its wheels but no engine.

Automated post-restore validation. If the drill is manual, you won't do it when you're tired, and you definitely won't do it at 2 AM. Wire it up: the restore job triggers a verification job, the verification job runs the integrity checks and the app-level probes, and the result is a pass/fail that lands somewhere you will actually look. The industry has shipped pieces of this for years; the homelab version is a cron job, a script, and a notification that says "restore test FAILED" in the same channel where you already read alerts. You can build that weekend. The question is whether you will.

What breaks in practice

Let me be concrete, because the failure modes are boring and specific, and boring is where they live.

Deduplicated repositories rot in ways their own check misses. Borg issue #4777 in the project's tracker documents a case of widespread corruption across multiple clients and a hypervisor sharing one repository, with clean memory tests and clean ZFS scrubs on the hardware. The repository's integrity check was not enough to catch what was wrong. The same thread of problems exists in restic: repository-level checks can pass while pack files on disk are corrupt, and the failure only surfaces when you read the data back. The lesson is not that these tools are broken. The lesson is that a metadata check and a data check are different instruments, and the metadata check is the one everyone runs, because it is cheap.

Snapshots go stale in a way that is worse than missing. A backup taken on day eight of a slowly spreading problem is not a clean backup. It is a faithful record of the compromise, and it will restore exactly what the system looked like when it was already sick. The "good" snapshot you think you have is the one from day four, and you will not know which is which until you diff the contents. Staleness is the failure mode that the success log never flags, because from the job's point of view, a system that is quietly compromised is still a system that reads fine.

Send-and-receive tools break across different layouts. ZFS send and receive is one of the cleanest integrity-verified backup mechanisms there is, with checksums in the stream. It still fails across machines, documented in OpenZFS issue #5528 and a spread of community threads, when the receiving pool has a different vdev layout or device tree than the one that wrote the snapshot. The stream is fine. The pool it lands in is not the pool it was meant for. Corrupted snapshots can also propagate down the pipe, which is why the receive-side readonly import trick exists as a stopgap.

Restores lose the things that are not the file. ACLs. This is the NAS share you restored to the new box and then found that no user could access it, because the permission metadata never made the trip. Vendor knowledge bases, including Kaseya's own, document that agent-based CIFS and NFS backups cannot restore per-file ACLs; the data is in the backup, the permissions are not. A restore that gets the files back and not the access back has failed in the only way that matters to the people who were supposed to use the data.

Credential rotation silently kills your backups. This one is insidious because the failure is a log line you never read. Your backup job authenticates to the repository with a service account. That account gets rotated, or the certificate behind it expires, or the storage provider changes its auth scheme. The next 40 runs fail to authenticate. Some tools report that as a network error, or worse, keep reporting success against a cached state, or fail so quietly that the alert gets buried under a week of noise. The backup you think you are making in June was last actually made in February. When the fire comes in August, you restore from February, and the missing six weeks are why this is worse than any single data loss. If you have ever changed a password and not checked the next day's backup log, this one is yours to check today.

How often should you test

Not on a calendar. That's the honest answer, and it's annoying, because a calendar is what you can put on the wall.

The right frequency is per data class, keyed to how much you can afford to lose, not to the date on the month.

Databases and transactional data: test restore monthly, at minimum, with an integrity check on the restored instance, not just a file comparison. This is your highest-stakes class, and the one most likely to be logically corrupt while the file layer looks clean. If you can, test point-in-time restore specifically, because that is the exact capability you need when the corruption started on March 15th and you need to land on March 14th 23:59.

Full-system or VM backups: test quarterly with an actual boot. A VM restore that has never been booted is an untested restore. The boot test is the whole value, and it takes twenty minutes, which is the entire cost of the class.

File and NAS shares: test semi-annually with a tree diff and a permission spot-check. Pick ten files across the tree, not just the top level, and verify content hash and ACL against a known-good snapshot.

The immutable copy itself: test annually, and test it differently from the rest. Restore a sample from the immutable copy, not the primary, because the failure you are checking for is "the copy that was supposed to be your last line of defense is the one that is broken." Include the KMS key and the retention dates in that test. If the annual test restores from the primary instead, you have not tested the immutability. You have tested the primary again.

Automate what you can, and keep a manual pass. Automation gets you the daily "is the pipeline alive" signal: job ran, verification job ran, result was pass. It does not get you the boot test, or the judgment call about whether the restored service is actually behaving. The manual pass is where you catch the thing the script was not written to look for, and it is also where the drill stays a skill in your hands instead of a black box. When the fire comes, you will not be running the script. You will be doing the thing, by hand, tired, and the last time you did it by hand determines how well you do it then.

The 2 AM log, again

Come back to the opening. Your backup ran at 2 AM. It returned "Success."

A success log is a receipt. It proves a transaction happened. It does not prove the goods were what the receipt said they were. The difference is not a gap in your setup, and it is not a gap a better vendor closes. It is a gap in a habit: the habit of opening the box before the fire, while the box is still just a box, and failing the test costs an afternoon instead of a quarter.

The 3-2-1-1-0 rule, at its "0", is asking you to build that habit into the infrastructure instead of leaving it to willpower. Most of us are one failed restore away from finding out what our real number is, and it is never the one we thought.

The drill is not optional. The drill is the backup. Everything else is the part you hope is enough.

Sources

Organized by section, in the order the claims appear. Community-sourced failure modes are framed as documented in project trackers and vendor knowledge bases, not peer-reviewed. Survey statistics are industry research, not academic studies.

1. The 3-2-1-1-0 rule

1. Backblaze, "The 3-2-1 Backup Rule and Beyond" — https://www.backblaze.com/blog/whats-the-diff-3-2-1-vs-3-2-1-1-0-vs-4-3-2 — 2024 Claim: the 3-2-1 rule originated in photography; Peter Krogh popularized it in the 2009 DAM Book.
2. Veeam, "3-2-1 Backup Rule Explained" — https://www.veeam.com/blog/321-backup-rule.html Claim: the rule extended to 3-2-1-1-0; the "1" is an immutable copy, the "0" is zero recovery errors.
3. AvePoint, "What Is the 3-2-1 Backup Rule and Why It's Evolved to 3-2-1-1-0?" — https://www.avepoint.com/blog/backup/what-is-the-3-2-1-backup-rule — Apr 15, 2026 Claim: the "0" defined as verification after every job, a different emphasis than Veeam's.
4. Datto, "What Is the 3-2-1-1-0 Backup Rule?" — https://www.datto.com/blog/3-2-1-1-0-backup-rule Claim: adds immutable copy and verification to the traditional rule.
5. Cohesity, "What Is the 3-2-1 Backup Rule?" glossary — https://www.cohesity.com/glossary/321-backup-rule Claim: immutability plus validated accuracy layered onto the base rule.

2. Immutability at the storage layer

6. Eon, "How to Protect Data with AWS Immutable Backups" — https://www.eon.io/blog/aws-immutable-backups — May 28, 2026 Claims: delete markers can hide locked versions from backup tools; Object Lock does not protect keys or metadata.
7. Scality/Solved Magazine, "S3 Object Lock: Immutability and WORM" — https://www.solved.scality.com/how-aws-s3-object-lock-can-help-your-data-stay-safe Claims: retention expiry un-locks but does not delete objects; compliance vs governance mode differences.
8. Cloudian, "S3 Object Lock: Meet Compliance and Protect Against Ransomware" — https://cloudian.com/blog/s3-object-lock-protecting-data-for-ransomware-threats-and-compliance Claim: Object Lock must be enabled at bucket creation and cannot be added to an existing bucket.
9. Veeam Forums, "Advice on immutable S3 storage with a 'no delete' S3 bucket policy" — https://forums.veeam.com/viewtopic.php?f=52&t=100074 Claim: direct-to-object backups hit by attackers placing delete markers across millions of objects while the lock stayed intact.

3. Real-world incidents

10. Halcyon AI, "Why Ransomware Deletes Your Backups Before You Know You've Been Hit" — https://www.halcyon.ai/blog/why-ransomware-deletes-your-backups-before-you-know-youve-been-hit Claim: the systematic backup-destruction routine (enumerate repositories, purge older generations, delete the catalog, kill backup services) documented as a named TTP.
11. Sophos, "Ransomware Payments Increase 500% In the Last Year, Finds Sophos State of Ransomware Report" — https://www.sophos.com/en-us/press/press-releases/2024/04/ransomware-payments-increase-500-last-year-finds-sophos-state — Apr 2024; State of Ransomware 2024 whitepaper PDF: https://assets.sophos.com/X24WTUEQ/at/9brgj5n44hqvgsp5f5bqcps/sophos-state-of-ransomware-2024-wp.pdf Claims: 94% of organizations hit by ransomware saw backup-compromise attempts during the attack; 57% of attempts succeeded. Official Sophos press release for the State of Ransomware 2024 whitepaper (https://assets.sophos.com/X24WTUEQ/at/9brgj5n44hqvgsp5f5bqcps/sophos-state-of-ransomware-2024-wp.pdf), which contains the same figures.
12. Wikipedia, "Kaseya VSA ransomware attack" — https://en.wikipedia.org/wiki/Kaseya_VSA_ransomware_attack Claims: the July 2021 supply chain attack exploited CVE-2021-30116; roughly 50 MSPs and up to 1,500 downline businesses affected.
13. CISA, "Kaseya Ransomware Attack Guidance: Affected MSPs and Their Customers" — https://www.cisa.gov/news-events/news/kaseya-ransomware-attack-guidance-affected-msps-and-their-customers — Jul 2021 Claim: official U.S. government acknowledgment of the Kaseya VSA attack and guidance for affected MSPs and their customers.
14. UK National Audit Office, "Investigation: WannaCry cyber attack and the NHS" — https://www.nao.org.uk/reports/investigation-wannacry-cyber-attack-and-the-nhs — 2017 Claim: 81 of 236 NHS hospital sites affected; patching had lapsed despite alerts issued a year before the attack.
15. Fortune, "Ransomware attack on hospital may be first to cause a death" — https://fortune.com/2020/09/18/ransomware-police-investigating-hospital-cyber-attack-death — Sep 18, 2020 Claim: the Düsseldorf University Hospital attack (September 2020) forced diversion of a patient to another facility, delaying treatment by about an hour; the patient died. See also HIPAA Journal, "Hospital Ransomware Attack Results in Patient Death," https://www.hipaajournal.com/hospital-ransomware-attack-results-in-patient-death.

4. Failure modes

16. Borg issue #4777, "borg check --repair: Data integrity error" — https://github.com/borgbackup/borg/issues/4777 Claim: widespread corruption across multiple clients and a hypervisor sharing one repository, with clean memory tests and ZFS scrubs.
17. restic forum, "restic backup errors after running check --read-data" — https://forum.restic.net/t/restic-backup-errors-after-running-check-read-data/3030 Claim: pack files failing cryptographic verification while a metadata-only check reported the repository healthy.
18. OpenZFS issue #5528, "Force send a corrupt snapshot?" — https://github.com/openzfs/zfs/issues/5528 Claim: zfs send/receive failures across machines with different vdev layouts or device trees; the readonly-import stopgap.
19. Proxmox forum thread #37191, "VM backup restore and RDB additional disk" — https://forum.proxmox.com/threads/vm-backup-restore-and-rdb-additional-disk.37191 Claim: a VM with a disk marked "no backup" restores with a missing disk; fstab references a nonexistent device and the guest drops to a rescue shell.
20. Proxmox forum thread #163655, "Backup restore fails on new machine" — https://forum.proxmox.com/threads/backup-restore-fails-on-new-machine.163655 Claim: restore fails when the target storage's content type does not match the source's.
21. Kaseya KB, "Access Control List (ACL) Permissions are not restoring with file restores" — https://helpdesk.kaseya.com/hc/en-gb/articles/4407521628817 Claim: agent-based Windows CIFS and NFS backups cannot restore per-file ACLs; the data is in the backup, the permissions are not.

5. Statistics

22. At-Bay, "Report reveals businesses fail to recover backup when hit by ransomware" — https://www.at-bay.com/press_releases/report-reveals-businesses-fail-to-recover-backup-when-hit-by-ransomware/ — Oct 17, 2023 Claims: 92% of businesses had backups in place; 31% failed to restore data during a ransomware attack; failed restorers were 3x more likely to pay a ransom; effective backups reduced claim severity by 41%.
23. Avast Blog, "Why we (still) need World Backup Day" — https://blog.avast.com/why-we-still-need-world-backup-day — Mar 31, 2017 Claims: 60% of backups are incomplete; roughly half of restore attempts fail.