Disaster recovery

This is the plan for when something goes badly wrong. Disaster recovery is not about avoiding disasters, it's about making sure you can come back from one with the smallest possible loss of data and time. The starting point for everything on this page is good backups.

Three failure scenarios to plan for

Most real incidents fall into one of three buckets.

• Hardware failure. A disk dies, the GPU fails, the host won't boot. The platform is offline but the data on surviving disks is still good. Recovery is mechanical: replace the failed component, bring the host back, verify everything comes up green.
• Data corruption. Something writes bad data, or the right data gets overwritten. The host is fine, the platform is running, but one dataset (a knowledge base, a config table, the audit log) is in a state you need to roll back. Recovery is surgical: restore just that subset from a clean backup.
• Total site loss. Fire, flood, ransomware, theft. The host and everything on it are gone. Recovery means rebuilding the platform on new hardware from the off-site backups you've been quietly maintaining. This is the scenario your backup discipline is really for.

What recovery looks like in each scenario

Recovering from hardware failure

1. Diagnose which component failed. System status and host logs usually tell you.
2. Replace the failed component (disk, GPU, RAM, PSU, the whole host if needed).
3. Bring the host back up.
4. If the data volume survived, the platform should come up where it left off. Verify on System status.
5. If the data volume didn't survive, restore from your most recent backup. See Backups for the restore workflow.
6. Run health checks: sign in, open a chat, query a KB, confirm citations come back, spot-check the audit log.

TIP

Hardware failures are the most common DR scenario and the easiest to recover from, as long as your backups are recent. A failed GPU is a one-day incident. A failed disk without a recent backup is a much longer one.

Recovering from data corruption

1. Identify the affected dataset. The audit log usually points at the bad change and when it happened.
2. Stop further writes to that dataset if you can without taking the whole platform down.
3. Restore the affected subset from the most recent clean backup, ideally from before the corruption was introduced. A full restore is rarely necessary.
4. Verify: open the affected KB or settings, confirm it looks right, confirm related data still references it correctly.
5. Document what happened in the audit trail so the next person knows why the data jumped backward.

Recovering from total site loss

1. Provision replacement hardware that matches your previous tier or better. See Hardware sizing.
2. Install Stoa on the new host. Your delivery engineer handles this. See Installation.
3. Restore the full backup set: documents, indexes, database, configuration, audit log.
4. Reconnect external integrations (directory, email, webhooks) and confirm each one tests green.
5. Walk through a post-recovery checklist: sign in as admin and as a regular user, open a chat with KB grounding, run a research job, confirm the audit log picks up where the backup left off.
6. Communicate the recovery to your users. They've been waiting.

RTO and RPO in plain terms

Two numbers shape your DR plan.

• Recovery time objective (RTO): how long can you be down? An hour, a day, a week? The smaller the number, the more standby capacity you need (warm spare hardware, near-real-time replication, automated restores).
• Recovery point objective (RPO): how much data can you afford to lose? An hour's worth, a day's worth, a week's? The smaller the number, the more frequent your backups need to be.

Pick honest numbers for your environment, then check that your backup cadence and off-site policy actually meet them. A daily backup with a weekly off-site copy is a 24-hour RPO on a good day and a 7-day RPO on a bad one. If your stated RPO is 4 hours, your current strategy doesn't match it.

The role of your delivery engineer

For major recoveries, your delivery engineer travels on-site (or coordinates remotely) to assist. You don't have to do total-site recovery alone, and you shouldn't try. Stoa is delivered as a partnership precisely because the worst day to learn the restore procedure is the day you actually need it. Document how to reach your delivery engineer outside business hours and what your contractual response time is.

Document your specific recovery plan

Every Stoa deployment is different. Storage layouts, network topologies, integration partners, and backup destinations all vary. The pages in this docs site can't write your DR runbook for you. They can give you the shape. Write down, for your environment:

• Where backups live (paths, hostnames, storage system, off-site destination).
• Who has access, and how it's granted.
• The exact restore steps, including any environment-specific commands.
• The contact list: delivery engineer, infrastructure team, security team, leadership.
• The communications plan: who tells users, how, and when.
• The post-recovery checklist.

Keep this document somewhere that survives the disaster. A copy on the failed host is useless.

Test your recovery

You have never tested recovery if you have never restored from a backup. The first time you discover that a backup is incomplete should not be the day you need it. Once a quarter, restore one of your backups into a non-production environment and walk through the steps. Time it. Note what was missing, confusing, or slower than expected. Update your DR plan. If your compliance regime requires a documented restore drill, this exercise satisfies it. If it doesn't, do it anyway.

Next

• Backups for the foundation that DR depends on.
• Updates & on-site delivery for the planned-maintenance counterpart of an unplanned recovery.
• What to monitor to catch problems before they become disasters.