AI-Development-Server/docs/DISASTER-RECOVERY.md

# Disaster Recovery

This document defines expected recovery scenarios for Atlas. It does not claim that backups currently exist unless they have been verified.

Backup destination, retention, encryption, restore testing, and ownership are `TODO: Confirm during the relevant implementation phase.`

## Recovery Principles

- Protect source repositories and infrastructure history first.
- Keep secrets outside this repository.
- Prefer rebuild from documented automation over manual reconstruction.
- Validate restored services before returning them to use.
- Record recovery actions in [BUILD-LOG.md](BUILD-LOG.md).

## OS Disk Failure

### Impact

- Ubuntu installation unavailable.
- System packages and live configuration on the OS disk unavailable.
- `/sql` and `/srv` may remain intact if their disks are healthy.

### Required Backups

- Ubuntu installation media.
- Repository access through Gitea or another verified copy.
- Copies of live system configuration not yet generated from this repository.
- Secrets and SSH keys from an approved location outside this repository.

### Recovery Outline

1. Replace the OS disk if required.
2. Reinstall Ubuntu 26.04 LTS.
3. Restore hostname, network, SSH, and sudo access.
4. Remount `/sql` and `/srv`.
5. Clone or use the existing infrastructure repository.
6. Run future installation automation when available.
7. Restore live configuration and services.

### Validation

- Hostname is `atlas`.
- Static IP address is `192.168.10.10`.
- `/sql` and `/srv` are mounted.
- Git can reach Gitea.
- Required services start and pass smoke tests.

### Remaining Gaps

- OS rebuild automation is not yet implemented.
- Live configuration inventory is not yet complete.

## SQL Disk Failure

### Impact

- SQL data on `/sql` may be unavailable or lost.
- Applications depending on SQL Server may be unavailable.

### Required Backups

- Verified SQL backups from outside this repository.
- SQL Server installation and configuration documentation.
- SQL credentials from approved secret storage.

### Recovery Outline

1. Replace or repair the SQL disk.
2. Recreate filesystem and mount at `/sql`.
3. Reinstall or validate SQL Server.
4. Restore databases from verified backups.
5. Reapply permissions and maintenance jobs.

### Validation

- `/sql` is mounted.
- SQL Server starts.
- Databases are online.
- Application connectivity succeeds.
- Restore validation queries pass.

### Remaining Gaps

- SQL Server is not yet documented as installed by this repository.
- Backup location, retention, and restore procedure are not yet confirmed.

## Development Disk Failure

### Impact

- `/srv` may be unavailable or lost.
- Local repository working copies, scripts in progress, build artefacts, and deployment material on `/srv` may be lost.

### Required Backups

- Gitea remote repositories.
- Any non-committed work that was backed up outside `/srv`.
- Deployment artefacts only if they cannot be rebuilt from source.

### Recovery Outline

1. Replace or repair the development disk.
2. Recreate filesystem and mount at `/srv`.
3. Recreate `/srv/repos`.
4. Clone required repositories from Gitea.
5. Recreate build and deployment directories from documented automation.

### Validation

- `/srv` is mounted.
- `/srv/repos` exists.
- Infrastructure repository is cloned.
- Required application repositories are cloned.
- Smoke tests pass where available.

### Remaining Gaps

- Application repository inventory is not yet documented.
- Build artefact and deployment location policy is not yet confirmed.

## Complete Server Loss

### Impact

- All local OS, SQL, development, configuration, and service state may be lost.
- Recovery depends on external repositories, backups, installation media, and secret storage.

### Required Backups

- Gitea-hosted repositories or another off-server Git copy.
- SQL backups.
- Secret material from approved secret storage.
- Configuration not yet represented in this repository.
- Documentation needed to rebuild storage, network, and services.

### Recovery Outline

1. Replace or rebuild hardware.
2. Install Ubuntu 26.04 LTS.
3. Recreate storage layout with `/sql` and `/srv`.
4. Restore network identity.
5. Restore SSH and Gitea access.
6. Clone this repository.
7. Follow [REBUILD-GUIDE.md](REBUILD-GUIDE.md).
8. Restore databases and services from verified backups.

### Validation

- Server identity matches documented values.
- Repositories are restored.
- Required services pass validation.
- Applications pass smoke tests.
- Data restore is verified.

### Remaining Gaps

- Full rebuild is not yet automated.
- Off-server backup coverage has not been verified in this repository.

## Accidental Repository Deletion

### Impact

- Local infrastructure working copy under `/srv/repos/AI-Development-Server` is lost.
- Uncommitted changes may be lost.

### Required Backups

- Gitea remote repository.
- Any external backup containing uncommitted work, if available.

### Recovery Outline

1. Recreate `/srv/repos` if needed.
2. Clone the repository from Gitea.
3. Confirm branch and remote.
4. Reapply any known uncommitted work only from trusted sources.

### Validation

- `git remote -v` shows the expected Gitea remote.
- `git status` is clean or only shows expected work.
- Required documentation and scripts are present.

### Remaining Gaps

- Uncommitted work recovery depends on external backups or manual reconstruction.

## Failed Application Deployment

### Impact

- A deployed application may be unavailable, partially updated, or running an unintended version.
- Source repositories and this infrastructure repository should remain intact.

### Required Backups

- Previous known-good deployment artefact or source revision.
- Database backup if the deployment includes schema or data changes.
- Previous live configuration if changed.

### Recovery Outline

1. Stop or isolate the failed deployment if required.
2. Restore the previous artefact or redeploy the previous source revision.
3. Restore database state if required and approved.
4. Restore previous configuration if changed.
5. Validate application behaviour.

### Validation

- Service status is healthy.
- nginx routing works if applicable.
- Application smoke tests pass.
- Logs show no repeated startup or request failures.

### Remaining Gaps

- Deployment paths, rollback scripts, and smoke tests are not yet defined.

## Configuration Corruption

### Impact

- Services may fail to start or may run with unintended settings.
- Secret-bearing live configuration may need manual restoration.

### Required Backups

- Repository-managed templates.
- Backup of live configuration files.
- Secret values from approved secret storage.

### Recovery Outline

1. Identify the corrupted configuration.
2. Compare live configuration with repository templates.
3. Restore from a known-good backup or regenerate from templates.
4. Reinsert secrets from approved storage without committing them.
5. Validate and reload affected services.

### Validation

- Service configuration validation passes.
- Service reload or restart succeeds.
- Logs show expected behaviour.
- Application smoke tests pass where available.

### Remaining Gaps

- Live configuration inventory and backup process are not yet complete.

## Credential Compromise

### Impact

- Unauthorised access may be possible to Atlas, Gitea, databases, deployments, external APIs, or backups.
- Affected credentials must be rotated.

### Required Backups

- Not primarily a backup-driven recovery scenario.
- Requires access to credential authorities and secret storage.

### Recovery Outline

1. Identify affected credentials and access scope.
2. Revoke or rotate compromised credentials.
3. Review logs for misuse.
4. Update live systems with new credentials.
5. Confirm no secret values were committed to Git.
6. If a secret was committed, treat Git history and all clones as compromised for that secret.

### Validation

- Old credentials no longer work.
- New credentials work only where intended.
- Services using rotated credentials pass validation.
- Repository search finds no committed live secret values.

### Remaining Gaps

- Credential inventory, rotation schedule, and incident process are not yet confirmed.