AI-Development-Server/docs/REBUILD-GUIDE.md

# Rebuild Guide

This guide describes the planned rebuild path for Atlas. At Phase 1, the repository contains documentation but no executable installation phases. Steps marked as TODO are not yet automated.

## Automation Status

- Repository structure and documentation: present.
- Shared installation framework: present.
- System discovery script: present.
- Host validation script: present.
- .NET installation script: present.
- Docker installation script: present.
- SQL Server container installation script: present.
- nginx installation script: present.
- PlotDirector development deployment script: present.
- Other component installation scripts: not yet created.
- Configuration templates: not yet created.
- Operational scripts: not yet created.
- Smoke tests: not yet created.
- Backup and restore automation: not yet created.

## Preconditions

- Physical server hardware is available.
- Ubuntu 26.04 LTS installation media is available.
- Network details for Atlas are known.
- Gitea SSH access can be restored or re-created.
- Required secrets and private keys are available from an approved location outside this repository.
- Backups are available where required.

Backup inventory and credential recovery process are `TODO: Confirm during the relevant implementation phase.`

## 1. Ubuntu Installation

1. Install Ubuntu 26.04 LTS on the OS SSD.
2. Set hostname to `atlas`.
3. Create the dedicated development user.
4. Configure administrative access for the development user.
5. Apply operating system updates.

Exact installer choices, partitioning profile, and user creation procedure are `TODO: Confirm during the relevant implementation phase.`

## 2. Storage Preparation

Target storage layout:

- OS volume mounted at `/`
- SQL data volume mounted at `/sql`
- Development volume mounted at `/srv`
- Repository root at `/srv/repos`

Current Phase 1 status:

- `/sql` is the intended SQL data volume.
- `/srv` is the primary development volume.
- Idempotent storage preparation scripts do not exist yet.

Future rebuild work must document:

- Disk identification method.
- Filesystem labels or UUIDs.
- `/etc/fstab` entries.
- Mount validation commands.
- Ownership and permissions.

These details are `TODO: Confirm during the relevant implementation phase.`

## 3. Network Setup

Target identity:

- Hostname: `atlas`
- Static IP address: `192.168.10.10`
- Default gateway: `192.168.10.1`
- Primary interface: `enp3s0`

Configure DNS, host mappings, firewall rules, and public exposure only after the relevant policy is confirmed.

DNS and firewall details are `TODO: Confirm during the relevant implementation phase.`

## 4. SSH

1. Install or enable OpenSSH.
2. Confirm SSH access from trusted administration clients.
3. Restore or create the SSH key needed for Gitea access.
4. Keep private keys outside this repository.

SSH hardening policy, trusted client list, and key rotation procedure are `TODO: Confirm during the relevant implementation phase.`

## 5. Repository Checkout

Create the repository root if needed:

```bash
sudo mkdir -p /srv/repos
```

Clone the infrastructure repository:

```bash
cd /srv/repos
git clone git@git.catherinelynwood.com:atlas-bot/AI-Development-Server.git
cd AI-Development-Server
```

If the remote repository is unavailable, follow the disaster recovery guidance in [DISASTER-RECOVERY.md](DISASTER-RECOVERY.md).

## 6. Bootstrap Process

At Phase 1, bootstrap is manual. Future bootstrap should be reduced to a documented command sequence that runs idempotent scripts from this repository.

Planned bootstrap flow:

1. Review [README.md](../README.md).
2. Review [SERVER-STANDARDS.md](../SERVER-STANDARDS.md).
3. Review this rebuild guide.
4. Run `scripts/system/discover-system.sh`.
5. Run `scripts/system/validate-host.sh`.
6. Run `install/bootstrap.sh --dry-run`.
7. Run `install/phases/10-dotnet.sh` when .NET SDKs are required.
8. Run `install/phases/30-docker.sh`.
9. Run `install/phases/20-sql-server.sh` after Docker is available.
10. Run `install/phases/40-nginx.sh`.
11. Apply the UFW baseline rules.
12. Clone PlotDirector to `/srv/repos/PlotDirector` on branch `onboarding`.
13. Create protected PlotDirector runtime configuration at `/etc/plotdirector/plotdirector.env`.
14. Initialise the `PlotDirector_Development` database using the numbered SQL scripts.
15. Run `install/phases/50-plotdirector-dev.sh`.
16. Run additional ordered installation scripts from `install/phases/` when they exist.
17. Run smoke tests from `tests/` when they exist.

Bootstrap command sequence is `TODO: Confirm during the relevant implementation phase.`

## 7. Running Installation Phases

Planned scripts may include:

```text
install/phases/10-dotnet.sh
install/phases/20-sql-server.sh
install/phases/30-docker.sh
install/phases/40-nginx.sh
install/phases/50-plotdirector-dev.sh
```

Current Docker and SQL Server commands:

```bash
install/phases/30-docker.sh
install/phases/20-sql-server.sh
```

SQL Server operations:

```bash
sudo docker compose -p atlas-sql -f config/sql-server/compose.yml up -d
sudo docker compose -p atlas-sql -f config/sql-server/compose.yml stop
sudo docker compose -p atlas-sql -f config/sql-server/compose.yml restart
sudo docker ps --filter name=atlas-sql-server
sudo docker logs atlas-sql-server
sudo docker inspect -f '{{.State.Health.Status}}' atlas-sql-server
```

SQL Server live secrets are stored at `/etc/atlas/sql-server.env` and must not be committed.

Current nginx validation commands:

```bash
sudo nginx -t
systemctl is-active nginx
curl -fsS http://127.0.0.1/health
curl -fsS http://192.168.10.10/health
```

Current firewall inspection command:

```bash
sudo ufw status numbered
```

Current PlotDirector development deployment command:

```bash
install/phases/50-plotdirector-dev.sh
```

PlotDirector runtime details:

- Source repository: `/srv/repos/PlotDirector`
- Required branch: `onboarding`
- Entry project: `PlotLine/PlotLine.csproj`
- Published path: `/srv/apps/plotdirector-dev/current`
- Runtime environment file: `/etc/plotdirector/plotdirector.env`
- Service: `plotdirector-dev`
- Kestrel endpoint: `http://127.0.0.1:5050`
- nginx hostname: `dev.plotdirector.com`
- Database: `PlotDirector_Development`
- SQL login: `plotdirector_dev`

Use this Windows hosts-file entry for LAN testing:

```text
192.168.10.10 dev.plotdirector.com
```

PlotDirector validation commands:

```bash
systemctl is-active plotdirector-dev
journalctl -u plotdirector-dev --no-pager -n 100
curl --noproxy '*' -fsS http://127.0.0.1:5050/ >/dev/null
curl --noproxy '*' -fsS -H 'Host: dev.plotdirector.com' http://127.0.0.1/ >/dev/null
sudo nginx -t
```

Installation scripts must:

- Be idempotent.
- Validate prerequisites.
- Log concise progress.
- Avoid printing secrets.
- Document rollback expectations.
- Include or reference validation steps.

Current .NET installation command:

```bash
install/phases/10-dotnet.sh
```

Current .NET verification commands:

```bash
dotnet --info
dotnet --list-sdks
dotnet --list-runtimes
```

The current installed SDKs are .NET 8 SDK `8.0.128` and .NET 10 SDK `10.0.109`.

## 8. Validation

Minimum rebuild validation should confirm:

- Hostname is `atlas`.
- Static IP address is `192.168.10.10`.
- `/sql` is mounted.
- `/srv` is mounted.
- `/srv/repos` exists.
- Git can reach Gitea over SSH.
- This repository is checked out.
- Required installation phases have run successfully.
- Smoke tests pass.

Validation scripts are currently available at `scripts/system/discover-system.sh` and `scripts/system/validate-host.sh`.

Phase 2A host validation currently checks `/` has at least 10 GiB free, `/srv` has at least 20 GiB free, and `/sql` has at least 20 GiB free.

## 9. Restoring Services and Data

Restoration order should be:

1. Restore base OS access.
2. Restore storage mounts.
3. Restore repository access.
4. Run installation automation.
5. Restore configuration generated from templates.
6. Restore SQL data from verified backups.
7. Restore deployed applications or rebuild them from source.
8. Validate services and application behaviour.

SQL backup location, deployment artefact policy, restore commands, and service validation are `TODO: Confirm during the relevant implementation phase.`