# Story Intelligence Phase 20X Audit

## Purpose

This audit records the current state of Story Intelligence after the Phase 20 admin proving work and the first commit foundation. It is intentionally diagnostic. It does not propose new experimental tooling, prompt tuning, or entity import expansion.

## Executive Summary

Story Intelligence has successfully proven the AI pipeline: chapter text can be split into scene boundaries, each scene can be analysed with `gpt-5-mini`, results can be validated, persisted, reviewed, resolved into an import preview, and partially committed into PlotDirector scenes.

The system is not yet a usable author-facing manuscript import flow. Most working entry points are admin/test pages. The only real commit path is admin-only and currently blocked for normal existing chapters because PlotDirector creates a default scene for every chapter. The next phase should fix commit into chapters that contain only the untouched default placeholder scene, then expose the shortest author-facing paste/upload path.

## 1. Existing Entry Points

### Admin Story Intelligence Diagnostics

- Route: `GET /admin/story-intelligence-diagnostics`
- Controller/action: `AdminController.StoryIntelligenceDiagnostics`
- View: `Views/Admin/StoryIntelligenceDiagnostics.cshtml`
- Service: `StoryIntelligenceDiagnosticsService`
- Purpose: Shows build/config/live API diagnostics for Story Intelligence configuration.
- Access: Admin-only.
- Still needed: Yes, but as admin diagnostics only.
- Writes data: No.

### Admin Scene Intelligence Test

- Route: `GET /admin/story-intelligence-test`
- Route: `POST /admin/story-intelligence-test`
- Controller/actions: `AdminController.StoryIntelligenceTest`
- View: `Views/Admin/StoryIntelligenceTest.cshtml`
- Service: `StoryIntelligenceDryRunService`
- Purpose: Single-scene prompt/API/parse/validation test.
- Access: Admin-only.
- Still needed: Useful for low-level prompt/API debugging, not part of the future author flow.
- Writes data: No.

### Admin Chapter Structure Test

- Route: `GET /admin/chapter-structure-test`
- Route: `POST /admin/chapter-structure-test`
- Controller/actions: `AdminController.ChapterStructureTest`
- View: `Views/Admin/ChapterStructureTest.cshtml`
- Service: `ChapterStructureDryRunService`
- Purpose: Tests chapter structure prompt and validation.
- Access: Admin-only.
- Still needed: Useful for debugging boundary quality, not part of author flow.
- Writes data: No.

### Admin Combined Chapter-To-Scene Test

- Route: `GET /admin/chapter-story-intelligence-test`
- Route: `POST /admin/chapter-story-intelligence-test`
- Controller/actions: `AdminController.ChapterStoryIntelligenceTest`
- View: `Views/Admin/ChapterStoryIntelligenceTest.cshtml`
- Service: `ChapterStoryIntelligenceDryRunService`
- Purpose: Runs chapter structure, splits text, runs Scene Intelligence sequentially, and displays combined dry-run output.
- Access: Admin-only.
- Still needed: Useful as an in-memory pipeline debugger.
- Writes data: No on normal test post.

### Save Displayed Combined Dry Run

- Route: `POST /admin/chapter-story-intelligence-test/save`
- Controller/action: `AdminController.SaveChapterStoryIntelligenceRun`
- Service: `StoryIntelligenceResultPersistenceService.SaveDisplayedRunAsync`
- Purpose: Saves a displayed admin dry run into Story Intelligence audit tables.
- Access: Admin-only.
- Still needed: Maybe temporarily, but it is test tooling.
- Writes data: Yes, Story Intelligence audit rows only.

### Queue Admin Text Run

- Route: `POST /admin/chapter-story-intelligence-test/queue`
- Controller/action: `AdminController.QueueChapterStoryIntelligenceRun`
- Service: `StoryIntelligenceResultPersistenceService.QueueAdminTextRunAsync`
- Purpose: Queues pasted text for the persisted background runner.
- Access: Admin-only.
- Still needed: Useful until an author-facing import wizard exists.
- Writes data: Yes, queues a `StoryIntelligenceRuns` row; background worker later persists chapter and scene results.

### Queue From Existing Chapter

- Route: `GET /admin/story-intelligence-existing-chapter`
- Route: `POST /admin/story-intelligence-existing-chapter`
- Controller/actions: `AdminController.StoryIntelligenceExistingChapter`, `QueueStoryIntelligenceExistingChapter`
- View: `Views/Admin/StoryIntelligenceExistingChapter.cshtml`
- Services: `StoryIntelligenceExistingChapterQueueService`, `StoryIntelligenceSourceRepository`
- Purpose: Attempts to queue a persisted run from server-side text on an existing PlotDirector chapter.
- Access: Admin-only.
- Still needed: Diagnostic only unless server-side chapter text becomes reliable.
- Writes data: Yes, queues a Story Intelligence run if source text exists.

### Full Chapter Test

- Route: `GET /admin/story-intelligence-full-chapter-test`
- Route: `POST /admin/story-intelligence-full-chapter-test`
- Controller/actions: `AdminController.StoryIntelligenceFullChapterTest`
- View: `Views/Admin/StoryIntelligenceFullChapterTest.cshtml`
- Service: `StoryIntelligenceFullChapterTestService`
- Purpose: Admin paste/file full-chapter pipeline with configurable stage models.
- Access: Admin-only.
- Still needed: Useful until replaced by author-facing import.
- Writes data: Yes when queued; preflight/analyse mode does not write.

### Saved Runs List

- Route: `GET /admin/story-intelligence-runs`
- Controller/action: `AdminController.StoryIntelligenceRuns`
- View: `Views/Admin/StoryIntelligenceRuns.cshtml`
- Service: `StoryIntelligenceResultPersistenceService.ListRunsAsync`
- Purpose: Lists saved/persisted Story Intelligence runs.
- Access: Admin-only.
- Still needed: Yes for audit/admin monitoring.
- Writes data: No.

### Saved Run Detail, Import Preview, Monitor

- Route: `GET /admin/story-intelligence-runs/{id}`
- Controller/action: `AdminController.StoryIntelligenceRunDetails`
- View: `Views/Admin/StoryIntelligenceRunDetails.cshtml`
- Service: `StoryIntelligenceResultPersistenceService.GetRunDetailAsync`
- Resolver: `SceneImportResolver`
- Purpose: Shows run metadata, raw/parsed AI output, validation, scene results, import preview, and commit status.
- Access: Admin-only.
- Still needed: Yes for audit and troubleshooting.
- Writes data: No.

### Cancel Run

- Route: `POST /admin/story-intelligence-runs/{id}/cancel`
- Controller/action: `AdminController.CancelStoryIntelligenceRun`
- Purpose: Requests cancellation of queued/running persisted Story Intelligence jobs.
- Access: Admin-only.
- Still needed: Yes for long-running jobs.
- Writes data: Yes, updates run cancellation fields.

### Commit Confirmation

- Route: `GET /admin/story-intelligence-runs/{id}/commit`
- Controller/action: `AdminController.ConfirmStoryIntelligenceImportCommit`
- View: `Views/Admin/ConfirmStoryIntelligenceImportCommit.cshtml`
- Service: `StoryIntelligenceImportCommitService.BuildConfirmationAsync`
- Purpose: Shows deliberate confirmation before creating real PlotDirector scenes.
- Access: Admin-only.
- Still needed: Yes until author-facing import has its own equivalent confirmation.
- Writes data: No.

### Commit Import

- Route: `POST /admin/story-intelligence-runs/{id}/commit`
- Controller/action: `AdminController.CommitStoryIntelligenceImport`
- Service: `StoryIntelligenceImportCommitService.CommitAsync`
- Repository: `StoryIntelligenceResultRepository.CommitImportAsync`
- Purpose: Creates PlotDirector scenes, metrics, notes and commit audit from a completed Story Intelligence run.
- Access: Admin-only.
- Still needed: Yes, but currently blocked for default-scene chapters.
- Writes data: Yes, real PlotDirector scene data and commit audit.

### Author-Facing Story Intelligence Onboarding

- Routes:
  - `GET /onboarding/story-intelligence`
  - `POST /onboarding/story-intelligence/start`
  - `POST /onboarding/story-intelligence/skip`
  - `GET /onboarding/story-intelligence/progress`
  - `POST /onboarding/story-intelligence/cancel`
  - `GET /onboarding/story-intelligence/complete`
- Controller/actions: `OnboardingController`
- Service: `StoryIntelligenceService`
- Purpose: Optional onboarding framework/stub job for Story Intelligence readiness.
- Access: Signed-in author.
- Still needed: Conceptually yes, but it is not the real manuscript import flow.
- Writes data: Yes, onboarding/story intelligence job state. It does not analyse manuscript text or import scenes.

### Generic JSON Import

- Routes under `ImportsController`
- Purpose: Existing project backup/import tooling, unrelated to Story Intelligence manuscript import.
- Access: Author-facing.
- Still needed: Yes, but not part of Story Intelligence.
- Writes data: Yes, generic import data when used.

### Word Companion API Import

- Routes under `/word-companion/...`, especially:
  - `POST /word-companion/manuscript/analyse`
  - `POST /word-companion/manuscript/import`
  - `POST /word-companion/books/{bookId}/chapters`
  - `POST /word-companion/books/{bookId}/chapters/{chapterId}/scenes`
- Controller: `WordCompanionController`
- Repository: `WordCompanionRepository`
- Purpose: Word add-in/runtime bridge for manuscript structure and sync.
- Access: Signed-in user via Word Companion API.
- Still needed: Yes, but it is separate from the Story Intelligence AI pipeline.
- Writes data: Yes, can create/link books, chapters, scenes and manuscript binding records.

## 2. Current Pipeline

### Text Input

- Admin text/file input:
  - `ChapterStoryIntelligenceDryRunForm`
  - `StoryIntelligenceFullChapterTestForm`
  - `StoryIntelligenceRunQueueRequest`
- Existing chapter input:
  - `StoryIntelligenceExistingChapterQueueService`
  - `StoryIntelligenceSourceRepository`
- Status: Works for admin paste/file. Existing chapter mode only works when server-side chapter text exists.

### Paragraph Splitting

- Class: `StoryIntelligenceParagraphs`
- Used by:
  - `ChapterStoryIntelligenceDryRunService`
  - `PersistedStoryIntelligenceRunner`
  - run detail display helpers
- Output: deterministic paragraph list and numbered chapter text.
- Persistence: paragraph count is stored on `StoryIntelligenceRuns.SourceParagraphCount`.
- Status: Working and aligned with validation.

### Chapter Structure

- Prompt: `/Docs/AI/Chapter-Structure-Prompt-V2.md` when current services use prompt v2.
- Services/classes:
  - `IStoryPromptRepository`
  - `IStoryPromptBuilder`
  - `IStoryIntelligenceClient`
  - `ChapterStructureDryRunService`
  - `ChapterStoryIntelligenceDryRunService`
  - `PersistedStoryIntelligenceRunner`
- Models:
  - `ChapterStructureModel`
  - `ChapterStructureSceneBoundary`
- Validation:
  - `ChapterStructureValidator`
  - `ChapterStructureBoundaryNormaliser`
- Persistence:
  - `StoryIntelligenceChapterResults`
- Status: Working. It supports explicit numbered paragraphs, validation, JSON repair, retry, and trailing placeholder boundary normalisation.

### Scene Splitting

- Services/classes:
  - `ChapterStoryIntelligenceDryRunService`
  - `PersistedStoryIntelligenceRunner`
- Input: validated/normalised chapter boundaries plus paragraph list.
- Output: in-memory scene text blocks with temporary scene numbers and paragraph ranges.
- Persistence:
  - Scene result rows later store `TemporarySceneNumber`, `StartParagraph`, `EndParagraph`.
- Status: Working in the admin/background pipeline.

### Scene Intelligence

- Prompt: `/Docs/AI/Scene-Prompt-V2.md`
- Services/classes:
  - `IStoryIntelligenceClient`
  - `StoryIntelligenceOpenAIInfrastructure`
  - `StoryIntelligenceAiJsonParser`
  - `StoryIntelligenceJsonRepair`
  - `StorySceneValidator`
- Model:
  - `SceneIntelligenceScene`
- Persistence:
  - `StoryIntelligenceSceneResults`
- Status: Working. `gpt-5-mini` output quality is acceptable after prompt v2 and JSON repair/retry.

### Validation

- Classes:
  - `ChapterStructureValidator`
  - `StorySceneValidator`
  - `ValidationResult`
- Status: Working. Validation still exposes schema deviations instead of silently repairing semantic data.

### Persistence / Background Runner

- Worker:
  - `PersistedStoryIntelligenceWorker`
- Runner:
  - `PersistedStoryIntelligenceRunner`
- Repository:
  - `StoryIntelligenceResultRepository`
- Tables:
  - `StoryIntelligenceRuns`
  - `StoryIntelligenceChapterResults`
  - `StoryIntelligenceSceneResults`
- Status: Working for queued admin runs, progressive scene result persistence, cancellation, token/cost audit, raw output preservation.

### Resolver / Import Preview

- Class:
  - `SceneImportResolver`
- Output:
  - `StoryIntelligenceImportPreview`
- Preview categories:
  - characters
  - locations
  - assets
  - relationships
  - knowledge
  - unresolved knowledge
  - timeline clues
- Status: Useful for review. Still preview-only for all entities except scenes/metrics/notes in the commit foundation.

### Commit Foundation

- Service:
  - `StoryIntelligenceImportCommitService`
- Repository:
  - `StoryIntelligenceResultRepository.CommitImportAsync`
- View:
  - `ConfirmStoryIntelligenceImportCommit.cshtml`
- SQL:
  - `121_Phase20V_StoryIntelligenceImportCommit.sql`
- Writes:
  - scenes
  - scene source fields
  - scene purpose labels where matching existing labels
  - scene metrics where matching existing metric definitions
  - timeline/import notes
  - import commit audit row
- Status: Implemented but not yet usable for normal target chapters because of default-scene blocking.

## 3. Database Writes

### Story Intelligence Audit Writes

The system can write:

- `StoryIntelligenceRuns`
  - queued/persisted run metadata
  - user/project/book/chapter/source metadata
  - status/progress/cancellation
  - prompt/model/token/cost/audit fields
- `StoryIntelligenceChapterResults`
  - raw response JSON
  - extracted output JSON
  - parsed/normalised JSON
  - validation counts
  - tokens/duration
- `StoryIntelligenceSceneResults`
  - per-scene raw output
  - parsed JSON
  - validation counts
  - paragraph ranges
  - tokens/duration
- `StoryIntelligenceImportCommits`
  - commit audit row
  - committed user/time
  - scenes/metrics created
  - status/notes

### PlotDirector Writes Currently Possible From Story Intelligence Commit

The Phase 20V commit foundation can write:

- `Scenes`
  - `SceneNumber`
  - `SceneTitle`
  - `Summary`
  - `RelativeTimeText`
  - `ScenePurposeNotes`
  - `SceneOutcomeNotes`
  - `POVCharacterID` only when resolved to an existing character
  - `ImportSource`
  - `ImportRunID`
  - `SourceStartParagraph`
  - `SourceEndParagraph`
- `ScenePurposes`
  - existing purpose labels only when matched
- `SceneMetricValues`
  - existing metric types only
- `SceneNotes`
  - timeline/import notes only, using existing `Timeline Note` type

### Not Currently Written By Story Intelligence

Story Intelligence does not currently write:

- characters
- character aliases
- locations
- location aliases
- assets
- asset aliases
- relationships
- knowledge records
- real timeline events
- warnings
- PlotDirector scene-location links
- PlotDirector scene-character appearances
- PlotDirector scene-asset links

## 4. Commit Blocker Analysis

### What Query Counts Existing Scenes?

There are two relevant checks:

1. Confirmation preparation uses `SceneRepository.ListByChapterAsync`, which calls `dbo.Scene_ListByChapter`.
2. Commit transaction uses `dbo.StoryIntelligenceImportCommit_CountChapterScenes`.

Both currently count only active scenes. `dbo.Scene_ListByChapter` filters `s.IsArchived = 0`. `dbo.StoryIntelligenceImportCommit_CountChapterScenes` also filters `IsArchived = 0`.

### Does It Include Archived Scenes?

By implementation, no. Archived scenes should not be counted by either check.

If a commit is still blocked after a scene appears to be archived, likely causes are:

- the scene was not actually archived;
- the archive attempt failed because PlotDirector prevents a chapter from having zero active scenes;
- there is another active scene in the chapter;
- the deployed database has not received the latest Phase 20V script or application code;
- a different check outside the current commit path is being hit.

### Does It Include The Automatic Default Scene?

Yes, if the default scene is active.

The default scene is created by `TR_Chapters_CreateDefaultScene` in `093_Phase16A_BaseIdentityAndSceneFoundation.sql`. For every inserted chapter, it creates active `Scene 1` when no active scenes exist.

### Is The Default Scene Required By The Rest Of PlotDirector?

Yes, historically. The database enforces a working assumption that a chapter must contain at least one active scene:

- `dbo.Scene_Archive` throws `A chapter must contain at least one scene.` when attempting to archive the last active scene.
- `Archive_Entity` applies the same rule for scenes.

This means a user normally cannot archive the only default scene through the standard archive path.

### Can Story Intelligence Safely Replace/Remove The Default Scene?

Probably yes, but only under a strict placeholder rule.

It is not safe to remove arbitrary existing scenes. It is likely safe to replace or archive exactly one untouched default placeholder when all of these are true:

- the chapter has exactly one active scene;
- the scene title is `Scene 1`;
- scene number is `1`;
- summary is null/empty;
- purpose/outcome notes are null/empty;
- no imported source tracking exists;
- no purpose labels, metric edits, notes, characters, assets, locations, relationships, dependencies, attachments or checklist/workflow user data are attached.

If those checks fail, import should remain blocked.

### Should Import Commit Ignore Archived Scenes?

Yes. It already does in the current implementation. The bigger problem is that the default scene usually cannot become archived through normal workflows because archiving the last active scene is blocked.

### Should The Wizard Create The Chapter At Commit Time Instead Of Setup Time?

For a future author-facing Story Intelligence import wizard, yes, this is likely cleaner. The import wizard should analyse manuscript text first, then create chapters/scenes at commit time. That avoids placeholder chapters/scenes entirely.

For existing projects/chapters, commit still needs a safe placeholder replacement rule.

### Lowest-Risk Fix

The lowest-risk immediate fix is not a broad merge mode. It is:

1. Detect an untouched default placeholder scene.
2. Allow commit when the only active existing scene is that placeholder.
3. In the same transaction, create imported scenes and then archive/delete the placeholder, or update the placeholder into imported scene 1 and create the remaining scenes.
4. Keep blocking any chapter with real author-created scene content.

The safest variant is probably to update the placeholder into imported scene 1, then create imported scenes 2..N. That avoids temporarily having a zero-scene chapter and avoids bypassing the "chapter must contain one scene" invariant. It still needs strict checks that the placeholder is untouched.

## 5. Current Author-Facing Import Status

No usable author-facing Story Intelligence manuscript import flow currently exists.

What exists:

- author-facing onboarding Story Intelligence consent/progress pages;
- Word Companion manuscript structure import API;
- generic JSON import page;
- admin-only Story Intelligence analysis and commit tooling.

What does not exist:

- an author-facing page where a user can paste or upload a manuscript and run Story Intelligence;
- an author-facing confirmation page for Story Intelligence scene import;
- an author-facing background progress page tied to real AI analysis;
- author-facing commit from Story Intelligence analysis into PlotDirector scenes.

## 6. Word Companion Onboarding / Import Status

### Onboarding UI

- Route: `/onboarding`
- Controller: `OnboardingController`
- View: `Views/Onboarding/Index.cshtml`
- Purpose: account/project/book setup plus Word Companion-assisted manuscript scan/review/build.
- Status: Partially implemented.

The onboarding scan flow uses:

- SignalR hub: `WordCompanionFollowHub`
- Services/repositories:
  - `OnboardingService`
  - `OnboardingRepository`
  - `OnboardingBuildRepository`
  - `ManuscriptScanPreviewStore`
- Pages:
  - `ScanReview.cshtml`
  - `BuildComplete.cshtml`

It can scan/review and then build approved structure. Earlier failures in this thread indicate the build path has had issues around disconnected companion state and project structure creation.

### Word Companion API

- Controller: `WordCompanionController`
- Repository: `WordCompanionRepository`
- Relevant routes:
  - `POST /word-companion/manuscript/analyse`
  - `POST /word-companion/manuscript/import`
  - `POST /word-companion/books/{bookId}/chapters`
  - `POST /word-companion/books/{bookId}/chapters/{chapterId}/scenes`
  - `POST /word-companion/books/{bookId}/sync-manuscript`

This API can create/link books, chapters, scenes and manuscript document bindings. It overlaps with Story Intelligence at the structural import layer, but it does not run the AI Story Intelligence pipeline.

### Overlap With Story Intelligence

Both systems can produce or consume manuscript structure:

- Word Companion focuses on Word document structure and sync.
- Story Intelligence focuses on AI-observed scene boundaries and scene intelligence.

They are not yet unified. There is no single author-facing route that says: "submit manuscript text/document -> run Story Intelligence -> review -> create scenes".

## 7. Recommended Next Implementation Phase

Recommended next phase: **Phase 20Y - Commit Into Empty/Default Chapters**.

Do this before building a new author-facing wizard.

Reason:

- The AI pipeline already works.
- The commit foundation exists.
- The immediate blocker prevents even admin validation of the first real bridge.
- A small, strict placeholder replacement fix will make the existing commit path testable.
- Building a new author-facing wizard before commit works would add another layer over a blocked core operation.

### Specific Recommendation

Implement the smallest safe fix:

1. Add a repository check for "chapter contains only untouched default placeholder scene".
2. Permit Story Intelligence commit when the chapter has either:
   - zero active scenes; or
   - exactly one untouched default placeholder scene.
3. During commit:
   - update the placeholder into imported scene 1 and source-track it; then
   - create imported scenes 2..N; or
   - create imported scenes first and archive the placeholder only after more scenes exist.
4. Continue to block any chapter with real existing scene content.
5. Keep this admin-only.
6. Do not import characters, locations, assets, relationships, knowledge or real timeline events.

After that, build the author-facing import wizard as the next separate phase.

### Shortest Route To A User-Usable Flow

The shortest practical route is:

1. Fix commit into default-placeholder chapters.
2. Add an author-facing "Import Manuscript With Story Intelligence" page that accepts pasted text or `.txt` upload.
3. Let the user choose project/book and either existing target chapter or "create chapter at commit".
4. Queue the existing persisted Story Intelligence runner.
5. Show progress using the existing run data.
6. Show a simplified confirmation page using the same commit service.
7. Commit scenes only.

Longer-term Word document extraction can come later. It should not block the first usable paste/upload path.

## 8. Proposed Acceptance Criteria For Next Phase

Phase 20Y should be narrowly scoped.

Acceptance criteria:

- Existing Story Intelligence commit remains admin-only.
- A chapter with zero active scenes can still be committed.
- A chapter with exactly one untouched default placeholder scene can be committed.
- A chapter with any real author-created scene content remains blocked.
- Placeholder detection checks scene title, scene number, summary, notes/source tracking and attached scene data conservatively.
- Commit updates/replaces or removes the placeholder transactionally.
- A committed run still cannot be committed twice.
- Characters, locations, assets, relationships, knowledge and real timeline events remain preview-only.
- Existing prompts and model settings are unchanged.
- Existing resolver rules are unchanged.
- New/updated SQL is in a new migration only.
- `dotnet build` passes.
- A live admin commit test proves scenes are created from one completed Story Intelligence run into a chapter that previously contained only the default placeholder.

## Final State

Story Intelligence is technically promising but operationally incomplete. The system has enough proven pieces to move toward a real user import, but the immediate need is to make the first commit path work safely against PlotDirector's existing "every chapter has one scene" invariant.

Do not add more admin experiment pages until the default-scene commit blocker is resolved.