From f4e6af844bd5d44636b231bc99b1919e995613e6 Mon Sep 17 00:00:00 2001 From: Nick Beckley Date: Sat, 11 Jul 2026 19:34:00 +0000 Subject: [PATCH] docs: add phase 21 story intelligence visualisation docs --- ...Story_Intelligence_Visualisation_Design.md | 0 ...Story_Intelligence_Experience_Design_v2.md | 514 +++++++++++++++ ...-Story-Intelligence-Visualisation-Audit.md | 596 ++++++++++++++++++ 3 files changed, 1110 insertions(+) rename PlotLine/Docs/{Features => Archive}/PlotDirector_Story_Intelligence_Visualisation_Design.md (100%) create mode 100644 PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Experience_Design_v2.md create mode 100644 docs/phases/Phase-21A-Story-Intelligence-Visualisation-Audit.md diff --git a/PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Visualisation_Design.md b/PlotLine/Docs/Archive/PlotDirector_Story_Intelligence_Visualisation_Design.md similarity index 100% rename from PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Visualisation_Design.md rename to PlotLine/Docs/Archive/PlotDirector_Story_Intelligence_Visualisation_Design.md diff --git a/PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Experience_Design_v2.md b/PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Experience_Design_v2.md new file mode 100644 index 0000000..b0bb136 --- /dev/null +++ b/PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Experience_Design_v2.md @@ -0,0 +1,514 @@ +# PlotDirector Story Intelligence Experience + +**Version:** Draft 2\ +**Status:** Phase 21 Design Vision\ +**Priority:** High + +------------------------------------------------------------------------ + +# Purpose + +Replace the traditional Story Intelligence progress page with a living +visual experience that allows authors to watch PlotDirector build an +understanding of their manuscript. + +This is **not** simply a better progress bar. + +It is intended to become one of PlotDirector's defining features, giving +authors confidence that the AI genuinely understands their story while +analysis is taking place. + +------------------------------------------------------------------------ + +# Design Philosophy + +## Story before structure + +The Story Intelligence Experience is **not** intended to display a +complete technical graph of every entity in the manuscript. + +Its purpose is to visualise PlotDirector's current understanding of the +story in a way that feels alive, engaging and emotionally recognisable. + +The screen should always answer: + +> **"What story is PlotDirector understanding right now?"** + +rather than: + +> **"What data currently exists?"** + +------------------------------------------------------------------------ + +## Narrative focus + +The display should behave like a camera following the narrative. + +As each scene completes, the visual emphasis naturally shifts towards +the people, places, objects and relationships that matter in that part +of the story. + +Everything should not receive equal importance. + +The visualisation should constantly rebalance around the current +narrative focus. + +------------------------------------------------------------------------ + +## Current understanding only + +The visualisation always represents PlotDirector's **best current +understanding**. + +It is **not** intended to preserve historical graph states. + +As new scenes are analysed, discoveries simply update the current +picture. + +When Story Intelligence completes and the author approves the imported +data, the visualisation naturally transitions to using canonical +PlotDirector data. + +------------------------------------------------------------------------ + +## Demonstrate understanding, not AI + +The purpose of the experience is not to impress users with AI +processing. + +It is to demonstrate understanding. + +The author should finish watching believing: + +> PlotDirector genuinely understands my novel. + +------------------------------------------------------------------------ + +# Overall User Flow + +Welcome + +↓ + +Project + +↓ + +Book + +↓ + +Word Companion + +↓ + +Scan manuscript + +↓ + +Review detected chapters + +↓ + +Start Story Intelligence + +At this point the wizard ends. + +The user chooses one of two experiences. + +------------------------------------------------------------------------ + +# Experience One -- Continue in Background + +Ideal for users who simply wish to import. + +Analysis continues entirely server-side. + +Closing Word or the browser never interrupts analysis. + +Completion provides: + +- Email notification +- In-app notification +- Resume from the Projects page +- Review Imported Scenes + +------------------------------------------------------------------------ + +# Experience Two -- Watch Story Intelligence + +A dedicated full-screen experience. + +Minimal chrome. + +No wizard. + +No forms. + +No distractions. + +Think: + +- Mission Control +- A modern strategy game +- Apple-quality animation +- Calm, cinematic motion + +This is the visual representation of PlotDirector reading and +understanding the author's story. + +------------------------------------------------------------------------ + +# Screen Layout + +## Header + +- PlotDirector +- Story Intelligence +- Current Project +- Current Book +- Analysis Stage +- Continue in Background + +------------------------------------------------------------------------ + +## Left Panel + +Administrative information. + +Contains: + +- Overall progress +- Percentage complete +- Estimated remaining time +- Chapters analysed +- Scenes analysed +- Current stage +- Latest discoveries +- Live activity feed + +This area should feel informative rather than technical. + +------------------------------------------------------------------------ + +## Centre + +The Living Story View. + +This is the heart of the experience. + +It should not attempt to display the whole book equally. + +Instead it presents the current centre of gravity of the narrative. + +The display should naturally revolve around: + +- Current POV character +- Important scene characters +- Active location +- Story-significant assets +- Current relationships +- Active knowledge threads + +------------------------------------------------------------------------ + +## Bottom Centre + +Recent scenes. + +The current scene occupies the centre. + +As scenes complete they gently move left. + +New scenes appear on the right. + +This should feel like the story is progressing rather than a queue being +processed. + +Beneath this is the timeline, gradually filling with dates and important +events. + +------------------------------------------------------------------------ + +## Right Panel + +Higher-level understanding. + +Contains: + +- Recent relationship discoveries +- Knowledge threads +- Story observations +- Interesting narrative insights + +These should feel insightful rather than diagnostic. + +------------------------------------------------------------------------ + +# Living Story View + +The visualisation should feel like a camera following the story. + +Characters, locations and assets should dominate the display using +imagery rather than abstract icons. + +The scene currently being analysed determines the focus. + +As new discoveries occur: + +- Characters appear. +- Locations appear. +- Assets appear. +- Relationships grow. +- Knowledge threads emerge. +- Timeline events appear. + +The graph gently reorganises itself. + +Nothing should feel abrupt. + +------------------------------------------------------------------------ + +# Visual Hierarchy + +Priority should naturally be: + +1. POV character +2. Main characters in the current scene +3. Current location +4. Important story assets +5. Active relationships +6. Knowledge threads +7. Less significant supporting entities + +Minor characters may remain visible but should naturally appear smaller +and less prominent. + +The display should never become visually cluttered. + +------------------------------------------------------------------------ + +# Imagery First + +Portraits, locations and assets are fundamental to the emotional impact +of the experience. + +Generic illustrated portraits and representative imagery are entirely +acceptable during Story Intelligence. + +The author may later replace them with custom images. + +The visualisation should avoid reducing important entities to anonymous +circles wherever imagery is available. + +People connect emotionally with faces, places and objects. + +------------------------------------------------------------------------ + +# Character Portraits + +Initially use a curated built-in library. + +Approximately 300--500 illustrations. + +Categorised by: + +- Age +- Gender +- Hair colour +- General appearance + +These are visual identifiers, not canonical artwork. + +------------------------------------------------------------------------ + +# Location Images + +Representative imagery chosen from a curated library. + +Examples: + +- Detached house +- Victorian home +- Farm +- School +- Church +- Canal +- Flat +- Police station + +The purpose is recognition rather than perfect accuracy. + +------------------------------------------------------------------------ + +# Asset Images + +Representative images for common story assets. + +Examples: + +- Car +- Letter +- Passport +- Suitcase +- Notebook +- Jewellery +- Backpack +- Weapon + +Again, these are visual identifiers only. + +------------------------------------------------------------------------ + +# Live Discovery Feed + +Examples: + +- Character discovered +- Location discovered +- Asset discovered +- Relationship detected +- Timeline event identified +- Knowledge thread opened + +Older discoveries gently fade as new discoveries arrive. + +------------------------------------------------------------------------ + +# Story Observations + +These should feel intelligent. + +Examples: + +- Rosie has become central to the story. +- Beth now appears in seven scenes. +- Several names may refer to the same person. +- A mystery thread has emerged. + +Avoid technical language. + +------------------------------------------------------------------------ + +# Current Scene + +Display: + +- Scene number +- Scene title (if available) +- Brief summary +- Current progress + +When a scene completes it joins the timeline. + +------------------------------------------------------------------------ + +# Timeline + +A growing ribbon across the bottom. + +Scenes move naturally from right to left. + +Dates and events accumulate. + +Later this same component becomes PlotDirector's permanent timeline. + +------------------------------------------------------------------------ + +# Animation Philosophy + +Every animation should communicate meaning. + +Examples: + +- Character appears → newly discovered. +- Relationship line grows → connection identified. +- Character grows → increased narrative importance. +- Timeline moves → scene completed. +- Knowledge thread appears → mystery recognised. + +Avoid decorative animation. + +Every movement should reinforce understanding. + +------------------------------------------------------------------------ + +# Current-State Model + +The visualisation always displays the current accumulated understanding. + +There is no requirement to replay previous graph states. + +If understanding changes, the visualisation simply updates. + +The browser always shows the latest truth. + +------------------------------------------------------------------------ + +# Technology Direction + +Current preferred architecture: + +- ASP.NET Core +- SQL Server +- SignalR +- Existing Story Intelligence results +- Interactive graph renderer +- HTML/CSS dashboard +- Smooth animation + +The renderer itself is an implementation detail. + +The user experience described in this document is the priority. + +------------------------------------------------------------------------ + +# Completion + +When Story Intelligence completes: + +Story Intelligence Complete + +Your manuscript is ready for review. + +Buttons: + +- Review Imported Scenes +- Explore Story Map + +------------------------------------------------------------------------ + +# Long-Term Vision + +The technology developed here becomes the foundation for: + +- Story Map +- Character Explorer +- Relationship Explorer +- Location Explorer +- Asset Explorer +- Timeline Explorer +- Knowledge Explorer +- Continuity Explorer + +The Story Intelligence experience is cinematic. + +The permanent Story Map is analytical. + +Both share the same underlying story data while presenting it +differently. + +------------------------------------------------------------------------ + +# Ultimate Design Goal + +The author should never feel they are watching a progress bar. + +They should feel they are watching their world come alive. + +By the time analysis finishes they should have greater confidence that +PlotDirector genuinely understands the novel they have spent months or +years creating. + +The visualisation should become one of PlotDirector's signature features +and something authors remember, recommend and enjoy watching. diff --git a/docs/phases/Phase-21A-Story-Intelligence-Visualisation-Audit.md b/docs/phases/Phase-21A-Story-Intelligence-Visualisation-Audit.md new file mode 100644 index 0000000..204e35b --- /dev/null +++ b/docs/phases/Phase-21A-Story-Intelligence-Visualisation-Audit.md @@ -0,0 +1,596 @@ +# Phase 21A - Story Intelligence Visualisation Architecture Audit + +## 1. Executive summary + +PlotDirector already has most of the server-side ingredients needed for a durable Story Intelligence visualisation: ASP.NET Core MVC, SQL Server persistence, Dapper repositories, SignalR, a persisted background worker, canonical story entities, and a current progress page that survives browser and Word closure. + +The current live experience is not yet graph-ready. SignalR messages are progress notifications, not durable events. Provisional discoveries are mostly reconstructed from saved scene-analysis JSON and transient in-memory batch decisions. Scene/chapter AI results have stable SQL identifiers, but provisional characters, locations, assets, relationships, knowledge items, confidence changes, merges and rejections do not have durable discovery records or ordered event history. + +Sigma.js, Graphology and SignalR remain suitable for Phase 21 if SQL Server remains authoritative and SignalR is only a transport. The biggest prerequisite for Phase 21B is a durable visualisation snapshot/event model that can be rebuilt after refresh, replayed after reconnect, and joined to both provisional and canonical PlotDirector IDs. + +No code, package, database, deployment or workflow changes were made for this audit. + +## 2. Current Story Intelligence workflow + +```mermaid +flowchart TD + A["Author starts onboarding import"] --> B["Website-led onboarding wizard"] + B --> C["Project and book selected or created"] + C --> D["Word Companion registers over SignalR"] + D --> E["Website asks Companion to scan document"] + E --> F["Companion reports scan preview"] + F --> G["Author reviews chapters, scenes, character candidates"] + G --> H["Onboarding builds approved chapter shells"] + H --> I["One StoryIntelligenceRun queued per approved chapter"] + I --> J["PersistedStoryIntelligenceWorker claims pending runs"] + J --> K["Chapter Structure prompt detects scene boundaries"] + K --> L["Scene Intelligence prompt analyses each suggested scene"] + L --> M["Chapter and scene results persisted in SQL"] + M --> N["SignalR progress updates current page"] + N --> O["Author reviews/imports scenes"] + O --> P["Characters, locations, assets, relationships, knowledge reviewed in sequence"] + P --> Q["Canonical PlotDirector records created or linked"] +``` + +Important components: + +| Component | Path | Responsibility | Current state | +|---|---|---|---| +| `OnboardingController` | `PlotLine/Controllers/OnboardingController.cs` | Onboarding routes, scan review, Story Intelligence start/progress/review/import stages. | Current. Central workflow controller. | +| `WordCompanionFollowHub` | `PlotLine/Hubs/WordCompanionFollowHub.cs` | Companion presence, scan command, scan progress/completion, build marker updates. | Current. User-group scoped. | +| `ManuscriptScanPreviewStore` | `PlotLine/Services/ManuscriptScanPreviewStore.cs` | In-memory scan preview/review/build state. | Current but non-durable. Important recovery weakness. | +| `OnboardingService` | `PlotLine/Services/OnboardingService.cs` | Wizard state, scan review validation, approved chapter build. | Current. Creates chapter shells before AI run. | +| `OnboardingStoryIntelligenceService` | `PlotLine/Services/OnboardingStoryIntelligenceService.cs` | Queues per-chapter runs, builds progress/review models, orchestrates staged imports. | Current. Depends on in-memory batch store. | +| `OnboardingStoryIntelligenceBatchStore` | `PlotLine/Services/OnboardingStoryIntelligenceService.cs` | Keeps batch-to-run mapping and review decisions in a `ConcurrentDictionary`. | Current but volatile. Key gap. | +| `PersistedStoryIntelligenceWorker` | `PlotLine/Services/PersistedStoryIntelligenceWorker.cs` | Background service polling every 3 seconds for pending persisted runs. | Current. Browser-independent. | +| `PersistedStoryIntelligenceRunner` | `PlotLine/Services/PersistedStoryIntelligenceRunner.cs` | Executes chapter and scene prompts, validates, saves results, publishes progress. | Current. Per-run processing. | +| `StoryIntelligenceResultRepository` | `PlotLine/Data/StoryIntelligenceResultRepository.cs` | SQL access for runs, chapter/scene results, import commits. | Current. Core persistence path. | +| `StoryIntelligenceProgressNotifier` | `PlotLine/Services/StoryIntelligenceProgressNotifier.cs` | Broadcasts run progress to SignalR user group. | Current. Ephemeral messages. | +| `StoryIntelligenceImportCommitService` | `PlotLine/Services/StoryIntelligenceImportCommitService.cs` | Converts completed scene results into canonical chapters/scenes/metrics. | Current. Transaction is inside SQL proc. | +| Entity import services | `StoryIntelligenceCharacterImportService.cs`, `StoryIntelligenceLocationImportService.cs`, `StoryIntelligenceAssetImportService.cs`, `StoryIntelligenceRelationshipImportService.cs`, `StoryIntelligenceKnowledgeImportService.cs` | Build review candidates from committed scenes and saved AI JSON; create/link/ignore canonical entities. | Current. Decisions are partly transient. | + +The older `StoryIntelligenceJobs` model still exists through `IStoryIntelligenceRepository`, `StoryIntelligenceService`, and legacy progress support, but the onboarding analysis flow now primarily uses `StoryIntelligenceRuns`. + +## 3. Current background-processing architecture + +`Program.cs` registers `StoryIntelligenceWorker`, `PersistedStoryIntelligenceWorker`, `EmailQueueWorker`, and `WordCompanionPresenceMonitor` as hosted services. The persisted runner claims SQL-backed pending runs via `StoryIntelligenceRun_ClaimNextPending` using `UPDLOCK, READPAST`, then processes a single run. + +Reliability strengths: + +- Processing is independent of the browser and Word once runs are queued. +- Runs persist status, stage, message, token totals, scene counts, failure stage, error text, cancellation timestamps, start/completion timestamps and source text. +- Cancellation is stored with `CancellationRequestedUtc` and checked by the runner. +- Failed chapter/scene JSON parse attempts can be persisted as result rows. + +Weaknesses: + +- No durable visualisation event stream exists. +- No explicit retry count or backoff model was found for failed Story Intelligence runs. +- No stale-running recovery was found for `StoryIntelligenceRuns` if the app dies after a run is claimed and marked `Running`. +- The active batch mapping from onboarding batch ID to run IDs is in memory only. +- Multiple app instances would need careful review: SQL claim locking helps workers, but in-memory batch state and SignalR user groups are process-local. + +## 4. Current provisional discovery model + +| Discovery type | Current storage before approval | Stable provisional ID? | Notes | +|---|---|---:|---| +| Chapters | Scan preview/review in memory; approved chapter shells become canonical before analysis. | Temporary scan keys only. | Chapter shells are canonical before scene analysis starts. | +| Scenes | `StoryIntelligenceSceneResults` with `SceneResultID` and temporary scene number. | Yes for scene results. | Canonical scenes created during scene commit. | +| Characters | Parsed scene JSON and review candidates derived by service. | Derived key only. | Character decisions are in in-memory batch. | +| Character aliases | Derived/imported during character review. | No durable provisional alias ID. | Alias decisions not durable. | +| Locations | Parsed scene JSON and review candidates. | Derived key only. | Uses normalisation rules, no provisional table. | +| Assets | Parsed scene JSON and review candidates. | Derived key only. | Ownership/custody inferred during import service. | +| Relationships | Parsed scene JSON and review candidates. | Derived key only. | No persisted suspected/confirmed relationship lifecycle. | +| Timeline clues | Parsed scene JSON/import notes. | No. | Imported indirectly as scene notes/timeline data. | +| Knowledge | Parsed scene JSON and review candidates. | Derived key only. | Knowledge selections/decisions live in batch. | +| Confidence | Stored inside parsed AI JSON and some review previews. | Attached to JSON only. | No confidence history. | +| Evidence/excerpts | Source text and raw/parsed prompt output are retained. | Scene result IDs help. | User-facing evidence is not normalised. | +| Rejections | In-memory batch decisions while process lives. | No durable rejection ID. | Lost on application restart unless inferred from canonical state. | +| Duplicate/merge candidates | Derived at review time. | No. | Alias/merge operations do not retain durable provisional lineage. | + +The current system can answer "what chapter/scene is being analysed now?" and "what scene results exist?" reasonably well. It cannot reliably answer "in what order were characters discovered?", "when did confidence change?", "which provisional identities merged?", or "what did a reconnecting graph client miss?" without adding durable event/discovery storage. + +## 5. Canonical data model relevant to the graph + +Major graph-ready canonical records: + +| Record | Tables / procedures | Models / repositories | Suitability | +|---|---|---|---| +| Projects, books, chapters, scenes | `Projects`, `Books`, `Chapters`, `Scenes`; `Project_*`, `Book_*`, `Chapter_*`, `Scene_*` | `ProjectRepository`, `BookRepository`, `ChapterRepository`, `SceneRepository` | Strong node base. Ordering by book/chapter/scene is significant. Soft archive applies. | +| Characters and appearances | `Characters`, `CharacterAliases`, `SceneCharacters` | `CharacterRepository` | Strong character nodes and scene edges. Alias graph needs interpretation. | +| Relationships | `CharacterRelationships`, `RelationshipEvents`, relationship categories/types | `CharacterRepository`, `RelationshipMapService` | Already feeds Cytoscape relationship map. Good edge source. | +| Character knowledge | `CharacterKnowledge`, knowledge states | `CharacterRepository` | Useful for knowledge nodes/edges, but evidence lineage limited. | +| Locations | `Locations`, `LocationAliases`, `LocationRelationships`, scene-location/cross refs | `LocationRepository` | Good location nodes; hierarchy and relationships need graph mapping. | +| Assets | `StoryAssets`, `AssetAliases`, `AssetEvents`, `AssetCustodyEvents`, dependencies | `AssetRepository` | Good asset nodes and custody/ownership edges. | +| Timeline/scene metrics | Timeline procedures, `SceneMetric*`, scene date/time fields | `TimelineRepository`, metric repositories | Good timeline layering, not a standalone graph yet. | +| Plot lines/threads | `PlotLines`, plot thread procedures/events | `PlotRepository` | Good thread nodes/edges when events are included. | +| Continuity warnings | `ContinuityWarnings`, acknowledgements | `WarningRepository`, `ContinuityWarningAcknowledgementRepository` | Useful issue nodes attached to scenes/entities. | +| Project activity | `ProjectActivity` | `ProjectActivityRepository` | Audit/activity feed, not detailed enough for graph replay. | + +Existing canonical data is suitable for permanent Story Map snapshots, but the graph service should translate domain records into graph nodes/edges rather than exposing database DTOs directly. + +## 6. Existing SignalR infrastructure + +SignalR is registered in `Program.cs` with `MaximumReceiveMessageSize = 1 MB`. + +Hubs: + +- `/hubs/word-companion-follow` -> `WordCompanionFollowHub`, `[Authorize]` +- `/hubs/story-intelligence` -> `StoryIntelligenceHub`, `[Authorize]` + +Group model: + +- Word Companion uses `word-companion-presence:{userId}`. +- Story Intelligence uses `story-intelligence:{userId}`. + +Client usage: + +- `PlotLine/Views/Shared/_Layout.cshtml` includes SignalR from CDN for authenticated users. +- `PlotLine/Views/WordCompanionHost/Index.cshtml` includes SignalR from CDN. +- `PlotLine/wwwroot/js/story-intelligence-progress.js` connects to `/hubs/story-intelligence`, uses automatic reconnect, invokes watch methods after connect/reconnect, and handles `StoryIntelligenceProgressChanged` plus `StoryIntelligenceRunProgressChanged`. + +Assessment: + +- Suitable for progress updates and first visualisation transport. +- Current group scope is user-wide, not job/book-specific. +- Watch methods validate run ownership by `UserID`, but future graph groups should validate project/book/job access before joining. +- Messages are ephemeral. Missed recovery currently happens by reloading current run state, not by replaying events. +- No scale-out/backplane or sticky-session configuration was found. Deployment docs only cover upload storage; nginx/WebSocket settings were not present in the repo. + +## 7. Existing frontend architecture + +PlotLine is mostly server-rendered Razor with plain JavaScript and CSS: + +- No PlotLine `package.json` was found. +- `PlotDirector.WordCompanion/package.json` exists only for the Office add-in manifest tooling. +- `bundleconfig.json` minifies selected CSS/JS files. +- Bootstrap is served from `wwwroot/lib/bootstrap`; the CSS appears to be Bootstrap 5. +- jQuery and validation libraries exist under `wwwroot/lib`. +- Theme support exists through `data-theme` / `data-bs-theme`, `site.css`, `plotline-theme.css`, and localStorage. +- Current JS is directly served/minified; no TypeScript build or module bundler is present for the MVC app. +- Existing components include Bootstrap modals, popovers, dropdowns, accordions, details-based drawers, a scene inspector drawer, and rich CSS theme audit pages. +- `prefers-reduced-motion` appears in CSS. Relationship Map disables layout animation. + +Graph-related precedent: + +- `RelationshipMap/Index.cshtml` loads Cytoscape from CDN and renders an interactive character relationship graph with pan/zoom, layout, edge selection, detail panel, resize handling and dark-theme styles. +- No Sigma.js or Graphology usage was found. + +Sigma.js/Graphology integration options: + +- Best long-term option: add an MVC frontend package/build pipeline and bundle Sigma/Graphology/worker modules with cache busting. +- Short-term prototype option: load from CDN like Cytoscape, but that conflicts with a durable production-grade graph engine and CSP/offline control. +- TypeScript is not currently established in PlotLine; Phase 21B should decide whether to introduce it with a minimal build step. + +## 8. Relevant pages and navigation + +Relevant routes: + +- `/onboarding` -> onboarding wizard and Word scan controls. +- `/onboarding/scan-review` -> chapter/scene scan review. +- `/onboarding/story-intelligence` -> pre-analysis overview. +- `/onboarding/story-intelligence/progress?batchId=...` -> current progress page. +- `/onboarding/story-intelligence/review?batchId=...` -> scene review/import. +- `/onboarding/story-intelligence/characters|locations|assets|relationships|knowledge` -> staged entity review. +- `/onboarding/story-intelligence/book/{bookId}/continue` -> resume from book. +- `/RelationshipMap?projectId=...` -> current graph visualisation. +- `/Projects`, `/Books/Details`, `/Chapters/Details`, `/Scenes/Edit`, `/Characters`, `/Locations`, `/StoryAssets`, `/Timeline`, `/PlotThreads`, `/ContinuityExplorer`, `/Warnings`. + +The current layout can host a graph, but the full-screen visualisation should probably use a minimal-chrome shell. No established full-screen graph canvas shell was found. Relationship Map has a graph workspace plus details panel that can be reused conceptually. + +## 9. Security and authorisation assessment + +Current strengths: + +- Controllers are generally `[Authorize]`. +- `ProjectAccessFilter` globally checks project/entity access based on controller parameters and model properties. +- Project collaborators and owners are handled through `ProjectAccess_*` stored procedures. +- Story Intelligence hub methods check authenticated user ID; `WatchStoryIntelligenceRun` checks `run.UserID == userId`. +- Razor output is encoded by default. + +Risks/gaps for the visualisation: + +- Future graph APIs cannot rely only on `run.UserID`; collaborative project access and book-level access must be checked. +- User-group SignalR broadcasts could deliver all a user's Story Intelligence progress to every tab. A graph should use job/book groups after explicit authorisation. +- Manuscript excerpts in snapshots/evidence drawers need strict project/book/run ownership checks. +- CDN scripts complicate future CSP. +- Uploaded portrait/cover URLs should be served through existing upload path rules and not blindly embedded from AI output. + +Recommended graph access: + +- Snapshot endpoint: require authenticated user and `ProjectAccess_UserCanAccessEntity` for the book/run/project. +- Hub group join: `WatchStoryIntelligenceGraph(runId, afterSequence)` validates the same access, then joins `story-intelligence-run:{runId}`. +- Do not expose raw prompt internals or private reasoning; expose short manuscript evidence and structured user-facing justifications. + +## 10. Stable identity assessment + +Canonical IDs are stable (`project-{id}`, `book-{id}`, `chapter-{id}`, `scene-{id}`, `character-{id}`, `location-{id}`, `asset-{id}`, `relationship-{id}`, `knowledge-{id}`). + +Current provisional stability: + +- `StoryIntelligenceRunID`, `ChapterResultID`, and `SceneResultID` are stable. +- Temporary scan keys are stable only within the scan preview memory store. +- Review candidate keys for characters/locations/assets/relationships/knowledge are derived and not durable. +- Merges/aliases do not have a durable provisional-to-canonical lineage. + +Recommended convention, pending future storage: + +- `run-{runId}` +- `chapter-result-{chapterResultId}` +- `scene-result-{sceneResultId}` +- `provisional-character-{discoveryId}` -> merge/replace to `character-{characterId}` +- `provisional-location-{discoveryId}` -> `location-{locationId}` +- `provisional-asset-{discoveryId}` -> `asset-{assetId}` +- `provisional-relationship-{discoveryId}` -> `relationship-{characterRelationshipId}` +- `provisional-knowledge-{discoveryId}` -> `knowledge-{characterKnowledgeId}` + +The client will need an explicit `IdentityMerged` or `NodePromoted` event to preserve visual continuity when provisional nodes become canonical. + +## 11. Evidence and confidence assessment + +Stored today: + +- Source text on `StoryIntelligenceRuns`. +- Raw response JSON, assistant output JSON and parsed JSON for chapter/scene results. +- Validation errors/warnings counts. +- Confidence values inside parsed scene JSON for many extracted facts. +- Source scene/chapter through run, chapter result and scene result linkage. +- Canonical import commit metadata and warnings. + +Not stored in a graph-ready way: + +- Normalised evidence records per claim. +- Short source excerpts per discovery. +- Confidence history. +- Alternative candidates. +- Confirmation or contradictory evidence across later scenes. +- Manual override/rejection reason as durable data. +- User-facing reasoning summaries separate from raw AI output. + +The future evidence drawer can be supported if Phase 21 adds durable discovery/evidence rows derived from parsed scene results, not from private model reasoning. + +## 12. Event-stream assessment + +Existing mechanisms: + +- `ProjectActivity` and `ProjectAudit_Record`: useful general audit trail, not fine-grained Story Intelligence replay. +- SignalR progress events: ephemeral. +- Import commit rows: durable finalisation record, not incremental discovery history. +- SQL result tables: durable snapshots of prompt outputs, not ordered graph events. + +Needed event model: + +- Dedicated `StoryIntelligenceVisualisationEvents` table. +- Monotonic `SequenceNumber` per run/job. +- Event type, timestamp, project/book/run/scene/discovery references. +- JSON payload with a version. +- Idempotency key. +- Retention rules. +- Optional processed/published metadata if an outbox publisher is introduced. + +Candidate events: + +`AnalysisStarted`, `StageChanged`, `ChapterAnalysisStarted`, `SceneAnalysisStarted`, `SceneAnalysisCompleted`, `NodeDiscovered`, `NodeUpdated`, `EdgeDiscovered`, `EdgeUpdated`, `IdentityMerged`, `RelationshipSuspected`, `RelationshipConfirmed`, `TimelineEventDiscovered`, `KnowledgeThreadOpened`, `KnowledgeThreadResolved`, `ObservationCreated`, `AnalysisCompleted`, `AnalysisFailed`, `NodePromotedToCanonical`, `DiscoveryRejected`. + +## 13. Performance and scale assessment + +Estimated graph sizes: + +| Scope | Scenes | Characters | Locations | Assets | Relationships | Knowledge/events | Nodes | Edges | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| Small test | 5-15 | 5-12 | 3-8 | 3-10 | 5-20 | 10-40 | 30-90 | 50-150 | +| Normal novel | 60-120 | 25-80 | 30-120 | 40-180 | 80-250 | 150-600 | 300-1,100 | 700-3,000 | +| Large novel | 150-250 | 80-180 | 100-250 | 150-400 | 250-700 | 500-1,500 | 1,000-3,000 | 3,000-10,000 | +| Trilogy | 300-600 | 150-350 | 200-600 | 300-1,000 | 700-2,000 | 1,500-5,000 | 3,000-9,000 | 10,000-35,000 | +| Large multi-book project | 800+ | 400+ | 800+ | 1,500+ | 3,000+ | 8,000+ | 12,000+ | 50,000+ | + +Primary risks: + +- JSON payload size for snapshots and replay. +- Browser layout cost for dense relationship/evidence edges. +- Portrait textures and labels. +- Replaying thousands of small events after reconnect. +- SQL DTO construction from many canonical tables. + +Recommendation: Phase 21B should target one book / one active analysis batch, with filters and level-of-detail from the start. Project-wide exploration should come later. + +## 14. Reconnection and recovery assessment + +Current recovery: + +- Progress JS reconnects and invokes watch methods to get current run state. +- Browser refresh can reload progress from SQL if the batch still exists. +- Closing Word does not stop queued analysis. + +Gaps: + +- In-memory scan preview and batch store do not survive application restart. +- SignalR missed events are not replayable. +- No event sequence exists. +- No graph snapshot endpoint exists. + +Required Phase 21 design: + +```mermaid +flowchart LR + A["Browser opens visualisation"] --> B["GET graph snapshot"] + B --> C["Snapshot includes lastSequence"] + C --> D["Join authorised SignalR run group"] + D --> E["Receive events with sequence numbers"] + E --> F["Apply idempotently to Graphology"] + F --> G["Reconnect"] + G --> H["Request events after last applied sequence"] + H --> F + H --> I["If too old, reload snapshot"] +``` + +## 15. Approval-to-canonical transition assessment + +Scene import: + +- `StoryIntelligenceImportCommitService` prepares completed run results. +- SQL commit creates canonical scenes/metrics and records `StoryIntelligenceImportCommits`. +- Pipeline state records scene import. +- Duplicate scene import is blocked by readiness checks. + +Entity imports: + +- Review candidates are derived from committed scenes and saved parsed JSON. +- Create/link/alias/ignore actions call existing canonical repositories. +- Pipeline state advances per book. +- Decisions and last import summaries are in the batch object, not durable review tables. + +Risks: + +- Provisional node IDs will disappear unless durable discovery IDs are introduced. +- Rejections/aliases are not reliably reconstructable after restart. +- Partial approval can change graph topology without an event record. +- Relationships can reference provisional characters unless promotion/merge ordering is explicit. + +```mermaid +flowchart TD + A["Parsed scene result"] --> B["Provisional discovery row"] + B --> C["Graph provisional node"] + C --> D{"Author decision"} + D -->|Create| E["Canonical record created"] + D -->|Link| F["Existing canonical record linked"] + D -->|Alias/Merge| G["Discovery merged into target"] + D -->|Ignore| H["Discovery rejected"] + E --> I["NodePromotedToCanonical event"] + F --> I + G --> J["IdentityMerged event"] + H --> K["DiscoveryRejected event"] +``` + +## 16. Proposed Phase 21 architecture + +```mermaid +flowchart TD + A["Story Intelligence analysis worker"] --> B["Persist discoveries and state"] + B --> C["SQL Server"] + C --> D["Ordered visualisation event table"] + D --> E["Story Intelligence event publisher"] + E --> F["SignalR hub"] + F --> G["Browser client"] + G --> H["Snapshot loader"] + G --> I["Event applier"] + H --> J["Graphology graph model"] + I --> J + J --> K["Sigma.js renderer"] + G --> L["Panels, feeds, filters, evidence drawer"] +``` + +Principles: + +- SQL Server is authoritative. +- SignalR is transport only. +- Browser never analyses manuscript. +- Graph DTOs are versioned and separate from database DTOs. +- The same graph engine supports live analysis and completed Story Map. + +## 17. Proposed server-side components + +- `StoryIntelligenceVisualisationSnapshotService`: builds snapshot for run/book/project. +- `StoryIntelligenceVisualisationEventRepository`: persists ordered events. +- `StoryIntelligenceDiscoveryRepository`: persists provisional nodes/edges/evidence. +- `StoryIntelligenceVisualisationPublisher`: publishes stored events to SignalR after commit. +- `StoryIntelligenceGraphHub`: job/book-scoped groups with explicit access validation. +- `StoryIntelligenceGraphController` or API controller: snapshot and missed-event endpoints. +- Integration points in `PersistedStoryIntelligenceRunner` and import services to write discovery/event rows. + +## 18. Proposed browser-side components + +```mermaid +flowchart LR + A["Razor page"] --> B["graph bootstrap"] + B --> C["snapshot API client"] + B --> D["SignalR client"] + C --> E["graph adapter"] + D --> E + E --> F["Graphology"] + F --> G["Sigma renderer"] + E --> H["activity feed"] + E --> I["evidence drawer"] + E --> J["filters/search"] + K["layout worker"] --> F +``` + +Suggested modules: + +- `story-graph-client` +- `story-graph-adapter` +- `story-graph-renderer` +- `story-graph-layout-worker` +- `story-graph-panels` +- `story-graph-api` + +## 19. Proposed API contracts + +Snapshot: + +```http +GET /api/story-intelligence/runs/{runId}/graph-snapshot?scope=book +``` + +Response shape: + +```json +{ + "runId": 123, + "projectId": 10, + "bookId": 20, + "lastSequence": 456, + "nodes": [], + "edges": [], + "activeStage": "SceneIntelligence", + "permissions": { "canReview": true }, + "generatedUtc": "2026-07-11T00:00:00Z" +} +``` + +Missed events: + +```http +GET /api/story-intelligence/runs/{runId}/graph-events?afterSequence=456 +``` + +## 20. Proposed SignalR message contracts + +Hub route proposal: `/hubs/story-intelligence-graph` + +Client invokes: + +- `WatchRunGraph(runId, afterSequence)` +- `LeaveRunGraph(runId)` + +Server sends: + +```json +{ + "runId": 123, + "sequence": 457, + "eventId": "01J...", + "type": "NodeDiscovered", + "occurredUtc": "2026-07-11T00:00:00Z", + "payloadVersion": 1, + "payload": {} +} +``` + +## 21. Proposed event categories + +- Analysis lifecycle +- Progress and stage +- Scene/chapter lifecycle +- Discovery node lifecycle +- Relationship/edge lifecycle +- Evidence/confidence updates +- Merge/promotion/rejection +- Review/approval +- Failure/cancellation/recovery + +## 22. Proposed implementation sequence + +1. Add durable event/discovery design and SQL script. +2. Add read-only snapshot service over existing canonical plus run-result data. +3. Add event writer around current runner progress and scene result persistence. +4. Add graph hub with authorised job groups and replay-after-sequence. +5. Build browser graph adapter with mocked renderer data from real snapshots. +6. Integrate Sigma.js/Graphology and layout worker. +7. Add live progress/discovery events. +8. Add approval/promotion events for scene and entity imports. +9. Add completed Story Map route reusing the graph engine. + +## 23. Database changes likely to be required + +Follow existing SQL script convention in `PlotLine/Sql`, where the current latest script is `134_Phase20AV_ArchivedBookProjectPermanentDeletion.sql`. + +Likely future script: + +- `135_Phase21B_StoryIntelligenceVisualisationEvents.sql` + +Likely tables: + +- `StoryIntelligenceDiscoveries` +- `StoryIntelligenceDiscoveryEvidence` +- `StoryIntelligenceDiscoveryAliases` +- `StoryIntelligenceVisualisationEvents` +- Possibly `StoryIntelligenceGraphSnapshots` if snapshot materialisation is needed later. + +Conventions to follow: + +- Idempotent `IF OBJECT_ID` / `COL_LENGTH` checks. +- `CREATE OR ALTER PROCEDURE`. +- `PK_`, `FK_`, `DF_`, `CK_`, `IX_` naming. +- User/project/book isolation columns and indexes. +- Do not alter historical scripts. + +## 24. Risks and mitigations + +| Risk | Mitigation | +|---|---| +| Volatile batch state prevents recovery | Persist batch/run mappings and review decisions or derive them from durable pipeline rows. | +| SignalR events missed | Store ordered events and replay after sequence. | +| Graph too dense | One-book scope, filters, clustering, label level-of-detail. | +| Provisional/canonical ID churn | Add discovery IDs and explicit promotion/merge events. | +| Manuscript data leakage | Snapshot/hub access checks against project/book/run. | +| Package integration churn | Decide MVC frontend build approach before adding Sigma/Graphology. | +| CDN/CSP concerns | Prefer bundled assets for production. | +| Existing progress workflow regression | Add visualisation beside current progress page first. | + +## 25. Decisions still required + +- Whether Phase 21B introduces an MVC `package.json` and bundler. +- Whether to persist batch state or replace batch IDs with durable book/run pipeline IDs. +- Exact discovery/event SQL schema. +- Whether to keep Cytoscape for Relationship Map or migrate it later to the new engine. +- Retention policy for raw source excerpts and visualisation events. +- Whether snapshots are generated on demand or materialised. + +## 26. Files inspected + +Primary files inspected: + +- `PlotLine/Program.cs` +- `PlotLine/Hubs/StoryIntelligenceHub.cs` +- `PlotLine/Hubs/WordCompanionFollowHub.cs` +- `PlotLine/Controllers/OnboardingController.cs` +- `PlotLine/Controllers/RelationshipMapController.cs` +- `PlotLine/Services/OnboardingService.cs` +- `PlotLine/Services/OnboardingStoryIntelligenceService.cs` +- `PlotLine/Services/PersistedStoryIntelligenceWorker.cs` +- `PlotLine/Services/PersistedStoryIntelligenceRunner.cs` +- `PlotLine/Services/StoryIntelligenceProgressNotifier.cs` +- `PlotLine/Services/StoryIntelligenceImportCommitService.cs` +- `PlotLine/Services/SceneImportResolver.cs` +- `PlotLine/Services/StoryIntelligenceCharacterImportService.cs` +- `PlotLine/Services/StoryIntelligenceLocationImportService.cs` +- `PlotLine/Services/ProjectAccessFilter.cs` +- `PlotLine/Services/ProjectAccessServices.cs` +- `PlotLine/Data/StoryIntelligenceResultRepository.cs` +- `PlotLine/Data/StoryIntelligenceRepository.cs` +- `PlotLine/Data/StoryIntelligencePipelineRepository.cs` +- `PlotLine/Data/Repositories.cs` +- `PlotLine/Models/StoryIntelligenceModels.cs` +- `PlotLine/Models/StoryIntelligencePersistenceModels.cs` +- `PlotLine/ViewModels/OnboardingViewModels.cs` +- `PlotLine/Views/Onboarding/StoryIntelligenceProgress.cshtml` +- `PlotLine/Views/Onboarding/StoryIntelligenceReview.cshtml` +- `PlotLine/Views/RelationshipMap/Index.cshtml` +- `PlotLine/Views/Shared/_Layout.cshtml` +- `PlotLine/wwwroot/js/story-intelligence-progress.js` +- `PlotLine/bundleconfig.json` +- `PlotDirector.WordCompanion/package.json` +- SQL scripts `001`, `004`, `005`, `006`, `007`, `020`, `033`, `039`, `042`, `046`, `088`, `093`, `111`, `114`, `115`, `117`, `121`, `123`, `124`, `127` through `134` +- `PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Visualisation_Design.md` +- `PlotLine/Docs/Architecture/DeploymentConfiguration.md` + +## 27. Recommended next step for Phase 21B + +Phase 21B should be a narrow foundation phase: + +1. Design and add durable graph discovery/event storage for one active `StoryIntelligenceRun`. +2. Add authorised snapshot and event replay endpoints. +3. Add a SignalR graph hub that joins one run group after access validation. +4. Emit lifecycle, stage, scene-start, scene-complete and basic discovery events from the existing persisted runner. +5. Do not replace the current progress page yet; add the visualisation as an optional "Watch Story Intelligence" experience backed by the same SQL state.