PlotDirector/docs/phases/Phase-21A-Story-Intelligence-Visualisation-Audit.md

Phase 21A - Story Intelligence Visualisation Architecture Audit

1. Executive summary

PlotDirector already has most of the server-side ingredients needed for a durable Story Intelligence visualisation: ASP.NET Core MVC, SQL Server persistence, Dapper repositories, SignalR, a persisted background worker, canonical story entities, and a current progress page that survives browser and Word closure.

The current live experience is not yet graph-ready. SignalR messages are progress notifications, not durable events. Provisional discoveries are mostly reconstructed from saved scene-analysis JSON and transient in-memory batch decisions. Scene/chapter AI results have stable SQL identifiers, but provisional characters, locations, assets, relationships, knowledge items, confidence changes, merges and rejections do not have durable discovery records or ordered event history.

Sigma.js, Graphology and SignalR remain suitable for Phase 21 if SQL Server remains authoritative and SignalR is only a transport. The biggest prerequisite for Phase 21B is a durable visualisation snapshot/event model that can be rebuilt after refresh, replayed after reconnect, and joined to both provisional and canonical PlotDirector IDs.

No code, package, database, deployment or workflow changes were made for this audit.

2. Current Story Intelligence workflow

flowchart TD
    A["Author starts onboarding import"] --> B["Website-led onboarding wizard"]
    B --> C["Project and book selected or created"]
    C --> D["Word Companion registers over SignalR"]
    D --> E["Website asks Companion to scan document"]
    E --> F["Companion reports scan preview"]
    F --> G["Author reviews chapters, scenes, character candidates"]
    G --> H["Onboarding builds approved chapter shells"]
    H --> I["One StoryIntelligenceRun queued per approved chapter"]
    I --> J["PersistedStoryIntelligenceWorker claims pending runs"]
    J --> K["Chapter Structure prompt detects scene boundaries"]
    K --> L["Scene Intelligence prompt analyses each suggested scene"]
    L --> M["Chapter and scene results persisted in SQL"]
    M --> N["SignalR progress updates current page"]
    N --> O["Author reviews/imports scenes"]
    O --> P["Characters, locations, assets, relationships, knowledge reviewed in sequence"]
    P --> Q["Canonical PlotDirector records created or linked"]

Important components:

Component	Path	Responsibility	Current state
OnboardingController	PlotLine/Controllers/OnboardingController.cs	Onboarding routes, scan review, Story Intelligence start/progress/review/import stages.	Current. Central workflow controller.
WordCompanionFollowHub	PlotLine/Hubs/WordCompanionFollowHub.cs	Companion presence, scan command, scan progress/completion, build marker updates.	Current. User-group scoped.
ManuscriptScanPreviewStore	PlotLine/Services/ManuscriptScanPreviewStore.cs	In-memory scan preview/review/build state.	Current but non-durable. Important recovery weakness.
OnboardingService	PlotLine/Services/OnboardingService.cs	Wizard state, scan review validation, approved chapter build.	Current. Creates chapter shells before AI run.
OnboardingStoryIntelligenceService	PlotLine/Services/OnboardingStoryIntelligenceService.cs	Queues per-chapter runs, builds progress/review models, orchestrates staged imports.	Current. Depends on in-memory batch store.
OnboardingStoryIntelligenceBatchStore	PlotLine/Services/OnboardingStoryIntelligenceService.cs	Keeps batch-to-run mapping and review decisions in a ConcurrentDictionary.	Current but volatile. Key gap.
PersistedStoryIntelligenceWorker	PlotLine/Services/PersistedStoryIntelligenceWorker.cs	Background service polling every 3 seconds for pending persisted runs.	Current. Browser-independent.
PersistedStoryIntelligenceRunner	PlotLine/Services/PersistedStoryIntelligenceRunner.cs	Executes chapter and scene prompts, validates, saves results, publishes progress.	Current. Per-run processing.
StoryIntelligenceResultRepository	PlotLine/Data/StoryIntelligenceResultRepository.cs	SQL access for runs, chapter/scene results, import commits.	Current. Core persistence path.
StoryIntelligenceProgressNotifier	PlotLine/Services/StoryIntelligenceProgressNotifier.cs	Broadcasts run progress to SignalR user group.	Current. Ephemeral messages.
StoryIntelligenceImportCommitService	PlotLine/Services/StoryIntelligenceImportCommitService.cs	Converts completed scene results into canonical chapters/scenes/metrics.	Current. Transaction is inside SQL proc.
Entity import services	StoryIntelligenceCharacterImportService.cs, StoryIntelligenceLocationImportService.cs, StoryIntelligenceAssetImportService.cs, StoryIntelligenceRelationshipImportService.cs, StoryIntelligenceKnowledgeImportService.cs	Build review candidates from committed scenes and saved AI JSON; create/link/ignore canonical entities.	Current. Decisions are partly transient.

The older StoryIntelligenceJobs model still exists through IStoryIntelligenceRepository, StoryIntelligenceService, and legacy progress support, but the onboarding analysis flow now primarily uses StoryIntelligenceRuns.

3. Current background-processing architecture

Program.cs registers StoryIntelligenceWorker, PersistedStoryIntelligenceWorker, EmailQueueWorker, and WordCompanionPresenceMonitor as hosted services. The persisted runner claims SQL-backed pending runs via StoryIntelligenceRun_ClaimNextPending using UPDLOCK, READPAST, then processes a single run.

Reliability strengths:

• Processing is independent of the browser and Word once runs are queued.
• Runs persist status, stage, message, token totals, scene counts, failure stage, error text, cancellation timestamps, start/completion timestamps and source text.
• Cancellation is stored with CancellationRequestedUtc and checked by the runner.
• Failed chapter/scene JSON parse attempts can be persisted as result rows.

Weaknesses:

• No durable visualisation event stream exists.
• No explicit retry count or backoff model was found for failed Story Intelligence runs.
• No stale-running recovery was found for StoryIntelligenceRuns if the app dies after a run is claimed and marked Running.
• The active batch mapping from onboarding batch ID to run IDs is in memory only.
• Multiple app instances would need careful review: SQL claim locking helps workers, but in-memory batch state and SignalR user groups are process-local.

4. Current provisional discovery model

Discovery type	Current storage before approval	Stable provisional ID?	Notes
Chapters	Scan preview/review in memory; approved chapter shells become canonical before analysis.	Temporary scan keys only.	Chapter shells are canonical before scene analysis starts.
Scenes	StoryIntelligenceSceneResults with SceneResultID and temporary scene number.	Yes for scene results.	Canonical scenes created during scene commit.
Characters	Parsed scene JSON and review candidates derived by service.	Derived key only.	Character decisions are in in-memory batch.
Character aliases	Derived/imported during character review.	No durable provisional alias ID.	Alias decisions not durable.
Locations	Parsed scene JSON and review candidates.	Derived key only.	Uses normalisation rules, no provisional table.
Assets	Parsed scene JSON and review candidates.	Derived key only.	Ownership/custody inferred during import service.
Relationships	Parsed scene JSON and review candidates.	Derived key only.	No persisted suspected/confirmed relationship lifecycle.
Timeline clues	Parsed scene JSON/import notes.	No.	Imported indirectly as scene notes/timeline data.
Knowledge	Parsed scene JSON and review candidates.	Derived key only.	Knowledge selections/decisions live in batch.
Confidence	Stored inside parsed AI JSON and some review previews.	Attached to JSON only.	No confidence history.
Evidence/excerpts	Source text and raw/parsed prompt output are retained.	Scene result IDs help.	User-facing evidence is not normalised.
Rejections	In-memory batch decisions while process lives.	No durable rejection ID.	Lost on application restart unless inferred from canonical state.
Duplicate/merge candidates	Derived at review time.	No.	Alias/merge operations do not retain durable provisional lineage.

The current system can answer "what chapter/scene is being analysed now?" and "what scene results exist?" reasonably well. It cannot reliably answer "in what order were characters discovered?", "when did confidence change?", "which provisional identities merged?", or "what did a reconnecting graph client miss?" without adding durable event/discovery storage.

5. Canonical data model relevant to the graph

Major graph-ready canonical records:

Record	Tables / procedures	Models / repositories	Suitability
Projects, books, chapters, scenes	Projects, Books, Chapters, Scenes; Project_*, Book_*, Chapter_*, Scene_*	ProjectRepository, BookRepository, ChapterRepository, SceneRepository	Strong node base. Ordering by book/chapter/scene is significant. Soft archive applies.
Characters and appearances	Characters, CharacterAliases, SceneCharacters	CharacterRepository	Strong character nodes and scene edges. Alias graph needs interpretation.
Relationships	CharacterRelationships, RelationshipEvents, relationship categories/types	CharacterRepository, RelationshipMapService	Already feeds Cytoscape relationship map. Good edge source.
Character knowledge	CharacterKnowledge, knowledge states	CharacterRepository	Useful for knowledge nodes/edges, but evidence lineage limited.
Locations	Locations, LocationAliases, LocationRelationships, scene-location/cross refs	LocationRepository	Good location nodes; hierarchy and relationships need graph mapping.
Assets	StoryAssets, AssetAliases, AssetEvents, AssetCustodyEvents, dependencies	AssetRepository	Good asset nodes and custody/ownership edges.
Timeline/scene metrics	Timeline procedures, SceneMetric*, scene date/time fields	TimelineRepository, metric repositories	Good timeline layering, not a standalone graph yet.
Plot lines/threads	PlotLines, plot thread procedures/events	PlotRepository	Good thread nodes/edges when events are included.
Continuity warnings	ContinuityWarnings, acknowledgements	WarningRepository, ContinuityWarningAcknowledgementRepository	Useful issue nodes attached to scenes/entities.
Project activity	ProjectActivity	ProjectActivityRepository	Audit/activity feed, not detailed enough for graph replay.

Existing canonical data is suitable for permanent Story Map snapshots, but the graph service should translate domain records into graph nodes/edges rather than exposing database DTOs directly.

6. Existing SignalR infrastructure

SignalR is registered in Program.cs with MaximumReceiveMessageSize = 1 MB.

Hubs:

• /hubs/word-companion-follow -> WordCompanionFollowHub, [Authorize]
• /hubs/story-intelligence -> StoryIntelligenceHub, [Authorize]

Group model:

• Word Companion uses word-companion-presence:{userId}.
• Story Intelligence uses story-intelligence:{userId}.

Client usage:

• PlotLine/Views/Shared/_Layout.cshtml includes SignalR from CDN for authenticated users.
• PlotLine/Views/WordCompanionHost/Index.cshtml includes SignalR from CDN.
• PlotLine/wwwroot/js/story-intelligence-progress.js connects to /hubs/story-intelligence, uses automatic reconnect, invokes watch methods after connect/reconnect, and handles StoryIntelligenceProgressChanged plus StoryIntelligenceRunProgressChanged.

Assessment:

• Suitable for progress updates and first visualisation transport.
• Current group scope is user-wide, not job/book-specific.
• Watch methods validate run ownership by UserID, but future graph groups should validate project/book/job access before joining.
• Messages are ephemeral. Missed recovery currently happens by reloading current run state, not by replaying events.
• No scale-out/backplane or sticky-session configuration was found. Deployment docs only cover upload storage; nginx/WebSocket settings were not present in the repo.

7. Existing frontend architecture

PlotLine is mostly server-rendered Razor with plain JavaScript and CSS:

• No PlotLine package.json was found.
• PlotDirector.WordCompanion/package.json exists only for the Office add-in manifest tooling.
• bundleconfig.json minifies selected CSS/JS files.
• Bootstrap is served from wwwroot/lib/bootstrap; the CSS appears to be Bootstrap 5.
• jQuery and validation libraries exist under wwwroot/lib.
• Theme support exists through data-theme / data-bs-theme, site.css, plotline-theme.css, and localStorage.
• Current JS is directly served/minified; no TypeScript build or module bundler is present for the MVC app.
• Existing components include Bootstrap modals, popovers, dropdowns, accordions, details-based drawers, a scene inspector drawer, and rich CSS theme audit pages.
• prefers-reduced-motion appears in CSS. Relationship Map disables layout animation.

Graph-related precedent:

• RelationshipMap/Index.cshtml loads Cytoscape from CDN and renders an interactive character relationship graph with pan/zoom, layout, edge selection, detail panel, resize handling and dark-theme styles.
• No Sigma.js or Graphology usage was found.

Sigma.js/Graphology integration options:

• Best long-term option: add an MVC frontend package/build pipeline and bundle Sigma/Graphology/worker modules with cache busting.
• Short-term prototype option: load from CDN like Cytoscape, but that conflicts with a durable production-grade graph engine and CSP/offline control.
• TypeScript is not currently established in PlotLine; Phase 21B should decide whether to introduce it with a minimal build step.

8. Relevant pages and navigation

Relevant routes:

• /onboarding -> onboarding wizard and Word scan controls.
• /onboarding/scan-review -> chapter/scene scan review.
• /onboarding/story-intelligence -> pre-analysis overview.
• /onboarding/story-intelligence/progress?batchId=... -> current progress page.
• /onboarding/story-intelligence/review?batchId=... -> scene review/import.
• /onboarding/story-intelligence/characters|locations|assets|relationships|knowledge -> staged entity review.
• /onboarding/story-intelligence/book/{bookId}/continue -> resume from book.
• /RelationshipMap?projectId=... -> current graph visualisation.
• /Projects, /Books/Details, /Chapters/Details, /Scenes/Edit, /Characters, /Locations, /StoryAssets, /Timeline, /PlotThreads, /ContinuityExplorer, /Warnings.

The current layout can host a graph, but the full-screen visualisation should probably use a minimal-chrome shell. No established full-screen graph canvas shell was found. Relationship Map has a graph workspace plus details panel that can be reused conceptually.

9. Security and authorisation assessment

Current strengths:

• Controllers are generally [Authorize].
• ProjectAccessFilter globally checks project/entity access based on controller parameters and model properties.
• Project collaborators and owners are handled through ProjectAccess_* stored procedures.
• Story Intelligence hub methods check authenticated user ID; WatchStoryIntelligenceRun checks run.UserID == userId.
• Razor output is encoded by default.

Risks/gaps for the visualisation:

• Future graph APIs cannot rely only on run.UserID; collaborative project access and book-level access must be checked.
• User-group SignalR broadcasts could deliver all a user's Story Intelligence progress to every tab. A graph should use job/book groups after explicit authorisation.
• Manuscript excerpts in snapshots/evidence drawers need strict project/book/run ownership checks.
• CDN scripts complicate future CSP.
• Uploaded portrait/cover URLs should be served through existing upload path rules and not blindly embedded from AI output.

Recommended graph access:

• Snapshot endpoint: require authenticated user and ProjectAccess_UserCanAccessEntity for the book/run/project.
• Hub group join: WatchStoryIntelligenceGraph(runId, afterSequence) validates the same access, then joins story-intelligence-run:{runId}.
• Do not expose raw prompt internals or private reasoning; expose short manuscript evidence and structured user-facing justifications.

10. Stable identity assessment

Canonical IDs are stable (project-{id}, book-{id}, chapter-{id}, scene-{id}, character-{id}, location-{id}, asset-{id}, relationship-{id}, knowledge-{id}).

Current provisional stability:

• StoryIntelligenceRunID, ChapterResultID, and SceneResultID are stable.
• Temporary scan keys are stable only within the scan preview memory store.
• Review candidate keys for characters/locations/assets/relationships/knowledge are derived and not durable.
• Merges/aliases do not have a durable provisional-to-canonical lineage.

Recommended convention, pending future storage:

• run-{runId}
• chapter-result-{chapterResultId}
• scene-result-{sceneResultId}
• provisional-character-{discoveryId} -> merge/replace to character-{characterId}
• provisional-location-{discoveryId} -> location-{locationId}
• provisional-asset-{discoveryId} -> asset-{assetId}
• provisional-relationship-{discoveryId} -> relationship-{characterRelationshipId}
• provisional-knowledge-{discoveryId} -> knowledge-{characterKnowledgeId}

The client will need an explicit IdentityMerged or NodePromoted event to preserve visual continuity when provisional nodes become canonical.

11. Evidence and confidence assessment

Stored today:

• Source text on StoryIntelligenceRuns.
• Raw response JSON, assistant output JSON and parsed JSON for chapter/scene results.
• Validation errors/warnings counts.
• Confidence values inside parsed scene JSON for many extracted facts.
• Source scene/chapter through run, chapter result and scene result linkage.
• Canonical import commit metadata and warnings.

Not stored in a graph-ready way:

• Normalised evidence records per claim.
• Short source excerpts per discovery.
• Confidence history.
• Alternative candidates.
• Confirmation or contradictory evidence across later scenes.
• Manual override/rejection reason as durable data.
• User-facing reasoning summaries separate from raw AI output.

The future evidence drawer can be supported if Phase 21 adds durable discovery/evidence rows derived from parsed scene results, not from private model reasoning.

12. Event-stream assessment

Existing mechanisms:

• ProjectActivity and ProjectAudit_Record: useful general audit trail, not fine-grained Story Intelligence replay.
• SignalR progress events: ephemeral.
• Import commit rows: durable finalisation record, not incremental discovery history.
• SQL result tables: durable snapshots of prompt outputs, not ordered graph events.

Needed event model:

• Dedicated StoryIntelligenceVisualisationEvents table.
• Monotonic SequenceNumber per run/job.
• Event type, timestamp, project/book/run/scene/discovery references.
• JSON payload with a version.
• Idempotency key.
• Retention rules.
• Optional processed/published metadata if an outbox publisher is introduced.

Candidate events:

AnalysisStarted, StageChanged, ChapterAnalysisStarted, SceneAnalysisStarted, SceneAnalysisCompleted, NodeDiscovered, NodeUpdated, EdgeDiscovered, EdgeUpdated, IdentityMerged, RelationshipSuspected, RelationshipConfirmed, TimelineEventDiscovered, KnowledgeThreadOpened, KnowledgeThreadResolved, ObservationCreated, AnalysisCompleted, AnalysisFailed, NodePromotedToCanonical, DiscoveryRejected.

13. Performance and scale assessment

Estimated graph sizes:

Scope	Scenes	Characters	Locations	Assets	Relationships	Knowledge/events	Nodes	Edges
Small test	5-15	5-12	3-8	3-10	5-20	10-40	30-90	50-150
Normal novel	60-120	25-80	30-120	40-180	80-250	150-600	300-1,100	700-3,000
Large novel	150-250	80-180	100-250	150-400	250-700	500-1,500	1,000-3,000	3,000-10,000
Trilogy	300-600	150-350	200-600	300-1,000	700-2,000	1,500-5,000	3,000-9,000	10,000-35,000
Large multi-book project	800+	400+	800+	1,500+	3,000+	8,000+	12,000+	50,000+

Primary risks:

• JSON payload size for snapshots and replay.
• Browser layout cost for dense relationship/evidence edges.
• Portrait textures and labels.
• Replaying thousands of small events after reconnect.
• SQL DTO construction from many canonical tables.

Recommendation: Phase 21B should target one book / one active analysis batch, with filters and level-of-detail from the start. Project-wide exploration should come later.

14. Reconnection and recovery assessment

Current recovery:

• Progress JS reconnects and invokes watch methods to get current run state.
• Browser refresh can reload progress from SQL if the batch still exists.
• Closing Word does not stop queued analysis.

Gaps:

• In-memory scan preview and batch store do not survive application restart.
• SignalR missed events are not replayable.
• No event sequence exists.
• No graph snapshot endpoint exists.

Required Phase 21 design:

flowchart LR
    A["Browser opens visualisation"] --> B["GET graph snapshot"]
    B --> C["Snapshot includes lastSequence"]
    C --> D["Join authorised SignalR run group"]
    D --> E["Receive events with sequence numbers"]
    E --> F["Apply idempotently to Graphology"]
    F --> G["Reconnect"]
    G --> H["Request events after last applied sequence"]
    H --> F
    H --> I["If too old, reload snapshot"]

15. Approval-to-canonical transition assessment

Scene import:

• StoryIntelligenceImportCommitService prepares completed run results.
• SQL commit creates canonical scenes/metrics and records StoryIntelligenceImportCommits.
• Pipeline state records scene import.
• Duplicate scene import is blocked by readiness checks.

Entity imports:

• Review candidates are derived from committed scenes and saved parsed JSON.
• Create/link/alias/ignore actions call existing canonical repositories.
• Pipeline state advances per book.
• Decisions and last import summaries are in the batch object, not durable review tables.

Risks:

• Provisional node IDs will disappear unless durable discovery IDs are introduced.
• Rejections/aliases are not reliably reconstructable after restart.
• Partial approval can change graph topology without an event record.
• Relationships can reference provisional characters unless promotion/merge ordering is explicit.

flowchart TD
    A["Parsed scene result"] --> B["Provisional discovery row"]
    B --> C["Graph provisional node"]
    C --> D{"Author decision"}
    D -->|Create| E["Canonical record created"]
    D -->|Link| F["Existing canonical record linked"]
    D -->|Alias/Merge| G["Discovery merged into target"]
    D -->|Ignore| H["Discovery rejected"]
    E --> I["NodePromotedToCanonical event"]
    F --> I
    G --> J["IdentityMerged event"]
    H --> K["DiscoveryRejected event"]

16. Proposed Phase 21 architecture

flowchart TD
    A["Story Intelligence analysis worker"] --> B["Persist discoveries and state"]
    B --> C["SQL Server"]
    C --> D["Ordered visualisation event table"]
    D --> E["Story Intelligence event publisher"]
    E --> F["SignalR hub"]
    F --> G["Browser client"]
    G --> H["Snapshot loader"]
    G --> I["Event applier"]
    H --> J["Graphology graph model"]
    I --> J
    J --> K["Sigma.js renderer"]
    G --> L["Panels, feeds, filters, evidence drawer"]

Principles:

• SQL Server is authoritative.
• SignalR is transport only.
• Browser never analyses manuscript.
• Graph DTOs are versioned and separate from database DTOs.
• The same graph engine supports live analysis and completed Story Map.

17. Proposed server-side components

• StoryIntelligenceVisualisationSnapshotService: builds snapshot for run/book/project.
• StoryIntelligenceVisualisationEventRepository: persists ordered events.
• StoryIntelligenceDiscoveryRepository: persists provisional nodes/edges/evidence.
• StoryIntelligenceVisualisationPublisher: publishes stored events to SignalR after commit.
• StoryIntelligenceGraphHub: job/book-scoped groups with explicit access validation.
• StoryIntelligenceGraphController or API controller: snapshot and missed-event endpoints.
• Integration points in PersistedStoryIntelligenceRunner and import services to write discovery/event rows.

18. Proposed browser-side components

flowchart LR
    A["Razor page"] --> B["graph bootstrap"]
    B --> C["snapshot API client"]
    B --> D["SignalR client"]
    C --> E["graph adapter"]
    D --> E
    E --> F["Graphology"]
    F --> G["Sigma renderer"]
    E --> H["activity feed"]
    E --> I["evidence drawer"]
    E --> J["filters/search"]
    K["layout worker"] --> F

Suggested modules:

• story-graph-client
• story-graph-adapter
• story-graph-renderer
• story-graph-layout-worker
• story-graph-panels
• story-graph-api

19. Proposed API contracts

Snapshot:

GET /api/story-intelligence/runs/{runId}/graph-snapshot?scope=book

Response shape:

{
  "runId": 123,
  "projectId": 10,
  "bookId": 20,
  "lastSequence": 456,
  "nodes": [],
  "edges": [],
  "activeStage": "SceneIntelligence",
  "permissions": { "canReview": true },
  "generatedUtc": "2026-07-11T00:00:00Z"
}

Missed events:

GET /api/story-intelligence/runs/{runId}/graph-events?afterSequence=456

20. Proposed SignalR message contracts

Hub route proposal: /hubs/story-intelligence-graph

Client invokes:

• WatchRunGraph(runId, afterSequence)
• LeaveRunGraph(runId)

Server sends:

{
  "runId": 123,
  "sequence": 457,
  "eventId": "01J...",
  "type": "NodeDiscovered",
  "occurredUtc": "2026-07-11T00:00:00Z",
  "payloadVersion": 1,
  "payload": {}
}

21. Proposed event categories

• Analysis lifecycle
• Progress and stage
• Scene/chapter lifecycle
• Discovery node lifecycle
• Relationship/edge lifecycle
• Evidence/confidence updates
• Merge/promotion/rejection
• Review/approval
• Failure/cancellation/recovery

22. Proposed implementation sequence

1. Add durable event/discovery design and SQL script.
2. Add read-only snapshot service over existing canonical plus run-result data.
3. Add event writer around current runner progress and scene result persistence.
4. Add graph hub with authorised job groups and replay-after-sequence.
5. Build browser graph adapter with mocked renderer data from real snapshots.
6. Integrate Sigma.js/Graphology and layout worker.
7. Add live progress/discovery events.
8. Add approval/promotion events for scene and entity imports.
9. Add completed Story Map route reusing the graph engine.

23. Database changes likely to be required

Follow existing SQL script convention in PlotLine/Sql, where the current latest script is 134_Phase20AV_ArchivedBookProjectPermanentDeletion.sql.

Likely future script:

• 135_Phase21B_StoryIntelligenceVisualisationEvents.sql

Likely tables:

• StoryIntelligenceDiscoveries
• StoryIntelligenceDiscoveryEvidence
• StoryIntelligenceDiscoveryAliases
• StoryIntelligenceVisualisationEvents
• Possibly StoryIntelligenceGraphSnapshots if snapshot materialisation is needed later.

Conventions to follow:

• Idempotent IF OBJECT_ID / COL_LENGTH checks.
• CREATE OR ALTER PROCEDURE.
• PK_, FK_, DF_, CK_, IX_ naming.
• User/project/book isolation columns and indexes.
• Do not alter historical scripts.

24. Risks and mitigations

Risk	Mitigation
Volatile batch state prevents recovery	Persist batch/run mappings and review decisions or derive them from durable pipeline rows.
SignalR events missed	Store ordered events and replay after sequence.
Graph too dense	One-book scope, filters, clustering, label level-of-detail.
Provisional/canonical ID churn	Add discovery IDs and explicit promotion/merge events.
Manuscript data leakage	Snapshot/hub access checks against project/book/run.
Package integration churn	Decide MVC frontend build approach before adding Sigma/Graphology.
CDN/CSP concerns	Prefer bundled assets for production.
Existing progress workflow regression	Add visualisation beside current progress page first.

25. Decisions still required

• Whether Phase 21B introduces an MVC package.json and bundler.
• Whether to persist batch state or replace batch IDs with durable book/run pipeline IDs.
• Exact discovery/event SQL schema.
• Whether to keep Cytoscape for Relationship Map or migrate it later to the new engine.
• Retention policy for raw source excerpts and visualisation events.
• Whether snapshots are generated on demand or materialised.

26. Files inspected

Primary files inspected:

• PlotLine/Program.cs
• PlotLine/Hubs/StoryIntelligenceHub.cs
• PlotLine/Hubs/WordCompanionFollowHub.cs
• PlotLine/Controllers/OnboardingController.cs
• PlotLine/Controllers/RelationshipMapController.cs
• PlotLine/Services/OnboardingService.cs
• PlotLine/Services/OnboardingStoryIntelligenceService.cs
• PlotLine/Services/PersistedStoryIntelligenceWorker.cs
• PlotLine/Services/PersistedStoryIntelligenceRunner.cs
• PlotLine/Services/StoryIntelligenceProgressNotifier.cs
• PlotLine/Services/StoryIntelligenceImportCommitService.cs
• PlotLine/Services/SceneImportResolver.cs
• PlotLine/Services/StoryIntelligenceCharacterImportService.cs
• PlotLine/Services/StoryIntelligenceLocationImportService.cs
• PlotLine/Services/ProjectAccessFilter.cs
• PlotLine/Services/ProjectAccessServices.cs
• PlotLine/Data/StoryIntelligenceResultRepository.cs
• PlotLine/Data/StoryIntelligenceRepository.cs
• PlotLine/Data/StoryIntelligencePipelineRepository.cs
• PlotLine/Data/Repositories.cs
• PlotLine/Models/StoryIntelligenceModels.cs
• PlotLine/Models/StoryIntelligencePersistenceModels.cs
• PlotLine/ViewModels/OnboardingViewModels.cs
• PlotLine/Views/Onboarding/StoryIntelligenceProgress.cshtml
• PlotLine/Views/Onboarding/StoryIntelligenceReview.cshtml
• PlotLine/Views/RelationshipMap/Index.cshtml
• PlotLine/Views/Shared/_Layout.cshtml
• PlotLine/wwwroot/js/story-intelligence-progress.js
• PlotLine/bundleconfig.json
• PlotDirector.WordCompanion/package.json
• SQL scripts 001, 004, 005, 006, 007, 020, 033, 039, 042, 046, 088, 093, 111, 114, 115, 117, 121, 123, 124, 127 through 134
• PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Visualisation_Design.md
• PlotLine/Docs/Architecture/DeploymentConfiguration.md

27. Recommended next step for Phase 21B

Phase 21B should be a narrow foundation phase:

1. Design and add durable graph discovery/event storage for one active StoryIntelligenceRun.
2. Add authorised snapshot and event replay endpoints.
3. Add a SignalR graph hub that joins one run group after access validation.
4. Emit lifecycle, stage, scene-start, scene-complete and basic discovery events from the existing persisted runner.
5. Do not replace the current progress page yet; add the visualisation as an optional "Watch Story Intelligence" experience backed by the same SQL state.