nick/PlotDirector
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
PlotDirector/docs/phases/Phase-21A-Story-Intelligence-Visualisation-Audit.md

597 lines
31 KiB
Markdown
Raw Blame History

# Phase 21A - Story Intelligence Visualisation Architecture Audit
## 1. Executive summary
PlotDirector already has most of the server-side ingredients needed for a durable Story Intelligence visualisation: ASP.NET Core MVC, SQL Server persistence, Dapper repositories, SignalR, a persisted background worker, canonical story entities, and a current progress page that survives browser and Word closure.
The current live experience is not yet graph-ready. SignalR messages are progress notifications, not durable events. Provisional discoveries are mostly reconstructed from saved scene-analysis JSON and transient in-memory batch decisions. Scene/chapter AI results have stable SQL identifiers, but provisional characters, locations, assets, relationships, knowledge items, confidence changes, merges and rejections do not have durable discovery records or ordered event history.
Sigma.js, Graphology and SignalR remain suitable for Phase 21 if SQL Server remains authoritative and SignalR is only a transport. The biggest prerequisite for Phase 21B is a durable visualisation snapshot/event model that can be rebuilt after refresh, replayed after reconnect, and joined to both provisional and canonical PlotDirector IDs.
No code, package, database, deployment or workflow changes were made for this audit.
## 2. Current Story Intelligence workflow
```mermaid
flowchart TD
A["Author starts onboarding import"] --> B["Website-led onboarding wizard"]
B --> C["Project and book selected or created"]
C --> D["Word Companion registers over SignalR"]
D --> E["Website asks Companion to scan document"]
E --> F["Companion reports scan preview"]
F --> G["Author reviews chapters, scenes, character candidates"]
G --> H["Onboarding builds approved chapter shells"]
H --> I["One StoryIntelligenceRun queued per approved chapter"]
I --> J["PersistedStoryIntelligenceWorker claims pending runs"]
J --> K["Chapter Structure prompt detects scene boundaries"]
K --> L["Scene Intelligence prompt analyses each suggested scene"]
L --> M["Chapter and scene results persisted in SQL"]
M --> N["SignalR progress updates current page"]
N --> O["Author reviews/imports scenes"]
O --> P["Characters, locations, assets, relationships, knowledge reviewed in sequence"]
P --> Q["Canonical PlotDirector records created or linked"]
```
Important components:
| Component | Path | Responsibility | Current state |
|---|---|---|---|
| `OnboardingController` | `PlotLine/Controllers/OnboardingController.cs` | Onboarding routes, scan review, Story Intelligence start/progress/review/import stages. | Current. Central workflow controller. |
| `WordCompanionFollowHub` | `PlotLine/Hubs/WordCompanionFollowHub.cs` | Companion presence, scan command, scan progress/completion, build marker updates. | Current. User-group scoped. |
| `ManuscriptScanPreviewStore` | `PlotLine/Services/ManuscriptScanPreviewStore.cs` | In-memory scan preview/review/build state. | Current but non-durable. Important recovery weakness. |
| `OnboardingService` | `PlotLine/Services/OnboardingService.cs` | Wizard state, scan review validation, approved chapter build. | Current. Creates chapter shells before AI run. |
| `OnboardingStoryIntelligenceService` | `PlotLine/Services/OnboardingStoryIntelligenceService.cs` | Queues per-chapter runs, builds progress/review models, orchestrates staged imports. | Current. Depends on in-memory batch store. |
| `OnboardingStoryIntelligenceBatchStore` | `PlotLine/Services/OnboardingStoryIntelligenceService.cs` | Keeps batch-to-run mapping and review decisions in a `ConcurrentDictionary`. | Current but volatile. Key gap. |
| `PersistedStoryIntelligenceWorker` | `PlotLine/Services/PersistedStoryIntelligenceWorker.cs` | Background service polling every 3 seconds for pending persisted runs. | Current. Browser-independent. |
| `PersistedStoryIntelligenceRunner` | `PlotLine/Services/PersistedStoryIntelligenceRunner.cs` | Executes chapter and scene prompts, validates, saves results, publishes progress. | Current. Per-run processing. |
| `StoryIntelligenceResultRepository` | `PlotLine/Data/StoryIntelligenceResultRepository.cs` | SQL access for runs, chapter/scene results, import commits. | Current. Core persistence path. |
| `StoryIntelligenceProgressNotifier` | `PlotLine/Services/StoryIntelligenceProgressNotifier.cs` | Broadcasts run progress to SignalR user group. | Current. Ephemeral messages. |
| `StoryIntelligenceImportCommitService` | `PlotLine/Services/StoryIntelligenceImportCommitService.cs` | Converts completed scene results into canonical chapters/scenes/metrics. | Current. Transaction is inside SQL proc. |
| Entity import services | `StoryIntelligenceCharacterImportService.cs`, `StoryIntelligenceLocationImportService.cs`, `StoryIntelligenceAssetImportService.cs`, `StoryIntelligenceRelationshipImportService.cs`, `StoryIntelligenceKnowledgeImportService.cs` | Build review candidates from committed scenes and saved AI JSON; create/link/ignore canonical entities. | Current. Decisions are partly transient. |
The older `StoryIntelligenceJobs` model still exists through `IStoryIntelligenceRepository`, `StoryIntelligenceService`, and legacy progress support, but the onboarding analysis flow now primarily uses `StoryIntelligenceRuns`.
## 3. Current background-processing architecture
`Program.cs` registers `StoryIntelligenceWorker`, `PersistedStoryIntelligenceWorker`, `EmailQueueWorker`, and `WordCompanionPresenceMonitor` as hosted services. The persisted runner claims SQL-backed pending runs via `StoryIntelligenceRun_ClaimNextPending` using `UPDLOCK, READPAST`, then processes a single run.
Reliability strengths:
- Processing is independent of the browser and Word once runs are queued.
- Runs persist status, stage, message, token totals, scene counts, failure stage, error text, cancellation timestamps, start/completion timestamps and source text.
- Cancellation is stored with `CancellationRequestedUtc` and checked by the runner.
- Failed chapter/scene JSON parse attempts can be persisted as result rows.
Weaknesses:
- No durable visualisation event stream exists.
- No explicit retry count or backoff model was found for failed Story Intelligence runs.
- No stale-running recovery was found for `StoryIntelligenceRuns` if the app dies after a run is claimed and marked `Running`.
- The active batch mapping from onboarding batch ID to run IDs is in memory only.
- Multiple app instances would need careful review: SQL claim locking helps workers, but in-memory batch state and SignalR user groups are process-local.
## 4. Current provisional discovery model
| Discovery type | Current storage before approval | Stable provisional ID? | Notes |
|---|---|---:|---|
| Chapters | Scan preview/review in memory; approved chapter shells become canonical before analysis. | Temporary scan keys only. | Chapter shells are canonical before scene analysis starts. |
| Scenes | `StoryIntelligenceSceneResults` with `SceneResultID` and temporary scene number. | Yes for scene results. | Canonical scenes created during scene commit. |
| Characters | Parsed scene JSON and review candidates derived by service. | Derived key only. | Character decisions are in in-memory batch. |
| Character aliases | Derived/imported during character review. | No durable provisional alias ID. | Alias decisions not durable. |
| Locations | Parsed scene JSON and review candidates. | Derived key only. | Uses normalisation rules, no provisional table. |
| Assets | Parsed scene JSON and review candidates. | Derived key only. | Ownership/custody inferred during import service. |
| Relationships | Parsed scene JSON and review candidates. | Derived key only. | No persisted suspected/confirmed relationship lifecycle. |
| Timeline clues | Parsed scene JSON/import notes. | No. | Imported indirectly as scene notes/timeline data. |
| Knowledge | Parsed scene JSON and review candidates. | Derived key only. | Knowledge selections/decisions live in batch. |
| Confidence | Stored inside parsed AI JSON and some review previews. | Attached to JSON only. | No confidence history. |
| Evidence/excerpts | Source text and raw/parsed prompt output are retained. | Scene result IDs help. | User-facing evidence is not normalised. |
| Rejections | In-memory batch decisions while process lives. | No durable rejection ID. | Lost on application restart unless inferred from canonical state. |
| Duplicate/merge candidates | Derived at review time. | No. | Alias/merge operations do not retain durable provisional lineage. |
The current system can answer "what chapter/scene is being analysed now?" and "what scene results exist?" reasonably well. It cannot reliably answer "in what order were characters discovered?", "when did confidence change?", "which provisional identities merged?", or "what did a reconnecting graph client miss?" without adding durable event/discovery storage.
## 5. Canonical data model relevant to the graph
Major graph-ready canonical records:
| Record | Tables / procedures | Models / repositories | Suitability |
|---|---|---|---|
| Projects, books, chapters, scenes | `Projects`, `Books`, `Chapters`, `Scenes`; `Project_*`, `Book_*`, `Chapter_*`, `Scene_*` | `ProjectRepository`, `BookRepository`, `ChapterRepository`, `SceneRepository` | Strong node base. Ordering by book/chapter/scene is significant. Soft archive applies. |
| Characters and appearances | `Characters`, `CharacterAliases`, `SceneCharacters` | `CharacterRepository` | Strong character nodes and scene edges. Alias graph needs interpretation. |
| Relationships | `CharacterRelationships`, `RelationshipEvents`, relationship categories/types | `CharacterRepository`, `RelationshipMapService` | Already feeds Cytoscape relationship map. Good edge source. |
| Character knowledge | `CharacterKnowledge`, knowledge states | `CharacterRepository` | Useful for knowledge nodes/edges, but evidence lineage limited. |
| Locations | `Locations`, `LocationAliases`, `LocationRelationships`, scene-location/cross refs | `LocationRepository` | Good location nodes; hierarchy and relationships need graph mapping. |
| Assets | `StoryAssets`, `AssetAliases`, `AssetEvents`, `AssetCustodyEvents`, dependencies | `AssetRepository` | Good asset nodes and custody/ownership edges. |
| Timeline/scene metrics | Timeline procedures, `SceneMetric*`, scene date/time fields | `TimelineRepository`, metric repositories | Good timeline layering, not a standalone graph yet. |
| Plot lines/threads | `PlotLines`, plot thread procedures/events | `PlotRepository` | Good thread nodes/edges when events are included. |
| Continuity warnings | `ContinuityWarnings`, acknowledgements | `WarningRepository`, `ContinuityWarningAcknowledgementRepository` | Useful issue nodes attached to scenes/entities. |
| Project activity | `ProjectActivity` | `ProjectActivityRepository` | Audit/activity feed, not detailed enough for graph replay. |
Existing canonical data is suitable for permanent Story Map snapshots, but the graph service should translate domain records into graph nodes/edges rather than exposing database DTOs directly.
## 6. Existing SignalR infrastructure
SignalR is registered in `Program.cs` with `MaximumReceiveMessageSize = 1 MB`.
Hubs:
- `/hubs/word-companion-follow` -> `WordCompanionFollowHub`, `[Authorize]`
- `/hubs/story-intelligence` -> `StoryIntelligenceHub`, `[Authorize]`
Group model:
- Word Companion uses `word-companion-presence:{userId}`.
- Story Intelligence uses `story-intelligence:{userId}`.
Client usage:
- `PlotLine/Views/Shared/_Layout.cshtml` includes SignalR from CDN for authenticated users.
- `PlotLine/Views/WordCompanionHost/Index.cshtml` includes SignalR from CDN.
- `PlotLine/wwwroot/js/story-intelligence-progress.js` connects to `/hubs/story-intelligence`, uses automatic reconnect, invokes watch methods after connect/reconnect, and handles `StoryIntelligenceProgressChanged` plus `StoryIntelligenceRunProgressChanged`.
Assessment:
- Suitable for progress updates and first visualisation transport.
- Current group scope is user-wide, not job/book-specific.
- Watch methods validate run ownership by `UserID`, but future graph groups should validate project/book/job access before joining.
- Messages are ephemeral. Missed recovery currently happens by reloading current run state, not by replaying events.
- No scale-out/backplane or sticky-session configuration was found. Deployment docs only cover upload storage; nginx/WebSocket settings were not present in the repo.
## 7. Existing frontend architecture
PlotLine is mostly server-rendered Razor with plain JavaScript and CSS:
- No PlotLine `package.json` was found.
- `PlotDirector.WordCompanion/package.json` exists only for the Office add-in manifest tooling.
- `bundleconfig.json` minifies selected CSS/JS files.
- Bootstrap is served from `wwwroot/lib/bootstrap`; the CSS appears to be Bootstrap 5.
- jQuery and validation libraries exist under `wwwroot/lib`.
- Theme support exists through `data-theme` / `data-bs-theme`, `site.css`, `plotline-theme.css`, and localStorage.
- Current JS is directly served/minified; no TypeScript build or module bundler is present for the MVC app.
- Existing components include Bootstrap modals, popovers, dropdowns, accordions, details-based drawers, a scene inspector drawer, and rich CSS theme audit pages.
- `prefers-reduced-motion` appears in CSS. Relationship Map disables layout animation.
Graph-related precedent:
- `RelationshipMap/Index.cshtml` loads Cytoscape from CDN and renders an interactive character relationship graph with pan/zoom, layout, edge selection, detail panel, resize handling and dark-theme styles.
- No Sigma.js or Graphology usage was found.
Sigma.js/Graphology integration options:
- Best long-term option: add an MVC frontend package/build pipeline and bundle Sigma/Graphology/worker modules with cache busting.
- Short-term prototype option: load from CDN like Cytoscape, but that conflicts with a durable production-grade graph engine and CSP/offline control.
- TypeScript is not currently established in PlotLine; Phase 21B should decide whether to introduce it with a minimal build step.
## 8. Relevant pages and navigation
Relevant routes:
- `/onboarding` -> onboarding wizard and Word scan controls.
- `/onboarding/scan-review` -> chapter/scene scan review.
- `/onboarding/story-intelligence` -> pre-analysis overview.
- `/onboarding/story-intelligence/progress?batchId=...` -> current progress page.
- `/onboarding/story-intelligence/review?batchId=...` -> scene review/import.
- `/onboarding/story-intelligence/characters|locations|assets|relationships|knowledge` -> staged entity review.
- `/onboarding/story-intelligence/book/{bookId}/continue` -> resume from book.
- `/RelationshipMap?projectId=...` -> current graph visualisation.
- `/Projects`, `/Books/Details`, `/Chapters/Details`, `/Scenes/Edit`, `/Characters`, `/Locations`, `/StoryAssets`, `/Timeline`, `/PlotThreads`, `/ContinuityExplorer`, `/Warnings`.
The current layout can host a graph, but the full-screen visualisation should probably use a minimal-chrome shell. No established full-screen graph canvas shell was found. Relationship Map has a graph workspace plus details panel that can be reused conceptually.
## 9. Security and authorisation assessment
Current strengths:
- Controllers are generally `[Authorize]`.
- `ProjectAccessFilter` globally checks project/entity access based on controller parameters and model properties.
- Project collaborators and owners are handled through `ProjectAccess_*` stored procedures.
- Story Intelligence hub methods check authenticated user ID; `WatchStoryIntelligenceRun` checks `run.UserID == userId`.
- Razor output is encoded by default.
Risks/gaps for the visualisation:
- Future graph APIs cannot rely only on `run.UserID`; collaborative project access and book-level access must be checked.
- User-group SignalR broadcasts could deliver all a user's Story Intelligence progress to every tab. A graph should use job/book groups after explicit authorisation.
- Manuscript excerpts in snapshots/evidence drawers need strict project/book/run ownership checks.
- CDN scripts complicate future CSP.
- Uploaded portrait/cover URLs should be served through existing upload path rules and not blindly embedded from AI output.
Recommended graph access:
- Snapshot endpoint: require authenticated user and `ProjectAccess_UserCanAccessEntity` for the book/run/project.
- Hub group join: `WatchStoryIntelligenceGraph(runId, afterSequence)` validates the same access, then joins `story-intelligence-run:{runId}`.
- Do not expose raw prompt internals or private reasoning; expose short manuscript evidence and structured user-facing justifications.
## 10. Stable identity assessment
Canonical IDs are stable (`project-{id}`, `book-{id}`, `chapter-{id}`, `scene-{id}`, `character-{id}`, `location-{id}`, `asset-{id}`, `relationship-{id}`, `knowledge-{id}`).
Current provisional stability:
- `StoryIntelligenceRunID`, `ChapterResultID`, and `SceneResultID` are stable.
- Temporary scan keys are stable only within the scan preview memory store.
- Review candidate keys for characters/locations/assets/relationships/knowledge are derived and not durable.
- Merges/aliases do not have a durable provisional-to-canonical lineage.
Recommended convention, pending future storage:
- `run-{runId}`
- `chapter-result-{chapterResultId}`
- `scene-result-{sceneResultId}`
- `provisional-character-{discoveryId}` -> merge/replace to `character-{characterId}`
- `provisional-location-{discoveryId}` -> `location-{locationId}`
- `provisional-asset-{discoveryId}` -> `asset-{assetId}`
- `provisional-relationship-{discoveryId}` -> `relationship-{characterRelationshipId}`
- `provisional-knowledge-{discoveryId}` -> `knowledge-{characterKnowledgeId}`
The client will need an explicit `IdentityMerged` or `NodePromoted` event to preserve visual continuity when provisional nodes become canonical.
## 11. Evidence and confidence assessment
Stored today:
- Source text on `StoryIntelligenceRuns`.
- Raw response JSON, assistant output JSON and parsed JSON for chapter/scene results.
- Validation errors/warnings counts.
- Confidence values inside parsed scene JSON for many extracted facts.
- Source scene/chapter through run, chapter result and scene result linkage.
- Canonical import commit metadata and warnings.
Not stored in a graph-ready way:
- Normalised evidence records per claim.
- Short source excerpts per discovery.
- Confidence history.
- Alternative candidates.
- Confirmation or contradictory evidence across later scenes.
- Manual override/rejection reason as durable data.
- User-facing reasoning summaries separate from raw AI output.
The future evidence drawer can be supported if Phase 21 adds durable discovery/evidence rows derived from parsed scene results, not from private model reasoning.
## 12. Event-stream assessment
Existing mechanisms:
- `ProjectActivity` and `ProjectAudit_Record`: useful general audit trail, not fine-grained Story Intelligence replay.
- SignalR progress events: ephemeral.
- Import commit rows: durable finalisation record, not incremental discovery history.
- SQL result tables: durable snapshots of prompt outputs, not ordered graph events.
Needed event model:
- Dedicated `StoryIntelligenceVisualisationEvents` table.
- Monotonic `SequenceNumber` per run/job.
- Event type, timestamp, project/book/run/scene/discovery references.
- JSON payload with a version.
- Idempotency key.
- Retention rules.
- Optional processed/published metadata if an outbox publisher is introduced.
Candidate events:
`AnalysisStarted`, `StageChanged`, `ChapterAnalysisStarted`, `SceneAnalysisStarted`, `SceneAnalysisCompleted`, `NodeDiscovered`, `NodeUpdated`, `EdgeDiscovered`, `EdgeUpdated`, `IdentityMerged`, `RelationshipSuspected`, `RelationshipConfirmed`, `TimelineEventDiscovered`, `KnowledgeThreadOpened`, `KnowledgeThreadResolved`, `ObservationCreated`, `AnalysisCompleted`, `AnalysisFailed`, `NodePromotedToCanonical`, `DiscoveryRejected`.
## 13. Performance and scale assessment
Estimated graph sizes:
| Scope | Scenes | Characters | Locations | Assets | Relationships | Knowledge/events | Nodes | Edges |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Small test | 5-15 | 5-12 | 3-8 | 3-10 | 5-20 | 10-40 | 30-90 | 50-150 |
| Normal novel | 60-120 | 25-80 | 30-120 | 40-180 | 80-250 | 150-600 | 300-1,100 | 700-3,000 |
| Large novel | 150-250 | 80-180 | 100-250 | 150-400 | 250-700 | 500-1,500 | 1,000-3,000 | 3,000-10,000 |
| Trilogy | 300-600 | 150-350 | 200-600 | 300-1,000 | 700-2,000 | 1,500-5,000 | 3,000-9,000 | 10,000-35,000 |
| Large multi-book project | 800+ | 400+ | 800+ | 1,500+ | 3,000+ | 8,000+ | 12,000+ | 50,000+ |
Primary risks:
- JSON payload size for snapshots and replay.
- Browser layout cost for dense relationship/evidence edges.
- Portrait textures and labels.
- Replaying thousands of small events after reconnect.
- SQL DTO construction from many canonical tables.
Recommendation: Phase 21B should target one book / one active analysis batch, with filters and level-of-detail from the start. Project-wide exploration should come later.
## 14. Reconnection and recovery assessment
Current recovery:
- Progress JS reconnects and invokes watch methods to get current run state.
- Browser refresh can reload progress from SQL if the batch still exists.
- Closing Word does not stop queued analysis.
Gaps:
- In-memory scan preview and batch store do not survive application restart.
- SignalR missed events are not replayable.
- No event sequence exists.
- No graph snapshot endpoint exists.
Required Phase 21 design:
```mermaid
flowchart LR
A["Browser opens visualisation"] --> B["GET graph snapshot"]
B --> C["Snapshot includes lastSequence"]
C --> D["Join authorised SignalR run group"]
D --> E["Receive events with sequence numbers"]
E --> F["Apply idempotently to Graphology"]
F --> G["Reconnect"]
G --> H["Request events after last applied sequence"]
H --> F
H --> I["If too old, reload snapshot"]
```
## 15. Approval-to-canonical transition assessment
Scene import:
- `StoryIntelligenceImportCommitService` prepares completed run results.
- SQL commit creates canonical scenes/metrics and records `StoryIntelligenceImportCommits`.
- Pipeline state records scene import.
- Duplicate scene import is blocked by readiness checks.
Entity imports:
- Review candidates are derived from committed scenes and saved parsed JSON.
- Create/link/alias/ignore actions call existing canonical repositories.
- Pipeline state advances per book.
- Decisions and last import summaries are in the batch object, not durable review tables.
Risks:
- Provisional node IDs will disappear unless durable discovery IDs are introduced.
- Rejections/aliases are not reliably reconstructable after restart.
- Partial approval can change graph topology without an event record.
- Relationships can reference provisional characters unless promotion/merge ordering is explicit.
```mermaid
flowchart TD
A["Parsed scene result"] --> B["Provisional discovery row"]
B --> C["Graph provisional node"]
C --> D{"Author decision"}
D -->|Create| E["Canonical record created"]
D -->|Link| F["Existing canonical record linked"]
D -->|Alias/Merge| G["Discovery merged into target"]
D -->|Ignore| H["Discovery rejected"]
E --> I["NodePromotedToCanonical event"]
F --> I
G --> J["IdentityMerged event"]
H --> K["DiscoveryRejected event"]
```
## 16. Proposed Phase 21 architecture
```mermaid
flowchart TD
A["Story Intelligence analysis worker"] --> B["Persist discoveries and state"]
B --> C["SQL Server"]
C --> D["Ordered visualisation event table"]
D --> E["Story Intelligence event publisher"]
E --> F["SignalR hub"]
F --> G["Browser client"]
G --> H["Snapshot loader"]
G --> I["Event applier"]
H --> J["Graphology graph model"]
I --> J
J --> K["Sigma.js renderer"]
G --> L["Panels, feeds, filters, evidence drawer"]
```
Principles:
- SQL Server is authoritative.
- SignalR is transport only.
- Browser never analyses manuscript.
- Graph DTOs are versioned and separate from database DTOs.
- The same graph engine supports live analysis and completed Story Map.
## 17. Proposed server-side components
- `StoryIntelligenceVisualisationSnapshotService`: builds snapshot for run/book/project.
- `StoryIntelligenceVisualisationEventRepository`: persists ordered events.
- `StoryIntelligenceDiscoveryRepository`: persists provisional nodes/edges/evidence.
- `StoryIntelligenceVisualisationPublisher`: publishes stored events to SignalR after commit.
- `StoryIntelligenceGraphHub`: job/book-scoped groups with explicit access validation.
- `StoryIntelligenceGraphController` or API controller: snapshot and missed-event endpoints.
- Integration points in `PersistedStoryIntelligenceRunner` and import services to write discovery/event rows.
## 18. Proposed browser-side components
```mermaid
flowchart LR
A["Razor page"] --> B["graph bootstrap"]
B --> C["snapshot API client"]
B --> D["SignalR client"]
C --> E["graph adapter"]
D --> E
E --> F["Graphology"]
F --> G["Sigma renderer"]
E --> H["activity feed"]
E --> I["evidence drawer"]
E --> J["filters/search"]
K["layout worker"] --> F
```
Suggested modules:
- `story-graph-client`
- `story-graph-adapter`
- `story-graph-renderer`
- `story-graph-layout-worker`
- `story-graph-panels`
- `story-graph-api`
## 19. Proposed API contracts
Snapshot:
```http
GET /api/story-intelligence/runs/{runId}/graph-snapshot?scope=book
```
Response shape:
```json
{
"runId": 123,
"projectId": 10,
"bookId": 20,
"lastSequence": 456,
"nodes": [],
"edges": [],
"activeStage": "SceneIntelligence",
"permissions": { "canReview": true },
"generatedUtc": "2026-07-11T00:00:00Z"
}
```
Missed events:
```http
GET /api/story-intelligence/runs/{runId}/graph-events?afterSequence=456
```
## 20. Proposed SignalR message contracts
Hub route proposal: `/hubs/story-intelligence-graph`
Client invokes:
- `WatchRunGraph(runId, afterSequence)`
- `LeaveRunGraph(runId)`
Server sends:
```json
{
"runId": 123,
"sequence": 457,
"eventId": "01J...",
"type": "NodeDiscovered",
"occurredUtc": "2026-07-11T00:00:00Z",
"payloadVersion": 1,
"payload": {}
}
```
## 21. Proposed event categories
- Analysis lifecycle
- Progress and stage
- Scene/chapter lifecycle
- Discovery node lifecycle
- Relationship/edge lifecycle
- Evidence/confidence updates
- Merge/promotion/rejection
- Review/approval
- Failure/cancellation/recovery
## 22. Proposed implementation sequence
1. Add durable event/discovery design and SQL script.
2. Add read-only snapshot service over existing canonical plus run-result data.
3. Add event writer around current runner progress and scene result persistence.
4. Add graph hub with authorised job groups and replay-after-sequence.
5. Build browser graph adapter with mocked renderer data from real snapshots.
6. Integrate Sigma.js/Graphology and layout worker.
7. Add live progress/discovery events.
8. Add approval/promotion events for scene and entity imports.
9. Add completed Story Map route reusing the graph engine.
## 23. Database changes likely to be required
Follow existing SQL script convention in `PlotLine/Sql`, where the current latest script is `134_Phase20AV_ArchivedBookProjectPermanentDeletion.sql`.
Likely future script:
- `135_Phase21B_StoryIntelligenceVisualisationEvents.sql`
Likely tables:
- `StoryIntelligenceDiscoveries`
- `StoryIntelligenceDiscoveryEvidence`
- `StoryIntelligenceDiscoveryAliases`
- `StoryIntelligenceVisualisationEvents`
- Possibly `StoryIntelligenceGraphSnapshots` if snapshot materialisation is needed later.
Conventions to follow:
- Idempotent `IF OBJECT_ID` / `COL_LENGTH` checks.
- `CREATE OR ALTER PROCEDURE`.
- `PK_`, `FK_`, `DF_`, `CK_`, `IX_` naming.
- User/project/book isolation columns and indexes.
- Do not alter historical scripts.
## 24. Risks and mitigations
| Risk | Mitigation |
|---|---|
| Volatile batch state prevents recovery | Persist batch/run mappings and review decisions or derive them from durable pipeline rows. |
| SignalR events missed | Store ordered events and replay after sequence. |
| Graph too dense | One-book scope, filters, clustering, label level-of-detail. |
| Provisional/canonical ID churn | Add discovery IDs and explicit promotion/merge events. |
| Manuscript data leakage | Snapshot/hub access checks against project/book/run. |
| Package integration churn | Decide MVC frontend build approach before adding Sigma/Graphology. |
| CDN/CSP concerns | Prefer bundled assets for production. |
| Existing progress workflow regression | Add visualisation beside current progress page first. |
## 25. Decisions still required
- Whether Phase 21B introduces an MVC `package.json` and bundler.
- Whether to persist batch state or replace batch IDs with durable book/run pipeline IDs.
- Exact discovery/event SQL schema.
- Whether to keep Cytoscape for Relationship Map or migrate it later to the new engine.
- Retention policy for raw source excerpts and visualisation events.
- Whether snapshots are generated on demand or materialised.
## 26. Files inspected
Primary files inspected:
- `PlotLine/Program.cs`
- `PlotLine/Hubs/StoryIntelligenceHub.cs`
- `PlotLine/Hubs/WordCompanionFollowHub.cs`
- `PlotLine/Controllers/OnboardingController.cs`
- `PlotLine/Controllers/RelationshipMapController.cs`
- `PlotLine/Services/OnboardingService.cs`
- `PlotLine/Services/OnboardingStoryIntelligenceService.cs`
- `PlotLine/Services/PersistedStoryIntelligenceWorker.cs`
- `PlotLine/Services/PersistedStoryIntelligenceRunner.cs`
- `PlotLine/Services/StoryIntelligenceProgressNotifier.cs`
- `PlotLine/Services/StoryIntelligenceImportCommitService.cs`
- `PlotLine/Services/SceneImportResolver.cs`
- `PlotLine/Services/StoryIntelligenceCharacterImportService.cs`
- `PlotLine/Services/StoryIntelligenceLocationImportService.cs`
- `PlotLine/Services/ProjectAccessFilter.cs`
- `PlotLine/Services/ProjectAccessServices.cs`
- `PlotLine/Data/StoryIntelligenceResultRepository.cs`
- `PlotLine/Data/StoryIntelligenceRepository.cs`
- `PlotLine/Data/StoryIntelligencePipelineRepository.cs`
- `PlotLine/Data/Repositories.cs`
- `PlotLine/Models/StoryIntelligenceModels.cs`
- `PlotLine/Models/StoryIntelligencePersistenceModels.cs`
- `PlotLine/ViewModels/OnboardingViewModels.cs`
- `PlotLine/Views/Onboarding/StoryIntelligenceProgress.cshtml`
- `PlotLine/Views/Onboarding/StoryIntelligenceReview.cshtml`
- `PlotLine/Views/RelationshipMap/Index.cshtml`
- `PlotLine/Views/Shared/_Layout.cshtml`
- `PlotLine/wwwroot/js/story-intelligence-progress.js`
- `PlotLine/bundleconfig.json`
- `PlotDirector.WordCompanion/package.json`
- SQL scripts `001`, `004`, `005`, `006`, `007`, `020`, `033`, `039`, `042`, `046`, `088`, `093`, `111`, `114`, `115`, `117`, `121`, `123`, `124`, `127` through `134`
- `PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Visualisation_Design.md`
- `PlotLine/Docs/Architecture/DeploymentConfiguration.md`
## 27. Recommended next step for Phase 21B
Phase 21B should be a narrow foundation phase:
1. Design and add durable graph discovery/event storage for one active `StoryIntelligenceRun`.
2. Add authorised snapshot and event replay endpoints.
3. Add a SignalR graph hub that joins one run group after access validation.
4. Emit lifecycle, stage, scene-start, scene-complete and basic discovery events from the existing persisted runner.
5. Do not replace the current progress page yet; add the visualisation as an optional "Watch Story Intelligence" experience backed by the same SQL state.
Reference in New Issue View Git Blame Copy Permalink