PlotDirector/docs/story-intelligence/import-visualisation-preservation.md

Story Intelligence Import Visualisation Preservation Audit

1. Purpose and History

This document preserves the current Story Intelligence import visualisation before any decision to remove it from the active import workflow. The intended future boundary is:

IMPORT / STORY INTELLIGENCE
    -> persisted results and domain progress
    -> read-only visualisation

The visualisation must not cause extra analysis, extraction, persistence, AI calls, or image generation just to make the display interesting.

This audit is based on source inspection only. No application code, SQL, or runtime data was changed. Screenshots were not captured because doing that safely would require an authenticated development session and existing import data; the existing design reference is /srv/repos/PlotDirector/PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Visualisation_Design.png.

2. Current UX

There are two related experiences:

• Active onboarding progress page: /srv/repos/PlotDirector/PlotLine/Views/Onboarding/StoryIntelligenceProgress.cshtml. This is the production-facing progress page. It shows "Reading your manuscript", a plain import progress card, chapters read, scenes read, failures, elapsed time, estimated remaining, current chapter, current scene, latest scene summary, recent discoveries, cancel, continue in background, replay, and review actions.
• Development experience visualisation: /srv/repos/PlotDirector/PlotLine/Views/Development/StoryIntelligenceExperience.cshtml. This is admin/development-only and uses the full visual theatre: dark full-screen shell, header status strip, circular progress ring, left-side discovery/activity panels, central "living story view", character/location stage, SVG relationship lines, scene ribbon, timeline, right-side knowledge/observation/relationship panels, replay controls, and diagnostics.

The production page explicitly says processing can continue after the browser tab is closed. The full visualisation is reachable from the production progress page by a "Replay Story Intelligence" link when an import session exists, but the controller route is protected by [Authorize(Policy = "AdminOnly")] and environment.IsDevelopment().

The distinctive visual language lives mostly in /srv/repos/PlotDirector/PlotLine/wwwroot/css/story-intelligence-experience-prototype.css: dark navy/blue-gold stage, blurred translucent panels, conic progress ring, animated SVG connection lines, circular character/location portraits, text-only evidence cards when portraits are not trusted, glowing timeline markers, compact side panels, and reduced-motion handling. Client behaviour is in /srv/repos/PlotDirector/PlotLine/wwwroot/js/story-intelligence-experience-prototype.js: it polls snapshots, reconciles nodes, animates progress, preloads images, draws collision-aware connections, updates scene ribbon/timeline, and opens insight detail cards.

3. Architecture

The active import pipeline is separate from the full visualisation:

• /srv/repos/PlotDirector/PlotLine/Services/OnboardingStoryIntelligenceService.cs queues one persisted Story Intelligence run per included chapter.
• /srv/repos/PlotDirector/PlotLine/Services/PersistedStoryIntelligenceWorker.cs is the background service that repeatedly asks the runner to process pending runs.
• /srv/repos/PlotDirector/PlotLine/Services/PersistedStoryIntelligenceRunner.cs performs the expensive work: one Chapter Structure call, one Scene Intelligence call per detected scene, JSON repair when needed, validation, persistence, progress publication, Story Memory processing, and Character Intelligence profile updates.
• /srv/repos/PlotDirector/PlotLine/Services/StoryIntelligenceVisualisationSnapshotService.cs builds read models from persisted run/scene results and Story Memory assignments.
• /srv/repos/PlotDirector/PlotLine/Controllers/StoryIntelligenceVisualisationController.cs exposes read-only JSON snapshots for authenticated users.
• /srv/repos/PlotDirector/PlotLine/Controllers/DevelopmentController.cs serves the full experience page and its replay/rebuild action, development-only.
• /srv/repos/PlotDirector/PlotLine/Hubs/StoryIntelligenceHub.cs plus /srv/repos/PlotDirector/PlotLine/Services/StoryIntelligenceProgressNotifier.cs provide reusable SignalR progress updates.

The full visualisation is mostly a read-only observer when simply opened. The dangerous coupling is that Story Memory and illustration-demand machinery can be used by the import pipeline and by the development replay/rebuild path, so "visualisation support" has grown into durable derived state.

4. Component Inventory

Component	Classification	Notes
/srv/repos/PlotDirector/PlotLine/Controllers/StoryIntelligenceVisualisationController.cs	VISUALISATION ONLY	Read-only snapshot API for run/import session visualisation.
/srv/repos/PlotDirector/PlotLine/Controllers/DevelopmentController.cs methods StoryIntelligenceExperience, RebuildStoryIntelligenceExperience, diagnostics	VISUALISATION ONLY / UNCERTAIN	Development-only UI and rebuild tools. Rebuild calls Story Memory and can create demand records.
/srv/repos/PlotDirector/PlotLine/Views/Development/StoryIntelligenceExperience.cshtml	VISUALISATION ONLY	Full-screen experience/replay UI.
/srv/repos/PlotDirector/PlotLine/wwwroot/js/story-intelligence-experience-prototype.js	VISUALISATION ONLY	Snapshot polling, animation, layout, drawing, replay controls.
/srv/repos/PlotDirector/PlotLine/wwwroot/css/story-intelligence-experience-prototype.css	VISUALISATION ONLY	Full visual design system for the prototype.
/srv/repos/PlotDirector/PlotLine/ViewModels/StoryIntelligenceExperiencePrototypeViewModels.cs	VISUALISATION ONLY	Snapshot DTOs: scenes, characters, locations, assets, relationships, diagnostics, image resolution.
/srv/repos/PlotDirector/PlotLine/Services/StoryIntelligenceVisualisationSnapshotService.cs	DO NOT REMOVE UNTIL DECOUPLED	Builds visualization snapshots and contains visual-only inference/story-memory logic.
/srv/repos/PlotDirector/PlotLine/Services/StoryIntelligenceExperiencePrototypeData.cs	VISUALISATION ONLY	Simulation data.
/srv/repos/PlotDirector/PlotLine/wwwroot/images/story-intelligence/prototype/*	VISUALISATION ONLY	Static fallback prototype assets.
/srv/repos/PlotDirector/PlotLine/Views/Onboarding/StoryIntelligenceProgress.cshtml	IMPORT PIPELINE / SHARED UX	Active import status screen; contains replay link.
/srv/repos/PlotDirector/PlotLine/wwwroot/js/story-intelligence-progress.js	SHARED INFRASTRUCTURE	Production progress SignalR client for onboarding and legacy dashboards.
/srv/repos/PlotDirector/PlotLine/Hubs/StoryIntelligenceHub.cs	SHARED INFRASTRUCTURE	Authenticated SignalR hub for progress.
/srv/repos/PlotDirector/PlotLine/Services/StoryIntelligenceProgressNotifier.cs	SHARED INFRASTRUCTURE	Broadcasts persisted run progress events.
/srv/repos/PlotDirector/PlotLine/Services/PersistedStoryIntelligenceRunner.cs	IMPORT PIPELINE / STORY INTELLIGENCE	AI execution, persistence, progress, Story Memory, Character Intelligence.
/srv/repos/PlotDirector/PlotLine/Docs/AI/Chapter-Structure-Prompt-V2.md	STORY INTELLIGENCE	Boundary prompt; explicitly excludes story entities/observations.
/srv/repos/PlotDirector/PlotLine/Docs/AI/Scene-Prompt-V2.md	STORY INTELLIGENCE / VISUALISATION COUPLED	Main scene schema includes many fields consumed by import and visualisation, including characterAppearance.
/srv/repos/PlotDirector/PlotLine/Services/StoryMemoryServices.cs	STORY INTELLIGENCE / DO NOT REMOVE UNTIL DECOUPLED	Durable derived scene memory and illustration assignment/demand.
/srv/repos/PlotDirector/PlotLine/Data/StoryMemoryRepository.cs	STORY INTELLIGENCE / SHARED INFRASTRUCTURE	Persistence for Story Memory. Note: currently uses inline SQL.
/srv/repos/PlotDirector/PlotLine/Services/IllustrationLibraryServices.cs	SHARED INFRASTRUCTURE	Reusable image library and image generation worker integration.
/srv/repos/PlotDirector/PlotLine/Services/StoryIntelligenceIllustrationMatchingService.cs	VISUALISATION COUPLED / UNCERTAIN	Older project-level assignment/demand based on visualisation DTOs.
/srv/repos/PlotDirector/PlotLine/Data/StoryIntelligenceIllustrationMatchingRepository.cs	VISUALISATION COUPLED / UNCERTAIN	Project-level visualisation assignment/demand tables.
SQL 114, 115, 117, 121, 127, 137, 142, 144, 145, 146, 147	IMPORT/STORY INTELLIGENCE	Persisted runs, result storage, pipeline state, Story Memory, character context, safe image policy.
SQL 135, 136, 138, 139, 140, 141	IMAGE/VISUALISATION COUPLED	Illustration library, matching, demand, audit.

5. Event Catalogue

Event/update	Example	Source	Persisted?	AI derived?	Required elsewhere?
Import started	"Story Intelligence has started reading this chapter."	PersistedStoryIntelligenceRunner.PublishAsync	Run progress fields yes	No	Yes, production progress
Chapter started	"PlotDirector is finding the scene boundaries..."	Runner before Chapter Structure call	Run progress fields yes	No	Yes
Scene boundaries detected	"Detected 4 suggested scenes."	Runner after Chapter Structure response	Yes, chapter result and run counts	Yes	Yes
Scene analysis started	"Reading scene 2."	Runner before Scene Intelligence call	Run progress fields yes	No	Yes
Scene analysis completed	"Scene 2 analysed."	Runner after scene persistence	Yes, scene result and run counts	Yes	Yes
Latest scene summary	Scene short summary	Scene prompt summary.short	Yes in ParsedJson; SignalR transient copy	Yes	Yes, review/progress
Recent discoveries	"Possible character found: Beth"	Runner derives from parsed characters/locations/assets	SignalR transient; source persisted	Yes source, No derivation	Nice-to-have
Failure	"Scene 2 needs attention."	Runner catch blocks	Yes	Sometimes	Yes
Cancellation	"Analysis was cancelled."	Runner cancellation path	Yes	No	Yes
Completion	"Chapter analysis complete."	Runner completion path	Yes	No	Yes
Snapshot change token	hash of run/result state	Visualisation snapshot service	No	No	Visualisation only
Current scene/window	latest persisted scene result	Visualisation snapshot service	No	No	Visualisation only
Character node	name, role, relevance, weight	Visualisation snapshot service from scene characters plus story memory	No as DTO; source persisted	Mostly yes	Review/import uses source, not node
Location node	active location image/text	Snapshot service from scene locations/setting	No as DTO; source persisted	Mostly yes	Review/import uses source
Asset state	important assets, often not rendered stage nodes	Snapshot service from scene assets	No as DTO; source persisted	Mostly yes	Review/import uses source
Relationship lines	labels/states/weights	Snapshot service from scene relationships and visual memory inference	Some source persisted; inferred visual memory no	Mixed	Explicit relationships yes; hard-coded visual inference no
Knowledge threads	questions and knowledge changes	Snapshot service from scene JSON	No as DTO; source persisted	Yes	Knowledge import uses knowledge changes
Observations	title/detail evidence cards	Snapshot service from scene.Observations	No as DTO; source persisted	Yes	Unclear/no strong current consumer outside visualisation
Image demand status	demand recorded/queued/satisfied	Story Memory or old matching service	Yes	No direct text AI; may trigger image generation later	Images useful to visualisation, maybe library
Image generated	library item status generated	Illustration worker/library	Yes	Image model call	Shared library, but demand originates from SI memory/visualisation coupling

6. Visualisation-Driven Intelligence

Information already required by PlotDirector:

• Scene boundaries, scene summaries, scene references, and detected scene counts are core import outputs.
• Characters, aliases, locations, assets, relationships, knowledge changes, metrics, source limits, and continuity signals are used by review/import services.
• Character aliases and known-character context are used by the prompt to reduce identity drift.
• Character appearance has a downstream consumer: Story Memory and illustration matching. It is not required for scene creation itself, but it is required by the current image assignment/demand system.

Information that appears primarily visualisation-oriented:

• StoryIntelligenceExperience* DTOs, image resolution diagnostics, current visual scene window, display weights, stage labels, timeline labels, feed items, SVG connection labels, replay indexes, panel counts, and change tokens.
• Snapshot-service in-memory StoryMemory used to carry characters, point-of-view, and relationships through the visual replay.
• Hard-coded/thematic inference in StoryIntelligenceVisualisationSnapshotService, including Beth/Grace/Elen family inference and summary-language inference. This is not the same as canonical relationship import.
• scene.Observations as a displayed "Story observations" panel. It is generated by Scene Prompt V2 and persisted in scene JSON, but source searches found no strong downstream consumer outside visualisation/Story Memory evidence extraction.

Information with uncertain value:

• Character appearance and deterministic appearance inference. It is useful only if reusable illustrations remain part of Story Intelligence or future passive visualisation.
• Durable Story Memory entities. They may become useful for future context/intent or import review, but current normal import review services still primarily read persisted scene results rather than requiring Story Memory.

7. AI Cost Attribution

AI operation	Required by Import	Required by SI	Required by Visualisation	Could remove?
Chapter Structure prompt	Yes	Yes	No	No
Scene Intelligence prompt per scene	Yes	Yes	Partially: it includes fields the visualisation displays	Not wholesale
JSON repair retry	Yes for reliability	Yes	No	No, unless schema/output complexity is reduced enough to lower failures
Character Intelligence profile update	No for basic import	Yes/current architecture	No direct	Possibly defer or make opt-in
Story Memory deterministic processing	No for basic import	Partially/current architecture	Yes for image/visual continuity	Decouple before removing
Illustration library image generation	No for manuscript import	No for textual SI	Yes/partially for visual experience	Yes for simplified import
Development replay/rebuild	No	No	Yes	Yes, keep as offline/admin-only if needed

Answer: the visualisation partially increases AI cost. Opening the snapshot endpoint does not itself call the text AI. But the current scene prompt asks for characterAppearance specifically for illustration matching, Story Memory can queue image demand during import, and development rebuild can create demand records. If automatic image generation is enabled and approval is not required, those demands can lead to image-generation calls.

8. Evidence and Observation System

The "Beth pours tea" style evidence comes from Scene Prompt V2 and the persisted SceneIntelligenceScene JSON, not from a separate visualisation prompt. Relevant fields include:

• characters[].actions, characters[].notes
• relationships[].relationshipSignal, relationships[].evidence
• knowledgeChanges[].evidence
• questionsRaised[].evidence
• observations[].description, observations[].predicate, observations[].objectName, observations[].evidence
• characterAppearance[].evidence

Representation:

• Models are in /srv/repos/PlotDirector/PlotLine/Models/StoryIntelligencePersistenceModels.cs and /srv/repos/PlotDirector/PlotLine/Models/StoryIntelligenceImportPreviewModels.cs.
• Scene results are stored by /srv/repos/PlotDirector/PlotLine/Data/StoryIntelligenceResultRepository.cs via dbo.StoryIntelligenceSceneResult_Save.
• The visualisation reads ParsedJson, deserializes SceneIntelligenceScene, and projects observations into StoryIntelligenceExperienceTextItem.

Consumers:

• Character/location/asset/relationship/knowledge import services read scene JSON.
• Story Memory reads characters, appearance, locations, assets, and relationships.
• The visualisation explicitly renders observations. No clear normal user-facing consumer of scene.Observations was identified outside the visualisation-style experience.

Classification:

• CORE STORY INTELLIGENCE: scene summary, characters, relationships, locations, assets, knowledge changes, source limits.
• USEFUL DERIVED INTELLIGENCE: known-character aliases, deterministic appearance facts if image matching stays, Story Memory identities if future context/intent uses them.
• VISUALISATION TELEMETRY: visual DTOs, panel state, change token, replay index, display weights, latest-discovery phrasing, activity feed.
• UNUSED / AI EXHAUST: observations appears likely unless a downstream product consumer is added.

9. Image-Generation Lifecycle

Current route:

Scene result saved
  -> StoryMemoryService.ProcessSceneResultAsync
  -> Upsert durable memory characters/locations/assets/relationships
  -> ResolveIllustrationsAsync
  -> match approved/generated illustration library items
  -> if none, upsert demand
  -> if QueueDemand enabled, create/queue planned library item
  -> IllustrationGenerationWorker claims queued item
  -> OpenAI image generation
  -> uploaded file persisted to illustration library
  -> future snapshots/replays can reuse assignment/library item

Findings:

• Existing library images are checked first. Story Memory filters active, non-archived, non-fallback, generated/approved items and scores hard compatibility before queueing demand.
• Duplicate demand is limited by ImportSessionID + EntityType + ArchetypeKey; alias drift can still create separate canonical identity keys if alias resolution has not merged them.
• Image demands are tied to Story Memory temporary/import-session entities, not normal canonical Character records.
• The generated image is stored in the Story Intelligence illustration library, not automatically assigned to the canonical Characters table or normal character gallery/avatar.
• Subsequent chapters in the same import can reuse Story Memory assignments and library items.
• Subsequent imports can reuse the global library item if compatible, but per-import Story Memory assignments are separate.
• Failed/partial imports can leave generated or queued library items and demand records because the image library is independent of final import commit.
• The visualisation itself benefits from images, but normal manuscript import does not require generated images.

Every route that can incur image generation cost:

• Story Memory during the active runner, when AutomaticImageGenerationDuringImportEnabled == true, RequireImageGenerationApproval == false, and caps allow queueing.
• Development replay/rebuild path if called with queueDemand=true, followed by the image worker.
• Admin illustration library/starter/regeneration workflows outside import.
• Older StoryIntelligenceIllustrationMatchingService can upsert planned items/queue demand from visualisation DTOs; it should be treated as visualisation-coupled until proven otherwise.

10. SignalR and Reusable Infrastructure

Preserve:

• /srv/repos/PlotDirector/PlotLine/Hubs/StoryIntelligenceHub.cs
• /srv/repos/PlotDirector/PlotLine/Services/StoryIntelligenceProgressNotifier.cs
• /srv/repos/PlotDirector/PlotLine/wwwroot/js/story-intelligence-progress.js
• StoryIntelligenceRunProgressEvent

These are not obsolete visualisation code. They power the production progress screen and are a natural fit for the simplified unattended import.

The full visualisation does not use SignalR directly; it polls read-only snapshot endpoints. The onboarding/progress UI uses SignalR and should remain as lightweight progress infrastructure.

11. Preservation Inventory

PRESERVE AS ACTIVE INFRASTRUCTURE:

• Persisted run/result pipeline, background worker, queue/claim/complete/fail/cancel stored procedures.
• SignalR hub/notifier and production progress page.
• Story Intelligence review/import services for scenes, characters, locations, assets, relationships, and knowledge.
• Email queue infrastructure, although no completion email for Story Intelligence was found.
• Illustration library infrastructure if retained for user/admin image workflows.

PRESERVE FOR FUTURE VISUALISATION:

• Full visual design ideas from StoryIntelligenceExperience.cshtml, story-intelligence-experience-prototype.css, and story-intelligence-experience-prototype.js.
• Read-only snapshot endpoint pattern.
• Replay controls and ability to reconstruct from persisted results.
• Timeline/ribbon, relationship-line drawing, text fallback cards, insight detail panels, and diagnostics.
• Static fallback prototype assets.

SAFE TO REMOVE after documentation and route cleanup:

• Development-only simulation data and static prototype images, if no future visual replay is planned.
• Replay link from active onboarding, if the full visualisation is removed from the workflow.
• Visualisation-specific DTOs and snapshot endpoints, after confirming no external client uses them.

DO NOT REMOVE UNTIL DECOUPLED:

• Story Memory processing from the active runner, because it is currently intertwined with illustration assignment/demand and durable derived state.
• characterAppearance in the scene prompt until image matching/Story Memory decisions are made.
• Old project-level StoryIntelligenceIllustrationMatchingService and repository until references and tables are audited for production use.
• Any SQL tables/procs in the 135-147 range without mapping them to current consumers.

12. Future Visualisation Boundary

A future visualisation should subscribe only to:

• ImportSessionStatusChanged
• ChapterRunQueued
• ChapterRunStarted
• ChapterRunCompleted
• SceneResultPersisted
• ReviewStageAvailable
• ImportFailed
• ImportCompleted
• ImageAssignmentAvailable only when an image exists naturally

Each event should contain IDs, counts, status, display-safe summary, and links to persisted records. It should not contain prompt-only display demands. Visualisation should load data through read-only queries against persisted scene results, canonical import previews, and existing image assignments.

The future visualisation must not:

• Add prompt fields.
• Trigger Story Memory rebuild.
• Queue image demand.
• Persist visual-only inferred relationships.
• Request JSON repair.
• Cause additional AI calls.

13. Simplified Unattended-Import Requirements

Already exists:

• Durable pending/running/completed/failed runs.
• Background processing independent of the browser.
• Persisted progress/status fields.
• Cancellation.
• Production progress page that says users can leave and return later.
• Email queue infrastructure and user email addresses.

Missing or incomplete:

• Story Intelligence completion email was not found.
• A single authoritative import-session completion event/worker action should queue email once.
• Production UX should return users directly to review/import results, not development replay.
• AI/image cost gates should be explicit before queueing text/image work.
• Story Memory and image demand should be optional/deferred for unattended import.
• Progress should rely on persisted pipeline state, not in-browser visual state.

14. Screenshots and References

• Existing design reference: /srv/repos/PlotDirector/PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Visualisation_Design.png
• Experience design document: /srv/repos/PlotDirector/PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Experience_Design_v2.md
• Active progress page: /srv/repos/PlotDirector/PlotLine/Views/Onboarding/StoryIntelligenceProgress.cshtml
• Full experience page: /srv/repos/PlotDirector/PlotLine/Views/Development/StoryIntelligenceExperience.cshtml
• Full experience CSS: /srv/repos/PlotDirector/PlotLine/wwwroot/css/story-intelligence-experience-prototype.css
• Full experience JS: /srv/repos/PlotDirector/PlotLine/wwwroot/js/story-intelligence-experience-prototype.js

15. Known Good Design Ideas Worth Preserving

• "Living story view" as a stage, not a table.
• Progress ring plus plain status numbers.
• Character/location prominence based on relevance and point of view.
• Relationship lines with visual tone for support/conflict/uncertainty.
• Scene ribbon and timeline to show motion through the manuscript.
• Text-only evidence cards when image confidence is low.
• Replay from persisted results.
• Compact diagnostic panel for engineering trust.
• Respect for reduced motion.
• "Return to Import" route back to the actual workflow.

Recommended Subsequent Task

Do not remove anything yet. The next implementation task should:

1. Remove the development replay link from the active onboarding flow or hide it behind an admin/development flag.
2. Add a completion-email path driven by persisted import-session completion.
3. Disable automatic image demand during normal import by default and require explicit user/admin approval before image generation.
4. Split Story Memory into two modes: minimal domain memory required by import/context, and optional visual/image memory.
5. Trim Scene Prompt V2 so observations and characterAppearance are included only when a downstream product feature genuinely consumes them.
6. Replace visualisation DTO dependencies in image matching with domain DTOs, or retire the old project-level matching service.
7. Keep a passive snapshot/replay prototype that reads persisted results only.