diff --git a/docs/story-intelligence/PlotDirector-Story-Intelligence-Refactor-Plan.md b/docs/story-intelligence/PlotDirector-Story-Intelligence-Refactor-Plan.md new file mode 100644 index 0000000..4285ecc --- /dev/null +++ b/docs/story-intelligence/PlotDirector-Story-Intelligence-Refactor-Plan.md @@ -0,0 +1,1067 @@ +# PlotDirector Story Intelligence Refactor Plan + +## Purpose + +This document records the agreed direction for the next Story +Intelligence refactor. + +The current system is producing genuinely useful analysis, particularly +canonical scene boundaries, structural summaries, generated scene titles +and scene metrics. The remaining work is primarily about establishing +stronger context and canonical identity earlier, improving downstream +analysis quality, and replacing the long linear review wizard with a +much lighter exception-based review model. + +The guiding principle is: + +> **Establish reliable context as early as possible, preserve narrative +> progression, and only ask the author to resolve information +> PlotDirector cannot safely determine itself.** + +A second equally important principle is: + +> **Later analysis must consume facts PlotDirector already knows rather +> than independently rediscovering them.** + +The work should be implemented in dependency order. Do not begin by +redesigning the review UI around data that is still ambiguous or poorly +contextualised. + +------------------------------------------------------------------------ + +# Core Architectural Principles + +## Canonical scenes remain the structural foundation + +The existing Core Import architecture remains authoritative: + +- Chapters are established first. +- Canonical scenes are detected and created. +- `SceneManuscriptSources` contains the canonical source text. +- Structural summaries are generated once and reused. +- Optional descriptive scene titles are generated in the same + structural call. +- Word markers use canonical ChapterIDs and SceneIDs. +- Story Intelligence enriches those existing scenes rather than + redetecting or recreating them. + +Do not reintroduce duplicate scene-boundary analysis or duplicate scene +summaries. + +## POV must become early structural context + +POV is sufficiently important that it should be established during +structural analysis, before detailed Story Intelligence enrichment. + +Knowing the POV character allows PlotDirector to resolve otherwise +ambiguous references such as: + +- Narrator +- I / me / my +- my bedroom +- my car +- my mother +- my handbag + +This in turn improves character identity, locations, assets, +relationships, knowledge and emotional attribution. + +## CharacterID represents physical identity + +A canonical `CharacterID` represents the actual person in the story, +regardless of the names, descriptions or aliases used for that person. + +For example, all of the following may ultimately refer to one +CharacterID: + +- Margaret Grant +- Maggie +- Miss Grant +- Rusty +- Grace +- Mrs Jackson + +Similarly, an initially unidentified character such as `The Man in Grey` +may later be established as `Victor Harrington`. + +Canonical identity is important not only for naming but for physical +continuity. The same CharacterID cannot normally occupy two incompatible +locations at the same time merely because different aliases are being +used. + +## Alias identity must not erase narrative history + +When two character identities are merged, PlotDirector must establish +the physical/canonical truth without destroying the way the character +was presented in the manuscript. + +If `The Man in Grey` is later resolved as Victor Harrington: + +- Victor becomes the canonical CharacterID. +- `The Man in Grey` remains an alias/reference identity. +- Earlier scene evidence that used `The Man in Grey` remains + preserved. +- Knowledge about that identity can vary by character and scene. + +The existing Knowledge system is responsible for facts such as: + +`Maggie learns that The Man in Grey is Victor Harrington` + +There is no need for a parallel identity-reveal event system if the +existing Knowledge model can express this. + +## Relationships are progressive, not static labels + +This is critical. + +A canonical relationship may persist between two CharacterIDs while its +**state changes through the story**. + +For example, Maggie and Rob may progress through states such as: + +- strangers / first meeting +- acquaintances +- friendship +- attraction +- romantic interest +- couple +- engaged + +PlotDirector must not flatten this into a single final relationship type +and lose the progression. + +The relationship model must preserve: + +1. **Canonical relationship identity** between the two CharacterIDs. +2. **Relationship events / state changes** tied to canonical SceneIDs. +3. **Evidence** supporting each change. +4. The ability to determine the relationship state at any point in the + story. + +A later conclusion that Maggie and Rob are a couple must not rewrite +earlier scenes as though they were already a couple. + +Story Intelligence should accumulate evidence over the book and identify +meaningful progression. + +------------------------------------------------------------------------ + +# Stage 1 - Establish POV as Structural Context + +POV should be determined during the existing chapter-level structural +analysis. + +## Data model + +POV ultimately belongs to the Scene because mixed-POV chapters exist. + +However, each chapter may have a strong **default POV** which is used to +seed its scenes. + +The desired model is: + +`Chapter default POV -> scene inheritance -> explicit per-scene override` + +The chapter default must not permanently override individual scene +values. + +## Structural analysis + +The existing chapter-level structural AI call should attempt to +identify: + +- default chapter POV; +- confidence/evidence; +- per-scene POV override where a change is clearly detected. + +Explicit manuscript cues such as: + +`Narrated by Maggie` + +should be treated as extremely strong evidence. + +Do not add separate AI calls solely for POV if it can be returned by the +existing structural analysis call. + +## Core Import review + +The chapter review/structural flow should expose the proposed POV +compactly. + +For example: + +`Chapter 6 · Danger in the Dark POV: Maggie Grant` + +The author should normally do nothing. + +If POV cannot be established: + +`POV: Not detected` + +with a dropdown allowing the author to select a canonical character. + +For mixed chapters, structural analysis may propose per-scene overrides. + +## Scene persistence + +When canonical scenes are created: + +- scenes inherit the confirmed chapter POV by default; +- structural per-scene overrides are applied where appropriate; +- provenance should distinguish generated/inherited values from + explicit author changes. + +## Normal Chapter UI + +The Chapter page should eventually expose: + +`POV: Maggie` + +or: + +`POV: Mixed` + +and provide a convenience action such as: + +`Set POV for all scenes: [Character]` + +This must protect explicit author overrides appropriately. + +## Downstream rule + +Once a Scene has a known POV CharacterID, later Story Intelligence must +treat that as authoritative context. + +`Narrator` should not survive as a separate character candidate when the +narrator is already known. + +------------------------------------------------------------------------ + +# Stage 2 - Build Proper Character Identity Merge and Alias Support + +PlotDirector needs a reusable canonical character merge mechanism. + +This is not an import-only feature. + +## Required behaviour + +Given: + +`Character B -> merge into Character A` + +PlotDirector must consolidate all identity-dependent data onto Character +A. + +Investigate all current and future CharacterID references before +implementation. + +Likely affected areas include: + +- SceneCharacters +- character aliases +- relationships +- relationship events +- knowledge +- asset ownership/associations +- location associations where character-based +- evidence/provenance +- Story Intelligence review candidates +- other CharacterID-backed tables + +Do not simply copy the alias string and delete the old character. + +## Preserve provenance + +If the manuscript previously referred to Character B as +`The Man in Grey`, preserve that reference evidence against the +canonical character after merge. + +The canonical physical identity becomes Character A, while the +manuscript reference remains historically meaningful. + +## Normal Character UI + +The standard Character pages should support: + +`Merge with another character` + +This operation must use the same canonical merge service used by Story +Intelligence review. + +Do not create a special import-only merge implementation. + +## Safety principle + +False merges are worse than temporary duplicate identities. + +When identity is genuinely uncertain, PlotDirector should prefer to keep +identities separate and ask the author later. + +------------------------------------------------------------------------ + +# Stage 3 - Redefine Character Review as Mandatory Identity Resolution + +Characters are the one required Story Intelligence review prerequisite. + +The purpose is no longer: + +> Review every character found by AI. + +It becomes: + +> Resolve the character identities PlotDirector could not determine +> confidently. + +## Existing canonical characters are the anchor + +Story Intelligence must receive and respect: + +- canonical CharacterIDs; +- canonical names; +- known aliases; +- deterministic SceneCharacter mappings; +- Scene POV; +- scene evidence. + +## Automatic cases + +Examples that should normally require no human decision: + +- known canonical character matches; +- known aliases; +- punctuation variants such as `Mrs Patterson` / `Mrs. Patterson`; +- `Narrator` where Scene POV is known; +- obvious case/spacing/title normalisation. + +## Genuine review cases + +Examples: + +### Rusty + +Possible match: Maggie Grant. + +Choices: + +- Alias of Maggie Grant +- Keep as separate character +- Another existing character +- Ignore + +### Creepy Guy + +If recurring/significant, this may legitimately remain an unidentified +canonical character. + +Choices might include: + +- Keep as unidentified character +- Alias of existing character +- Ignore + +Do not discard unnamed characters merely because they lack a +conventional name. + +### Pat + +If identity is known but a useful core attribute cannot be determined: + +- Female +- Male +- Other/appropriate supported value +- Unknown + +Only ask where the information is genuinely useful and unresolved. + +## Review workload + +The UI should communicate something like: + +`91 identities resolved automatically` `9 need your help` + +not present 100 already-resolved characters for approval. + +## Gate + +Once required character ambiguities are resolved, the remainder of Story +Intelligence review becomes available independently. + +------------------------------------------------------------------------ + +# Stage 4 - Improve Story Intelligence Scene Context + +Detailed Story Intelligence currently appears to lose contextual +information that the chapter-level structural pass understands. + +Each canonical scene analysis should receive a compact context package +built from existing persisted data. + +At minimum investigate supplying: + +- BookID / ChapterID / SceneID +- chapter title +- canonical scene title +- `Scenes.StructuralSummary` +- full current `SceneManuscriptSources.SourceText` +- resolved Scene POV CharacterID/name +- structural Setting +- deterministic SceneCharacter mappings +- relevant canonical characters and aliases +- previous scene title + structural summary +- next scene title + structural summary +- Story Era +- Primary Locale + +If a concise chapter structural summary already exists, consider +including it. + +Do not resend the complete chapter for every scene unless evidence +proves that is necessary. + +The aim is to provide narrative surroundings cheaply. + +## Established facts are facts + +Later prompts should be explicitly told which supplied values are +canonical/authoritative and must not be rediscovered or competed with. + +Examples: + +- canonical SceneID +- POV +- structural summary +- canonical character identity +- existing aliases +- structural setting where established + +------------------------------------------------------------------------ + +# Stage 5 - Rework Location Discovery Around Structural Setting + +Investigate the existing structural Scene `Setting` information first. + +Current observations suggest it is often more realistic than the later +Location extraction. + +## Desired approach + +Use: + +`structural setting -> canonicalise -> additional significant location discovery -> book-wide consolidation` + +rather than treating every place-like noun as a new Location. + +## Contextual resolution + +Examples: + +`bedroom` + +should become something meaningful where context supports it: + +- Beth's bedroom +- Maggie's bedroom + +`Narrator's home` + +should resolve through Scene POV. + +Generic references such as: + +- road +- garage +- bedroom +- car park + +must not automatically become useless canonical Locations. + +However, generic-sounding locations can be significant. A car park may +be the site of a major story event. + +Therefore significance and context matter more than vocabulary alone. + +## Normalisation + +Resolve obvious variants such as: + +- garage / garages +- Mrs Patterson's house / Mrs. Patterson's house + +before author review where confidence is high. + +## Review + +Only meaningful uncertain candidates should require author attention. + +Scene title, chapter, structural summary and evidence should be shown so +the author has context. + +------------------------------------------------------------------------ + +# Stage 6 - Improve Asset Discovery Using Structural Evidence + +Asset Intelligence should consume what earlier structural analysis +already knows. + +The current six-chapter test missed Beth's Memory Tin even though +structural scene information recognised it. + +Investigate this specifically. + +## Asset significance evidence + +Potential signals include: + +- repeated mentions across scenes; +- presence in `StructuralSummary`; +- presence in generated/manual scene title; +- ownership references; +- involvement in knowledge/events; +- recurrence across chapters; +- plot significance. + +An object important enough to survive structural summarisation is a +useful signal that it may be a significant Asset. + +## Canonicalisation + +Resolve obvious aliases/variants such as: + +`Triumph TR6` `TR6` + +where evidence shows they are the same physical asset. + +As with characters, avoid unsafe merging when uncertain. + +------------------------------------------------------------------------ + +# Stage 7 - Make Relationships Progressive and Evidence-Based + +Relationship Intelligence must operate against canonical CharacterIDs. + +This stage must preserve **relationship progression over time**. + +## Canonical relationship + +There should be one canonical relationship identity for a pair of +characters where appropriate. + +For example: + +`Maggie <-> Rob` + +## Relationship events + +The relationship then accumulates scene-based events and state changes. + +Conceptual example: + +- Scene 12: first meaningful interaction +- Scene 24: friendship deepens +- Scene 39: mutual attraction becomes evident +- Scene 51: romantic relationship begins +- Scene 88: engagement + +The exact taxonomy should follow PlotDirector's existing +relationship/event model rather than inventing a parallel system +unnecessarily. + +## Do not flatten progression + +If the final relationship state is `Engaged`, earlier scenes must not be +rewritten as though Maggie and Rob were engaged throughout the book. + +PlotDirector should be able to determine: + +- current/final relationship state; +- relationship state at a specific scene/time; +- events that caused progression; +- evidence supporting those events. + +## Book-wide inference + +Do not classify the relationship independently from each scene. + +Use: + +`scene evidence -> canonical pair -> accumulate evidence -> relationship events/state progression` + +For example, repeated evidence of Maggie and Rosie: + +- hugs; +- presents; +- social outings; +- trust; +- repeated positive interaction; + +should support `Friendship` with reasonable confidence. + +Do not overclaim romantic/familial relationships without sufficient +evidence. + +## Review presentation + +Show: + +- canonical character pair; +- proposed relationship state/event; +- relevant chapter/scene; +- structural scene summary; +- concise supporting evidence; +- confidence where useful. + +For large books, support grouping/pagination. + +------------------------------------------------------------------------ + +# Stage 8 - Fix Knowledge Provenance and Presentation + +The existing Knowledge system remains the correct home for what +characters know and when they learn it. + +## Fix provenance centrally + +The current `Chapter 0` issue appears across multiple review categories. + +Investigate and fix the common ChapterID/SceneID provenance path rather +than patching each Razor page separately. + +Knowledge entries should be traceable to: + +- canonical ChapterID +- canonical SceneID +- scene number/title +- structural summary +- knowledge holder +- fact +- acquisition/change point + +## Identity knowledge + +Character alias/identity revelations belong here. + +Example: + +`Maggie learns that The Man in Grey is Victor Harrington` + +The canonical CharacterID establishes objective physical identity. + +Knowledge records establish subjective awareness. + +Do not conflate the two. + +## Review + +Large unpaginated lists are unacceptable for full novels. + +Use sensible pagination/grouping/filtering with scene context. + +Only uncertain/consequential knowledge should require explicit review +where possible. + +------------------------------------------------------------------------ + +# Stage 9 - Deep Audit and Fix Plot Lines and Threads + +Plot Lines and Threads are a central PlotDirector feature and require a +dedicated pass. + +## Current positives + +The AI is finding plausible narrative signals. + +## Current problems observed + +- some proposed Plot Lines are too trivial and are really Threads; +- Plot Lines and Threads are visually mixed; +- Threads suggest assigning a related Plot Line but proposed Plot + Lines are not available; +- candidates cannot merge with other candidates; +- ordering appears to follow discovery rather than narrative + hierarchy; +- supporting scene evidence appears incorrect; +- a thread only present in Chapter 1 claimed support across all six + chapters; +- after review, only two Plot Lines and zero Threads were created. + +## Definitions + +### Plot Line + +A major sustained narrative arc. + +### Plot Thread + +A smaller recurring question, clue, objective, subplot, secret, +relationship issue, promise or unresolved matter, often belonging +beneath a Plot Line. + +## Desired review model + +### Proposed Plot Lines + +Show major arcs. + +Each may contain: + +### Suggested Threads + +Also allow: + +### Unassigned Threads + +The author should be able to: + +- accept; +- rename; +- reject; +- merge with existing canonical Plot Line/Thread; +- merge with another candidate; +- move a proposed Thread beneath a proposed Plot Line; +- reclassify Plot Line \<-\> Thread where appropriate. + +## Evidence + +Supporting evidence must use actual canonical SceneIDs. + +Only scenes that genuinely support the candidate may be listed. + +Do not infer support merely because a candidate exists somewhere within +the same chapter/book. + +This stage should be undertaken after POV, identity and provenance +foundations are reliable. + +------------------------------------------------------------------------ + +# Stage 10 - Replace the Story Intelligence Review Wizard with a Review Centre + +The linear Story Intelligence review wizard should be retired after +mandatory Character Resolution. + +The Core Import remains a wizard because it has genuine sequential +dependencies. + +Detailed Story Intelligence review does not. + +## Entry experience + +Core Import completes and the book is immediately usable. + +The author can: + +- browse chapters; +- inspect scenes; +- read structural summaries; +- see scene titles; +- edit characters; +- add character portraits; +- use Word Companion/multiscreen tools. + +Story Intelligence continues server-side without requiring Word to +remain open. + +When analysis completes, the existing email notification invites the +author back. + +## Mandatory gate + +The author first resolves required Character ambiguities. + +After this, open the Review Centre. + +## Review Centre + +Conceptually: + + Analysis Status + ---------------------- ------------------ + Characters Resolved + Locations 6 need attention + Assets 3 need attention + Relationships 8 suggestions + Knowledge 2 need attention + Metrics Available + Plot Lines & Threads 7 proposals + +The exact visual design should follow PlotDirector's established UI. + +## Independent modules + +After Character Resolution, the author can review categories in any +order. + +They may: + +- review Plot Lines first; +- leave Locations until later; +- close the browser halfway through Relationships; +- return another day. + +Progress is persisted. + +The book remains usable. + +There is no requirement for Story Intelligence review to reach 100%. + +------------------------------------------------------------------------ + +# Stage 11 - Change Review Philosophy to Exception Handling + +The number of AI observations must not equal the number of human +decisions. + +This is the key scalability principle. + +## Three broad classes + +### Confident / deterministic + +Resolve automatically. + +Examples: + +- known canonical character; +- known alias; +- punctuation/case normalisation; +- Narrator when POV is known. + +### Low consequence and easily editable + +Save automatically. + +Example: + +- scene metric values. + +The author can edit them later. + +### Ambiguous or consequential + +Require review. + +Examples: + +- uncertain character identity; +- possible character merge; +- ambiguous significant location; +- new major Plot Line; +- uncertain relationship state change. + +## Target workload + +A 50-chapter / 250-scene novel may generate thousands of internal +observations. + +The author should ideally make **tens of meaningful decisions**, not +hundreds or thousands of repetitive approvals. + +Always provide `View all` or equivalent for authors who want complete +control. + +------------------------------------------------------------------------ + +# Stage 12 - Standardise Busy/Save Behaviour + +All review actions that perform server/database work must use one +consistent busy-state pattern. + +Examples: + +- Create/resolve characters +- Save locations +- Save assets +- Apply relationships +- Apply knowledge +- Save Plot Lines/Threads + +On activation: + +- disable the triggering control; +- prevent duplicate submission; +- show a spinner/progress state; +- change the label appropriately (`Saving...`, `Creating...`, etc.); +- complete; +- return to the Review Centre or next meaningful destination. + +Remove repetitive interstitial pages such as: + +- Characters created +- Locations created +- Assets created + +A small inline/transient confirmation is enough. + +------------------------------------------------------------------------ + +# Stage 13 - Pagination, Grouping and Context + +For categories that can still contain substantial data: + +- paginate; +- filter; +- group where useful; +- preserve selections/review state across pages. + +Relationships and Knowledge should particularly benefit from +chapter/scene grouping. + +Where a candidate is tied to manuscript evidence, display useful +context: + +- Chapter number/title +- Scene number/title +- StructuralSummary +- concise evidence + +Pagination supports usability but is **not** a substitute for reducing +unnecessary decisions. + +------------------------------------------------------------------------ + +# Stage 14 - Preserve Existing Successful Features + +Do not destabilise the parts that are now working well. + +Preserve: + +- canonical chapter/scene import; +- structural scene boundaries; +- `SceneManuscriptSources`; +- structural summaries as the single factual summary; +- optional descriptive scene titles; +- Word marker creation/save/acknowledgement; +- Word Companion existing-book linking; +- multiscreen SceneID following; +- deterministic SceneCharacter mapping; +- configured Scene Metric Types; +- current realistic metric scoring; +- Story Intelligence operating against canonical scenes; +- Story Era / Primary Locale direction; +- existing Knowledge architecture; +- existing relationship-event architecture where it already supports + progression. + +Metrics in particular are producing useful values and should not become +a mandatory approval exercise. + +------------------------------------------------------------------------ + +# Recommended Implementation Order + +The work should be split into multiple focused Codex tasks. + +## Job 1 - POV and Character Identity Foundation + +Implement/investigate: + +- chapter default POV detection in structural analysis; +- scene POV inheritance and per-scene overrides; +- author POV review/editing; +- normal Chapter POV controls; +- canonical character merge; +- alias consolidation; +- provenance-preserving identity merge; +- Character Review redesigned around unresolved identity; +- propagation of canonical CharacterID references. + +This is the prerequisite for everything else. + +## Job 2 - Story Intelligence Context and Provenance + +Refactor detailed analysis so it receives canonical: + +- POV; +- Setting; +- StructuralSummary; +- scene title; +- previous/next scene context; +- characters and aliases; +- Era/Locale. + +Also fix the shared Chapter 0/provenance problem. + +Do not yet undertake broad review UI redesign. + +## Job 3 - Entity and Relationship Intelligence Quality + +Rework: + +- Locations; +- Assets; +- Relationships; +- Knowledge. + +Specifically investigate: + +- reuse of structural Setting; +- Memory Tin omission; +- entity canonicalisation; +- relationship classification; +- **relationship progression through scene-based events rather than a + flattened final state**. + +## Job 4 - Plot Lines and Threads + +Perform a dedicated deep audit/fix covering: + +- candidate evidence; +- book-wide consolidation; +- Plot Line versus Thread classification; +- hierarchy; +- candidate-to-candidate merge; +- assignment of proposed Threads to proposed Plot Lines; +- canonical persistence; +- incorrect supporting-scene claims. + +## Job 5 - Review Centre and Human Workload + +Only after the intelligence foundations are cleaner: + +- retire the linear Story Intelligence review wizard; +- retain mandatory Character Resolution; +- add independent Review Centre modules; +- implement exception-based review; +- add bulk actions; +- add pagination/grouping; +- standardise busy states; +- remove success interstitials; +- allow leaving and resuming at any time. + +------------------------------------------------------------------------ + +# Success Criteria + +The refactor is successful when: + +1. Core Import gives the author a useful book quickly. +2. POV is established early enough to improve downstream + interpretation. +3. Canonical CharacterIDs represent physical identity regardless of + aliases. +4. Alias merges preserve manuscript/narrative history. +5. Knowledge continues to model who knows what and when. +6. Relationships preserve **progression over time**, not merely the + final relationship type. +7. Later AI analysis reuses structural knowledge instead of + rediscovering it. +8. Locations/assets/relationships receive sufficient context to avoid + generic low-value candidates. +9. Plot Line and Thread evidence is trustworthy. +10. The author only has to resolve genuine uncertainty. +11. After Character Resolution, review categories are independent and + resumable. +12. A full novel does not require hours of repetitive approval clicking. +13. Existing good features, particularly scene summaries, titles and + metrics, remain intact. + +------------------------------------------------------------------------ + +# Key Product Principle + +PlotDirector should not demonstrate intelligence by showing the author +everything it detected. + +It should demonstrate intelligence by knowing **what it can safely +resolve itself, what changes through the story, and what genuinely +requires the author's judgement**.