PlotDirector/Docs/Story_Intelligence_Illustration_Selection_Technical_Explanation.md

Story Intelligence Illustration Selection Technical Explanation

This document explains how Story Intelligence currently chooses illustrations, tracing the execution path from scene analysis through Story Memory, illustration matching, demand generation, snapshot construction, and browser rendering.

No code changes are described here as recommendations only; this is documentation of the current implementation.

Complete Execution Path

flowchart TD
    A[PersistedStoryIntelligenceRunner] -->|Chapter prompt AI call| B[Chapter Structure JSON]
    B -->|Scene boundaries| C[Scene Intelligence prompt AI calls]
    C -->|ParsedJson saved| D[StoryIntelligenceSceneResults]
    D --> E[StoryIntelligenceVisualisationSnapshotService]
    E -->|Build scene states and Story Memory| F[Prototype ViewModel]
    F --> G[IllustrationLibraryService.ApplyApprovedIllustrationsAsync]
    G -->|Exact stable-code lookup| H[Starter/generated image if code matches]
    H --> I[StoryIntelligenceIllustrationMatchingService]
    I -->|Characters| J[Assignment revalidation and scoring]
    I -->|Locations/assets| K[Demand lookup/creation]
    J --> L[StoryIntelligenceCharacterIllustrationAssignments]
    K --> M[StoryIntelligenceIllustrationDemands and queued IllustrationLibraryItems]
    L --> N[Visualisation JSON]
    M --> N
    N --> O[Browser visualisation]

sequenceDiagram
    participant Scene as Scene Analysis
    participant Memory as Story Memory
    participant Match as Illustration Matcher
    participant Demand as Demand Generator
    participant Snap as Snapshot Builder
    participant UI as Visualisation

    Scene->>Scene: OpenAI chapter + scene prompts
    Scene->>Snap: Persisted scene result JSON
    Snap->>Memory: Replay analysed scenes in order
    Memory-->>Snap: Carried characters, relationships, POV
    Snap->>Match: Character/location/asset view model
    Match->>Match: Read library + previous assignments
    Match->>Demand: No compatible illustration
    Demand->>Demand: Upsert demand + queue planned image
    Match-->>Snap: Assigned/fallback image resolution
    Snap-->>UI: Snapshot JSON
    UI->>UI: Render imagePath/fallback

Main Services and Responsibilities

PersistedStoryIntelligenceRunner

Responsible for running Story Intelligence analysis and persisting results.

Input:

• Queued StoryIntelligenceQueuedRun
• Chapter source text
• Chapter Structure prompt
• Scene Intelligence prompt
• OpenAI model settings

Output:

• Persisted chapter structure result
• Persisted scene intelligence result
• Run progress/status

Database reads:

• Queued Story Intelligence run via StoryIntelligenceRun_ClaimNextPending
• Current run state for cancellation checks

Database writes:

• StoryIntelligenceRuns
• StoryIntelligenceChapterResults
• StoryIntelligenceSceneResults

AI calls:

• Chapter Structure prompt
• Scene Intelligence prompt per scene
• JSON repair retry prompt if parsing fails

Illustration role:

• None directly. It only creates the structured scene data later consumed by the visualisation and matcher.

StoryIntelligenceVisualisationSnapshotService

Responsible for reconstructing visualisation state from persisted analysis.

Input:

• Import session ID or run ID
• Persisted run and scene-result records
• Parsed scene JSON

Output:

• StoryIntelligenceExperiencePrototypeViewModel
• Scene states with characters, location, assets, relationships, knowledge, activity, discoveries
• In-memory Story Memory

Database reads:

• StoryIntelligenceBookPipelines
• StoryIntelligenceRuns
• StoryIntelligenceSceneResults

Database writes:

• None directly.

Cached/derived values:

• ChangeToken
• Session change token
• Replay cursor
• In-memory StoryMemory
• Stable visual IDs
• Stable fallback library codes

AI calls:

• None.

IllustrationLibraryService.ApplyApprovedIllustrationsAsync

Responsible for the first image pass, using exact stable-code lookup.

Input:

• Prototype view model with LibraryCode values on characters, locations and assets

Output:

• Updated image paths and ImageResolution objects where a matching library item exists

Database reads:

• IllustrationLibraryItems

Database writes:

• None.

Matching behaviour:

• Exact StableCode match only.
• Uses Approved before Generated.
• Then newest item first.

StoryIntelligenceIllustrationMatchingService

Responsible for final character assignment, revalidation, scoring, and demand creation.

Input:

• Project ID
• Fully built visualisation prototype view model
• Existing illustration library
• Existing active character assignments

Output:

• Character image assignments
• Character assignment diagnostics
• Demand records when no suitable illustration exists
• Queued planned illustration items when demand threshold is reached

Database reads:

• IllustrationLibraryItems
• StoryIntelligenceCharacterIllustrationAssignments

Database writes:

• StoryIntelligenceCharacterIllustrationAssignments
• StoryIntelligenceIllustrationDemands
• IllustrationLibraryItems through planned/queued demand records

AI calls:

• None directly.
• It queues illustration generation. A background image-generation worker later calls OpenAI Images.

IllustrationGenerationRunner and OpenAIIllustrationImageProvider

Responsible for generating queued illustration-library images.

Input:

• Queued IllustrationLibraryItems
• Prompt/specification JSON

Output:

• Stored generated image files
• Updated illustration-library item status

Database reads:

• IllustrationLibraryItems

Database writes:

• IllustrationLibraryItems

AI calls:

• POST https://api.openai.com/v1/images/generations

Character Illustration Selection

Character matching is a two-layer process.

First, StoryIntelligenceVisualisationSnapshotService gives each character a likely starter LibraryCode. Then IllustrationLibraryService.ApplyApprovedIllustrationsAsync resolves exact library-code matches. After that, StoryIntelligenceIllustrationMatchingService.ApplyCharacterIllustrationsAsync performs the newer assignment and revalidation pass.

Where Age Is Determined

Age is determined in two places.

Initial stable-code age:

• Class: StoryIntelligenceVisualisationSnapshotService.CharacterProfile
• Method: DetectAgeBand
• Used only to pick a starter code.
• Defaults to YoungAdult.

Final matcher age:

• Class: StoryIntelligenceIllustrationCompatibility
• Method: InferCharacterEvidence
• Produces a CharacterEvidenceProfile.
• Used by StoryIntelligenceIllustrationMatchingService.Score.

Where Inferred Age Is Determined

Inferred age comes from:

• Explicit age terms: 13, 14, 15, 16, 17, 18, 19
• Words: toddler, baby, infant, child, schoolboy, schoolgirl, teenager, young adult, middle-aged, elderly
• Role/title terms: Mr, Mrs, Miss, Ms, Dr, Professor, teacher, detective, police officer, nurse, receptionist, solicitor, landlord, manager
• Relationship terms: mother, father, parent, aunt, uncle, grandmother, grandfather

Where Gender/Presentation Is Determined

Presentation is determined by StoryIntelligenceIllustrationCompatibility.InferCharacterEvidence.

Sources, in priority order:

1. Title presentation
2. Entity relationship/presentation words in scoped text
3. Relationship evidence
4. Pronouns
5. Hard-coded name list
6. Unknown

Whether Names Influence Gender

Yes. Names are hard-coded.

Examples:

• Feminine: Beth, Maggie, Rosie, Grace, Zoe, Rebecca, Annie, Mary, Sharon, Catherine, Susan, Helen, Elen
• Masculine: Adam, Colin, David, Kevin, Simon, Graham, Gareth, John, Michael, Peter, Richard, Rick, George

There is also a presentation guard that can override leaked relationship/pronoun evidence using name/title evidence unless explicit contradictory evidence appears.

Whether Titles Influence Gender

Yes.

• Mrs, Miss, Ms, Lady infer feminine.
• Mr, Sir infer masculine.
• Dr and Professor infer adult age but not masculine/feminine presentation by themselves.

Whether Occupations Influence Age

Yes. Occupations such as teacher, driving examiner, detective, police officer, nurse, receptionist, solicitor, landlord, manager, professor and doctor infer adult age.

Hair Colour, Eye Colour and Ethnicity

Hair colour is extracted in StoryIntelligenceIllustrationCompatibility.HairColour.

Recognised values include:

• Bald
• Black
• Brown/brunette/dark hair
• Blonde/fair hair
• Red/auburn/ginger
• Grey/gray
• White

Hair length is also extracted.

Eye colour is not currently extracted for matching.

Ethnicity is not currently extracted for matching. The matcher uses skin tone, not ethnicity.

Previously Learned Evidence

Previously learned Story Memory evidence can affect the character states that enter the matcher, but there is no persistent Story Memory evidence table read by the matcher.

Previous assignment evidence is stored as JSON in StoryIntelligenceCharacterIllustrationAssignments, but ListAssignmentsAsync currently returns EvidenceJson and CandidateDiagnosticsJson as null. That means previous evidence is not rehydrated and used as a true evidence source during matching.

Conflict Resolution

Conflict handling is mostly rule-based:

• FirstKnown picks the first non-unknown signal from prioritised evidence.
• The presentation guard prevents obvious gender leakage from overriding strong name/title evidence.
• Hard rejections eliminate impossible illustration candidates.
• Existing assignments are revalidated against current evidence.
• Replacement requires hard failure, material wrongness, or a better score by margin.

Hard Rules for Characters

A candidate is rejected if:

• Named character uses Unknown Figure artwork.
• Candidate is marked unsuitable for named-character matching.
• Age is incompatible.
• Strong presentation evidence contradicts candidate presentation.
• Known hair colour contradicts candidate hair colour.
• Candidate is already assigned to another significant named character.

Additional rule:

• If character age evidence is unknown, child/teen candidate artwork is rejected until explicit child/teen evidence appears.

Character Weighted Scoring

Final matcher score in StoryIntelligenceIllustrationMatchingService.Score:

• Age exact match: +25
• Age evidence unknown: +12
• Candidate age unknown: +8 plus rejection note
• Presentation exact match with strong evidence: +42
• Presentation exact match without strong evidence: +25
• Presentation evidence unknown: half weight
• Candidate presentation unknown: one-third weight plus rejection note
• Hair colour exact match: +20
• Hair colour evidence unknown: +10
• Candidate hair colour unknown: +6 plus rejection note
• Skin tone exact match: +20
• Skin tone evidence unknown: +10
• Candidate skin tone unknown: +6 plus rejection note
• Hair length exact match: +8
• Glasses exact match: +7
• Facial hair exact match: +7
• Approved library item: +3

Score is clamped between 0 and 100.

Minimum suitable score:

• 48

Reassignment margin:

• 24

Older stable-code scoring before the final matcher:

• Age exact: +80
• Compatible age: +30
• Age mismatch: -45
• Presentation exact: +24
• Hair colour exact: +18
• Hair length exact: +10
• Signal keyword match: +14 each

Why Existing Illustrations Are Retained

An existing illustration is retained when:

• Existing assignment still resolves to a public image URL.
• Existing candidate has no hard rejections.
• Existing candidate is not materially wrong for known age, presentation, hair colour or skin tone.
• No better candidate exists, or the best candidate does not beat existing score by the reassignment margin.
• Existing score is at least 48.

Why Existing Illustrations Are Replaced

An existing illustration is replaced when:

• It violates hard constraints.
• It is materially wrong for age, presentation, hair colour or skin tone.
• A better compatible candidate beats the existing score by at least 24.
• The existing assignment was fallback/null and a generated valid item is now available.

When Demand Is Created

A character demand record is created when no hard-compatible unused illustration scores at least 48.

Demand key:

character|age|presentation|hairColour|skinTone

Writes:

• StoryIntelligenceIllustrationDemands

If request count reaches threshold, currently 1, it creates or queues:

char-demand-{hash}

When Generated Illustrations Become Eligible

A generated illustration becomes eligible when:

• It exists in IllustrationLibraryItems
• Status is Generated or Approved
• It has a valid thumbnail or file path
• A later snapshot/matcher pass reads it from the library

Why Duplicate Portraits Sometimes Occur

Duplicate portraits can occur because:

1. Exact stable-code resolution can apply the same starter code before final assignment.
2. Duplicate assignment cleanup only handles assignment records, not every transient fallback/stable-code image.
3. Low-significance or unresolved identities can share generic fallbacks.
4. Character identity canonicalisation can split one person into multiple keys.
5. Existing duplicate assignment handling keeps the first retained item and clears later duplicates only during the matcher pass.

Why Unknown Figure Can Still Appear

Unknown Figure can still appear when:

• The initial stable-code phase chooses char-unknown-figure.
• The character is unresolved/generic, so named-character hard rejection does not apply.
• The matcher records demand and falls back to an existing SVG/fallback.
• The library contains generated/approved unknown figure art used by exact code before the final assignment overrides it.

Location Illustration Selection

Classification

Locations are built by StoryIntelligenceVisualisationSnapshotService.BuildLocation.

It chooses:

1. Present-in-scene locations before mentioned-only locations.
2. More specific names.
3. Higher confidence.
4. Fallback to scene setting.

Semantic Typing

Semantic type is determined by StoryIntelligenceIllustrationCompatibility.LocationType.

Examples:

• Bathroom
• Kitchen
• Bedroom
• LivingRoom
• Hallway
• Stairwell
• Hospital
• Pub
• Beach
• RailwayStation
• Canal
• Garage
• HouseExterior
• Office
• School
• WaitingRoom
• Road
• CarPark
• Shop
• StreetFurniture
• FlatInterior
• UnknownInterior

Matching

Location matching has two phases:

1. Initial stable-code mapping in LocationLibraryCode.
2. Demand-based exact generated lookup in ApplyLocationDemandAsync.

Important current behaviour:

• Bathroom, WaitingRoom and Road intentionally return no starter code.
• This prevents borrowing wrong generic interiors.
• Generated demand selection uses exact code loc-demand-{hash}.

There is a LocationCompatible helper, but generated supporting image selection currently uses exact demand stable-code lookup rather than weighted compatibility scoring.

Demand Creation

Location demand is created if:

• Location type is not UnknownInterior.
• Current image is still fallback.
• No generated demand image exists for the exact demand code.

Demand key:

location|type

Writes:

• StoryIntelligenceIllustrationDemands
• IllustrationLibraryItems planned/queued demand item

Reuse Rules

Locations are reusable by type/demand key. They are not unique per manuscript location.

Asset Illustration Selection

Classification

Assets are built by StoryIntelligenceVisualisationSnapshotService.BuildAssets.

It:

• Excludes mentioned-only assets.
• Excludes low-value/background objects.
• Groups by cleaned asset name.
• Picks highest-confidence entry per name.
• Takes top five by asset weight.

Semantic Typing

Asset type is determined by StoryIntelligenceIllustrationCompatibility.AssetType.

Examples:

• Ambulance
• Stretcher
• BodyBag
• Bottle
• Money
• Clothing
• Document
• Jewellery
• Weapon
• Hatchback
• FamilyCar
• SportsCar
• Van
• Car
• CarPart
• Tin
• Parcel
• Rucksack
• Suitcase
• Note
• Letter
• Notebook
• Book
• Telephone
• Keys
• Photograph
• UnknownObject

Colour Handling

Colour is determined by AssetColour.

Recognised values:

• Blue
• Red
• Green
• Yellow
• Black
• White
• Grey
• Brown
• Unknown

Matching

Asset matching has two phases:

1. Initial stable-code mapping in AssetLibraryCode.
2. Demand-based exact generated lookup in ApplyAssetDemandAsync.

Starter mappings include:

• Letter/Note/Document -> asset-folded-letter
• Notebook/Book -> asset-worn-notebook
• Car -> asset-red-classic-car
• Keys -> asset-old-keys
• Telephone -> asset-phone-recorder
• Photograph -> asset-photo-print
• Rucksack -> asset-rucksack

Generated demand selection uses exact stable code:

asset-demand-{hash}

There is an AssetCompatible helper, but generated supporting-image selection currently uses exact demand code rather than weighted compatibility scoring.

Demand Creation

Asset demand is created if:

• Asset type is not UnknownObject.
• Current image is fallback.
• No generated image exists for exact demand code.

Demand key:

asset|type|colour

Writes:

• StoryIntelligenceIllustrationDemands
• IllustrationLibraryItems planned/queued demand item

Database Table Summary

Reads:

• StoryIntelligenceRuns
• StoryIntelligenceChapterResults
• StoryIntelligenceSceneResults
• StoryIntelligenceBookPipelines
• IllustrationLibraryItems
• StoryIntelligenceCharacterIllustrationAssignments

Writes:

• StoryIntelligenceRuns
• StoryIntelligenceChapterResults
• StoryIntelligenceSceneResults
• StoryIntelligenceCharacterIllustrationAssignments
• StoryIntelligenceIllustrationDemands
• IllustrationLibraryItems

No dedicated persistent Story Memory table is currently used in the illustration matcher.

Cached Values Used

The system uses these cached or derived values:

• Stored ParsedJson in scene results
• Stored illustration library metadata/specification JSON
• Stored generated file/thumbnail paths
• Stored active character assignments
• Stored demand keys/request counts
• Snapshot ChangeToken
• Replay cursor index
• In-memory Story Memory during snapshot construction
• Stable hashes for visual entity IDs and demand codes

Previous Assignments Considered

The matcher considers active previous assignments from StoryIntelligenceCharacterIllustrationAssignments.

Used fields include:

• CharacterKey
• IllustrationLibraryItemID
• StableCode
• MatchConfidence
• MatchingScore
• ReasonChosen
• ReasonReassigned

Current limitation:

• Previous EvidenceJson and CandidateDiagnosticsJson are not returned by ListAssignmentsAsync; they are selected as null.

Story Memory Evidence Considered

Story Memory can influence:

• Added remembered characters
• POV identity
• Character role/relevance
• Carried relationships
• Inferred kinship relationships
• Rough presentation and age band

Story Memory does not directly persist or supply a structured evidence object to the final matcher. It affects the visual character state, and the matcher then infers evidence from that state.

Character Evidence Ignored

Currently ignored or mostly lost for matching:

• Eye colour
• Ethnicity
• Clothing details
• Body type
• Exact age beyond broad age band
• Many action-level descriptors
• Raw manuscript text
• Prior assignment evidence JSON
• Scene observations unless transformed into role/relevance/relationship text

Five Biggest Weaknesses

1. Two image systems compete: stable-code starter matching and final assignment matching.
2. Story Memory is rebuilt in memory and is not a durable structured evidence store.
3. Character evidence is compressed before matching.
4. Locations and assets do not have a real weighted matcher.
5. Previous evidence is stored but not reloaded as evidence for conflict resolution.

Five Places Evidence Can Be Lost

1. AI scene JSON may omit physical details.
2. Snapshot character state drops raw action and descriptor detail.
3. Story Memory stores only simplified role/relevance/age/presentation.
4. Matcher ignores eye colour, ethnicity, clothing, build and exact age.
5. Assignment repository does not return previous evidence JSON to the matcher.

Five Places Incorrect Assignments Can Occur

1. Initial stable-code fallback picks a plausible but wrong starter image.
2. Hard-coded name gender lists misclassify unusual or ambiguous names.
3. Relationship/family terms leak onto the wrong character.
4. Asset/location keyword order maps vague text too broadly.
5. Canonical identity splitting creates multiple assignment keys for one person.

Five Quickest Improvements

1. Store structured evidence per canonical identity and reload it during matching.
2. Remove or demote pre-matcher stable-code character assignment for live data.
3. Add weighted matchers for locations and assets.
4. Add eye colour, ethnicity/skin-tone source, clothing and exact-age evidence fields.
5. Expand diagnostics to show winning candidate, rejected candidates and exact evidence source per entity.