Commit remaining Story Intelligence changes
This commit is contained in:
parent
08a014e2ce
commit
d40c607d9b
@ -0,0 +1,745 @@
|
||||
# Story Intelligence Illustration Selection Technical Explanation
|
||||
|
||||
This document explains how Story Intelligence currently chooses illustrations, tracing the execution path from scene analysis through Story Memory, illustration matching, demand generation, snapshot construction, and browser rendering.
|
||||
|
||||
No code changes are described here as recommendations only; this is documentation of the current implementation.
|
||||
|
||||
## Complete Execution Path
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[PersistedStoryIntelligenceRunner] -->|Chapter prompt AI call| B[Chapter Structure JSON]
|
||||
B -->|Scene boundaries| C[Scene Intelligence prompt AI calls]
|
||||
C -->|ParsedJson saved| D[StoryIntelligenceSceneResults]
|
||||
D --> E[StoryIntelligenceVisualisationSnapshotService]
|
||||
E -->|Build scene states and Story Memory| F[Prototype ViewModel]
|
||||
F --> G[IllustrationLibraryService.ApplyApprovedIllustrationsAsync]
|
||||
G -->|Exact stable-code lookup| H[Starter/generated image if code matches]
|
||||
H --> I[StoryIntelligenceIllustrationMatchingService]
|
||||
I -->|Characters| J[Assignment revalidation and scoring]
|
||||
I -->|Locations/assets| K[Demand lookup/creation]
|
||||
J --> L[StoryIntelligenceCharacterIllustrationAssignments]
|
||||
K --> M[StoryIntelligenceIllustrationDemands and queued IllustrationLibraryItems]
|
||||
L --> N[Visualisation JSON]
|
||||
M --> N
|
||||
N --> O[Browser visualisation]
|
||||
```
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Scene as Scene Analysis
|
||||
participant Memory as Story Memory
|
||||
participant Match as Illustration Matcher
|
||||
participant Demand as Demand Generator
|
||||
participant Snap as Snapshot Builder
|
||||
participant UI as Visualisation
|
||||
|
||||
Scene->>Scene: OpenAI chapter + scene prompts
|
||||
Scene->>Snap: Persisted scene result JSON
|
||||
Snap->>Memory: Replay analysed scenes in order
|
||||
Memory-->>Snap: Carried characters, relationships, POV
|
||||
Snap->>Match: Character/location/asset view model
|
||||
Match->>Match: Read library + previous assignments
|
||||
Match->>Demand: No compatible illustration
|
||||
Demand->>Demand: Upsert demand + queue planned image
|
||||
Match-->>Snap: Assigned/fallback image resolution
|
||||
Snap-->>UI: Snapshot JSON
|
||||
UI->>UI: Render imagePath/fallback
|
||||
```
|
||||
|
||||
## Main Services and Responsibilities
|
||||
|
||||
### PersistedStoryIntelligenceRunner
|
||||
|
||||
Responsible for running Story Intelligence analysis and persisting results.
|
||||
|
||||
Input:
|
||||
|
||||
- Queued `StoryIntelligenceQueuedRun`
|
||||
- Chapter source text
|
||||
- Chapter Structure prompt
|
||||
- Scene Intelligence prompt
|
||||
- OpenAI model settings
|
||||
|
||||
Output:
|
||||
|
||||
- Persisted chapter structure result
|
||||
- Persisted scene intelligence result
|
||||
- Run progress/status
|
||||
|
||||
Database reads:
|
||||
|
||||
- Queued Story Intelligence run via `StoryIntelligenceRun_ClaimNextPending`
|
||||
- Current run state for cancellation checks
|
||||
|
||||
Database writes:
|
||||
|
||||
- `StoryIntelligenceRuns`
|
||||
- `StoryIntelligenceChapterResults`
|
||||
- `StoryIntelligenceSceneResults`
|
||||
|
||||
AI calls:
|
||||
|
||||
- Chapter Structure prompt
|
||||
- Scene Intelligence prompt per scene
|
||||
- JSON repair retry prompt if parsing fails
|
||||
|
||||
Illustration role:
|
||||
|
||||
- None directly. It only creates the structured scene data later consumed by the visualisation and matcher.
|
||||
|
||||
### StoryIntelligenceVisualisationSnapshotService
|
||||
|
||||
Responsible for reconstructing visualisation state from persisted analysis.
|
||||
|
||||
Input:
|
||||
|
||||
- Import session ID or run ID
|
||||
- Persisted run and scene-result records
|
||||
- Parsed scene JSON
|
||||
|
||||
Output:
|
||||
|
||||
- `StoryIntelligenceExperiencePrototypeViewModel`
|
||||
- Scene states with characters, location, assets, relationships, knowledge, activity, discoveries
|
||||
- In-memory Story Memory
|
||||
|
||||
Database reads:
|
||||
|
||||
- `StoryIntelligenceBookPipelines`
|
||||
- `StoryIntelligenceRuns`
|
||||
- `StoryIntelligenceSceneResults`
|
||||
|
||||
Database writes:
|
||||
|
||||
- None directly.
|
||||
|
||||
Cached/derived values:
|
||||
|
||||
- `ChangeToken`
|
||||
- Session change token
|
||||
- Replay cursor
|
||||
- In-memory `StoryMemory`
|
||||
- Stable visual IDs
|
||||
- Stable fallback library codes
|
||||
|
||||
AI calls:
|
||||
|
||||
- None.
|
||||
|
||||
### IllustrationLibraryService.ApplyApprovedIllustrationsAsync
|
||||
|
||||
Responsible for the first image pass, using exact stable-code lookup.
|
||||
|
||||
Input:
|
||||
|
||||
- Prototype view model with `LibraryCode` values on characters, locations and assets
|
||||
|
||||
Output:
|
||||
|
||||
- Updated image paths and `ImageResolution` objects where a matching library item exists
|
||||
|
||||
Database reads:
|
||||
|
||||
- `IllustrationLibraryItems`
|
||||
|
||||
Database writes:
|
||||
|
||||
- None.
|
||||
|
||||
Matching behaviour:
|
||||
|
||||
- Exact `StableCode` match only.
|
||||
- Uses `Approved` before `Generated`.
|
||||
- Then newest item first.
|
||||
|
||||
### StoryIntelligenceIllustrationMatchingService
|
||||
|
||||
Responsible for final character assignment, revalidation, scoring, and demand creation.
|
||||
|
||||
Input:
|
||||
|
||||
- Project ID
|
||||
- Fully built visualisation prototype view model
|
||||
- Existing illustration library
|
||||
- Existing active character assignments
|
||||
|
||||
Output:
|
||||
|
||||
- Character image assignments
|
||||
- Character assignment diagnostics
|
||||
- Demand records when no suitable illustration exists
|
||||
- Queued planned illustration items when demand threshold is reached
|
||||
|
||||
Database reads:
|
||||
|
||||
- `IllustrationLibraryItems`
|
||||
- `StoryIntelligenceCharacterIllustrationAssignments`
|
||||
|
||||
Database writes:
|
||||
|
||||
- `StoryIntelligenceCharacterIllustrationAssignments`
|
||||
- `StoryIntelligenceIllustrationDemands`
|
||||
- `IllustrationLibraryItems` through planned/queued demand records
|
||||
|
||||
AI calls:
|
||||
|
||||
- None directly.
|
||||
- It queues illustration generation. A background image-generation worker later calls OpenAI Images.
|
||||
|
||||
### IllustrationGenerationRunner and OpenAIIllustrationImageProvider
|
||||
|
||||
Responsible for generating queued illustration-library images.
|
||||
|
||||
Input:
|
||||
|
||||
- Queued `IllustrationLibraryItems`
|
||||
- Prompt/specification JSON
|
||||
|
||||
Output:
|
||||
|
||||
- Stored generated image files
|
||||
- Updated illustration-library item status
|
||||
|
||||
Database reads:
|
||||
|
||||
- `IllustrationLibraryItems`
|
||||
|
||||
Database writes:
|
||||
|
||||
- `IllustrationLibraryItems`
|
||||
|
||||
AI calls:
|
||||
|
||||
- `POST https://api.openai.com/v1/images/generations`
|
||||
|
||||
## Character Illustration Selection
|
||||
|
||||
Character matching is a two-layer process.
|
||||
|
||||
First, `StoryIntelligenceVisualisationSnapshotService` gives each character a likely starter `LibraryCode`. Then `IllustrationLibraryService.ApplyApprovedIllustrationsAsync` resolves exact library-code matches. After that, `StoryIntelligenceIllustrationMatchingService.ApplyCharacterIllustrationsAsync` performs the newer assignment and revalidation pass.
|
||||
|
||||
### Where Age Is Determined
|
||||
|
||||
Age is determined in two places.
|
||||
|
||||
Initial stable-code age:
|
||||
|
||||
- Class: `StoryIntelligenceVisualisationSnapshotService.CharacterProfile`
|
||||
- Method: `DetectAgeBand`
|
||||
- Used only to pick a starter code.
|
||||
- Defaults to `YoungAdult`.
|
||||
|
||||
Final matcher age:
|
||||
|
||||
- Class: `StoryIntelligenceIllustrationCompatibility`
|
||||
- Method: `InferCharacterEvidence`
|
||||
- Produces a `CharacterEvidenceProfile`.
|
||||
- Used by `StoryIntelligenceIllustrationMatchingService.Score`.
|
||||
|
||||
### Where Inferred Age Is Determined
|
||||
|
||||
Inferred age comes from:
|
||||
|
||||
- Explicit age terms: `13`, `14`, `15`, `16`, `17`, `18`, `19`
|
||||
- Words: `toddler`, `baby`, `infant`, `child`, `schoolboy`, `schoolgirl`, `teenager`, `young adult`, `middle-aged`, `elderly`
|
||||
- Role/title terms: `Mr`, `Mrs`, `Miss`, `Ms`, `Dr`, `Professor`, `teacher`, `detective`, `police officer`, `nurse`, `receptionist`, `solicitor`, `landlord`, `manager`
|
||||
- Relationship terms: `mother`, `father`, `parent`, `aunt`, `uncle`, `grandmother`, `grandfather`
|
||||
|
||||
### Where Gender/Presentation Is Determined
|
||||
|
||||
Presentation is determined by `StoryIntelligenceIllustrationCompatibility.InferCharacterEvidence`.
|
||||
|
||||
Sources, in priority order:
|
||||
|
||||
1. Title presentation
|
||||
2. Entity relationship/presentation words in scoped text
|
||||
3. Relationship evidence
|
||||
4. Pronouns
|
||||
5. Hard-coded name list
|
||||
6. Unknown
|
||||
|
||||
### Whether Names Influence Gender
|
||||
|
||||
Yes. Names are hard-coded.
|
||||
|
||||
Examples:
|
||||
|
||||
- Feminine: Beth, Maggie, Rosie, Grace, Zoe, Rebecca, Annie, Mary, Sharon, Catherine, Susan, Helen, Elen
|
||||
- Masculine: Adam, Colin, David, Kevin, Simon, Graham, Gareth, John, Michael, Peter, Richard, Rick, George
|
||||
|
||||
There is also a presentation guard that can override leaked relationship/pronoun evidence using name/title evidence unless explicit contradictory evidence appears.
|
||||
|
||||
### Whether Titles Influence Gender
|
||||
|
||||
Yes.
|
||||
|
||||
- `Mrs`, `Miss`, `Ms`, `Lady` infer feminine.
|
||||
- `Mr`, `Sir` infer masculine.
|
||||
- `Dr` and `Professor` infer adult age but not masculine/feminine presentation by themselves.
|
||||
|
||||
### Whether Occupations Influence Age
|
||||
|
||||
Yes. Occupations such as teacher, driving examiner, detective, police officer, nurse, receptionist, solicitor, landlord, manager, professor and doctor infer adult age.
|
||||
|
||||
### Hair Colour, Eye Colour and Ethnicity
|
||||
|
||||
Hair colour is extracted in `StoryIntelligenceIllustrationCompatibility.HairColour`.
|
||||
|
||||
Recognised values include:
|
||||
|
||||
- Bald
|
||||
- Black
|
||||
- Brown/brunette/dark hair
|
||||
- Blonde/fair hair
|
||||
- Red/auburn/ginger
|
||||
- Grey/gray
|
||||
- White
|
||||
|
||||
Hair length is also extracted.
|
||||
|
||||
Eye colour is not currently extracted for matching.
|
||||
|
||||
Ethnicity is not currently extracted for matching. The matcher uses skin tone, not ethnicity.
|
||||
|
||||
### Previously Learned Evidence
|
||||
|
||||
Previously learned Story Memory evidence can affect the character states that enter the matcher, but there is no persistent Story Memory evidence table read by the matcher.
|
||||
|
||||
Previous assignment evidence is stored as JSON in `StoryIntelligenceCharacterIllustrationAssignments`, but `ListAssignmentsAsync` currently returns `EvidenceJson` and `CandidateDiagnosticsJson` as null. That means previous evidence is not rehydrated and used as a true evidence source during matching.
|
||||
|
||||
### Conflict Resolution
|
||||
|
||||
Conflict handling is mostly rule-based:
|
||||
|
||||
- `FirstKnown` picks the first non-unknown signal from prioritised evidence.
|
||||
- The presentation guard prevents obvious gender leakage from overriding strong name/title evidence.
|
||||
- Hard rejections eliminate impossible illustration candidates.
|
||||
- Existing assignments are revalidated against current evidence.
|
||||
- Replacement requires hard failure, material wrongness, or a better score by margin.
|
||||
|
||||
### Hard Rules for Characters
|
||||
|
||||
A candidate is rejected if:
|
||||
|
||||
- Named character uses Unknown Figure artwork.
|
||||
- Candidate is marked unsuitable for named-character matching.
|
||||
- Age is incompatible.
|
||||
- Strong presentation evidence contradicts candidate presentation.
|
||||
- Known hair colour contradicts candidate hair colour.
|
||||
- Candidate is already assigned to another significant named character.
|
||||
|
||||
Additional rule:
|
||||
|
||||
- If character age evidence is unknown, child/teen candidate artwork is rejected until explicit child/teen evidence appears.
|
||||
|
||||
### Character Weighted Scoring
|
||||
|
||||
Final matcher score in `StoryIntelligenceIllustrationMatchingService.Score`:
|
||||
|
||||
- Age exact match: `+25`
|
||||
- Age evidence unknown: `+12`
|
||||
- Candidate age unknown: `+8` plus rejection note
|
||||
- Presentation exact match with strong evidence: `+42`
|
||||
- Presentation exact match without strong evidence: `+25`
|
||||
- Presentation evidence unknown: half weight
|
||||
- Candidate presentation unknown: one-third weight plus rejection note
|
||||
- Hair colour exact match: `+20`
|
||||
- Hair colour evidence unknown: `+10`
|
||||
- Candidate hair colour unknown: `+6` plus rejection note
|
||||
- Skin tone exact match: `+20`
|
||||
- Skin tone evidence unknown: `+10`
|
||||
- Candidate skin tone unknown: `+6` plus rejection note
|
||||
- Hair length exact match: `+8`
|
||||
- Glasses exact match: `+7`
|
||||
- Facial hair exact match: `+7`
|
||||
- Approved library item: `+3`
|
||||
|
||||
Score is clamped between `0` and `100`.
|
||||
|
||||
Minimum suitable score:
|
||||
|
||||
- `48`
|
||||
|
||||
Reassignment margin:
|
||||
|
||||
- `24`
|
||||
|
||||
Older stable-code scoring before the final matcher:
|
||||
|
||||
- Age exact: `+80`
|
||||
- Compatible age: `+30`
|
||||
- Age mismatch: `-45`
|
||||
- Presentation exact: `+24`
|
||||
- Hair colour exact: `+18`
|
||||
- Hair length exact: `+10`
|
||||
- Signal keyword match: `+14` each
|
||||
|
||||
### Why Existing Illustrations Are Retained
|
||||
|
||||
An existing illustration is retained when:
|
||||
|
||||
- Existing assignment still resolves to a public image URL.
|
||||
- Existing candidate has no hard rejections.
|
||||
- Existing candidate is not materially wrong for known age, presentation, hair colour or skin tone.
|
||||
- No better candidate exists, or the best candidate does not beat existing score by the reassignment margin.
|
||||
- Existing score is at least `48`.
|
||||
|
||||
### Why Existing Illustrations Are Replaced
|
||||
|
||||
An existing illustration is replaced when:
|
||||
|
||||
- It violates hard constraints.
|
||||
- It is materially wrong for age, presentation, hair colour or skin tone.
|
||||
- A better compatible candidate beats the existing score by at least `24`.
|
||||
- The existing assignment was fallback/null and a generated valid item is now available.
|
||||
|
||||
### When Demand Is Created
|
||||
|
||||
A character demand record is created when no hard-compatible unused illustration scores at least `48`.
|
||||
|
||||
Demand key:
|
||||
|
||||
```text
|
||||
character|age|presentation|hairColour|skinTone
|
||||
```
|
||||
|
||||
Writes:
|
||||
|
||||
- `StoryIntelligenceIllustrationDemands`
|
||||
|
||||
If request count reaches threshold, currently `1`, it creates or queues:
|
||||
|
||||
```text
|
||||
char-demand-{hash}
|
||||
```
|
||||
|
||||
### When Generated Illustrations Become Eligible
|
||||
|
||||
A generated illustration becomes eligible when:
|
||||
|
||||
- It exists in `IllustrationLibraryItems`
|
||||
- Status is `Generated` or `Approved`
|
||||
- It has a valid thumbnail or file path
|
||||
- A later snapshot/matcher pass reads it from the library
|
||||
|
||||
### Why Duplicate Portraits Sometimes Occur
|
||||
|
||||
Duplicate portraits can occur because:
|
||||
|
||||
1. Exact stable-code resolution can apply the same starter code before final assignment.
|
||||
2. Duplicate assignment cleanup only handles assignment records, not every transient fallback/stable-code image.
|
||||
3. Low-significance or unresolved identities can share generic fallbacks.
|
||||
4. Character identity canonicalisation can split one person into multiple keys.
|
||||
5. Existing duplicate assignment handling keeps the first retained item and clears later duplicates only during the matcher pass.
|
||||
|
||||
### Why Unknown Figure Can Still Appear
|
||||
|
||||
Unknown Figure can still appear when:
|
||||
|
||||
- The initial stable-code phase chooses `char-unknown-figure`.
|
||||
- The character is unresolved/generic, so named-character hard rejection does not apply.
|
||||
- The matcher records demand and falls back to an existing SVG/fallback.
|
||||
- The library contains generated/approved unknown figure art used by exact code before the final assignment overrides it.
|
||||
|
||||
## Location Illustration Selection
|
||||
|
||||
### Classification
|
||||
|
||||
Locations are built by `StoryIntelligenceVisualisationSnapshotService.BuildLocation`.
|
||||
|
||||
It chooses:
|
||||
|
||||
1. Present-in-scene locations before mentioned-only locations.
|
||||
2. More specific names.
|
||||
3. Higher confidence.
|
||||
4. Fallback to scene setting.
|
||||
|
||||
### Semantic Typing
|
||||
|
||||
Semantic type is determined by `StoryIntelligenceIllustrationCompatibility.LocationType`.
|
||||
|
||||
Examples:
|
||||
|
||||
- Bathroom
|
||||
- Kitchen
|
||||
- Bedroom
|
||||
- LivingRoom
|
||||
- Hallway
|
||||
- Stairwell
|
||||
- Hospital
|
||||
- Pub
|
||||
- Beach
|
||||
- RailwayStation
|
||||
- Canal
|
||||
- Garage
|
||||
- HouseExterior
|
||||
- Office
|
||||
- School
|
||||
- WaitingRoom
|
||||
- Road
|
||||
- CarPark
|
||||
- Shop
|
||||
- StreetFurniture
|
||||
- FlatInterior
|
||||
- UnknownInterior
|
||||
|
||||
### Matching
|
||||
|
||||
Location matching has two phases:
|
||||
|
||||
1. Initial stable-code mapping in `LocationLibraryCode`.
|
||||
2. Demand-based exact generated lookup in `ApplyLocationDemandAsync`.
|
||||
|
||||
Important current behaviour:
|
||||
|
||||
- Bathroom, WaitingRoom and Road intentionally return no starter code.
|
||||
- This prevents borrowing wrong generic interiors.
|
||||
- Generated demand selection uses exact code `loc-demand-{hash}`.
|
||||
|
||||
There is a `LocationCompatible` helper, but generated supporting image selection currently uses exact demand stable-code lookup rather than weighted compatibility scoring.
|
||||
|
||||
### Demand Creation
|
||||
|
||||
Location demand is created if:
|
||||
|
||||
- Location type is not `UnknownInterior`.
|
||||
- Current image is still fallback.
|
||||
- No generated demand image exists for the exact demand code.
|
||||
|
||||
Demand key:
|
||||
|
||||
```text
|
||||
location|type
|
||||
```
|
||||
|
||||
Writes:
|
||||
|
||||
- `StoryIntelligenceIllustrationDemands`
|
||||
- `IllustrationLibraryItems` planned/queued demand item
|
||||
|
||||
### Reuse Rules
|
||||
|
||||
Locations are reusable by type/demand key. They are not unique per manuscript location.
|
||||
|
||||
## Asset Illustration Selection
|
||||
|
||||
### Classification
|
||||
|
||||
Assets are built by `StoryIntelligenceVisualisationSnapshotService.BuildAssets`.
|
||||
|
||||
It:
|
||||
|
||||
- Excludes mentioned-only assets.
|
||||
- Excludes low-value/background objects.
|
||||
- Groups by cleaned asset name.
|
||||
- Picks highest-confidence entry per name.
|
||||
- Takes top five by asset weight.
|
||||
|
||||
### Semantic Typing
|
||||
|
||||
Asset type is determined by `StoryIntelligenceIllustrationCompatibility.AssetType`.
|
||||
|
||||
Examples:
|
||||
|
||||
- Ambulance
|
||||
- Stretcher
|
||||
- BodyBag
|
||||
- Bottle
|
||||
- Money
|
||||
- Clothing
|
||||
- Document
|
||||
- Jewellery
|
||||
- Weapon
|
||||
- Hatchback
|
||||
- FamilyCar
|
||||
- SportsCar
|
||||
- Van
|
||||
- Car
|
||||
- CarPart
|
||||
- Tin
|
||||
- Parcel
|
||||
- Rucksack
|
||||
- Suitcase
|
||||
- Note
|
||||
- Letter
|
||||
- Notebook
|
||||
- Book
|
||||
- Telephone
|
||||
- Keys
|
||||
- Photograph
|
||||
- UnknownObject
|
||||
|
||||
### Colour Handling
|
||||
|
||||
Colour is determined by `AssetColour`.
|
||||
|
||||
Recognised values:
|
||||
|
||||
- Blue
|
||||
- Red
|
||||
- Green
|
||||
- Yellow
|
||||
- Black
|
||||
- White
|
||||
- Grey
|
||||
- Brown
|
||||
- Unknown
|
||||
|
||||
### Matching
|
||||
|
||||
Asset matching has two phases:
|
||||
|
||||
1. Initial stable-code mapping in `AssetLibraryCode`.
|
||||
2. Demand-based exact generated lookup in `ApplyAssetDemandAsync`.
|
||||
|
||||
Starter mappings include:
|
||||
|
||||
- Letter/Note/Document -> `asset-folded-letter`
|
||||
- Notebook/Book -> `asset-worn-notebook`
|
||||
- Car -> `asset-red-classic-car`
|
||||
- Keys -> `asset-old-keys`
|
||||
- Telephone -> `asset-phone-recorder`
|
||||
- Photograph -> `asset-photo-print`
|
||||
- Rucksack -> `asset-rucksack`
|
||||
|
||||
Generated demand selection uses exact stable code:
|
||||
|
||||
```text
|
||||
asset-demand-{hash}
|
||||
```
|
||||
|
||||
There is an `AssetCompatible` helper, but generated supporting-image selection currently uses exact demand code rather than weighted compatibility scoring.
|
||||
|
||||
### Demand Creation
|
||||
|
||||
Asset demand is created if:
|
||||
|
||||
- Asset type is not `UnknownObject`.
|
||||
- Current image is fallback.
|
||||
- No generated image exists for exact demand code.
|
||||
|
||||
Demand key:
|
||||
|
||||
```text
|
||||
asset|type|colour
|
||||
```
|
||||
|
||||
Writes:
|
||||
|
||||
- `StoryIntelligenceIllustrationDemands`
|
||||
- `IllustrationLibraryItems` planned/queued demand item
|
||||
|
||||
## Database Table Summary
|
||||
|
||||
Reads:
|
||||
|
||||
- `StoryIntelligenceRuns`
|
||||
- `StoryIntelligenceChapterResults`
|
||||
- `StoryIntelligenceSceneResults`
|
||||
- `StoryIntelligenceBookPipelines`
|
||||
- `IllustrationLibraryItems`
|
||||
- `StoryIntelligenceCharacterIllustrationAssignments`
|
||||
|
||||
Writes:
|
||||
|
||||
- `StoryIntelligenceRuns`
|
||||
- `StoryIntelligenceChapterResults`
|
||||
- `StoryIntelligenceSceneResults`
|
||||
- `StoryIntelligenceCharacterIllustrationAssignments`
|
||||
- `StoryIntelligenceIllustrationDemands`
|
||||
- `IllustrationLibraryItems`
|
||||
|
||||
No dedicated persistent Story Memory table is currently used in the illustration matcher.
|
||||
|
||||
## Cached Values Used
|
||||
|
||||
The system uses these cached or derived values:
|
||||
|
||||
- Stored `ParsedJson` in scene results
|
||||
- Stored illustration library metadata/specification JSON
|
||||
- Stored generated file/thumbnail paths
|
||||
- Stored active character assignments
|
||||
- Stored demand keys/request counts
|
||||
- Snapshot `ChangeToken`
|
||||
- Replay cursor index
|
||||
- In-memory Story Memory during snapshot construction
|
||||
- Stable hashes for visual entity IDs and demand codes
|
||||
|
||||
## Previous Assignments Considered
|
||||
|
||||
The matcher considers active previous assignments from `StoryIntelligenceCharacterIllustrationAssignments`.
|
||||
|
||||
Used fields include:
|
||||
|
||||
- `CharacterKey`
|
||||
- `IllustrationLibraryItemID`
|
||||
- `StableCode`
|
||||
- `MatchConfidence`
|
||||
- `MatchingScore`
|
||||
- `ReasonChosen`
|
||||
- `ReasonReassigned`
|
||||
|
||||
Current limitation:
|
||||
|
||||
- Previous `EvidenceJson` and `CandidateDiagnosticsJson` are not returned by `ListAssignmentsAsync`; they are selected as null.
|
||||
|
||||
## Story Memory Evidence Considered
|
||||
|
||||
Story Memory can influence:
|
||||
|
||||
- Added remembered characters
|
||||
- POV identity
|
||||
- Character role/relevance
|
||||
- Carried relationships
|
||||
- Inferred kinship relationships
|
||||
- Rough presentation and age band
|
||||
|
||||
Story Memory does not directly persist or supply a structured evidence object to the final matcher. It affects the visual character state, and the matcher then infers evidence from that state.
|
||||
|
||||
## Character Evidence Ignored
|
||||
|
||||
Currently ignored or mostly lost for matching:
|
||||
|
||||
- Eye colour
|
||||
- Ethnicity
|
||||
- Clothing details
|
||||
- Body type
|
||||
- Exact age beyond broad age band
|
||||
- Many action-level descriptors
|
||||
- Raw manuscript text
|
||||
- Prior assignment evidence JSON
|
||||
- Scene observations unless transformed into role/relevance/relationship text
|
||||
|
||||
## Five Biggest Weaknesses
|
||||
|
||||
1. Two image systems compete: stable-code starter matching and final assignment matching.
|
||||
2. Story Memory is rebuilt in memory and is not a durable structured evidence store.
|
||||
3. Character evidence is compressed before matching.
|
||||
4. Locations and assets do not have a real weighted matcher.
|
||||
5. Previous evidence is stored but not reloaded as evidence for conflict resolution.
|
||||
|
||||
## Five Places Evidence Can Be Lost
|
||||
|
||||
1. AI scene JSON may omit physical details.
|
||||
2. Snapshot character state drops raw action and descriptor detail.
|
||||
3. Story Memory stores only simplified role/relevance/age/presentation.
|
||||
4. Matcher ignores eye colour, ethnicity, clothing, build and exact age.
|
||||
5. Assignment repository does not return previous evidence JSON to the matcher.
|
||||
|
||||
## Five Places Incorrect Assignments Can Occur
|
||||
|
||||
1. Initial stable-code fallback picks a plausible but wrong starter image.
|
||||
2. Hard-coded name gender lists misclassify unusual or ambiguous names.
|
||||
3. Relationship/family terms leak onto the wrong character.
|
||||
4. Asset/location keyword order maps vague text too broadly.
|
||||
5. Canonical identity splitting creates multiple assignment keys for one person.
|
||||
|
||||
## Five Quickest Improvements
|
||||
|
||||
1. Store structured evidence per canonical identity and reload it during matching.
|
||||
2. Remove or demote pre-matcher stable-code character assignment for live data.
|
||||
3. Add weighted matchers for locations and assets.
|
||||
4. Add eye colour, ethnicity/skin-tone source, clothing and exact-age evidence fields.
|
||||
5. Expand diagnostics to show winning candidate, rejected candidates and exact evidence source per entity.
|
||||
|
||||
@ -276,10 +276,12 @@ public sealed class OnboardingStoryIntelligenceService(
|
||||
var assetReview = await assetImport.BuildReviewAsync(batch);
|
||||
var relationshipReview = await relationshipImport.BuildReviewAsync(batch);
|
||||
var knowledgeReview = await knowledgeImport.BuildReviewAsync(batch);
|
||||
var pipeline = await pipelineState.GetForBookAsync(batch.BookID, batch.UserID);
|
||||
return new StoryIntelligenceProgressViewModel
|
||||
{
|
||||
BatchID = batch.BatchID,
|
||||
PreviewID = batch.PreviewID,
|
||||
ImportSessionID = pipeline?.StoryIntelligenceBookPipelineID,
|
||||
ProjectID = batch.ProjectID,
|
||||
BookID = batch.BookID,
|
||||
ProjectName = batch.ProjectName,
|
||||
|
||||
@ -11,6 +11,7 @@ namespace PlotLine.Services;
|
||||
public interface IStoryIntelligenceIllustrationMatchingService
|
||||
{
|
||||
Task ApplyCharacterIllustrationsAsync(int projectId, StoryIntelligenceExperiencePrototypeViewModel prototype);
|
||||
Task ApplyPersistedCharacterIllustrationsAsync(int projectId, StoryIntelligenceExperiencePrototypeViewModel prototype);
|
||||
Task ApplySupportingIllustrationDemandAsync(int projectId, StoryIntelligenceExperiencePrototypeViewModel prototype);
|
||||
}
|
||||
|
||||
@ -153,6 +154,67 @@ public sealed class StoryIntelligenceIllustrationMatchingService(
|
||||
}
|
||||
}
|
||||
|
||||
public async Task ApplyPersistedCharacterIllustrationsAsync(int projectId, StoryIntelligenceExperiencePrototypeViewModel prototype)
|
||||
{
|
||||
if (projectId <= 0 || prototype.Scenes.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var existingAssignments = (await assignments.ListAssignmentsAsync(projectId))
|
||||
.Where(item => item.IllustrationLibraryItemID.HasValue && !string.IsNullOrWhiteSpace(item.StableCode))
|
||||
.ToDictionary(item => item.CharacterKey, StringComparer.OrdinalIgnoreCase);
|
||||
if (existingAssignments.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var catalogue = (await library.ListAsync(new IllustrationLibraryFilter { Category = IllustrationLibraryCategories.Character }))
|
||||
.Where(item => item.Status is IllustrationLibraryStatuses.Approved or IllustrationLibraryStatuses.Generated)
|
||||
.Where(item => !string.IsNullOrWhiteSpace(item.StableCode))
|
||||
.GroupBy(item => item.IllustrationLibraryItemID)
|
||||
.ToDictionary(group => group.Key, group => group.First());
|
||||
|
||||
foreach (var character in prototype.Scenes.SelectMany(scene => scene.Characters))
|
||||
{
|
||||
if (!existingAssignments.TryGetValue(character.Id, out var assignment)
|
||||
|| assignment.IllustrationLibraryItemID is not int itemId
|
||||
|| !catalogue.TryGetValue(itemId, out var item))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var publicUrl = ResolveExistingPublicUploadUrl(item.ThumbnailPath) ?? ResolveExistingPublicUploadUrl(item.FilePath);
|
||||
if (string.IsNullOrWhiteSpace(publicUrl))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var fallbackImageUrl = string.IsNullOrWhiteSpace(character.ImageResolution.FallbackImageUrl)
|
||||
? character.ImagePath
|
||||
: character.ImageResolution.FallbackImageUrl;
|
||||
character.LibraryCode = item.StableCode;
|
||||
character.ImagePath = publicUrl;
|
||||
character.ImageResolution = new StoryIntelligenceExperienceImageResolution
|
||||
{
|
||||
EntityId = character.Id,
|
||||
StableCode = item.StableCode,
|
||||
IllustrationLibraryItemId = item.IllustrationLibraryItemID,
|
||||
Status = item.Status,
|
||||
FinalImageUrl = publicUrl,
|
||||
FallbackImageUrl = fallbackImageUrl,
|
||||
FallbackUsed = false,
|
||||
AssignmentConfidence = assignment.MatchConfidence,
|
||||
MatchingScore = assignment.MatchingScore,
|
||||
ReasonChosen = FirstConfigured(assignment.ReasonChosen, "Replay used persisted character illustration assignment."),
|
||||
ReasonReassigned = assignment.ReasonReassigned,
|
||||
PreviousIllustration = assignment.StableCode,
|
||||
DemandStatus = "Replay read-only; demand generation disabled",
|
||||
IllustrationAllocationStatus = "Persisted replay assignment"
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
public async Task ApplySupportingIllustrationDemandAsync(int projectId, StoryIntelligenceExperiencePrototypeViewModel prototype)
|
||||
{
|
||||
if (projectId <= 0 || prototype.Scenes.Count == 0)
|
||||
@ -905,6 +967,9 @@ public sealed class StoryIntelligenceIllustrationMatchingService(
|
||||
private static bool IsUnknown(string? value)
|
||||
=> string.IsNullOrWhiteSpace(value) || string.Equals(value, "Unknown", StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
private static string FirstConfigured(params string?[] values)
|
||||
=> values.FirstOrDefault(value => !string.IsNullOrWhiteSpace(value)) ?? string.Empty;
|
||||
|
||||
private static bool IsUnresolvedName(string? value)
|
||||
{
|
||||
var clean = (value ?? string.Empty).Trim().ToLowerInvariant();
|
||||
|
||||
@ -138,6 +138,7 @@ public sealed class StoryIntelligenceProgressViewModel
|
||||
public StoryIntelligenceJobProgress Job { get; init; } = new();
|
||||
public Guid? BatchID { get; init; }
|
||||
public Guid? PreviewID { get; init; }
|
||||
public int? ImportSessionID { get; init; }
|
||||
public int ProjectID { get; init; }
|
||||
public int BookID { get; init; }
|
||||
public string? ProjectName { get; init; }
|
||||
|
||||
@ -66,6 +66,17 @@ public sealed class StoryIntelligenceExperienceSnapshotDiagnostics
|
||||
public string CurrentChangeToken { get; init; } = string.Empty;
|
||||
public string? AnalysisCursor { get; init; }
|
||||
public string? Warning { get; init; }
|
||||
public int ReplayIndex { get; init; }
|
||||
public int TotalReplayableScenes { get; init; }
|
||||
public string ReplayMilestone { get; init; } = string.Empty;
|
||||
public string ReplayState { get; init; } = string.Empty;
|
||||
public string SnapshotCacheStatus { get; init; } = string.Empty;
|
||||
public string RebuildVersion { get; init; } = string.Empty;
|
||||
public int FailedChaptersSkipped { get; init; }
|
||||
public string FailedChapterSummary { get; init; } = string.Empty;
|
||||
public string PanelVisibleCounts { get; init; } = string.Empty;
|
||||
public string PanelHiddenCounts { get; init; } = string.Empty;
|
||||
public int AiCallsDuringReplay { get; init; }
|
||||
}
|
||||
|
||||
public sealed class StoryIntelligenceExperienceRunOption
|
||||
@ -81,6 +92,11 @@ public sealed class StoryIntelligenceExperienceSceneState
|
||||
{
|
||||
public string Id { get; init; } = string.Empty;
|
||||
public bool IsCurrent { get; init; }
|
||||
public int? ChapterRunId { get; init; }
|
||||
public decimal? ChapterNumber { get; init; }
|
||||
public int? SceneResultId { get; init; }
|
||||
public int ReplayIndex { get; init; }
|
||||
public int GlobalSceneIndex { get; init; }
|
||||
public string ChapterLabel { get; init; } = string.Empty;
|
||||
public int SceneNumber { get; init; }
|
||||
public string Title { get; init; } = string.Empty;
|
||||
|
||||
@ -6,6 +6,7 @@
|
||||
var statusLabel = Model.Mode switch
|
||||
{
|
||||
"simulation" => "Simulation",
|
||||
"replay" => "Replay",
|
||||
"real" when Model.IsTerminal => "Analysis Complete",
|
||||
"real" => "Live Analysis",
|
||||
"error" => "Import Session Error",
|
||||
@ -14,6 +15,7 @@
|
||||
var sourceLabel = Model.Mode switch
|
||||
{
|
||||
"simulation" => "Simulation only",
|
||||
"replay" => $"Saved analysis from Import Session {Model.ImportSessionId:N0}",
|
||||
"real" => "Real snapshot",
|
||||
"error" => "No live snapshot",
|
||||
_ => "Choose a session"
|
||||
@ -55,6 +57,7 @@
|
||||
data-run-id="@Model.RunId"
|
||||
data-book-id="@Model.BookId"
|
||||
data-import-session-id="@Model.ImportSessionId"
|
||||
data-failed-chapters="@Model.Diagnostics.FailedChapterSummary"
|
||||
data-snapshot-url="@(Model.ImportSessionId.HasValue ? $"/api/story-intelligence/import-sessions/{Model.ImportSessionId.Value}/visualisation-snapshot" : Model.RunId.HasValue ? $"/api/story-intelligence/runs/{Model.RunId.Value}/visualisation-snapshot" : string.Empty)"
|
||||
data-change-token="@Model.ChangeToken"
|
||||
data-terminal="@Model.IsTerminal.ToString().ToLowerInvariant()">
|
||||
@ -70,7 +73,7 @@
|
||||
<span data-stage-label>Preparing</span>
|
||||
<span>Scene <strong data-current-scene>1</strong></span>
|
||||
</div>
|
||||
@if (Model.Mode == "real" && Model.BookId.HasValue)
|
||||
@if ((Model.Mode == "real" || Model.Mode == "replay") && Model.BookId.HasValue)
|
||||
{
|
||||
<a class="story-exp-background" href="/onboarding/story-intelligence/book/@Model.BookId.Value/continue">Return to Import</a>
|
||||
}
|
||||
@ -86,7 +89,7 @@
|
||||
<strong>Story Intelligence import sessions</strong>
|
||||
@foreach (var run in Model.AvailableRuns)
|
||||
{
|
||||
<a href="/Development/StoryIntelligenceExperience?importSessionId=@run.ImportSessionId">@run.Label</a>
|
||||
<a href="/Development/StoryIntelligenceExperience?importSessionId=@run.ImportSessionId&mode=replay">Replay @run.Label</a>
|
||||
}
|
||||
</nav>
|
||||
}
|
||||
@ -97,6 +100,26 @@
|
||||
@Model.SnapshotMessage
|
||||
</aside>
|
||||
}
|
||||
@if (TempData["StoryIntelligenceReplayMessage"] is string replayMessage)
|
||||
{
|
||||
<aside class="story-exp-status-message" role="status">
|
||||
@replayMessage
|
||||
</aside>
|
||||
}
|
||||
@if (Model.Mode == "replay" && Model.ImportSessionId.HasValue)
|
||||
{
|
||||
<form class="story-exp-rebuild" asp-action="RebuildStoryIntelligenceExperience" method="post">
|
||||
<input type="hidden" name="importSessionId" value="@Model.ImportSessionId.Value" />
|
||||
<label>
|
||||
Rebuild Visualisation from Saved Analysis
|
||||
<select name="rebuildScope">
|
||||
<option value="snapshots only">Snapshots only</option>
|
||||
<option value="all derived visualisation state">All derived visualisation state</option>
|
||||
</select>
|
||||
</label>
|
||||
<button type="submit">Rebuild</button>
|
||||
</form>
|
||||
}
|
||||
|
||||
<section class="story-exp-shell" aria-label="Story Intelligence visual prototype">
|
||||
<aside class="story-exp-left" aria-label="Analysis progress">
|
||||
@ -170,6 +193,23 @@
|
||||
</div>
|
||||
|
||||
<section class="story-exp-ribbon-wrap" aria-label="Recent scenes and timeline">
|
||||
@if (Model.Mode == "replay")
|
||||
{
|
||||
<div class="story-exp-replay-browser" aria-label="Replay scene browser">
|
||||
<button type="button" data-control="first">First</button>
|
||||
<button type="button" data-control="previous">Previous</button>
|
||||
<label>
|
||||
Chapter
|
||||
<select data-replay-chapter></select>
|
||||
</label>
|
||||
<label>
|
||||
Scene
|
||||
<select data-replay-scene></select>
|
||||
</label>
|
||||
<button type="button" data-control="next">Next</button>
|
||||
<button type="button" data-control="last">Last</button>
|
||||
</div>
|
||||
}
|
||||
<div class="story-exp-ribbon" data-scene-ribbon></div>
|
||||
<div class="story-exp-timeline" data-timeline></div>
|
||||
</section>
|
||||
@ -191,11 +231,11 @@
|
||||
</aside>
|
||||
</section>
|
||||
|
||||
@if (Model.Mode == "simulation")
|
||||
@if (Model.Mode == "simulation" || Model.Mode == "replay")
|
||||
{
|
||||
<aside class="story-exp-controls" aria-label="Prototype controls">
|
||||
<div>
|
||||
<span>Prototype controls</span>
|
||||
<span>@(Model.Mode == "replay" ? "Replay controls" : "Prototype controls")</span>
|
||||
<small>@sourceLabel</small>
|
||||
</div>
|
||||
<button type="button" data-control="restart">Restart</button>
|
||||
@ -204,7 +244,13 @@
|
||||
<button type="button" data-control="next">Next</button>
|
||||
<label>
|
||||
Speed
|
||||
<input type="range" min="3500" max="10000" step="500" value="6200" data-speed-control />
|
||||
<select data-speed-control>
|
||||
<option value="8000">0.5x</option>
|
||||
<option value="4200" selected>1x</option>
|
||||
<option value="2200">2x</option>
|
||||
<option value="1100">4x</option>
|
||||
<option value="550">8x</option>
|
||||
</select>
|
||||
</label>
|
||||
</aside>
|
||||
}
|
||||
@ -356,6 +402,30 @@
|
||||
<dt>Snapshot source</dt>
|
||||
<dd data-diagnostic-key="snapshotSource">@Model.Diagnostics.SnapshotSource</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Replay index</dt>
|
||||
<dd data-diagnostic-key="replayIndex">@Model.Diagnostics.ReplayIndex</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Replayable scenes</dt>
|
||||
<dd data-diagnostic-key="totalReplayableScenes">@Model.Diagnostics.TotalReplayableScenes</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Replay state</dt>
|
||||
<dd data-diagnostic-key="replayState">@Model.Diagnostics.ReplayState</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Failed chapters skipped</dt>
|
||||
<dd data-diagnostic-key="failedChaptersSkipped">@Model.Diagnostics.FailedChaptersSkipped</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Failed chapter gaps</dt>
|
||||
<dd data-diagnostic-key="failedChapterSummary">@Model.Diagnostics.FailedChapterSummary</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>AI calls during replay</dt>
|
||||
<dd data-diagnostic-key="aiCallsDuringReplay">@Model.Diagnostics.AiCallsDuringReplay</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt>Current Snapshot Version</dt>
|
||||
<dd data-diagnostics-current>0</dd>
|
||||
@ -454,6 +524,13 @@
|
||||
</div>
|
||||
</dl>
|
||||
</details>
|
||||
|
||||
<aside class="story-exp-detail" data-detail-panel hidden>
|
||||
<button type="button" class="story-exp-detail__close" data-detail-close>Close</button>
|
||||
<span data-detail-tone></span>
|
||||
<h2 data-detail-title></h2>
|
||||
<p data-detail-body></p>
|
||||
</aside>
|
||||
</main>
|
||||
|
||||
<script src="~/js/story-intelligence-experience-prototype.js" asp-append-version="true"></script>
|
||||
|
||||
@ -141,6 +141,11 @@
|
||||
{
|
||||
<a class="btn btn-outline-primary" asp-controller="Projects" asp-action="Index">Continue in background</a>
|
||||
}
|
||||
@if (Model.ImportSessionID.HasValue)
|
||||
{
|
||||
<a class="btn btn-outline-primary"
|
||||
href="/Development/StoryIntelligenceExperience?importSessionId=@Model.ImportSessionID.Value&mode=replay">Replay Story Intelligence</a>
|
||||
}
|
||||
@if (!Model.HasActiveRuns && Model.Chapters.Any(chapter => chapter.IsRunComplete || chapter.IsFailed || chapter.HasCompletedCommit))
|
||||
{
|
||||
<a class="btn btn-primary"
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user