Phase 20AG – Complete Scene Intelligence Import

PlotLine/Docs/Features/PlotDirector_Story_Intelligence_Visualisation_Design.md

@@ -0,0 +1,269 @@
+# PlotDirector Story Intelligence Visualisation
+
+## Design Vision (Future Phase)
+
+**Version:** Draft 1\
+**Status:** Future Feature\
+**Priority:** High
+
+## Purpose
+
+Replace the traditional progress page with a live visualisation of
+PlotDirector understanding the user's story.
+
+------------------------------------------------------------------------
+
+# Overview
+
+The current Story Intelligence progress page successfully reports
+progress, but it behaves like an engineering dashboard.
+
+It shows percentages, counters and diagnostics. Although technically
+informative, it does not demonstrate the value of PlotDirector nor
+encourage the user to remain engaged during what may be a lengthy
+analysis.
+
+Instead, Story Intelligence should become an experience.
+
+The objective is to allow users to literally watch PlotDirector build an
+understanding of their story.
+
+A living knowledge map rather than a loading screen.
+
+------------------------------------------------------------------------
+
+# Overall User Flow
+
+Welcome → Project → Book → Word Companion → Scan manuscript → Review
+chapters → Start Story Intelligence
+
+The wizard ends here.
+
+The user sees:
+
+-   Story Intelligence has begun.
+-   Analysis continues even if Word or the browser is closed.
+-   Typical novels may take between 30 minutes and several hours.
+-   We'll email you when analysis is complete.
+
+Buttons:
+
+-   Continue in Background
+-   Watch Story Intelligence
+
+------------------------------------------------------------------------
+
+# Experience 1 -- Background Processing
+
+Ideal for users who simply wish to import.
+
+Analysis continues entirely server-side.
+
+Completion provides:
+
+-   Email notification
+-   Notification inside PlotDirector
+-   Resume button from the Projects page
+
+------------------------------------------------------------------------
+
+# Experience 2 -- Watch Story Intelligence
+
+A dedicated full-screen experience.
+
+Not part of the wizard.
+
+Minimal chrome.
+
+Mission Control for authors.
+
+------------------------------------------------------------------------
+
+# Visual Style
+
+Dark theme.
+
+Deep blues.
+
+Subtle animated particles.
+
+Glowing connection lines.
+
+Calm motion.
+
+Inspired by:
+
+-   NASA Mission Control
+-   Obsidian Graph
+-   Neo4j Browser
+-   Apple-quality animation
+
+------------------------------------------------------------------------
+
+# Screen Layout
+
+Top: - PlotDirector - Story Intelligence - Current Book - Analysis
+Status - Continue in Background
+
+Centre: - Living knowledge graph
+
+Left: - Progress - Latest discoveries - Live activity feed
+
+Right: - Story observations - Knowledge threads - Interesting
+discoveries
+
+Bottom: - Current scene - Timeline - Recent scenes
+
+------------------------------------------------------------------------
+
+# Living Knowledge Graph
+
+Initially almost empty.
+
+As Story Intelligence progresses:
+
+-   Characters appear.
+-   Locations appear.
+-   Assets appear.
+-   Relationships appear.
+-   Timeline events appear.
+-   Knowledge threads appear.
+
+Connections animate into existence.
+
+The graph constantly grows.
+
+------------------------------------------------------------------------
+
+# Character Portraits
+
+Initially use a built-in library of approximately 300--500 illustrated
+portraits.
+
+These are generic visual identifiers, not canonical artwork.
+
+Categorised by:
+
+-   Gender
+-   Age
+-   Hair colour
+-   General appearance
+
+Authors may later replace them.
+
+------------------------------------------------------------------------
+
+# Live Discovery Feed
+
+Examples:
+
+-   Character discovered: Rosie MacDonald
+-   Location discovered: The Doweries
+-   Relationship discovered: Beth ↔ Rosie
+-   Asset discovered: Red TR6
+-   Timeline event: 24 December 1982
+-   Mystery thread: Who left the note?
+
+Older discoveries gently fade away.
+
+------------------------------------------------------------------------
+
+# Story Observations
+
+Examples:
+
+-   Beth now appears in seven scenes.
+-   Rosie has become central to the story.
+-   Three names may refer to the same person.
+-   A mystery thread has emerged.
+
+These should feel insightful rather than like warnings.
+
+------------------------------------------------------------------------
+
+# Current Scene
+
+Display:
+
+-   Scene number
+-   Short summary
+-   Progress
+
+Each completed scene joins the graph.
+
+------------------------------------------------------------------------
+
+# Timeline
+
+A growing ribbon along the bottom.
+
+Dates, scenes and events appear naturally.
+
+Later this becomes the PlotDirector timeline.
+
+------------------------------------------------------------------------
+
+# Animation Style
+
+Nodes gently appear.
+
+Connections draw themselves.
+
+The graph subtly rebalances.
+
+The camera slowly pans.
+
+Everything should feel elegant and calm.
+
+------------------------------------------------------------------------
+
+# SignalR
+
+All updates are pushed live using SignalR.
+
+No polling.
+
+------------------------------------------------------------------------
+
+# Completion
+
+When analysis finishes:
+
+Story Intelligence Complete
+
+Your manuscript is ready for review.
+
+Button:
+
+Review Imported Scenes
+
+------------------------------------------------------------------------
+
+# Long-Term Vision
+
+The same graph engine becomes:
+
+-   Relationship Explorer
+-   Character Explorer
+-   Knowledge Explorer
+-   Continuity Explorer
+-   Timeline Explorer
+
+During Story Intelligence it displays temporary AI discoveries.
+
+After import it displays real PlotDirector data.
+
+------------------------------------------------------------------------
+
+# Design Goal
+
+The user should feel:
+
+'I can actually see PlotDirector understanding my novel.'
+
+not
+
+'I'm waiting for a progress bar.'
+
+The visualisation should become one of PlotDirector's defining features
+and something authors genuinely enjoy watching.

PlotLine/Docs/StoryIntelligence_Phase20AG_SceneImport.md

@@ -0,0 +1,35 @@
+# Phase 20AG - Scene Intelligence Import
+
+Story Intelligence scene import now uses the stored Scene Intelligence JSON to populate the existing PlotDirector scene fields. This phase does not add new OpenAI requests or new analysis passes.
+
+## Imported Scene Fields
+
+- Scene title: generated from the imported scene number and short summary.
+- Short summary: saved to the same scene summary field used by manually authored scenes.
+- Detailed summary: saved to scene outcome notes.
+- Scene purpose: `observedFunction` is saved to scene purpose notes.
+- Purpose labels: matched to existing PlotDirector scene purpose labels when the observed purpose text or returned purpose labels match existing labels. New purpose labels are not created.
+- Point of view: linked only when the returned POV name resolves to an existing project character or alias. Unresolved POV is left blank and noted.
+- Metrics: supported project metric values are imported for Action, Emotion, Tension, Conflict, Mystery, Romance, Humour, Revelation and Pacing when matching metric types exist.
+- Time reference: the returned setting time is saved as the scene relative time text.
+- Timeline clues: factual timeline clues are saved as scene notes. Timeline events are not created.
+- Scene notes: useful non-duplicative atmosphere, ambiguity and descriptive observation notes are saved as Story Intelligence import notes.
+- Source tracking: each scene records the Story Intelligence run and source paragraph range.
+
+## Confidence
+
+Confidence remains stored in Story Intelligence audit data and metric notes. Author-facing onboarding review displays confidence as High, Medium or Low rather than numeric values.
+
+## Not Imported Yet
+
+The following Story Intelligence areas remain future phases:
+
+- Characters
+- Locations
+- Assets
+- Relationships
+- Knowledge changes
+- Continuity warnings
+- Real timeline events
+
+These are intentionally not created during scene import so that this phase only completes the scene-level import surface.

PlotLine/Services/OnboardingStoryIntelligenceService.cs

@@ -415,7 +415,7 @@ public sealed class OnboardingStoryIntelligenceService(
             {
                 SceneNumber = scene.TemporarySceneNumber,
                 Summary = Trim(FirstConfigured(scene.ParsedScene.Summary?.Short, scene.ParsedScene.Summary?.Detailed, $"Scene {scene.TemporarySceneNumber:N0}")!, 180),
-                Pov = FirstConfigured(scene.ParsedScene.PointOfView?.CharacterName, scene.ParsedScene.PointOfView?.NarrativeMode),
+                Pov = scene.PovCharacterID.HasValue ? FirstConfigured(scene.ParsedScene.PointOfView?.CharacterName, scene.ParsedScene.PointOfView?.NarrativeMode) : null,
                 Setting = BuildSetting(scene.ParsedScene.Setting),
                 Confidence = scene.ParsedScene.Summary?.Confidence ?? scene.ParsedScene.Setting?.Confidence,
                 HasWarnings = false

PlotLine/Services/StoryIntelligenceImportCommitService.cs

@@ -37,8 +37,10 @@ public sealed class StoryIntelligenceImportCommitService(
         ["mystery"] = ["Mystery"],
         ["romance"] = ["Romance", "Emotional Intimacy"],
         ["humour"] = ["Humour", "Humor", "Comedy", "Comic Relief"],
+        ["humor"] = ["Humour", "Humor", "Comedy", "Comic Relief"],
         ["revelation"] = ["Revelation", "Reveal"],
-        ["pacingIntensity"] = ["PacingIntensity", "Pacing Intensity", "Overall Intensity"]
+        ["pacing"] = ["Pacing", "PacingIntensity", "Pacing Intensity", "Overall Intensity"],
+        ["pacingIntensity"] = ["Pacing", "PacingIntensity", "Pacing Intensity", "Overall Intensity"]
     };
 
     public async Task<StoryIntelligenceImportConfirmationViewModel?> BuildConfirmationAsync(int runId)
@@ -152,7 +154,7 @@ public sealed class StoryIntelligenceImportCommitService(
         var timelineNoteTypeId = noteTypes.FirstOrDefault(type => string.Equals(type.TypeName, "Timeline Note", StringComparison.OrdinalIgnoreCase))?.SceneNoteTypeID;
         if (timelineNoteTypeId is null)
         {
-            warnings.Add("Timeline clues will not be imported because the Timeline Note type is not available.");
+            warnings.Add("Timeline clues and Story Intelligence import notes will not be imported because the Timeline Note type is not available.");
         }
 
         var purposeMap = lookupData.ScenePurposeTypes.ToDictionary(type => Clean(type.PurposeName), type => type.ScenePurposeTypeID, StringComparer.OrdinalIgnoreCase);
@@ -242,12 +244,12 @@ public sealed class StoryIntelligenceImportCommitService(
             SourceLabel = sceneResult.SourceLabel,
             ParsedScene = parsed,
             PovCharacterID = povCharacterId,
-            PurposeTypeIDs = MatchPurposes(parsed.ScenePurpose?.ObservedFunction, purposeMap),
+            PurposeTypeIDs = MatchPurposes(parsed, purposeMap),
             Metrics = BuildMetrics(parsed, metricMap),
             SceneTitle = sceneTitle,
             Summary = string.IsNullOrWhiteSpace(summary) ? null : summary,
             PurposeNotes = BuildPurposeNotes(parsed, sceneResult, povName, povCharacterId),
-            OutcomeNotes = string.IsNullOrWhiteSpace(parsed.Summary?.Detailed) ? null : $"Detailed summary: {parsed.Summary.Detailed}",
+            OutcomeNotes = string.IsNullOrWhiteSpace(parsed.Summary?.Detailed) ? null : Clean(parsed.Summary.Detailed),
             ImportNoteText = BuildImportNote(parsed, povName, povCharacterId)
         };
     }
@@ -356,6 +358,7 @@ public sealed class StoryIntelligenceImportCommitService(
         }
 
         var values = new List<SceneMetricValue>();
+        var importedMetricTypeIds = new HashSet<int>();
         foreach (var (name, metric) in parsed.Metrics)
         {
             if (!metric.Score.HasValue || !metricMap.TryGetValue(name, out var metricType))
@@ -363,6 +366,11 @@ public sealed class StoryIntelligenceImportCommitService(
                 continue;
             }
 
+            if (!importedMetricTypeIds.Add(metricType.MetricTypeID))
+            {
+                continue;
+            }
+
             values.Add(new SceneMetricValue
             {
                 MetricTypeID = metricType.MetricTypeID,
@@ -375,16 +383,16 @@ public sealed class StoryIntelligenceImportCommitService(
         return values;
     }
 
-    private static IReadOnlyList<int> MatchPurposes(string? observedFunction, IReadOnlyDictionary<string, int> purposeMap)
+    private static IReadOnlyList<int> MatchPurposes(SceneIntelligenceScene parsed, IReadOnlyDictionary<string, int> purposeMap)
     {
-        var value = Clean(observedFunction);
+        var values = ReadPurposeTexts(parsed).Select(Clean).Where(value => !string.IsNullOrWhiteSpace(value)).Distinct(StringComparer.OrdinalIgnoreCase).ToList();
-        if (string.IsNullOrWhiteSpace(value))
+        if (values.Count == 0)
         {
             return [];
         }
 
         var matches = purposeMap
-            .Where(pair => value.Contains(pair.Key, StringComparison.OrdinalIgnoreCase) || pair.Key.Contains(value, StringComparison.OrdinalIgnoreCase))
+            .Where(pair => values.Any(value => value.Contains(pair.Key, StringComparison.OrdinalIgnoreCase) || pair.Key.Contains(value, StringComparison.OrdinalIgnoreCase)))
             .Select(pair => pair.Value)
             .Distinct()
             .ToList();
@@ -403,6 +411,12 @@ public sealed class StoryIntelligenceImportCommitService(
             lines.Add($"Observed function: {parsed.ScenePurpose.ObservedFunction}");
         }
 
+        var purposeLabels = ReadPurposeLabels(parsed.ScenePurpose).ToList();
+        if (purposeLabels.Count > 0)
+        {
+            lines.Add($"Purpose labels observed: {string.Join(", ", purposeLabels)}");
+        }
+
         if (parsed.Setting is not null)
         {
             var setting = string.Join("; ", new[]
@@ -430,10 +444,23 @@ public sealed class StoryIntelligenceImportCommitService(
     private static string? BuildImportNote(SceneIntelligenceScene parsed, string povName, int? povCharacterId)
     {
         var lines = new List<string>();
-        if (parsed.TimelineClues?.Count > 0)
+        var timelineClues = ReadFactualTimelineClues(parsed).ToList();
+        if (timelineClues.Count > 0)
         {
             lines.Add("Timeline clues:");
-            lines.AddRange(parsed.TimelineClues.Select(clue => $"- {Clean(clue.AbsoluteDate ?? clue.RelativeOrder ?? clue.Clue)}"));
+            lines.AddRange(timelineClues.Select(clue => $"- {clue}"));
+        }
+
+        var sceneNotes = ReadUsefulSceneNotes(parsed).ToList();
+        if (sceneNotes.Count > 0)
+        {
+            if (lines.Count > 0)
+            {
+                lines.Add(string.Empty);
+            }
+
+            lines.Add("Scene notes:");
+            lines.AddRange(sceneNotes.Select(note => $"- {note}"));
         }
 
         if (!string.IsNullOrWhiteSpace(povName) && !povCharacterId.HasValue)
@@ -444,6 +471,178 @@ public sealed class StoryIntelligenceImportCommitService(
         return lines.Count == 0 ? null : string.Join(Environment.NewLine, lines);
     }
 
+    private static IEnumerable<string> ReadPurposeTexts(SceneIntelligenceScene parsed)
+    {
+        if (!string.IsNullOrWhiteSpace(parsed.ScenePurpose?.ObservedFunction))
+        {
+            yield return parsed.ScenePurpose.ObservedFunction;
+        }
+
+        foreach (var label in ReadPurposeLabels(parsed.ScenePurpose))
+        {
+            yield return label;
+        }
+    }
+
+    private static IEnumerable<string> ReadPurposeLabels(SceneIntelligenceScenePurpose? purpose)
+    {
+        if (purpose?.ExtensionData is null)
+        {
+            yield break;
+        }
+
+        foreach (var propertyName in new[] { "purposeLabels", "labels", "label" })
+        {
+            if (!purpose.ExtensionData.TryGetValue(propertyName, out var element))
+            {
+                continue;
+            }
+
+            if (element.ValueKind == JsonValueKind.String)
+            {
+                var label = Clean(element.GetString());
+                if (!string.IsNullOrWhiteSpace(label))
+                {
+                    yield return label;
+                }
+            }
+            else if (element.ValueKind == JsonValueKind.Array)
+            {
+                foreach (var item in element.EnumerateArray())
+                {
+                    if (item.ValueKind == JsonValueKind.String)
+                    {
+                        var label = Clean(item.GetString());
+                        if (!string.IsNullOrWhiteSpace(label))
+                        {
+                            yield return label;
+                        }
+                    }
+                }
+            }
+        }
+    }
+
+    private static IEnumerable<string> ReadFactualTimelineClues(SceneIntelligenceScene parsed)
+    {
+        if (parsed.TimelineClues is null)
+        {
+            yield break;
+        }
+
+        foreach (var clue in parsed.TimelineClues)
+        {
+            var value = Clean(clue.AbsoluteDate ?? clue.RelativeOrder ?? clue.Clue);
+            if (string.IsNullOrWhiteSpace(value))
+            {
+                continue;
+            }
+
+            var source = $"{clue.Clue} {clue.RelativeOrder} {clue.AbsoluteDate} {clue.Evidence}";
+            if (source.Contains("speculat", StringComparison.OrdinalIgnoreCase)
+                || source.Contains("uncertain", StringComparison.OrdinalIgnoreCase)
+                || source.Contains("unknown", StringComparison.OrdinalIgnoreCase))
+            {
+                continue;
+            }
+
+            yield return value;
+        }
+    }
+
+    private static IEnumerable<string> ReadUsefulSceneNotes(SceneIntelligenceScene parsed)
+    {
+        var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+
+        foreach (var note in ReadSettingNotes(parsed.Setting))
+        {
+            if (seen.Add(note))
+            {
+                yield return note;
+            }
+        }
+
+        if (parsed.SourceLimits?.AmbiguityNotes is not null)
+        {
+            foreach (var ambiguity in parsed.SourceLimits.AmbiguityNotes.Select(Clean).Where(value => !string.IsNullOrWhiteSpace(value)))
+            {
+                var note = $"Ambiguity: {ambiguity}";
+                if (seen.Add(note))
+                {
+                    yield return note;
+                }
+            }
+        }
+
+        if (parsed.Observations is null)
+        {
+            yield break;
+        }
+
+        foreach (var observation in parsed.Observations)
+        {
+            var note = Clean(observation.Description);
+            if (string.IsNullOrWhiteSpace(note) || !IsUsefulSceneObservation(observation))
+            {
+                continue;
+            }
+
+            if (seen.Add(note))
+            {
+                yield return note;
+            }
+        }
+    }
+
+    private static IEnumerable<string> ReadSettingNotes(SceneIntelligenceSetting? setting)
+    {
+        if (setting is null)
+        {
+            yield break;
+        }
+
+        var atmosphere = ReadStringExtension(setting.ExtensionData, "atmosphere", "mood", "tone");
+        if (!string.IsNullOrWhiteSpace(atmosphere))
+        {
+            yield return $"Atmosphere: {atmosphere}";
+        }
+    }
+
+    private static bool IsUsefulSceneObservation(SceneIntelligenceObservation observation)
+    {
+        var type = Clean(observation.ObservationType);
+        var predicate = Clean(observation.Predicate);
+        return type.Contains("setting", StringComparison.OrdinalIgnoreCase)
+            || type.Contains("atmosphere", StringComparison.OrdinalIgnoreCase)
+            || type.Contains("scene", StringComparison.OrdinalIgnoreCase)
+            || predicate.Contains("observes", StringComparison.OrdinalIgnoreCase)
+            || predicate.Contains("reveals", StringComparison.OrdinalIgnoreCase)
+            || predicate.Contains("confirms", StringComparison.OrdinalIgnoreCase)
+            || predicate.Contains("discovers", StringComparison.OrdinalIgnoreCase);
+    }
+
+    private static string? ReadStringExtension(Dictionary<string, JsonElement>? data, params string[] names)
+    {
+        if (data is null)
+        {
+            return null;
+        }
+
+        foreach (var name in names)
+        {
+            if (data.TryGetValue(name, out var element) && element.ValueKind == JsonValueKind.String)
+            {
+                var value = Clean(element.GetString());
+                if (!string.IsNullOrWhiteSpace(value))
+                {
+                    return value;
+                }
+            }
+        }
+
+        return null;
+    }
+
     private static SceneIntelligenceScene? TryReadScene(StoryIntelligenceSavedSceneResult result)
     {
         if (string.IsNullOrWhiteSpace(result.ParsedJson))

PlotLine/Views/Onboarding/StoryIntelligenceReview.cshtml

@@ -285,7 +285,19 @@
     }
 
     private static string FormatConfidence(decimal? confidence)
-        => confidence.HasValue ? $"{Math.Clamp(confidence.Value, 0m, 1m):P0}" : "Not available";
+    {
+        if (!confidence.HasValue)
+        {
+            return "Not available";
+        }
+
+        var value = Math.Clamp(confidence.Value, 0m, 1m);
+        return value >= 0.75m
+            ? "High"
+            : value >= 0.45m
+                ? "Medium"
+                : "Low";
+    }
 
     private static string FormatCost(decimal? cost)
         => cost.HasValue ? $"About ${cost.Value:0.00}" : "Not available";