Phase 20AF – Scene Analysis Reliability and Import Experience Reality Check

Docs/StoryIntelligence_Phase20AF_Report.md

@@ -0,0 +1,100 @@
+# Phase 20AF - Scene Analysis Reliability and Import Experience Reality Check
+
+## Summary
+
+This pass found that scene failures were mostly not analysis-quality failures. The saved run data shows scene intelligence often returned useful content, but the import preparation path rejected it because of parse audit wrappers, strict validation counts, or malformed/incomplete JSON in optional sections.
+
+The review/import path now prepares scenes from the same import objects the commit service consumes. Optional scene intelligence problems no longer automatically prevent a scene from being created.
+
+## Failure Diagnosis
+
+Local completed onboarding runs using `Chapter=gpt-5-mini; Scene=gpt-5-mini` show these patterns:
+
+- Some scene responses parsed after JSON repair or retry and were saved inside a `parsedScene` audit wrapper.
+- The importer only attempted to read direct `SceneIntelligenceScene` JSON and skipped the wrapped repaired result.
+- The importer filtered out any scene with `ValidationErrorsCount > 0`, even when the scene had enough information to create a PlotDirector scene.
+- Fully failed parse cases were usually malformed or incomplete JSON:
+  - `No complete JSON object was found.`
+  - invalid numeric confidence such as a decimal followed by whitespace.
+  - retry attempted once but still returned incomplete JSON.
+
+The failures were therefore caused by:
+
+- malformed or incomplete OpenAI JSON from `gpt-5-mini`;
+- importer/parser logic that was too strict for repaired/audited results;
+- validation errors being treated as fatal even when minimum scene import data existed.
+
+## Tolerance Changes Made
+
+Scene import preparation now:
+
+- reads both direct parsed scene JSON and repaired/retry audit wrappers containing `parsedScene`;
+- no longer skips a scene solely because `ValidationErrorsCount > 0`;
+- creates a limited-detail fallback scene when paragraph range is valid but optional analysis cannot be read;
+- uses generated fallback title/summary when needed;
+- skips optional metrics/POV/setting details when they cannot be read;
+- keeps developer diagnostics available through saved run details.
+
+Minimum scene creation now depends on:
+
+- scene number;
+- paragraph range;
+- source run/chapter reference;
+- usable summary or generated fallback summary.
+
+## User Experience Changes
+
+The progress page now has an explicit `Continue in background` action and explains:
+
+> PlotDirector will keep analysing your manuscript. You can close Word, close this browser tab, and return later. We’ll show this import on your Projects page while it is running.
+
+The Projects page now shows a compact active import card while an onboarding Story Intelligence batch is active:
+
+- book title;
+- chapter progress label;
+- `View progress` button.
+
+This gives users a route back after closing the browser tab.
+
+## Developer Diagnostics
+
+Failed or limited scenes remain auditable through Admin run details. For each scene result, the system stores:
+
+- raw OpenAI response;
+- assistant output;
+- parsed or repaired JSON audit;
+- retry flags and retry errors when available;
+- validation error/warning counts;
+- paragraph range and scene number.
+
+The onboarding review screen keeps author language, while admin diagnostics link to the technical run detail.
+
+## GPT-5 Comparison
+
+No local completed onboarding run using `Scene=gpt-5` was present in the database at the time of this pass.
+
+Local aggregate for completed onboarding runs:
+
+| Model | Runs | Detected scenes | Completed scenes | Failed scenes |
+|---|---:|---:|---:|---:|
+| Chapter=gpt-5-mini; Scene=gpt-5-mini | 17 | 94 | 84 | 13 |
+
+A real comparison still needs to be run with:
+
+- Chapter Structure: `gpt-5-mini`
+- Scene Intelligence: `gpt-5`
+
+against the same source chapters.
+
+## Expected Success Rate Change
+
+Before this pass, saved mini-model runs could show analysed scenes as unavailable for import because repaired/audited JSON or validation-warning scenes were excluded.
+
+After this pass, those scenes are prepared if they have valid paragraph ranges. In the inspected local failed cases, the failed scene records had valid ranges, so they should now be created as either normal or limited-detail scenes rather than being blocked solely by optional analysis defects.
+
+## Remaining Risks
+
+- A response with no usable paragraph range still cannot be safely imported.
+- The active Projects card currently uses the in-memory onboarding batch, so it covers closing the browser tab but not an application restart.
+- GPT-5 comparison has not yet been executed locally.
+- Fallback scenes may need author review because optional POV, setting, metrics or timeline detail may be missing.

PlotLine/Services/OnboardingStoryIntelligenceService.cs

@@ -535,6 +535,7 @@ public interface IOnboardingStoryIntelligenceBatchStore
     Task SaveAsync(OnboardingStoryIntelligenceBatch batch);
     Task<OnboardingStoryIntelligenceBatch?> GetAsync(int userId, Guid batchId);
     Task<OnboardingStoryIntelligenceBatch?> GetLatestForPreviewAsync(int userId, Guid previewId);
+    Task<OnboardingStoryIntelligenceBatch?> GetLatestActiveForUserAsync(int userId);
 }
 
 public sealed class OnboardingStoryIntelligenceBatchStore : IOnboardingStoryIntelligenceBatchStore
@@ -555,6 +556,12 @@ public sealed class OnboardingStoryIntelligenceBatchStore : IOnboardingStoryInte
             .Where(batch => batch.UserID == userId && batch.PreviewID == previewId)
             .OrderByDescending(batch => batch.CreatedUtc)
             .FirstOrDefault());
+
+    public Task<OnboardingStoryIntelligenceBatch?> GetLatestActiveForUserAsync(int userId)
+        => Task.FromResult(batches.Values
+            .Where(batch => batch.UserID == userId)
+            .OrderByDescending(batch => batch.CreatedUtc)
+            .FirstOrDefault(batch => batch.Items.Count > 0));
 }
 
 public sealed class OnboardingStoryIntelligenceBatch

PlotLine/Services/StoryIntelligenceImportCommitService.cs

@@ -158,14 +158,20 @@ public sealed class StoryIntelligenceImportCommitService(
         var purposeMap = lookupData.ScenePurposeTypes.ToDictionary(type => Clean(type.PurposeName), type => type.ScenePurposeTypeID, StringComparer.OrdinalIgnoreCase);
         var metricMap = BuildMetricMap(activeMetricTypes);
         var importScenes = new List<StoryIntelligenceSceneImportItem>();
-        foreach (var sceneResult in sceneResults
-            .Where(scene => scene.ValidationErrorsCount == 0)
-            .OrderBy(scene => scene.TemporarySceneNumber))
+        foreach (var sceneResult in sceneResults.OrderBy(scene => scene.TemporarySceneNumber))
         {
             var parsed = TryReadScene(sceneResult);
             if (parsed is null)
             {
-                warnings.Add($"Scene {sceneResult.TemporarySceneNumber:N0} could not be prepared automatically and needs review.");
+                var fallback = BuildFallbackImportScene(sceneResult, metricMap);
+                if (fallback is null)
+                {
+                    warnings.Add($"Scene {sceneResult.TemporarySceneNumber:N0} could not be prepared automatically and needs review.");
+                    continue;
+                }
+
+                warnings.Add($"Scene {sceneResult.TemporarySceneNumber:N0} will be created with limited detail because some analysis could not be read.");
+                importScenes.Add(fallback);
                 continue;
             }
 
@@ -246,6 +252,61 @@ public sealed class StoryIntelligenceImportCommitService(
         };
     }
 
+    private static StoryIntelligenceSceneImportItem? BuildFallbackImportScene(
+        StoryIntelligenceSavedSceneResult sceneResult,
+        IReadOnlyDictionary<string, SceneMetricType> metricMap)
+    {
+        if (!sceneResult.StartParagraph.HasValue || !sceneResult.EndParagraph.HasValue || sceneResult.StartParagraph > sceneResult.EndParagraph)
+        {
+            return null;
+        }
+
+        var summary = ExtractSummaryFromJsonText(sceneResult.OutputTextJson)
+            ?? ExtractSummaryFromJsonText(sceneResult.ParsedJson)
+            ?? $"Scene {sceneResult.TemporarySceneNumber:N0} from paragraphs {DisplayRange(sceneResult.StartParagraph, sceneResult.EndParagraph)}.";
+        var parsed = new SceneIntelligenceScene
+        {
+            SchemaVersion = "fallback",
+            Summary = new SceneIntelligenceSummary
+            {
+                Short = summary,
+                Detailed = summary,
+                Confidence = null
+            },
+            PointOfView = new SceneIntelligencePointOfView
+            {
+                CharacterName = null,
+                NarrativeMode = "Unknown",
+                Confidence = null
+            },
+            Setting = new SceneIntelligenceSetting
+            {
+                LocationName = null,
+                LocationType = "Unknown",
+                Confidence = null
+            },
+            Metrics = new Dictionary<string, SceneIntelligenceMetric>()
+        };
+
+        return new StoryIntelligenceSceneImportItem
+        {
+            SceneResultID = sceneResult.SceneResultID,
+            TemporarySceneNumber = sceneResult.TemporarySceneNumber,
+            StartParagraph = sceneResult.StartParagraph,
+            EndParagraph = sceneResult.EndParagraph,
+            SourceLabel = sceneResult.SourceLabel,
+            ParsedScene = parsed,
+            PovCharacterID = null,
+            PurposeTypeIDs = [],
+            Metrics = [],
+            SceneTitle = GenerateTitle(sceneResult.TemporarySceneNumber, summary),
+            Summary = summary,
+            PurposeNotes = $"Story Intelligence import source: run {sceneResult.StoryIntelligenceRunID:N0}, paragraphs {DisplayRange(sceneResult.StartParagraph, sceneResult.EndParagraph)}.{Environment.NewLine}Some optional analysis could not be read, so this scene was created with limited detail.",
+            OutcomeNotes = null,
+            ImportNoteText = "Scene created with limited Story Intelligence detail. Review the source manuscript for POV, setting, metrics and timeline notes."
+        };
+    }
+
     private async Task<Dictionary<string, int>> BuildCharacterMapAsync(int projectId)
     {
         var map = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
@@ -397,6 +458,12 @@ public sealed class StoryIntelligenceImportCommitService(
             {
                 return direct;
             }
+
+            using var document = JsonDocument.Parse(result.ParsedJson);
+            if (document.RootElement.TryGetProperty("parsedScene", out var parsedScene))
+            {
+                return parsedScene.Deserialize<SceneIntelligenceScene>(JsonOptions);
+            }
         }
         catch (JsonException)
         {
@@ -405,6 +472,78 @@ public sealed class StoryIntelligenceImportCommitService(
         return null;
     }
 
+    private static string? ExtractSummaryFromJsonText(string? value)
+    {
+        if (string.IsNullOrWhiteSpace(value))
+        {
+            return null;
+        }
+
+        try
+        {
+            using var document = JsonDocument.Parse(value);
+            if (TryReadSummary(document.RootElement, out var summary))
+            {
+                return TrimTo(summary!, 500);
+            }
+        }
+        catch (JsonException)
+        {
+        }
+
+        var shortIndex = value.IndexOf("\"short\"", StringComparison.OrdinalIgnoreCase);
+        if (shortIndex >= 0)
+        {
+            var colonIndex = value.IndexOf(':', shortIndex);
+            if (colonIndex >= 0)
+            {
+                var afterColon = value[(colonIndex + 1)..].TrimStart();
+                if (afterColon.Length > 1 && afterColon[0] == '"')
+                {
+                    var endIndex = afterColon.IndexOf('"', 1);
+                    if (endIndex > 1)
+                    {
+                        return TrimTo(afterColon[1..endIndex], 500);
+                    }
+                }
+            }
+        }
+
+        return null;
+    }
+
+    private static bool TryReadSummary(JsonElement element, out string? summary)
+    {
+        summary = null;
+        if (element.ValueKind != JsonValueKind.Object)
+        {
+            return false;
+        }
+
+        if (element.TryGetProperty("parsedScene", out var parsedScene))
+        {
+            return TryReadSummary(parsedScene, out summary);
+        }
+
+        if (!element.TryGetProperty("summary", out var summaryElement) || summaryElement.ValueKind != JsonValueKind.Object)
+        {
+            return false;
+        }
+
+        foreach (var propertyName in new[] { "short", "detailed" })
+        {
+            if (summaryElement.TryGetProperty(propertyName, out var valueElement)
+                && valueElement.ValueKind == JsonValueKind.String
+                && !string.IsNullOrWhiteSpace(valueElement.GetString()))
+            {
+                summary = valueElement.GetString();
+                return true;
+            }
+        }
+
+        return false;
+    }
+
     private static int FindId<T>(IReadOnlyList<T> items, Func<T, string> name, Func<T, int> id, params string[] preferredNames)
     {
         foreach (var preferred in preferredNames)

PlotLine/Services/StoryIntelligenceService.cs

@@ -68,6 +68,8 @@ public sealed class StubStoryIntelligenceProvider : IStoryIntelligenceProvider
 
 public sealed class StoryIntelligenceService(
     IStoryIntelligenceRepository repository,
+    IStoryIntelligenceResultRepository persistedRuns,
+    IOnboardingStoryIntelligenceBatchStore onboardingBatches,
     IOnboardingRepository onboarding,
     IProjectRepository projects,
     IBookRepository books,
@@ -146,6 +148,41 @@ public sealed class StoryIntelligenceService(
             return new StoryIntelligenceDashboardViewModel();
         }
 
+        var activeBatch = await onboardingBatches.GetLatestActiveForUserAsync(currentUser.UserId.Value);
+        if (activeBatch is not null)
+        {
+            var runs = new List<StoryIntelligenceSavedRun>();
+            foreach (var item in activeBatch.Items)
+            {
+                var run = await persistedRuns.GetRunAsync(item.RunID);
+                if (run is not null)
+                {
+                    runs.Add(run);
+                }
+            }
+
+            var activeRun = runs
+                .Where(run => run.Status is StoryIntelligenceRunStatuses.Pending or StoryIntelligenceRunStatuses.Running)
+                .OrderBy(run => run.ChapterNumber ?? decimal.MaxValue)
+                .ThenBy(run => run.StoryIntelligenceRunID)
+                .FirstOrDefault();
+            if (activeRun is not null)
+            {
+                var completedChapters = runs.Count(run => run.Status is StoryIntelligenceRunStatuses.Completed or StoryIntelligenceRunStatuses.CompletedWithWarnings);
+                var currentChapter = Math.Clamp(completedChapters + 1, 1, Math.Max(1, activeBatch.Items.Count));
+                return new StoryIntelligenceDashboardViewModel
+                {
+                    ShouldShow = true,
+                    ActiveBatchID = activeBatch.BatchID,
+                    ActiveImportBookTitle = activeBatch.BookTitle,
+                    ActiveImportProgressLabel = $"Chapter {currentChapter:N0} of {activeBatch.Items.Count:N0}",
+                    Title = "Story Intelligence is reading",
+                    Description = "PlotDirector is analysing your manuscript in the background.",
+                    ButtonText = "View progress"
+                };
+            }
+        }
+
         var state = await onboarding.GetAsync(currentUser.UserId.Value);
         return state?.ProjectID.HasValue == true && state.BookID.HasValue
             ? new StoryIntelligenceDashboardViewModel

PlotLine/ViewModels/OnboardingViewModels.cs

@@ -216,6 +216,10 @@ public sealed class StoryIntelligenceDashboardViewModel
     public string Description { get; init; } = "Prepare the optional analysis framework for your manuscript.";
     public string ButtonText { get; init; } = "Set up Story Intelligence";
     public StoryIntelligenceJobProgress? Job { get; init; }
+    public Guid? ActiveBatchID { get; init; }
+    public string? ActiveImportBookTitle { get; init; }
+    public string? ActiveImportProgressLabel { get; init; }
+    public bool HasActiveImport => ActiveBatchID.HasValue;
 }
 
 public sealed class CompanionPresenceViewModel

PlotLine/Views/Onboarding/StoryIntelligenceProgress.cshtml

@@ -30,7 +30,7 @@
         }
 
         <section class="alert alert-info">
-            You can close Word or this browser tab once analysis has started. PlotDirector is working from a saved manuscript snapshot.
+            PlotDirector will keep analysing your manuscript. You can close Word, close this browser tab, and return later. We’ll show this import on your Projects page while it is running.
         </section>
 
         <section class="story-live-progress-card" aria-label="Import progress">
@@ -133,6 +133,10 @@
                     <input type="hidden" name="batchId" value="@Model.BatchID" />
                     <button class="btn btn-outline-secondary" type="submit" disabled="@(!Model.HasActiveRuns)">Cancel analysis</button>
                 </form>
+                @if (Model.HasActiveRuns)
+                {
+                    <a class="btn btn-outline-primary" asp-controller="Projects" asp-action="Index">Continue in background</a>
+                }
                 @if (!Model.HasActiveRuns && Model.Chapters.Any(chapter => chapter.IsRunComplete || chapter.IsFailed || chapter.HasCompletedCommit))
                 {
                     <a class="btn btn-primary"

PlotLine/Views/Onboarding/StoryIntelligenceReview.cshtml

@@ -167,6 +167,7 @@
                                 <div><dt>Failure stage</dt><dd>@(chapter.FailureStage ?? "-")</dd></div>
                                 <div><dt>Tokens</dt><dd>@(chapter.TotalTokens?.ToString("N0") ?? "-")</dd></div>
                             </dl>
+                            <a class="btn btn-outline-secondary btn-sm" asp-controller="Admin" asp-action="StoryIntelligenceRunDetails" asp-route-id="@chapter.RunID">Open diagnostics</a>
                         </details>
                     }
                 </details>

PlotLine/Views/Projects/Index.cshtml

@@ -34,6 +34,25 @@
     </section>
 }
 
+@if (Model.StoryIntelligence.HasActiveImport)
+{
+    <section class="story-intelligence-dashboard-card story-intelligence-dashboard-card--compact"
+             aria-label="Active manuscript import">
+        <div>
+            <p class="eyebrow">Manuscript import</p>
+            <h2>Story Intelligence is reading</h2>
+            <p>
+                <strong>@(Model.StoryIntelligence.ActiveImportBookTitle ?? "Your manuscript")</strong>
+                <span class="d-block">Progress: @(Model.StoryIntelligence.ActiveImportProgressLabel ?? "In progress")</span>
+            </p>
+        </div>
+        <a class="btn btn-primary"
+           asp-controller="Onboarding"
+           asp-action="StoryIntelligenceProgress"
+           asp-route-batchId="@Model.StoryIntelligence.ActiveBatchID">View progress</a>
+    </section>
+}
+
 @if (TempData["ArchiveMessage"] is string archiveMessage)
 {
     <div class="alert alert-success">@archiveMessage</div>