Phase 20K – Scene Intelligence Validation
This commit is contained in:
parent
d2c1841853
commit
6932518ec6
@ -0,0 +1,176 @@
|
||||
# Phase 20 -- Story Intelligence User Experience
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the intended user experience for Story
|
||||
Intelligence.
|
||||
|
||||
It complements the technical architecture documents by describing how
|
||||
Story Intelligence should feel from an author's perspective.
|
||||
|
||||
The goal is not simply to import a manuscript into PlotDirector.
|
||||
|
||||
The goal is to transform an existing manuscript into a rich, structured
|
||||
PlotDirector project with the minimum possible effort from the author.
|
||||
|
||||
Throughout this document the author remains the final authority.
|
||||
|
||||
Story Intelligence assists the author.
|
||||
|
||||
It never replaces them.
|
||||
|
||||
------------------------------------------------------------------------
|
||||
|
||||
# Design Philosophy
|
||||
|
||||
Story Intelligence should feel effortless.
|
||||
|
||||
The author should never feel that they are performing data entry simply
|
||||
so PlotDirector can understand their manuscript.
|
||||
|
||||
Instead, PlotDirector should quietly perform hours of structured work on
|
||||
the author's behalf.
|
||||
|
||||
The author should feel that they have handed PlotDirector their
|
||||
manuscript and, after a short wait, received a populated writing project
|
||||
ready to explore.
|
||||
|
||||
The ideal experience is:
|
||||
|
||||
> "Upload my manuscript. Make a cup of tea. Come back to a project that
|
||||
> already knows my story."
|
||||
|
||||
------------------------------------------------------------------------
|
||||
|
||||
# Core Principles
|
||||
|
||||
## The manuscript is the source of truth.
|
||||
|
||||
Story Intelligence never rewrites the author's work. It observes,
|
||||
organises and suggests. The original manuscript always remains the
|
||||
authoritative version of the story.
|
||||
|
||||
## AI assists.
|
||||
|
||||
AI never decides. Every imported item is a suggestion. The author may
|
||||
change, merge, split or delete anything created during import.
|
||||
|
||||
## Structure before detail.
|
||||
|
||||
Before PlotDirector can understand a story it must understand its
|
||||
structure. Story Intelligence therefore builds the structural framework
|
||||
first before extracting detailed information.
|
||||
|
||||
## Review uncertainty, not certainty.
|
||||
|
||||
Authors should never be asked to review hundreds of AI decisions. Only
|
||||
items with low confidence should be highlighted.
|
||||
|
||||
## Creativity should never be interrupted.
|
||||
|
||||
Story Intelligence performs its work in the background. It should never
|
||||
repeatedly stop the author to request approval.
|
||||
|
||||
# Import Modes
|
||||
|
||||
## Automatic Import (Recommended)
|
||||
|
||||
The author supplies a manuscript and PlotDirector performs the complete
|
||||
Story Intelligence pipeline automatically. No manual approval is
|
||||
required during import.
|
||||
|
||||
## Guided Import
|
||||
|
||||
Guided Import performs the same analysis pipeline but pauses at selected
|
||||
stages to allow the author to review AI suggestions such as scene
|
||||
boundaries, character aliases, ambiguous locations and low-confidence
|
||||
relationships.
|
||||
|
||||
# Story Intelligence Pipeline
|
||||
|
||||
## Stage One -- Structural Analysis
|
||||
|
||||
Outputs include chapters, suggested scene boundaries, narrative
|
||||
structure, point-of-view changes and significant time shifts.
|
||||
|
||||
## Stage Two -- Semantic Analysis
|
||||
|
||||
Outputs include scene summaries, characters, locations, assets,
|
||||
relationships, knowledge changes, timeline clues, questions, story
|
||||
metrics and structured observations.
|
||||
|
||||
# Progress Experience
|
||||
|
||||
During processing PlotDirector should continuously communicate progress
|
||||
using friendly, reassuring messages such as:
|
||||
|
||||
- Reading Chapter 12...
|
||||
- Identifying dramatic scene changes...
|
||||
- Building character profiles...
|
||||
- Discovering important locations...
|
||||
- Tracking significant story objects...
|
||||
- Connecting relationships...
|
||||
- Analysing emotional pacing...
|
||||
- Mapping timeline events...
|
||||
- Checking story continuity...
|
||||
- Finalising your project...
|
||||
|
||||
# Import Completion
|
||||
|
||||
Rather than simply stating that the import has finished, PlotDirector
|
||||
should summarise what has been created.
|
||||
|
||||
Example:
|
||||
|
||||
- [x] 47 Chapters
|
||||
- [x] 183 Scenes
|
||||
- [x] 64 Characters
|
||||
- [x] 29 Locations
|
||||
- [x] 17 Story Assets
|
||||
- [x] Timeline created
|
||||
- [x] Story metrics generated
|
||||
|
||||
# Import Report
|
||||
|
||||
The Import Report should highlight only lower-confidence items, such as
|
||||
large scenes, possible aliases, ambiguous locations, low-confidence
|
||||
relationships and scene boundaries requiring review.
|
||||
|
||||
# Confidence Philosophy
|
||||
|
||||
Confidence guides review effort. It should never prevent import.
|
||||
|
||||
# Author Review
|
||||
|
||||
Review is always optional. Authors can begin using PlotDirector
|
||||
immediately after import and refine imported data over time.
|
||||
|
||||
# Future Improvements
|
||||
|
||||
Future capabilities may include additional AI providers, improved
|
||||
prompts, better scene boundary detection, richer continuity analysis,
|
||||
story health reports, character journey analysis and conversational "Ask
|
||||
My Story" features.
|
||||
|
||||
# Success Criteria
|
||||
|
||||
Story Intelligence succeeds when:
|
||||
|
||||
- A first-time user can import an existing manuscript without prior
|
||||
PlotDirector knowledge.
|
||||
- The resulting project immediately feels recognisable as their own
|
||||
book.
|
||||
- Manual setup has largely disappeared.
|
||||
- Review effort focuses only on uncertainty.
|
||||
- The author remains in complete creative control.
|
||||
- PlotDirector saves hours of repetitive project setup.
|
||||
|
||||
# Guiding Principle
|
||||
|
||||
Story Intelligence should never interrupt creativity.
|
||||
|
||||
It should quietly perform hours of structured work while the author
|
||||
makes a cup of tea.
|
||||
|
||||
When the author returns, PlotDirector should already understand the
|
||||
structure of their story.
|
||||
223
PlotLine/Models/SceneIntelligenceModels.cs
Normal file
223
PlotLine/Models/SceneIntelligenceModels.cs
Normal file
@ -0,0 +1,223 @@
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace PlotLine.Models;
|
||||
|
||||
public sealed class SceneIntelligenceScene
|
||||
{
|
||||
public string? SchemaVersion { get; init; }
|
||||
public SceneIntelligenceSceneReference? SceneReference { get; init; }
|
||||
public SceneIntelligenceSummary? Summary { get; init; }
|
||||
public SceneIntelligenceScenePurpose? ScenePurpose { get; init; }
|
||||
public SceneIntelligencePointOfView? PointOfView { get; init; }
|
||||
public SceneIntelligenceSetting? Setting { get; init; }
|
||||
public List<SceneIntelligenceCharacter>? Characters { get; init; }
|
||||
public List<SceneIntelligenceLocation>? Locations { get; init; }
|
||||
public List<SceneIntelligenceAsset>? Assets { get; init; }
|
||||
public List<SceneIntelligenceRelationship>? Relationships { get; init; }
|
||||
public List<SceneIntelligenceKnowledgeChange>? KnowledgeChanges { get; init; }
|
||||
public List<SceneIntelligenceTimelineClue>? TimelineClues { get; init; }
|
||||
public List<SceneIntelligenceQuestionRaised>? QuestionsRaised { get; init; }
|
||||
public List<SceneIntelligenceQuestionAnswered>? QuestionsAnswered { get; init; }
|
||||
public List<SceneIntelligenceObservation>? Observations { get; init; }
|
||||
public Dictionary<string, SceneIntelligenceMetric>? Metrics { get; init; }
|
||||
public SceneIntelligenceSourceLimits? SourceLimits { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceSceneReference
|
||||
{
|
||||
public int? ProjectId { get; init; }
|
||||
public int? BookId { get; init; }
|
||||
public int? ChapterId { get; init; }
|
||||
public int? SceneId { get; init; }
|
||||
public decimal? ChapterNumber { get; init; }
|
||||
public decimal? SceneNumber { get; init; }
|
||||
public string? SourceLabel { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceSummary
|
||||
{
|
||||
public string? Short { get; init; }
|
||||
public string? Detailed { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceScenePurpose
|
||||
{
|
||||
public string? ObservedFunction { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligencePointOfView
|
||||
{
|
||||
public string? CharacterName { get; init; }
|
||||
public string? NarrativeMode { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
public string? Evidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceSetting
|
||||
{
|
||||
public string? TimeOfDay { get; init; }
|
||||
public string? DateOrTimeReference { get; init; }
|
||||
public string? LocationName { get; init; }
|
||||
public string? LocationType { get; init; }
|
||||
public string? GenericRoomType { get; init; }
|
||||
public string? ParentLocationHint { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceCharacter
|
||||
{
|
||||
public string? Name { get; init; }
|
||||
public string? RoleInScene { get; init; }
|
||||
public bool? MentionedOnly { get; init; }
|
||||
public List<string>? Aliases { get; init; }
|
||||
public List<string>? Actions { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
public string? Notes { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceLocation
|
||||
{
|
||||
public string? Name { get; init; }
|
||||
public string? LocationType { get; init; }
|
||||
public string? GenericRoomType { get; init; }
|
||||
public string? ParentLocationHint { get; init; }
|
||||
public bool? PresentInScene { get; init; }
|
||||
public bool? MentionedOnly { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
public string? Notes { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceAsset
|
||||
{
|
||||
public string? Name { get; init; }
|
||||
public string? AssetType { get; init; }
|
||||
public string? Status { get; init; }
|
||||
public string? OwnerOrHolder { get; init; }
|
||||
public bool? MentionedOnly { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
public string? Notes { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceRelationship
|
||||
{
|
||||
public string? CharacterA { get; init; }
|
||||
public string? CharacterB { get; init; }
|
||||
public string? RelationshipSignal { get; init; }
|
||||
public string? Evidence { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceKnowledgeChange
|
||||
{
|
||||
public string? RecipientCharacter { get; init; }
|
||||
public string? KnowledgeItem { get; init; }
|
||||
public string? ChangeType { get; init; }
|
||||
public string? SourceCharacter { get; init; }
|
||||
public string? SourceType { get; init; }
|
||||
public string? Evidence { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceTimelineClue
|
||||
{
|
||||
public string? Clue { get; init; }
|
||||
public string? RelativeOrder { get; init; }
|
||||
public string? AbsoluteDate { get; init; }
|
||||
public string? Evidence { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceQuestionRaised
|
||||
{
|
||||
public string? Question { get; init; }
|
||||
public string? Scope { get; init; }
|
||||
public string? Evidence { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceQuestionAnswered
|
||||
{
|
||||
public string? Question { get; init; }
|
||||
public string? Answer { get; init; }
|
||||
public string? Evidence { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceObservation
|
||||
{
|
||||
public string? ObservationType { get; init; }
|
||||
public string? SubjectEntityType { get; init; }
|
||||
public string? SubjectName { get; init; }
|
||||
public string? ObjectEntityType { get; init; }
|
||||
public string? ObjectName { get; init; }
|
||||
public string? Predicate { get; init; }
|
||||
public string? Description { get; init; }
|
||||
public string? Evidence { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceMetric
|
||||
{
|
||||
public int? Score { get; init; }
|
||||
public decimal? Confidence { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
|
||||
public sealed class SceneIntelligenceSourceLimits
|
||||
{
|
||||
public bool? ContainsAmbiguity { get; init; }
|
||||
public List<string>? AmbiguityNotes { get; init; }
|
||||
|
||||
[JsonExtensionData]
|
||||
public Dictionary<string, JsonElement>? ExtensionData { get; init; }
|
||||
}
|
||||
17
PlotLine/Models/ValidationResult.cs
Normal file
17
PlotLine/Models/ValidationResult.cs
Normal file
@ -0,0 +1,17 @@
|
||||
namespace PlotLine.Models.StoryIntelligence;
|
||||
|
||||
public sealed class ValidationResult
|
||||
{
|
||||
public bool IsValid => Errors.Count == 0;
|
||||
public List<ValidationIssue> Warnings { get; init; } = [];
|
||||
public List<ValidationIssue> Errors { get; init; } = [];
|
||||
public List<ValidationIssue> Information { get; init; } = [];
|
||||
}
|
||||
|
||||
public sealed class ValidationIssue
|
||||
{
|
||||
public string Severity { get; init; } = string.Empty;
|
||||
public string Path { get; init; } = string.Empty;
|
||||
public string Message { get; init; } = string.Empty;
|
||||
public string SuggestedFix { get; init; } = string.Empty;
|
||||
}
|
||||
@ -187,6 +187,7 @@ public class Program
|
||||
builder.Services.AddSingleton<IStoryPromptBuilder, StoryPromptBuilder>();
|
||||
builder.Services.AddSingleton<IStoryPromptVersionService, StoryPromptVersionService>();
|
||||
builder.Services.AddHttpClient<IStoryIntelligenceClient, StoryIntelligenceClient>();
|
||||
builder.Services.AddScoped<IStorySceneValidator, StorySceneValidator>();
|
||||
builder.Services.AddScoped<IStoryIntelligenceDiagnosticsService, StoryIntelligenceDiagnosticsService>();
|
||||
builder.Services.AddScoped<IStoryIntelligenceDryRunService, StoryIntelligenceDryRunService>();
|
||||
builder.Services.AddScoped<IProjectAccessService, ProjectAccessService>();
|
||||
|
||||
@ -3,6 +3,7 @@ using System.Net;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Options;
|
||||
using PlotLine.Models;
|
||||
@ -185,7 +186,9 @@ public sealed class StoryIntelligenceClient(
|
||||
RawResponseText = responseText,
|
||||
Model = settings.Model,
|
||||
Duration = stopwatch.Elapsed,
|
||||
RetryCount = retryCount
|
||||
RetryCount = retryCount,
|
||||
InputTokens = GetTokenUsage(responseText).InputTokens,
|
||||
OutputTokens = GetTokenUsage(responseText).OutputTokens
|
||||
};
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
@ -274,6 +277,19 @@ public sealed class StoryIntelligenceClient(
|
||||
|
||||
private static string TrimForLog(string value)
|
||||
=> string.IsNullOrWhiteSpace(value) || value.Length <= 500 ? value : value[..500];
|
||||
|
||||
private static (int? InputTokens, int? OutputTokens) GetTokenUsage(string responseText)
|
||||
{
|
||||
try
|
||||
{
|
||||
var response = JsonSerializer.Deserialize<OpenAIResponseEnvelope>(responseText, JsonOptions);
|
||||
return (response?.Usage?.InputTokens, response?.Usage?.OutputTokens);
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return (null, null);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class StoryIntelligenceDiagnosticsService(
|
||||
@ -327,12 +343,14 @@ public sealed class StoryIntelligenceDryRunService(
|
||||
IStoryPromptVersionService versions,
|
||||
IStoryPromptBuilder builder,
|
||||
IStoryIntelligenceClient client,
|
||||
IStorySceneValidator validator,
|
||||
ILogger<StoryIntelligenceDryRunService> logger) : IStoryIntelligenceDryRunService
|
||||
{
|
||||
private const string ScenePromptFile = "Scene-Prompt-V1.md";
|
||||
private static readonly JsonSerializerOptions JsonOptions = new()
|
||||
{
|
||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||
PropertyNameCaseInsensitive = true,
|
||||
WriteIndented = true
|
||||
};
|
||||
|
||||
@ -356,6 +374,10 @@ public sealed class StoryIntelligenceDryRunService(
|
||||
var template = await prompts.LoadPromptAsync(ScenePromptFile, cancellationToken);
|
||||
var completedPrompt = builder.BuildPrompt(template, contextJson, form.SceneText);
|
||||
var result = await client.ExecutePromptAsync(completedPrompt, promptVersion, cancellationToken);
|
||||
var sceneJson = ExtractSceneJson(result.RawResponseText);
|
||||
var parsedScene = DeserializeScene(sceneJson);
|
||||
var validation = validator.Validate(parsedScene);
|
||||
|
||||
model.Result = new StoryIntelligenceDryRunResultViewModel
|
||||
{
|
||||
Success = true,
|
||||
@ -364,17 +386,46 @@ public sealed class StoryIntelligenceDryRunService(
|
||||
SceneContextJson = contextJson,
|
||||
Duration = FormatDuration(result.Duration),
|
||||
RetryCount = result.RetryCount,
|
||||
RawResponseText = result.RawResponseText
|
||||
InputTokens = result.InputTokens,
|
||||
OutputTokens = result.OutputTokens,
|
||||
RawResponseText = result.RawResponseText,
|
||||
SceneJsonText = sceneJson,
|
||||
ParsedScene = parsedScene,
|
||||
Validation = validation
|
||||
};
|
||||
|
||||
logger.LogInformation(
|
||||
"Admin Story Intelligence dry run completed. PromptVersion={PromptVersion} Model={Model} DurationMs={DurationMs} RetryCount={RetryCount} Success={Success}",
|
||||
"Admin Story Intelligence dry run completed. PromptVersion={PromptVersion} Model={Model} DurationMs={DurationMs} RetryCount={RetryCount} InputTokens={InputTokens} OutputTokens={OutputTokens} ValidationPassed={ValidationPassed} ValidationWarnings={ValidationWarnings} ValidationErrors={ValidationErrors} Success={Success}",
|
||||
promptVersion,
|
||||
result.Model,
|
||||
result.Duration.TotalMilliseconds,
|
||||
result.RetryCount,
|
||||
result.InputTokens,
|
||||
result.OutputTokens,
|
||||
validation.IsValid,
|
||||
validation.Warnings.Count,
|
||||
validation.Errors.Count,
|
||||
true);
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
model.Result = new StoryIntelligenceDryRunResultViewModel
|
||||
{
|
||||
Success = false,
|
||||
PromptVersion = promptVersion,
|
||||
Model = clientStatus.Model,
|
||||
SceneContextJson = contextJson,
|
||||
ErrorMessage = $"Scene Intelligence JSON could not be deserialised: {ex.Message}",
|
||||
Validation = validator.Validate(null)
|
||||
};
|
||||
|
||||
logger.LogWarning(
|
||||
ex,
|
||||
"Admin Story Intelligence dry run deserialisation failed. PromptVersion={PromptVersion} Model={Model} Success={Success}",
|
||||
promptVersion,
|
||||
clientStatus.Model,
|
||||
false);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
model.Result = new StoryIntelligenceDryRunResultViewModel
|
||||
@ -413,6 +464,53 @@ public sealed class StoryIntelligenceDryRunService(
|
||||
=> duration.TotalSeconds < 1
|
||||
? $"{duration.TotalMilliseconds:0} ms"
|
||||
: $"{duration.TotalSeconds:0.00} sec";
|
||||
|
||||
private static string ExtractSceneJson(string rawResponseText)
|
||||
{
|
||||
var response = JsonSerializer.Deserialize<OpenAIResponseEnvelope>(rawResponseText, JsonOptions)
|
||||
?? throw new JsonException("OpenAI response envelope was empty.");
|
||||
var outputText = response.Output?
|
||||
.SelectMany(item => item.Content ?? [])
|
||||
.FirstOrDefault(content => string.Equals(content.Type, "output_text", StringComparison.OrdinalIgnoreCase)
|
||||
&& !string.IsNullOrWhiteSpace(content.Text))
|
||||
?.Text;
|
||||
|
||||
if (string.IsNullOrWhiteSpace(outputText))
|
||||
{
|
||||
throw new JsonException("OpenAI response did not contain output_text content.");
|
||||
}
|
||||
|
||||
return outputText.Trim();
|
||||
}
|
||||
|
||||
private static SceneIntelligenceScene? DeserializeScene(string sceneJson)
|
||||
=> JsonSerializer.Deserialize<SceneIntelligenceScene>(sceneJson, JsonOptions);
|
||||
}
|
||||
|
||||
internal sealed class OpenAIResponseEnvelope
|
||||
{
|
||||
public List<OpenAIResponseOutputItem>? Output { get; init; }
|
||||
public OpenAIResponseUsage? Usage { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class OpenAIResponseOutputItem
|
||||
{
|
||||
public List<OpenAIResponseContentItem>? Content { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class OpenAIResponseContentItem
|
||||
{
|
||||
public string? Type { get; init; }
|
||||
public string? Text { get; init; }
|
||||
}
|
||||
|
||||
internal sealed class OpenAIResponseUsage
|
||||
{
|
||||
[JsonPropertyName("input_tokens")]
|
||||
public int? InputTokens { get; init; }
|
||||
|
||||
[JsonPropertyName("output_tokens")]
|
||||
public int? OutputTokens { get; init; }
|
||||
}
|
||||
|
||||
public sealed class StoryIntelligenceClientConfigurationStatus
|
||||
|
||||
560
PlotLine/Services/StorySceneValidator.cs
Normal file
560
PlotLine/Services/StorySceneValidator.cs
Normal file
@ -0,0 +1,560 @@
|
||||
using System.Text.Json;
|
||||
using PlotLine.Models;
|
||||
using PlotLine.Models.StoryIntelligence;
|
||||
|
||||
namespace PlotLine.Services;
|
||||
|
||||
public interface IStorySceneValidator
|
||||
{
|
||||
ValidationResult Validate(SceneIntelligenceScene? scene);
|
||||
}
|
||||
|
||||
public sealed class StorySceneValidator : IStorySceneValidator
|
||||
{
|
||||
private static readonly HashSet<string> LocationTypes = CreateSet("named", "generic", "unclear");
|
||||
private static readonly HashSet<string> CharacterRoles = CreateSet("present", "pointOfView", "speaker", "mentioned", "unclear");
|
||||
private static readonly HashSet<string> KnowledgeChangeTypes = CreateSet("Learns", "Realises", "IsTold", "Discovers", "Confirms", "Suspects", "Misunderstands");
|
||||
private static readonly HashSet<string> ObservationTypes = CreateSet("CharacterAction", "CharacterKnowledge", "CharacterLocation", "RelationshipSignal", "AssetInteraction", "LocationSignal", "TimelineSignal", "QuestionSignal");
|
||||
private static readonly HashSet<string> EntityTypes = CreateSet("Character", "Location", "Asset", "Knowledge", "Relationship", "Timeline", "Scene", "Unknown");
|
||||
private static readonly HashSet<string> PreferredMetrics = CreateSet("action", "emotion", "tension", "conflict", "mystery", "romance", "humour", "revelation", "pacingIntensity");
|
||||
private static readonly HashSet<string> RelationshipSignals = CreateSet("trust", "mistrust", "affection", "fear", "authority", "secrecy", "conflict", "dependence", "protection", "rivalry", "obligation");
|
||||
|
||||
public ValidationResult Validate(SceneIntelligenceScene? scene)
|
||||
{
|
||||
var result = new ValidationResult();
|
||||
|
||||
try
|
||||
{
|
||||
if (scene is null)
|
||||
{
|
||||
AddError(result, "$", "Scene Intelligence JSON could not be deserialised into the V1 model.", "Return one valid Scene Intelligence JSON object.");
|
||||
return result;
|
||||
}
|
||||
|
||||
ValidateRoot(scene, result);
|
||||
ValidateSceneReference(scene.SceneReference, result);
|
||||
ValidateSummary(scene.Summary, result);
|
||||
ValidateScenePurpose(scene.ScenePurpose, result);
|
||||
ValidatePointOfView(scene.PointOfView, result);
|
||||
ValidateSetting(scene.Setting, result);
|
||||
ValidateCharacters(scene.Characters, result);
|
||||
ValidateLocations(scene.Locations, result);
|
||||
ValidateAssets(scene.Assets, result);
|
||||
ValidateRelationships(scene.Relationships, result);
|
||||
ValidateKnowledgeChanges(scene.KnowledgeChanges, result);
|
||||
ValidateTimelineClues(scene.TimelineClues, result);
|
||||
ValidateQuestionsRaised(scene.QuestionsRaised, result);
|
||||
ValidateQuestionsAnswered(scene.QuestionsAnswered, result);
|
||||
ValidateObservations(scene.Observations, result);
|
||||
ValidateMetrics(scene.Metrics, result);
|
||||
ValidateSourceLimits(scene.SourceLimits, result);
|
||||
|
||||
if (result.IsValid)
|
||||
{
|
||||
AddInfo(result, "$", "Validation passed.", "No action needed.");
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
AddError(result, "$", $"Validator failed internally: {ex.Message}", "Review validator implementation for this response shape.");
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
private static void ValidateRoot(SceneIntelligenceScene scene, ValidationResult result)
|
||||
{
|
||||
RequiredString(scene.SchemaVersion, "schemaVersion", result);
|
||||
if (!string.Equals(scene.SchemaVersion, "1.0", StringComparison.Ordinal))
|
||||
{
|
||||
AddError(result, "schemaVersion", "Schema version must be \"1.0\".", "Return schemaVersion as \"1.0\".");
|
||||
}
|
||||
|
||||
RequiredObject(scene.SceneReference, "sceneReference", result);
|
||||
RequiredObject(scene.Summary, "summary", result);
|
||||
RequiredObject(scene.ScenePurpose, "scenePurpose", result);
|
||||
RequiredObject(scene.PointOfView, "pointOfView", result);
|
||||
RequiredObject(scene.Setting, "setting", result);
|
||||
RequiredArray(scene.Characters, "characters", result);
|
||||
RequiredArray(scene.Locations, "locations", result);
|
||||
RequiredArray(scene.Assets, "assets", result);
|
||||
RequiredArray(scene.Relationships, "relationships", result);
|
||||
RequiredArray(scene.KnowledgeChanges, "knowledgeChanges", result);
|
||||
RequiredArray(scene.TimelineClues, "timelineClues", result);
|
||||
RequiredArray(scene.QuestionsRaised, "questionsRaised", result);
|
||||
RequiredArray(scene.QuestionsAnswered, "questionsAnswered", result);
|
||||
RequiredArray(scene.Observations, "observations", result);
|
||||
RequiredObject(scene.Metrics, "metrics", result);
|
||||
RequiredObject(scene.SourceLimits, "sourceLimits", result);
|
||||
Unknown(scene.ExtensionData, "$", result);
|
||||
}
|
||||
|
||||
private static void ValidateSceneReference(SceneIntelligenceSceneReference? value, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
RequiredScalar(value.ProjectId, "sceneReference.projectId", result, allowNull: true);
|
||||
RequiredScalar(value.BookId, "sceneReference.bookId", result, allowNull: true);
|
||||
RequiredScalar(value.ChapterId, "sceneReference.chapterId", result, allowNull: true);
|
||||
RequiredScalar(value.SceneId, "sceneReference.sceneId", result, allowNull: true);
|
||||
RequiredScalar(value.ChapterNumber, "sceneReference.chapterNumber", result, allowNull: true);
|
||||
RequiredScalar(value.SceneNumber, "sceneReference.sceneNumber", result, allowNull: true);
|
||||
RequiredString(value.SourceLabel, "sceneReference.sourceLabel", result);
|
||||
Unknown(value.ExtensionData, "sceneReference", result);
|
||||
}
|
||||
|
||||
private static void ValidateSummary(SceneIntelligenceSummary? value, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
RequiredString(value.Short, "summary.short", result);
|
||||
RequiredString(value.Detailed, "summary.detailed", result);
|
||||
Confidence(value.Confidence, "summary.confidence", result);
|
||||
Unknown(value.ExtensionData, "summary", result);
|
||||
}
|
||||
|
||||
private static void ValidateScenePurpose(SceneIntelligenceScenePurpose? value, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
RequiredString(value.ObservedFunction, "scenePurpose.observedFunction", result);
|
||||
Confidence(value.Confidence, "scenePurpose.confidence", result);
|
||||
Unknown(value.ExtensionData, "scenePurpose", result);
|
||||
}
|
||||
|
||||
private static void ValidatePointOfView(SceneIntelligencePointOfView? value, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
RequiredScalar(value.CharacterName, "pointOfView.characterName", result, allowNull: true);
|
||||
RequiredScalar(value.NarrativeMode, "pointOfView.narrativeMode", result, allowNull: true);
|
||||
Confidence(value.Confidence, "pointOfView.confidence", result);
|
||||
RequiredScalar(value.Evidence, "pointOfView.evidence", result, allowNull: true);
|
||||
Unknown(value.ExtensionData, "pointOfView", result);
|
||||
}
|
||||
|
||||
private static void ValidateSetting(SceneIntelligenceSetting? value, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
RequiredScalar(value.TimeOfDay, "setting.timeOfDay", result, allowNull: true);
|
||||
RequiredScalar(value.DateOrTimeReference, "setting.dateOrTimeReference", result, allowNull: true);
|
||||
RequiredScalar(value.LocationName, "setting.locationName", result, allowNull: true);
|
||||
Enum(value.LocationType, "setting.locationType", LocationTypes, result);
|
||||
RequiredScalar(value.GenericRoomType, "setting.genericRoomType", result, allowNull: true);
|
||||
RequiredScalar(value.ParentLocationHint, "setting.parentLocationHint", result, allowNull: true);
|
||||
Confidence(value.Confidence, "setting.confidence", result);
|
||||
|
||||
if (string.Equals(value.LocationType, "generic", StringComparison.Ordinal) && string.IsNullOrWhiteSpace(value.GenericRoomType))
|
||||
{
|
||||
AddWarning(result, "setting.genericRoomType", "Generic setting has no generic room type.", "Include genericRoomType when the manuscript supports it.");
|
||||
}
|
||||
|
||||
Unknown(value.ExtensionData, "setting", result);
|
||||
}
|
||||
|
||||
private static void ValidateCharacters(List<SceneIntelligenceCharacter>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
DuplicateNames(values.Select((item, index) => (item.Name, Path: $"characters[{index}].name")), "characters", result);
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"characters[{i}]";
|
||||
RequiredString(item.Name, $"{path}.name", result);
|
||||
Enum(item.RoleInScene, $"{path}.roleInScene", CharacterRoles, result);
|
||||
RequiredScalar(item.MentionedOnly, $"{path}.mentionedOnly", result);
|
||||
RequiredArray(item.Aliases, $"{path}.aliases", result);
|
||||
RequiredArray(item.Actions, $"{path}.actions", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
RequiredScalar(item.Notes, $"{path}.notes", result, allowNull: true);
|
||||
if (item.MentionedOnly == true && !string.Equals(item.RoleInScene, "mentioned", StringComparison.Ordinal))
|
||||
{
|
||||
AddWarning(result, $"{path}.roleInScene", "Mentioned-only character should usually use roleInScene \"mentioned\".", "Use roleInScene \"mentioned\" for absent references.");
|
||||
}
|
||||
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateLocations(List<SceneIntelligenceLocation>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
DuplicateNames(values.Select((item, index) => (item.Name, Path: $"locations[{index}].name")), "locations", result);
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"locations[{i}]";
|
||||
RequiredString(item.Name, $"{path}.name", result);
|
||||
Enum(item.LocationType, $"{path}.locationType", LocationTypes, result);
|
||||
RequiredScalar(item.GenericRoomType, $"{path}.genericRoomType", result, allowNull: true);
|
||||
RequiredScalar(item.ParentLocationHint, $"{path}.parentLocationHint", result, allowNull: true);
|
||||
RequiredScalar(item.PresentInScene, $"{path}.presentInScene", result);
|
||||
RequiredScalar(item.MentionedOnly, $"{path}.mentionedOnly", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
RequiredScalar(item.Notes, $"{path}.notes", result, allowNull: true);
|
||||
if (item.PresentInScene == false && item.MentionedOnly == false)
|
||||
{
|
||||
AddWarning(result, $"{path}.mentionedOnly", "Location is not present but is not marked mentioned-only.", "Set mentionedOnly to true when a location is only discussed.");
|
||||
}
|
||||
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateAssets(List<SceneIntelligenceAsset>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
DuplicateNames(values.Select((item, index) => (item.Name, Path: $"assets[{index}].name")), "assets", result);
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"assets[{i}]";
|
||||
RequiredString(item.Name, $"{path}.name", result);
|
||||
RequiredScalar(item.AssetType, $"{path}.assetType", result, allowNull: true);
|
||||
RequiredScalar(item.Status, $"{path}.status", result, allowNull: true);
|
||||
RequiredScalar(item.OwnerOrHolder, $"{path}.ownerOrHolder", result, allowNull: true);
|
||||
RequiredScalar(item.MentionedOnly, $"{path}.mentionedOnly", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
RequiredScalar(item.Notes, $"{path}.notes", result, allowNull: true);
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateRelationships(List<SceneIntelligenceRelationship>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"relationships[{i}]";
|
||||
RequiredString(item.CharacterA, $"{path}.characterA", result);
|
||||
RequiredString(item.CharacterB, $"{path}.characterB", result);
|
||||
RequiredString(item.RelationshipSignal, $"{path}.relationshipSignal", result);
|
||||
if (!string.IsNullOrWhiteSpace(item.RelationshipSignal) && !RelationshipSignals.Contains(item.RelationshipSignal))
|
||||
{
|
||||
AddWarning(result, $"{path}.relationshipSignal", "Relationship signal is outside the preferred V1 neutral terms.", "Use a supported neutral signal when one fits.");
|
||||
}
|
||||
|
||||
RequiredString(item.Evidence, $"{path}.evidence", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
var key = string.Join("|", new[] { item.CharacterA, item.CharacterB, item.RelationshipSignal }.Select(value => value?.Trim().ToLowerInvariant()));
|
||||
if (!string.IsNullOrWhiteSpace(key) && !seen.Add(key))
|
||||
{
|
||||
AddWarning(result, path, "Duplicate relationship signal found.", "Merge duplicate scene-local relationship signals when appropriate.");
|
||||
}
|
||||
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateKnowledgeChanges(List<SceneIntelligenceKnowledgeChange>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"knowledgeChanges[{i}]";
|
||||
RequiredString(item.RecipientCharacter, $"{path}.recipientCharacter", result);
|
||||
RequiredString(item.KnowledgeItem, $"{path}.knowledgeItem", result);
|
||||
Enum(item.ChangeType, $"{path}.changeType", KnowledgeChangeTypes, result);
|
||||
RequiredScalar(item.SourceCharacter, $"{path}.sourceCharacter", result, allowNull: true);
|
||||
RequiredScalar(item.SourceType, $"{path}.sourceType", result, allowNull: true);
|
||||
RequiredString(item.Evidence, $"{path}.evidence", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateTimelineClues(List<SceneIntelligenceTimelineClue>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"timelineClues[{i}]";
|
||||
RequiredString(item.Clue, $"{path}.clue", result);
|
||||
RequiredScalar(item.RelativeOrder, $"{path}.relativeOrder", result, allowNull: true);
|
||||
RequiredScalar(item.AbsoluteDate, $"{path}.absoluteDate", result, allowNull: true);
|
||||
RequiredString(item.Evidence, $"{path}.evidence", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateQuestionsRaised(List<SceneIntelligenceQuestionRaised>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"questionsRaised[{i}]";
|
||||
RequiredString(item.Question, $"{path}.question", result);
|
||||
RequiredString(item.Scope, $"{path}.scope", result);
|
||||
RequiredString(item.Evidence, $"{path}.evidence", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateQuestionsAnswered(List<SceneIntelligenceQuestionAnswered>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"questionsAnswered[{i}]";
|
||||
RequiredString(item.Question, $"{path}.question", result);
|
||||
RequiredString(item.Answer, $"{path}.answer", result);
|
||||
RequiredString(item.Evidence, $"{path}.evidence", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateObservations(List<SceneIntelligenceObservation>? values, ValidationResult result)
|
||||
{
|
||||
if (values is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var item = values[i];
|
||||
var path = $"observations[{i}]";
|
||||
Enum(item.ObservationType, $"{path}.observationType", ObservationTypes, result);
|
||||
Enum(item.SubjectEntityType, $"{path}.subjectEntityType", EntityTypes, result);
|
||||
RequiredString(item.SubjectName, $"{path}.subjectName", result);
|
||||
NullableEnum(item.ObjectEntityType, $"{path}.objectEntityType", EntityTypes, result);
|
||||
RequiredScalar(item.ObjectName, $"{path}.objectName", result, allowNull: true);
|
||||
RequiredString(item.Predicate, $"{path}.predicate", result);
|
||||
RequiredString(item.Description, $"{path}.description", result);
|
||||
RequiredString(item.Evidence, $"{path}.evidence", result);
|
||||
Confidence(item.Confidence, $"{path}.confidence", result);
|
||||
if (string.IsNullOrWhiteSpace(item.ObjectEntityType) != string.IsNullOrWhiteSpace(item.ObjectName))
|
||||
{
|
||||
AddWarning(result, path, "Observation object entity type and object name are inconsistent.", "Use both objectEntityType and objectName, or set both to null when no object applies.");
|
||||
}
|
||||
|
||||
Unknown(item.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateMetrics(Dictionary<string, SceneIntelligenceMetric>? metrics, ValidationResult result)
|
||||
{
|
||||
if (metrics is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var (name, metric) in metrics)
|
||||
{
|
||||
var path = $"metrics.{name}";
|
||||
if (!PreferredMetrics.Contains(name))
|
||||
{
|
||||
AddWarning(result, path, "Metric is outside the preferred V1 metric set.", "Use preferred V1 metrics unless the schema is intentionally expanded.");
|
||||
}
|
||||
|
||||
if (metric is null)
|
||||
{
|
||||
AddError(result, path, "Metric object is required when a metric is present.", "Return { \"score\": 1-10, \"confidence\": 0.0-1.0 }.");
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!metric.Score.HasValue)
|
||||
{
|
||||
AddError(result, $"{path}.score", "Metric score is required.", "Return a score from 1 to 10.");
|
||||
}
|
||||
else if (metric.Score < 1 || metric.Score > 10)
|
||||
{
|
||||
AddError(result, $"{path}.score", "Metric score must be from 1 to 10.", "Use the V1 1-10 descriptive intensity scale.");
|
||||
}
|
||||
|
||||
Confidence(metric.Confidence, $"{path}.confidence", result);
|
||||
Unknown(metric.ExtensionData, path, result);
|
||||
}
|
||||
}
|
||||
|
||||
private static void ValidateSourceLimits(SceneIntelligenceSourceLimits? value, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
RequiredScalar(value.ContainsAmbiguity, "sourceLimits.containsAmbiguity", result);
|
||||
RequiredArray(value.AmbiguityNotes, "sourceLimits.ambiguityNotes", result);
|
||||
Unknown(value.ExtensionData, "sourceLimits", result);
|
||||
}
|
||||
|
||||
private static void Confidence(decimal? value, string path, ValidationResult result)
|
||||
{
|
||||
if (!value.HasValue)
|
||||
{
|
||||
AddError(result, path, "Confidence is required.", "Return a confidence value from 0.0 to 1.0.");
|
||||
return;
|
||||
}
|
||||
|
||||
if (value < 0 || value > 1)
|
||||
{
|
||||
AddError(result, path, "Confidence must be from 0.0 to 1.0.", "Use the V1 confidence scale.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void Enum(string? value, string path, HashSet<string> allowed, ValidationResult result)
|
||||
{
|
||||
RequiredString(value, path, result);
|
||||
if (!string.IsNullOrWhiteSpace(value) && !allowed.Contains(value))
|
||||
{
|
||||
AddError(result, path, $"Value \"{value}\" is not allowed.", $"Use one of: {string.Join(", ", allowed)}.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void NullableEnum(string? value, string path, HashSet<string> allowed, ValidationResult result)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(value) && !allowed.Contains(value))
|
||||
{
|
||||
AddError(result, path, $"Value \"{value}\" is not allowed.", $"Use null or one of: {string.Join(", ", allowed)}.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void RequiredString(string? value, string path, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
AddError(result, path, "Required string property is missing or null.", "Return a supported string value.");
|
||||
}
|
||||
else if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
AddError(result, path, "Required string property is empty.", "Return a non-empty string supported by the scene.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void RequiredScalar<T>(T? value, string path, ValidationResult result, bool allowNull = false)
|
||||
{
|
||||
if (value is null && !allowNull)
|
||||
{
|
||||
AddError(result, path, "Required property is missing or null.", "Return the required property with a supported value.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void RequiredObject<T>(T? value, string path, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
AddError(result, path, "Required object is missing or null.", "Return the required V1 object.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void RequiredArray<T>(List<T>? value, string path, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
AddError(result, path, "Required array is missing or null.", "Return an empty array when no items are supported.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void RequiredObject<TValue>(Dictionary<string, TValue>? value, string path, ValidationResult result)
|
||||
{
|
||||
if (value is null)
|
||||
{
|
||||
AddError(result, path, "Required object is missing or null.", "Return an empty object when no metrics are supported.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void DuplicateNames(IEnumerable<(string? Name, string Path)> values, string collectionPath, ValidationResult result)
|
||||
{
|
||||
var seen = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var (name, path) in values)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(name))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var key = name.Trim();
|
||||
if (seen.TryGetValue(key, out var existingPath))
|
||||
{
|
||||
AddWarning(result, path, $"Duplicate entity name also appears at {existingPath}.", $"Avoid duplicate names in {collectionPath} unless they represent distinct supported entities.");
|
||||
}
|
||||
else
|
||||
{
|
||||
seen[key] = path;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static void Unknown(Dictionary<string, JsonElement>? values, string path, ValidationResult result)
|
||||
{
|
||||
if (values is null || values.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var propertyName in values.Keys.Order(StringComparer.Ordinal))
|
||||
{
|
||||
AddWarning(result, path == "$" ? propertyName : $"{path}.{propertyName}", "Unknown property is not part of Scene Intelligence Schema V1.", "Remove this property or update the schema version intentionally.");
|
||||
}
|
||||
}
|
||||
|
||||
private static void AddError(ValidationResult result, string path, string message, string suggestedFix)
|
||||
=> result.Errors.Add(new ValidationIssue { Severity = "Error", Path = path, Message = message, SuggestedFix = suggestedFix });
|
||||
|
||||
private static void AddWarning(ValidationResult result, string path, string message, string suggestedFix)
|
||||
=> result.Warnings.Add(new ValidationIssue { Severity = "Warning", Path = path, Message = message, SuggestedFix = suggestedFix });
|
||||
|
||||
private static void AddInfo(ValidationResult result, string path, string message, string suggestedFix)
|
||||
=> result.Information.Add(new ValidationIssue { Severity = "Information", Path = path, Message = message, SuggestedFix = suggestedFix });
|
||||
|
||||
private static HashSet<string> CreateSet(params string[] values)
|
||||
=> new(values, StringComparer.Ordinal);
|
||||
}
|
||||
@ -133,7 +133,12 @@ public sealed class StoryIntelligenceDryRunResultViewModel
|
||||
public string SceneContextJson { get; init; } = string.Empty;
|
||||
public string Duration { get; init; } = string.Empty;
|
||||
public int RetryCount { get; init; }
|
||||
public int? InputTokens { get; init; }
|
||||
public int? OutputTokens { get; init; }
|
||||
public string RawResponseText { get; init; } = string.Empty;
|
||||
public string SceneJsonText { get; init; } = string.Empty;
|
||||
public SceneIntelligenceScene? ParsedScene { get; init; }
|
||||
public PlotLine.Models.StoryIntelligence.ValidationResult? Validation { get; init; }
|
||||
public string? ErrorMessage { get; init; }
|
||||
}
|
||||
|
||||
|
||||
@ -101,12 +101,289 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="mt-3">
|
||||
<label class="form-label" for="sceneContextJson">Scene context JSON</label>
|
||||
<textarea class="form-control font-monospace" id="sceneContextJson" rows="8" readonly>@result.SceneContextJson</textarea>
|
||||
<dl class="mb-0">
|
||||
<dt>Input tokens</dt>
|
||||
<dd>@(result.InputTokens.HasValue ? result.InputTokens.Value.ToString("N0") : "Not reported")</dd>
|
||||
<dt>Output tokens</dt>
|
||||
<dd>@(result.OutputTokens.HasValue ? result.OutputTokens.Value.ToString("N0") : "Not reported")</dd>
|
||||
</dl>
|
||||
</div>
|
||||
<div class="mt-3">
|
||||
<label class="form-label" for="rawResponseText">Raw response text</label>
|
||||
<textarea class="form-control font-monospace" id="rawResponseText" rows="18" readonly>@result.RawResponseText</textarea>
|
||||
|
||||
<ul class="nav nav-tabs mt-4" id="storyIntelligenceTabs" role="tablist">
|
||||
<li class="nav-item" role="presentation">
|
||||
<button class="nav-link active" id="raw-response-tab" data-bs-toggle="tab" data-bs-target="#raw-response" type="button" role="tab" aria-controls="raw-response" aria-selected="true">Raw Response</button>
|
||||
</li>
|
||||
<li class="nav-item" role="presentation">
|
||||
<button class="nav-link" id="validation-tab" data-bs-toggle="tab" data-bs-target="#validation" type="button" role="tab" aria-controls="validation" aria-selected="false">Validation</button>
|
||||
</li>
|
||||
<li class="nav-item" role="presentation">
|
||||
<button class="nav-link" id="parsed-model-tab" data-bs-toggle="tab" data-bs-target="#parsed-model" type="button" role="tab" aria-controls="parsed-model" aria-selected="false">Parsed Model</button>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<div class="tab-content border border-top-0 rounded-bottom p-3" id="storyIntelligenceTabContent">
|
||||
<div class="tab-pane fade show active" id="raw-response" role="tabpanel" aria-labelledby="raw-response-tab" tabindex="0">
|
||||
<div class="mb-3">
|
||||
<label class="form-label" for="sceneContextJson">Scene context JSON</label>
|
||||
<textarea class="form-control font-monospace" id="sceneContextJson" rows="8" readonly>@result.SceneContextJson</textarea>
|
||||
</div>
|
||||
@if (!string.IsNullOrWhiteSpace(result.SceneJsonText))
|
||||
{
|
||||
<div class="mb-3">
|
||||
<label class="form-label" for="sceneJsonText">Extracted Scene Intelligence JSON</label>
|
||||
<textarea class="form-control font-monospace" id="sceneJsonText" rows="12" readonly>@result.SceneJsonText</textarea>
|
||||
</div>
|
||||
}
|
||||
<div>
|
||||
<label class="form-label" for="rawResponseText">Raw OpenAI response</label>
|
||||
<textarea class="form-control font-monospace" id="rawResponseText" rows="18" readonly>@result.RawResponseText</textarea>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="tab-pane fade" id="validation" role="tabpanel" aria-labelledby="validation-tab" tabindex="0">
|
||||
@if (result.Validation is null)
|
||||
{
|
||||
<div class="alert alert-warning">Validation did not run.</div>
|
||||
}
|
||||
else
|
||||
{
|
||||
<h3>@ValidationSummary(result.Validation)</h3>
|
||||
@RenderIssues("Errors", result.Validation.Errors)
|
||||
@RenderIssues("Warnings", result.Validation.Warnings)
|
||||
@RenderIssues("Information", result.Validation.Information)
|
||||
}
|
||||
</div>
|
||||
|
||||
<div class="tab-pane fade" id="parsed-model" role="tabpanel" aria-labelledby="parsed-model-tab" tabindex="0">
|
||||
@if (result.ParsedScene is null)
|
||||
{
|
||||
<div class="alert alert-warning">No parsed Scene Intelligence model is available.</div>
|
||||
}
|
||||
else
|
||||
{
|
||||
var scene = result.ParsedScene;
|
||||
<div class="row g-3">
|
||||
<div class="col-lg-6">
|
||||
<section class="border rounded p-3 h-100">
|
||||
<h3>Scene Reference</h3>
|
||||
<dl class="mb-0">
|
||||
<dt>Schema</dt>
|
||||
<dd>@scene.SchemaVersion</dd>
|
||||
<dt>Project</dt>
|
||||
<dd>@Display(scene.SceneReference?.ProjectId)</dd>
|
||||
<dt>Book</dt>
|
||||
<dd>@Display(scene.SceneReference?.BookId)</dd>
|
||||
<dt>Chapter</dt>
|
||||
<dd>@Display(scene.SceneReference?.ChapterId) (@Display(scene.SceneReference?.ChapterNumber))</dd>
|
||||
<dt>Scene</dt>
|
||||
<dd>@Display(scene.SceneReference?.SceneId) (@Display(scene.SceneReference?.SceneNumber))</dd>
|
||||
<dt>Source</dt>
|
||||
<dd>@Display(scene.SceneReference?.SourceLabel)</dd>
|
||||
</dl>
|
||||
</section>
|
||||
</div>
|
||||
<div class="col-lg-6">
|
||||
<section class="border rounded p-3 h-100">
|
||||
<h3>Summary</h3>
|
||||
<p><strong>Short:</strong> @Display(scene.Summary?.Short)</p>
|
||||
<p><strong>Detailed:</strong> @Display(scene.Summary?.Detailed)</p>
|
||||
<p class="mb-0"><strong>Confidence:</strong> @Display(scene.Summary?.Confidence)</p>
|
||||
</section>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="row g-3 mt-1">
|
||||
<div class="col-lg-4">
|
||||
<section class="border rounded p-3 h-100">
|
||||
<h3>Purpose</h3>
|
||||
<p>@Display(scene.ScenePurpose?.ObservedFunction)</p>
|
||||
<p class="mb-0"><strong>Confidence:</strong> @Display(scene.ScenePurpose?.Confidence)</p>
|
||||
</section>
|
||||
</div>
|
||||
<div class="col-lg-4">
|
||||
<section class="border rounded p-3 h-100">
|
||||
<h3>Point Of View</h3>
|
||||
<dl class="mb-0">
|
||||
<dt>Character</dt>
|
||||
<dd>@Display(scene.PointOfView?.CharacterName)</dd>
|
||||
<dt>Mode</dt>
|
||||
<dd>@Display(scene.PointOfView?.NarrativeMode)</dd>
|
||||
<dt>Evidence</dt>
|
||||
<dd>@Display(scene.PointOfView?.Evidence)</dd>
|
||||
<dt>Confidence</dt>
|
||||
<dd>@Display(scene.PointOfView?.Confidence)</dd>
|
||||
</dl>
|
||||
</section>
|
||||
</div>
|
||||
<div class="col-lg-4">
|
||||
<section class="border rounded p-3 h-100">
|
||||
<h3>Setting</h3>
|
||||
<dl class="mb-0">
|
||||
<dt>Location</dt>
|
||||
<dd>@Display(scene.Setting?.LocationName)</dd>
|
||||
<dt>Type</dt>
|
||||
<dd>@Display(scene.Setting?.LocationType)</dd>
|
||||
<dt>Room</dt>
|
||||
<dd>@Display(scene.Setting?.GenericRoomType)</dd>
|
||||
<dt>Time</dt>
|
||||
<dd>@Display(scene.Setting?.TimeOfDay) / @Display(scene.Setting?.DateOrTimeReference)</dd>
|
||||
<dt>Confidence</dt>
|
||||
<dd>@Display(scene.Setting?.Confidence)</dd>
|
||||
</dl>
|
||||
</section>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@RenderCollection("Characters", scene.Characters, character =>
|
||||
$"{Display(character.Name)} - {Display(character.RoleInScene)} - mentioned only: {Display(character.MentionedOnly)} - confidence: {Display(character.Confidence)}")
|
||||
@RenderCollection("Locations", scene.Locations, location =>
|
||||
$"{Display(location.Name)} - {Display(location.LocationType)} - present: {Display(location.PresentInScene)} - confidence: {Display(location.Confidence)}")
|
||||
@RenderCollection("Assets", scene.Assets, asset =>
|
||||
$"{Display(asset.Name)} - {Display(asset.AssetType)} - status: {Display(asset.Status)} - holder: {Display(asset.OwnerOrHolder)}")
|
||||
@RenderCollection("Relationships", scene.Relationships, relationship =>
|
||||
$"{Display(relationship.CharacterA)} / {Display(relationship.CharacterB)} - {Display(relationship.RelationshipSignal)} - confidence: {Display(relationship.Confidence)}")
|
||||
@RenderCollection("Knowledge Changes", scene.KnowledgeChanges, change =>
|
||||
$"{Display(change.RecipientCharacter)} {Display(change.ChangeType)}: {Display(change.KnowledgeItem)}")
|
||||
@RenderCollection("Timeline Clues", scene.TimelineClues, clue =>
|
||||
$"{Display(clue.Clue)} - {Display(clue.RelativeOrder)}")
|
||||
@RenderCollection("Questions Raised", scene.QuestionsRaised, question =>
|
||||
$"{Display(question.Question)} - {Display(question.Scope)}")
|
||||
@RenderCollection("Questions Answered", scene.QuestionsAnswered, answer =>
|
||||
$"{Display(answer.Question)} => {Display(answer.Answer)}")
|
||||
@RenderCollection("Observations", scene.Observations, observation =>
|
||||
$"{Display(observation.SubjectName)} {Display(observation.Predicate)} {Display(observation.ObjectName)} - {Display(observation.ObservationType)}")
|
||||
|
||||
<section class="border rounded p-3 mt-3">
|
||||
<h3>Metrics</h3>
|
||||
@if (scene.Metrics is null || scene.Metrics.Count == 0)
|
||||
{
|
||||
<p class="mb-0">No metrics reported.</p>
|
||||
}
|
||||
else
|
||||
{
|
||||
<dl class="mb-0">
|
||||
@foreach (var metric in scene.Metrics)
|
||||
{
|
||||
<dt>@metric.Key</dt>
|
||||
<dd>Score @Display(metric.Value?.Score), confidence @Display(metric.Value?.Confidence)</dd>
|
||||
}
|
||||
</dl>
|
||||
}
|
||||
</section>
|
||||
|
||||
<section class="border rounded p-3 mt-3">
|
||||
<h3>Source Limits</h3>
|
||||
<p><strong>Contains ambiguity:</strong> @Display(scene.SourceLimits?.ContainsAmbiguity)</p>
|
||||
@RenderStringList("Ambiguity Notes", scene.SourceLimits?.AmbiguityNotes)
|
||||
</section>
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
}
|
||||
|
||||
@functions {
|
||||
private static string Display(object? value) => value switch
|
||||
{
|
||||
null => "None",
|
||||
string text when string.IsNullOrWhiteSpace(text) => "None",
|
||||
decimal number => number.ToString("0.##"),
|
||||
bool flag => flag ? "Yes" : "No",
|
||||
_ => value.ToString() ?? "None"
|
||||
};
|
||||
|
||||
private static string ValidationSummary(PlotLine.Models.StoryIntelligence.ValidationResult validation)
|
||||
=> validation.IsValid
|
||||
? $"✓ Passed with {validation.Warnings.Count:N0} warning(s)"
|
||||
: $"✖ Failed with {validation.Errors.Count:N0} error(s) and {validation.Warnings.Count:N0} warning(s)";
|
||||
|
||||
private static string SeveritySymbol(string severity) => severity switch
|
||||
{
|
||||
"Error" => "✖",
|
||||
"Warning" => "⚠",
|
||||
_ => "✓"
|
||||
};
|
||||
|
||||
private static Microsoft.AspNetCore.Html.IHtmlContent RenderIssues(string heading, IReadOnlyList<PlotLine.Models.StoryIntelligence.ValidationIssue> issues)
|
||||
{
|
||||
var builder = new System.Text.StringBuilder();
|
||||
builder.Append("<section class=\"mt-3\">");
|
||||
builder.Append("<h4>").Append(Encode(heading)).Append(" (").Append(issues.Count.ToString("N0")).Append(")</h4>");
|
||||
if (issues.Count == 0)
|
||||
{
|
||||
builder.Append("<p class=\"mb-0\">None.</p>");
|
||||
}
|
||||
else
|
||||
{
|
||||
builder.Append("<div class=\"list-group\">");
|
||||
foreach (var issue in issues)
|
||||
{
|
||||
builder.Append("<div class=\"list-group-item\">");
|
||||
builder.Append("<p class=\"mb-1\"><strong>")
|
||||
.Append(Encode(SeveritySymbol(issue.Severity)))
|
||||
.Append(' ')
|
||||
.Append(Encode(issue.Severity))
|
||||
.Append("</strong> <code>")
|
||||
.Append(Encode(issue.Path))
|
||||
.Append("</code></p>");
|
||||
builder.Append("<p class=\"mb-1\">").Append(Encode(issue.Message)).Append("</p>");
|
||||
builder.Append("<p class=\"mb-0 muted\">").Append(Encode(issue.SuggestedFix)).Append("</p>");
|
||||
builder.Append("</div>");
|
||||
}
|
||||
|
||||
builder.Append("</div>");
|
||||
}
|
||||
|
||||
builder.Append("</section>");
|
||||
return new Microsoft.AspNetCore.Html.HtmlString(builder.ToString());
|
||||
}
|
||||
|
||||
private static Microsoft.AspNetCore.Html.IHtmlContent RenderCollection<T>(string heading, IReadOnlyList<T>? items, Func<T, string> formatter)
|
||||
{
|
||||
var builder = new System.Text.StringBuilder();
|
||||
builder.Append("<section class=\"border rounded p-3 mt-3\">");
|
||||
builder.Append("<h3>").Append(Encode(heading)).Append("</h3>");
|
||||
if (items is null || items.Count == 0)
|
||||
{
|
||||
builder.Append("<p class=\"mb-0\">No items reported.</p>");
|
||||
}
|
||||
else
|
||||
{
|
||||
builder.Append("<ol class=\"mb-0\">");
|
||||
foreach (var item in items)
|
||||
{
|
||||
builder.Append("<li>").Append(Encode(formatter(item))).Append("</li>");
|
||||
}
|
||||
|
||||
builder.Append("</ol>");
|
||||
}
|
||||
|
||||
builder.Append("</section>");
|
||||
return new Microsoft.AspNetCore.Html.HtmlString(builder.ToString());
|
||||
}
|
||||
|
||||
private static Microsoft.AspNetCore.Html.IHtmlContent RenderStringList(string heading, IReadOnlyList<string>? items)
|
||||
{
|
||||
var builder = new System.Text.StringBuilder();
|
||||
builder.Append("<h4>").Append(Encode(heading)).Append("</h4>");
|
||||
if (items is null || items.Count == 0)
|
||||
{
|
||||
builder.Append("<p class=\"mb-0\">No items reported.</p>");
|
||||
}
|
||||
else
|
||||
{
|
||||
builder.Append("<ul class=\"mb-0\">");
|
||||
foreach (var item in items)
|
||||
{
|
||||
builder.Append("<li>").Append(Encode(item)).Append("</li>");
|
||||
}
|
||||
|
||||
builder.Append("</ul>");
|
||||
}
|
||||
|
||||
return new Microsoft.AspNetCore.Html.HtmlString(builder.ToString());
|
||||
}
|
||||
|
||||
private static string Encode(string? value)
|
||||
=> System.Net.WebUtility.HtmlEncode(value ?? string.Empty);
|
||||
}
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user