101 lines
4.5 KiB
Markdown
101 lines
4.5 KiB
Markdown
# Phase 20AF - Scene Analysis Reliability and Import Experience Reality Check
|
||
|
||
## Summary
|
||
|
||
This pass found that scene failures were mostly not analysis-quality failures. The saved run data shows scene intelligence often returned useful content, but the import preparation path rejected it because of parse audit wrappers, strict validation counts, or malformed/incomplete JSON in optional sections.
|
||
|
||
The review/import path now prepares scenes from the same import objects the commit service consumes. Optional scene intelligence problems no longer automatically prevent a scene from being created.
|
||
|
||
## Failure Diagnosis
|
||
|
||
Local completed onboarding runs using `Chapter=gpt-5-mini; Scene=gpt-5-mini` show these patterns:
|
||
|
||
- Some scene responses parsed after JSON repair or retry and were saved inside a `parsedScene` audit wrapper.
|
||
- The importer only attempted to read direct `SceneIntelligenceScene` JSON and skipped the wrapped repaired result.
|
||
- The importer filtered out any scene with `ValidationErrorsCount > 0`, even when the scene had enough information to create a PlotDirector scene.
|
||
- Fully failed parse cases were usually malformed or incomplete JSON:
|
||
- `No complete JSON object was found.`
|
||
- invalid numeric confidence such as a decimal followed by whitespace.
|
||
- retry attempted once but still returned incomplete JSON.
|
||
|
||
The failures were therefore caused by:
|
||
|
||
- malformed or incomplete OpenAI JSON from `gpt-5-mini`;
|
||
- importer/parser logic that was too strict for repaired/audited results;
|
||
- validation errors being treated as fatal even when minimum scene import data existed.
|
||
|
||
## Tolerance Changes Made
|
||
|
||
Scene import preparation now:
|
||
|
||
- reads both direct parsed scene JSON and repaired/retry audit wrappers containing `parsedScene`;
|
||
- no longer skips a scene solely because `ValidationErrorsCount > 0`;
|
||
- creates a limited-detail fallback scene when paragraph range is valid but optional analysis cannot be read;
|
||
- uses generated fallback title/summary when needed;
|
||
- skips optional metrics/POV/setting details when they cannot be read;
|
||
- keeps developer diagnostics available through saved run details.
|
||
|
||
Minimum scene creation now depends on:
|
||
|
||
- scene number;
|
||
- paragraph range;
|
||
- source run/chapter reference;
|
||
- usable summary or generated fallback summary.
|
||
|
||
## User Experience Changes
|
||
|
||
The progress page now has an explicit `Continue in background` action and explains:
|
||
|
||
> PlotDirector will keep analysing your manuscript. You can close Word, close this browser tab, and return later. We’ll show this import on your Projects page while it is running.
|
||
|
||
The Projects page now shows a compact active import card while an onboarding Story Intelligence batch is active:
|
||
|
||
- book title;
|
||
- chapter progress label;
|
||
- `View progress` button.
|
||
|
||
This gives users a route back after closing the browser tab.
|
||
|
||
## Developer Diagnostics
|
||
|
||
Failed or limited scenes remain auditable through Admin run details. For each scene result, the system stores:
|
||
|
||
- raw OpenAI response;
|
||
- assistant output;
|
||
- parsed or repaired JSON audit;
|
||
- retry flags and retry errors when available;
|
||
- validation error/warning counts;
|
||
- paragraph range and scene number.
|
||
|
||
The onboarding review screen keeps author language, while admin diagnostics link to the technical run detail.
|
||
|
||
## GPT-5 Comparison
|
||
|
||
No local completed onboarding run using `Scene=gpt-5` was present in the database at the time of this pass.
|
||
|
||
Local aggregate for completed onboarding runs:
|
||
|
||
| Model | Runs | Detected scenes | Completed scenes | Failed scenes |
|
||
|---|---:|---:|---:|---:|
|
||
| Chapter=gpt-5-mini; Scene=gpt-5-mini | 17 | 94 | 84 | 13 |
|
||
|
||
A real comparison still needs to be run with:
|
||
|
||
- Chapter Structure: `gpt-5-mini`
|
||
- Scene Intelligence: `gpt-5`
|
||
|
||
against the same source chapters.
|
||
|
||
## Expected Success Rate Change
|
||
|
||
Before this pass, saved mini-model runs could show analysed scenes as unavailable for import because repaired/audited JSON or validation-warning scenes were excluded.
|
||
|
||
After this pass, those scenes are prepared if they have valid paragraph ranges. In the inspected local failed cases, the failed scene records had valid ranges, so they should now be created as either normal or limited-detail scenes rather than being blocked solely by optional analysis defects.
|
||
|
||
## Remaining Risks
|
||
|
||
- A response with no usable paragraph range still cannot be safely imported.
|
||
- The active Projects card currently uses the in-memory onboarding batch, so it covers closing the browser tab but not an application restart.
|
||
- GPT-5 comparison has not yet been executed locally.
|
||
- Fallback scenes may need author review because optional POV, setting, metrics or timeline detail may be missing.
|