What goes wrong in the books StoryBuddy draws, how often, what was tried to fix it, and what the study must state as a limit. About half of the reviewed pages have at least one major defect: 77 of 150. Every number names the file it comes from.
Read these numbers as provisional
The 150 pages and 55 reference pictures used AI-assisted visual assessment on 2026-10-08. The owner has not confirmed the scores, so every audit number here is provisional. Only synthetic stories were reviewed. No child's story is shown or counted on this page.
Pages tagged as containing non-human characters have more defects than human-only pages in this audit, including mixed human and non-human casts. Reference counts classify the reference pictures themselves; page counts use the audit's tags rather than the fine-tuning character slices.
Pages with a major defect
Reference pictures with a major defect
Pages with a duplicate or extra character
The audit tags 117 of 150 shipped pages as containing non-human characters; the other 33 are human-only. Donated stories are mostly about humans, so the overall rate likely overstates what a classroom story gets. The human-only group is too small to call a rate, and no donated story was reviewed.
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.3
Source: docs/capstone/pipeline-stage-audit-2026-10.md (opens on GitHub in a new tab), “Provisional results”, Shipped pages and References
Of the 150 reviewed pages, how many carry each defect. A page can carry more than one.
| Defect | Pages | What it looks like |
|---|---|---|
| Reference inherited | 34 | A mistake in the reference picture (for example six legs drawn as four) repeats on every page |
| Duplicate character | 15 | The same character appears twice on one page |
| Missing or wrong action | 9 | The character stands there instead of doing what the text says |
| Identity drift | 8 | The character's colours, hair or shape change between pages |
| Extra character | 5 | Someone who is not in the story appears |
Source: docs/capstone/pipeline-stage-audit-2026-10.md (opens on GitHub in a new tab), “Provisional results”, Where the 77 come from
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.3, for what each defect looks like
Hand-picked synthetic examples illustrate these failures; they do not establish their frequency or current performance. They are training-split captures from the provisional October 8, 2026 audit, made with generation prompt version 4. They are illustrations, not new independent human labels.
Paper Cutout · synthetic · training split
Requested
“Lala looks up at the sky.”
Visible in the image
Only Lala is in the named scene cast, but the page draws two copies of her side by side.
syn-021 · s6 · selected final page · attempt 1
Gouache · synthetic · training split
Requested
“It had six legs and its wool was green moss and little white flowers grew in it.”
Visible in the image
The moss and flowers are present, but only four legs are depicted. A scene matching this reference would not establish that the requested body count was correct.
syn-002 · c1 (Mopsi) · final canonical reference · draw 1
Cartoon · synthetic · training split
Requested
“He practised breathing fire every night and what came out was air, because he was a sock.”
Visible in the image
The page depicts flames where the story explicitly says only air comes out. Other body and cast defects are visible too; this example does not isolate their cause.
syn-019 · s1 · selected final page · attempt 3
Cartoon · synthetic · training split
Requested
“He called himself General Lint and he commanded an army of six smaller dust bunnies.”
Visible in the image
The scene direction requests Lint and his soldiers crouching under the vacuum. Only Lint and the vacuum have references; the page invents helmeted humanoid soldiers instead of smaller dust bunnies.
syn-016 · s2 · selected final page · attempt 4
See reference/page identity-drift examples
Source: docs/specs/research-limitations-examples.md (opens on GitHub in a new tab), Selected examples, requirements and source provenance
Source: docs/capstone/pipeline-stage-audit-2026-10.md (opens on GitHub in a new tab), Provisional review and its limitations
The pipeline checks each reference picture and each page with an AI judge before it ships them. This is what those stored checks had said about the images the reviewers scored.
Of the 17 defective references
Of the 38 acceptable references
The check finds most bad references. The pipeline then ships the best of the failing draws, so every defect it found still reached the pages.
Of the 77 defective pages
Of the 71 acceptable pages
The page check compares a page with its reference picture. A mistake copied from the reference matches the reference, so it passes.
Source: docs/capstone/pipeline-stage-audit-2026-10.md (opens on GitHub in a new tab), “Provisional results”, References and Page gate
The scene editor is the image model that draws each page from the reference pictures. StoryBuddy ships Qwen-Image-Edit. Two other editors were set against it, and neither replaced it.
30 random pages, drawn by both editors and scored with the editor's name hidden. klein's style may be recognisable, so the blinding is partial.
Pages with a duplicate character
Pages where the character lost its identity
klein drew as many duplicates and lost the character more often. It was rejected, and Qwen-Image-Edit stays (ADR-063).
One invented character, a three-eyed lizard-bird. How often it stayed recognisable on a page, with its reference picture and without it.
Qwen-Image-Edit
OmniGen2 (tried as a backup)
This was a build check, scored without blinding, and it is not a finding of the study. Both runs failed the gate set in advance. Qwen-Image-Edit stayed the editor, and the failed gate is carried as a limitation.
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.4 for klein and §5.1 for OmniGen2
Source: docs/capstone/model-probes-2026-10.md (opens on GitHub in a new tab), “Blinded klein screen”
Source: docs/product/PHASE_05_RESULTS.md (opens on GitHub in a new tab), Probe 1
The October tests the group guide lists, one row each. Tests that failed are listed with the rest.
| Tried | Result | Outcome |
|---|---|---|
| A different scene editor (FLUX.2 klein) | On 30 random pages it drew as many duplicates as Qwen (4 each) and lost character identity more often (10 against 3). Rejected (ADR-063). | Rejected |
| A prompt line asking for "one moment" | 1 of 10 draws free of duplicates. Not adopted. | Not adopted |
| Rewriting the direction as one frozen instant | 3 of 10 free of duplicates. Not adopted. It did fix both pages where the text described something absent or pretend. | Not adopted |
| Shipping whichever attempt the judge did not flag for duplicates | Helped one scene and hurt another. Not adopted. | Not adopted |
| A different text model | No open-weight model beat Mistral. | No change |
| Adding a character the direction names by species ("the moth") | Fixed the one story it targets. Merged on 2026-10-09 (PR #101). | Merged |
| Other reference-picture models for stated body counts | None drew counts like "six legs" correctly. Already a stated limitation (ADR-056). | Stated limitation |
| Editing the finished reference to fix the count (October 11) | 5 of 9 fixed with the character unchanged, 1 borderline, 3 missed. Pages redrawn from the fixed references then showed the right count on 5 of 6 (1 borderline). One try each, with instructions written by hand. Not adopted yet: it needs a test with automatic instructions and an ADR. | A lead, not adopted |
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.4
Source: docs/capstone/model-probes-2026-10.md (opens on GitHub in a new tab), “Findings and changes”, for each test's plan and result
What the study says out loud as a weakness, in the group guide's own words.
About 13% of page images show a character twice. Two different scene editors do this at the same rate, and four fixes failed. It is a limitation of this kind of edit model, measured on two of them, and it remains open.
"Six legs" or "seven stripes" usually come out wrong. No tested model draws them from text (ADR-056). Editing the finished reference fixed 5 of 9 in one small test, and pages redrawn from those references kept the fix. It is a lead and not yet a fix: one try each, with instructions written by hand.
Background groups like "his six dust-bunny soldiers" have no reference picture. The model guesses what they look like, and once drew them as human boys.
Some pages miss the action, or give it to the wrong character. Describing only what is visible helped where the text mentioned something absent or pretend. That kind is about 3 pages in 150, too few to measure a fix before the defense.
Most October probes used pages chosen because they had failed, with one or two draws each. Only the klein screen was blinded and used random pages.
Objective 4's result is partial, and the trained judge scored lower F1 than the untrained one.
Two groupmates labelled every pair and the owner settled disagreements. Pairs both raters got wrong were never re-checked, apart from ten audited ones. Hair length is not a rule, so raters could split on it. Object characters (such as a tin can) lean toward "different" because their pages add arms and legs their references never had.
One grade band, one school, one country. The base model changed after registration, and AI assistance in labelling is disclosed. Both are reported as limitations.
The "about half the pages" figure is from invented stories, with 117 of 150 pages tagged as containing non-human characters, including mixed casts. Human-only pages did better (11 of 33 against 66 of 117), but 33 pages is too few to call a rate, and no donated story was reviewed.
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §6
Source: docs/capstone/model-probes-2026-10.md (opens on GitHub in a new tab), “Limitations to carry into the write-up”
