Judge results
Training the judge did not make it better at catching a different character on this run. The untrained model has the higher F1: 0.472, against 0.310 for the trained judge picked in advance. This is a partial result from 2026-10-08. Every number names the file it comes from.
What was tested
A pair is one reference picture of a character and one book page. A judge answers Same when the page shows that character, and Different when it does not.
The test set has 329 pairs: 47 Different and 282 Same. They come from children's donated stories. No story or character in the test set is in the training data, so each judge is tested on characters it never saw.
Four judges were scored on these pairs. One is Qwen3.5-9B without our training. The other three are separate training runs of that model, called seed 0, seed 1 and seed 2. Seed 1 was picked in advance, on validation data, as the one to compare with the untrained model. No seed was picked again after the test results were seen.
- Precision.
- Of the pairs the judge called Different, how many really were.
- Recall.
- Of the pairs that really were Different, how many the judge caught.
- F1.
- One number that balances precision and recall. 1.0 is perfect.
F1 on the Different class is the registered primary metric. Registered means it was written down before the results were seen.
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Exact values of the partial comparison (copied 2026-10-11)” and “What the registered freeze contains”
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §8, for the three definitions
Source: docs/product/PREREGISTRATION_OBJ4.md (opens on GitHub in a new tab), §5, for F1 as the primary metric
How the judge was trained
The trained judges start from the same Qwen3.5-9B model as the untrained one. The method is QLoRA: the model's own weights stay frozen, and a small add-on called an adapter is trained on top. Each trained judge is the model plus one adapter.
The adapter learned from the 484 training pairs, all from synthetic stories. One pass over them is an epoch. Training ran for three epochs: 61 updates in each, 183 in a run.
| Setting | Value | In plain words |
|---|---|---|
| model_name_or_path | Qwen/Qwen3.5-9B | The model that is trained. The untrained judge is this same model. |
| finetuning_type | lora | The model's own weights stay frozen. A small add-on, the adapter, is trained. |
| quantization_bit | 4 | The frozen model is loaded in a compressed 4-bit form. This is the Q in QLoRA. |
| lora_rank | 16 | The size of the adapter. |
| lora_alpha | 32 | How strongly the adapter's output is scaled. |
| image_max_pixels | 262144 | Each picture is shrunk to at most 512 × 512 pixels. |
| num_train_epochs | 3.0 | The model sees every training pair three times. |
| per_device_train_batch_size | 1 | One pair is read at a time. |
| gradient_accumulation_steps | 8 | The adapter is updated once for every 8 pairs. |
| learning_rate | 1.0e-4 | The largest size of one update. |
| lr_scheduler_type | cosine | The update size falls smoothly toward zero by the end. |
| warmup_ratio | 0.1 | The update size rises from zero over the first tenth of training. |
| save_steps | 10 | A checkpoint, a saved copy of the adapter, is kept every 10 updates. |
Why three seeds
A seed fixes the random choices in a training run, such as the order the pairs are shown in. The same data and settings with another seed give a slightly different judge. The plan registered in advance says one run on a dataset this small is noise. So it requires three runs, with seeds 0, 1 and 2, and all three reported, including a bad one. Picking the best seed on the test set is forbidden.
Which saved copy was used
For each seed, the checkpoint with the best F1 on the validation pairs was used: checkpoint 40 for seed 0, checkpoint 40 for seed 1 and checkpoint 140 for seed 2. Checkpoint 40 is 40 of the 183 updates, before the first epoch ends at update 61. Seed 1 was then picked as the one to compare, also on validation data.
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Registered training settings (copied 2026-10-11)” and “Exact values of the partial comparison (copied 2026-10-11)”
Source: backend/finetune/train_qlora.yaml (opens on GitHub in a new tab), the settings file itself
Source: docs/specs/judge-finetune.md (opens on GitHub in a new tab), §6, for how a run is started
Source: docs/product/PREREGISTRATION_OBJ4.md (opens on GitHub in a new tab), §9.5, for the seeds and the choice of checkpoint
F1, precision and recall
- The untrained model has a higher F1 than each of the three trained judges.
- Seed 1 is more careful. When it says Different it is usually right, which is high precision. It misses most of the real differences, which is low recall.
- Seed 0 shows the same pattern. Seed 2 does not: its precision and its recall are both below the untrained model's.
F1, precision and recall of each judge on the 329 test pairs. Every bar runs from 0 to 1. The interval under each judge is the 95% interval of its F1, from 10,000 resamples clustered by character. Every value is also in the table below.
F1, the primary metricPrecision and recall
Untrained Qwen3.5-9B
- F1 0.472
- Precision 0.382
- Recall 0.617
F1 95% interval: 0.236 to 0.629
Trained, seed 1 (picked in advance)
- F1 0.310
- Precision 0.818
- Recall 0.191
F1 95% interval: 0.000 to 0.491
Trained, seed 0
- F1 0.361
- Precision 0.786
- Recall 0.234
F1 95% interval: 0.000 to 0.586
Trained, seed 2
- F1 0.417
- Precision 0.353
- Recall 0.511
F1 95% interval: 0.214 to 0.566
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Exact values of the partial comparison (copied 2026-10-11)”
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.2, for the reading in plain words
What each judge did with the pairs
A caught pair is a Different pair the judge called Different. A missed pair is a Different pair it did not catch. A false alarm is a Same pair it called Different.
Each judge's answers on the 47 Different pairs and on the 282 Same pairs. The two bars count different numbers of pairs, and each is drawn to its own full width. Every value is also in the table below.
Right answerWrong answer
Untrained Qwen3.5-9B
The 47 Different pairs
Caught 29Missed 18
The 282 Same pairs
False alarms 47Correct 235
Trained, seed 1 (picked in advance)
The 47 Different pairs
Caught 9Missed 38
The 282 Same pairs
False alarms 2Correct 280
Trained, seed 0
The 47 Different pairs
Caught 11Missed 36
The 282 Same pairs
False alarms 3Correct 279
Trained, seed 2
The 47 Different pairs
Caught 24Missed 23
The 282 Same pairs
False alarms 44Correct 238
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Exact values of the partial comparison (copied 2026-10-11)”
Every value in the charts
| Judge | F1 | F1 95% interval | Precision | Recall | Caught (of 47) | Missed (of 47) | False alarms (of 282) | Correct Same (of 282) | Unreadable (of 329) |
|---|---|---|---|---|---|---|---|---|---|
| Untrained Qwen3.5-9B | 0.472 | 0.236 to 0.629 | 0.382 | 0.617 | 29 | 18 | 47 | 235 | 3 |
| Trained, seed 1 (picked in advance) | 0.310 | 0.000 to 0.491 | 0.818 | 0.191 | 9 | 38 | 2 | 280 | 40 |
| Trained, seed 0 | 0.361 | 0.000 to 0.586 | 0.786 | 0.234 | 11 | 36 | 3 | 279 | 44 |
| Trained, seed 2 | 0.417 | 0.214 to 0.566 | 0.353 | 0.511 | 24 | 23 | 44 | 238 | 8 |
The three trained seeds have a mean F1 of 0.363, with a sample standard deviation of 0.054.
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Exact values of the partial comparison (copied 2026-10-11)”
Unreadable answers
An unreadable answer is output that did not parse. It is scored as Same, so on a Different pair it counts as a miss.
- Untrained Qwen3.5-9B
- 3 of 329
- Trained, seed 1 (picked in advance)
- 40 of 329
- Trained, seed 0
- 44 of 329
- Trained, seed 2
- 8 of 329
Seeds 0 and 1 gave unreadable answers on about one pair in eight, which is part of why their recall is low.
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.2
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Exact values of the partial comparison (copied 2026-10-11)”, for the counts
Seed 1 against the untrained model
This is the registered comparison: the F1 of seed 1 minus the F1 of the untrained model.
- Gap in F1 (ΔF1)
- −0.161
- 95% interval of the gap
- −0.358 to +0.003
- Exact McNemar p
- 0.0035
- The gap is below zero because seed 1 has the lower F1.
- The interval of the gap includes zero. It just touches it.
- These results do not show that training improved the judge.
What the McNemar p does and does not show
This paragraph is arithmetic on the table above. The result file does not store it.
The untrained model called 76 pairs Different and was wrong on 65 pairs: 47 false alarms and 18 misses. Seed 1 called 11 pairs Different and was wrong on 40: 2 false alarms and 38 misses. McNemar's test compares which pairs each judge got right, so its small p goes with seed 1 being wrong on 25 fewer pairs. It does not say seed 1 is the better detector. F1 is lower for seed 1 because it misses 38 of the 47 Different pairs. A judge that answered Same every time would be right on 282 of 329 pairs and catch none.
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Exact values of the partial comparison (copied 2026-10-11)” and “Partial untuned versus fine-tuned comparison (2026-10-08)”
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.2
Source: docs/product/PREREGISTRATION_OBJ4.md (opens on GitHub in a new tab), §1.2, for the comparison
With the ambiguous reference dropped
This check was registered in advance. It drops one test character whose reference picture was ambiguous (20 pairs). That leaves 309 pairs, 41 of them Different.
| Judge | F1 | Precision | Recall |
|---|---|---|---|
| Untrained Qwen3.5-9B | 0.476 | 0.391 | 0.610 |
| Trained, seed 1 (picked in advance) | 0.346 | 0.818 | 0.220 |
| Trained, seed 0 | 0.400 | 0.786 | 0.268 |
| Trained, seed 2 | 0.438 | 0.382 | 0.512 |
Seed 1 minus the untrained model: ΔF1 −0.130, 95% interval −0.333 to +0.053, exact McNemar p = 0.0075. The direction matches the primary analysis. The primary analysis, on every pair, stands. This check never replaces it.
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Registered agreement and sensitivity analyses (2026-10-09)”
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.2
What we take from this
These are readings of the numbers above and of the plan registered in advance. Each rests on a document named under the list.
- The objective is to evaluate the trained judge, not to show it wins. Its registered result is the trained judge's precision, recall and F1 against human labels, and the plan says that result does not depend on any comparison. A low F1 is still that result. It is partial because the prompted Gemma baseline is missing.
- The trained judge is not used in the product. The plan ships a trained judge only if it beats the untrained model. StoryBuddy keeps its prompted Gemma judge. No formal decision under that rule was recorded.
- The main work moved to the page-drawing pipeline. Paid evaluation stopped on 2026-10-07 for budget, and the owner made the pipeline the next workstream that day. This result came on 2026-10-08, so it did not cause the move.
- Training changed the judge's answers. It did not improve them. Seed 1 called 11 pairs Different where the untrained model called 76, and its unreadable answers went from 3 to 40. The plan describes a trained judge that does not beat its base as “The LoRA did nothing”. Here the adapter did change the output.
Why it happened is not known
No cause was tested. These four explanations fit the numbers, and none has been checked.
- Checkpoints chosen too early. Seeds 0 and 1 both use checkpoint 40, from before the first epoch ended. They are the two seeds with 40 and 44 unreadable answers. Seed 2 uses checkpoint 140 and has 8. The choice rested on 4 Different validation pairs, and three runs are too few to call this a pattern.
- One sentence for every constructed pair. 104 of the 157 Different training pairs are constructed, and all of them carry the same reason and the same sentence, written by a script. The model may have learned the sentence instead of the difference.
- Different kinds of stories. Training used synthetic stories: 264 of 484 training pairs (54.5%) are non-human, including constructed pairs. The test stories are children's, mostly about humans.
- Answers cut off. A judge's answer is capped at 256 tokens, and the judge writes its reasons before its verdict. Whether the unreadable answers were cut off was not checked.
The plan says what may come next. Debugging uses training and validation data only. One more read of the test set is allowed, and only after a defect is found and fixed. Trying new settings does not count as a defect. No second read has been made.
Source: docs/product/PREREGISTRATION_OBJ4.md (opens on GitHub in a new tab), §1.1 for the objective, §6 for the rule on shipping, §7 for what may come next
Source: docs/capstone/research-changelog.md (opens on GitHub in a new tab), §1 items 3, 4 and 9, §2 for the move to the pipeline, and the 2026-10-11 rows for the four explanations
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Registered training settings (copied 2026-10-11)”, “What the registered freeze contains” and “Held-out resume context”
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §5.3, for the kinds of stories
What this does and does not show
- The result is partial. The prompted Gemma judge is not in it: that run failed authentication, and paid evaluation has stopped for budget. The full registered report is incomplete.
- It does not say fine-tuning can never work. It says this training run did not improve the judge under these data and settings.
- Training was completed on all three seeds. That training can be run and that it improves the judge are separate findings.
- The checkpoint and the seed were chosen on the validation split, which has 4 Different pairs out of 86. The validation F1 behind that choice is 0.333 for seed 0, 0.444 for seed 1 and 0.364 for seed 2. Each of the three rests on four pairs.
- The trained judges learned from synthetic stories only. The test set is children's stories only.
- A constructed pair sets one character's reference against another character's page. It is Different by construction and no rater saw it. 2 of the 104 constructed training pairs contradict a rated pair: the raters had labelled the same two image pairs Same. That is 4 of the 484 training records. No validation or test pair is affected.
- Two raters labelled the test pairs. On the 329 pairs they gave the same answer 89.7% of the time. Cohen's κ, which corrects for chance agreement, is 0.634. Both raters were briefed by the owner, who also settled their disagreements, so κ cannot catch a misunderstanding they shared.
Source: docs/capstone/research_runbook.md (opens on GitHub in a new tab), “Partial untuned versus fine-tuned comparison (2026-10-08)”, “Exact values of the partial comparison (copied 2026-10-11)”, “What the registered freeze contains” and “Registered agreement and sensitivity analyses (2026-10-09)”
Source: docs/capstone/group-guide.md (opens on GitHub in a new tab), §4 and §5.2