What the consistency judge was trained and tested on, and how "same character" was decided. Every number names the file it comes from.
What is in it
A pair is one reference picture of a character and one book page. Its label says whether the page shows that same character. The registered dataset holds 899 pairs from 42 generated books. Synthetic stories were written by the researchers for this purpose. Donated stories were written by children, and they are used for the test only.
Pairs in each split
Split
Stories from
Stories
Characters
Pairs
Same
Different
Train
Synthetic
24
44
484
327
157
Validation
Synthetic
6
11
86
82
4
Held-out test
Donated
12
24
329
282
47
All
42
79
899
691
208
No story or character is in more than one split, so the judge is tested on characters it never saw.
Of the train split's Different pairs, 104 are constructed: one character's reference set against another character's page. The other 795 pairs came out of the pipeline.
Two of those constructed pairs reuse an image pair the raters had labelled Same, so the training file holds those two image pairs with both answers. The dataset is registered and was left as it is.
The judge trains on synthetic stories and is tested on children's. A gap between the two shows up in the test result.
A separate exploratory freeze excluded 16 disputed synthetic training pairs, leaving 468. The completed three-seed training used the registered freeze and kept all 484 pairs.
Three donated candidate stories were excluded after pipeline failures. They were test candidates, never training stories.
Human and non-human pairs
Character kind follows the intake-declared non-human roster matched to each generated character's name. These counts cover the generated characters and all pairs, including constructed ones. They differ from a page audit, where a page can contain both human and non-human characters.
Two group members each labelled every pipeline pair once.
Where their labels disagreed, the study owner settled the pair.
A rater saw only the two images, never the story. The judge sees the same.
Same when species, face, body colours and body features all match the reference.
Different needs a difference the rater can name from the list below. Without one, the answer is Same.
Clothing, art style and lighting are not identity. Each of them alone is Same.
Reasons a pair is Different
Reason
Counts
Does not count
Pairs
Different Face
Face shape, eye style, or facial markings belong to another individual
expression, mouth open, looking away
141
Wrong Body Feature
Countable or structural parts differ: eyes, ears, tail, horns, wings, limbs
a part hidden by the pose
50
Character Absent
The reference character is not on the page
other characters on the page
18
Wrong Clothing/Accessories
Clothing or accessories changed, added or missing
a clothing change on its own, which is Same. Tick it only when another reason already makes the pair Different
12
Wrong Color
Skin, fur, hair, eye or body colour differs from the reference
clothing colour (that is Wrong Clothing)
9
Wrong Style
Art style looks off
an art style change on its own, which is Same. Tick it only when another reason already makes the pair Different
4
Wrong Species
Animal type or core silhouette differs
pose, angle
2
A pair can carry more than one reason, so the last column adds up to more than the 208 Different pairs.
Two things to know before reading the rulebook
Its opening describes one rater labelling twice. That plan was replaced on 2026-09-23 by the two raters and owner adjudication described above. The rulebook is frozen, so its text was left as written.
Ten pairs that both raters had marked Different for clothing or style alone were reviewed with AI assistance and set to Same with the owner's approval. Nine are train pairs and one is validation. No test label changed. It is reported as a deviation.
These six were hand-picked to show the rules at work. They are not a random sample and say nothing about how often each case occurs. All six are synthetic pairs from the train and validation splits, shown with their final labels. Open "Frozen record" on a card to see what is stored for that pair.
No test pair is shown. The test set comes from children's donated stories, and its images are not published.
Same character
ReferenceBook page
The pose, the expression and the setting changed. The face, hair and build did not.
Both raters first answered Same.
Frozen record
{
"pair_id": "033b8372e89e4c06",
"char_id": "syn-025:c0",
"split": "val",
"provenance": "synthetic",
"pair_type": "pipeline",
"differences_observed": "Present on the page: human, brown hair, tan skin, human body with arms and legs, human face with eyes, nose, and mouth, human body, blue shirt, pants.",
"same_character": true,
"label": false,
"anatomy_intact": true,
"text_free": true,
"failure_reasons": [],
"ref_verdict_status": "passed"
}
A sad face is an expression, which does not count. Species, body colours and the teacup all match. The last card checks this same page against another tortoise's reference.
Both raters first answered Same.
Frozen record
{
"pair_id": "14126f2a535e8682",
"char_id": "syn-020:c0",
"split": "train",
"provenance": "synthetic",
"pair_type": "pipeline",
"differences_observed": "Present on the page: tortoise, white, blue, tortoise with a teacup shell, tortoise face, chipped teacup shell with a handle, back legs sticking out.",
"same_character": true,
"label": false,
"anatomy_intact": true,
"text_free": true,
"failure_reasons": [],
"ref_verdict_status": "passed"
}
Only the character in the reference is compared. The man on the page does not count, and an open mouth is an expression.
Both raters first answered Same.
Frozen record
{
"pair_id": "4198b9cd8341bd3f",
"char_id": "syn-022:c0",
"split": "train",
"provenance": "synthetic",
"pair_type": "pipeline",
"differences_observed": "Present on the page: dog, brown and white fur, one blue eye, one brown eye, four-legged canine, dog face with one blue eye and one brown eye, bent tail, none.",
"same_character": true,
"label": false,
"anatomy_intact": true,
"text_free": true,
"failure_reasons": [],
"ref_verdict_status": "passed"
}
The colours and the beak carried over, but the page drew a cat's head on the body. That is another animal and another face.
Both raters first answered Different.
Frozen record
{
"pair_id": "b865a5466cccc89f",
"char_id": "syn-009:c0",
"split": "train",
"provenance": "synthetic",
"pair_type": "pipeline",
"differences_observed": "The character is not the species the reference shows. The face is that of a different individual. Expected: bird, brown, orange, small round bird body with long beak, bird face with large eyes and curved beak, round body, long curved beak, none.",
"same_character": false,
"label": true,
"anatomy_intact": true,
"text_free": true,
"failure_reasons": [
"wrong_species",
"different_face"
],
"ref_verdict_status": "failed"
}
A constructed pair: a script set this tortoise's reference against the page of the other tortoise in the same story. The species matches and the individual does not. Every constructed pair carries this one reason and this one sentence, written by the script and not by a rater.
Not shown to the raters. A script set the label.
Frozen record
{
"pair_id": "848c636f47f8e342",
"char_id": "syn-020:c1",
"split": "train",
"provenance": "synthetic",
"pair_type": "constructed",
"differences_observed": "The face and build are those of a different individual; this is not the reference character.",
"same_character": false,
"label": true,
"anatomy_intact": true,
"text_free": true,
"failure_reasons": [
"different_face"
],
"ref_verdict_status": "failed"
}
Each pair is one row in the dataset's manifest. The cards above show that row as it is stored, without the two image file paths. The table says what each field is and who filled it in.
What each field of a record means
Field
Meaning
pair_id
The pair's id, made by the build script. It carries no meaning.
char_id
The story and the character the reference belongs to. Splits are made on it. It never reaches the training text.
split
train, val or test. Bookkeeping, not training text.
provenance
synthetic for a story the researchers wrote, donated for a child's story.
pair_type
pipeline for a page the pipeline drew for this character, which the two raters labelled. constructed for a pair a script made from two characters.
differences_observed
A sentence a script builds from the ticked reasons and the character's stored attributes. No rater wrote it.
same_character
The final answer. true means Same.
label
The opposite of same_character. true means Different, the class the judge is scored on.
anatomy_intact
false when a rater ticked Broken Anatomy. A constructed pair takes the default, true.
text_free
false when a rater ticked Text Visible. A constructed pair takes the default, true.
failure_reasons
The reasons ticked, from the fixed list of seven above.
ref_verdict_status
Whether the automatic check of the reference picture passed when the book was made. Bookkeeping, not training text.
The model is trained on a prompt and the answer it should give. Every training record uses the same prompt. This is the record for the constructed tortoise pair above.
Prompt
<image><image>The FIRST image is a canonical character reference. The SECOND image is one page of a picture book in which that character should appear drawn to match the reference.
First describe every difference you observe between the character on the page and the reference. Then say whether it is the same character; list which of the reference's attributes are actually present on the page; whether the page is drawn in the same art style as the reference; whether the character's anatomy is intact; whether the character is drawn exactly once; and whether the picture is free of any text. Finally list the failure reasons that apply, choosing only from the fixed set.
Answer the model is trained to give
{
"differences_observed": "The face and build are those of a different individual; this is not the reference character.",
"same_character": false,
"attributes_present": [],
"style_match": false,
"anatomy_intact": true,
"subjects_unique": true,
"text_free": true,
"failure_reasons": [
"different_face"
]
}
Three of the answer's fields were never labelled: attributes_present, style_match and subjects_unique. They hold the same fixed value in every training record (an empty list, false and true), so they say nothing about this pair.
