ISLAND · Wildfire drought task · 236 labelled students
Two Clusterings Compared
The researchers' original groupings, defined by hand as coding rules, against the clusters discovered from LLM labels: what each group is, why each scheme exists, where they agree, where one splits what the other lumps together, and how the students in each overlap did.
Background reading
Start here. This page compares two ways of grouping the same students: the researchers' original groups, defined by hand as coding rules, and five clusters discovered from LLM labels. Read the definitions, then use the flow and the matrix to see where the groupings agree and where one splits what the other lumps together. Skill tracing (BKT) is the next level and is left out here.
The groups, defined
How students flow from one grouping to the other
Original (rules)Discovered (LLM)
Band width = number of labelled students in both groups. Click a band or a cell to explain it.
The overlap matrix
Each cell: students in both groups, and their share of the original row. Rows (dark edge) are original groups; columns (orange edge) are discovered clusters.
This overlap
What each side sees, and misses
Original groupings (researchers' rules)
Sees
Exactly whether a student isolated drought, and whether they followed a fair test with a fair tweak: the control-of-variables streak the curriculum aims at.
The same answer every time, from rules anyone can read (reproduced exactly for all 852 students).
The cases Hazbot's feedback is built on.
Misses
A clean fair test without a follow-up tweak: those students are "Exploring", the same group as students who never compared anything.
Comparisons made without one spark per zone: sim_group ignores them.
Why a student is exploring: stuck with identical zones, or changing several settings at once, call for different coaching.
Discovered clusters (LLM labels)
Sees
Different kinds of exploring, which split the original "Exploring" group four ways, each with its own coaching focus.
Efficient students who got the comparison right once and stopped: the original streak coding gives them no credit, and they made correct claims at rates close to the streak students.
Behaviour the rules have no letter for: moved sparks, wind as a confound, loops of identical reruns.
Misses, or risks
Its ground truth is Sonnet's labelling. It was checked against human fair-test codes (κ 0.91), but no person has sorted students into these five groups.
It judges "extended a fair test" weakly (κ 0.43 against the human C streak), which is what the original scheme does best.
Moderate separation (silhouette 0.31), one task, 236 labelled students.
Read together, the two are complementary: the original rules decide the case exactly; the discovered clusters say what kind of help fits. Data: aggregates over the 236 students with LLM labels (from 852 with run logs); outcome figures are hidden for overlaps under 5 students.