Eli Item-Writing Academy
About 30 minutes · Academy module: Peer Review: The Cold-Solve Protocol.
The author cannot review their own item — they know the tested point, so every ambiguity is invisible to them. Cold solvers (colleagues who haven't seen the item) replicate the examinee's position. The protocol: solver reads the item once, selects an answer, and writes one sentence on why each distractor fails. No discussion before both solvers commit.
Four judgments per item: (1) Tested point — can the solver state it in one sentence? If not, the item is muddy. (2) Key — did both solvers converge? If not, diagnose (see the Single-Best-Answer module's two-key rescue). (3) Distractors — did each attract at least consideration, and can the solver name each one's error? (4) Sensitivity — any gratuitous identity detail or loaded scenario?
When solvers disagree, the item is telling you something. Common diagnoses: ambiguous facts (two readings), a distractor that's defensible on the call as written (sharpen the call), or a tested point the author never stated (write it down, rebuild). Log the disagreement and the fix — the log is what makes your bank's quality auditable.
For large banks, sample: every new item gets two solvers; a quarterly 10% re-sample catches drift. Track solver agreement rates per author — not to punish, but to target coaching. The Question Studio's built-in two-checker gate automates this protocol for items drafted with Eli.
no independent review — the author's blind spots ship with the items. Within a week, students report three disputed keys; the author spends more time firefighting than the review would have cost.
The minimum viable protocol: one colleague, cold, 20 items, one sitting. Even a single cold solver catches most key disputes and unfocused stems.
— the review log entry:
Item CIV-S07 (Erie): Solver A chose key (B), Solver B chose (D). Diagnosis: fact pattern ambiguous about whether the federal rule was "arguably procedural" — B and D defensible on different readings. Fix: added one sentence clarifying the rule's function; both solvers re-solved cold → B, B. Shipped.
The log shows the disagreement, the diagnosis, the fix, and the re-solve. That paper trail is the quality system.
Recording it adds the module to your Academy completion record on this device — module, track, level, and date, ready to download from your account page for faculty-development documentation.