Eli Item-Writing Academy
About 40 minutes · Academy module: Item Analysis: Reading What the Numbers Say.
The p-value is the proportion of examinees answering correctly. 0.30–0.80 is the informative band for most purposes. Below 0.30 the item may be too hard or flawed; above 0.90 it contributes little information (though easy items have their place early in a practice set for confidence).
The discrimination index (upper–lower 27%) and the point-biserial correlation ask the same question: do high scorers get this right more often than low scorers? Positive and healthy (point-biserial above ~0.30) means the item measures what the exam measures. Near zero means noise. Negative means the item is actively mismeasuring — strong students are drawn to a distractor — and it goes to faculty review immediately.
For each option: what proportion chose it, and what was their average total score? Healthy pattern: the key pulls the top, each distractor pulls some of the middle, none pulls the top harder than the key. Pathologies: a dead distractor (near-zero pull — replace it), a seductive distractor (pulls top scorers — possible miskey or second defensible answer), and the flat item (all options pull evenly — the stem is probably unfocused).
Data suggests; faculty decides. The revision menu: reword (a distractor is ambiguous), replace (a distractor is dead), rekey (the data plus re-solving shows the key was wrong — rare, documented, and never silent), retire (the item is unfixable or off-spec). Every change is logged with the reason. Items are never auto-deleted or auto-revised by the system — the analytics flags are review signals only.
The discrimination is negative: the students choosing B average 81% overall while those choosing the key average 62%. Strong students are systematically drawn to B. This item is mismeasuring — it goes to faculty review, not to the bank.
— the review decision:
Flag for faculty review: negative discrimination (point-biserial −0.16). Distractor B pulls high scorers (avg 81%) harder than the key (avg 62%). Action: re-solve cold with two independent solvers; if B is defensible, rekey or retire — do not ship as-is.
Hard ≠ good. A hard item that discriminates positively is valuable; a hard item with negative discrimination is broken. The numbers tell you which — if you read all of them.
Recording it adds the module to your Academy completion record on this device — module, track, level, and date, ready to download from your account page for faculty-development documentation.