Bias & Sensitivity Review: Items That Are Fair

Learning goals

Run a sensitivity review on every item before it ships.
Distinguish content-necessary detail from gratuitous identity signaling.
Write diverse fact patterns without stereotyping.

Lesson 1 — What Sensitivity Review Checks

Three questions: (1) Does the item require any mention of race, gender, religion, national origin, disability, or other identity characteristics to test its point? If not, cut them. (2) If identity is load-bearing (a discrimination claim, a hate-crime enhancement), is it handled with precision and without stereotype? (3) Across the bank, do roles distribute fairly — are judges, lawyers, and experts demographically varied, and are parties not always the same profile?

Lesson 2 — Gratuitous Detail Is A Defect

"The Hispanic defendant…" in an item about the mailbox rule is gratuitous — it adds nothing to the tested point and invites stereotype threat. Names, too: if every criminal defendant in your bank has the same ethnic name pattern, that's a pattern worth fixing even if no single item is "wrong." The default: names and descriptors should be varied and irrelevant to the answer.

Lesson 3 — Necessary Detail, Handled Precisely

Some items need identity: a Title VII disparate-treatment question, a Batson challenge, an asylum claim. There, precision is the sensitivity: name the legally relevant characteristic, keep the rest neutral, and make sure the fact pattern supports the legal analysis rather than trading on tropes.

Lesson 4 — The Bank-Level Audit

Quarterly, sample 20 items and tally: who are the lawyers, judges, parties, victims, defendants? Patterns invisible at the item level emerge at the bank level. Fix by varying new items deliberately — not by retrofitting old ones awkwardly.

Spot-the-flaw drill

(intended key: A)

Fact pattern: A wealthy businessman offers to sell his yacht to his immigrant housekeeper for $50,000. She accepts by email. Is there an enforceable contract?

A. Yes, if the essential terms are agreed and there is consideration B. No, because housekeepers cannot contract with employers C. No, because email acceptances are ineffective D. Yes, but only if a broker was involved

Flaw: "wealthy businessman" and "immigrant housekeeper" are gratuitous — neither affec

"wealthy businessman" and "immigrant housekeeper" are gratuitous — neither affects formation doctrine, and the pairing trades on class and immigration tropes. (Options B–D are also legally frivolous, compounding the problem.)

Correction

Strip identity to what's needed (two parties, a price, an email acceptance) and rebuild the distractors as genuine formation errors.

Model rewrite

— cold-solve verified: 2/2 (solvers chose A, A)

Fact pattern: A seller emails a buyer: "Will sell you my yacht for $50,000. Reply by Friday." The buyer replies Thursday: "I accept." Is there an enforceable contract?

A. Yes, if the parties agreed on the essential terms with consideration B. No, because offers by email are revocable at any time before Friday C. No, because email acceptances are ineffective under the mailbox rule D. Yes, but only if the buyer also gave new consideration for the reply

Key: A

Drill key

Identity stripped; distractors now capture real errors (revocability confusion, mailbox-rule misapplication, consideration confusion). Both solvers chose A.

Sources

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. (Standard 3 — fairness in testing.)
Ramsey, P. A. (1993). Sensitivity review: The ETS experience as a case study. In Holland, P. W., & Wainer, H. (Eds.), Differential item functioning. Lawrence Erlbaum Associates.