Which measurement pairs support the comparison?
You review a fictional Kestrel Survey science product before the science lead makes a comparison claim.
The question
A pair is comparable only if both readings use the approved reference version. The review requires at least 10 comparable pairs and at least 80% of them within an absolute difference of 2 units, inclusive.
| Pairs | |
|---|---|
| Marked comparable in the summary | 12 |
| Within 2 units | 10 |
The summary labels all 12 pairs as comparable. You can check the reference versions against their records.
“Decide whether the evidence supports the comparison and what would change your decision.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What must you check first?
What would you do here? Open the move you’d make.
Check only whether the rate reaches 80%
Tempting, but it misses something. A high rate can still come from too few comparable pairs.
Verify comparability before using the count and rate
The strong move. Both thresholds apply to the same eligible records.
Treat a matching number as proof of comparability
Tempting, but it misses something. An apparent match can use an outdated reference version.
See how a strong candidate says it
“Check which pairs use the approved reference version. Then apply both the 10-pair minimum and the 80% agreement threshold.”
Why it works: The version check determines which pairs can enter both counts.
Step 2: What is the provisional result?
What would you do here? Open the move you’d make.
Give a provisional pass and review the version records
The strong move. The count and rate meet the thresholds if all 12 pairs are eligible.
Publish the comparison from the summary
Tempting, but it misses something. The data that make the comparison valid have not been checked.
Call the comparison a fail because two pairs miss
Tempting, but it misses something. The policy allows some differences above the limit.
See how a strong candidate says it
“If all 12 pairs are comparable, 10 of 12 is 83.3%. Both requirements are provisionally met. I would verify the version records before accepting the claim.”
Why it works: The summary gives an initial result, conditional on its version labels.
Step 3: What do the eligible pairs show?
The version audit finds 3 of the apparent passes used the old reference version. Every other pair used the approved version.
What would you do here? Open the move you’d make.
Keep the original 10 of 12 result
Tempting, but it misses something. That result includes three pairs the audit found ineligible.
Calculate 7 of 12 as the agreement rate
Tempting, but it misses something. This leaves noncomparable pairs in the denominator.
Report 7 of 9 and withhold the comparison claim
The strong move. Nine eligible pairs and 77.8% both fall short.
See how a strong candidate says it
“The three excluded pairs were all apparent passes. That leaves 7 of 9 comparable pairs, or 77.8%. Both the count and rate miss the thresholds, so I would withhold the claim.”
Why it works: The version audit removes the same three pairs from the numerator and denominator.
Step 4: What result would change the decision?
What would you do here? Open the move you’d make.
Accept the claim as soon as any new pair arrives
Tempting, but it misses something. The additional pair must be comparable and within the limit.
Request a comparable pair and check its result
The strong move. One additional pass can reach both thresholds exactly.
Remove an eligible miss to increase the percentage
Tempting, but it misses something. The miss belongs in the denominator and the sample would get smaller.
See how a strong candidate says it
“One additional comparable passing pair would give 8 of 10, or 80%. I would request more comparable evidence and have the science lead review it before accepting the claim.”
Why it works: A replacement has to qualify and pass. A new pair outside the difference limit would fix the count but leave the rate below 80%.
The answer, in short
- Decision
- Withhold the claim and give the science lead the eligible count and rate.
- Because
- Seven of nine comparable pairs fall within the allowed difference. Both thresholds fail.
- What would change it
- One additional comparable passing pair would meet both thresholds at 8 of 10.
Common traps
- Assume a summary label proves comparison eligibility.
- Keep excluded pairs in the denominator.
- Assume a new pair passes before checking its result.
Check your answer
- Did I check the version before interpreting a difference?
- Did I report the provisional rate with its assumption?
- Did I remove the three pairs from both counts?
- Did I state what new result would satisfy both thresholds?
Last reviewed Nov 11, 2026