Statistical claim checklist for news editors

What to take away

• Name the measure, population, period, geography, and unit.
• Recalculate percentages and changes from source values.
• Check uncertainty before declaring a difference or trend.
• Inspect axes, missing values, and category definitions.
• Keep the headline within the study design and evidence.

Use this checklist on any draft whose central finding depends on a number. Some items will not apply to a census or administrative total. Record why a check was skipped instead of assuming the number's source resolves every question. The concepts behind each section are explained in the guide to reporting statistics and polls.

Source and release

• The issuing body and exact release are named.
• The source page is specific and stable.
• The release date, data period, and revision status are recorded.
• Tables, footnotes, methodology, and errata were reviewed.
• The article uses the correct vintage rather than a cached copy.

Measure and population

• The measured concept is defined.
• The eligible population or universe is clear.
• The geography and boundary version are correct.
• Counts, rates, percentages, indexes, and estimates are not mixed.
• The denominator matches the claim.

Write the denominator in words. "Thirty percent" is incomplete until the reader knows 30 percent of whom or what.

Sample and collection

• Sample size and selection method are available.
• Collection mode and field dates are recorded.
• Response and coverage limits are considered.
• Weighting variables and design effects are checked.
• Question wording and answer choices match the paraphrase.

For survey items, the step-by-step method in reading a public opinion poll covers sponsor, sample, wording, and weighting.

AAPOR's minimum disclosure standards call for details including dates, population, recruitment, mode, sample construction, weighting, question wording, and sampling-error treatment when applicable. The AAPOR disclosure standards support this method review. Disclosure supplies the basis for evaluation but does not itself prove a result accurate.

Arithmetic

• Totals add correctly, allowing for rounding and missing categories.
• Percentage change uses the old value as denominator.
• Percentage points are distinguished from percent change.
• Per-capita figures use compatible population estimates.
• Currency comparisons account for inflation when required.
• Averages are weighted correctly where groups differ in size.

Recalculate outside the source's presentation. Keep the calculation, inputs, and rounding rule with the reporting notes. Source documents themselves get the version checks in reading an official document.

Uncertainty and difference

• The confidence level and margin or interval are identified.
• Full-sample uncertainty is not assigned to subgroups.
• A difference is tested under the suitable method.
• Multiple comparisons are considered when many groups are ranked.
• Small-area estimates receive extra scrutiny.

Census training on ACS estimates says comparison involves more than checking which estimate is higher and describes significance testing that uses each estimate's margin of error. The Census presentation on ACS margins of error supports testing before declaring an ACS difference. Apply the correct method for other surveys.

Time and comparison

• Periods have the same length and seasonal treatment.
• Definitions and collection methods did not change unnoticed.
• Geographic boundaries are comparable.
• Dollar values use a stated price basis.
• The chosen baseline is justified.
• A record claim names the available time span.

Chart review

• Axes, units, zero points, and scales are labeled.
• Time intervals are regular or visibly marked.
• Color does not carry the only distinction.
• Missing data are shown rather than connected silently.
• The visual uses the same numbers as the prose.
• The caption states source, period, and notable transformation.

Interpretation

• Association is not written as causation.
• The finding stays within the sampled population.
• Alternative explanations are considered.
• A model result is labeled as modeled.
• The story distinguishes statistical and practical importance.
• The source had a chance to address a discovered discrepancy.

When the number describes government performance, the policy briefing checklist adds the program-side checks.

Headline and summary

• The headline names the population or place when needed.
• It does not use "proves" for observational evidence.
• It does not rank indistinguishable estimates as firm winners and losers.
• It states a revision if the story corrects an earlier conclusion.
• The summary includes the main limit beside the finding.

Common questions

Must every number include a margin of error?

No. Some totals are not sample estimates, and some uncertainty is expressed differently. Identify the source type and use its relevant quality measures.

What if percentages do not add to 100?

Check rounding, multiple-response questions, missing answers, and excluded categories. Explain the reason when readers could be confused.

Is a statistically significant difference important?

Not necessarily. Statistical evidence about a difference does not determine its practical size or consequence.

Should the editor rerun the source's model?

For a central or disputed finding, independent replication may be warranted. At minimum, verify inputs, assumptions, code availability, and reported diagnostics.