Statistics reporting problems: nine traps in the numbers

What to take away

Most numerical errors are not advanced mathematical failures. They are failures to define, compare, or qualify. The repairs below can be applied during reporting and editing.

These failures show up in recognizable news forms: a fresh opinion poll, the monthly employment report, an agency's annual data release, a chart redrawn after revisions. Statistics reporting problems follow nine patterns, each with a repair.

1. Missing denominator

Problem: "Complaints rose 40 percent" appears without the old and new counts.

Repair: Give both values, period, covered population, and whether reporting rules changed. A rise from 5 to 7 is 40 percent but carries a different scale from a rise of thousands. The developing story context checklist makes redoing that arithmetic a standing item before publication.

2. Percent and percentage points mixed

Problem: A rate moves from 20 percent to 25 percent and the story calls it a 5 percent increase.

Repair: Write "up 5 percentage points, or 25 percent relative to the earlier rate." The AP Stylebook entry on percent, percentage and percentage points keeps the two measures apart, so a headline or chart label cannot swap one for the other.

3. Tiny subgroup treated as exact

Problem: A full-sample margin is attached to a subgroup with far fewer respondents.

Repair: Find subgroup sample size and design-based uncertainty. If precision is too weak, describe the result as inconclusive rather than forcing a ranking. The American Association for Public Opinion Research asks poll reports to publish the sampling error and sample size behind each result, and those subgroup limits are covered step by step in reading a public opinion poll.

4. Point estimates turned into firm winners

Problem: Places are ranked 1 through 50 even though adjacent estimates overlap substantially.

Repair: Group estimates that cannot be distinguished, show uncertainty, or use ranges. Avoid "best" and "worst" when the evidence supports only broad tiers. The poll estimates comparison explains why adjacent point estimates so often overlap.

5. One-year movement called a trend

Problem: Two points define a trend without checking earlier years, revisions, seasonality, or method changes.

Repair: Plot a suitable series, mark breaks, and explain whether movement exceeds normal variation. Use "increased from last year" when that is all the evidence shows. The Bureau of Labor Statistics publishes both seasonally adjusted and unadjusted versions of its monthly employment figures, so a raw month-to-month move can be seasonal rather than a trend.

6. Correlation written as causation

Problem: Areas with higher Measure A also have higher Outcome B, so the article says A caused B.

Repair: Report the association and inspect study design, timing, confounding, reverse causation, and mechanism. The NIST/SEMATECH e-Handbook of Statistical Methods, in its scatter-plot section, states that an association in a plot cannot prove cause and effect, which is why the claim stays limited to an observed relationship unless other evidence establishes cause.

7. Chart scale drives the drama

Problem: A truncated vertical axis makes a small change look extreme, or uneven time intervals are spaced equally.

Repair: Choose a scale suited to the question, label it clearly, and show context. Include a zero baseline for bars because their lengths encode magnitude. A line chart may use a narrower range when readers can see the axis and the purpose is change, but test whether it distorts practical scale. Alberto Cairo's How Charts Lie works through the truncated-axis and uneven-interval cases and how those design choices change what a reader takes away.

8. Summary statistic hides structure

Problem: An average is reported without examining distribution, outliers, or groups.

Repair: Plot the data and inspect median, spread, missingness, and subgroups. The NIST/SEMATECH e-Handbook of Statistical Methods describes graphical methods as tools for finding structure, testing assumptions, selecting models, spotting outliers, and identifying relationships, so numeric summaries belong alongside visual inspection rather than on their own.

9. Revision handled as a cosmetic edit

Problem: A source revises the inputs, but the story swaps the number without checking derived percentages, rankings, charts, or conclusions.

Repair: Reproduce every dependent calculation, update graphics, compare the old and new finding, and add a correction if the earlier conclusion was materially wrong. The Bureau of Labor Statistics revises its initial monthly employment estimates as more data arrive, and the Census Bureau publishes new population estimates in annual vintages; both are routine cases where the first published number is not the last. Distribution-wide corrections follow the repair order in developing news briefing problems.

Repair sequence

Common questions

Is a zero-based axis always required?

Bars generally need zero because length represents magnitude. Lines can use a narrower scale for change if the axis is clear and the design does not misstate practical size.

Can two data points show a change?

Yes. They show a difference between two periods. They are usually insufficient to establish a sustained trend.

Should every article include a chart?

No. Use a chart when it makes pattern, distribution, or comparison easier to see. A short table may be clearer for a few exact values.

What if revised data strengthen the original conclusion?

Update the values and revision note. Transparency applies even when the direction stays the same.