Why studies disagree

Studies on the same topic disagree because they rarely ask exactly the same question. Different participants, doses, comparison groups, outcome measures, durations and analytical choices can all move a result. Chance and publication bias account for much of the rest.

Conflicting findings usually reflect different populations, doses, outcomes, durations and analytical choices - not dishonesty.

Key takeaways

They are not asking the same question

Headlines flatten detail. One trial may test twelve weeks of supervised aerobic exercise in adults with diagnosed depression against usual care; another tests a self-reported step target in healthy students against nothing. Both get reported as 'exercise and mood'. Different populations, doses, comparators and outcomes produce different answers legitimately.

Small studies are noisy

Estimates from small samples vary widely by chance alone, and small studies that happen to produce large effects are more likely to be published. This is why initial findings frequently shrink as larger, better-controlled work follows - a pattern common enough to have its own name, the decline effect.

Analytical choices

Which outcome is primary, how missing data are handled, which covariates are included and where cut-offs are set all shift results. When these choices are made after seeing the data, effects inflate. Pre-registration exists to limit this, and its absence is worth noticing.

Publication and reporting bias

Null results are less likely to be submitted, accepted or covered. The published record therefore overstates the average effect, and a genuinely mixed field can look positive until the unpublished studies are accounted for.

What to do with a contradiction

Look for the most recent good systematic review rather than adjudicating single studies yourself. Ask whether the studies tested the same population and dose. Persistent disagreement usually means the effect is small, conditional on context, or not yet established - and describing it that way is more accurate than picking a side.

Frequently asked questions

Should I trust the newest study?

Not automatically. Newer is not better; larger, pre-registered, independently funded and replicated is better. A new study is one more data point in a body of evidence.

Does disagreement mean the science is broken?

No. Disagreement is a normal stage in establishing an effect, and it is how error gets corrected. It becomes a problem only when the uncertainty is hidden from readers.

How does Beiing handle conflicting evidence?

We label it as mixed rather than choosing the more appealing side, and we say what would need to change for the label to move. Our approach is set out in our research methodology.

Related on Beiing

Sources

  1. Cochrane Handbook for Systematic Reviews of Interventions - assessing certainty of evidence - Cochrane
  2. Page et al. (2021), The PRISMA 2020 statement: an updated guideline for reporting systematic reviews - BMJ / PubMed
  3. Ioannidis (2005), Why most published research findings are false - PLOS Medicine / PubMed
  4. MedlinePlus - Evaluating Health Information - U.S. National Library of Medicine
  5. PubMed - National Library of Medicine - NIH

Written by Beiing Editorial Team · Editorially reviewed by Research Desk · Published · Last reviewed . Editorial and research review by Beiing writers and researchers. This is not medical or clinical review, and no clinician signs off on our content.

Educational content only - not medical advice. In a crisis, call or text 988 (US) or your local emergency number. See crisis resources.