The science says…or does it? What to know about interpreting studies like a researcher

We are bombarded with “facts” every day. Research says this, research says that. But how much of that is actually true? We all know headlines oversimplify, exaggerate, or misinterpret what research has actually found, yet we are still inclined to believe them. Well, the truth is, not all research is created equal, and understanding the difference between a compelling headline and solid scientific evidence is more important than ever.

The good news? You don’t need a PhD to think like a researcher. You just need to know what you’re reading and what questions to ask.

Know a headline from a finding

How they hook you

It’s happened to all of us - you’re scrolling through your social media feed or watching the news and claims like this grab your attention:

“Red wine is good for your heart health.”

“Scientists discover that waking up at 5 a.m. makes you more successful.”

“Studies show red meat is bad for you.”

They all sound convincing. They mention scientists, research, and studies. They could be true. Are they? Surely they wouldn’t be trying to mislead us.

The first thing to keep in mind is that the journey of a piece of scientific information rarely makes its way directly from a researcher’s computer to your phone screen unchanged. Results are obtained in a lab. Researchers analyze data, interpret what they found, and discuss the findings with peers before deciding whether to start all over again or submit something for publication. Publishing in an academic journal is a long process in itself where it’s possible information is changed or updated during peer review. A press release is then sometimes issued to summarize big findings. A journalist sees that release and writes an article, which is positioned a certain way, introducing bias. An editor comes up with a headline. Someone turns it into a social media post. You read it on your morning commute to work.

By the time it reaches your feed, a scientific conclusion that took years, even decades to come to, may sound much simpler or more certain than the original findings justify. And with each step of that game of telephone, information can disappear or be omitted.

It’s not always that simple

Take the last headline, “red meat is bad for you.” If you talk to any average joe, he will tell you that meat is bad, it raises cholesterol and blood pressure, it might even cause cancer. Indeed, papers have found associations between higher red-meat consumption and a range of health outcomes, including cardiovascular disease, type 2 diabetes, certain cancers, and mortality (1).

But when you look carefully, the data show a different story. The studies in question largely compared people who reported eating more red meat with those who ate less. These groups may also differ in smoking, exercise and other habits, making it difficult to separate the effects of meat from the rest of their lifestyle.

Meanwhile, other large reviews have found that eating less red or processed meat was linked to only very small reductions in certain health risks and this evidence was uncertain because other habits and inaccurate food reporting could affect the results (2), and that processed meat, not red meat, is what is linked to heart disease and diabetes (3). What’s more, a survey of over 2,000 people following the carnivore diet found that respondents reported few adverse effects, improvements in diabetes-related measures and better well-being (4). So it’s not always as simple as “red meat is bad for you.”

This also works in the reverse— sometimes a scientific claim is controversial precisely because popular sentiment is to disagree with what the evidence shows.

Vaccines are a good example. Claims that childhood vaccines cause autism circulate all over social media. This has been investigated extensively with no evidence in support. In 2002, a Danish study of more than 537,000 children found no association between the measles, mumps and rubella (MMR) vaccine and autism (5). In 2014, a meta-analysis combining five cohort studies involving more than 1.2 million children, alongside five case-control studies, also found no association between vaccination and autism (6). Then, in 2019, another Danish study followed more than 657,000 children and again found no increased risk after MMR vaccination—even among children with a family history of autism (7).

So, whether a headline tells us something we already tend to believe (like red meat being bad for you) or something we tend to disagree with (vaccines being totally safe), the same rule applies: look beyond the headline. The research will show evidence in support or challenging it, but the answer is never that simple.

Know what study you’re reading

Types of studies

Before interpreting data, you first need to know what kind of study you’re dealing with. There are many different ways of conducting scientific research, and they are designed to answer fundamentally different questions.

Here are some of the main types of studies you might encounter.

• Observational study. Researchers observe what happens naturally without assigning an intervention. These studies can identify associations between variables. Observational studies include:

  • Cohort study. Researchers follow a group of people over time and compare different exposures or characteristics to see whether they are associated with later outcomes.

  • Case-control study. Researchers compare people with a particular outcome to people without it, then look back at previous exposures.

  • Cross-sectional study. Researchers measure exposure and outcome at a single point in time, giving them a snapshot of a population.

• Randomized controlled trial (RCT). Researchers randomly assign participants to different interventions or groups, allowing them to investigate whether an intervention causes a difference in an outcome.

• Systematic review. Researchers systematically collect and evaluate existing studies addressing a particular research question.

• Meta-analysis. Researchers statistically combine the results of multiple studies to produce an overall estimate of an effect. Learn more about meta-analyses here.

• Case report. Researchers describe an unusual or interesting case, which can help identify potential new observations or hypotheses.

How they work in practice

An observational study can tell us that people who do X tend to experience Y. It cannot, by itself, tell us that X caused Y. The large cohort studies looking at red meat consumption found that people who reported eating more red meat tended to have higher rates of certain health outcomes. But because researchers didn't randomly assign people to eat different amounts of red meat, the studies can't completely separate the effect of red meat itself from everything else that differs between those groups.

An RCT works differently. Instead of observing what people do, researchers randomly assign participants to different interventions and then compare the outcomes. This makes it much easier to investigate whether the intervention itself caused a difference.

But that doesn’t mean RCTs are automatically good and observational studies are bad. Some questions simply can't be answered with an RCT. We can't ask thousands of children to receive or avoid a vaccine for years just to see whether it changes their risk of autism. Instead, researchers opt for the study design that would make the most sense realistically and ethically.

Then there are systematic reviews. Instead of conducting a new experiment, researchers systematically gather and evaluate all the relevant studies they can find on a particular question. This can give us a much broader picture of the evidence than any single study, but systematic reviews aren't automatically more reliable just because they include more research.

However, the quality of the review depends on which studies were included, how they were selected and assessed, and whether the underlying studies are themselves strong. Researchers can also reach different conclusions when studies use different methods, populations or definitions, making the results difficult to combine. And if the available research is biased or incomplete, a systematic review can only work with what is out there.

The point isn't to rank study designs from “good” to “bad.” It's to understand what each design can actually tell us. An observational study can show whether people who eat more red meat have different health outcomes from those who eat less. An RCT can tell us whether the medication causes a change in blood pressure. And, a systematic review might give us the clearest picture when we want to understand the evidence across many studies. Each comes with its own advantages and drawbacks.

Know basic statistics

Once you know the type of study you’re looking at, the next step is to look into the data. Most scientific studies involve statistics, but you don’t need to be a statistician to understand the most important numbers. You just need to know what they are supposed to be telling you.

Here are a few of the most useful ones to recognize.

Effect size

An effect size describes the size of a difference or relationship. A result can be statistically significant without being particularly large or meaningful in practical terms.

This is one reason headlines can be misleading. A hypothetical headline could read: “Eating red meat linked to a 50% higher risk of heart disease.” This sounds dramatic and scary. But percentages need context.

Suppose that over 10 years, heart disease develops in 2 out of every 1,000 people in one group and 3 out of every 1,000 in another. That is a 50% relative increase in risk, but the absolute increase is 0.1 percentage points—from 0.2% to 0.3%—or 1 additional case per 1,000 people.

Both numbers are mathematically correct, but they create very different impressions. So if you see a dramatic percentage in a headline, always ask: 50% of what, and over how long?

P-values

You will also come across p-values, particularly when researchers are comparing groups or testing whether an observed difference could have arisen under a specified statistical model.

A p-value measures how likely you would be to obtain results at least as extreme as those observed, assuming the null hypothesis is true and the statistical model’s assumptions hold. A commonly used threshold is if p < 0.05, there is a statistically significant effect.

But this does not mean there is a 95% chance that the researcher's hypothesis is true. It means that, assuming the null hypothesis and the statistical model’s assumptions are correct, results at least as extreme as those observed would occur less than 5% of the time.

It also doesn't tell you how large or important the effect is, which is why you need effect sizes. A p-value is essentially one piece of information about a result, not the full truth, so it should never be interpreted as THE result.

Correlation

Correlation is one of the most important concepts to understand when reading research.

A correlation is a measure of the relationship between two variables. If two variables are correlated, they tend to vary together. But that does not tell us that one caused the other.

For example, ice cream sales and drowning deaths both increase during the summer. That doesn't mean eating ice cream causes drowning. A third factor, warmer weather, affects both.

The same problem appears in scientific research.

If people who consume more red meat also have higher rates of a particular health outcome, that is an association worth investigating. But those people may differ from people who eat less red meat in many other ways. Their overall diet, physical activity, smoking habits, alcohol consumption, socioeconomic status, or other factors may make a huge difference.

The association between two variables should be used as the starting point for asking questions and delving deeper.

Confidence intervals

Scientific results are estimates, and estimates come with uncertainty. A confidence interval gives you information about how precise an estimate is. For example, you might see a result reported as a hazard ratio of 1.20 with a 95% confidence interval of 1.10–1.30.

The important thing is that the researchers aren't saying, “The effect is 1.20.” They are also showing you a range of values compatible with the data and their statistical assumptions, which are between 1.10 and 1.30.

A narrow interval suggests a more precise estimate. A wide interval indicates more uncertainty. For a hazard ratio, 1 means no difference between groups. If the interval includes 1, no difference is also compatible with the data, but that doesn’t prove there is no effect.

An important thing to remember is that a precise estimate doesn’t mean it’s correct. A study can have a narrow confidence interval and still be affected by various sources of bias.

Bayesian thinking

One final statistical concept is particularly useful for thinking about science: Bayesian reasoning.

The basic idea is simple. New evidence doesn't exist in a vacuum. How much you should update your belief depends partly on what was already known before that evidence appeared.

Think of it as: What did we think before? What does this new study tell us? How should our confidence change?

A single surprising study doesn't necessarily overturn decades of consistent evidence. Likewise, one study that agrees with an existing body of research doesn't suddenly prove the hypothesis beyond doubt.

Science is cumulative. Each study adds another piece to the picture. Read more here on Bayesian statistics.

Know what to ask next

Reading and interpreting data are the first step before thinking something is true. You then need to ask the right questions to get more information. A few things to keep in mind here.

First, be wary of strong claims. Words like “proves,” “always” or “miracle” deserve a closer look. A small, short-term study cannot establish that something works for everyone or is safe over a lifetime.

Always consider the wider evidence base. One study rarely settles a debate. Does the finding hold up in other studies, or has someone picked the one paper that supports their argument while ignoring the rest?

Check what was actually studied. Studies done in mice may not apply to humans. Feeling healthier is different from having fewer diagnosed illnesses, and improving a blood marker does not necessarily mean preventing disease.

Other questions to ask yourself:

• What was the comparison? Could differences between the groups explain the result?

• Is something missing? Were uncertainty, dropouts or inconvenient findings left out?

• Who funded it? Conflicts of interest deserve attention, though they do not automatically invalidate a study.

• Does the headline match the paper? Check whether the researchers actually made the claim being circulated.

Takeaways

A headline rarely tells the whole story, and a single study rarely gives us the final answer. Understanding the type of study helps you judge what its findings can actually tell you. Knowing basic statistics helps you look beyond dramatic numbers and headlines. Ask questions, consider the wider evidence and apply the same scrutiny to claims you agree with as those you doubt.

Above all, always be willing to change your mind in light of new evidence.

References

1. Pan, A., Sun, Q., Bernstein, A. M., Schulze, M. B., Manson, J. E., Stampfer, M. J., ... & Hu, F. B. (2012). Red meat consumption and mortality: results from 2 prospective cohort studies. Archives of internal medicine, 172(7), 555-563.

2. Zeraatkar, D., Han, M. A., Guyatt, G. H., Vernooij, R. W., El Dib, R., Cheung, K., ... & Johnston, B. C. (2019). Red and processed meat consumption and risk for all-cause mortality and cardiometabolic outcomes: a systematic review and meta-analysis of cohort studies. Annals of internal medicine, 171(10), 703-710.

3. Micha, R., Wallace, S. K., & Mozaffarian, D. (2010). Red and processed meat consumption and risk of incident coronary heart disease, stroke, and diabetes mellitus: a systematic review and meta-analysis. Circulation, 121(21), 2271-2283.

4. Lennerz, B. S., Mey, J. T., Henn, O. H., & Ludwig, D. S. (2021). Behavioral Characteristics and Self-Reported Health Status among 2029 Adults Consuming a "Carnivore Diet". Current developments in nutrition, 5(12), nzab133. https://doi.org/10.1093/cdn/nzab133

5. Madsen, K. M., Hviid, A., Vestergaard, M., Schendel, D., Wohlfahrt, J., Thorsen, P., Olsen, J., & Melbye, M. (2002). A population-based study of measles, mumps, and rubella vaccination and autism. The New England journal of medicine, 347(19), 1477–1482. https://doi.org/10.1056/NEJMoa021134

6. Taylor, L. E., Swerdfeger, A. L., & Eslick, G. D. (2014). Vaccines are not associated with autism: an evidence-based meta-analysis of case-control and cohort studies. Vaccine, 32(29), 3623–3629. https://doi.org/10.1016/j.vaccine.2014.04.085

7. Hviid, A., Hansen, J. V., Frisch, M., & Melbye, M. (2019). Measles, Mumps, Rubella Vaccination and Autism: A Nationwide Cohort Study. Annals of internal medicine, 170(8), 513–520. https://doi.org/10.7326/M18-2101