Critically Appraising Quantitative Evidence

Unit 4 gave you the framework for appraising a body of evidence. This unit goes underneath it, to the quantitative designs themselves and the numbers they produce. It is the largest unit in Part 1 and comes in three parts. This first part is about levels and bias. The idea to carry through all of it is one you have met twice already and will meet again: a level ranks a design by how much bias it admits, and says nothing whatever about how well the study was actually carried out. By the end you can say what a level describes and what it is silent about; say why randomisation is the thing that licenses a causal claim; tell statistical significance from clinical significance; name where bias enters a study and several named forms; and say what separates quasi-experimental research from an implementation project.

Seven videos on the logic of a hierarchy. The one to carry is that a level ranks the design's exposure to bias and says nothing at all about how well the study was done.

What determines a level.

EBP 0751 Levels Follow Design

A study's level follows from its design, not from its topic or its journal.

Why design is what matters.

EBP 0752 Bias Sets the Level

Designs are ranked by how much bias they admit. That is the whole logic of a hierarchy.

The half the level does not capture.

EBP 0753 Rigor of Execution

How well the study was actually carried out, which the level says nothing about.

How to work through a body.

EBP 0754 Highest Level Down, For Your Question

Start at the highest level available for your question and work down.

Two words, one idea.

EBP 0755 Hierarchy and Levels Are the Same Thing

Hierarchy of evidence and levels of evidence name the same idea.

The point unit 7 depends on.

EBP 0756 Different Questions, Different Hierarchies

Each question type has its own hierarchy. There is no single pyramid that ranks evidence for every question.

What a rank actually gives you.

EBP 0757 What Rank Buys You

A high rank buys a lower risk of bias, and nothing else. It does not buy correctness.

Three videos on the dividing line of quantitative appraisal. Randomisation is what licenses a causal claim, and everything else shows association.

The dividing line of the unit.

EBP 0758 Only Randomisation Establishes Cause

Randomisation is what licenses a causal claim. Nothing else in the toolkit does.

And the consequence.

EBP 0759 Everything Else Shows Association

Every other design shows association. Association is useful and it is not cause.

Where syntheses sit.

EBP 0760 Syntheses Rank Highest

A synthesis of studies outranks any single study, because it answers against the body.

Two videos on the judgement you are actually making, which is not whether a study has flaws.

The frame for appraisal.

EBP 0761 No Study Is Perfect

Every study has limitations. Finding one is not a reason to discard it.

The judgement you are making.

EBP 0762 Limited, or Limited Enough to Discount

The question is not whether a study is limited but whether it is limited enough to change what you would do.

Four videos returning to unit 4's triage questions, now pointed at quantitative designs.

The three questions, again.

EBP 0763 The Three Rapid Critical Appraisal Questions

The same three questions from unit 4, now applied to quantitative designs.

The first.

EBP 0764 Question One Is Validity

Are the results valid — did the methods produce a trustworthy answer.

The second.

EBP 0765 Question Two Is Reliability

Are the results reliable — the size and precision of what was found.

The third.

EBP 0766 Question Three Is Applicability

Will the results help my patients, which is the question the study cannot answer for you.

Three videos on a distinction that decides how you read almost every result you will meet.

The framing to hold.

EBP 0767 How Far From True?

Appraisal asks how far a result might sit from the truth, and in which direction.

The first kind of significance.

EBP 0768 Statistical Significance

Whether a result is unlikely to be chance. It says nothing about whether the result matters.

The second, and the one that decides.

EBP 0769 Clinical Significance

Whether the result is large enough to change what you do. A result can be statistically significant and clinically trivial.

Nine videos. Bias can enter at design, sampling, measurement, analysis or reporting, so checking one stage is not checking the study.

The word, defined.

EBP 0770 Bias

A systematic error that pushes a result away from the truth in a particular direction.

The neighbouring problem.

EBP 0771 Confounding Variables

A third factor related to both the intervention and the outcome, which can manufacture an apparent effect.

Where bias enters.

EBP 0772 Bias Can Enter at Any Point

Bias can enter at design, sampling, measurement, analysis or reporting. Checking one stage is not checking the study.

The first named bias.

EBP 0773 Selection Bias

Distortion from how participants came to be in the study.

A common source of it.

EBP 0774 Convenience Sampling and Reach

Sampling whoever was available, which shapes who the result can be said to apply to.

A subtler source.

EBP 0775 Gatekeeper Bias

Distortion introduced by whoever decided which patients could be approached at all.

The classic protection.

EBP 0776 Knowing Who Received What

When participants or assessors know the allocation, expectations affect what is reported and measured.

Bias in the measuring.

EBP 0777 Measurement Bias

Distortion from how the outcome was measured, including instruments that drift.

The last named bias.

EBP 0778 Recall Bias

People with an outcome remember exposures differently from people without one.

Seven videos on observation and quasi-experiment. The last two matter beyond this unit: they draw the line between research and an implementation project.

The weaker designs begin.

EBP 0779 Nonexperimental Designs

Designs that observe without intervening. They describe what happens rather than what a change causes.

The weakest arrangement.

EBP 0780 One-Group Designs

A single group with no comparison, so there is nothing to attribute a change against.

The middle ground.

EBP 0781 Quasi-Experimental Designs

An intervention is made but allocation is not randomised, so groups may differ at the start.

The commonest quasi-experimental form.

EBP 0782 Pre-Post Designs

Measuring before and after a change. Anything else that changed in between is inside the result.

What separates it from a project.

EBP 0783 Quasi-Experimental Work Aims at Generalisable Knowledge

Quasi-experimental work still aims at generalisable knowledge, which is what makes it research.

The distinction that follows.

EBP 0784 Evidence Implementation Is Not Research

An implementation project aims at local improvement, not at generalisable knowledge, and is a different activity.

Where such projects sit.

EBP 0785 Where Implementation Projects Rank

Implementation and improvement projects rank low as external evidence, and that is not a criticism of them — they were answering a local question.

That is levels and bias. Against the outcomes. A level follows from the design and ranks how much bias that design admits. It says nothing about rigor of execution, and a high rank buys a lower risk of bias rather than correctness. Randomisation is what licenses a causal claim. Every other design shows association, which is useful and is not cause. And each question type has its own hierarchy — there is no single pyramid, which is the point unit 7 is built on. Statistical significance says a result is unlikely to be chance. Clinical significance says it is large enough to change what you do. A result can be both statistically significant and clinically trivial, and that combination is common. Bias can enter at design, sampling, measurement, analysis or reporting — selection, gatekeeper, measurement and recall bias among the named forms. Checking one stage is not checking the study. And the line between quasi-experimental research and an implementation project is what it aims at: generalisable knowledge, or local improvement. Implementation projects rank low as external evidence, which is not a criticism — they were answering a local question. Part B takes up trials and systematic reviews, and how each design is appraised.