Evaluating a harm study
Evaluating the Validity of a Harm Study
Are the results valid?
FOR COHORT STUDIES: Aside from the exposure of interest, did the exposed and control groups start and finish with the same risk for the outcome?
1. Were patients similar for prognostic factors that are known to be associated with the outcome (or did statistical adjustment level the playing field)?
The two groups, those exposed to the harm and those not exposed, must begin with the same prognosis. The characteristics of the exposed and non-exposed patients need to be carefully documented and their similarity (except for the exposure) needs to be demonstrated. The choice of comparison groups has a significant influence on the credibility of the study results. The researchers should identify an appropriate control population before making a strong inference about a harmful agent. The two groups should have the same baseline characteristics. If there are differences investigators should use statistical techniques to adjust or correct for differences.
2. Were the circumstances and methods for detecting the outcome similar?
In cohort studies determination of the outcome is critical. It is important to define the outcome and use objective measures to avoid possible bias. Detection bias may be an issue for these studies, as unblinded researchers may look deeper to detect disease or an outcome.
3. Was follow-up sufficiently complete?
Patients unavailable for complete follow-up may compromise the validity of the research because often these patients have very different outcomes than those that stayed with the study. This information must be factored into the study results.
FOR CASE CONTROL STUDIES: Did the cases and control group have the same risk (chance) of being exposed in the past?
1. Were cases and controls similar with respect to the indication or circumstances that would lead to exposure?
The characteristics of the cases and controls need to be carefully documented and their similarity needs to be demonstrated. The choice of comparison groups has a significant influence on the credibility of the study results. The researchers should identify an appropriate control population that would be eligible or likely to have the same exposure as the cases.
2. Were the circumstances and methods for determining exposure similar for cases and controls?
In a case control study determination of the exposure is critical. The exposure in the two groups should be identified by the same method. The identification should avoid any kind of bias, such as recall bias. Sometimes using objective data, such as medical records, or blinding the interviewer can help eliminate bias.
Key issues for Harm Studies:
|
What are the results?
How strong is the association between exposure and outcome?
* What is the risk ratio or odds ratio?
* Is there a dose-response relationship between exposure and outcome?
How precise was the estimate of the risk?
* What is the confidence interval for the relative risk or odds ratio?
Strength of inference:
For RCT or Prospective cohort studies: Relative Risk
|
Cases Outcome present |
Controls Outcome not present |
|
| Exposure Yes | a | b |
| Exposure No | c | d |
Relative Risk (RR) = a /(a + b) / c/(c + d)
is the risk of the outcome in the exposed group divided by the risk of the outcome in the unexposed group:
RR = (exposed outcome yes / all exposed) / (not exposed outcome yes / all not exposed)
Example: “RR of 3.0 means that the outcome occurs 3 times more often in those exposed versus unexposed.”
For case-control or retrospective studies: Odds Ratio
|
Cases Outcome present |
Controls Outcome not present |
|
| Exposure Yes | a | b |
| Exposure No | c | d |
Odds Ratio (OR) = (a / c) / (b / d)
is the odds of previous exposure in a case divided by the odds of exposure in a control patient:
OR = (exposed - outcome yes / not exposed - outcome yes) / (exposed - outcome no / not exposed - outcome no)
Example: “OR of 3.0 means that cases were 3 times more likely to have been exposed than were control patients.”
Confidence Intervals are a measure of the precision of the results of a study. For example, “36 [95% CI 27-51]“, a 95%CI range means that if you were to repeat the same clinical trial a hundred times you can be sure that 95% of the time the results would fall within the calculated range of 27-51. Wider intervals indicate lower precision; narrow intervals show greater precision.
Confounding Variable is one whose influence distorts the true relationship between a potential risk factor and the clinical outcome of interest.
Read more on odds ratios: The odds ratio Douglas G Altman & J Martin Bland BMJ 2000;320:1468 (27 May)
Watch more on odds ratios: Understanding odds ratio with Gordon Guyatt. (21 minutes.)
How can I apply the results to patient care?
Were the study subjects similar to your patients or population?
Is your patient so different from those included in the study that the results may not apply?
Was the follow-up sufficiently long?
Were study participants followed-up long enough for important harmful effects to be detected?
Is the exposure similar to what might occur in your patient?
Are there important differences in exposures (dose, duration, etc) for your patients?
What is the magnitude of the risk?
What level of baseline risk for the harm is amplified by the exposure studied?
Are there any benefits known to be associated with the exposure?
What is the balance between benefits and harms for patients like yours?
Source: Guyatt, G. Rennie, D. Meade, MO, Cook, DJ. Users’ Guide to Medical Literature: A Manual for Evidence-Based Clinical Practice, 2nd Edition 2008.
Print the harm article worksheet.
