APPRAISE: Diagnosis Part 4
Consider the following questions when evaluating the validity of a prognosis study.
For future reference, download the Prognosis Critical Appraisal Worksheet.
Was the sample of patients representative?
Patients groups should be clearly defined and representative of the spectrum of disease found in most practices. Failure to clearly define the patients who entered the study increases the risk that the sample is unrepresentative. To help you decide about the appropriateness of the sample, look for a clear description of which patients were included and excluded from a study. The way the sample was selected should be clearly specified, along with the objective criteria used to diagnose the patients with the disorder.
Knowledge Check: When evaluating whether a study’s patient sample is representative, which factor best indicates appropriate sampling?
A) The study clearly defines inclusion and exclusion criteria, reflecting the typical spectrum of the disease seen in practice.
B) The study includes only patients with severe forms of the disease to ensure results are statistically significant.
C) The study avoids describing how patients were selected to prevent bias in interpretation.
Were the patients sufficiently homogeneous with respect to prognostic factors?
Prognostic factors are characteristics of a particular patient that can be used to more accurately predict the course of a disease. These factors, which can be demographic (age, gender, race, etc.) or disease specific (e.g., stage of a tumor or disease) or comorbid (other conditions existing in the patient at the same time), can also help predict good or bad outcomes.
In comparing the prognosis of the 2 study groups, researchers should consider whether or not the patient’s clinical characteristics are similar. It may be that adjustments have to made based on prognostic factors to get a true picture of the clinical outcome. This may require clinical experience or knowledge of the underlying biology to determine if all relevant factors were considered.
Knowledge Check: Why is it important for study patients to be sufficiently homogeneous with respect to prognostic factors?
A) It minimizes differences in outcomes that could be due to variations in age, disease stage, or comorbidities.
B) It guarantees that no clinical experience is needed to interpret prognostic influences on outcomes.
C) It ensures that results can be easily generalized to all populations without adjustment.
Was the follow-up sufficiently complete?
Follow-up should be complete and all patients accounted for at the end of the study. Patients who are lost to follow-up may often suffer the adverse outcome of interest and therefore, if not accounted for, may bias the results of the study. Determining if the number of patients lost to follow up affects the validity depends on the proportion of patients lost and the proportion of patients suffering the adverse outcome.
Patients should be followed until they fully recover or one of the disease outcomes occur. The follow-up should be long enough to develop a valid picture of the extent of the outcome of interest. Follow-up should include at least 80% of participants until the occurrence of a major study end point or to the end of the study.
Knowledge Check: Which situation would most threaten the validity of a study’s follow-up?
A) The study follows 90% of patients to a major endpoint.
B) The study loses 25% of participants, many of whom experienced an adverse outcome of interest.
C) The study defines outcomes like “death” or “full recovery” with clear, objective criteria.
Were objective and unbiased outcome criteria used?
Some outcomes are clearly defined, such as death or full recovery. In between, can exist a wide range of outcomes that may be less clearly defined. Investigators should establish specific criteria that define each possible outcome of the disease and use these same criteria during patient follow-up. Investigators making judgments about the clinical outcomes may have to be “blinded” to the patient characteristics and prognostic factors in order to eliminate possible bias in their observations.
How likely are the outcomes over time?
- What are the event rates at different points in time?
- If event rates vary with time, are the results shown using a survival curve?
How precise are the estimates of likelihood?
- What is the confident interval for the principle event rate?
- How do confidence intervals change over time?