Chapter 7. Clinical Trials

• A clinical trial is the direct comparison of two or more treatment modalities in human groups.
•  Evidence-based medicine is the integration of current best evidence with clinical expertise, pathophysiological knowledge, and patient preferences to make health decisions.
• The practice of evidence-based medicine is encouraged because it may lead to more consistent and objective clinical decisions.
• A type I error occurs when a study finds a difference in effectiveness between the treatments being compared when in fact no difference exists.
• A type II error occurs when a study fails to find a difference in treatment effectiveness between the treatments being compared when in fact a difference does exist.
•  Statistical power is the ability of a study to detect a true difference between groups being compared.
• When treatments are assigned by randomization, probability alone determines assignment, rather than the personal preferences of either physicians or patients.
• When patients are unaware of their treatment assignment in a clinical trial, it is referred to as a single-blinded study. In a double-blinded study, neither the patients nor the treating physicians know individual treatment assignments.
• A ratio of either rates or risks can be used to compare the outcomes in the experimental and control groups.
•  Meta-analysis is a statistical integration of the results of several independent studies.
• A sensitivity analysis can be used to determine whether differences across studies may be explained by characteristics of the various populations studied.

In the office, the pediatrician performs a urinalysis, which reveals a normal microscopic examination; the dipstick examination of the urine is negative for blood, white blood cells, and bilirubin, but is 4+ positive for glucose and 1+ positive for ketones. The pediatrician expresses concern to the parents that the girl may have diabetes mellitus. A complete blood count and blood chemistries are sent to a local laboratory, and the doctor arranges for the patient and her parents to see a pediatric endocrinologist within the hour.

In the endocrinologist’s office, the results of the blood tests confirm that the girl has a markedly elevated blood glucose (565 mg/dL) and a slightly elevated blood urea nitrogen (24 mg/dL). Her serum bicarbonate is normal. The endocrinologist discusses the diagnosis of type I diabetes mellitus with the patient and her parents and asks them to speak with a nurse educator in the office to begin to learn about diabetes mellitus, the self-administration of insulin, dietary control, and glucose monitoring.

In the United States, the prevalence of diabetes mellitus is estimated to be between 2% and 4% of the population. The two basic types of diabetes mellitus are contrasted in Table 7–1. Type I diabetes is characterized by a markedly decreased or absence of production of insulin, possibly due to an autoimmune destruction of the pancreatic beta cells. Without insulin to facilitate the entry of glucose into cells, plasma glucose levels rise. When the level of plasma glucose exceeds an amount that can be reabsorbed by the kidney (approximately 180 mg/dL), the resulting glucosuria obligates an osmotic diuresis. The concentration of glucose in the urine necessitates the elimination of excessive water in the urine. Excess glucose is also lost in the urine.

The resulting loss of water and calories leads to the cardinal signs of diabetes: increased urination, thirst, and appetite accompanied by weight loss and dehydration. Prior to the discovery of insulin, patients diagnosed with type I diabetes mellitus died of their disease within weeks. After the purification of insulin from animals, subcutaneous injections of insulin allowed patients with type I diabetes mellitus to survive. Today, type I diabetic patients inject synthetically manufactured human insulin. The development of machines that measure plasma glucose levels from small samples of blood and automatically adjust insulin levels has facilitated the management of diabetes. Although the discovery of insulin and the development of glucose monitoring virtually eliminated acute deaths due to insulin deficiency, persons with type I diabetes continued to suffer the long-term complications of the disease—premature atherosclerotic cardiovascular and peripheral vascular disease resulting in heart attacks, stroke, and amputations; retinopathy (ie, damage to the arterial vessels that can lead to blindness); and nephropathy (ie, damage to the microvasculature of the kidney ultimately leading to kidney failure).

During the 1980s, the cause of these long-term complications was debated within the medical community. One group argued that if blood glucose levels were controlled more tightly, a corresponding decrease in the long-term complications of diabetes would result. Others postulated that other physiologic abnormalities associated with diabetic disease contribute to the long-term vascular complications. This debate is of critical importance to our patient, a teenager with newly diagnosed diabetes. Conventional twice-a-day injections of regular insulin in combination with a longer acting insulin would control her glucose well enough to avoid complications associated with very high blood glucose levels. However, more intensive therapy with four injections of insulin per day and more frequent monitoring of blood glucose levels would more closely approximate secretion of insulin by the normal pancreas, and theoretically might reduce the risk of long-term vascular complications. At the same time, more intensive therapy could also result in more frequent episodes of low blood glucose levels (hypoglycemia), which, although it is rare, can result in coma. The pediatric endocrinologist might also be concerned that frequent episodes of hypoglycemia would interfere with the normal growth of an adolescent.

Although there are theoretical reasons why a physician would want to maintain normal blood glucose levels in our patient, there are also hazards associated with the intensive therapy described above. To determine the safety and efficacy of new therapies, researchers must conduct a clinical trial. A randomized, controlled clinical trial is a study design in which one treatment is compared directly with another treatment to determine which of the two treatments would be of greatest benefit.

In the Patient Profile, the clinician is faced with a decision concerning treatment: which of two possible regimens to use to control blood glucose levels. Hippocrates’ axiom—”First, do no harm”—is a time-honored warning to clinicians contemplating medical intervention. “One must attend in medical practice not primarily to plausible theories,” Hippocrates wrote, “but to experience combined with reason.” In other words, a treatment plan should be reasonable in theory but should also be tested experientially. More than 2000 years ago, Hippocrates noted that judging the benefits of a treatment should be based on the treatment’s effects on patients.

In modern medicine, a randomized, controlled clinical trial of one therapy versus another is the accepted gold standard by which the usefulness of a treatment is determined. To practice modern medicine and select appropriate therapy, it is necessary to understand the design and conduct of clinical trials.

This important method of evaluating treatments utilizes two types of knowledge, both alluded to by Hippocrates: reason and experience. “Reasonable” treatment is treatment suggested by a knowledge of basic biomedical science. For the clinical situation presented in the Patient Profile, a knowledge of the pathophysiology of diabetes mellitus and the long-term complications of this disease would indicate the need for tight control of blood glucose levels and the avoidance of any possible long-term complications of diabetes. However, as stated by Hippocrates, the clinician cannot base therapeutic decisions on theory alone; the dictates of reason must be submitted to the test of experience.

Clinicians employ two types of experience to assess a treatment regimen—their own personal experience and the written or orally conveyed experience of their colleagues. Written experience may take the form of a report of a single case, a series of cases, or a comparison of one treatment with another.

The use of evidence-based medicine is encouraged for several reasons:

• 1. It can provide an efficient framework for accessing and systematically interpreting the medical literature.
• 2. It can provide an objective basis for selecting a strategy for disease management, thereby enhancing clinical outcomes.
• 3. It can serve as a rational basis for modifying the practice of disease management by individual clinicians.
• 4. It can provide a coordinated plan for disease management when care is delivered by a multidisciplinary team.
• 5. It can identify gaps in current knowledge and thereby suggest prioritized areas for further investigation.
• 6. It can suggest areas in which flexibility in disease management is justified by a lack of evidence to support a particular approach.
• 7. It can facilitate assessment of the process of disease management, suggesting opportunities for improved quality and efficiency.

There are a number of barriers to the practice of evidence-based medicine. Many clinicians have limited training in formal assessment of research studies. Even if clinicians have the necessary skills, they may have little time to search and evaluate the literature. Even if time is available, there are many important issues about disease management for which there is a paucity of evidence to appraise.

Considering these obstacles, it may be surprising that evidence-based medicine has progressed so rapidly in recent years. In part, changes in the organization, financing, and delivery of health care have encouraged a greater focus on clinical outcomes and the cost effectiveness of different management strategies. In addition, easier and faster accessibility to the relevant medical literature is possible through automated searches of electronic databases. Also, the Internet has facilitated the rapid and widespread sharing of results from critical appraisals.

Although far from universally adopted, evidence-based medicine offers the promise of improved management of disease. A prerequisite to the practice of evidence-based medicine is the ability to critically assess clinical studies. The following sections focus on the basic elements of a clinical trial. At the conclusion of this chapter, the principles of combining information across clinical trials are discussed.

As in a laboratory experiment, the first step in performing a clinical trial is to formulate the major research question. This question, usually referred to as the hypothesis, is refined to determine important study parameters, for example, the types of interventions to be compared, the nature of the outcomes to be assessed, the number of subjects in each treatment group, and the eligibility requirements for enrollment.

The parameter that is measured that will provide the answer to the most important question of the clinical trial is the primary end point. When determining the primary end point, clinical researchers must consider the following questions:

• 1. Which end points are most important clinically?
• 2. Which end points can be measured in a reasonably unbiased manner?
• 3. What practical constraints exist, such as population size, financial resources of the research study, and ability to follow patients on a long-term basis?

Researchers in the United States and Canada designed a clinical trial to address the following question: Is intensive therapy, including more frequent injections of insulin and more frequent monitoring of blood glucose levels, “superior” to standard therapy for diabetes mellitus?

To assess treatment efficacy, more than one end point can be measured. Types of end points include measures of quality of life, length of survival, percentage of patients surviving, complication rate, and intermediate end points that are predictive of survival or quality of life (Table 7–2). In assessing “superiority” for the treatment of diabetes mellitus, possible outcome measures include the following:

• 1. The percentage of patients surviving at a specified time following the initiation of treatment
• 2. A patient’s ability to maintain an active life-style
• 3. The risk of having a major complication from treatment
• 4. The risk of experiencing one of the vascular events associated with diabetes mellitus
• 5. Measurement of hemoglobin A1-C (HgbA1C), which is an indicator of long-term blood glucose levels
• 6. Blood glucose levels at specific points in time.

Examples of two of these questions restated symbolically as hypotheses follow:

where H0 is the so-called null hypothesis and π0 and π1 are the percentages of persons in the standard therapy or intensive therapy groups, respectively, who develop diabetic retinopathy during the 5-year period following entry into the study.

where μ0 and μ1 are the mean HgbA1C levels at 1 year following entry into the trial for persons in the standard therapy or intensive therapy groups, respectively.

Note that these hypotheses are stated in the null form—there is no difference between the treatment groups regarding the specified end point. The purpose of the clinical trial is to test whether the observed outcomes are consistent with these null hypotheses. If the observed data are not consistent with the null hypothesis, the null hypothesis (H0) is rejected in favor of an alternative hypothesis (HA) such as the following:

where π0 and π1 again are the respective percentages of persons developing diabetic retinopathy during the 5-year period following entry into the study in the standard therapy and intensive therapy groups. This alternative hypothesis states that the two treatments will differ with respect to the development of diabetic retinopathy. Similarly, an alternative hypothesis for the HgbA1C outcome could be stated as follows:

where μ0 and μ1 again are the respective mean HgbA1c levels for the standard therapy and intensive therapy groups. In other words, this alternative hypothesis states that the two treatments will differ with respect to mean HgbA1c levels achieved in patients after treatment.

The most important end points for any therapy for a potentially fatal disease would be survival, or survival accounting for quality of life. However, researchers in this trial of therapy for diabetes chose diabetic retinopathy as the primary end point of the study for several reasons:

• 1. Death from diabetes often occurs decades after the onset of the disease. If death were the primary end point, the trial could not have been completed in a timely manner.
• 2.Retinopathy is a serious complication of diabetes. Vascular damage in one organ correlates with vascular damage at other sites of the body. Since vascular disease is the predominant physiologic process resulting in morbidity and mortality from the disease, the researchers felt that it was a reasonable choice as a primary end point.
• 3. The eye provides a unique portal through which damage to the vascular system can be visualized noninvasively. Furthermore, the measurement of the progression of retinopathy could be standardized and evaluated by taking photographs of each patient’s retina at various points in time; moreover, researchers could evaluate the amount of retinal damage as documented in the photographs without knowledge of the treatment group to which each patient had been randomized.
• 4. Retinopathy, an intermediate end point, is thought to be predictive of two important final end points: survival and quality of life.

The number of subjects to be enrolled in a clinical trial—the sample size—must be determined in relation to the primary research end point. At the conclusion of any experiment, data are analyzed and a statistical decision is made to reject or accept the null hypothesis. This decision is based on probabilities, and unfortunately may be correct or incorrect. The relationship between the possible results of the diabetes therapy trial and the “truth” is shown in Figure 7–1. For the purposes of this discussion, “truth” can be thought of as the results of the intervention, if applied correctly to the entire universe of patients with the clinical condition under study. A clinical trial can be considered a sample of the “truth.” Using a sample of a population, the hope is to make valid inferences about the entire population, but since only a sample can be evaluated, the risk of inaccurate conclusions exists.

In cells A and D of Figure 7–1, the results of the study agree with the “truth.” In cells B and C, however, the study results do not agree with the “truth”—errors are made. These two types of error differ in their origin and implications.

Falsely positive or falsely negative studies can occur because of faulty methodology, chance occurrences, or both. Whereas methodologic error can be minimized by careful attention to study design, errors due to chance can never be completely eliminated. Such errors, however, can be estimated. The notation used to denote the likelihood of a type I error—that the observed difference between groups is not a true difference but is due instead to chance—is the alpha level. Conversely, the notation used to denote the likelihood of a type II error—that the study did not find a difference when there actually is one—is the beta level. Researchers specify the alpha and beta levels when planning a study. The alpha level is specified commonly as 0.05, which means that the investigator is willing to accept a 5% risk of committing a type I error (falsely concluding that the groups differ when in reality they do not). The investigator must also specify beforehand the beta level, or risk of committing a type II error. Often a beta level of 0.20 is considered adequate—in other words a one in five chance of missing a true difference between the groups is allowed.

Once the alpha and beta levels have been specified, the research team must specify another extremely important study parameter before determining sample size—the magnitude of the difference in outcome between treatment groups that the study will be designed to detect. This difference between the treatments under comparison is of great importance and should be selected on the basis of clinical information. In deciding on the level of outcome difference worthy of detection, the investigator might consider one or more of the following questions:

• 1. What difference in outcome would be important to clinicians treating this type of patient?
• 2. What difference would be meaningful to a patient who may suffer the consequences of the disease?
• 3. What difference in outcome would justify use of the more effective treatment in spite of greater expense or greater side effects?

Formulas for the determination of sample size and illustrative calculations are provided in Appendix A. Suffice it to say here that all three factors just mentioned (acceptable levels of type I and type II errors and the expected magnitude of difference in outcome between groups) are inversely related to the required sample size (Table 7–3). That is, if only a 1%, rather than the more commonly accepted 5%, chance of committing a type I error can be tolerated, the sample size must be increased. Similarly, a decrease in the acceptable level of type II error (enhanced statistical power) necessitates studying more subjects.

As the expected magnitude of difference in outcome (eg, proportion of subjects developing retinopathy) between treatment groups decreases, a larger sample size is required to detect the difference. In contrast, as the variability of the outcome (eg, the standard deviation of HgbA1c level) diminishes, fewer subjects are required to demonstrate a difference in outcome between the groups.

A central tenet of the clinical trial is that patients should be assigned to treatment groups by a method that maximizes the probability that the two groups will be similar in background characteristics that may influence either the response to therapy or the primary outcome measure.

The assignment to a treatment group for each patient is independent of (not influenced by) the assignment for all other patients. When there are two possible treatment assignments with equal sample sizes (as in the diabetes trial), each patient has a 50% chance of being assigned to either treatment. Such an assignment could be decided by a coin toss: if the coin comes up heads, the patient is assigned to standard therapy; if the coin comes up tails, the patient is assigned to intensive therapy. Of course, tossing a coin is a rather inelegant way to assign patients to treatment groups, so many investigators use more sophisticated devices, such as tables of random numbers or computer-generated random assignments. The basic design of a randomized controlled clinical trial is illustrated in Figure 7–2, using the diabetes therapy trial study as an example.

As recently as 50 years ago, the primary means of evaluating the benefit of a new treatment was to treat a series of patients with the new method and then compare the outcome with that observed in the past for a group of patients who received the standard therapy. The patients who received the standard therapy are called nonconcurrent or historical controls. Consider how such a study might have been performed to address the question concerning treatment of diabetes. If researchers wanted to test the hypothesis concerning the clinical efficacy of intensive therapy (experimental treatment) versus standard therapy (or control treatment), a group of patients would be given intensive therapy and the proportion of patients who develop retinopathy in a specified period of time would then be compared with the proportion of patients who develop retinopathy among a group of patients treated at an earlier time with standard insulin therapy (Figure 7–3). There are several inherent problems with this approach:

• 1. The diagnostic criteria for progression of retinopathy may change over time.
• 2. The techniques used to measure retinopathy may change over the time period of the study.
• 3. Additional treatment modalities, such as advances in knowledge regarding dietary control, could become available over time and for ethical reasons would have to be employed on the patients during the second part of the study.
• 4. Most importantly, the patients who presented with diabetes during the time when patients were being treated with standard insulin therapy may not have been similar in prognostic characteristics (sex, age, socioeconomic status) to the patients who present during the intensive treatment period.

If concurrent controls are necessary to compare treatment groups, a basic question arises: How should patients be assigned to each treatment group?

Physicians or patients could choose which treatment a patient will be given. This technique, however, is seriously flawed. Although health care providers strive to be objective decision makers, they are empathetic to the needs of their patients and often have opinions concerning the efficacy of a treatment prior to the results of clinical trials. Patients—who of course have a direct personal interest in any treatment result—would very likely choose a treatment based on their assessment or expectation of its efficacy.

To avoid unfair comparisons that may result from these preferences of patients and physicians, the assignment to treatment group should therefore be determined by chance, independently of the wishes of clinicians and patients. The purpose of randomization is to achieve “equality” of baseline characteristics of treatment groups, so that the comparison of treatments is considered fair. To assess the equality of the treatment groups, demographic and prognostic factors may be compared between groups. If patients have been randomly allocated, it is expected that the groups will be similar in demographic and prognostic features.

A comparison of treatment groups after randomization from the diabetes therapy trial is given in Table 7–4. Note that two groups of patients were studied in this trial. The primary prevention group was defined by its complete lack of retinopathy at entry into the trial. The secondary prevention group included patients who already had signs of diabetic retinopathy at time of entry into the trial. The separation of the trial participants into groups is a technique known as stratification. The researchers chose to stratify by the presence or absence of retinopathy because they felt that this variable was of great importance in assessing the effect of treatment. By stratifying, the researchers ensured that they would have randomization within each of these two important but different groups.

As can be seen in Table 7–4, the two treatment groups are remarkably similar with regard to age, sex, duration of disease, insulin dose, glycosylated hemoglobin, and mean blood glucose level, and in regard to other characteristics known to be risk factors for vascular disease for subjects in both the primary and secondary prevention groups.

If the object is to ensure that similar numbers of patients with certain important prognostic characteristics are included in each treatment group, those characteristics can be accounted for in the randomization process. Suppose researchers wanted to guarantee that at the end of the diabetes therapy study there would be equal percentages of males and females in each treatment group. The order in which the two treatments were assigned to groups of four patients of the same sex could be random; the first four male patients would be assigned to either the standard or the intensive therapy group and the next four males would be assigned to the other group. Females would be assigned in the same manner. At the end of the study there would be an equal number of male and female patients in each treatment group. Assignment of patients in this manner is termed block randomization.The purpose of randomization in blocks of patients is to protect against imbalanced treatment assignment (due to “luck of the draw”) with respect to prognostically important patient subgroups.

In the year 1801, Haygarth reported the results of what may have been the first placebo-controlled trial. A popular treatment for many diseases at that time was to apply metal rods, known as Perkins tractors, to the body and thus relieve symptoms through a supposed electromagnetic influence of the metal. Haygarth treated five patients with imitation wooden tractors and found that four gained relief. The following day, he used the metal tractors on the same five patients and obtained identical results: relief in four of the five subjects. In describing the results of his experiment, Haygarth quoted James Lind, who is credited with performing the first clinical trial: “An important lesson in physic is here to be learnt, viz., the wonderful and powerful influence of the passions of the mind on the state and disorders of the body. This is too often overlooked in the cure of diseases.”

The influence of treatment of any kind on the perceptions patients have about their illness cannot be forgotten when designing a clinical trial. This concept is most important when the outcome measure is subjective. The patient’s desires may also influence decisions made by clinicians subsequent to the initial randomization in a clinical trial. An additional factor that could potentially lead to differential treatment of groups within a trial is the clinician’s decision-making process during a clinical trial. For example, will a clinician who has certain beliefs concerning a treatment’s efficacy be likely to discontinue a therapy believed to be inferior in favor of the alternative therapy?

To account for the placebo effect and to reduce the introduction of bias due to the conceptions of patients and clinicians, studies may be conducted in a blinded fashion (Table 7–5). “Blinding” means that the treatment assignment is not known to certain persons.

In the diabetes therapy trial, neither patients nor their physicians were blinded to their treatment group. Blinding was not possible because the patients and physicians are intimately involved in regulation of insulin dose on either arm of the trial. The researchers did blind the physicians who evaluated the photographs of the retina when assessing subjects for retinopathy. This was done to eliminate the possibility of observation bias.

Although blinding of both the patients and the treating physicians is desirable, there are trials, such as the diabetes therapy trial, that cannot be conducted in a blinded fashion because the treatments are so obviously different that it is not feasible to keep the assignment secret. Whenever possible, however, blinding should be employed, because lack of blinding could influence perceptions of outcome and reduce the confidence in study results.

The investigator who contemplates entering a patient into a randomized clinical trial is faced with several ethical dilemmas. First, is the method of the randomized clinical trial ethically acceptable? One of the most important ethical tenets in medicine is that the patient’s welfare is of primary concern, and a caregiver should prescribe the optimal treatment for a patient. It could be argued that even if a clinician has only a hunch or feeling that one treatment is superior, the patient should be offered that treatment. Therefore randomization between two treatments might be considered to be unethical. Given the seriousness and possible side effects of medical interventions, however, the axiom “first, do no harm” must always be borne in mind. The history of medicine is replete with examples of treatments now known to be either of no benefit or actually harmful. The clinical trial is considered to be the best method available to determine the benefits and potential harm of treatment regimens.

If the clinical trial method is accepted as appropriate, a decision must be made concerning how to perform trials as ethically as possible. The following is a list of guidelines for medical professionals who are conducting clinical trials:

• 1. None of the treatment options included in a randomized trial should be known to be inferior to another treatment option based on previous randomized studies, and if a standard treatment regimen exists, it should be used as the control.
• 2. The trial should address a question that is of clinical importance and seek to answer the question in a way that will be useful for future patients.
• 3. Patients should be told that they are part of a clinical experiment and should be informed in understandable language about all treatment options, the risks and benefits of participation, and the nature of randomization. The patient who then agrees to participate is said to have given informed consent, which implies that the patient freely chooses to be included in the trial.
• 4. The investigators undertaking the trial should be able to recruit in a timely manner the number of patients needed to meet the required sample size.

Relatively few physicians design clinical trials, and a limited number enter patients into clinical trials—but all clinicians read published accounts of clinical trials and use the results to guide the treatment of patients. A checklist of questions to help the physician interpret and evaluate clinical trials appears in Table 7–6.

Design Issues

The null hypothesis—and what would constitute a meaningful difference in outcome—should be stated in the methods section of a reported clinical trial. In the diabetes therapy trial, the primary outcome and the difference that was considered meaningful were clearly stated prior to beginning the trial. Trials should be designed to test one specific hypothesis or only a few hypotheses, and these should be evident to the reader.

The characteristics of the study population are particularly important when assessing the relevance of a specific trial to an individual practitioner’s patients. In the diabetes therapy trial, the eligibility and exclusion criteria are clearly stated, as summarized in Table 7–7. Summary data about sex, age, and medical histories of the participants appear in Table 7–4. The combination of the entry criteria and the demographic characteristics of the study’s entrants provides the reader with an adequate description of the study group. With this information, the reader is better able to judge whether the results of this study are applicable to a particular patient. Patients are often excluded from a trial for pragmatic reasons, such as inability to comply with treatment or lack of fluency in the language of the investigators. These exclusions, however, may limit the ability to generalize study findings to other patient groups.

Once randomized to a particular treatment regimen, a patient may adhere (comply) or elect not to follow the prescribed regimen. Possible reasons for noncompliance are listed in Table 7–8. The investigator cannot force participants to comply, since such coercion would violate the rights of subjects to participate of their own free choice.

There are several possible ways, however, to increase compliance in a clinical trial (Table 7–9). The investigator may be able to select subjects who can be expected to be compliant. Motivation to participate is likely to be enhanced if the patients perceive themselves to be at high risk of an adverse health consequence. In the diabetes therapy trial, all enrolled subjects had an illness that put them at risk for the long-term vascular complications of diabetes. Also, participants are apt to be motivated to comply if the treatments offered may reduce the need for painful or debilitating therapy. In the diabetes therapy trial, for instance, patients may have hoped that more intensive therapy, or participation in a clinical trial with stringent guidelines for treatment, would help reduce the long-term complications of disease.

In nonurgent situations, the investigator may be able to assess probable compliance before randomization is performed. To monitor compliance the investigator may ask eligible participants to take either an active or an inert medication. This pretest interval is referred to as a “run-in” test period. Individuals who show a likelihood of compliance are randomized to the treatments of interest, and those likely to be noncompliant are excluded from the trial. Other strategies to increase compliance include providing incentives for participation or maintaining frequent contact with subjects. Compliance also is likely to be enhanced by keeping the duration of the intervention as brief as possible.

Regardless of how carefully a clinical trial is designed, it is likely that some subjects will not adhere to the treatment regimen. The extent to which participants actually comply can be assessed by various approaches. Personal reports by subjects and family members provide a simple but possibly unreliable basis on which to determine compliance. In drug studies, a traditional approach to assessing compliance is to count the number of unused pills at regular intervals. However, because pills can disappear for reasons other than ingestion by the subject, pill counts provide suggestive but not definitive evidence of compliance. The most conclusive evidence of compliance with a drug regimen is likely to be obtained by measurement of the drug (or a metabolite) in the subject’s blood or urine. However, even this type of biological assay has limited utility, with the most obvious constraints being cost, inconvenience to subjects, and the difficulty of collecting specimens from some individuals. Moreover, long-term compliance typically cannot be assessed by measurements of such specimens, as the presence of most drugs is detectable in blood or urine for no more than a few days. The diabetes therapy trial was fortunate to be able to measure glycosylated hemoglobin, a long-term measure of blood glucose control, as a secondary end point. Although measurement of glycosylated hemoglobin levels is not a direct measure of compliance, among the two treatment groups it provided an independent measure of effect of treatment that was expected to differ if the treatments did differentially affect subjects’ plasma glucose levels.

Despite the difficulties inherent in assessing compliance, it is important to estimate the extent to which subjects adhere to the assigned regimens. The ability of a study to identify a true effect of treatment (statistical power) may be diminished if a substantial proportion of the participants do not comply with the assigned treatment. That is, the observed difference in outcomes between the study groups may be reduced because of noncompliance. Accordingly, it may be necessary to include a larger initial sample size to compensate for the loss of discriminatory power. As will be emphasized later in this chapter, it is important to include all randomized patients in the main analysis of a clinical trial. Therefore, every effort should be made to determine the outcomes of both compliant and noncompliant subjects.

Analysis of Results

Loss of some patients to follow-up is likely to occur in any clinical trial. The greater the number of patients who are lost and the less information available about them, the less confidence can be placed in the results of the trial. In the analysis of results from a clinical trial, patients should be left in the treatment group originally assigned by the study (intention to treat) even if they received one of the other treatments after the original treatment regimen failed. In the diabetes therapy study 95 women assigned to the standard therapy group received more intensive therapy during a pregnancy. In the analysis, these women remained in the standard therapy group. In another smaller subgroup of patients, researchers discontinued intensive therapy and resumed standard therapy. However, these patients remained in the intensive therapy group during analyses. This may seem counterintuitive, as these patients actually received some of both treatment regimens. The purpose of this trial, however, was to help clinicians determine the best treatment at the time of initial presentation for entry into the trial. Subsequent treatment decisions may include the alternative treatment, but those subsequent decisions have no bearing on the original clinical question posed by the trial and thus are not pertinent to group assignment. Design of the diabetes therapy trial with consideration of intention to treat is illustrated schematically in Figure 7–4.

All participants who are randomized should be included in the analysis of a clinical trial. Selective removal of subjects from the comparison of outcomes, even if it seems justified for pragmatic reasons, may lead to erroneous conclusions. Consider, for example, the question of whether to include noncompliers in the analysis. Because these individuals did not receive the assigned treatments as intended, it may seem illogical to leave them in the analysis. It has been shown, however, that noncompliers tend to have worse outcomes than compliers, regardless of their treatment assignment. If the treatment assignment affects the level of compliance, then excluding noncompliance from the analysis can produce a misleading result.

Removal of noncompliers from the analysis may also limit the ability to generalize study findings to clinical practice. In recommending treatment to a particular patient, the physician must consider the possibility that the treatment will not be completed as intended. The essential question of a clinical trial is whether a treatment should be offered at a particular point in time. Therefore the relevant information on treatment benefit is the outcome among all patients who were offered the treatment, rather than just those who completed it.

A well-reported clinical trial should contain enough of the primary data to enable the reader (1) to compare the main outcome measure between the treatment groups and (2) to perform basic statistical tests to determine whether it is reasonable to exclude chance variation as a cause of observed differences between the compared groups. For the clinical trial, it is useful to review three very common types of comparisons: the comparison of two risks, the comparison of time to an event (survival analysis), and the comparison of two means.

Many outcomes from a clinical trial are yes/no outcomes (eg, death or no death, cure or no cure, recurrence or no recurrence) and therefore can be displayed in simple tabular format. Whether a patient developed retinopathy in the primary prevention group in the diabetes therapy trial is presented in Table 7–10.

The incidence rate of developing retinopathy in each arm of the primary treatment group was determined by dividing the number of subjects who developed retinopathy by the number of person-years of follow-up. Person-years are calculated by summing the years that each subject is in the study prior to the development of retinopathy. This allows all subjects to be included in the results of a study, regardless of how long each subject was enrolled. This issue is common to the clinical trial, as recruitment of subjects into a trial often occurs over several years. The incidence rate (IR) of retinopathy for the standard therapy group was calculated to be 4.7 per 100 patient-years of follow-up, and 1.2 per 100 patient-years of follow-up for the intensive therapy group. There are several ways to compare these two rates.

One way to compare the rates is to calculate the percentage of incidence of retinopathy that would be avoided if intensive therapy were used instead of standard therapy. This percentage is known as the percentage rate reduction and is calculated as follows:

If the percentage rate reduction = 0, there is no reduction in incidence rate attributable to the new therapy, and the treatments are judged to be equivalent. The further the percentage rate reduction is from zero, the greater the difference between the two groups. For the diabetes therapy trial, the percentage of retinopathy that could have been prevented by patients using intensive therapy rather than standard therapy is calculated as follows:

That is, almost three fourths of the retinopathy that occurred in the standard therapy group could have been avoided if those patients had been treated with intensive therapy. The value of 74% is known as a point estimate because it is the single value along the scale from 0 to 100% that is most consistent with the results of the trial.

A useful method to gauge the precision of any point estimate is to calculate the 95% confidence intervals for the estimate. If the clinical trial were repeated many times, the values falling between the upper and lower bounds of the 95% confidence interval would include the true point estimate value 95% of the time. If the 95% confidence interval of the percentage risk reduction includes 0, the data are consistent with the null hypothesis, and the difference between the groups is not statistically significant at an alpha level of 0.05. If the 95% confidence interval does not include 0, the difference is statistically different at an alpha level of 0.05.

The approximate 95% confidence interval (CI) for the percentage rate reduction in the diabetes therapy trial calculated above is 60–83%. Since the interval does not include 0, this decreased rate is considered statistically significant at an alpha level of 0.05. This means that given the observed data, if the trial were repeated often, 95% of the time the percentage rate reduction would fall between 60% and 83%. The 95% confidence interval for the percentage rate reduction for retinopathy is illustrated schematically in Figure 7–5.

That is, the rate of developing retinopathy in the intensive treatment group is about one quarter that of the standard therapy group. The value of 0.26 is another example of a point estimate, because it is the single value along the rate ratio scale most consistent with the observed data. Similar to the percentage rate reduction, 95% confidence limits for the RR point estimate can also be calculated. If the 95% confidence interval includes the null value of 1.0, there is no statistical difference between rates in the two groups, and the null hypothesis would be accepted. If the 95% confidence interval does not include 1.0, the difference in rates between the two groups would be considered statistically significant at the 5% level. The point estimate and 95% confidence interval for the rate ratio of retinopathy in the diabetes therapy trial is illustrated schematically in Figure 7–6. Note that the point estimate of the RR does not lie at the midpoint of the 95% confidence interval. The asymmetry of the confidence interval occurs because the distribution of possible values of the RR is skewed toward the right (ie, all the values corresponding to a benefit for intensive therapy are compressed into the range of zero to 1, whereas the values corresponding to a benefit for standard therapy are spread from 1 to positive infinity).

The RR is a simple, easily understood method to evaluate results of clinical trials. For time-to-event data, however, survival analysis has several advantages over the RR. The survival curve, as described in Chapter 2: Epidemiologic Measures, is a graphic presentation of time-to-event data. Since it graphically depicts events as they occur over time, the survival curve provides information on the rapidity with which events occur. Furthermore, the survival curve can make use of data from patients who are followed for varying lengths of time. For the diabetes therapy trial, the cumulative risk of retinopathy was plotted over time (Figure 7–7). Although this figure is not a display of survival (life versus death), it does represent “survival” without the occurrence of the event of interest, in this case retinopathy. A patient who is followed for only 3 years provides useful information on risk of developing retinopathy for that period of time, but would provide no information pertinent to a comparison beyond 3 years. Survival analysis also allows the median survival duration (eg, time to retinopathy development) as well as the percentage of survivors (eg, persons without retinopathy) to be estimated at any time along the curve.

Time since first treatment is depicted along the horizontal axis, and the percentage of patients with retinopathy is displayed on the vertical axis. At the time of initial treatment (years = 0), 0% of the patients in each group have developed retinopathy (100% have survived without retinopathy). As time from treatment progresses, the percentage of patients who are diagnosed with retinopathy increases, although more rapidly in the control group. At the end of 9 years of follow-up, 14% of the patients treated with intensive therapy have been diagnosed with retinopathy compared with 55% of the standard therapy subjects.

The survival curves could be used to estimate the relative risk of being diagnosed with retinopathy at any point in time. For example, the intensive to conventional group relative risk of retinopathy at 5 years is as follows:

This relative risk indicates that the intensive therapy subjects have about 60% less risk than the conventional therapy subjects of developing retinopathy within 5 years. Alternatively, the median time to development of retinopathy for the two groups can be estimated and contrasted. The median time is the point at which half of an initial study group remains free of the occurrence of interest. In this example, the estimated median time to development of retinopathy for the standard therapy group is 8.5 years. The median time to development of retinopathy for the intensive therapy group has not been reached after 9 years of follow-up. That is, patients treated with intensive therapy are developing retinopathy at a slower rate than patients treated with standard therapy.

Several tests of significance can be used to compare survival curves (see Dawson and Trapp, 2004). As is demonstrated by the diabetes trial data, although this type of analysis is called survival analysis, it is not limited to the analysis of deaths. Any event that occurs over time (time to disease recurrence, time to return to work, etc) can be compared in this fashion.

Another comparison frequently used in clinical trials is the comparison of means. In the diabetes therapy trial, blood glucose levels were measured at the onset of the study and at regular intervals thereafter. In Table 7–4, one of the baseline characteristics compared between the standard therapy and intensive therapy groups is mean blood glucose level. It is possible to distribute patients into several discrete categories based on blood glucose level (eg, < 140, 140–179, 180–239, 240 or greater), but that would involve a loss of useful information about the actual observed glucose measurements. Instead, the mean or average blood glucose level can be compared. The null and alternative hypotheses for this comparison would be stated as follows:

A test of the equality of two means can be accomplished by performing a t test as follows:

where x1 and x2 are the observed mean blood glucose levels of the standard and intensive treatment groups, respectively, sp is an estimate of the pooled variance of the two means, and n1 and n2 are the sample sizes for each group. To illustrate the use of a t test, the observed mean blood glucose levels in patients treated with intensive and conventional therapy can be compared as follows:

A t statistic of 22.8 for this sample size corresponds to a p value of < 0.0005. In other words, there is less than a 0.05% chance that a difference in means as large as that observed with these sample sizes could have occurred by sampling variability alone. Accordingly, the null hypothesis of no difference between the means is rejected. If a study concludes that no difference exists between treatment regimens, the amount of difference the authors thought was important should be specified, as well as the likelihood that the study did not find a difference due to chance alone. The likelihood that a negative result is due to chance is the beta error; 1 minus the beta error (1 – β) is the statistical power (see Sample Size Determination).

Meta-Analysis

In the preceding section, the analysis of a single clinical trial was discussed. Rarely is a single study capable of providing the definitive answer to a clinical question. The diabetes therapy trial discussed in this chapter is one of the most important contributions to the evidence on the value of intensive treatment in reducing the complications of diabetes mellitus. The strengths of the study included its comparatively large sample size and long duration of follow-up. The substantial reduction in the risk of complications with intensive therapy suggests that a clinically meaningful benefit can be obtained and it was shown that this was unlikely to have occurred by chance alone.

However, the tight eligibility requirements for this study produced a study population that was relatively young, healthy, and had few minority participants. Whether these results apply to patients who are older, sicker, or members of minority groups cannot be determined from this study. To assess whether the results of this clinical trial are applicable to patients who did not meet the eligibility requirements of this study, it is useful to consider whether other clinical trials have been conducted on this topic. If other clinical trials exist, they may have included patients with characteristics different from those in this investigation. Treatment effects then can be compared across these studies. If the results of the various studies show a consistent pattern of results, then it is possible to be more confident that the benefits of intensive therapy are not confined to a narrow subset of patients.

A well-conducted meta-analysis offers several advantages over other types of reviews. First, a meta-analysis allows the direct presentation of the data on which the summary conclusions are made. Accordingly, the results of a meta-analysis are likely to be more data driven and objective than in other types of reviews. Second, the statistical summation in a meta-analysis produces a quantitative outcome measure and leads to precise estimates of effect. Third, if outcomes vary across studies, it may be possible to develop explanations for the pattern of variation in results, thereby gaining greater insight into the clinical question.

In essence, a meta-analysis is a study of other studies. As with any other type of investigation, a meta-analysis should be planned in advance and should follow a research protocol. The research plan should specify the hypotheses, the sampling strategy, the inclusion criteria, the data to be collected, and the approach to analysis of the information. At each of these steps, strategic decisions must be made that will impact the final product.

To illustrate this point, consider the seemingly straightforward issue of selecting the studies for inclusion. If interest is centered on clinical trials concerning intensive therapy for diabetes mellitus, a search of the published literature on this topic seems to be the logical sampling strategy. Because not all studies are published, the published literature may not yield a fair representation of all available information on the topic. It has been shown that studies with statistically significant findings are more likely to be published than studies with negative results. Also, studies sponsored by governments or nonprofit organizations are more likely to be published than those sponsored by the private sector (presumably because of the desire to protect proprietary information). Additional sampling distortion can result from the fact that papers published in certain influential journals are more likely to be cited, and therefore are easier to identify than works published elsewhere. Also, some studies result in multiple publications, which makes them easier to detect. In some instances, it may not even be possible for the meta-analyst to determine that two separate published papers arise from the same source clinical trial population. For pragmatic reasons, a meta-analysis may be confined to English language publications, which selects for more widely read journals and may result in an overrepresentation of studies with positive results.

Thus, the published literature on a topic may selectively exclude information that would affect the ultimate conclusion. This potential source of error is sometimes referred to as publication bias. Attempts can be made to identify and include unpublished studies in a meta-analysis, but there are obvious difficulties in locating such information and securing access to it. One approach to finding unpublished studies is to search for relevant investigations in a registry of clinical trials. Since studies are registered before they are completed, their inclusion in a registry is less likely to be influenced by whether the results were positive. If both published and unpublished studies are included, it is possible to analyze the results separately. An apparent difference in results between published and unpublished studies suggests the possibility of a publication bias.

Once the universe of clinical trials on a topic is identified, the meta-analyst must decide which specific studies to include in the study. This process is analogous to the need to establish eligibility criteria for enrolling patients in a clinical trial. For a meta-analysis, the decision about whether to include a particular study typically is based on (1) an assessment of its quality and (2) the ability to combine it with other studies based on respective patient populations, treatment regimens, and outcomes. Judging quality can be a subjective process, so it is advisable to limit inclusion criteria to basic features of a well-designed clinical trial. Examples of such inclusion criteria might include

• 1. Proper randomization of treatment assignments
• 2. Blinded assessment of outcome
• 3. Analysis based on the intention-to-treat principle.

Once the eligible source studies are selected, a standard abstract form should be used to extract key information. Independent abstraction of information by two reviewers is a useful practice to help reduce errors in data collection. Blinding of the abstractors to the identity of the original investigators, their affiliations, sources of support, and publication may help to reduce the potential of observation bias.

The outcome must be measured in a similar way across studies. For dichotomous outcomes (eg, the development of retinopathy or not), a rate ratio or relative risk may be the appropriate measure. For continuous outcomes (such as blood glucose level), a difference in means for the experimental and control groups may be employed. It should be noted, however, that the magnitude of a mean difference is affected by the distribution of the underlying population. For example, the mean difference in blood glucose levels between experimental and control subjects would be expected to be greater in a study population of diabetics with higher initial blood glucose levels than in a study population with lower initial levels. Accordingly, differences are often presented in units of standard deviation, thereby adjusting for the underlying distribution of values.

At the heart of a meta-analysis is the statistical combination of results from the individual clinical trials. The simplest approach to this combination would be to calculate the arithmetic average (mean) of the individual results. The simple mean of individual results would give equal weight to each of the studies. Because studies with smaller sample sizes are more prone to the effects of chance variation than studies with larger sample sizes, it is desirable to assign less influence to the smaller studies. By calculating a weighted average (in contrast to a simple or unweighted average) of results, more emphasis can be placed on the most statistically precise individual trial findings.

The statistical models used to combine results fall into two broad categories. The so-called fixed-effects model assumes that any differences in results across individual studies are attributable entirely to random variation. In contrast, the random-effects model assumes that the underlying relationship in question varies across studies in addition to the influences of random variation. When the results of the individual clinical trials being summarized are similar, minimal differences between the fixed- and random-effects models will be obtained. In general, however, the random-effects model will lead to somewhat less precise summary estimates, as reflected in wider confidence intervals. When the results of the underlying clinical trials vary widely, they are said to be heterogeneous. Under conditions of heterogeneity, the summary estimates obtained through the fixed- and random-effects models will differ, possibly to a considerable extent. It is important, therefore, to be able to determine whether the results of individual clinical trials are heterogeneous.

One approach to addressing the question of whether the variation across individual study results can be attributed to random variation alone is to perform a statistical test for heterogeneity. A statistically significant result for a test of heterogeneity would suggest that the variation across studies is greater than can be attributed to random variation alone. In such instances, therefore, a random-effects model would be the preferred approach to summarizing the data. On the other hand, if the test of heterogeneity is not statistically significant, the level of variation observed across individual study results can be explained by random variation. In such circumstances, a fixed-effects model would be an acceptable approach to summarizing the data.

Whether the question of heterogeneity of results should be addressed only, or even primarily, by a test of statistical significance is a matter of some debate. Because the tests of heterogeneity have limited statistical power, they are prone to type II errors, that is, they may lead to false-negative conclusions about the presence of heterogeneity in the underlying data. A statistically nonsignificant result may falsely reassure the meta-analyst (and the reader) that there is no evidence of heterogeneity in the underlying data. It is recommended, therefore, that the question of heterogeneity of results across studies be explored further during the course of the meta-analysis.

Heterogeneity of findings across studies can arise for a number of reasons in addition to random variation. The characteristics of the individual patient populations may be an important contributor to differences in observed effects. As previously noted, eligibility and exclusion criteria for a particular clinical trial may impose limitations on the selection of subjects based on age, sex, race, general health status, and other characteristics. To the extent that eligibility requirements vary across studies, the source populations may differ in fundamental ways that affect their likelihood of response to the treatment under investigation.

Aspects of the design of a clinical trial, for example, whether blinding was performed, the rate of compliance with treatment, the duration of follow-up, and the completeness of follow-up, may influence the results obtained. Approaches to analysis of individual study results, such as adhering to the intention-to-treat principle, can also impact the results.

Whether a clinical trial is stopped early is likely to be related to the findings. In general, clinical trials are not terminated prematurely unless an unexpectedly large difference in outcomes is observed between the experimental and control group. Such a difference could arise because the experimental treatment is much more effective than the control treatment (placebo or standard therapy). Alternatively, the complications or side effects of treatment may be greater in one treatment group than the other, causing the investigators to terminate the clinical trial early. In either case, the results of prematurely concluded trials are likely to differ from those trials brought to the originally intended conclusion.

Given all of the potential sources of heterogeneity, it is not surprising that variation in results across the selected studies is a common occurrence in meta-analyses. At one level, this variability is disconcerting because consistency of results across studies is a useful approach to establishing that an observed treatment effect is real. The ability to reproduce findings across independent investigators, study populations, and settings increases confidence in the inferences drawn concerning the effectiveness of the experimental treatment. Heterogeneity in results, particularly if it cannot be explained, raises uncertainty in the inferences that can be drawn from a meta-analysis.

It is also important to recognize circumstances in which a meta-analysis may not be advisable. For some questions of clinical interest, there may be an insufficient number of trials available to yield a conclusive summary. Even when a large number of trials on a topic are available, fundamental differences in their study populations, design, and outcome measures may make it unwise to attempt to combine results. Also, when there is considerable heterogeneity of treatment effect across studies that cannot be explained by a sensitivity analysis, it may be imprudent to calculate a weighted average of the individual study results.

In summary, meta-analysis is a powerful tool to help assess the cumulative evidence on the effectiveness of a particular treatment. By combining information across studies, it can lead to a statistically precise and objective estimate of the effect of interest, and it allows the consistency of findings across studies to be considered. On the other hand, the quality of the design and data of the original clinical trials, which is beyond the control of the meta-analyst, will affect the quality of the meta-analysis. Decisions made by the meta-analyst, such as which clinical trials to include and how the summary analyses are performed, will also affect the quality of the meta-analysis. Ultimately, the value of a meta-analysis will rest on its ability to help make informed treatment decisions. We will return to the topic of meta-analyses and systematic reviews in Chapter 13: Interpretation of Epidemiologic Literature in the context of discussing the interpretation of epidemiologic literature.

The Cochrane Collaboration

The Cochrane Collaboration is an international organization whose goal is to help people make well-informed decisions about health care. It addresses this goal by preparing, maintaining, and ensuring accessibility to current, rigorous systematic reviews of the benefits and risks of health care interventions. This organization was founded at a meeting in Oxford, England in late 1993. It is named in memory of Archie Cochrane (1909–1988), a physician epidemiologist who was a pioneer in the field of health services research. Cochrane was noted for his strong belief that health care decisions should be based on a critical synthesis of well-designed clinical trials of treatment effectiveness.

The Cochrane Collaboration involves six organizational units: Collaborative Review Groups, Fields, Centers, Methods Working Groups, Consumer Networks, and a Steering Committee. The core work of the organization is the preparation of systematic reviews, which are conducted by the Collaborative Review Groups. Each Review Group is comprised of individuals who share interests in particular health problems. The results of the systematic reviews are incorporated in the Cochrane Library, an electronic repository of evidence needed to make informed health care decisions. The Library was established in 1995 and is updated quarterly. It contains the following resources:

• 1. The Cochrane Database of Systematic Reviews—a regularly updated collection of completed systematic reviews and protocols of systematic reviews that are in progress.
• 2. The Database of Abstracts of Reviews of Effectiveness—an inventory of more than 2000 reviews that were completed outside of the Cochrane Collaboration.
• 3. The Cochrane Controlled Trials Register—a database that by 2003 contained citations for nearly 400,000 controlled trials identified by contributors to the Collaboration.
• 4. The Cochrane Review Methodology Database—a reference list of articles on the science of synthesizing evidence and the practical aspects of preparing systematic reviews.

Over its relatively short history, the Cochrane Collaboration has already produced a remarkable body of information. By 2003, 1837 systematic reviews were published. In spite of this impressive progress, the Collaboration faces many challenges. First, although the goal of the Collaboration is to produce reviews across the full spectrum of health care, it is dependent on the interests of persons who volunteer to prepare systematic reviews. Second, the use of such a wide range of reviewers of various backgrounds and skill levels makes it difficult to ensure a uniform high standard of work quality. Third, availability of the information in the Cochrane Library is not yet universal. Barriers to its availability include lack of knowledge about its existence and the subscription cost. However, with the progress already achieved by the Cochrane Collaboration, there is reason for optimism that this organization will play an increasingly important role in promoting well-informed health care decisions.

Evidence-based medicine can be defined as the integration of current best evidence with clinical expertise, pathophysiological knowledge, and patient preferences to make health care decisions. Although there are barriers to the practice of evidence-based medicine, such as the skills and time required to appraise the literature, this approach encourages effective management of disease. It can serve to optimize health outcomes and promote cost-effective disease management.

Fundamental to the practice of evidence-based medicine is the ability to assess the design, conduct, and analysis of clinical studies critically. For the purpose of assessing the comparative benefits of alternative treatments, the randomized controlled clinical trial is the gold standard approach. Therefore the evidence-based practitioner must be thoroughly familiar with this research method.

The advantages and disadvantages of randomized controlled clinical trials, as compared with alternative study designs, are set forth in Table 7–11. The principal strength of this approach derives from assigning treatments to patients by randomization, thereby tending to balance the study groups with respect to both known and unknown prognostic factors.

Before enrolling patients in a clinical trial, the investigator can determine the baseline and follow-up information that will be required on all subjects. Procedures can then be put in place to enable the researchers to collect data in a fairly complete and accurate manner. The investigator can also allocate subjects to desired dose levels rather than relying on physician or patient preferences. When blinding of the evaluators or patients is feasible, the assessment of clinical outcomes is less likely to be influenced by knowing which treatment was used.

However, randomized controlled clinical trials are subject to certain constraints. Restrictive criteria for inclusion of subjects may produce a very homogeneous study population, which may restrict the ability to extrapolate results to patients with other characteristics. Clinical trials—particularly those involving chronic processes—may require years of follow-up to determine the outcome of treatment. A prolonged observation period leads to higher costs, increases the likelihood that patients will be lost to follow-up, and delays the time at which a treatment recommendation can be made. The use of intermediate end points, such as measurement of blood glucose levels or glycosylated hemoglobin in the diabetes therapy trial, can help limit the length of required follow-up. Nevertheless, a definitive conclusion about treatment benefit often requires years of observation.

Large sample sizes typically are required in clinical trials when the magnitude of difference in responses between study groups is small. Furthermore, large numbers of subjects are likely to be required to demonstrate differences between study groups when there is wide variability in responses to treatment. Increasing the size of the study population not only raises the cost of a trial, but may also lead to pragmatic difficulties in locating a sufficiently large pool of eligible patients.

Ethical concerns may arise in a clinical trial if one or more of the treatment options has serious potential side effects or if early data suggest—but do not establish—a therapeutic advantage for one of the treatments. In this situation, a decision must be made about whether the trial should be continued until a definitive conclusion is reached or should be terminated early so that all patients have the opportunity to receive the apparently superior treatment. To minimize the possible influence of real or perceived conflicts of interest, an external advisory group should review these ethical questions.

An investigator cannot control the behavior of subjects enrolled in a clinical trial. Even after initial informed consent has been given to participate in a clinical trial, a subject has the right to withdraw at any time. Some subjects may elect to remain in the trial but not comply with the assigned regimen. Noncompliance can reduce the statistical power of a clinical trial and thereby lead to a false-negative conclusion. Accordingly, every effort must be made to achieve maximal compliance with assigned treatment without infringing on the patient’s right to refuse therapy. Ultimately, treatment decisions should be based on the best evidence available concerning therapeutic benefit. The standard approach to gathering this evidence is the randomized controlled clinical trial. Although this type of investigation is labor intensive, time consuming, and expensive, it can provide the most convincing evidence of the superiority of one treatment over another. Through the use of randomized clinical trials such as the diabetes therapy study, decisions concerning patient treatment and care can be based on rigorous scientific information.

A systematic review is any type of synthesis of medical evidence on a topic that has been prepared using strategies to minimize errors. A particular type of systematic review is referred to as a meta-analysis. A meta-analysis is a statistical combination or integration of the results of several independent studies that are considered to be combinable. A meta-analysis can provide a statistically precise estimate of the treatment effect in question. It also may help to explain any observed variation in the estimated treatment effect across different studies.

Important steps in a meta-analysis include the definition of the research question of interest, the sampling of eligible studies, the abstraction of key information from the studies, the choice of an analytical strategy, and the interpretation of the results obtained. The sampling strategy including clinical trials is particularly important, as the data available for the meta-analysis will be determined by these decisions. The possible influences of a variety of factors, such as publication bias, must be considered when designing the process for identifying and determining the eligibility of individual clinical trials.

In calculating the summary measure of treatment effect in a meta-analysis, the individual study results typically are weighted to give the most influence to larger studies with more statistically precise estimates. Statistical summarization of results across studies typically is performed with either a fixed-effects model or a random-effects model. A fixed-effects model assumes that any variation in observed treatment effects across studies is attributable only to random variation. A random-effects model assumes that in addition to random variation, there is a different underlying effect in each clinical trial included. The two models will yield different summary measures of effect if the findings of the underlying clinical trials are heterogeneous, that is to say, the findings vary beyond the expected influence of chance. The pattern of heterogeneity can be explored through a sensitivity analysis, and thereby may lead to the identification of subgroups of studies with homogeneous effects. In so doing, the overall pattern of heterogeneity of clinical trial results may be explained by characteristics of their population groups, study features, or other characteristics.

One massive undertaking in support of the conduct and use of high-quality systematic reviews is the Cochrane Collaboration. This organization is dedicated to promoting well-informed health care decisions. It strives to meet this goal by preparing, maintaining, and ensuring accessibility to current, rigorous, systematic reviews of the benefits and risks of health care interventions. Established in late 1993, the Collaboration has a series of Review Groups, each of which includes individuals with a shared interest in particular health problems. These Review Groups conduct systematic reviews, which are disseminated through the Cochrane Library. In addition to the completed reviews, the Library, which is updated quarterly, includes a database of systematic reviews performed outside of the Collaboration, a register of clinical trials identified by members of the Collaboration, and a database of methodological references.

By mid-1999, the Cochrane Collaboration had completed 576 systematic reviews and had prepared protocols to conduct 538 additional reviews. Although the Collaboration intends to address all areas of health care, at present it is constrained by the interests of the volunteer reviewers. Moreover, the large number of reviewers utilized worldwide, with varying levels of skills and experience, compounds the problem of quality assurance. In spite of these challenges, the Cochrane Collaboration offers great promise in promoting health care decisions that are well informed by the best available evidence.