Despite the existence of many prognostic models and their attractive potential utility, scoring systems have a limited clinical value at the bedside and are seldom used in clinical practice. There are a number of reasons for this.
Prognostic models should closely follow the advancement of medical sciences and be updated in a timely fashion to avoid becoming obsolete. There may be limitations of the predictive ability after modification of a variable. As diagnosis and treatment evolve scoring systems require repeated recalibration and validation, although this is difficult and time consuming to achieve. Also the lack of collaboration between researchers that might improve upon a single model leads to the multiplication of redundant, often competing and nonvalidated models.10,13,14,15
Most of the available information on scoring systems is based on internal validation, less studies report external validation and almost none report the impact on clinical decision-making and outcome.16 After the external validation process is completed, the impact of the scoring system on clinical decision-making, patient outcome, or costs should be assessed in a management study or randomized controlled trial12,13 before being used in clinical practice.
Scoring systems designed for general medical or surgical ICUs would be inappropriate if used in specialized units such as coronary, burn or pediatric where patients differ from the populations of the original and external validation samples from which the system was derived. Clinical judgment remains fundamental because not all relevant variables are included in the models.
The outcome and the time points might be different than those for which the scoring system was designed. For example, changes in hospital practices resulting in earlier discharge of patients to long-term acute care facilities thus reducing the hospital mortality or calculating scores outside of the intended and validated ranges, for example, daily versus within the first hour or first 24 hours.
To be used in clinical practice, the user should understand the existing models, be convinced of their utility and trained in their application. The barriers for the effective use of scoring systems may be related to the limited practical utility of the models. Some models such as APACHE IV are complex and require dedicated personnel and a specific training; they can be expensive to use, therefore, limiting their spread across the healthcare community.
The more data required by the scoring system, both in amount and complexity, the greater the opportunity for errors and missing data. In theory, different users with similar backgrounds in critical care should obtain similar scores for the same patient population, however, low interobserver reliability resulting in dissimilar scoring values by different users can occur.
Scoring systems provide only probabilities, not certainties. It would be inappropriate to deny or withdraw care based solely on a probability derived from any existing scoring system. The course of illness over time is important. Clinical variables may not be included in these models and clinical judgment and ethical principles need to be employed when making decisions regarding prognosis in an individual patient. For example, the physiologic response of sepsis which is linked to unknown genomic factors is not taken into account in any scoring system. Human physiology is complex and clinical assessment prevails over scoring system predictions.
Benchmarking Intensive Care Units Based on Scoring Systems
The ranking of intensive care units and institutions based on severity of illness versus mortality rate is controversial. Different populations of patients in different ICUs are not always comparable even when their severity of illness scores is identical. For example, a hospital with the same mean prognostic score but a higher mortality does not necessarily underperform when compared to another hospital as there may be other elements that are not represented in the dataset captured by these systems. For instance, as the lack of health insurance is associated with increased 30-day mortality and decreased use of common procedures for the critically ill, the insurance status may be an indicator of mortality not picked up by patient and clinical characteristic variables.17 In elective surgical admissions, there is an association between the socioeconomic status and hospital mortality that is not explained by case mix or the withdrawal of active treatment.18 Finally, among trauma patients, the uninsured receive less trauma-related care and have a higher mortality rate.19 These examples illustrate the fact that 2 different populations of patients can have similar severity of illness scores with very different mortality rates.