A warning on early-warning scores!
It is clinically intuitive that physiological deterioration precedes critical illness, and there is growing research evidence to support this belief.1–6 The use of track and trigger early-warning systems to facilitate the early recognition of the acutely deteriorating patient outside the critical care area of a hospital is based on this premise. The first such scoring system, based on a single parameter trigger, was developed in Liverpool, Australia as part of the Medical Emergency Team (MET) concept.7 This was followed by the development of a range of similar systems that have entered clinical practice around the world.89 These incorporate a variety of physiological variables in scoring, and trigger algorithms that are often more complex than the original MET's calling criteria. They use a wide range of weightings and cut-off points to develop a score designed to trigger activation of a response system. However, most parameters and cut-off points are still based on clinical intuition, rather than the use of rigorous derivation and validation methodologies.Recent work casts doubt on the clinical utility of the systems currently in use. The MERIT study, a randomized, controlled trial by Hillman and colleagues,10 failed to demonstrate benefit partly because of the calling criteria's sensitivity and specificity of <50%. Indeed, the majority of ‘deteriorating’ patients were not detected until <15 min before they suffered cardiac arrest, died or were admitted to an intensive care unit. The SOCCER study by Jacques and colleagues5 attempted to identify early and late predictors of adverse outcome and suggested that the failure of MERIT to demonstrate efficacy for METs may also be due to the current MET call criteria, which are actually late signs of deterioration.5 Two other studies have demonstrated the influence of relatively small changes in physiological cut-off points on improving the sensitivities and specificities of the MET criteria.1112
More complex early-warning systems have not convincingly demonstrated that they are useful for the early detection of the deteriorating patient. When applied in an ideal mathematical way, these types of scores have reasonable sensitivity and specificity.13 However, in clinical practice, there are major concerns over their accuracy. A recent systematic review9 concluded that there was ‘little evidence of reliability, validity, and utility’, and that their ‘sensitivity was poor, which might be due in part to the nature of the physiology monitored or to the choice of trigger threshold’. Other work has also demonstrated inaccuracies in the calculation of scores by staff,1415 and significant intra- and interrater reliability error.16
Unless scoring systems have appropriate sensitivity and specificity, and minimize errors associated with documentation and scoring, they will fail to identify patients who need additional care and will increase workload in circumstances where no intervention is required. There is a clear need to re-assess the use of early-warning scoring systems as they are currently used and to use a structured, scientific approach to their development and evaluation.
First, it is essential to identify which physiological variables are important as early identifiers of deterioration. This requires that study designs should ensure the collection of all potentially appropriate, routinely available items every time a measurement set is made. Restricting the data collection to objective variables is preferable, because using terms such as ‘worried’ as triggers7 has obvious problems with interpretation. Next, it is important to determine the correct cut-off values, and this requires studies to include the whole ‘at risk’ population (i.e. the denominator). The available literature appears to suggest that many existing scoring systems use the incorrect physiological variables (e.g. temperature),1317 fail to use all of the appropriate ones (e.g. oxygen saturation),1317–19 or use the wrong cut-off points (e.g. oxygen saturation).51317–19
Having identified the important variables and cut off values, it is then important to derive and validate a scoring system de novo. This requires the use of derivation and validation cohort methodology and has been undertaken in only a few studies to date, usually in emergency departments or assessment units.17–19 Studies in these clinical settings have the obvious advantage that the identification of a uniform sampling point (e.g. on admission) for physiological variables is easy. Identifying appropriate sampling points for use in derivation and validation cohort methodology on general hospital wards is more challenging. Which value should one use? Which value should one use on admission? Which value is the most deranged? Which value is the most recent before the arrival of an outreach or MET?
In this edition of the British Journal of Anaesthesia, Duckitt and colleagues20 describe a derivation and validation study of 4384 patients in a UK Medical Assessment Unit. They describe the values of physiological variables that were important as descriptors of in-hospital mortality (e.g. heart rate >101 beats min–1, ventilatory frequency >19 bpm, oxygen saturation <96%, and systolic arterial pressure <100 mm Hg) and have applied weightings to them in a new scoring system, which they then tested in a separate validation cohort. The new scoring system has reasonable diagnostic accuracy and is more accurate than another commonly used scoring system. Higher scores are also associated with higher mortalities and longer hospital length of stay. However, before the Worthing system could be adopted into clinical practice in other areas of their hospital, or in other hospitals, further validation using other data sources is required. This poses a clinical dilemma as this study20 used saturation values measured on air. It is questionable whether it is ethical to delay administering oxygen to, or perhaps worse still to remove it from, sick patients simply to perform an early-warning score. Patients on the steep curve of the oxygen dissociation curve, who are likely to be among those at greatest risk of an adverse outcome, may well suffer adversely if oxygen is withheld or removed. Ignoring the delivered oxygen concentration is equally problematic, because the treatment drives oxygen saturation towards a goal of 100%. Future studies will need to link oxygen saturation to delivered oxygen concentration.
Nevertheless, Duckitt and colleagues20 are to be congratulated on taking a robust approach to their study, which adds considerably to the understanding in this field. Their work, and that of others,17–19 demonstrates that scoring systems of high accuracy can be developed de novo and may, after appropriate development and validation in varied patient groups, be able to replace the existing scores that have low or poor diagnostic accuracy.
Finally, to overcome the inaccuracies and miscalculations related to manual data collection, it may be necessary to adopt continuous vital signs monitoring,6 or electronic data management and scoring systems.2122 Such systems should minimize intra- and interrater reliability error and facilitate the use of potentially more accurate discriminate functions that are less amenable to manual calculation error. However, to date, such systems have not demonstrated improved clinical or other outcomes, and further research is required.
In conclusion, the existing combinations of early-warning scoring systems and outreach/MET responses have failed to demonstrate improved outcomes in studies that attempt to eliminate bias.23 This may be because of the lack of sensitivity and specificity of current systems to allow accurate detection of early critical illness. The use of scores that use parameters and cut-off points that are inappropriate may be unhelpful and should be replaced by ones that are derived using reliable methodologies and have high diagnostic accuracy. Further work is required to derive and validate such scores and to explore the role of electronic data capture and scoring systems. Until this work is completed, we should not accept that the existing early-warning scoring systems are necessarily any better at detecting the deteriorating patient than high-quality clinical assessment and judgement by appropriately skilled and experienced personnel.
G.S. is leading a team of researchers who have contributed to the development of an electronic vital signs data gathering system for use in general ward areas of hospital, and the team is currently investigating the function of early-warning scoring systems. Professor Smith's wife is a creditor of the company that is developing the system.
Health Services Research Unit
Health Sciences Building
University of Aberdeen
Aberdeen
UK
Department of Critical Care
Portsmouth Hospitals NHS Trust and University of Bournemouth
Portsmouth
UK
* E-mail: b.h.cuthbertson{at}abdn.ac.uk
References
1 Schein RMH, Hazday N, Pena M, Ruben BH, Sprung CL. Clinical antecedents to in-hospital cardiac arrest. Chest (1990) 98:1388–92.[CrossRef][Web of Science][Medline]
2 Franklin C, Mathew J. Developing strategies to prevent in-hospital cardiac arrest: analysing responses of physicians and nurses in the hours before the event. Crit Care Med (1994) 22:244–7.[Web of Science][Medline]
3 Goldhill DR, McNarry AF, Mandersloot G, McGinley A. A physiologically-based early warning score for ward patients: the association between score and outcome. Anaesthesia (2005) 60:547–53.[CrossRef][Web of Science][Medline]
4 Kause J, Smith G, Prytherch D, et al. for the Intensive Care Society (UK) and Australian and New Zealand Intensive Care Society Clinical Trials Group ACADEMIA Study investigators. A comparison of antecedents to cardiac arrests, deaths and emergency intensive care admissions in Australia and New Zealand and the United Kingdom—the ACADEMIA study. Resuscitation (2004) 62:275–82.[CrossRef][Web of Science][Medline]
5 Jacques T, Harrison GA, McLaws ML, Kilborn G. Signs of critical conditions and emergency responses (SOCCER): a model for predicting adverse events in the inpatient setting. Resuscitation (2006) 69:175–83.[CrossRef][Web of Science][Medline]
6 Watkinson PJ, Barber VS, Price JD, Hann A, Tarassenko L, Young JD. A randomised controlled trial of the effect of continuous electronic physiological monitoring on the adverse event rate in high risk medical and surgical patients. Anaesthesia (2006) 61:1031–9.[CrossRef][Web of Science][Medline]
7 Lee A, Bishop G, Hillman KM, Daffurn K. The Medical Emergency Team. Anaesth Intensive Care (1995) 23:183–6.[Web of Science][Medline]
8 DeVita MA, Bellomo R, Hillman K, et al. Findings of the First Consensus Conference on Medical Emergency Teams. Crit Care Med (2006) 34:2463–78.[CrossRef][Web of Science][Medline]
9 Gao H, McDonnell A, Harrison DA, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med (2007) 33:667–79.[CrossRef][Web of Science][Medline]
10 Hillman K, Chen J, Cretikos M, et al. Introduction of the medical emergency team (MET) system: a cluster-randomised controlled trial. Lancet (2005) 365:2091–7.[CrossRef][Web of Science][Medline]
11 Bell MB, Konrad D, Granath F, Ekbom A, Martling CR. Prevalence and sensitivity of MET-criteria in a Scandinavian University Hospital. Resuscitation (2006) 70:66–73.[CrossRef][Web of Science][Medline]
12 Cretikos M, Chen J, Hillman K, Bellomo R, Finfer S, Flabouris A. the MERIT study investigators. The objective medical emergency team activation criteria: a case–control study. Resuscitation (2007) 73:62–72.[CrossRef][Web of Science][Medline]
13 Cuthbertson BH, Bourjerdi M, Aucott L, Prescott G. Can physiological variables and early warning scoring systems allow early recognition of the deteriorating surgical patient? Crit Care Med (2007) 35:402–9.[CrossRef][Web of Science][Medline]
14 Smith AF, Oakley RJ. Incidence and significance of errors in a patient ‘track and trigger’ system during an epidemic of Legionnaires' disease: retrospective case note analysis. Anaesthesia (2006) 61:222–8.[CrossRef][Web of Science][Medline]
15 Prytherch D, Smith GB, Schmidt P, et al. Calculating early warning scores—a classroom comparison of pen and paper and hand-held computer methods. Resuscitation (2006) 70:173–8.[CrossRef][Web of Science][Medline]
16 Subbe CP, Gao H, Harrison DA. Reproducibility of physiological track-and-trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med (2007) 33:619–24.[CrossRef][Web of Science][Medline]
17 Kellet J, Deane B. The Simple Clinical Score predicts mortality for 30 days after admission to an acute medical unit. Q J Med (2006) 99:771–81.[Web of Science]
18 Olsson T, Lind L. Comparison of the rapid emergency medicine score and APACHE II in non-surgical emergency department patients. Acad Emerg Med (2003) 10:1040–8.[CrossRef][Web of Science][Medline]
19 Olsson T, Terent A, Lind L. Rapid Emergency Medicine score: a new prognostic tool for in-hospital mortality in non-surgical emergency department patients. J Intern Med (2004) 255:579–87.[CrossRef][Web of Science][Medline]
20 Duckitt R, Buxton-Thomas R, Walker J, et al. The Worthing Physiological Scoring System: derivation and validation of a physiological early warning system for medical admissions. Observational population based single-centre study. Br J Anaesth (2007) 98:770–75.
21 Smith GB, Prytherch DR, Schmidt P, et al. Hospital-wide physiological surveillance—a new approach to the early identification and management of the sick patient. Resuscitation (2006) 71:19–28.[CrossRef][Web of Science][Medline]
22 Tarassenko L, Hann A, Young D. Integrated monitoring and analysis for early warning of patient deterioration. Br J Anaesth (2006) 97:64–8.
23 Esmonde L, McDonnell A, Ball C, et al. Investigating the effectiveness of critical care outreach services: a systematic review. Intensive Care Med (2006) 32:1713–21.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. Dosani, J. Lim, P. Yang, C. Brouse, J. Daniels, G. Dumont, and J. M. Ansermino Clinical evaluation of algorithms for context-sensitive physiological monitoring in children Br. J. Anaesth., May 1, 2009; 102(5): 686 - 691. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. J. M. Morgan and M. M. Wright In defence of early warning scores Br. J. Anaesth., November 1, 2007; 99(5): 747 - 748. [Full Text] [PDF]
|
|
E-letters:
Read all E-letters
- Early warning scores: a homogeneous score for a heterogenous population?
- Carolyn Johnston
- British Journal of Anaesthesia, 25 Jun 2007 [Full text]
- In Defense of Early Warning Scores
- Richard J M Morgan, et al.
- British Journal of Anaesthesia, 14 Aug 2007 [Full text]
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||