BJA Advance Access published online on August 17, 2007
British Journal of Anaesthesia, doi:10.1093/bja/aem240
Simulation as an additional tool for investigating the performance of standard operating procedures in anaesthesia
1 Department of Anaesthesiology, Emergency and Intensive Care Medicine, University of Goettingen, Robert-Koch-Strasse 40, 37075 Goettingen, Germany
2 Department of Psychology
3 Department of Anaesthesiology, University of Heidelberg, Heidelberg, Germany
4 Department of Anaesthesiology, Siloah and St Trudpert, Pforzheim, Germany
* Corresponding author. E-mail: yzausig{at}zari.de
Accepted for publication June 25, 2007.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Background: In medicine, the use of standard operating procedures (SOPs) is often evaluated using questionnaires (QUES). However, QUES can have limitations with regard to method, thus leading to errors. Simulation (SIM) offers another opportunity for evaluation. We hypothesized that medical errors in the evaluation of SOPs using QUES could be detected by SIM, and that SIM is better qualified to demonstrate applied medicine.
Methods: We investigated the use of SOPs in anaesthesia, rapid sequence induction (RSI), by means of a QUES (n=42) or SIM (n=42) among 84 anaesthesiologists. Seven measures for preventing aspiration during induction of anaesthesia were examined and evaluated according to a predetermined points system.
Results: The average number of times that precautionary measures to prevent aspiration were mentioned in the QUES [4.8 (0.9)] or performed during SIM [5.0 (1.1)] did not differ between the two groups. Pre-oxygenation was the most frequently described or performed measure (95% vs 93%). However, other measures, such as avoidance of positive pressure ventilation (45% vs 85%), differed significantly between the two groups.
Conclusions: QUES and SIM are powerful instruments for evaluating the implementation of SOPs such as RSI. SIM demonstrates automated behaviours and thus more clearly represents behaviours used in clinical practice than is possible to demonstrate using QUES. Using a combination of these two instruments, method errors resulting from the individual instruments can be reduced.
Keywords: anaesthetic techniques, induction; complications, aspiration; education, continuing; model, computer simulation; safety
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Standard operating procedures (SOPs) are components of clinical pathways that were introduced to improve diagnostic and therapeutic management within medicine. They are based on current studies and recommendations from experts and professional organizations. The introduction of these procedures into everyday use in hospitals has led to a reduction in morbidity, mortality, and hospital costs.1
In general, SOPs are often evaluated using instruments such as questionnaires (QUES) or observer studies.2 3 However, the reliability of an investigation using QUES alone depends on the response rate and the representativeness of the survey.4 Additionally, method errors can also occur, due to missing data, polarity reversal (1 instead of 6), contradictory, or inconsistent answers or due to answers that tend towards social desirability, all of which can lead to a bias in the results.5
Simulation (SIM) enables successful training of technical and non-technical skills.6 SIM also provides the opportunity to observe and evaluate technical performance.7–9 Thus, SIM represents an excellent means of evaluating the use of SOPs, with a special focus on applied practice. The objective of this study was to examine the results of the evaluation of an SOP with the generally recognized instrument, QUES, and to compare those with the results of a possible new tool, SIM. Currently, we are not aware of any other simulator-supported studies comparing the performance of QUES and SIM. We chose rapid sequence induction (RSI) procedure, which is indicated by any risk of aspiration of the contents of the stomach during induction of anaesthesia. We hypothesized that method errors in the evaluation using QUES could be detected using SIM, SIM is better qualified to demonstrate applied medicine, and that SIM is better suited than QUES to the implementation of medical standards.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
The study was approved by the medical faculty of the University of Heidelberg, Germany, and written informed consent was obtained from all of the participants. The investigation performed by QUES was blinded and the answers could not be traced back to the participants.
Our SIM centre is located in our Department of Anaesthesiology. It comprises an operating room that is separated from the control room by a two-way mirror. The operating theatre includes a standard anaesthetic apparatus (Sulla 808V, Draeger®, Luebeck, Germany), a physiological patient monitor (GE®, Eagle 5000, Sollingen, Germany), an anaesthesia cart, an operating table (Maquet®, Rastatt, Germany), a surgical instrument table, and a patient high-fidelity simulator mannequin (HPS, METI®, Sarasota, FL, USA) located on the operating table. This simulated world presents an environment for administering anaesthesia that is quite close to reality.10
An anaesthesia nurse from our department supported the participants from the SIM group during anaesthesia. The nurse was familiar with all of the medical devices. The participants were told ahead of time that the nurse could be trusted and would only provide correct information.
Eighty-four anaesthesiologists from external hospitals and from within our department participated in the study. They were randomized into two groups of equal size with 42 participants each. One group received a QUES; the other group participated in a session of SIM. Participants described anaesthesia or administered it to a 20-yr-old, 72 kg man in good general health who had not undergone any previous operations. For more than 3 h, the patient had been suffering with typical symptoms of acute appendicitis, with nausea and vomiting, and was at an increased risk of aspiration. Participants were told that a stomach tube had been inserted in the emergency room. The stomach had been suctioned, and the tube had been removed by the nurse shortly before anaesthesia induction. All participants had free choice of medical equipment and medications for anaesthesia.
The participants in the SIM group received a brochure about the options and limitations of SIM and the structure of the SIM centre. The SIM session was video-recorded using two cameras. Recording and camera set-up was performed by a member of the study group who had been briefed beforehand.
Participants in the QUES group received an open QUES based on a pilot study done by our department. This QUES contained only basic information as described above, and participants were asked to describe in detail their way of induction of anaesthesia with regard to precautions, choice of drugs, and the order of administration. The QUES were collected and analysed retrospectively.
Criteria for inclusion in the study were participation on a voluntary basis, professional experience of >0.5 yr and no previous experience with simulators. Only QUES that were filled out completely and video recordings that were technically acceptable were included in the analysis process.
To obtain the highest possible level of reliability, two independent anaesthesiologists who had not been involved in collecting the data evaluated the QUES and video recordings.
The participants indicated their degree of anaesthesiological training (anaesthesiologist with board certification vs anaesthesiologist without board certification), their gender, and their clinical experience. We used the video recordings and the QUES to evaluate the precautionary measures used to prevent aspiration during the RSI.2 3 For evaluation, we used investigations that had already been carried out regarding RSI.2 3 Seven precautions to prevent aspiration were examined.
- Pre-oxygenation: Detailed description in the QUES group, or performance in the SIM group, according to the current recommendations11 (e.g. sealed face mask for >3 min, an FEO2 of 0.9 was achieved).
- Change of the horizontal position of the patient: Detailed description in the QUES group, or performance in the SIM group according to the current recommendations12 (e.g. head down or semi-sitting position).
- Performance of a RSI that was truly ‘rapid’: a truly ‘rapid’ RSI was considered to be done, when terms such as ‘RSI’, ‘rapid’, ‘fast’, or other similar expressions were described in the QUES, or mentioned in the SIM.
- Prepared suction.
- Arranged endotracheal tube with stylet.
- Applying cricoid pressure.
- Avoidance of positive pressure ventilation (APPV) before securing the airway with an endotracheal tube.
The last four precautions were either detailed in the QUES group or judged on performance and orders to the nurse in the SIM group.
For each of the respective preventative measures implemented, one point was given (Table 1).
|
Additionally, we also examined the way the anaesthesia was performed with regard to choice of drugs (dose and substance). Classification was performed as follows: propofol 1.5–2.5 mg kg–1, thiopental 3–5 mg kg–1, fentanyl 1–5 µg kg–1, sufentanil 0.5–2 µg kg–1, rapifen 10–40 µg kg–1, rocuronium 0.6–0.9 mg kg–1, and succinylcholin chloride 1–1.5 mg kg–1.
Both the patients’ characteristics obtained and the performances of RSI were analysed statistically. We used a paired t-test for independent pairs to compare the number of points received when taking precautions towards aspiration in the two groups. Each precaution was compared between the two groups using the 
2 test. P-values of <0.05 were considered significant, and those <0.01 were considered highly significant. All data are presented as mean (SD).
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
All video cassettes from the SIM group (n=42) were of sufficient audiovisual quality for evaluation. The return rate for the QUES group was 95.2%, which means that 40 of the 42 QUES could be evaluated.
The two groups did not differ with regard to gender, clinical experience, and level of education in anaesthesia (Table 2). Twenty women and 22 men participated in the SIM study. In the QUES group, 22 women and 18 men answered the QUES. The average level of clinical experience in the QUES group was somewhat longer (5.8 yr) than that in the SIM group (4.0 yr), but this was not significant (P=0.09). In the SIM group, nine participants were anaesthesiologists with board certification and 33 were anaesthesiologists in training. There were 15 anaesthesiologists with board certification and 25 anaesthesiologists in training in the QUES group.
|
Clinical experience (Fig. 1), gender, and level of education (data not shown) in the two groups did not correlate with the number of points achieved for precautions against aspiration.
|
The average number of points was 5.0 (1.1) in the SIM group. This did not significantly differ from those in the QUES group [4.8 (0.9)] (Table 2). The frequency of generally accepted precautions against aspiration (Fig. 2) such as pre-oxygenation (SIM: 95.2% and QUES: 95.0%) or the application of cricoid pressure (83.3% and 92.5%, respectively) did not differ between the SIM and the QUES groups. There were significant differences, however, among the semi-sitting position, performance of a ‘rapid’ RSI, prepared suction, use of an endotracheal tube with stylet and APPV. However, there was a significant difference (P<0.01) between the two groups for the APPV (45% vs 85%, respectively) and performance of a ‘rapid’ RSI (23% vs 88%, respectively).
|
The medication used during the RSI is presented in Table 3. Only 17 of the 40 participants (42.5%) in the QUES group answered the question satisfactorily.
|
Fentanyl [mean 166.2 (53.8) µg] was the most preferred opioid in the SIM group (P<0.01) (Table 3). The second most frequently used opioid was sufentanil [mean dose: 15 (5) µg], followed by alfentanil [2.5 (0.5) mg] used in 4.8% of the cases.
Precurarization was performed in 31 cases in the SIM group. Rocuronium was used in 43.8% of the cases, atracurium in 15.7%, vecuronium in 15.7%, cisatracurium in 6.3%, and mivacurium in 3.1%. No prophylaxis was used in 15.7% of the cases when succinylcholine chloride was administered. With regard to frequency of administration, propofol [182.9 (32.8) mg] and thiopental [441.7 (60.7) mg] did not differ (P=0.19). Succinylcholine chloride [101.5 (9.7) mg] was significant as the preferred neuromuscular blocking drug in 80.9% (P<0.01) of the cases, followed by rocuronium [52.9 (6.9) mg] with 16.7%.
In the QUES group, fentanyl [181.3 (49.6) µg] was the most frequently described opioid (Table 3). In 41.2% of the cases, no opioid was specified. In 64.8% of the cases, succinylcholine chloride [110 (38.2) mg] was the most frequently used neuromuscular blocking drug, followed by rocuronium (35.3%). In 71.4% of the cases, a non-depolarizing neuromuscular blocking drug was administered prophylactically as part of precurarization. Two participants did not describe any precurarization. Thiopental [409.1 (51.4 mg)] and propofol [226.7 (37.3 mg)] were used for induction.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Our study demonstrated a significant difference in the results of the two test instruments when investigating the performance of an SOP. Although the average number of precautionary measures named or performed to prevent aspiration did not differ between the two groups, there were significant qualitative differences. In the QUES group, the semi-sitting position and prepared suction were described much more frequently, whereas in the SIM group, APPV, ‘rapid’ performance of RSI, and arranged endotracheal tube with stylet were applied most frequently.
Qualitative and quantitative differences in the implementation of precautions against aspiration during the RSI could result from a varying group of patients. Thus, surveys of anaesthesia in obstetrics demonstrated considerable differences in the application of precautionary measures and selection of medications when compared with non-obstetric operations.2 3 In the current study, however, both groups were to perform a RSI on an acute appendicitis case. There are also regional and international differences in the performance of RSIs.2 3 13 In a national comparison of anaesthesia standards, Fassoulaki and colleagues13 demonstrated that the performance of pre-oxygenization and application of cricoid pressure is subject to variations of 14–100% and 20–100%, respectively. This could be caused by varying recommendations from the individual anaesthesiological associations or regionally different discussions regarding the purpose and intent of the individual precautionary measures.12–14 Also, clinical experience may correlate with the application of medical standards.15 In this study, we were unable to prove a connection between professional experience and the selection of and the number of precautionary measures taken during the RSI. Byrne and Greaves16 recently described how difficult it is to find a correlation between clinical performance and clinical experience.
The implementation of medical standards is often investigated using surveys. Various methods have been established for this purpose. These include face-to-face interviews, postal or telephone surveys, and more recently, the Internet. QUES frequently serve as the basis of these investigations.5 However, low return rate or restricted representation can limit the value of a study.5 Methodological errors during an investigation with a QUES due to missing data, reversal of polarity (1 instead of 6), and contradictory or inconsistent answers can also lead to a skewing of the results.17 A very frequent error, answering questions according to social desirability, can lead to a discrepancy between the theories described in the QUES (with regard to an evidence-based medicine) and behaviour that would actually occur in practice. These errors become obvious when comparing the groups’ responses during implementation of semi-sitting position and preparing suction. Although these measures are generally considered very important3 8 and the participants using the QUES indicated them as such, in the clinical reality provided for the SIM group, they were much less frequently applied.
Normally, the QUES determines the extent of existing knowledge on the part of the person surveyed. Recurrent, standardized activities that are performed unconsciously, or automatically, are generally not stored in the brain as processes, but only as products. This decreases the reliability of a QUES.18 This explains the discrepancy between the two groups when performing the APPV, ‘rapid’ RSI, and the use of an endotracheal tube with stylet. Owing to the degree of realism it provides and to its ability to represent and quantify practical skills,6 15 SIM provides an appropriate means of testing these automatic behaviours, and in this regard, it has a clear advantage over the QUES.7
But SIM is not free of methodological errors. The current SIM used in this study represents a realistic anaesthetic situation10 in which technical and non-technical skills can be learnt and practiced.19 However, being observed can cause unusual stress, much like in a test. On the other hand, the Hawthorne effect can occur. These two effects can change the natural behaviour of the test subject in simulated practice, thus endangering the external validity of the study. Technical problems (such as computer crashes) and limits (such as no palpable body temperature and no visible cyanosis) can countermand the suggestion of reality and thus lead to changes in practical procedures.20
In both groups, fentanyl (SIM: 90.8% and QUES: 94.2%) was most frequently used at a sufficient dose.2 According to another survey, alfentanil was the preferred drug in 88% of the cases, whereas in this study, only two participants in the SIM group and none in the QUES group chose to administer it.3 This could be due to differences in the group of patients or to differing teaching methods in the study from Thwaites and colleagues.2 3 Sufentanil was administered in both groups (SIM: 11.9% and QUES: 5.9%); however, the dose selected [15 (5) µg] was well below the expected dose of 0.5–2 µg kg–1.21
The majority of participants in the SIM group used propofol, whereas the medication described most often in the QUES group was thiopental. The latter record is in keeping with surveys done in the UK.2 3 The two induction agents used here enable similar intubation conditions.22 Surprisingly, despite the known high rate of side-effects, succinylcholine was predominantly used in both groups. This is in agreement with the results of other studies.2 3
This study does not provide a general representation of the current situation during performance of a RSI in Germany. Participants came mostly from one region, and the number of participants was too low to make a generally valid, representative statement.5 We purposely did not evaluate participants using both instruments, thus decreasing the risk of a bias regarding training effect or recall.6 23 The open QUES was selected because although RSI is a standardized process, there is not sufficient existing information to enable a stringent construction of the QUES. But, these data should help future researchers to deal with these limitations. The clear methodical differences between the two evaluation instruments demonstrate the difficulties involved in developing adequate training based on the data attained. To establish precautionary methods and adequate teaching based on the results of the QUES, one would emphasize on APPV, rapid performance of RSI, and correct preparation of the intubation, although data acquired by SIM would put the semi-sitting position and preparation of suction in first place. One must be aware of this discrepancy if one wishes to measure performance. However, combination of the two investigating instruments could enable the creation of better, clearer training objectives.
In conclusion, the results of this study demonstrate that SIM is a suitable method to attain data. Compared with the established method, which is investigation via QUES, SIM can easily detect method errors, such as social desirability. Beyond that, SIM is more suitable for verifying automatic behaviours and thus more closely represents the applied practice, and so SIM could be better suited to the implementation of medical standard. A combination of the two evaluation methods can result in a strategy to better reproduce SOPs, to generate new SOPs, and thus to increase patient safety.
|
Funding |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
Medical Faculty of University, University of Heidelberg, Germany (f. 203 453).
|
Acknowledgements |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
We would like to thank other members of our simulator team: C. Busch, T. Boeker, H. Gahderi, A. Schmitz, and E. Zink.
|
Footnotes |
|---|
The data were presented in part as a poster at the German Anaesthesia Congress (Deutscher Anästhesiecongress—DAC) in Munich from April 16–19, 2005, at the SESAM meeting in Bristol, UK from May 20–22, 2005 and also at the annual meeting of the European Society of Anaesthesia in Vienna, Austria from May 28–31, 2005. 
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionFundingAcknowledgementsReferences |
|---|
1 Pearson SD, Goulart Fisher D, Lee TH. Critical pathways as a strategy for improving care: problems and potential. Ann Intern Med (1995) 123:941–8.
2 Morris J, Cook TM. Rapid sequence induction: a national survey of practice. Anaesthesia (2001) 56:1090–7.[Web of Science][Medline]
3 Thwaites AJ, Rice CP, Smith I. Rapid sequence induction: a questionnaire survey of its routine conduct and continued management during a failed intubation. Anaesthesia (1999) 54:376–81.[CrossRef][Web of Science][Medline]
4 Brown GW, Russell IF. A survey of anaesthesia for caesarean section. Int J Obstet Anesth (1995) 4:214–8.[CrossRef][Medline]
5 Krosnick JA. Survey research. Annu Rev Psychol (1999) 50:537–67.[CrossRef][Web of Science][Medline]
6 Gaba DM, Howard SK, Flanagan B, et al. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology (1998) 89:8–18.[CrossRef][Web of Science][Medline]
7 Fletcher G, Flin R, McGeorge P, et al. Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth (2003) 90:580–8.
8 Forrest FC, Taylor MA, Postlethwaite K, Aspinall R. Use of a high-fidelity simulator to develop testing of the technical performance of novice anaesthetists. Br J Anaesth (2002) 88:338–44.
9 Jacobsen J, Lindekaer AL, Ostergaard HT, et al. Management of anaphylactic shock evaluated using a full-scale anaesthesia simulator. Acta Anaesthesiol Scand (2001) 45:315–9.[CrossRef][Web of Science][Medline]
10 Grube C, Volk S, Zausig Y, Graf BM. Changing Culture—Simulatortraining als Mittel zur erhohten Patientensicherheit. Bericht vom International Meeting on Medical Simulation, Scottsdale, 12–14.1.2001 [Changing culture—simulator-training as a method to improve patient safety. Report on an international meeting on medical simulation. Scottsdale: January 12–14, 2001]. Anaesthesist (2001) 50:358–62.[CrossRef][Medline]
11 Bell MD. Routine pre-oxygenation—a new ‘minimum standard’ of care? Anaesthesia (2004) 59:943–5.[CrossRef][Web of Science][Medline]
12 Altermatt FR, Munoz HR, Delfino AE, Cortinez LI. Pre-oxygenation in the obese patient: effects of position on tolerance to apnoea. Br J Anaesth (2005) 95:706–9.
13 Fassoulaki A, Sarantopoulos C, Vassiliou M. Performance of the anesthesiologists in 12 Departments of Anesthesia in Greece. Acta Anaesthesiol Belg (1995) 46:79–86.[Medline]
14 Schlesinger S, Blanchfield D. Modified rapid-sequence induction of anesthesia: a survey of current clinical practice. AANA J (2001) 69:291–8.[Medline]
15 Devitt JH, Kurrek MM, Cohen MM, Cleave Hogg D. The validity of performance assessments using simulation. Anesthesiology (2001) 95:36–42.[CrossRef][Web of Science][Medline]
16 Byrne AJ, Greaves JD. Assessment instruments used during anaesthetic simulation: review of published studies. Br J Anaesth (2001) 86:445–50.
17 Woodward CA. Questionnaire construction and question writing for research in medical education. Med Educ (1988) 22:345–63.[Web of Science][Medline]
18 Baddeley AD. Human memory. In: Theory and Practice (1997) Hove: Psychology Press.
19 Blum RH, Raemer DB, Carroll JS, et al. Crisis resource management training for an anaesthesia faculty: a new approach to continuing education. Med Educ (2004) 38:45–55.[Web of Science][Medline]
20 Hotchkiss MA, Biddle C, Fallacaro M. Assessing the authenticity of the human simulation experience in anesthesiology. AANA J (2002) 70:470–3.[Medline]
21 Georgiou M, Parlapani A, Argiriadou H, et al. Sufentanil or clonidine for blunting the increase in intraocular pressure during rapid-sequence induction. Eur J Anaesthesiol (2002) 19:819–22.[CrossRef][Web of Science][Medline]
22 Dobson AP, McCluskey A, Meakin G, Baker RD. Effective time to satisfactory intubation conditions after administration of rocuronium in adults. Comparison of propofol and thiopentone for rapid sequence induction of anaesthesia. Anaesthesia (1999) 54:172–6.[CrossRef][Web of Science][Medline]
23 Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology (2005) 103:241–8.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  Y. A. Zausig, H. Kunzig, G. Roth, and B. M. Graf Anaesthetist's opinions on simulator-based training in continuing education Br. J. Anaesth., November 1, 2009; 103(5): 774 - 775. [Full Text] [PDF]
|
|
|
|
 |
|
 J. M. Rodriguez-Paz, L. J. Mark, K. R. Herzer, J. D. Michelson, K. L. Grogan, J. Herman, D. Hunt, L. Wardlow, E. P. Armour, and P. J. Pronovost A Novel Process for Introducing a New Intraoperative Program: A Multidisciplinary Paradigm for Mitigating Hazards and Improving Patient Safety Anesth. Analg., January 1, 2009; 108(1): 202 - 210. [Abstract] [Full Text] [PDF]
|
|
E-letters:
Read all E-letters
- Universal Patient-Protection Algorithm
- John George George Cherian, et al.
- British Journal of Anaesthesia, 6 Nov 2007 [Full text]
- Are Simulators an effective tool for teaching and training?
- Dr. Oswald D'Mello; DA, FRCA
- British Journal of Anaesthesia, 20 Nov 2007 [Full text]
- Expansion of standard operating procedures
- York A. Zausig et al.
- British Journal of Anaesthesia, 22 Jan 2008 [Full text]
- Need of further investigation in simulation
- York A. Zausig et al.
- British Journal of Anaesthesia, 29 Feb 2008 [Full text]
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||