Abstract
Improvements in safety culture have been postulated as one of the mechanisms underlying the association between the introduction of the World Health Organisation (WHO) Surgical Safety Checklist with perioperative briefings and debriefings, and enhanced patient outcomes. The 5 Steps to Safer Surgery (5SSS) incorporates pre-list briefings, the three steps of the WHO Surgical Safety Checklist (SSC) and post-list debriefings in one framework. We aimed to identify any changes in safety culture associated with the introduction of the 5SSS in orthopaedic operating theatres.
We assessed the safety culture in the elective orthopaedic theatres of a large UK teaching hospital before and after introduction of the 5SSS using a modified version of the Safety Attitude Questionnaire - Operating Room (SAQ-OR). Primary outcome measures were pre-post intervention changes in the six safety culture domains of the SAQ-OR. We also analysed changes in responses to two items regarding perioperative briefings.
The SAQ-OR survey response rate was 80% (60/75) at baseline and 74% (53/72) one yr later. There were significant improvements in both the reported frequency (P<0.001) and perceived importance (P=0.018) of briefings, and in five of the six safety culture domain scores (Working Conditions, Perceptions of Management, Job Satisfaction, Safety Climate and Teamwork Climate) of the SAQ-OR (P<0.001 in all cases). Scores in the sixth domain (Stress Recognition) decreased significantly (P=0.028).
Implementation of the 5SSS was associated with a significant improvement in the safety culture of elective orthopaedic operating theatres.
Efforts at improving the safety culture in hospitals are welcomed by most clinical staff.
Structured checklists and teamwork in healthcare are linked with improved patient outcome.
This study found the introduction of the Surgical Safety Checklist along with debriefings enhanced the safety culture of an orthopaedic theatre team.
It is more than a decade since the UK Department of Health noted that ‘Safety cultures can have a positive and quantifiable impact on the performance of organisations’,1 and the Institute of Medicine declared that ‘the healthcare organization must develop a culture of safety’.2 Since this time increasing evidence has been published supporting an association between improved safety culture and better patient outcomes.3–8
Subsequent to these reports, assessment of safety culture has been recommended as part of the quality framework applied to healthcare organisations.9 Safety culture measurement has also been recommended for the routine assessment of hospitals in the United States and to assess the effect of targeted interventions designed to improve safety.10,11 Two such interventions are the World Health Organisation (WHO) Surgical Safety Checklist (SSC) and perioperative briefings and debriefings.12
In 2009 Haynes and colleagues13 published the results of a study demonstrating a reduction in morbidity and mortality associated with the introduction of the SSC. They acknowledged that the mechanism for improvement is multifactorial, involving theatre systems and processes and a change in the culture within the operative teams. They also reported that the individual steps of the checklist were frequently omitted. This raises the question of how much of the effect is as a result of the checklist itself and how much was attributable to a change in culture caused by its introduction. The importance of the cultural shift was strengthened in a subsequent study, which found no correlation between the number of items of the checklist completed and the improvement in outcomes.5 However recent research has suggested a more limited improvement in the safety culture associated with the introduction of the SSC.14
Preoperative briefings have been shown to be positively associated with the attitudes of staff towards patient safety, to improve communication and team-working, and to reduce unexpected delays.8,15–18 These adaptive changes influence the safety culture and support the possibility that the shift in safety culture, rather than the technical changes associated with the introduction of a checklist, is important in the improvement of perioperative outcomes.
After pilot studies, Patient Safety First recommended combining preoperative list briefings, the three steps of the SSC and postoperative debriefings into the 5 Steps for Safer Surgery (5SSS). There is a limited amount of research into what effect the 5SSS has in changing culture within healthcare organisations.19 The main aim of this study was to assess whether there was a change in operating theatre safety climate associated with the introduction of the 5SSS.
Methods
The Cornwall and Plymouth Ethics Committee was consulted and deemed that ethical approval was not required.
Study design
This was a prospective study using pre- and post-intervention surveys in an intervention group. The primary outcome measure was the pre-post change in safety culture in operating theatre staff associated with the introduction of the 5SSS.
Study population
The study was conducted in the four elective orthopaedic theatres of Plymouth Hospitals NHS Trust, UK, a tertiary care hospital with more than 900 beds. The elective orthopaedic theatres were selected, as this was the largest functional unit within the theatre suite. This gave the maximum possible sample size for a single unit. No sample size calculation was performed before undertaking the study. The target population comprised all staff working in the elective orthopaedic operating theatres including surgeons, anaesthetists and nursing staff.
SAQ-OR
The survey instrument used was the SAQ-OR.20 This version has been adapted for the theatre setting from the original Safety Attitude Questionnaire and was chosen because the psychometric properties have been extensively tested and validated and it has been widely used in the United States and Europe.21
The SAQ-OR (Supplementary Appendix 1 Fig.) comprises 59 items that use a 5-point Likert response scale (disagree strongly, disagree slightly, neutral, agree slightly and agree strongly). These items include six subscales measuring particular safety cultural domains – job satisfaction (question numbers 2,8,15,29 and 41), perceptions of management (question numbers 10,17, 18 and 26), teamwork climate (question numbers 3, 24, 30, 34, 35 and 38), safety climate (question numbers 4, 5, 11, 12, 20, 21 and 28), stress recognition (question numbers 16, 25, 31 and 32) and working conditions (question numbers 6, 7, 22 and 42).
Two items that were not in the domain subscales, but were of particular interest in this study, were ‘Briefing theatre personnel before a surgical procedure is important for patient safety’ and ‘Briefings are common in the theatre’. The SAQ-OR also requests basic demographic data.
Before administration, the wording of the questions was reviewed and minor linguistic modifications were made to ensure that United Kingdom participants could understand the questions. Examples of such modifications include ‘physicians’ being changed to ‘doctors’ and ‘OR’ to ‘theatres’.
Administration of surveys
The surveys were administered 12 months apart with the first survey immediately before implementation of the 5SSS. The SAQ-OR was hand delivered by the investigator to all staff groups in the four elective orthopaedic operating theatres of the hospital over a three-week period. An addressed envelope was attached for the return of the questionnaire via the internal hospital mail system. Identical methodology was used for the post-intervention survey in the same theatres.
Anonymity of the respondents was guaranteed and maintained.
Intervention
The 5SSS were introduced into the orthopaedic theatres at the end of April 2009 as the first stage of an incremental roll-out throughout the hospital. Theatre staff were trained in the use of the 5SSS via multimedia small group sessions led by patient safety champions. Laminated copies of the checklist were placed on the walls of the relevant anaesthetic rooms and theatres.
Data analysis
Data from the survey were entered into Microsoft Excel and analysed using SPSS version 20. We calculated response rates in the pre- and post-intervention samples and tested for differences in their demographic and professional profiles (age, gender, job role, experience in specialty and length of service in current hospital) using χ2 and t-tests as appropriate. Following the SAQ Scale Computation Instructions, the individual item responses were converted to a numerical scale – Strongly disagree=1, disagree slightly=2, etc, and all negatively worded items (item 12 ‘In the theatre it is difficult to discuss errors’ and item 24 ‘In the theatres here, it is difficult to speak up if I perceive a problem with patient care’) were reverse scored. The domain scores were calculated by taking the mean score across the items in the domain subscale, subtracting 1 and multiplying by 25 to give scores on a 0–100 scale. Differences between the pre- and post-intervention safety climate domain scores were tested using two-sided unpaired t-tests and differences in the two briefing item scores were tested using Mann-Whitney U-tests because of non-normality. We controlled for multiple comparisons using the Holm-Bonferroni procedure to ensure a maximum family-wise error rate of 5%.22 Respondents were regarded as having a ‘positive’ response to an individual item if they returned an item score of 4 or 5 (after reverse scoring where necessary) and as having a ‘positive’ domain score if that score was 75 or greater. The percentages of respondents in the pre- and post-intervention samples who returned a positive climate score in each domain were compared graphically, as were the percentages of positive responses to the two briefing items and each of the items within the climate score scales.
Results
The overall response rates were 80% (60/75) in the pre-test survey and 74% (53/72) in the post-test survey. It was not possible to identify how many staff completed the questionnaire on both occasions because of the anonymity of the survey.
A breakdown of respondents' job roles in the two yrs of the survey is shown in Table 1. The mix of job roles did not change significantly between the two yrs (χ2 test, P=0.90). The percentage of female respondents in the sample rose from 36% in 2009 to 45% in 2010, though the difference was not statistically significant (χ2 test, P=0.31). Further evidence of similarity between the 2009 and 2010 respondent samples was furnished by independent-samples t-tests of the difference in mean ages (P=0.81), mean experience in specialty (P=0.86), and mean length of service in the current hospital (P=0.89). A full table of demographic data broken down by job role is given in Supplementary Appendix 2 Table.
Respondents by job role and group
| Pre-intervention | Post-intervention | |
|---|---|---|
| Health care assistant | 4 | 5 |
| Nurse | 13 | 13 |
| Operating department assistant | 11 | 7 |
| Anaesthetist | 16 | 15 |
| Surgeon | 16 | 12 |
| Total | 60 | 52 |
| Pre-intervention | Post-intervention | |
|---|---|---|
| Health care assistant | 4 | 5 |
| Nurse | 13 | 13 |
| Operating department assistant | 11 | 7 |
| Anaesthetist | 16 | 15 |
| Surgeon | 16 | 12 |
| Total | 60 | 52 |
Respondents by job role and group
| Pre-intervention | Post-intervention | |
|---|---|---|
| Health care assistant | 4 | 5 |
| Nurse | 13 | 13 |
| Operating department assistant | 11 | 7 |
| Anaesthetist | 16 | 15 |
| Surgeon | 16 | 12 |
| Total | 60 | 52 |
| Pre-intervention | Post-intervention | |
|---|---|---|
| Health care assistant | 4 | 5 |
| Nurse | 13 | 13 |
| Operating department assistant | 11 | 7 |
| Anaesthetist | 16 | 15 |
| Surgeon | 16 | 12 |
| Total | 60 | 52 |
A yr after the introduction of the 5SSS there was a significant improvement in how important staff believed briefings to be and in how common they were (Table 2). There were significant improvements in scores for five of the six domains of the SAQ-OR - teamwork climate, safety climate, job satisfaction, perceptions of management and working conditions but a significant deterioration in the perception of stress recognition by staff (Table 2). All eight of these changes remained statistically significant after applying the Holm-Bonferroni procedure. The corresponding changes in the percentages of ‘positive’ domain scores are shown in Fig. 1, while Supplementary Appendix 3 Fig. contains graphical comparisons of the percentages of ‘positive’ responses to individual items.
Pre- and post-5SSS mean (SD), mean differences (95% confidence intervals) and associated P values for briefing item and SAQ-OR domain scores. *95% confidence intervals for briefing items obtained by bootstrapping; †Mann-Whitney U-test for briefing items, independent samples t-test for safety climate scores; ‡Briefing item scores are measured on a 1–5 scale; ¶Domain scores are measured on a 0–100 scale
| Pre-5SSS | Post-5SSS | Difference* | P value† | |
|---|---|---|---|---|
| Briefing items‡ | ||||
| Briefing important | 4.53 (0.85) | 4.87 (0.39) | 0.34 (0.11, 0.59) | 0.018 |
| Briefing common | 2.22 (1.12) | 4.55 (0.57) | 2.33 (1.99, 2.63) | <0.001 |
| Safety attitude domain scores¶ | ||||
| Working conditions | 43.0 (22.9) | 63.5 (17.4) | 20.5 (12.8, 28.2) | <0.001 |
| Perception of management | 31.4 (17.7) | 45.1 (18.5) | 13.8 (7.0, 20.5) | <0.001 |
| Stress recognition | 77.0 (20.9) | 67.8 (23.2) | –9.2 (–17.4, –1.0) | 0.028 |
| Job satisfaction | 48.9 (21.8) | 64.4 (18.1) | 15.5 (8.0, 23.0) | <0.001 |
| Safety climate | 57.3 (16.4) | 67.4 (13.8) | 10.1 (4.4, 15.8) | 0.001 |
| Teamwork climate | 58.4 (17.3) | 71.2 (14.4) | 12.8 (6.8, 18.8) | <0.001 |
| Pre-5SSS | Post-5SSS | Difference* | P value† | |
|---|---|---|---|---|
| Briefing items‡ | ||||
| Briefing important | 4.53 (0.85) | 4.87 (0.39) | 0.34 (0.11, 0.59) | 0.018 |
| Briefing common | 2.22 (1.12) | 4.55 (0.57) | 2.33 (1.99, 2.63) | <0.001 |
| Safety attitude domain scores¶ | ||||
| Working conditions | 43.0 (22.9) | 63.5 (17.4) | 20.5 (12.8, 28.2) | <0.001 |
| Perception of management | 31.4 (17.7) | 45.1 (18.5) | 13.8 (7.0, 20.5) | <0.001 |
| Stress recognition | 77.0 (20.9) | 67.8 (23.2) | –9.2 (–17.4, –1.0) | 0.028 |
| Job satisfaction | 48.9 (21.8) | 64.4 (18.1) | 15.5 (8.0, 23.0) | <0.001 |
| Safety climate | 57.3 (16.4) | 67.4 (13.8) | 10.1 (4.4, 15.8) | 0.001 |
| Teamwork climate | 58.4 (17.3) | 71.2 (14.4) | 12.8 (6.8, 18.8) | <0.001 |
Pre- and post-5SSS mean (SD), mean differences (95% confidence intervals) and associated P values for briefing item and SAQ-OR domain scores. *95% confidence intervals for briefing items obtained by bootstrapping; †Mann-Whitney U-test for briefing items, independent samples t-test for safety climate scores; ‡Briefing item scores are measured on a 1–5 scale; ¶Domain scores are measured on a 0–100 scale
| Pre-5SSS | Post-5SSS | Difference* | P value† | |
|---|---|---|---|---|
| Briefing items‡ | ||||
| Briefing important | 4.53 (0.85) | 4.87 (0.39) | 0.34 (0.11, 0.59) | 0.018 |
| Briefing common | 2.22 (1.12) | 4.55 (0.57) | 2.33 (1.99, 2.63) | <0.001 |
| Safety attitude domain scores¶ | ||||
| Working conditions | 43.0 (22.9) | 63.5 (17.4) | 20.5 (12.8, 28.2) | <0.001 |
| Perception of management | 31.4 (17.7) | 45.1 (18.5) | 13.8 (7.0, 20.5) | <0.001 |
| Stress recognition | 77.0 (20.9) | 67.8 (23.2) | –9.2 (–17.4, –1.0) | 0.028 |
| Job satisfaction | 48.9 (21.8) | 64.4 (18.1) | 15.5 (8.0, 23.0) | <0.001 |
| Safety climate | 57.3 (16.4) | 67.4 (13.8) | 10.1 (4.4, 15.8) | 0.001 |
| Teamwork climate | 58.4 (17.3) | 71.2 (14.4) | 12.8 (6.8, 18.8) | <0.001 |
| Pre-5SSS | Post-5SSS | Difference* | P value† | |
|---|---|---|---|---|
| Briefing items‡ | ||||
| Briefing important | 4.53 (0.85) | 4.87 (0.39) | 0.34 (0.11, 0.59) | 0.018 |
| Briefing common | 2.22 (1.12) | 4.55 (0.57) | 2.33 (1.99, 2.63) | <0.001 |
| Safety attitude domain scores¶ | ||||
| Working conditions | 43.0 (22.9) | 63.5 (17.4) | 20.5 (12.8, 28.2) | <0.001 |
| Perception of management | 31.4 (17.7) | 45.1 (18.5) | 13.8 (7.0, 20.5) | <0.001 |
| Stress recognition | 77.0 (20.9) | 67.8 (23.2) | –9.2 (–17.4, –1.0) | 0.028 |
| Job satisfaction | 48.9 (21.8) | 64.4 (18.1) | 15.5 (8.0, 23.0) | <0.001 |
| Safety climate | 57.3 (16.4) | 67.4 (13.8) | 10.1 (4.4, 15.8) | 0.001 |
| Teamwork climate | 58.4 (17.3) | 71.2 (14.4) | 12.8 (6.8, 18.8) | <0.001 |
Percentage of respondents reporting a positive score in each domain of the SAQ-OR pre- and post-intervention.
Discussion
We believe that there has been no comprehensive assessment of the effects of the introduction of the 5SSS on safety culture. This paper demonstrates that the introduction of the 5SSS in the elective orthopaedic theatres of a large teaching hospital was associated with significant improvements in the frequency with which briefings occurred and the importance that staff attached to them, and in five of the six domains of the SAQ-OR. These were teamwork climate, safety climate, job satisfaction, perceptions of management and working conditions. There was a significant worsening in the climate score for stress recognition.
The increase in the number of positive responses to the question ‘Briefings are common in this theatre’ indicates that there was good compliance with the 5SSS protocol put in place for the study. In addition, staff indicated higher values with respect to the perceived importance of briefings, which is likely to support a cultural change.
Previous studies have shown an improvement in the safety culture associated with the introduction of briefings,8 but only limited impact regarding any change in safety climate associated with the introduction of the SSC.5 In the most comprehensive study published to date the intervention group included different surgical specialties but did not address whether there are differences between the functional units within the hospital nor whether these showed an improvement.14 Previous studies looking at the variation between hospitals have demonstrated a variation of 18–100% of respondents reporting a positive safety climate.23 It is suggested that the variation between different individual units within a hospital may be as great as that between hospitals. It is unclear whether examining the results of all surgical specialities as a single entity masks important changes within the functional units of the different surgical specialities. Identifying the functional units and the safety culture within them is crucial in allowing improvement work to be designed and targeted at the staff within each unit.
While the domain scores appear low overall even after the introduction of the 5SSS, they fall within the reported range. However it is not the absolute value attained that demonstrates the importance of an individual intervention, but the improvement from baseline. Pronovost and colleagues24 have suggested that an alternative means of assessing the success of an intervention, such as the 5SSS is to aim for a 10 percentage point improvement in the climate scores. Interpretation of the results in this way would indicate that there had been a meaningful increase in climate scores in teamwork climate, safety climate, job satisfaction and working conditions but not in the perception of management. This percentage point increase is more readily understood by staff than statistical analysis and maybe more useful in setting goals for safety culture improvement.
The worsening in the perception of stress by staff may have been a corollary of the improvements in the other domains causing a perceived improved ability to cope with stress. Indeed, improved teamwork has been shown to mitigate some of the effects of stress on performance.25 The minor improvement in the perception of management is likely to reflect ongoing problems in how management is perceived to support front line clinical areas. The perception of management is likely to be of growing importance as efficiency pressures continue to influence decision making and these factors should intensify the consideration of interventions to ameliorate this (e.g. safety walkrounds).
The results show an improvement in aspects of the safety culture associated with the introduction of the 5SSS but do not explain the reasons for the improvement in the domains of safety culture. Haynes and colleagues 13 reported that the benefits that they observed in the reduction in morbidity and mortality following the introduction of the WHO checklist did not correlate with the improvements in the ‘specific processes of care’ (i.e. completion of the individual steps on the checklist). They suggest that the change in outcome may be attributable to a different process that enhances team functioning and safety climate. The transformation may have come about as the process of completing the 5SSS facilitates clear communication within each team and encourages the development of a flattened hierarchy allowing concerns regarding patient safety to be raised. An improvement in the safety culture as a mechanism for improved patient outcomes would fit with previous studies that have demonstrated associations between better safety climates and lower rates of adverse patient safety indicators.3–8
If it is a cultural shift that is important in improving patient outcomes, then the engagement of frontline staff and their commitment to the process may be more important than the completion of individual steps on the checklist. Understanding the factors that impact on the benefits to patients is crucial in understanding where the emphasis should be on the implementation and success of the 5SSS. It follows that the adaptive changes in the unit culture brought about through the introduction of these targeted interventions, although taking longer to achieve, may have a greater impact and bring about longer lasting improvements than simply concentrating on the technical aspects of the checklist. The engagement of frontline staff is crucial in getting them to support the introduction of the checklist and local adaptation was recommended by the National Patient Safety Agency to try to help to overcome some of the resistance that might otherwise ensue.26 This commitment to the process rather than a focus on completing each step is likely to drive the improvements in culture and safety.
There are several limitations to this study. It is impossible to identify if the improvement in safety climate was because of either the introduction of briefings or the WHO SSC as both were introduced together. What is clear is the importance that staff in orthopaedic theatres attached to pre-list briefings and that their importance increased after their formal introduction.
Changes of staff in the orthopaedic theatres could not be controlled for within the study and this may have contributed to the observed change in safety climate. However it is likely that staff that joined a theatre adopted the behaviours of that unit rather than imposing their own standards on the theatre. Another potential limitation is that the study was conducted in elective theatres in one speciality and in one hospital. However we believe that the results would be generalizable to other hospitals and specialties. Future research has been suggested to investigate the effect that conducting safety climate surveys has on safety culture and we did not control for this within the study.6 We believe however, that the effect of the relatively short intervention of completing the SAQ-OR would be less than any effect from the much higher intensity and daily use of the 5SSS.
The study did not attempt to assess an improvement in patient outcomes and further work is required to investigate the effect on patient outcomes associated with the introduction of the 5 Steps to Safer Surgery and to assess the effect of briefings and debriefings on safety culture.
Conclusion
Our study showed significant improvements in safety culture after the introduction of the 5SSS in elective orthopaedic theatres. We would expect the results to generalise to other theatres and hospitals. The adaptive changes associated with the introduction of the 5SSS may be the crucial factor in improving patient safety, rather than simple adherence to a tick-box completion of the checklist. We recommend that safety climate should be routinely measured to assess the impact of interventions on the safety climate.
Authors' contributions
M.R.H. contributed to the study design and conception, administration of the Safety Attitude Questionnaire, analysis and interpretation of the data, first draft of the paper and critical revision. M.L.A. and T.C.E.G. contributed to the introduction of the 5 Steps to Safer Surgery to the orthopaedic theatres and critical revision of the article. M.J.R. contributed to the analysis and interpretation of the data and critical revision of the article. All authors approved the version of the manuscript for publication.
Supplementary material
Supplementary material is available at British Journal of Anaesthesia online.
Declaration of interest
MRH: This work formed the basis of the dissertation submitted for the MSc in Quality and Safety in Healthcare, Imperial College.
Acknowledgements
Thanks to Nick Sevdalis for his support throughout this project.
References
