BJA Advance Access published online on May 16, 2007
British Journal of Anaesthesia, doi:10.1093/bja/aem117
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Quality of anaesthesia-related information accessed via Internet searches
Pôle d'Anesthésie Réanimation, CHU d'Angers, 49933 Angers Cedex 9, France
* Corresponding author. E-mail: lbeydon.angers{at}invivo.edu
Accepted for publication March 10, 2007.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionAppendixReferences |
|---|
Background: We conducted a study to examine the quality and stability of information available from the Internet on four anaesthesia-related topics.
Methods: In January 2006, we searched using four key words (porphyria, scleroderma, transfusion risk, and epidural analgesia risk) with five search engines (Google®, HotBot®, AltaVista®, Excite®, and Yahoo®). We used a published scoring system (NetScoring®) to evaluate the first 15 sites identified by each of these 20 searches. We also used a simple four-point scale to assess the first 100 sites in the Google® search on one of our four topics (‘epidural analgesia risk’). In November 2006, we conducted a second evaluation, using three search engines (Google®, AltaVista®, and Yahoo®) with 14 synonyms for ‘epidural analgesia risk’.
Results: The five search engines performed similarly. NetScoring® scores were lower for transfusion risk (P<0.001). One or more high-quality sites was identified consistently among the first 15 sites in each search. Quality scored using the simple scale correlated closely with medical content and design by NetScoring® and with the number of references (P<0.05). Synonyms of ‘epidural analgesia risk’ yielded similar results. The quality of accessed information improved somewhat over the 11 month period with Yahoo® and AltaVista®, but declined with Google®.
Conclusions: The Internet is a valuable tool for obtaining medical information, but the quality of websites varies between different topics. A simple rating scale may facilitate the quality scoring on individual websites. Differences in precise search terms used for a given topic did not appear to affect the quality of the information obtained.
Keywords: assessment, preanaesthetic; computers; education
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
The Internet has made a huge amount of information readily available to both the public and the medical community. In 2001, the number of health-related websites was estimated at more than 100 000.1 The quality of the information available on the web is variable.2 Health-related sites range from peer-reviewed medical journals and not-for-profit physician organizations to commercial sites and weblogs (online personal journals). Several studies have assessed the quality of health-related information on specific topics available on the Internet; for example, treatments for depression,3 laparoscopy,4 fever in children,5 6 diarrhoea,7 menorrhagia,8 asthma,9 and neurosurgery.10 Three studies of Internet information about anaesthesia produced limited data.11–13 There is no consensus about the quality of the first 15 sites in search retrieval lists, to which most Internet users confine their searches.14 Other issues include whether the quality varies according to the topic, according to the level of transparency about the source, over time, and whether the search terms used influence the quality of accessed information. A simple subjective score that reliably evaluates the quality of health-related information would be useful. Available scoring systems are complex and time-consuming, and therefore better suited to research than to information access by individual physicians and patients.
Our study focused on issues of potential interest to both patients and anaesthesiologists. Our primary objective was to determine the quality of the first 15 websites accessed using anaesthesia-related search terms and five widely used search engines. Website transparency and design were assessed using an established scoring system (NetScoring®).15 Medical content quality was evaluated by comparison with reference publications. Several search engines were compared. Our secondary objective was to determine whether a simple scale showed promise as a faster tool than NetScoring® for evaluating the quality of health-related information on the Internet. Finally, we evaluated the stability of Internet-accessed information over time, and we compared the quality of websites retrieved using roughly synonymous search terms.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
Two anaesthesiologists independently conducted two search sessions 11 months apart, in January and November 2006.
January 2006 session: quality of information retrieved by five search engines about four anaesthesia-related topics
Search strategy
On a single day in January 2006, we used four terms (‘porphyria’, ‘scleroderma’, ‘transfusion risk’, and ‘epidural analgesia risk’) to look for information using the five search engines16 (Yahoo®, Google®, AltaVista®, HotBot®, and Excite®). Thus, 20 searches were conducted in all. We selected these four terms as relevant to common concerns for patients or anaesthesiologists. Furthermore, the Internet may be a valuable source of information when time is short, for instance before emergency surgery in a patient with porphyria or scleroderma. We appraised the first 15 sites retrieved with each term and each search engine, which yielded 300 sites in all. Duplicates were skipped.
NetScoring® evaluation of the first 15 sites
In a preliminary evaluation, we found that most anaesthesiologists read only the first 15 results of their searches, in agreement with data reported by Eysenbach and Köhler.14 Therefore, for this study, two anaesthesiologists independently assessed the first 15 sites retrieved by each of the 20 searches using main NetScoring® criteria. NetScoring® was developed for medical websites,1517 and includes scores for transparency, design, and accuracy of the medical content (for details, see Appendix). Medical content was scored by comparison with up-to-date information obtained from published recommendations and medical textbooks.18–21 The transparency, design, and medical content scores are added to obtain a global score, which was expressed as the percentage of the maximum achievable score. The presence of bibliographic references supporting the information in the site was noted. Incorrect or unbalanced (e.g. simplistic) information led to subtraction of 1 point from the medical content quality score. For each site, we compared the global scores assigned by the two investigators. When the difference was greater than 10%, the two investigators appraised the site together in order to reach an agreement on the score.
Simplified evaluation of the first 100 sites for ‘epidural analgesia risk’ using a four-point scale
The first 100 sites retrieved by Google® with the term ‘epidural analgesia risk’' were assessed by each of the two anaesthesiologists in a separate session, using three criteria, each scored separately from 1 to 4 (1, poor; 2, fair; 3, good; and 4, professional): global quality of the website (simple global), quality of the medical content (simple medical content), and quality of the website design (simple design). The number of bibliographic references in each website was also recorded. We also recorded whether the site was created by a nurse, physician, healthcare institution, organization, publisher, or healthcare product manufacturer; or whether the site was an article by a journalist, an article in a medical journal, or an abstract from a medical journal without free access to the full text. Duplicate sites were not used for the analysis.
Comparison of sites retrieved by two searches using the same term 10 days apart
We repeated the ‘epidural analgesia risk’ search on Google® 10 days after the first search, and we recorded the number of discrepancies in retrieved sites.
November 2006 session: quality of information retrieved by three search engines (Google®, Yahoo®, and AltaVista®) using 14 synonyms for ‘risk of epidural analgesia’
In this second session, only medical content was assessed. The two main goals of this session were to assess stability of medical information over an 11 month period and to compare results across synonymous searching terms. We chose ‘risk of epidural analgesia’ for this session and used the 14 terms listed in Table 2 with three search engines (Google®, Yahoo®, and AltaVista®). Within each search, defined as the combination of a search term used with a search engine, we considered the first 15 websites, which yielded 210 sites with each engine. Within each search, each duplicate was skipped and replaced with the next site on the list. Across searches, repeats were kept, so that the total number of sites was 630. Medical content was assessed, for each of the first 15 sites retrieved by each search.
Analysis strategy and statistics
We assessed the quality of websites retrieved using different search engines and search terms, using the first 15 sites retrieved by each search. NetScoring® results and simple scores are reported as medians and ranges. We compared the median score obtained with each search term for transparency, design and medical content, pooling data from all the five search engines. To evaluate the impact of the search strategy, we compared the median number of sites with medical-content scores 
95% across search terms, with the five engines pooled. Similarly, to evaluate the impact of the search engine, we compared the median number of sites with medical-content scores 95% across search engines, with all search terms pooled. Medians were compared using Kruskal–Wallis test with Dunn's test to compare groups when the overall P-value was <0.05. Finally, we assessed the simple subjective score using the first 100 websites retrieved by Google® (from which duplicates were excluded) with the term ‘epidural analgesia risk’. To this end, we first used the first 15 sites to evaluate linear correlations between the simple score on the global criterion (simple global), the global Netscoring® value, the simple score on the medical content criterion (simple medical content), and the score for medical content accuracy. To evaluate internal consistency of the simple scale, we used the first 100 websites (after excluding duplicates) to assess correlations linking simple design, simple medical content, and number of bibliographic references to simple global.
We used the data from the November 2006 session to look for differences across search terms and search engines in the number of sites with medical-content scores 95%. The Kruskal–Wallis test with Dunn's test was used.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
January 2006 session
NetScoring® evaluation of the first 15 sites
The global NetScoring® value was not significantly different across search engines. Therefore, we pooled scores for all the five search engines for the comparison of subscores (transparency, design, and medical content) across the four search terms. The medical content scores for the sites on transfusion risk were significantly lower than those for the sites on the other three topics (Table 1). The sites retrieved by the two technical terms (‘porphyria’ and ‘scleroderma’) probably scored higher on medical content because nearly all of them were created by professionals. The transfusion risk websites also had significantly lower scores for transparency and design. In each of the 20 searches (four terms and five engines), the first 15 sites included at least one site with a medical content score >95%. When all the data were pooled, we found a small but significant decrease in medical content quality with increasing rank (n=300, r=0.14, P<0.05).
|
Simplified evaluation of ‘epidural analgesia risk’ sites using a simple scale to assess three items
Five sites were duplicates and were excluded from the analysis, leaving 95 sites for the evaluation. Among them, 45% were authored by health professionals (Fig. 1). Our evaluation of the first 15 sites showed that simple global correlated with the global Netscoring® value (r2=0.61, P<0.001) and that the simple medical content moderately correlated with the medical content accuracy score determined in the January 2006 session (r2=0.50, P<0.01). In the first 95 sites, simple medical content, simple design, and the number of bibliographical references each correlated significantly with simple global (r2=0.4, 0.67, and 0.18, respectively; P<0.001 for all correlations).
|
Comparison of sites retrieved by two searches using the same term 10 days apart
We found a 16% discrepancy between the two searches. The first difference occurred at the 16th site.
November 2006 session
Table 2 lists the median and range for the medical content scores of the first 15 websites retrieved using 14 synonyms for ‘epidural analgesia risk’ and three search engines. Pooling all search terms, we found that the number of websites having medical content scores 95% was significantly lower (P<0.05) with Google® than with AltaVista® and Yahoo®. [Kruskal–Wallis one-way ANOVA on rank-test with multiple comparison procedures (Dunn's method)].
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
Our main findings are that the first 15 sites retrieved using health-related terms contained at least one site with high-quality medical content (score >95%), that quality decreased with increasing rank in the search retrieval list, and that a simple subjective score is effective for rating sites. Furthermore, we noted a decline in quality of information retrieved by Google®, as assessed using our criteria, contrasting with the stability of information retrieved by Yahoo® and AltaVista®. Finally, synonyms retrieved sites of comparable medical content quality.
We chose four terms that reflected a broad range of anaesthesia-related issues. This choice of terms gives our study a broader scope than many previously published studies on medical information accessed via the Internet. For instance, previous studies of Internet information on anaesthesia focused on a single topic.1113 Internet- available information retrieved using two of these terms, porphyria and scleroderma, may be of special interest to anaesthesiologists looking for information before emergency surgery in patients with these rare diseases. The other two other terms related to widely performed procedures about which anaesthesiologists are likely to be fully conversant yet interested in finding material to support the information they provide to their patients.
We used the five most popular search engines. Identifying differences in site quality across engines may allow physicians to give valuable advice to patients about which engines are best.
We conducted two assessments 11-months apart. Because the Internet is a fast-growing and unregulated tool, stability over time of the information it contains is a major concern. No study can claim to supply permanently valid information about medical content on the Internet, and survival of truth in this field may well be briefer than in other areas of medicine. Clearly, studies of medical content on the Internet will have to be repeated at closely spaced intervals, and the development by the medical community of an Internet monitoring system may deserve consideration. Interestingly, we found that medical content accuracy, as assessed in our study, did not differ significantly across search engines in January 2006, whereas Google® performed significantly less well than Yahoo® and AltaVista® in November 2006.
We evaluated 14 synonyms for one of our areas of interest (epidural analgesia), to evaluate whether the choice of the search term affected the quality of retrieved websites. We selected ‘epidural analgesia risk’, because during our first session this term was extensively tested and showed marked variations in medical content quality.
When scoring medical content accuracy, we compared the material in the sites with material in textbooks and recommendations issued by learned societies.18–21 This method allowed us to evaluate the ‘accuracy’ item, which is among the key items in the Netscoring® system. Among earlier studies of medical content on the Internet, all but six failed to indicate the source of reference data. However, the reference sources obviously influence the results of an analysis.5 8 22–24
We asked two physicians to evaluate the sites independently and to develop a consensus about sites for which their evaluations diverged. Fremont and colleagues25 found a high level of intraobserver and interobserver agreement regarding scoring of medical website quality, and suggested that the best procedure may be assessment by two independent observers followed by resolution of differences.
An encouraging finding from our study is that we consistently retrieved at least one website having a medical content score >95% within the first 15 websites on the retrieval list, during two sessions done 11 months apart. Moreover, medical content quality correlated negatively with rank in the retrieval list. The presence of at least one high-quality website within the first 15 limits the impact of the variations in mean quality scores found across the four main search terms. Similarly, high-quality sites were retrieved with all 14 terms for ‘epidural analgesia risk’. We retrieved higher-quality websites than did many earlier studies. In a systematic review by Eysenbach and colleagues,26 70% of studies of health-related websites concluded that quality was inadequate. However, most of these studies were conducted more than 4 yr ago, and improvements have probably occurred in the interval. In a study of sites on fever management in children,6 a slight improvement was noted between 1997 and 2001,5 6 although 46% of the pages retrieved in 1997 were retrieved unchanged in 2001.
Medical content accuracy may vary across topics. Topics for which the patient is also a consumer may generate a larger number of sites created by commercial entities, which may provide information of poorer quality than sites for which no conflict of interest occurs. For instance, among the sites on nutrition, 89% provided inaccurate information compared with fewer than 10% of sites on cancer.27 Professional organizations may be another valuable source of accurate information. In a letter, Theodosiou and Theodosiou11 reported that the best content retrieved by a Google® search using the term ‘having an anaesthetic’ came from the Royal College of Anaesthetists, which was the 10th retrieved site. Institutions and organizations may be more likely to follow guidelines for building high-quality health-related websites.1 28 29 General purpose labels such as Health on the Net (HON) may promote quality, although they are awarded on the basis of formal aspects rather than accuracy of content.11 The source of the information on the site should be clearly specified in order to help readers assess credibility. This criterion correlated with quality of medical content in our study.
Stability of quality over time is a major concern. The two searches done 11 months apart showed a good stability with two engines (Yahoo® and AltaVista®). With Google®, however, a decline in quality, as assessed using our criteria, was noted, as a result of a sharp increase in the number of abstracts of articles published in medical journals, which focused on highly specific points rather than providing comprehensive information on the risk of epidural analgesia.
We assessed a non-validated simple scale for rapid assessment of health-related websites. Our goal was to obtain a score reflecting intuitive criteria that non-health professionals may use to select sites. The scores on each of the three items (medical content, design, and the number of references supplied in the sites) correlated with the quality assessed by Netscoring®, suggesting a good performance of the scale. Simple scales that are fast to use and relevant to criteria used by the general public would be extremely useful, especially to help non-healthcare professionals to select websites. However, in this study assessment of website quality and the preliminary validation of simple scale parameters were performed by anaesthesiologists. Further studies are needed to investigate how well non-professionals identify high-quality websites.
In conclusion, the Internet contains good quality information on several anaesthesia-related topics. We found that the first 15 sites always contained at least one high-quality site. Quality was stable across synonyms, and also over time with two of three search engines. Future studies should monitor information quality over time, compare search engines, and validate simple scores that could be used by non-professionals to select reliable sites.
|
Appendix |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
|
|
|
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendixReferences |
|---|
1 Risk A, Dzenowagis J. Review of Internet health information quality initiatives. In: J Med Internet Res (2001) E28. Available from http//www.jmir.org/2001/4/c28/3.
2 Delamothe T. Quality of websites: kitemarking the west wind. Br Med J (2000) 321:843–4.
3 Griffiths KM, Christensen H. Quality of web-based information on treatment of depression: cross sectional survey. Br Med J (2000) 321:1511–5.
4 Allen JW, Finch RJ, Coleman MG, Nathanson LK, O'Rourke NA, Fielding GA. The poor quality of information about laparoscopy on the world wide web as indexed by popular search engines. Surg Endosc (2002) 16:170–2.[CrossRef][Web of Science][Medline]
5 Pandolfini C, Bonati M. Follow up of quality of public oriented health information on the world wide web: systematic re-evaluation. Br Med J (2002) 324:582–3.
6 Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the world wide web: systematic survey of advice on managing fever in children at home. Br Med J (1997) 314:1875–9.
7 McClung HJ, Murray RD, Heitlinger LA. The Internet as a source for current patient information. Pediatrics (1998) 101:pE2.
8 Latthe PM, Latthe M, Khan KS. Quality of medical information about menorrhagia on the worldwide web. BJOG (2000) 107:39–43.[CrossRef][Medline]
9 Croft DR, Peterson MW. An evaluation of the quality and contents of asthma education on the world wide web. Chest (2002) 121:1301–7.[CrossRef][Web of Science][Medline]
10 Ellamushi H, Narenthiran G, Kitchen ND. Is current information available useful for patients and their families? Ann R Coll Surg Engl (2001) 83:292–4.[Web of Science][Medline]
11 Theodosiou CA, Theodosiou LJ. Does the Internet provide safe information for pre-anaesthetic patients? Anaesthesia (2003) 58:805–6.[CrossRef][Web of Science][Medline]
12 Lim JM, Ho KM. A comparison of the Internet and the standard textbook in preparing for the professional anaesthetic examination. J Clin Monit Comput (1999) 15:449–53.[CrossRef][Web of Science][Medline]
13 Harmon D, Duggan M, Flynn N. Anaesthesia on the world wide web: is reliable patient information available on the Internet? Anaesthesia (2000) 55:728–9.[Web of Science][Medline]
14 Eysenbach G, Kohler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. Br Med J (2002) 324:573–7.
15 Chu-rouen.fr [homepage on the Internet]. Net Scoring ©: criteria to assess the quality of Health Internet information. (2000) Available from http://www.chu-rouen.fr/netscoring/netscoringeng.html.
16 Poremsky D. Google and Other Search Engines (2004) Berkeley, CA: Peachpit Press.
17 Darmoni SJ, Leroux V, Daigne M, Thirion B, Santamaria P, Duvaux C. Critères de qualité, de l'information de santé, sur l'Internet. In: Santé et Réseaux Informatiques, Informatique et Santé (1998) France: Springer-Verlag. 162–74.
18 SFAR (Société française d'Anesthésie Réanimation). (Homepage on the Internet). Information médicale sur l'analgésie péridurale en obstétrique. Ann Fr Anesth Réanim (1998) 17:fi170–1. Available from http://www.sfar.org/infoapdobst.html.[Web of Science]
19 Thirion B, Diedorf S. Porhyrias. In: Anesthesia and Co-existing Disease—Stoelting RK, Dierdorf SF, eds. (2002) 4th. Philadelphia, PA: Churchill Livingstone. 455–60.
20 Thirion B, Diedorf S. Sclerorderma. In: Anesthesia and Co-existing Disease—Stoelting RK, Dierdorf SF, eds. (2002) 4th edn. Philadelphia, PA: Churchill Livingstone. 511–2.
21 Sfar.org (SFAR: Société française d'Anesthésie Réanimation). (Homepage on the Internet). Feuillet d'information sur la transfusion sanguine. Available from http://www.sfar.org/s/IMG/pdf/infosgSFAR06.pdf.
22 Roupi P, van de Lei J. Rx medication information for the public and the WWW: quality issues. Med Inform Internet Med (1999) 24:171–9.[CrossRef][Web of Science][Medline]
23 Tamm EP, Raval BK, Huynh PT. Evaluation of the quality of self-education mammography material available for patients on the Internet. Acad Radiol (2000) 7:137–41.[CrossRef][Web of Science][Medline]
24 Latthe M, Latthe PM, Charlton R. Quality information on emergency contraception on the internet. Br J Fam Plann (2000) 26:39–43.[Web of Science][Medline]
25 Fremont P, Labrecque M, Legare F, Baillargeon L, Misson L. Evaluation of medical web sites. Interobserver and intraobserver reliability of an evaluation tool. Can Fam Physician (2001) 47:2270–8.[Web of Science][Medline]
26 Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA (2002) 287:2691–700.
27 Miles J, Petrie C, Steel M. Slimming on the Internet. J R Soc Med (2000) 93:254–7.
28 Winker MA, Flanagin A, Chi-Lum B, et al. Guidelines for medical and health information sites on the internet: principles governing AMA web sites. American Medical Association. JAMA (2000) 283:1600–6.
29 Wilson P. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the Internet. Br Med J (2002) 324:598–602.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  G. B. Drummond and M. F. Dozier Just give me the facts Br. J. Anaesth., November 1, 2007; 99(5): 608 - 610. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||