Brittain H. Tulbert, MD; Clint W. Snyder, PhD; Robert T. Brodell, MD
Dr. Tulbert is from Summa Health System, Akron City Hospital, Akron, Ohio. Drs. Snyder and Brodell are from bNortheastern Ohio Universities College of Medicine, Rootstown, Ohio. Dr. Brodell is also from Case Western Reserve University School of Medicine, Cleveland, Ohio, and University of Rochester School of Medicine, Rochester, New York.
Disclosure: Dr. Brodell is a Fellow of the American Academy of Dermatology and has served on numerous committees of the AAD, which produces some of the educational materials that were studied. Drs. Tulbert and Snyder report no relevant conflicts of interest.
Background: Supplemental educational reading material is of no value to patients if it is not read and comprehended. Objective: Using standardized research tools, online patient education materials were comparatively assessed for readability and length in words to identify the strengths and weaknesses of widely utilized sources. Methods: Three sources of patient-education material on the internet (WebMD.com, Wikipedia.org, and MedicineOnline.com) were compared with materials produced by the American Academy of Dermatology for readability utilizing Flesch-Kincaid Grade Level and Flesch Reading Ease Scale. Automated word counts were used to determine the length of each educational piece. Results: The information presented in American Academy of Dermatology electronic pamphlets on the internet is significantly harder to comprehend than MedicineOnline.com, but easier than Wikipedia.org. The latter site proved significantly harder to comprehend than all other sources. The American Academy of Dermatology electronic pamphlets and MedicineOnline.com materials were the most concise, averaging 1,200 words or less, although this was not a statistically significant difference in length compared to other online patient-education resources. No single source of online patient-education material demonstrates optimal features with regard to each of these parameters. Limitations: Only 15 topic areas in the four most commonly accessed sources of patient information were analyzed in this study. Conclusion: No single source of commonly used internet patient-education material demonstrates optimal features with regard to readability, length, and presence of photographic illustrations. These educational materials should target a length of 1,200 words, be illustrated with clinical images, and readability should correspond with the national average reading level. (J Clin Aesthet Dermatol. 2011;4(3):27–33.)
Health literacy has been defined as, “The degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.” The educational component of the dermatological encounter is of critical importance in fulfilling this capacity. Unfortunately, time restraints, patient anxiety, and disparities in educational level of the patient and physician impact patient education in a negative manner. Educational materials are designed to reinforce and supplement the information provided during the patient visit and may serve to increase patient adherence. Utilizing these tools is associated with improved patient adherence and completion of colorectal screening tests and improved asthma management. In order to improve health literacy, it is of great importance to provide the patient with readable and comprehensible information. In addition, physician recommendation of a reliable website reduces the likelihood that patients will be influenced by inaccurate information.
Readability, the ease with which written materials are read, is an essential factor in assessing the utility of a patient-oriented information resource. Increasing the readability of dermatological education materials, judged by scientific indices, has been associated with improved overall patient comprehension.[7,8] The Flesch-Kincaid Grade Level (FKGL) approximates the readability of a passage of text to a United States grade level. The FKGL, however, accurately measures reading materials only in the 3rd to 12th grade range. The Flesch Reading Ease Scale (FRES) grades readability on a 0-to-100 scale rather than on an educational grade scale. Documents with higher numbers are more easily read. Scores ranging from 70 to 100 are considered “fairly easy” to “very easy” to read, respectively. Scores from 60 to 70 are considered to be of normal or standard difficulty (Table 1).
The average reading level in the United States is between the eighth and ninth grade level,[10,13] and 36 percent of all Americans fall into the basic or below basic health literacy levels. A 2003 study by the National Center for Educational Statistics showed that patients with basic and below basic levels of health literacy were more likely to gather information from books, pamphlets, newspapers, and magazines rather than the internet/web sources. This demographic, however, may rapidly change in the future. The percentage of homes with Internet connections in the United States has increased to 62 percent, up from 18 percent in 1997.
This study is designed to provide the evidence needed to make recommendations regarding online resources. The readability and length of the American Academy of Dermatology (AAD) online pamphlets, WebMD.com (WebMD), Wikipedia.org (Wikipedia), and Medicine Online.com (MedicineOnline) were assessed. The length of the article was included as an important characteristic because length impacts the readability of educational materials. Even motivated patients have a finite amount of time to read information and a limited attention span.
Materials and Methods
The AAD educational pamphlets cover 62 topics and have been available free of charge online for the past 10 years. Eleven of the 15 topics included in the study were chosen from the list of 20 “top-selling” AAD pamphlets (AAD Internal Report, Pamphlet Editorial Board, March 2009). These pamphlets include Seborrheic Keratoses, Actinic Keratoses, Seborrheic Dermatitis, Eczema, Basal Cell Carcinoma, Acne, Psoriasis, Squamous Cell Carcinoma, Molluscum Contagiosum, Malignant Melanoma, and Tinea Versicolor. Four topics (Herpes Simplex, Herpes Zoster, Genital Warts, and Alopecia Areata) were chosen because they represent a spectrum of common and treatable dermatological disorders that may be particularly distressing to the patient due to their social ramifications.
The internet sources for this study were chosen using a Google search (“Online Medical Information”) in April 2009. Approximately 107 million pages were identified. The top five sources of information (WebMD.com, Medicinenet.com, The Online Merck Manual Library, Emedicine.medscape.com, and MedicineOnline.com) were considered for the study. Medicinenet was eliminated because it is a part of the WebMD network and contains similar information, although it is accessed less frequently on the web.[17,18] The Online Merck Manual Library and Emedicine.com were eliminated because they are references for medical professionals rather than patients.
Wikipedia.org is an online encyclopedia that may be edited by anyone, regardless of his or her qualifications. It was chosen for inclusion in the study because a Google search of the 15 chosen topics returned Wikipedia articles within the top five results for every topic, and as the top article (the “I’m Feeling Lucky” article) for 5 of the 15 topics. In addition, a study in the American Journal of Informatics concluded that Wikipedia is a highly utilized source of health information on the internet.
The entire body of text from each of the 15 topics was entered into Microsoft Word to evaluate the FKGL, the FRE score, and word count. The FKGL equation uses the number of words in a sentence and the quantity of syllables per word to estimate the American grade level education required for comprehension of a passage of written text. Using this formula, lower calculated results correlate with higher readability. Flesch-Kincaid Grade Level = 0.39 (words/sentence) + 11.8 (syllables/word) – 15.59. The Flesch Reading Ease formula similarly uses the number of syllables per 100-word block and the number of words per sentence to calculate the readability of a block of text. Unique to this formula, higher calculated results imply more readable text.[11,12] Flesch Reading Ease = 206.835 – 0.846 (word length) – 1.015 (sentence length). All related text for each source was utilized without references, unrelated information, self-assessment tools, or links to third-party sites.
The presence or absence of clinical photographs was recorded. There was no attempt made to judge the accuracy of material presented in these websites that would have required a committee of experts to grade the content of each educational piece. No human subjects were used in this research and, therefore, the Helsinki Protocol does not apply.
The three variables (FRE, FKGL, and word count) were analyzed according to source using SPSS version 17.0.1. The sources were then compared using a One-way Analysis of Variance (ANOVA) test with a Post-Hoc Scheffe test.
Prior to analyzing the data, the authors established “ideal” criteria for the readability of the resources. As the national readability has been stated to be between the 8th to 9th grade level,[10,13] the authors set the level of acceptable readability to be less than or equal to a FKGL of 8.9 and greater than or equal to 60.0 for the FRE.
The FKGL, FRE, and word count for each topic are summarized in Table 2a and Table 2b. Average and median statistics for each website are summarized in Table 3. All tested materials fell between 6.90 and 12.00 on the Flesch-Kincaid Grade Level (Figure 1). Considering all 58 individual educational materials, 23 had a FKGL below the established 8.9 criteria of FKGL. These included 3 of 23 (13%) of AAD Internet pamphlets, 8 of 23 (35%) found on WebMD, and 12 of 23 (52%) on MedicineOnline. None of the Wikipedia articles was below the ninth grade level, and 11 of 15 Wikipedia articles exceeded 12.0 on the FKGL.
When examining the entire body of educational materials, only 8 of 58 (14%) of the articles reached a FRE of 60.0. Of these, two were AAD pamphlets, two were from WebMD, and four were from MedicineOnline. No articles from Wikipedia exceeded a FRE of 60.0 (Figure 2).
One-way ANOVA for the four sources across the FRE and FKGL measures showed that there was at least one significant difference for all compared measures among the sources (p<0.01) (Table 3). The Scheffe test, which allowed a further delineation of these significant differences, produced the most striking finding. In all three measures, the AAD pamphlets, the WebMD re-sources, and the information on MedicineOnline were not significantly different from one another, with just one exception: Medicine Online materials demonstrated sig-nificantly better readability scores (p=0.02) regarding the FKGL (8.1) than the AAD pamphlets (9.6) and WebMD (9.3). MedicineOnline’s FRE score, although better (58.2), proved to be not statistically more readable than the AAD pamphlets (51.6) and WebMD (52.9).
Wikipedia had the worst readability with respect to both FKGL (11.8) and FRE (35.5). All of these values indicate significantly worse readability than the AAD Pamphlets, WebMD, and Medicine Online (p<0.01). The median results for each of the information sources had a similar profile (Table 2a and Table 2b). Interestingly, the AAD pamphlets had neither the best nor the worst readability according to any included parameter.
WebMD and Wikipedia had the longest individual articles. The average length of a WebMD article was 3,737 words, approximately three times longer than the average AAD pamphlet (1,208 words). Wikipedia entries averaged 2,748 words, approximately two times longer than the AAD articles. MedicineOnline had the shortest entries with an average length of only 1,047 words (Figure 3). While One-way ANOVA testing yielded a statistically significant difference between the groups (p=0.023), this difference was primarily due to the highly variable lengths of the individual articles. The Scheffe test demonstrated that the differences between the individual groups did not reach statistical significance. The online AAD pamphlets are liberally illustrated with clinical photographs. Wikipedia uses clinical images sparingly, and MedicineOnline and WebMD have no clinical photographs.
The internet has revolutionized patient education. The availability of millions of pages of material is, however, both a benefit and a detriment. Medical misinformation is disappointingly common on the internet. Many postings represent opinion with no peer review. On the other hand, Wikipedia is usually not considered an academic resource, yet a 2005 study in Nature stated that the accuracy of the articles in Wikipedia rivals that of Encyclopedia Britannica. Accepting the fact that educated patients can best manage their own care, it is the physician’s duty to guide their patients to appropriate resources for further reading at home. In addition, guiding patients to proper information sources can potentially decrease medical liability.
AAD pamphlets are written by dermatologists and edited by a professional staff. They make liberal use of excellent quality, topic-specific photographs. Photographs may reinforce the patient’s awareness, understanding, and recall of information presented in the pamphlets.
Unfortunately, AAD pamphlets did not score as highly on readability indices as would be ideal. The average Flesch-Kincaid Grade Level for the pamphlets was 9.6, indicating a difficulty that exceeds the national reading average. They are also significantly more difficult to read than the MedicineOnline sources. Only three of the AAD pamphlets were assessed at or below the ideal 8.9 FKGL. On the other hand, they were concisely written (average 1,208 words; median 934 words).
MedicineOnline had the best overall readability indices. Twelve of the 13 available articles had readability of less than 8.9 on the FKGL, a significant advantage over AAD materials. Writing is concise, averaging near 1,000 words. MedicineOnline, however, had some significant shortcomings. The website contains complete information for only 12 of the common dermatological disease topics studied and no clinical images are included. No information was found on two of the topics (squamous cell carcinoma, alopecia areata) and information specific only to women was available for the genital warts topic. This website also contains multiple dead links.
WebMD articles tended to have better readability indices than the AAD pamphlets; however, these findings were not statistically significant. Of the 15 articles on WebMD, eight were assessed at or below the 8.9 Flesch-Kincaid Grade Level. These articles were significantly easier to read than Wikipedia resources with respect to all readability parameters. Information was available pertaining to each of the 15 AAD topics. WebMD has a panel of physician editors who routinely review the material posted on the website, but it is not clear if dermatologists review all dermatology-related material. WebMD articles also had the highest word count (average 3,737 words; median 2,069 words), requiring 2 to 3 times as long to read this information when compared to the AAD materials. Differences in the length of these articles, however, did not reach statistical significance. No clinical images are included.
Wikipedia is a well-known source of information on virtually any topic. Unfortunately, it had the worst readability indices of all of the studied resources, decreasing the utility of this resource for less-educated patients. The FKGL of the Wikipedia text (mean 11.8; median 12.0) suggests high school completion would be required to adequately comprehend this material. In addition, many of the FKGL for Wikipedia exceeded the maximum value of the index, 12.0. The collaborative authorship of Wikipedia articles by the general public without professional editing provides an explanation for the lack of attention to targeting the eighth grade reading level that is most appropriate. Clinical images are used inconsistently.
Considering the expansion of internet-based patient-education materials, it is critical that these sites be scrutinized with scientific rigor. The authors’ overall assessment of these educational materials is presented in Table 3. There is room for improvement for all of the resources included in this study. Each site should consider rewriting materials aiming for FKGL less than or equal to 8.9 (average national reading level) and FRE scores between 60 and 70. This goal is attainable for the AAD, as MedicineOnline and WebMD have demonstrated the ability to draft these resources in more readable formats. This goal will be more difficult to attain for Wikipedia unless professional editors are introduced. Wikipedia, in particular, needs to clearly define portals with more readable text and shorter articles specifically for patient education. These articles should be separated from professionally developed educational materials in order to gain the authors’ recommendation for patient education. In addition, many physicians will never be comfortable with sources that do not perform a professional, independent, peer-review process of all web material.
As always, patients should be advised to “ask their doctor” to clarify any information they read, whether it be from a pamphlet, on the internet, or via any other form of communication.
1. Healthy People 2010. 2nd ed. Washington (DC): Department of Health and Human Services (US); 2000.
2. Winnick S, Lucas DO, Hartman AL, Toll D. How do you improve compliance? Pediatrics. 2005;115:718–724.
3. Stokamer CL, Tenner CT, Chaudhuri J, et al. Randomized controlled trial of the impact of intensive patient education on compliance with fecal occult blood testing. J Gen Intern Med. 2005;20:278–282.
4. Bhogal SK, Zemek RL, Ducharme F. Written action plans for asthma in children. Cochrane Database of Systematic Reviews. 2006; Issue 3; Art. No.: CD005306.DOI10.1002/ 14651858.CD005306.pub2.
5. Zirwas MJ, Holder JL. Patient education strategies in dermatology—Part 2: Methods. J Clin Aesthetic Dermatol. 2009;2(12):28–34.
6. Jackson RH, Davis TC, Bairnsfather LE, et al. Patient reading ability: an overlooked problem in health care. South Med J. 1991:84:1172–1176.
7. Baker GC, Newton DE, Bergstresser PR. Increased readability improves the comprehension of written information for patients with skin disease. J Am Acad Dermatol. 1988;19(6):1135–1141.
8. Feldman SR, Vanarthos J, Fleischer AB. The readability of patient education materials designed for patients with psoriasis. J Am Acad Dermatol. 1994;30:284–286.
9. Kincaid JP, Fishburne RP Jr., Rogers RL, Chissom BS. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy-enlisted personnel. Research Branch Report 8-75. Millington, TN: Naval Technical Training, U.S. Naval Air Station; Memphis, TN; 1975.
10. D’Allessandro DM, Kingsley P, Johnson-West J. The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med. 2001;155: 807–812.
11. Flesch R. A new readability yardstick. J Appl Psychol. 1948;32:221–233.
12. Flesch R. How to Test Readability. New York: Harper and Brothers; 1951
13. Doak CC, Doak LG, Root JH. Teaching Patients With Low Literacy Skills. 2nd ed. Philadelphia, Pa: JB Lippincott; 1996.
14. Kutner M, Greenberg E, Jin Y, Paulsen C. The Health Literacy of America’s Adults: Results From the 2003 National Assessment of Adult Literacy (NCES2006–483). U.S. Department of Education. Washington, DC: National Center for Education Statistics; 2006.
15. U.S. Census Bureau. Internet use triples in decade, Census Bureau reports [internet]. U.S. Census Bureau News. Washington D.C. U.S. Department of Commerce; 2009 June. http://www.census.gov/Pressrelease/www/releases/archives/communications_industries/013849.
16. Baker GC. Writing easily read patient education handouts, a computerized approach. Seminars in Dermatology. 1991;10(2):102–106.
17. Alexa Web Information Service. WebMD.com – Traffic Details from Alexa [Internet]. San Francisco, CA: Alexa Internet; c1996–2009 [Modified 2009 Oct 7; Cited 2009 Oct 7]. http://www.alexa.com/siteinfo/webmd.com.
18. Alexa Web Information Service. Medicinenet.com – Traffic Details from Alexa [Internet]. San Francisco, CA: Alexa Internet; c1996–2009 [Modified 2009 Oct 7; Cited 2009 Oct 7]. Available from: http://www.alexa.com/siteinfo/medicinenet.com.
19. The Wikimedia Foundation. About Wikipedia.org [Internet]. San Francisco, CA: The Wikimedia Foundation; c2003–2009 [modified 2009 Aug 8, cited 2009 Aug 8]. http://en.wikipedia.org/ wiki/Wikipedia:About.
20. Laurent MR, Vickers TJ. Seeking health information online: does Wikipedia matter? J Am Med Inform Assoc. 2009; 16(4):471–479. Epub 2009 Apr 23.
21. Giles J. Internet encyclopedias go head to head. Nature. 2005;438(7070):900–901.
22. Katz LG. The use of printed instruction sheets to enhance patient compliance. Seminars in Dermatology. 1991;10(2): 91–95.
23. Newell S. Writing effective pamphlets: a basic guide, prepared for Hunter Centre for Health Advancement. Newcastle, NSW; 1996. http://epubs.scu.edu.au/educ pubs/143.
24. WebMD LLC. About WebMD [Internet]. Atlanta, GA: WebMD LLC; c2005–2009. [Cited 2009 Jun 6]. http:// www.webmd.com/about-webmd-policies/default.htm?ss=ftr.