J Clin Aesthet Dermatol. 2024;17(11):14.
Analysis of ChatGPT’s Responses to Patient Questions on Hidradenitis Suppurativa
Dear Editor:
Since ChatGPT’s release in November 2022, the large language chatbot has been used to diagnose medical conditions, write research papers, and assess clinical skills.1 Hidradenitis suppurativa (HS) is a medical condition characterized by painful skin abscesses and scarring of the skin.2 Because of the associated pain, sensitive anatomic locations, drainage, odor, and scarring, HS has a powerful psychosocial impact and decreases quality of life for patients.2
With the rise of artificial intelligence, the goal of this study was to analyze ChatGPT’s responses to patient questions on HS, particularly those dealing with psychosocial impact. Readability was assessed by number of sentences, words per sentence, characters per sentence, percentage of difficult words, Flesch Reading Ease (FRE), and the Flesch-Kincaid Grade Level (FKGL). The FRE score ranges from 0 (unreadable) to 100 (extremely easy to read).3 The FKGL provides a score corresponding to a United States education grade level, a lower score means a text is easier to read.3 For example, a text with a FKGL of 8.5 means that individuals with an education level of 8th grade or higher should be able to comprehend the text fully.3 This is the level that is suggested for health-related information to be considered accessible to all levels of literacy.3
ChatGPT’s responses were compared to responses to these questions available from the Hidradenitis Suppurativa Foundation website.4 Statistical significance was determined by a p-value <0.05 and 95% confidence intervals.
In January 2023, various questions were used to prompt ChatGPT, such as:
- “Will people think I am gross if I have hidradenitis suppurativa?”
- “In what ways can hidradenitis suppurativa impact my personal or work life?”
- “What can I do at home to help treat or prevent hidradenitis suppurativa flares?”
- “What should I expect over time if I have hidradenitis suppurativa?”
- “How do I explain hidradenitis suppurativa to my friends and family?”
- “Will hidradenitis suppurativa affect my romantic life?”
There was a statistically significant difference in the responses provided by ChatGPT and the HS Foundation in the words per sentence (p-value=0.0153, 95 CI: 1.58–10.05), percentage of difficult words (p-value=0.0002, 95% CI: 9.15–17.22), FRE (p-value=0.0002, 95% CI: -51.71– -27.27), and the FKGL (p-value=0.0004, 95% CI: 4.55–9.34).
The responses provided by the HS foundation to popular patient questions on HS are statistically significantly more accessible and easier to read than the responses provided by ChatGPT as determined by the percentage of difficult words, the Flesch Reading Ease, and The Flesch-Kincaid Grade Level. ChatGPT may still nevertheless be a source for patient information, however the responses provided by the HS foundation are more accessible and may reach a larger patient population. Limitations of this study include restrictions of the FRE and FKGL, these scoring methods do not take consider document factors such as, layout, pictures, chart, color, font, spacing, personal factors including motivation, information needs, or anxiety level, and style of writing such as cultural sensitivity.3 Other limitations include a relatively small size of questions asked and differing AI responses to the same prompt.
With regard,
Kripa Ahuja, MS, and Peter Lio, MD
Keywords. Chatbots, artificial intelligence, hidradenitis suppurativa, patient education, emotional support
Affiliations. Ms. Ahuja is with the Eastern Virginia Medical School in Norfolk, Virginia. Dr. Lio is with the Department of Dermatology at Northwestern University in Chicago, Illionois.
Funding. No funding was provided for this article.
Disclosures. Some of the data from this manuscript was presented in a poster presentation at the January 2024 North Carolina Dermatology Association Annual Meeting in Greensboro, North Carolina. Dr. Lio reports research grants/funding from AbbVie, AOBiome; is on the speaker’s bureau for AbbVie, Arcutis, Eli Lilly, Galderma, Hyphens Pharma, Incyte, La Roche-Posay/L’Oreal, MyOR Diagnostics, ParentMD, Pfizer, Pierre-Fabre Dermatologie, Regeneron/Sanofi Genzyme, Verrica; reports consulting/advisory boards for Alphyn, AbbVie, Almirall, Amyris, Arcutis, ASLAN, Boston Skin Science, Bristol-Myers Squibb, Burt’s Bees, Castle Biosciences, Codex Labs, Concerto Biosci, Dermavant, Eli Lilly, Galderma, Janssen, Johnson & Johnson, Kimberly-Clark, LEO Pharma, Lipidor, L’Oreal, Merck, Micreos, MyOR Diagnostics, Regeneron/Sanofi Genzyme, Skinfix, Theraplex, UCB, Unilever, Verrica Yobee Care; stock options with Codex, Concerto Biosciences and Yobee Care. In addition, Dr. Lio has a patent pending for a Theraplex product with royalties paid and is a Board member and Scientific Advisory Committee Member of the National Eczema Association. Ms. Ahuja has no conflicts of interest or relationships to disclose.
References
- Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595.
- Schneider-Burrus S, Tsaousi A, Barbus S, et al. Features associated with quality of life impairment in hidradenitis suppurativa patients. Front Med. 2021;8:676241.
- Jindal P, MacDermid J. Assessing reading levels of health information: uses and limitations of flesch formula. Educ Health. 2017;30(1):84.
- HS Foundation. https://www.hs-foundation.org/