Availability and readability of online patient information on clubfoot: assessment of paediatric hospital clubfoot web pages

Abstract

Purpose

To determine the availability and readability of online patient information (OPI) provided by paediatric hospitals in the United States using clubfoot as a model condition

Methods

The websites of the top 95 paediatric hospitals identified using US News & World Report were included. The names of paediatric hospitals and the terms “clubfoot”, “clubfeet” and “talipes equinovarus” were entered into the Google search engine. Readability was assessed using five validated metrics and the composite grade level (CGL). The number of unpaid monthly visits was calculated with the Ahrefs Organic Traffic Score (OTS) tool. Data for paediatric hospitals were compared with the same metrics for the top ten Google search results.

Results

Of 95 paediatric hospitals, 29 (30.5%) did not have at least one web page dedicated to clubfoot. The 128 web pages representing 66 paediatric hospitals had an average CGL of 9.4, representing a readability level requiring some high school education. The mean OTS for all paediatric hospitals was 116 estimated visits per month, which was significantly less than that for the top ten Google clubfoot search results (3035.1; p < 0.0001).

Conclusion

Paediatric hospital web pages on clubfoot were visited much less frequently than those from the top ten Google search results. Only two web pages (1.6%) from paediatric hospitals offered OPI on clubfoot that met the American Medical Association recommended reading level (sixth-grade level). Paediatric hospitals should create OPI on clubfoot with appropriate readability and accessibility for patient families.

Level of Evidence

N/A

Introduction

Online patient information (OPI) is becoming increasingly important in the field of medicine. Recent surveys show that over 70% of Americans use the internet to search for information about their health, and that a majority state that online information influences their decisions about healthcare treatment.^1,2 OPI can help increase knowledge of health conditions in some cases, but information on the internet can also be inaccurate, esoteric or sensational, leading to disinformation or distrust in the physician-patient relationship.³ More specifically within paediatrics, mothers are very likely to use OPI in the first few years after their child’s birth, especially to seek more details on the diagnosis of paediatric conditions.⁴

Quality of OPI is generally lacking, creating a digital information ecosystem that could potentially misguide patients.⁵ In one meta-review of orthopaedic OPI, only around 3% to 15% of websites were rated as high-quality.⁶ The availability of OPI from reputable healthcare institutions, such as paediatric hospitals, is not well-known, but prior work has indicated that OPI coming from academic centres and hospitals tend to be among the highest quality available to patients, as compared with commercial websites. However, prior work has suggested that OPI provided by academic centres represents a small minority of OPI accessed online.⁷ Furthermore, the vast majority of OPI, including that for orthopaedics, has generally been shown to be above both the American Medical Association (AMA)-recommended sixth grade reading level and the average US reading level (eighth grade), making access to comprehensible, accurate information challenging for patients.^8-10

Clubfoot is one of the most common congenital deformities of the skeletal system and one that is associated with tremendous anxiety for patient families. Prenatal diagnosis of congenital deformities has been shown to increase acute parental psychological distress after birth.¹¹ Without any social support, young mothers of children with clubfoot are more likely to report stress and depressive symptoms. Availability of accurate medical information has been shown to provide support for families dealing with clubfoot.¹² As there remain gaps in our knowledge regarding the aetiology and long-term management of clubfoot, access to and availability of accurate information becomes all the more important for guiding families deciding on treatment options.¹³ Consequently, the goal of this study is to provide a much-needed examination of the quality, availability and readability of OPI for clubfoot in order to better inform clubfoot care and improve understanding of how patients use online health information.

Materials and Methods

Identification of institutions

All paediatric hospitals in this study were identified using the U.S. News & World Report rankings website,¹⁴ and all paediatric hospitals ranked for orthopaedics were included in our study.

Website search methodology

The names of the paediatric hospitals along with either the term “clubfoot” “clubfeet” or “talipes equinovarus” were entered into the Google search engine sometime between 03 September 2020 and 20 September 2020. A single author (MX) manually evaluated search results to verify relevance of web page content. Up to five web pages were included for each institution in our analysis. If a paediatric hospital did not have any web pages with at least one paragraph dedicated to clubfoot aimed at patient use, it was discarded from the rest of the analysis. Pages listing providers treating clubfoot, any patient testimonials or blog entries and advertisements were also omitted. To compare web pages from paediatric hospitals with the most popular results being accessed by patients, a Google search blinded to prior search history of “clubfoot” was undertaken on 26 September 2020; the first page (i.e. the top ten links that were not adverts) were used for analysis, as previous research has shown that patients rarely go beyond the first page of search results.¹⁵

Assessment of OPI availability

Web pages from the search process outlined above were assessed for availability of OPI on clubfoot as following: 1) nothing at all on clubfoot or congenital malformations of the lower extremity; 2) some information on congenital malformations of the lower extremity but no specific mention of clubfoot; 3) website has one or less paragraphs; 4) website has more than one paragraph but less than one full web page; and 5) have at least one standard web page.

Assessment of OPI readability

Readability of each web page with at least some information on clubfoot was assessed using the WebFX online tool (Harrisburg, PA, USA) using the text presented on each web page. This tool assesses presented text for readability using five validated metrics that take into consideration the number of total words, sentences and syllables to estimate the minimum grade level required to comprehend the text. The metrics analysed were the Flesch-Kincaid Grade Level, the Gunning Fog Score, the Simple Measure of Gobbledygook Index, the Coleman-Liau Index and the Automated Readability Index.^16,17 A higher score in each metric corresponds to a higher grade level and greater difficulty of comprehension. To account for the variability among readability metrics, the mean of all five metrics listed above was rounded to the nearest grade level to find the corresponding grade level (CGL).

Assessment of web traffic access

Website traffic was assessed using the website analytics tool from Ahrefs, a website analysis company (Singapore, Malaysia). Ahrefs represents website traffic using the Organic Traffic Score (OTS) metric, which estimates the number of unpaid visits per month to a particular uniform resource locator by calculating the traffic generated by the top 100 keywords associated with searches leading to that particular site over a 30-day average.

Statistical analysis

Comparative analyses were performed using two-sided Student’s t-tests (α = 0.05).

Results

Assessment of availability

Of the 95 paediatric hospitals, 29 (30.5%) had no information on clubfoot. For the other 66 (69.5%) that provided at least some information, there were a total of 128 web pages that mentioned clubfoot to varying degrees. Out of these 128 web pages, 123 (96.1%) were full web pages on clubfoot, one (0.8%) had more than one paragraph but less than one full web page, and three (2.3%) had at least one paragraph on clubfoot (Fig. 1).

Fig. 1. The availability of online patient information (OPI) for paediatric hospitals. ‘With clubfoot OPI’ = at least one page dedicated to clubfoot; ‘multiple paragraphs on clubfoot’ = at least one paragraph on clubfoot; ‘one (or fewer) paragraph on clubfoot’ = less than one paragraph for clubfoot; and ‘no information’ = no information on clubfoot provided.

Assessment of readability

A total of 128 web pages from 66 paediatric hospitals were assessed for readability. The mean corresponding grade level (CGL) for all web pages was 9.4 (range = 8), corresponding to some high school education. The lowest CGL was fifth grade, and the highest was seventeenth grade (i.e. requires a college degree). In all, 37% of websites from paediatric hospitals required a middle school education to comprehend (i.e. sixth to eighth grades), 45% required some high school education (i.e. nineth to eleventh grades), 11% required a high school degree (i.e. completing tweltfth grade) and 7% were at the level of some college or holding a college degree (i.e. greater than twelfth grade; Fig. 2). The average of all five metrics and the CGL for paediatric hospital web pages and Google top ten search results were higher than the sixth-grade level recommended by AMA (Fig. 3). In fact, 126 (98.4%) of the web pages had a CGL above 6 (sixth-grade level) and 60 (46.9%) above 9 (equivalent to high school education). A mean of around 12.0% of the words on each web page were considered complex (range = 19.4).

Fig. 2. Grade-level distributions of clubfoot-specific web pages for paediatric hospital and Google top ten search results.

Fig. 3. The readability of clubfoot online patient information found on paediatric hospital web pages and pages from Google top ten search results. Mean values are shown. The bolded horizontal line indicates the American Medical Association-recommended sixth grade reading level.

For the top ten search results on Google, five were from academic hospitals, three were private foundations, one was from a general knowledge website (Wikipedia), and one web page was from the American Academy of Orthopedic Surgeons (see Table 1) for all Google top ten results listed out in greater detail). There were no statistically significant differences in any of the readability metrics between the top ten search results from Google and the paediatric hospital web pages. The mean CGL for all Google top ten web pages was around 9.3 (range = 6), or corresponding to some high school education, just as for the paediatric hospital web pages. In all, 40% of the web pages from the Google search required a middle school education or lower to understand, while half (50%) required some high school and 10% required a high school degree; no sites required any college education. All but one website (90%) required an education above sixth grade, which is the AMA-recommended reading level. As with paediatric hospital web pages, all five of the readability metrics and the CGL were higher than the sixth-grade level recommended by AMA (Fig. 4). Within the ten websites found on Google, between those from paediatric hospitals and those not from paediatric hospitals, there were no statistically significant differences in any of the readability metrics.

Table 1 Out of the top 10 search results for clubfoot on Google, five are from academic institutions or hospitals. This table shows the type of web page and name of web page for all 10 results

Type of website	Number	Web page source
Academic institutions or hospitals	5	Mayo Clinic, Johns Hopkins School of Medicine, Cedars-Sinai Medical Center, Boston Children’s Hospital, Nemours Children’s Health System (KidsHealth)
Private organizations	3	Medical News Today, March of Dimes, WebMD
General information site	1	Wikipedia
Other	1	American Academy of Orthopedic Surgeons

Fig. 4. The readability of clubfoot online patient information found on paediatric hospital web pages and non-paediatric hospital pages, comparing only websites from the Google top ten search results. Mean values are shown. The bolded horizontal line indicates the American Medical Association-recommended sixth grade reading level.

Assessment of web traffic access

The mean web traffic score for all paediatric hospital web pages with information on clubfoot was around 116.2, which means that, on average, around 116 visits were made to each web page per month. There was a wide range of web traffic scores for paediatric hospital web pages, from 0 to 9000, with 79 web pages having a score of 0. In contrast, the mean web traffic score for the top ten search results on Google was much greater, at 3035.1 (range = 8686), which means that, on average, around 3035 visits were made to each web page from the first page of Google search results per month (Fig. 5). A two-sided t-test was performed, and the difference between the two averages were found to be statistically significant (p < 0.0001).

Fig. 5. Comparison of mean monthly traffic estimates for the top ten Google “clubfoot” search results and paediatric hospital web pages dedicated to clubfoot. The difference in the Organic Traffic Score was statistically significant (p < 0.0001).

For the top ten search results on Google, there were web pages from five paediatric hospitals: number 2, Mayo Clinic (web traffic score: 9000); number 5, Johns Hopkins (web traffic score: 2100); number 6, Neymours (web traffic score: 2300); number 8, Cedars-Sinai (web traffic score: 314); and number 10, Children’s National (web traffic score: 1100). There was no statistically significant difference between the paediatric hospitals and non-paediatric hospitals within the top ten search results on Google in terms of web traffic.

Discussion

OPI plays a crucial role in educating families of children with clubfoot when they make treatment decisions, but so far, very little is understood regarding the availability, readability or accessibility of clubfoot OPI, especially from high quality sources like hospitals and academic institutions. Our analysis suggests that clubfoot OPI provided by paediatric hospitals is not universally available, is written at a level too difficult to read and of the information that is available is not readily accessed by patient families.

Assessment of availability

Our results suggest that clubfoot OPI provided by paediatric hospitals is limited in availability. Of the paediatric hospitals included in this analysis, 29 of 95 (30.5%) do not provide any information on their website about clubfoot. Accurate medical information has been shown to support the psychological wellbeing of families dealing with clubfoot;¹² as such, the lack of readily available information provided by treating institutions represents a potential opportunity for improvement in providing patients with usable health information as a part of their care. This finding also aligns with prior research indicating that hospital websites are, as a whole, typically poor in accessibility, do not feature the information most desired by potential patients efficiently, and are not designed with patients and their families in mind.^18-22 Potential solutions have been proposed, such as having hospitals and medical institutions involve patient feedback during any website design process in order to provide higher-quality and more accessible OPI.^22,23 Since many Americans use the internet to learn more about their illnesses and possible procedures and medications, improved access to OPI from hospitals can be transformative, as patients tend to view hospitals with better and more comprehensive websites more favourably,²⁴ and those equipped with accurate OPI help families feel more comfortable with their provider, leading to increased confidence in their physician’s recommendations.²⁵

Assessment of readability

In our analysis, we found that web pages from paediatric hospitals are at a reading level around high school level (mean = 9.425, around nineth grade). Additionally, the vast majority of web pages from hospitals were written at a reading level far higher than AMA recommendations (125 web pages or 98.4%). These challenges may ultimately prove to be a barrier in the effective use of OPI.²⁵ Low health literacy, which can result from failing to understand OPI if the OPI is not ‘readable’ enough, has been linked to poorer health information comprehension and health outcomes.^26,27 Prior work has found that patients with greater health illiteracy may not completely comprehend and may misuse medications, skip appointments, among other negative effects, which leads to worse healthcare outcomes.²⁸ While low levels of health literacy have been linked to poorer comprehension of orthopaedic care, it remains unclear what the specific effect of low comprehensibility of OPI might be on patient outcomes in clubfoot care and paediatric orthopaedic care more generally.²⁹

Assessment of web traffic

OPI from academic institutions and paediatric hospitals were accessed at a very low rate, and at a much lower rate than the top ten search results from Google, with 84% (107) of all web pages from hospitals receiving fewer than ten visits per month. Given the variability of the types of sources in the top ten Google search results, the degree to which patients are receiving high-quality information is uncertain. The most commonly accessed online information (i.e. from Google) can be of uncertain quality and may not be providing patients with accurate information, since OPI from academic institutions has been previously shown to be of higher quality and reliability than those from private or media sources.³⁰ Furthermore, given the rapid monetization of internet search engines, online advertisements that are prominent on the first page of search results can easily be mistaken for valuable and accurate health information, which poses a grave danger to patients’ understanding of medical conditions and procedures.³¹ Potential solutions have previously been described, including developing a website quality assessment tool as a guide to help websites (from paediatric hospitals) rank higher in Google search engine result pages, since the average Internet user does not go past the first page of search results. Additionally, clinicians may use their time with patients to discuss what online sources are trustworthy and which ones they should avoid, and to encourage them to prioritize their hospital’s website for information.^15,32 (Before doing this, clinicians should ensure that their hospitals are providing accessible and readable information online.) Ultimately, additional work is needed to better assess effective solutions that connect patients and their families to OPI supplied by healthcare professionals or institutions.

Limitations

This study was not without its limitations. While our study was not limited to an analysis of clubfoot centres, our methodology allows us to evaluate OPI as provided by a wide range of institutions providing paediatric care in the United States. Future studies may clarify differences in care between dedicated centres and paediatric orthopaedic care at large. Our reported web traffic score from Ahrefs is an estimate based on Ahrefs’ proprietary web traffic algorithm and does not track actual count of page visits. However, the algorithm itself ensures good comparisons between web pages of the same niche, such as clubfoot information on hospital websites in this study.³³ Readability measures are also imperfect, but without other validated metrics, the readability indices used in this study represent the best available tool to assess reading difficulty.³⁴ We did not systematically assess the accuracy or reliability of information from paediatric hospitals included in our analysis, though prior work has indicated that OPI from hospitals tends to be among the highest quality available.³⁰ Top ten Google search results may vary from person to person based on previous search history, location, saved cookies and cache. For our analysis, Google Chrome’s incognito mode was used for all searches to minimize this source of variability. Finally, while clubfoot represents only a single condition within paediatric orthopaedics, we believe this topic to be both germane to orthopaedic surgeons and to highlight the problems facing OPI in the current digital health landscape.

Conclusion

Our study highlights the challenges faced by medical institutions and hospitals in presenting and providing readable OPI for patients and their families within the context of clubfoot. We have indicated that the information about clubfoot from paediatric hospitals is not visible or comprehensible enough for the general population to make the most use. Improving the availability, accessibility and readability of such information through transformative, patient-centred design at a hospital-level has the potential to improve patient comprehension about clubfoot and potentially patient outcomes in this field. Additional work is needed to evaluate the impact of such an intervention and to ensure hospital communication with patients is the most efficient and effective in improving the quality of medical care.

OA Licence Text

This article is distributed under the terms of the Creative Commons Attribution-Non Commercial 4.0 International (CC BY-NC 4.0) licence (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed.

Compliance with ethical standards

Funding statement

No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article.

Ethical statement

Ethical approval: There was no research involving humans or animals.

Informed consent: Not applicable, since this research did not involve human participants.

ICMJE Conflict of interest statement

The authors declare that they have no conflict of interest.

Author Contributions

MX: Study design, Performed measurements, Statistical analysis, Manuscript ­preparation.

JY: Study design, Manuscript preparation.

CM: Manuscript revision.

References

1. Fox S Duggan M. Health Online 2013. Pew Research Center; 2013. https://www.pewresearch.org/internet/2013/01/15/health-online-2013/ (date last accessed 10 October 2020 ).

2. Fox S Rainie L Horrigan J et al. The online health care revolution: how the web helps Americans take better care of themselves. https://www.pewinternet.org/wp-content/uploads/sites/9/media/Files/Reports/2000/PIP_Health_Report.pdf.pdflast accessed 19 April 2021 ).

3. Tonsaker T Bartlett G Trpkov C. Health information on the Internet: gold mine or minefield? Can Fam Physician 2014;60:407-408.

4. Bernhardt JM Felter EM. Online pediatric information seeking among mothers of young children: results from a qualitative study using focus groups. J Med Internet Res 2004;6:e7.

5. Young J Dee EC May C. Availability and readability of online patient information on osteosarcoma: assessment of paediatric hospital and National Cancer Institute-Designated Cancer Center (NCIDCC) osteosarcoma web pages. JBJS Open Access 2020;5:e20.00054-e20.00054.

6. Cassidy JT Baker JF. Orthopaedic patient information on the world wide web: an essential review. J Bone Joint Surg [Am] 2016;98:325-338.

7. LaValley SA Kiviniemi MT Gage-Bouchard EA. Where people look for online health information. Health Info Libr J 2017;34:146-155.

8. Eltorai AEM Sharma P Wang J Daniels AH. Most American Academy of Orthopaedic Surgeons’ online patient education material exceeds average patient reading level. Clin Orthop Relat Res 2015;473:1181-1186.

9. Feghhi DP Komlos D Agarwal N Sabharwal S. Quality of online pediatric orthopaedic education materials. J Bone Joint Surg [Am] 2014;96:e194.

10. Daraz L Morrow AS Ponce OJ et al. Readability of online health information: a meta-narrative systematic review. Am J Med Qual 2018;33:487-492.

11. Skari H Malt UF Bjornland K et al. Prenatal diagnosis of congenital malformations and parental psychological distress—a prospective longitudinal cohort study. Prenat Diagn 2006;26:1001-1009.

12. Coppola G Costantini A Tedone R et al. The impact of the baby’s congenital malformation on the mother’s psychological well-being: an empirical contribution on the clubfoot. J Pediatr Orthop 2012;32:521-526.

13. Dobbs MB Gurnett CA. Update on clubfoot: etiology and treatment. Clin Orthop Relat Res 2009;467:1146-1153.

14. No authors listed. Best Children’s Hospitals for Orthopedics. US News and World Report. https://health.usnews.com/best-hospitals/paediatric-rankings/orthopedics (Date last accessed 21 December 2020 ).

15. Morahan-Martin JM. How internet users find evaluate and use online health information: a cross-cultural review. Cyberpsychol Behav 2004;7:497-510.

16. Smith EA Kincaid JP. Derivation and Validation of the Automated Readability Index for Use with Technical Materials. Hum Factors. 1970;12(5):457-564.

17. Karmakar S Zhu Y. Visualizing multiple text readability indexes. In: 2010 International Conference on Education and Management Technology.IEEE ; 2010:133-137. doi:10.1109/ICEMT.2010.5657684

18. Llinás G Rodríguez-Iñesta D Mira JJ Lorenzo S Aibar C. A comparison of websites from Spanish American and British hospitals. Methods Inf Med 2008;47:124-130.

19. Kim Y-S Oh K-S. A study on current state of web content accessibility on general hospital websites in Korea. J Internet Computing and Services 2010;11:87-103.

20. Alhuwail D AlMeraj Z Boujarwah F. Evaluating hospital websites in Kuwait to improve consumer engagement and access to health information: a cross-sectional analytical study. BMC Med Inform Decis Mak 2018;18:82.

21. Acosta-Vargas P Acosta T Luján-Mora S. Framework for accessibility evaluation of hospital websites. In:2018 International Conference on E-Democracy and E-Government (ICEDEG). IEEE ; 2018:9-15.

22. Mukhopadhyay RS Waller A Franklin VL. What do UK children’s hospital websites offer patients and families? Arch Dis Child 2008;93(2):179-180.

23. Schenker Y London AJ. Risks of imbalanced information on US hospital websites. JAMA Intern Med 2015;175:441-443.

24. Gümüş R Sönmez Y. Quality of online communication tools at hospitals and their effects on health service consumers’ preferences. Int J Healthc Manag 2020;13:35-44.

25. Tan SS-L Goonawardene N. Internet health information seeking and the patient-physician relationship: a systematic review. J Med Internet Res 2017;19:e9.

26. Diviani N van den Putte B Giani S van Weert JC. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res 2015;17:e112.

27. Berkman ND Sheridan SL Donahue KE Halpern DJ Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011;155:97-107.

28. Battineni G Baldoni S Chintalapudi N et al. Factors affecting the quality and reliability of online health information. Digit Health 2020;6:1-11.

29. Graham S. Do patients understand? permj 2008;12(3): 67-69.

30. Kadakia RJ Tsahakis JM Issar NM et al. Health literacy in an orthopedic trauma patient population: a cross-sectional survey of patient comprehension. J Orthop Trauma 2013;27:467-471.

31. Storino A Castillo-Angeles M Watkins AA et al. Assessing the accuracy and readability of online health information for patients with pancreatic cancer. JAMA Surg 2016;151:831-837.

32. Dunne S Cummins NM Hannigan A et al. A method for the design and development of medical or health care information websites to optimize search engine results page rankings on Google. J Med Internet Res 2013;15:e183.

33. Soulo T. Organic traffic. Ahrefs’ SEO metrics: what they mean and how to use them. https://ahrefs.com/blog/seo-metrics/#section7 (Date last accessed 21 December 2020 ).

34. Redish JC. Understanding the limitations of readability formulas. IEEE Trans Prof Commun 1981;24:46-48.