Social media as health educator: An assessment of the understandability and accuracy of tiktok content about contraception

Abstract

Background

Contraception knowledge and attitudes are largely formed from conversations within one’s social network. More recently, this network has expanded to include social media. As the fastest growing social media platform, we aimed to assess popular contraception videos on TikTok for content understandability, actionability and accuracy.

Methods

This is a secondary analysis of the most viewed, contraception-specific TikTok videos that were previously coded as containing educational messaging in a content-analysis study. We assessed videos for understandability (ability to explain a key message) and actionability (identify what they can do to act on that message) using the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V) instrument and for Currency, Relevance, Authority, Accuracy, and Purpose (CRAAP) using a modified instrument within the health information education literature.

Results

The 174 videos with educational content scored as understandable but not actionable in PEMAT-A/V assessment, with videos created by healthcare providers (HCPs) performing better than those by non-HCPs in six of eleven domains of understandability. Videos overall scored well in relevance and purpose within the CRAAP assessment. Videos created by HCP’s (n = 99) scored higher than those by non-HCPs (n = 75) in relevance (score = 3.9 versus 3.3), authority (score = 4.8 versus 1.7), accuracy (score = 6.0 versus 3.5), and purpose (score = 7.1 versus 5.7) [all p < 0.001]. Regarding video engagement, the median number of views among the videos in the cohort was 604,450. Of all video views, 79.2% were of HCP-created videos and 20.8% were of non-HCP videos.

Discussion

Much of the contraceptive educational messaging on TikTok is understandable, relevant, and accurate. HCP-created videos overall scored better as compared to other creators, though even HCP-created videos score fail to provide actionable recommendations. The videos created by HCP also had greater metrics of engagement. Given this, we encourage HCPs to use social media to better inform its users.

Background

Patients rely on social networks for contraception decision making [1], networks that are now largely comprised of people who post on social media [2]. Family members, peers, and social media have all affect reproductive-aged individuals’ contraception attitudes and behaviors [3]. TikTok is the world’s fastest growing social media platform and users ages 18–24 years are its largest demographic [4, 5]. A prior content analysis study [6] of the 700 most viewed contraception TikTok videos categorized 36% of the videos as educational, defined as informing or teaching about a birth control method. Close to half of the educational videos (49.6%) were created by healthcare providers (HCPs), and these videos had much higher levels of user engagement (views, shares, likes, comments). The other half of educational content videos were created by non-HCPs informed, most commonly, by the personal experience as current or former birth control users rather than by professional expertise. Recent reports suggest a rise of contraceptive misinformation on social media, often urging users to switch to less effective contraception, particularly non-hormonal methods, such as “natural cycles” [7, 8]. Furthermore, those who primarily derive birth control information from the internet and social media tend to have greater concerns about hormonal methods than those whose main source of information is a healthcare provider or a school setting [9]. Younger people aged 18–24 years, the same demographic who most use TikTok, report lower levels of contraceptive access and use in recent years within the United States [10]. The potential for influential misinformation on social media is particularly alarming when reproductive options, particularly abortion, are limited through restrictive U.S. state laws. To gauge the degree of misinformation on TikTok, we performed a secondary analysis of the TikTok content analysis study by Stoddard et al. (2024) to evaluate the understandability, actionability, and scientific accuracy of those most-viewed videos containing contraception educational content [6]. These results represent the next step in understanding the potential cultural and personal influence of social media on people’s contraceptive decision making.

Methods

In Stoddard et al. (2024), we performed a content analysis of the 700 most viewed contraception TikTok videos at the time of download in December 2021 [6]. We searched based on a previously published list of hashtags associated with contraception on social media for each of the six birth control methods: intrauterine devices (#iud), implant (#nexplanon), oral contraceptive pills (#birthcontrolpills), vaginal ring (#nuvaring), birth control patch (#birthcontrolpatch) and birth control injection (#depo) [11]. In addition to this list, we added the search terms #birthcontrol and #planb [6]. From this original cohort, we coded 252 videos (36%) as educational, defined as “containing at least one take-home point or fact the creator is educating the viewer on” [6]. We limited this new analysis to just these videos coded in Stoddard et al. [6] as educational as we recognized we could assess accuracy only for those videos with factually derived, as opposed to emotionally derived, content (examples of the latter include jokes about birth control or political messaging).

To measure video understandability and actionability, we used fourteen validated measures within the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V) [12], an instrument that has previously been used to evaluate the understandability of social media content related to the IUD [13] and vasectomy [14]. PEMAT-A/V describes understandable materials as those that allow individuals with varying educational backgrounds and levels of health literacy to “process and explain key messages” of the materials. Actionable materials allow for these individuals to “identify what they can do” with the information [12]. We rated each item on a binary scale of yes (1) or no (0). PEMAT scores were summed across individual videos and calculated as a percentage of the sum of the total points scored by the total points possible; higher percentages suggest greater understandability and actionability [12]. Two reviewers (JKC, MEH-K) independently scored videos, and a third (DB) adjudicated discrepancies reaching substantial agreement for all items (Cohen’s Kappa > 0.7).

To assess video reliability and accuracy, we used relevant questions from a previously modified version of the Currency, Relevance, Authority, Accuracy, and Purpose (CRAAP) test [15], which has been employed in other studies evaluating the accuracy of online health information [16,17,18]. Initially developed to qualitatively assess the reliability of online educational resources [19], the CRAAP test has been adapted for quantitative assessment of health information scored with points within the domains of currency (0–5), relevance (0–4), authority (0–5), accuracy (0–7), and purpose (0–8) (Supplement) [15]. Compared to other instruments to evaluate medical information, the CRAAP test’s criteria focus on broad patient education rather than discrete, specific medical content, and thus is more suitable for social media content [19]. Two board-certified gynecologists (DB, NJ) coded the videos, with discrepancies adjudicated if Cohen’s Kappa was < 0.7.

We reported frequencies for each question within each domain for both PEMAT-A/V and CRAAP assessments. We performed a Chi-square test to compare groups for each question. We calculated means and standard deviations and medians and interquartile ranges (IQR) for summative domain scores. We used an independent samples t-test to evaluate differences in means and a Wilcoxon rank sum test to test medians between groups. We used Stata version 16.0 for analyses. This study was declined for review by Mass General Brigham IRB as non-human research.

Results

Of the 252 videos coded as educational within the original dataset [6], 174 videos were included in this secondary analysis. Seventy-eight of the original videos were ineligible due to deletion (n = 46) or restriction (n = 16) of creators’ accounts since the original analysis, content being too brief or in a form that precluded assessment by the PEMAT or CRAAP instruments (n = 11), or nearly duplicate videos from the same creator (n = 5). Not all videos could be evaluated within each question of each domain, leading some videos to be scored as “not applicable.” Physicians, physician assistants, midwives, and nurses as HCPs created 57% (n = 99) of the eligible videos: these videos on averaged had more views than those created by non-HCPs. The total number of views for all videos was 424,306,029 and the median was 604,450 per video with 9.8% of videos having over 5 million views. Of the 17 videos having over 5 million views, 15 (88.2%) were HCP-created and 2 (11.8%) were non-HCP created videos (p = 0.008). Of all views, 79.2% were of HCP-created videos and 20.8% were of non-HCP videos. The total number of likes was 32,647,492 and the median number was 58,100 per video with 60.5% coming from HCP-created videos and 39.5% from non-HCP videos. The total number of shares was 1,192,437 and the median number was 915 per video with 55.4% from HCP-created videos and 44.6% from non-HCP videos.

The mean score of all educational videos on PEMAT-A/V criteria for understandability was 74.7% (SD ± 17.4%, range 28.6–100) and for actionability was 37.7% (SD ± 40.7%, range 0-100%) of the total possible points, with established thresholds set at “understandable” and “actionable” if scored at 70% or higher [12]. Specific domains of understandability and actionability are presented for all videos overall and when comparing HCP with non-HCPs. Educational videos created by HCPs were significantly better in three domains (purpose, logical sequence, and clear images) and significantly worse in three domains (using common language, defined medical terms, and clear wording) of understandability. Healthcare providers were significantly better in the actionability domain of breaking down information. There were no differences in the PEMAT-A/V summative scores by creator type (Table 1).

Table 1 Patient education materials assessment tool for audiovisual materials differences in mean scores by TikTok video creator type, N = 174

On the CRAAP assessment, HCP-created content demonstrated significantly greater summative scores for relevance (3.9 versus 3.3), authority (4.8 versus 1.7), accuracy (6.0 versus 3.5), and purpose (7.1 versus 5.7) than non-HCP-created content (all p<0.001) [Table 2]. All videos scored well on domains of currency, no matter the creator type. Overall, videos scored well in domains of relevance, though 33% of non-HCP created videos had overgeneralized their content beyond the specific relevance of the video topic. As would be expected, videos created by HCP scored very differently on all three domains of authority since the video content represents their professional expertise. Overall, the majority of videos scored well on domains of accuracy, though 26.7% of non-HCP content contradicted or could not be corroborated by published evidence. Almost 99% of videos were clear in stating their intended purpose, though the purpose differed between groups, with 99% of HCP creating content stating facts compared to non-HCP who presented a mix of facts (48.7%), opinion (43.2%), and propaganda (8.1%).

Table 2 Currency, relevance, authority, accuracy and purpose domains and differences in mean scores for contraception TikTok videos by creator type, N = 174

Discussion

Contraception content posted on TikTok for the purpose of teaching or informing was understandable, but not actionable, according to PEMAT-A/V thresholds when created by both HCPs and non-HCPs. Thus, according to this validated metric to evaluate health information for people with varying levels of literacy, most videos conveyed a clear message but did not provide viewers with further resources to act on or implement this message. Overall, videos demonstrated good currency, relevance, authority, accuracy, and purpose, though, as would be expected, HCP-created content had greater scores of relevance, authority, accuracy, and purpose. Given the U.S. cultural mores about reproduction and birth control, sometimes resulting in mis- or even deliberate dis-information, we specifically used the CRAAP instrument for assessment to address the nature, partiality, and biases of the information provided in videos. It was reassuring to find the strong overall scores on the CRAAP assessment. It is not surprising that the videos created by HCPs provided more accurate, objective, and evidence-based information in comparison to content by non-HCP rooted more often in personal experience. Prior studies have evaluated the quality and reliability of TikTok content focused on specific contraceptive methods, such as oral contraceptives and implants [20, 21]. Others have evaluated accuracy of contraception content on social media without examining quality or understandability of content [22]. This study analyzes both accuracy and understandability of a wide range of contraception content on TikTok created for and by people with varying levels of health literacy.

Our study has several implications related to TikTok’s utility in disseminating reliable contraception educational content. Given the platform’s popularity [4] and users’ high level of contraception video engagement [6, 23], TikTok has potential to strongly influence contraception decision making, particularly among younger users. Our cohort contained about 56% of videos created by HCPs, but these videos were quite popular, making up close to 80% of all video views. In an investigation of over 1,100 U.S. young people assigned female at birth aged 15–29, 82% said they prefer to get their information about birth control from medical professionals. Yet only 43% of that cohort had received contraception information from medical professionals in the past year [24]. Of note when considering our engagement metrics, we saw many of the HCP videos were made by a small handful of content creators who could be considered “influencers”. While it is beyond the scope of this paper to discuss engagement metrics or reasons behind the virality of these videos, the high viewership of these videos still indicates the reach of this content and the number of TikTok users, especially young women, who are exposed to the information provided.

Given this viewership coupled with peoples’ stated desire to get their birth control information from medical professionals and with our findings of greater accuracy, reliability and objectivity of those videos created by HCPs, we should consider using this platform more to reach and teach the public at large. In an age of long wait times for healthcare appointments and short appointment visit times due to overloaded clinic templates, medical professionals should consider public education through social media as part of their educational mission. Other studies of TikTok have demonstrated strong engagement with videos created by medical professionals for dermatologic conditions [4] and sex education [23]. Direct to consumer education and communication are common in modern healthcare with the rise of electronic patient chart messaging and direct to consumer medical advertising. Our findings demonstrate that even HCP-created videos score low on providing actionable recommendations. Resources have been developed to help guide medical professionals to create TikTok and other social media content [25, 26]. Given the large consumption of social media, this platform should not be overlooked or underestimated as a means of more effectively imparting our expertise and informing the public at large.

Strengths of this study include the high popularity of the videos with large engagement metrics and the use of two validated scoring metrics to assess both accessibility to users and the accuracy of information. This study was limited by the difficulties in adapting some of these validated questions to the short-form videos of TikTok. We attempted to minimize subjectivity of PEMAT and CRAAP assessment through the use of two coders and assessment of coder agreement. Of the original dataset from Stoddard et al. [6], only the videos that contained educational content could be assessed for understandability and accuracy with the PEMAT and CRAAP surveys, with another 4% of those excluded for being too short to be assessed using these instruments. This smaller cohort affects the conclusions that can be made from our results. Furthermore, since this is a secondary analysis of prior data collected in 2021, it is possible that changes have occurred in the content posted on TikTok. A next step in investigation will be to collect a new snapshot of contraception content on TikTok today and assess for changes between now and 2021 regarding content themes and accuracy. Lastly, our analysis is strictly on the content available and, to a limited degree what content TikTok users find appealing, as assessed through degree of views, likes, and shares. However, this line of inquiry does not assess what information they retain or what type of specific messaging they find interesting or helpful for contraceptive decision making.

Conclusion

In an era of more extreme U.S. abortion restrictions, contraceptive use as primary prevention for unintended pregnancy is particularly important. Contraception content on social media has the potential to inform, either positively or negatively, the contraceptive decision-making process. Our findings are overall reassuring, with the most viewed contraception videos on TikTok demonstrating understandable and accurate educational content. HCPs must be aware of the inaccurate content that their patients might be seeing and that may be informing their healthcare decisions. As informed reproductive health professionals, we should consider using this and other social media platforms ourselves to create and promote highly reliable contraception education to the public.