Social Media Index: Entering the Debate

On February 1, 2016, the St Emlyn’s blog published a post by notable FOAMite Dr. Simon Carley (@EMManchester) that critiqued the Social Media Index (SMi) [/fusion_builder_column]

[1]. The SMi is an impact metric for emergency medicine (EM) blogs and podcasts that has been hosted on ALiEM for the past few years [2]. This post provides background on its creation and evolution and responds to some of these criticisms. It will be followed by a more philosophical post by Teresa Chan (@TChanMD) on CanadiEM tomorrow.

The Conception and Evolution of the Social Media Index

In June of 2013 while attending the Canadian Association of Emergency Physicians Annual Conference, I discussed the scholarly legitimacy of medical education blogs like ALiEM and BoringEM (now known as CanadiEM) with noted Canadian EM educator Dr. Jonathan Sherbino (@sherbino). While this topic has since been addressed in much more depth [3,4], one of the key quotes that I took away from our conversation was:

It’s all about the impact! – Sherbino

By this he meant that if the impact of a medical education blog could be quantified in some way it would be much easier to justify it as a form of academic scholarship. This quantification would demonstrate to the academic world that the work myself and others put into our blogs and podcasts went beyond what “anyone could put on the internet.” My musings about an impact factor for blogs and podcast began shortly after and ultimately resulted in the development of the Social Media Index. I hoped that by developing an impact metric:

• Learners, practitioners, and educators would be able to identify high-impact sites
• Bloggers and podcasters would be able to quantify the relative impact of their blogs and podcasts
• Institutions would be able to understand the impact of their faculty members
• Researchers would have a way to identify blogs and podcasts for study

While the index has changed markedly since its inception, often in response to criticism [5], these goals have not. The extent to which it has met them is debatable, but I have been pleased to see it make a mark on the scholarship of FOAM. It entered the literature with a description of SMi’s derivation and validation in WestJEM [6] and has since played a role in a study evaluating blog and podcast quality [7,8] and identification of resources for the ALiEM AIR Series [JGME in press]. These studies spurred a research agenda on the evaluation of blogs and podcasts that already exceeds what has been done to evaluate the quality of textbooks and lectures [9].

Responding to the Critique

As outlined in the post on St. Emlyn’s, the SMi has flaws. This critique focused primarily on the components used to calculate the SMi:

1. Alexa Score
2. Facebook Likes
3. Twitter Followers

While the conversion of these metrics to the SMi is complicated, the rationale for using them is quite simple. I believe two things:

• Medical professionals are the primary audience of FOAM websites, and
• Medical professionals do not visit (Alexa) or follow (Facebook and Twitter) low impact or low quality websites.
  • The Alexa Score is the most widely accepted publicly available marker of web traffic. The gathering of individual analytic data from each website is not feasible so this widely used metric is the next best thing. Facebook and Twitter are the two most widely used social networks in the FOAM world and “Likes” and “Follows” on these networks seem to be an even stronger endorsement. By combining these data points and comparing the results between websites I believe that we can paint a reasonably accurate picture of the impact of individual blogs or podcasts.

Another criticism was that the SMi can be gamed by purchasing followers on Facebook and Twitter. This is true for the SMi as well as every other metric. If we keep waiting for someone to develop a metric that cannot be gamed we will never have one. Given that the purchase of followers does not seem to be widespread in the FOAM community I do not think it invalidates the entire metric, although it is certainly something to consider. Ultimately, I suspect that if you ask knowledgeable and engaged members of the FOAM community whether they feel that the websites at the top of the SMi are impactful and respected resources, I think that they would agree. While face validity represents validity in its weakest form, in a world quickly being overpopulated by websites the ability of someone outside of the FOAM community to take a quick look at a list and determine where to start has some power and is worthy of further investigation.

Beyond the Social Media Index

FOAM has evolved from an amorphous, unstructured, rebel entity into a movement that is widely recognized. For better or worse, further growth will result in additional organization, measurement, and scholarship (check out CanadiEM tomorrow for more on this). To that end, the team behind the SMi has been far from idle following its development [8]. While we feel that the SMi serves a purpose, we strongly believe that developing ways to assist learners and practicing clinicians to critically appraise the quality of online resources will take on increasing importance in the future. We have published multiple studies looking past impact into quality and are now recruiting medical student, EM resident, and EM attendings/consultants for our biggest studies yet (@METRIQstudy). If you would be willing to assist us in this endeavor, please consider signing up for the METRIQ study which will involve the ranking of blogs (part 1) and podcasts (part 2) using several quality evaluation tools.

In closing, I would like to thank the crew at St Emlyn’s Blog for featuring a post on the SMi on their website. As someone who often critiques the research of others online, I am humbled that this influential group deems our work worthy of coverage and happy to see this discussion elevated.

References

1. Carley S. The Social Media Index (SMi): Can & should we measure #FOAMed? St Emlyn’s Blog. February 1, 2016.
2. Thoma B. The Social Media Index.Acad Life Emerg Med. November 2015.
3. Thoma B, Chan TM, Benitez J, Lin M. Educational Scholarship in the Digital Age: A Scoping Review and Analysis of Scholarly Products. The Winnower. 2014; 3:e141827.77297. DOI:10.15200/winn.141827.77297.
4. Sherbino J, Arora VM, Van Melle E, Rogers R, Frank JR, Holmboe ES. Criteria for social media-based scholarship in health professions education. Postgrad Med J. 2015; 91(1080): 551-5. PMID: 26275426
5. Thoma B. Social Media Index: Controversy and Evolution. Acad Life Emerg Med. November 24, 2013.
6. Thoma B, Sanders JL, Lin M, Paterson QS, Steeg J, Chan TM. The social media index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015; 16(2): 242-9. PMID: 25834664
7. Thoma B, Chan TM, Paterson QS, Milne WK, Sanders JL, Lin M. Emergency Medicine and Critical Care Blogs and Podcasts: Establishing an International Consensus on Quality. Ann Emerg Med. 2015; 66(4): 396-402.e4. PMID: 2584084
8. Lin M, Thoma B, Trueger NS, Ankel F, Sherbino J, Chan T. Quality indicators for blogs and podcasts used in medical education: modified Delphi consensus recommendations by an international cohort of health professions educators. Postgrad Med J. 2015; 91(1080): 546-50. PMID: 26275428
9. Thoma B. The METRIQ Research Agenda. METRIQ Study. 2016.

Author information

The post Social Media Index: Entering the Debate appeared first on ALiEM.