While talking with various CME industry stakeholders about peer review of CME content during the past months, I observed a lack of consistency in the definition of the term, employment of a process, and reasons for using or not using reviews.
As a first step to better understanding the role of peer review in CME, I conducted three online surveys with subsets of accredited providers, pharmaceutical companies, and CME consultants. The surveys addressed the same issues from each of the three different perspectives.
My goal was to get a point-in-time snapshot of where stakeholders are regarding the adoption of peer review of CME content since the Accreditation Council for CME first put it forth in October 2004 as an example of a mechanism for resolving conflicts of interest. Although sample sizes are limited, I found the results interesting and thought others might as well.
Asked whether they use peer review, the answer from providers was a resounding “yes.” Of those providers surveyed, 90 percent indicated that they have or are implementing a process for peer review of CME content, or have either a case-by-case content review process or use content review as a second step for conflict resolution. Only 8 percent said that they have no plans to implement a peer-review process. (Two percent answered “other.”) Among consultant respondents, 77 percent indicated that they strongly recommend to their clients that they include peer review of CME content; 23 percent neither recommend nor discourage it.
Respondents articulated their commitment to peer review, despite the challenges it poses. “We always run our activities through content validation and peer review,” wrote one provider. Another stated: “It is costly, time-consuming, but worth it.” A consultant respondent detailed the obstacles faced with peer review: “Peer review provides valuable feedback for improving the quality and effectiveness of the activity, prior to it being implemented. One of the greatest challenges is obtaining a sufficient representation of content to send to peer review. Speakers complain that no one else requires a description of content, much less with identification of evidence-based literature findings.”
With the advent of the Accreditation Council for CME Content Validation Policy in 2002, the promulgation of the updated Standards for Commercial Support in 2004, and the ever-increasing scrutiny of bias in CME, providers' reasons for employing peer review were not surprising. Participants were asked to rank their reasons, using a rating scale of 1 to 5, with 1 being the primary reason. The primary reason cited was conflict-of-interest resolution (compliance with Standard 2.1), followed by verification of balance and absence of commercial bias (compliance with Standard 5) and content validation (compliance with ACCME's Content Validation Policy).
Consultants were asked to rank their reasons for recommending the inclusion of a peer-review process. Their responses differed only slightly from those of the providers: They ranked content validation first, followed closely by verification of balance and absence of commercial bias, and as a mechanism for resolving conflict of interest.
Providers who indicated that they have “no plans to implement a peer-review process” were asked to rank their reasons for not employing one. The top three reasons indicated were “staff hours required to coordinate,” followed closely by “peer review is not required,” and “cost” to implement.
Consultants were asked to identify, based on their experience, why clients choose not to employ a peer-review process. Their responses varied somewhat from those of the providers. Although the primary reason — “staff hours to coordinate” — was consistent with that of providers, they identified “cost” and “difficulty identifying reviewers who are without conflicts,” as second and third respectively.
One provider commented that the process just doesn't apply to certain CME settings. “The questionnaire reflects a misperception of the informal processes at medical schools and hospitals. Most of our speakers are local and present to each other regularly. The content of their presentations is informally assessed all of the time. This is generally an excellent predictor of future performance. With this informal, ongoing peer assessment/review process in place, we have little need for formal, proactive peer review of most CME presentations. The speakers also know that a biased presentation will affect their peers' perceptions of the speaker in the future. The only times we are likely to use formal, proactive peer review are when we produce a meeting where the speakers are mostly not from our institution and the activity has sufficient commercial money involved and audience size (e.g., a satellite meeting) that the risks are higher for bias. This is a very small fraction of the more than 300 activities our institution provides annually.”
Survey participants were asked to rate whether they strongly agreed, agreed, disagreed, or strongly disagreed with the statement: “One way to improve CME and the CME/pharmaceutical industry relationship is to have full peer review of all CME materials.” Respondents from all three groups overwhelmingly agreed.
Seventy-five percent of provider participants indicated that they either agreed (55 percent) or strongly agreed (20 percent); 20 percent disagreed; and 5 percent strongly disagreed.
Ninety-two percent of consultant respondents said that they strongly agreed (50 percent) or agreed (42 percent); 8 percent disagreed.
Seventy-seven percent of pharmaceutical company participants either strongly agreed (38.5 percent) or agreed (38.5 percent); 23 percent either strongly disagreed (15 percent) or disagreed (8 percent).
I asked providers, based on their experience, if grantors currently view peer review of CME content as a requirement, an advantage, a disadvantage, or neither an advantage nor a disadvantage. Providers were nearly evenly divided in viewing it as a requirement (30 percent), an advantage (33 percent), and neither an advantage nor disadvantage (34 percent). Only 2 percent indicated that grantors view it as a disadvantage. (Figures do not add up to 100 percent due to rounding.)
Consultants were asked, based on their interactions with grantor representatives and the collective experience of their clients, how they think grantors view peer review of CME content. Seventy-five percent of participants said grantors currently view it as an “advantage”; 25 percent said as neither an advantage nor a disadvantage.
The majority (72 percent) of pharma respondents said that their grant request committee considers peer review of CME content either a requirement (39 percent) or an advantage (33 percent); 22 percent said that it was neither an advantage nor a disadvantage. Six percent (just one respondent) said it was a disadvantage.
As indicated in the numbers of providers doing peer review, they're committed to the process regardless of whether it is a funding advantage. As one provider stated: “We have not experienced requests by grantors to conduct peer reviews — we do it to protect the integrity of our activities. I doubt that our grantors even know we are doing this on a regular basis.”
So, if a significant number of providers have processes in place, all of the stakeholders are in agreement that peer review is one way to improve CME and the CME/pharmaceutical industry relationship, the vast majority agree that it is an advantage to include information about the process in the grant request, and we have fairly consistent information that the cost of implementing such a process is an issue for providers, then one would expect that there would also be consistent information on the funding of reviews. Right? Not necessarily.
Providers, except for those who had no plans to implement a peer-review process, were asked if, in their experience, grantors are consistently funding, sometimes funding, not funding, or haven't been asked to fund peer review as part of the grant amount. Given concerns regarding the cost of conducting reviews, the results were surprising. Fifty-seven percent indicated that either they had not requested funding (37 percent) or they did not know if grantors were funding (20 percent), 22 percent indicated that grantors are consistently funding, 15 percent said they are sometimes funding, and 6 percent said they are not funding.
Consultant responses were consistent with those of providers. Based on their interactions with grantor representatives and their clients, 50 percent of consultant participants either indicated that providers were not requesting funding (25 percent) or acknowledged that they did not know (25 percent). Forty-two percent said that grantors were funding peer review either sometimes (25 percent) or consistently (17 percent); 8 percent indicated that peer review is not currently being funded by grantors.
Pharma participants were asked if they are funding the cost of peer review for CME grants. As one might expect — given the current environment and the small sample size — the results for this question were very fragmented. Twenty-three percent indicated that they were funding “sometimes, depending on the quality of the review”; and 23 percent said that their company has not taken a stance on this matter. Fifteen percent said that they currently fund grants “always”; 15 percent said they sometimes fund, depending on cost; and 15 percent said they did not know. Only 8 percent said that they “never” fund the cost of peer review. (Figures do not add up to 100 percent due to rounding.)
Not surprisingly, the provider perspective on who should pay for peer review can differ considerably from industry's perspective. One provider wrote: “If we are to utilize peer review for everything, it would be costly in time and money. Would industry support [peer review]?”
As if to answer that question, a pharma respondent commented: “Peer review should be an integral part of the grant request and process. It should not cost extra for this to be completed. Providers should use this process to their advantage to show fair balance and lack of bias in their programs.”
What surprised me most about the results of the survey were the number of organizations that are conducting peer review of CME content. For a process that is not “required” by any regulatory body, the practice is pervasive. I was also surprised by the number of provider respondents and the time they took to provide comments. To me, this demonstrates their need to identify best practices, which I will attempt to do in the next installment of this two-part series.
Jane M. Ruppenkamp, a 15-year CME veteran, is founder and CEO of CME Peer Review, LLC, Alexandria, Va., an independent peer-review/content-validation service. She is also a partner in PTR Educational Consultants, LLC, Bristow, Va. Contact her at jruppenkamp@cmepeerreview.com. For complete survey results, free of charge, visit www.cmepeerreview.com.
Ruppenkamp thanks all the survey respondents, as well as Jacqueline Parochka, EdD, and Richard Tischler, PhD, for their contributions.
The accredited-provider survey was e-mailed to 429 individuals at 265 organizations representing all provider types on April 3, 2006. Individuals were past participants in industry conferences, selected based primarily on their organizational title. A reminder e-mail was sent on April 7 to those who had not yet responded. A total of 368 e-mails were delivered and 133 responses from 122 companies were received for an individual response rate of 36 percent and a company response rate of 46 percent. The survey consisted of 13 questions and an open-ended comment section.
The consultant survey was e-mailed to 27 industry consultants on March 20, 2006. Individuals were past participants and speakers at industry conferences who identified themselves as consultants; all e-mails were delivered. A reminder e-mail was sent on March 22 to those who had not yet responded. A total of 13 responses were received for a response rate of 48 percent. The survey consisted of 10 questions and an open-ended comment section.
This survey was e-mailed to 117 individuals at 32 companies on March 17, 2006. Individuals were past participants in industry conferences, selected based primarily on their title in their organization. A reminder e-mail was sent on March 22 to those who had not yet responded. A total of 113 e-mails were delivered and 21 responses from 12 companies were received, for an individual response rate of 17.9 percent and a company response rate of 37.5 percent. The survey consisted of 10 questions and an open-ended comment section.