While peer review can help you document the integrity of your activity and provide a mechanism for resolving conflicts of interest with faculty and planners, setting up and maintaining an effective process can be cumbersome and time consuming, and requires thoughtful planning. But as one CME provider commented: “It's worth it.”
In “Do You Peer Review?” (June 2006 issue, page 22) I shared the results of the survey I conducted this past spring with accredited providers, consultants, and commercial supporters. As a follow-up, this article provides a 10-step checklist for developing a peer-review process.
- Clarify Your Purpose
It is clear from the survey results that providers have many different ways of implementing review processes; however, their primary reasons are the same — resolution of conflicts of interest (Accreditation Council for CME Standard for Commercial Support 2.1), verification of balance and the absence of bias (Standard 5), and validation of content (ACCME policy). According to “Ask ACCME” (on its Web site at www.ACCME.org), peer review may also be used after the activity, in which case “it is a monitoring process-and could produce information on the effectiveness of the provider's mechanism to resolve conflicts of interest. System improvements may follow, if warranted.”
- Define peer
Survey results also demonstrated that providers are inconsistently defining the term peer. Some seek subject matter experts and others seek members of the target audience. In addition to physician specialists, providers also utilize nonspecialist physicians, pharmacists, nurses, and other healthcare providers to conduct reviews. Some use members of the CME staff, program committee, advisory board, editorial board, and association — to name just a few other resource pools.
Just as important as conducting the review is the provider's responsibility to consider reviewer qualifications and potential conflicts. If the definition you develop for peer is broad, identify the circumstances that will determine the type of reviewer you will use in various circumstances. Will you always utilize an expert or a member of the target audience? Would the type of reviewer be determined by content, target audience, availability, etc.?
- Qualify and train reviewers
Here we can borrow from the faculty selection process. First, establish selection criteria. Determine what knowledge, skills, and experience the reviewer will need to effectively review the content. Then, solicit healthcare providers and review curricula vitae based on selection criteria to identify qualified reviewers.
Just as we train faculty, we should prepare reviewers. Create training materials and educate reviewers to ensure they understand the reason for and context of the review.
- Decide how reviewers will evaluate content
Is it your policy to review all CME content or only the activities that have known conflicts? What will the review document — e.g., balance, objectivity, absence of commercial bias, scientific rigor? Establish criteria based on your purpose for conducting the review, and develop a tool that reviewers use to evaluate the criteria. If corrective actions are necessary, is a subsequent review required? If so, determine what warrants a subsequent review, how it will be conducted, and by whom. Establish what information the reviewer will receive regarding conflict of interest that may be present.
- Require reviewers to disclose
For the same reasons that you require faculty to disclose, you must also require reviewers to do so in accordance with the ACCME Standards for Commercial Support and your internal disclosure policy, and you need to provide regular updates to their disclosures.
- Identify reviewers' conflicts of interest
Determine conflicts of interest in accordance with the ACCME Standards and your internal policy. Immediately disqualify reviewers who have an identified conflict with the activity.
- Build peer review into the CME planning process
Who on your staff will be responsible for coordinating the review process? Identify turnaround time for reviews (and subsequent reviews, if necessary). Make sure timeframes are realistic for the reviewers who you plan to use. Allow time for corrective changes, as well as discussions with your faculty, activity director, and/or program committee after the review has been completed.
- Make changes as a result of the review
Determine who will be responsible for reviewing the feedback, deciding changes, and communicating potential changes to faculty. For example, will it be the director of CME, the medical director, activity director, chair of the CME advisory board, or another member of the organizational leadership?
- Document the process
Those who are familiar with the accreditation process understand that it is not enough to have a mechanism in place; it must also be documented. Establish who will sign the completed review in addition to the reviewer (e.g., CME leadership). Place a copy of the review in the activity file along with a copy of changes made as a result of the review to demonstrate that you both have a mechanism and you follow the process that has been outlined.
- Measure effectiveness
Revisit your purpose for implementing the process and determine how you will know if it is successful. Consider how you will monitor results, measure outcomes, and continually improve the process. For example, a relatively easy way to start might be to monitor feedback on commercial bias gathered from the post-activity evaluation.
To share your best practices for peer review, contact me at jruppenkamp@cmepeerreview.com. I will compile and share your responses.
Jane M. Ruppenkamp, a 15-year CME veteran, is founder and CEO of CME Peer Review, LLC, Alexandria, Va. She is also a partner in PTR Educational Consultants, LLC, based in Bristow, Va. For complete survey results, visit www.cmepeerreview.com
Emerging Best Practice: External Review Process
The vast majority of CME providers who responded to our survey reported that they have an internal peer-review process — one in which the reviewer is a staff member. However, the majority of consultants recommended an external process, such as a contract with external healthcare providers, a physician group, or an independent third party. Most commercial supporter respondents indicated that an external process would be considered an advantage in the grant review process. The potential ethical implications of an internal process (e.g., perceived conflicts of interest created by the reviewers' relationships with their employer including revenue goals, grantor relationships, timeline constraints, and internal politics) make external review an emerging best practice.
For more articles on CME, click here.
