How you can develop a good educational outcomes measurement plan?
Accreditation agencies, government, and grantors are requiring continuing medical education providers to demonstrate return on education, or ROE, for their activities. Providers are being asked to submit reports with information on their CME activities' reach (the number of physicians who participated), and evidence that learners improved their knowledge and competence and that physician practice patterns/clinical performance was enhanced. With commercial support for CME on the decline and demands from government for CME transparency increasing, we need strategies to incorporate educational outcomes measurement, or EOM, that will demonstrate ROE in terms of physician knowledge, competence, performance, and patient health improvements.
Optimal EOM begins with identifying the educational need. What is the practice or knowledge gap? What are the barriers to physicians practicing according to the ideal standard of care? Consider the use of control surveys to validate the need and uncover the practice patterns/knowledge levels of your target audience. On these surveys, ask if your audience is familiar with the information you plan to teach them to determine their knowledge and competence levels. Discovering how close they are to established clinical guidelines will help develop a baseline. Are there barriers previously unforeseen that prevent them from practicing according to the established standard of care? Keep the questions simple.
When formulating an EOM plan, consider:
What is the educational intervention designed to achieve? To build awareness for new treatment options, reinforce established evidence-based practice, or encourage the adoption of updated clinical guidelines?
Are the learning objectives measurable in terms of physician knowledge, competence, performance improvement, and/or enhanced patient outcomes?
How much change in knowledge or behavior can be expected to result from the activity or activities?
Once you have done this, you can design the learning objectives and EOM modalities to address these needs. Work with faculty to develop one or two primary learning objectives in each activity.
As providers and faculty collaborate to develop the EOM plan, it is important to link these questions to the demonstrated need. For example, if the educational gap is related to understanding the symptoms of cardiovascular disease in order to make an accurate diagnosis, consider a multiple-choice question asking participants to identify the most common symptom(s) of this disease. Case-based learning provides a direct measurement path between the adoption of new knowledge and self-reported practice performance. Providers and faculty must ensure the EOM modalities are appropriately matched to the educational design/format.
Most participants are going to consider the information provided in CME activities and balance it against their own clinical experience, peer-to-peer discussions, complementary self-education/assessment, and the unique needs of a diverse patient population. Therefore, it is important to note that shifts in knowledge of greater than 0.5 on a 6-point Likert-type scale are rare.
No activity accomplishes all of its goals in a single exposure. Often, multiple interventions are required.
No matter what EOM plan CME providers implement, we must refine our measurement tools and move toward developing standardized EOM modalities that our industry can integrate easily into practice. Standardized measurement will help providers make the results of our activities transparent to accreditation agencies, grantors, and the public.
Ann C. Lichti, CCMEP, is the vice president of Health Care Education Strategy for Veritas Institute for Medical Education Inc. Reach her at ann.lichti@veritasime.com. The opinions expressed are those of the author and do not constitute the views of Veritas Institute for Medical Education Inc.