TWO YEARS AGO, our strategic partner, Axdev Global, an educational company in Norfolk, Va., told me that return on investment is only a tangential concept in CME; but what we can demonstrate is return on educational investment. Now with the PhRMA code and the OIG Advisory, the concept of ROEI is a reality with commercial supporters making decisions based on educational excellence instead of marketing savvy. Academic and private sector organizations offering outcomes evaluation services (see box) in Accreditation Council for CME Element 2.4, evaluating the effectiveness of CME activities in meeting identified educational needs, have become de rigueur.

Measure More Than Satisfaction

Standard evaluations offered at the end of an activity typically measure satisfaction and knowledge gained. What we really want to determine is whether the educational activity affected physician behavior. And, in some cases, we can determine if the activity had an impact on patient care.

If you read my article, “Linking Needs to Outcomes,” (December 2003 MM, available online at mm.meetingsnet.com), you might remember that CME activities are based on needs, and needs are linked with desired educational results, learning objectives, educational design, and outcomes. A few tips: Prepare your outcomes questions at the same time that you are planning the event — don't wait until after the activity. For a one-time, short CME activity, keep the questionnaire to one page to make it easy to respond.

From Evaluations to Action

Offering an outcomes evaluation is not enough. If you're an accredited provider, the Accreditation Council for CME will want to know what you do with the information gathered. At a minimum, you should do the following, and document the process:

• Review the results of the evaluation with your staff and CME Committee or Advisory Board. Set a criterion as to what level of implementation is considered effective. There is no set answer to this. If the results are not acceptable, chart a course of change in your educational process and see what happens.

• Aggregate the results of your outcomes evaluations over a period of time (e.g., quarterly, annually) and review them with staff and the CME Committee. Look for trends and anomalies and analyze them.

You don't have to perform outcomes evaluations on all activities. It is up to you to determine which activities you will evaluate. Once you establish the criteria, evaluate consistently. You might consider inviting just a sampling of attendees to participate in a panel to perform a battery of evaluations.

While it is true that our current methods of outcomes evaluation rely on self-reported information, this still provides information on the effectiveness of CME. Moreover, the literature shows that learners typically do not change behavior with only one intervention, but this is a first step, and it does provide those who invest in CME with some indication of ROEI. And, it does begin to respond to those nay-sayers who believe that CME is a poor relative within the family of medical education.

Steven M. Passin is president of the CME consulting firm, Steve Passin & Associates in Newtown, Pa. He has also served as deputy health secretary for California. Contact him at Passin@PassinAssociates.com.

OUTCOMES RESOURCES

Axdev Global: www.axdevgroup.com
Outcomes Inc.: www.ceoutcomes.com
Veritas Institute: www.veritasime.com
Designing Solutions: www.dspesg.com