Every five years the CME office at the University of Wisconsin-Madison Medical School holds a strategic planning retreat. At the most recent one, held in 2000, George Mejicano, MD, assistant dean, CME, asked the group, “What are we doing well and what are we doing poorly?”
Despite the fact that the participants included a diverse range of CME stakeholders from the regional healthcare community — CME instructors; medical school faculty members; CE directors from the school of pharmacy, the school of nursing, and other medical schools; clinicians who represented physician learners; and people who represented patients — their answers were the same. They said: We wonder why you don't do research.
“Let me put it little bit differently,” explains Mejicano. “At a place like Wisconsin, research is king.
The Three-Year Plan
Mejicano, who has a master's degree in adult education, took the feedback seriously. He brought in Henry B. Slotnick, PhD, PhD, who has been researching how physicians learn for 15 years, to create a research facility. They developed an ambitious three-year plan. During year one, Slotnick and his colleagues began a qualitative study of how physicians learn across the medical school and career continuum, by interviewing about 40 medical students, residents, and practicing clinicians. The data collection should be done by early 2003. During this year, Slotnick also reviewed all the world literature on physician learning. And, because one of the goals of the project is to increase the CME office's visibility, they sponsored an international conference in October about changing clinician behavior, which attracted 60 experts.
The overall goal of the research plan, says Mejicano, is to develop a “universal theory” that can be applied to physician education across the spectrum. “I would not be surprised if CME research affects how we train medical students and residents. The sooner we start training people in ways that can affect their behavior, the more likely they will accept [those methods] as the way they live and learn during their entire careers.”
During year two, they plan to create a proposal for a major demonstration project, which would put the research results into practice. They will design and implement educational activities aimed at changing physician treatment behavior for a specific disease state, an area where the evidence-based literature indicates physicians frequently practice in a suboptimal manner. During year three, they will collaborate with hospital groups to conduct the project. They are working with hospital-based groups rather than individual doctors because “we want to change entire communities,” says Mejicano. To see if the project succeeded at changing physician behavior, they will document the outcomes by collecting information from doctors and from patient records.
Slotnick's excitement and confidence bursts forth as he describes the tasks ahead. “By dint of our conference last month, I have on my desk an integration of the most current thinking about how physician behavior might be changed — at the individual physician level, the community of practice level, and the system level. We have a better theoretical base as a rationale for our activities than anyone in the past. Our armamentarium will have more weapons in it than have been used before.”
If such a project sounds prohibitively expensive for the average CME program, you're right. Mejicano is fortunate that his predecessors developed a reserve fund which provided the project's seed money; the demonstration project will require outside funding. “The idea is that the research enterprise will eventually be self sustaining over a several year period,” says Mejicano. “The investment upfront will more than pay for itself both in terms of exposure and in terms of giving back to the community of learners. Hopefully, in the end, we will bring in some big grants. It's a long-term investment. It's not something you plunk your dime in today and see the return tomorrow. We'll see [the results] in five to 15 years.”
It may seem like research is a luxury you can't possibly afford. But Slotnick, Mejicano, and other research experts say that given the current environment, CME providers can't afford not to develop strategies, even if they begin on a small scale, to make their programs more effective.
Follow the Money
In fact, your ability to demonstrate in-depth needs assessments and evaluations can make a difference in getting commercial support. With the July release of the Pharmaceutical Research and Manufacturers of America's Code on Interactions with Healthcare Professionals, “the days of the dinner meeting at the local restaurant are numbered,” says Mejicano, “so those resources [will be redirected into] legitimate CME enterprises, and the drug companies are going to want a return on their investment. I'm optimistic that this change will promote research efforts.”
Other CME professionals are observing the same trend. “From what I've been seeing around the country there is a substantial change in what pharma is willing to buy now,” says Steve Passin, president of Steve Passin & Associates, a CME consulting firm in Newtown Square, Pa.” And they want proof that their money has been invested in an activity that has a high need, and they want proof that there's a return on educational investment. I underline the word educational investment because it's different than ROI.”
In response, says Passin, “we're suggesting to our clients that they put together what we call a needs assessment dossier for the pharma companies. It shows the level of needs assessment and the education's outcome in terms of physician behavior change. This positions our clients as the kind of CME providers that should be receiving money from pharma.”
“Pharmaceutical companies are beginning to be more vocal about their interests in CME effectiveness,” agrees Linda Casebeer PhD, associate professor, division of CME, University of Alabama School of Medicine in Birmingham. Interest is so strong, says Casebeer, who has spent 12 years in evaluations and outcomes research, that she and her colleagues have started a consulting firm called Outcomes Inc., that helps pharmaceutical companies measure CME results.
A research project may enable you to get government funds as well. “At the government level, there is a great deal more interest right now in research,” she says. The National Institutes of Health want to know what interventions are really effective in changing physician behavior and patient outcomes. UAB's CME office is currently working on four NIH grants.
Proving CME's Worth
Another reason to consider doing research to document your program's effectiveness is that organizations from the American Board of Medical Specialties to the Federation of State Medical Boards are demanding more accountability from CME. And so is the Accreditation Council for CME, which released new Essentials several years ago to help providers improve their programs. A number of CME offices, including the University of Wisconsin's (with the help of Passin and Slotnick), have received exemplary accreditation in Essential Area 2, which covers needs assessment and evaluation. Now the bar is being raised again by the American Academy of Family Physicians and the ACCME with their push towards evidence-based CME. All of those forces, says Passin, mean that providers have to “do good needs assessment upfront and measure their outcomes against that.”
Before Outcomes
Operative words: good needs assessment upfront. Note that Slotnick spent the first year of the current research project conducting needs assessments. But some providers, in their eagerness to demonstrate outcomes, skip doing their homework.
“Everyone in CME is tooting their horns about ROI, but you have to know where your starting point is to be able to optimize your ROI,” says Suzanne Murray, president, AXDEV Global, an educational company in Norfolk, Va., and Montréal, Québec. A prominent medical association recently asked her to measure the effectiveness of an activity that had already been designed. That approach, says Murray, is self-defeating. “If you're not doing the appropriate research upfront because the acquisition cost is much too high, you're not going to hit your target.”
Before even thinking about ROI, says Murray, providers need to identify the barriers that prevent healthcare professionals from changing their behavior. “It's complex. What are their attitudes? Their attitudes and their personal values are driving their judgment, which is driving their clinical experience and clinical decision-making. That's what you have to go after.”
First, Face to Face Interviews
It is exactly that complex, multilayered physician behavior process that Slotnick's research model aims to uncover.
His first step is to interview physicians face-to-face in nominal group process (NGP) sessions, which allow physicians to quickly identify and prioritize clinical problems. Slotnick and his colleagues used this process during a year-long research project with the American Epilepsy Society, which won a 2002 William Campbell Felch/Wyeth-Ayerst Award for Research in CME. They explored the learning needs of four groups of epileptologists in Boston — both AES members and nonmembers. About 800 people were surveyed; the impressive response rates varied from 40 to 60 percent.
“He doesn't just sit down and say, “What are your problems?'” says Sandra Pizzoferrato, associate director, AES, West Hartford, Conn. “He has a way of digging deep to find out what their issues and concerns are. If they think there's a problem, he asks, ‘Why is it a problem, and what can we do to help you overcome that problem?’”
Getting to the deeper layer involves phrasing questions differently than a typical needs assessment. Instead of asking, “What do you need to know?” Slotnick asked, “If you're going to hire somebody to do this job what are the basic issues that you expect this person to handle?” Says Patricia Shafer, RN, chair, allied healthcare subcommittee, AES, “That opened up discussions about the issues the participants struggle with day in and day out.”
And the results were a lot different than they were with previous needs assessments, says Shafer. “With prior assessments I've done with epilepsy nurses, patient and family education didn't come out high. People think, I can do that; I do that everyday. But when faced with examples of difficult situations, people realize, “That's the stuff that takes up my time. Maybe I do need more on that.” The nurses' need for more education about dealing with patients and families, “was the surprising but very positive and important message that I got out of the nurses' and physicians' data.” In fact, if she were doing it again, Schafer says she would include patients in the group process.
Another lesson she learned was that nurses who work in drug trials have very different needs than those who work in clinical settings. Looking back, she said that it would have been a good idea to conduct those interview groups separately. “I think that we need to pay more attention to the needs of nurses in clinical drug trials because almost 40 to 50 percent of nurses who attend AES meetings are hired under research grants.”
Ego Essentials
Some providers may be reluctant to convene focus groups for specialists because they are afraid of insulting them. However, Murray says that in her experience, high-end experts welcome the process. “They frequently tell us, ‘We're not used to being asked what our needs are. We're used to being asked what other people need.’” However, she adds the process will work only if it is done respectfully. “They're not necessarily comfortable in traditional formats just discussing and sharing their challenges. Because they're highly educated professionals, there are egos involved, so the focus groups have to be done in a credible research context — meaning you're not going to put their comments on the Web, you're not going to share them with everybody within your organization.” When compiling data, make sure it's anonymous and confidential, so that specific statements can't be linked back to specific respondents, she says.
But if you do it right, physicians do enjoy the process, agrees Marianna Shershneva, MD, a rheumatologist from Russia. Now Slotnick's research assistant, she is doing her dissertation at the University of Wisconsin. “Almost everybody said how pleased they were to have this opportunity to reflect, and that they know more about themselves than they knew before. It's important to reflect on your learning, and physicians don't do it often,” she says. They seemed to be relaxed — some of them spent more than [the allotted] one hour. It's amazing that they talked about some very sensitive things, such as when a patient died.”
Next Step: Vignettes
Slotnick next step is to design questionnaires, using vignettes developed from the problems that come out of NGP sessions. At the current University of Wisconsin project, almost 1,000 primary care physicians were sent surveys with vignettes; about 41 percent returned usable responses.
Here's one vignette example: “A middle-aged female refugee from Kosovo is brought to your office by a relative who explains the woman is a cousin who can't recall the last time she saw a doctor. They request that you do a “physical” and recommend whatever is needed for her to be healthy now that she is living in the United States. The woman claims to have no health problems or current illnesses.”
The physician chooses from the following answers:
Not Applicable: I wouldn't encounter a problem like [that one].
Confident: I'm confident my skills and knowledge are sufficient to address the problem posed in the vignette. All I might be interested in is information so I can decide when next to update.
Update: I need to update my skills and knowledge so I can handle the problem posed in the vignette.
Completed: I have recently updated the skills and knowledge needed to handle the problem posed in the vignette. All I might be interested in is hearing the experiences others have had handling such problems.
This type of survey is effective, says Slotnick, because “there is research evidence that physicians do a better job of recognizing problems they have than identifying their [learning needs.] Clinical descriptions call to mind not only the medical aspects of the problems, but the psychosocial and emotional aspects as well.”
One lesson learned: Customize the vignettes. Some pediatricians felt the vignettes did not apply to them and refused to participate, as the examples were of adult patients, even though a child could have the same problem.
Complicated and time-consuming as it is to gather the data, some of the toughest work comes afterwards. “The biggest challenge was asking, how does the data fit together,” says Shafer. When doing such in-depth needs assessments, make sure to plan enough “thinking time,” she advises CME providers. “Doing the groups is the easy part.” While the AES education council looked at the results, Shafer says there are more layers to be mined. “I'd like to see how nurse researchers would interpret it. I'd like to bring in independent researchers, maybe even from some different disciplines. It could be helpful to give you another take on it.” Unlike standard needs assessments, this more in-depth kind has a longer shelf life, says Shafer. “There's lots here that will serve us for the next few years.”
Commitment to Change
Once you've interpreted the data, then you have to take the leap and make changes in your program. “If you embark on a real, solid needs assessment but then don't take the findings seriously, and you end up offering something that fits your needs more [than the survey participants' needs], that is much more detrimental than not doing it at all,” says Murray. “What makes a needs assessment succeed is your organization's commitment right up front.”
The AES demonstrated that commitment. Changes were implemented quickly at the annual meeting. “In past years, particularly in the major sessions, the physicians were the educators and what came out of the study is that the nurses needed to hear more from their nursing colleagues,” says Shafer. As sometimes happens with research results, this wasn't news to Shafer, but it reinforced what she and many of the other nurses had thought for a long time. And the reinforcement gave them clout. “It allowed us to get our own symposium for Allied Health. We started it last year and we focused on one of the biggest needs, the cognitive and behavioral issues.”
Overall, Pizzoferrato found the project, which was funded through the society, quite valuable. “In the past we would do standard yearly evaluations that were conducted at our conference. We would ask, What topics are most important to you on the scale of 1 to 5. We found ourselves doing the same things over and over again. This study provided a new dimension. It's very useful as opposed to using expert opinion — actually when you think about it — it is expert opinion.”
That's exactly the point, says Murray. Needs assessments should not be solely based on, for example, specialists' opinions about what primary care physicians need. She uses this analogy: “Would you like us to ask your boss what your training needs are? Or decide what your education needs are without talking to you directly?”
Convincing Faculty to Change Formats
Needs assessments also result in changing education formats as well as topics, but persuading faculty to change their style can be difficult. At the Allied Health symposium, Shafer instructed faculty to use the same type of case-based approach that was used in the needs assessment. “It was challenging helping people refocus,” she says. “It means presenting for shorter periods of time, being more concise in presenting your content, and then using your expertise in a different way with the panel format.” Adds Murray, “We need to focus more on what's beneficial to the learner versus how can we make it easier for the teacher.”
Widening Circles of Change
Pizzoferrato also found that the AES project increased awareness about CME in the wider healthcare community. “It got the CME committee involved. It got the board of directors involved. The folks who participated in the [nominal group process] didn't think about CME as much before as they might now.”
Raising your office's profile benefits you in another way: It enables you to access the resources in your community. The Wisconsin CME office has been holding pizza lunches with people from the University and wider medical community. Quite a range of people have attended, including David Kindig, MD, PhD, a healthcare policy expert who served in the Clinton administration. “We have phenomenal resources available in this community,” says Slotnick.
Those kinds of discussions spark creativity and research efforts beyond the boundaries of an individual CME program. The University of Wisconsin project is generating interest in the Ukraine, adds Shershneva. “My colleague in the Ukraine and I are discussing the possibility of a study that may be related to [this] work. It's important to learn from each other.”
Meanwhile, Mejicano intends to share the lessons learned from the research project with other providers. “This isn't a proprietary activity. We genuinely believe that it is part of our responsibility to disseminate the knowledge and skill that we produce here to everyone.”
So You Want to Start a Research Project
If you're interested in starting a research project, here's some advice from the pros.
Ask yourself why
“Think very hard about why it is you wish to do research,” says Henry B. Slotnick, PhD, PhD, visiting professor, CME office, University of Wisconsin-Madison Medical School. “Do you wish to contribute new knowledge to the medical education community? Contribute to the profile of the shop? Because it's a way to bring in more money? Then you begin making plans that would address those reasons.
Consider collaboration
“The larger question is what is your role in the research world?” says George Mejicano, MD, assistant dean, CME, University of Wisconsin-Madison Medical School. “The hallmark is collaboration. You may not have the resources to do it on your own but I think that almost every provider would have the opportunity to collaborate with another institution that does research.”
Take Needs Assessment 101
Get a basic training on needs assessments, says Suzanne Murray, president, AXDEV Global, Norfolk, Va., and Montréal, Québec. An excellent resource, says Slotnick, is the Society for Academic CME. SACME puts on summer institutes for training CME researchers and also provides funding for research projects.
More than paper pushers
Research projects are extremely time intensive and it's important to get buy-in from your staff. “Get the staff to understand that they are no longer just pushing papers and doing logistics for CME — they have to become educators,” says Steve Passin, president of Steve Passin & Associates, Newtown Square, Pa. “They've been doing it so long they may not want to change. You've got to get them on board and interested in how they can grow themselves. It makes the job so much more satisfying. There's a new set of skills that you need today. We're not donut and coffeehouses anymore.”
Eyes wide shut
Staff apprehension is perfectly understandable, as research projects can add a huge burden to their workload, especially if there is no additional staff support. That's why it's a good idea to really understand what you're getting into, says Ann Bailey, associate director of CME, University of Wisconsin-Madison Medical School.” We needed to have a better understanding of what was really involved with this project. We went in a little blind,” she says. The nuts and bolts — scheduling focus groups, organizing mailings for surveys, and tracking the results so that respondents were not identified — “took a tremendous amount of staff time.”
Start with baby steps
To start with, do research in your core area of competency, suggests Passin. Adds Murray, “If you're targeting community-based primary care physicians as the audience for an educational intervention, [survey] them first.”
Research with a small r
“Sometimes we describe CME research — with a big R — when probably all of us are doing some research on a regular basis,” says Linda Casebeer, PhD, associate professor, division of CME, University of Alabama School of Medicine in Birmingham. “Much of what providers are already doing is research. They need to recognize that as important data, and use what they know already. I have a lot of faith in CME providers that they're already doing as much as they can with their resources.”
Starting Small: Matching Physicians' Personalities to Their Speciality
While the kind of research project described in the main story is far beyond the reach of most CME offices, research experts advise that you start small. Here's one example.
“I was curious. I never read anything about whether physicians are drawn to a specialty based on their personality type,” says Gordon West, PhD, CE specialist, Annenberg Center for Health Sciences in Rancho Mirage, Calif. “If that proves to be true, then maybe we can tailor educational offerings to meet the needs of specific specialties by dealing with their personality types and the educational preferences of those types.”
West decided to find out. He randomly pulled 1,000 physicians names from the database of about 13,000 physicians who earned CE certificates last year from Annenberg, and invited them to take the Myers-Briggs personality test. The costs were minimal as West is certified to give and interpret the Myers-Briggs test, and “Annenberg can afford the stamps.” He is tracking respondents not only by their scores and specialty, but also by their age, gender, number of years in practice, practice setting, and type of community in which they live and practice. His goal is to have at least 200 responses. When we talked to him, the numbers in any particular specialty were too small to draw conclusions, however he is beginning to see some trends. “In a couple of areas, the physician averages and the general population averages are diametrically opposed to one another.”
If the results are promising, West would like to expand the scope of this study. He is exploring possible partnerships with another CME provider and a business school. “The logical next step is to take a control group that goes through a normal program and another group that goes through a specifically designed one [matching personality types to learning formats], and measure outcomes right away and maybe at three months and six months.”
The results could also be used to customize how needs assessment questions are phrased, he adds.
