“Putting courses up on the Web is like feeding an elephant,” says Jann Balmer, PhD, director of continuing medical education at the University of Virginia School of Medicine, a registered nurse, and long-time advocate of online learning. “It's an activity that never ends. Why? Because part of the reason people like it is that it is immediate. That means we can't afford to let it become stale. You have to keep your finger on the pulse of it.”
Balmer seems to be succeeding in keeping the elephant fed. CardioVillage, her organization's CME Web site, has received 166,360 “educational” hits since its inception in November 1999 — that figure refers to people who actually went to the lessons, rather than those who just hit the home page. During the month we researched this story, April, the site registered 231 new users.
And that's despite various studies showing that physicians are a tad on the slow side in adapting online solutions of any sort. But CME may be the magnet other services aren't. In a survey released in March by WebSurveyMD.com, one-third or fewer of the respondents showed interest in communicating with patients or colleagues on the Internet; but 45 percent of respondents did use Web CME.
When talking with Balmer and others about how to draw physicians to online CME, it becomes apparent that a successful Web-based program cannot simply be a byproduct of your other CME activities. Here are Web Essentials based on the lessons learned — sometimes the hard way — by online CME veterans.
Part 1 Organizational Essentials
Research your audience carefully — and then take the numbers with a grain of salt. When the Philadelphia-based American College of Physicians — American Society of Internal Medicine (ACP — ASIM), was first considering offering online clinical problem-solving cases, it gathered a lot of needs-assessment information.
“The data showed there was very high interest among our members in online CME and CD-ROM,” said Theresa Kanya, vice president of education for the ACP — ASIM, at a session on online learning held at the Alliance for CME annual meeting in January in San Francisco. “We also had factual information about the number of physicians ordering CD-ROM products and signing up for our Web site.”
Based on initial research, the ACP — ASIM projected sales of 9,000 courses over four years. Since the launch in August 1999, exactly 850 physicians had signed up as of January 2001. Those who signed up liked the course, and ACP — ASIM has since broadened its marketing effort from individual physicians to group practices and distribution of the course through a partnership with a private firm, but the disconnect between the research and the outcome clearly stung. Especially after The Journal of the American Medical Association gave a very positive review to the course in April 2000, recommending it “without reservation.”
Along with determining interest by your audience in participation in online CME, it is important to determine what Balmer calls “their collateral skills,” that is, their ability to use a computer and a browser. “There will be a wide variation in your audience, from those who can barely point-and-click to those who could probably program their own sites,” she says. Talk to your Web partners, Balmer advises, who can help you figure out how to find a middle ground and provide support services for users.
“You can't let [pharmaceutical companies] control access to the educational activity. That's when the nuances of all this Web stuff get really dicey.”
— Jann Balmer, PhD
Find a physician who will champion the project. At the University of Virginia, Balmer considers herself extremely fortunate in having two cardiologists who are also expert at Web site development participating in CardioVillage. “We're lucky. We've got two cardiologists who know how to write code,” she says. Larry Gimple, MD, and Ralph Buckley, MD, both of whom teach cardiology at the University of Virginia, are admittedly extraordinary examples. They own their own company for designing online medical education, called CardioConcepts. Buckley has even written proprietary software for creating medical education Web pages — while continuing his work as an assistant professor of internal medicine in the university's division of cardiovascular medicine. How do you find champions like these? “These are the guys who teach themselves HTML at home and put their cases up on the Web, then come ask you whether they can get credit for it,” says Balmer.
For the American College of Cardiology's (ACC) efforts in online CME, the champion was Kim Eagle, MD, chief of the division of cardiology in the University of Michigan (Ann Arbor) Health System. “One of the first things we learned was that when you're trying something new, it's good to try it with someone who is respected in your organization and is also eager to try something new and take a risk,” said Marcia Jackson, PhD, ACC's senior associate executive vice president, at the Alliance meeting. Eagle is known as an advocate for establishing myocardial infarction guidelines on the national level.
The most important quality a physician-champion can possess, says Balmer, is that he or she make daily use of the software. Buckley and Gimple are practicing cardiologists, so they are always looking for new features for the site that will help them in their practice.
Prepare for an unending commitment, and organize accordingly. “You have to look at ways to streamline the process by creating an organizational structure and planning system that will accommodate itself to the nuances of an activity that, basically, never ends,” says Balmer. She warns that without proper guidelines and protocols, the round-the-clock nature of the Internet will quickly lead to staff burnout. In fact, presenting online CME requires a different mind-set than that which makes a successful meeting organizer. Being project-oriented is necessary, but not sufficient, according to Mark Wieting, vice president of educational programs for the American Academy of Orthopaedic Surgeons (AAOS), based in Rosemont, Ill. “The tendency is to concentrate on getting the thing done, then look forward to the next [project],” he said, while sharing the lectern at the Alliance for CME with the ACP — ASIM's Kanya. “It's hard to get back to the ones you've already done. You have to commit to maintenance.”
“[Tech companies] know CME is for physicians, but they want to please the pharmaceutical company because that's where the money came from…. Things can get out of hand very quickly; you have to be really assertive.”
— Jann Balmer, PhD
Stick to the Standards. “I prefer to have the CME office more involved rather than less involved,” says Balmer. “If you're not at the table to hear [partner organization] conversations, you can't cut them off at the pass. Especially with the Web-based stuff, the half-life of decision-making is incredibly short. Things can get out of hand very quickly; you have to be really assertive.”
She adds that many of the private firms that want to partner with UVA, and whose expertise is technological, can't make the distinction between physician and pharmaceutical firm in deciding who the customer is.
“They know CME is for physicians, but they want to please the pharmaceutical company because that's where the money came from.” Having just ended a stint as chair of the Accreditation Review Committee of the Accreditation Council for CME, Balmer is possibly better-informed than most; nonetheless, she says she's never needed anything beyond the current Standards for guidance.
“There are touchstones that you look for to validate any CME activity, whether it's live or an enduring material or on the Web ,” she says. For a Web-specific example, she cites pharmaceutical firms that want to link directly to education on the UVA site. She prohibits such practices, because she knows that giving a firm control over the context in which information is presented is just as bad as giving them control over content.
“You can't let them control access to the educational activity,” she says. “That's when the nuances of all this Web stuff get really dicey.”
Despite her reservations about working with private-sector companies, she thinks public/private partnerships have a lot of potential. “We're working [on a project separate from CardioVillage] with mypatient.com [a partnership between UVA and Montreal-based I.C. Axon, a medical Web site developer]. There have been some bumps in the road, because they're a venture-capital — driven company and don't always understand how CME works, but it's a really cool product.”
Expect intellectual property problems and disclosure dilemmas. “Suppose amember from a medical school makes a presentation at our program,” says the ACC's Jackson. “They've used their time to make the intellectual property which is their presentation. Who owns that presentation? Do we own it? Does the presenter own it? Does the presenter's school own it? As we put more content from our annual scientific session on the Web [and grant Category 1 credit for it], this becomes a major issue for us.”
She foresees no easy answers to these questions. Jackson adds that the latest wrinkle in disclosure is the situation in which ACC members donate their time and services to meetings where content is presented on the Web, but are also acting in advisory capacities or as contributors to dot-com companies that want to partner with the ACC to present CME on the Web. “We have fears that we may lose some of the intellectual property produced at our meetings — property that we enjoy because of the professionalism of our experts — to these companies.” She adds that, to date, presenters still consider it an honor, not a fee-generating task, to participate in ACC activities.
Tackle the technology. There are many ways to construct Web sites, and many vendors offering a plethora of solutions. For example, the latest in Web site-building technology is the dynamically created Web page, which draws text and graphics from a central database to create pages on the fly. The ACC is in the middle of building such a database for its online education. This requires tagging and indexing bits of information so they can be retrieved as needed. Jackson is finding that she needs to learn about tagging and indexing standards, and that her vocabulary is increasing as a result.
“‘Granularity of chunking’ is a phrase I've learned in the last year, and it is an important issue,” she says. “You want small enough units of information that you can mix and match, but large enough so each one maintains cohesiveness in its own right.”
This example is offered by way of saying that technology decisions can be delegated up to a point, but only to a point. Database software vendors are not education experts, much less CME experts. That role belongs to the CME provider.
Part 2 Execution Essentials
Presenting CME on the Web is a lot like presenting a live conference: The devil is in the details. Here are five points about practices to follow and pitfalls to avoid in the actual execution of an online CME course.
Make it familiar. For its first online CME course, back in 1997, the American Academy of Orthopaedic Surgeons offered a course on managing malpractice risk. It was prepared by members of the AAOS committee on professional liability; they were highly qualified experts in the field.
“We have fears that we may lose some of the intellectual property produced at our meetings…to these [dot-com] companies.”
— Marcia Jackson, PhD
“Each member wrote a chapter,” recalled the AAOS's Wieting. “The learner had the option of either reading it from the screen or listening to the presenters read it. It took about two hours to complete. Not many people started it, and almost everybody who started it stopped. We learned that what we had in mind for the Web was not much of a learning experience.”
Since then, AAOS has launched “orthopedic flash cards,” where x-rays that can be clicked on to enlarge are presented, and learners make a diagnosis, then find out what the experts had to say.
More recently, AAOS discovered the power of metaphor: The main Web page is now presented to the viewer as a “campus,” with a student union, an auditorium, a home room, and other symbols of learning that are familiar to physicians.
The meta-aphor for CardioVillage is a meeting. “CardioVillage has an exhibit hall,” says Balmer. We have it because we want to blend the familiar with the new. If physicians see things that are familiar — they can say to themselves that they know what an exhibit hall is — then whether it's real or virtual becomes a non-issue. What's important is that they have a conceptual framework for dealing with it.”
Similarly, at mypatient.com, the UVA's joint project with Web site deverloper I.C. Axon, the central metaphor is a doctor's waiting room, Balmer says.
“The idea is to not make it so foreign that they think they can't do it,” she says. Neither of these sites features “talking-head” lecturers.
On the other hand, the American College of Cardiology has had some success with repackaging live presentations from its annual meeting and presenting them on the Web, although the actual translation from straight video to streaming video is handled by an outside publisher.
Make it friendly — for faculty, too. A recent online course offered by the AAOS allowed participants a lot of leeway in the way they structured their coursework. They could look at expert opinions, read cases, conduct literature searches on Medline, and participate in online discussions. They could choose from reading, listening to, or reading and listening to lecturers. Bios and photos of lecturers were included.
There was just one problem — the course took four hours to complete, but there was no feature that would let physicians save their work and then retrieve it at the spot where they last left off working. Seemingly a minor detail — but only a third of the physicians who started the course actually finished it, and this despite e-mail reminders to those who started the course and then abandoned it.
Another aspect of “user-friendliness” is that the education design reflect the informational desires of the physicians involved. This is where having physician-champions comes in handy, according to Balmer. “They know what physicians want to know.”
For example, when you look at an EKG image on the CardioVillage Web site, a click of the mouse will blow it up to full-screen size. Buckley and Gimple knew that other cardiologists would want to be able to look closely at specific leads on the EKG, so they designed the course presentation to make that possible.
“User-friendliness” extends to faculty, too, as ACP-ASIM's Kanya discovered. “We created an Internet-based authoring tool to help authors write and submit their cases,” she said. “We thought it would make getting presentations to the Web easier, faster, and more convenient than the traditional approach, which for us meant many face-to-face meetings with authors, which was expensive.”
A planning meeting with 10 physician-authors was held, and, as Kanya said, “Everyone was excited and motivated to become online CME pioneers.” In fact, the first set of cases arrived ahead of deadline. But that was the end of the good news. The authoring tool turned out to be harder to use than expected, nor was it as functional as expected. It nearly upended the project, with two otherwise enthusiastic physician-authors resigning out of frustration. It was quickly replaced with a Microsoft Word template, which while not as great for the Web page producers, was far easier for the authors to use. (It wasn't all bad — Kanya said the authoring tool is still being used — successfully — at the production end.)
Don't skip the sign-in. It's good to be user-friendly, but this does not extend to allowing visitors to gain access to your online education without first identifying them-selves. “We didn't require [registration] at first,” says Balmer. “Buckley and Gimple were afraid the doctors wouldn't want to — except then we couldn't track usage the way wanted to. You must be able to track users through multiple uses of your site. You may have multiple funders, which means coming back several times to measure return on investment. If you can't track user hits and membership services, you'll be at a disadvantage.”
Testing, testing… “When we add a new feature, we analyze it to make sure it's right before it goes live,” says Balmer. “We test on a production server and when it works the way we want it to, and the educational content is what we want it to be, then we'll move it over to the live server.” In fact, she points out, the very first CardioVillage program went live (in November 2000) only after two and a half years of development.
Tweak forever. As discussed earlier, putting CME on the Web successfully means making a strong commitment to ongoing improvement. This means more than adding new content — it means improving the way you present the content you already have. The CardioVillage site is “tweaked every month,” says Balmer. “We reformat all the time.” Their efforts pay off — 5,431 visitors participated in learning sessions from January through April in 2001, up from 3008 during the same period the year before.
Part 3 Back to the Future
“The Internet is a world with a ton of potential, but to realize it, you've got to get people out of their routine habits of doing business — to take a step back and look from a different angle,” says Balmer.
At the ACC, the CME office is definitely taking a step back and looking from a different angle. Jackson says her organization's membership is interested in what amounts to just-in-time CME.
“They want it delivered where they are, when they want it. They are essentially nomadic — they don't sit around in offices all day. We think the Palm Pilot could be an excellent delivery system for CME.”
On the other hand, CME providers are also learning that sometimes the preferred medium of information is still meetings or print. Theresa Kanya pointed out that when young residents were surveyed about a new education product, the majority wanted it in print.
“We were planning it as a CD-ROM product only,” she said. “We changed course and offered it in print, too, and that's the way they're buying it.”
Finally, what about those market research studies mentioned at the beginning of this article that indicate that physicians are more interested in online CME than they are in, say, online patient record management? If that's so, then CME providers may have more clout with online partners than they thought. Big medical portals that are trying to lure physicians to partake of a host of services may discover that it's the CME that is the main show, not the other services.
Winning Web Sites
Here are the CME Web sites discussed in this story:
American Academy of Orthopaedic Surgeons www3.aaos.org/courses/campus.cfm
American College of Physicians-American Society of Internal Medicine www.acponline.org/
American College of Cardiology www.acc.org/education/education.htm