THE SURGICAL TEAM has scrubbed up and donned their gowns; press 1 on your keypad if you're ready to operate. A script during the annual Transcatheter Cardiovascular Therapeutics meeting in Washington, D.C., might read just that way. Like many other medical meetings, TCT broadcasts live surgery via satellite as part of its educational programming. What's unusual is that all the attendees — about 15,000 cardiologists — can participate remotely in the cardiovascular procedure using an audience response keypad.

“At certain points during the case presentations, the doctors ask the audience what they would do next. They vote live, real-time while the procedure is taking place,” says Anthony Cosimano, director of sales and marketing at RSi Communications Group, Hackensack, N.J., the company behind TCT's audience response system. “And at times it changes the way these doctors do the procedures,” he says.

That thought might make any patient a little uneasy, but the application certainly earns TCT a place among the most innovative users of keypad technology. While the most basic uses of response systems — garnering audience opinions and testing their knowledge — are welcome adjuncts to a meeting, used creatively, ARS can also be a consensus-building and brainstorming tool, an evaluation system, and even a teambuilding tool.

Beyond Demographic Data

But many observers agree that planners and speakers aren't taking advantage of the systems' capabilities. The potential of ARS is often sorely under-explored, with presenters simply gathering demographic data and asking basic questions related to the meeting session. “There are a lot of people who put it into a program because it's a novelty or their client wants it. But they give no guidance to their presenters about what they want to or can accomplish with the system,” Cosimano says. “A medical society might make the keypad system available to 20 or 30 speakers as part of a major symposium, but left on their own, only three or four of those speakers are inventive and will come up with something interesting to do [with the keypads]. Not enough guidance is given to people about how they can go beyond the novelty stage, documenting and reinforcing the learning that's taking place, using the audience's responses as a tool to help make a point, gathering data that can have real meaning for you or your client going forward.”

Speaker Feedback

Marc Dickstein, MD, is one medical educator who definitely gets it. Dickstein was introduced to ARS at a meeting of the Society for Cardiovascular Anesthesiologists a few years ago and brought his enthusiasm back to Columbia University College of Physicians and Surgeons, where he is associate professor of clinical anesthesiology in the Division of Cardiothoracic Anesthesia, and co-director, Science Basic to the Practice of Medicine and Dentistry. Working with Turning Technologies, Youngstown, Ohio, the school purchased 250 keypads and a base unit with a grant from the Gladys Brooks Foundation and began using them with students at the beginning of the 2001 school year.

“My initial interest in this was to increase active learning and decrease passive learning [in large lecture classes], but I found it to be useful for many other reasons,” says Dickstein. “It gave incredible feedback to lecturers about what students understood. Some-times it was kind of startling for them to see what points did or did not get through, and it changed the way they taught. The other advantage is that the students got a better sense of whether they were keeping up with the material, so they could pace themselves and not fall behind. If they were keeping up, they could reduce their stress level.”

Dickstein has experimented with the system to get the most out of it. “The most elementary application is to give students pre-tests and post-tests to see if they got the key points [of a lecture],” he says. “We also couple it with demonstrations and have the students predict what's going to happen. We can have a video link from an exercise lab and ask them to try to guess what's going to happen next or what the response will be. They can guess themselves, see what the rest of the class predicts, and observe what actually happens, and they get really invested in the demonstration.

“The other use of it — and this is really interesting — is when I pose a question and there's a pretty good distribution [of answers]. When most of the class does not know the right answer, I may give the class a hint, or I may have them talk to each other and try to convince each other of the answer, and then I'll re-poll and see if there's a change.”

Dickstein says he's looking forward to the 2003 version of Turning Technologies software, which will allow users to see how many people changed their answer in the second polling. “If answers originally were split 50-50, and I re-poll and get the same split, [with the current software] I don't know if everyone stuck to their own camp,” he says. “I also use the keypads before a large exam to go over questions from a prior year's exam, and then I review the questions based on the feedback.

“My use of this technology is in its infancy,” says Dickstein, who has had so much positive feedback and so much interest from other lecturers that the university is now getting another base unit and plans to hire someone to train faculty and run the system.

Key In Creativity

The success of a keypad program depends on the creativity of the meeting organizer and the presenters, but they don't have to go it alone. Here are a half dozen examples of the applications, for corporate and association meetings, that go beyond the boring.

  • Strategic Planning

    RSi suggests a two-tier questioning approach. First, ask how important a particular trait is to the success of your product or company, rating it on a scale from one to five. The follow-up question would be: How well do we accomplish this trait at our company? “Do this a number of times, and you come up with an x/y quadrant map that shows what is important to the organization and how well the audience believes they're doing it,” says Cosimano.

  • Brainstorming

    Like the strategic planning approach, meeting leaders can ask the audience about a series of ideas. First asking, “How good is the idea?” and following up with “How achievable is the idea?” “This is a good strategy to focus people and stimulate productive conversation in a meeting,” says Cosimano.

  • Anonymity

    A lot of time can be wasted at meetings when votes need to be tallied. Electing officers and voting on motions can be handled quickly with ARS. And focus groups are another natural fit, especially when the audience might be more honest if they can share their thoughts anonymously.

  • Feedback

    If you have trouble getting meeting evaluations returned at the end of a session, ask the questions via the ARS system. Answers do not have to be displayed to the audience.

  • Icebreaker

    “Jeopardy”-style ARS games can be used to mix education with some fun (the audience's answers can be sorted by region or division to promote friendly competition). Trace Kuhn, formerly with Extreme Meetings!, an audience response consulting firm in Tampa, Fla., says his company organized a twist on the idea for a sales operations group of a medical supply company. Before the meeting, they had attendees send in fun facts about themselves, and they then used those facts as the “Jeopardy” answers.

  • Sales Training

    RSi worked with Johnson & Johnson on a sales training program in which more than 800 people moved through five different sessions, one of which was a pre- and post-training product-knowledge exam. The keypads were coded with each participant's name, region, and sales specialty so the data could be examined in various ways.

“At the end of the day, we provided an Excel spreadsheet that specifically told them how each attendee had scored and showed which areas they were weakest in,” Cosimano says. “We provided that data immediately so they could use it the next day.”

Incorporating audience response systems into a meeting is now as easy and routine as ordering any other audiovisual component. But unlike a video projector that will play whatever you put into it, these systems take some planning and organization. Given that effort, however, the tool is unlike any other for assessing the feelings, knowledge, and opinions of large groups.

Assessing PhRMA Code Compliance

Audience response keypads are no longer too costly for small meetings. “We have made some breakthroughs and are now doing dinner meetings for 20 to 30 people,” says Anthony Cosimano, director of sales and marketing, RSi Communications Group, Hackensack, N.J. “We have made it [the software] much less cumbersome so that a person who is not a keypad technician can successfully run a meeting. Trained technicians do not have to travel to an event. Basically, if a person can run a computer and set up a video projector, they have sufficient skill level to run a keypad meeting.”

While creating a more interesting experience for the dinner meeting attendee is one benefit of incorporating ARS technology, the bigger draw may be the ability to document educational content, says Cosimano. “In the medical field, under the new guidelines [the PhRMA code] with respect to educational programs, many of our corporate clients are concerned about what is going on in the meetings that they're underwriting,” he says. “The keypads give the product team or corporate management a continuing stream of information about how their message is getting across and how it needs to be tweaked, and it also allows a brand manager to assure the legal department that fair, balanced information is being covered in these meetings.”

Typically, says Cosimano, “the firm that gave a grant got a subjective report back from the CME provider: who showed up, what they talked about. We can supplement that subjective report by documenting what people learned.”

Reports from the Field

Here are a few examples of how CME providers are using audience response systems.

“Audience response systems have affected the way our speakers approach their audience. For example, at a workshop that we anticipated would attract primarily surgeons with substantial experience in a certain procedure, the ARS results showed that the majority of respondents performed this procedure fewer than five times per year, allowing the lecturer to adapt his talk for a less experienced audience. We have also used the system for evaluation purposes and needs assessment.”

Caryl Herrington Worthington

Director, CME and Fellowship Review
American Academy of Facial Plastic and Reconstructive Surgery
Alexandria, Va.

“We have used the audience response system for a single speaker and multiple speakers. For both formats, we engage the audience to ‘vote’ on multiple-choice answers as to their recommendation for treatment, best of …, etc. We have not modified the speaker evaluation form to include questions such as: Does the audience response format add to your enjoyment of the presentation? Does the ARS provide added benefits to the educational format? [However], we did ask these types of questions when originally testing the equipment in order to survey our medical staff. For doctors, the perception of benefit was 50-50; other medical classifications (nurses, technicians, etc.) seemed to enjoy the experience more. That may be because the other medical professionals attending CME activities feel too intimidated to provide input or ask questions in front of MDs.”

Leslie Ann Cyril

Oncology Education Coordinator
Hoag Cancer Center
Newport Beach, Calif.