Models that Work: The Nuts and Bolts of Faculty Development

Distance-Based Learning for General Internal Medicine, Family Medicine, and Pediatrics

Introduction

Recent research suggests that the commitments of family medicine faculty have changed over the years as have the future needs of these faculty members (Bland & Simpson, 1997). However, there is little corresponding research or generalizable evaluation to guide the development of programs to prepare faculty for these changes. Over the years, money has been given to support faculty development training programs, but little has been done to support research or generalizable evaluation on faculty development. Although internal program evaluations are regularly conducted, these evaluations do not provide information that generalizes beyond the program and the specific questions of the program. Furthermore, many of the evaluations of faculty development have lacked the rigor necessary to measure outcomes (Reid, Stritter & Arndt, 1977) and those involved with the programs have not been committed to publishing results (Bland & Simpson, 1997).

While there is some information about prior investments in faculty development, there is much to gain from conducting research or generalizable evaluation to identify which components, competencies, and strategies of faculty development are effective for which situations given the ever growing demands of our faculty development programs (Bland & Simpson, 1997). It has been argued (Irby & Hekelman, 1997) that the first step to comprehensively measuring the success of faculty development programs is to evaluate the situations in which they are implemented (i.e., to perform needs assessments) and then to assess the strategies used and the outcomes achieved. In doing so, it becomes possible to link needs to outcomes, thereby providing a way to identify the conditions for which certain programs are effective. This information would allow future planners to strike the best match between the focus of a faculty development program and the goals and responsibilities of departments, programs, and faculty (Bland & Simpson, 1997).

In December of 1998, the Models that Work conference presented successful local, regional, and national programs that have approached faculty development in a variety of ways. The intent of the conference was to help participants choose the model of faculty development most appropriate for their institution and gain skills to begin implementation of faculty development programs at their institutions.

The goal of the Distance Based Learning project is to increase the effectiveness of these faculty development efforts that the conference participants have undertaken at their institutions. The comprehensive approach I'm about to describe is intended to give participants the skills necessary to contribute to the broader understanding of faculty development. It is our hope that participating sites will produce program evaluations, of publishable quality, of the effectiveness of specific faculty development models in a range of institutional environments.

Team/ Pilot Site Selection

To gain maximal benefit from the Conference, participants were encouraged to attend the Conference as a team of two or more faculty. All sites were encouraged to apply to be a "pilot team" which meant they would receive consultation from national experts on needs assessment, post conference planning, and program evaluation. Participants interested in being a pilot team were asked to provide a brief description of their faculty development needs; the commitment of their institution to implement any needs assessment, faculty development, and/or evaluation designs, and the capacity of their institution to implement any needs assessment, faculty development, and/or evaluation designs. Pilot teams were selected before the teleconference based on these features and on the desire for the pilot groups to represent a range of institution types (e.g., community-oriented, consortium, research-oriented, etc.) and departmental interests (e.g., family medicine, general internal medicine, and pediatrics). Of the 17 institutions selected as pilot sites, 11 family medicine, 9 internal medicine, and 9 pediatrics departments are represented.

Initially six pilot sites were to be selected. Because we had an overwhelming response from institutions wishing to be selected as "pilot sites" for the Distance Based Learning project (we received 34 applications) and all applications were quite impressive, the Selection Committee suggested that we submit a request for additional funds to provide dedicated technical assistance to more than six sites. The Committee recommended selection of 17 pilot sites based on the superb scores these institutions received from raters reviewing the application materials. Expansion of the pilot group in this way allowed us to acknowledge the superior ratings of a larger portion of applicants and to include in the pilot "mix" a more diverse range of academic institutions and a broader representation of geographic locales.

The selected sites are:

• University of Hawaii, Honolulu, HI
• Gulf Coast Consortium for Community-Based Education, Galveston, TX
• Michigan State University College of Osteopathic Medicine, East Lansing, MI
• University of Utah Health Sciences Center, Salt Lake City, UT
• University of Iowa College of Medicine, Iowa City, IA
• University of Kansas School of Medicine, Wichita, KS
• University of Minnesota, Minneapolis, MN
• Children's Hospital Medical Center, Cincinnati, OH
• Georgetown University School of Medicine, Dept. of Family Medicine, Washington, DC
• Tulane University School of Medicine, New Orleans, LA
• Mayo Clinic, Kasson, MN
• University of Puerto Rico, San Juan, PR
• University of South Carolina School of Medicine, Columbia, SC
• University of South Dakota School of Medicine, Sioux Falls, SD
• University of Ottawa, Ontario, Canada*
• George Washington University School of Medicine, Washington, DC
• University of Pennsylvania, Philadelphia, PA

*Technical assistance for any site outside the United States cannot be supported by the federal funds directing this project. However, since ratings by the pilot site selection committee for the University of Ottawa were commendable, this team will receive technical assistance as part of the outreach efforts supported by the University of Minnesota.

Technical Assistance: Pre-conference

Prior to the Conference, all participants were encouraged to conduct an institutional assessment of their faculty development needs in order to helps them identify what issues to address and to best use the conference to plan their faculty development program (i.e., help them identify the most appropriate faculty development strategy for their situation).

To assist these efforts, a pre-conference video teleconference, "Performing A Needs Assessment," was broadcast on October 22, 1998. The teleconference featured national program evaluation experts who provided information on conducting an institutional needs assessment and responded to specific audience questions on needs assessment. Videotapes were made for and distributed to pilot sites and other conference registrants as well as conference faculty, presenters, and committee members. These videotapes were distributed prior to the Models that Work conference in December. Additional copies have been distributed upon request. To date, a total of 142 tapes have been sent. A limited number of videotape copies still remain. Please contact my office if you are interested.

In the six weeks following the teleconference, the Project Director, Project Evaluator, and Evaluation consultant reviewed needs assessment outlines and progress updates from any pilot site sending such information. Arrangements were also made during this time for the pilot sites to meet each other and the evaluation team at a special lunch at the conference and to have site participants talk informally with members of the conference faculty at one of the evening receptions. In preparation for the special lunch, packets were prepared for the pilot sites describing each of the teams and their faculty development plans. In addition, research was done on peer coaching so that sites could be encouraged to not only access the technical assistance team with their questions and concerns but to network with the wider group of pilot site participants in order to share their successes, lessons, and information learned.

Technical Assistance: Conference

The material presented by the evaluators at the conference introduced participants to evaluating the effectiveness of their faculty development programs. It is hoped that such targeted instruction, followed by one year of evaluation assistance, will enable individuals from various institutions, with various faculty needs, to identify which faculty development approaches work best when and to produce research on faculty development that can guide the development of programs to best meet both the roles of faculty participants and their departments' goals.

In preparation for the Models that Work conference in December, the Project Director and Evaluation Consultant reviewed background research, respectively, on effective approaches to faculty development and evaluating faculty development programs and publishing the results. They also prepared invited presentations (as well as presentation handouts and slides) on these topics. These presentations opened and closed the Models that Work conference.

As noted above, the Project Evaluator planned a special lunch for the pilot site participants at the conference. An effort was made to match sites for the luncheon based on project characteristics and targets of faculty development efforts. The lunch was well-received and gave participants a chance to network with each other and with the technical assistance team. The Project Evaluator also developed a pilot site resource packet with brief descriptions of each planned faculty development project and contact information for each site. The packet also detailed the kinds of assistance that might be provided, helpful references on faculty development, and information about the technical assistance team. Participants were encouraged to use the network of resources within the pilot site group, as well as the technical team contacts, after the conference. In addition, the Project Evaluator made arrangements for participants from each site to have the opportunity to talk informally about their faculty development plans, ideas, and issues with conference faculty at one of the evening receptions. She also provided information to the sites prior to the conference about the consultations they could arrange with conference faculty on the final afternoon of the conference.

Technical Assistance: Post Conference

As I said earlier, material was presented at the conference on evaluating the effectiveness of the faculty development programs. Presently, we are providing long-distance technical assistance to all participants for one year (with dedicated time to the pilot sites), especially with regard to the evaluation of the program implemented at each institution. An emphasis is being placed on conducting meaningful evaluations and publishing the evaluation results. Again, publishing the evaluation results will help future faculty development efforts to identify which faculty development approaches work best to facilitate success in various faculty roles and for various types of achievement.

We have had requests from many of the pilot participants. The assistance requested has been varied but has included, for example: helping sites think through faculty development questions; reviewing needs assessment outlines and progress updates; reviewing a manuscript on faculty development; providing advice on an instructional book and workshop being built to facilitate writing skills; providing advice on an evaluation design to assess a faculty development program; reviewing a school wide faculty development strategy; reviewing and commenting on survey instruments to be used to assess needs and to structure faculty development programs; and providing leads on things such as where to find information on a particular faculty development approach or sources of funding for faculty development programs.

Plans for Follow-Up Evaluation of the Pilot Sites

To assess whether or not the objectives of this project have been accomplished, the pilot teams will be asked to complete a pilot site evaluation and offer their recommendations and degree of satisfaction with the process of distance based learning used to improve the design and evaluation of their faculty development programs.

A comparison will also be conducted of the manuscripts in preparation, manuscripts submitted, publications, and presentations completed by conference participants and personnel on a sample of faculty development and faculty development evaluation projects funded by HRSA for the period December 4, 1998 to December 31, 1999. The process for selecting comparison projects has yet to be determined. It will most likely involve reviewing a list provided by HRSA of such projects and contacting principal investigators individually regarding publications, etc. Projects may also be selected following a review of conference proceedings or agendas from professional meetings or a search of journals publishing faculty development abstracts and articles during the aforesaid time period.

In addition, we are tracking the types of consultation we provide to each site and we will have quarterly updates from each site on the progress of their projects. While no data is presently available to indicate the effectiveness of our intervention on improving the likelihood that faculty development programs will be successful, the reports will enable us to provide specific, intermediate evaluation soon and outcomes assessment at the end of the project.