Reports and References - COPE Outcomes and Assessment Committee Draft Report

For April 9, 2003 meeting of COPE

[download Word file]

THE BASIC QUESTIONS
- In what ways is the University relevant to the public, in our ongoing teaching and research?
- How can we document and measure that relevance?
- How can we use those measures to track improvements in public engagement?

OUR CHARGE IS TO DEVELOP
- specific short term and long term expected outcomes of COPE activities
- ways that University of Minnesota would be different in five years as a fully engaged university
- measures of public engagement as an indicator of institutional performance

DOHERTY CLASSIFICATION
Prof. Bill Doherty has suggested a useful classification of three categories of civic engagement, as they relate to faculty work.

1. The ability of faculty to articulate the public value of their work
2. "Outreach", in which we bring our expertise to people in ways that they see as relevant (the traditional notion of outreach)
3. Collaborative working relationship with community partners to define problems and work together to solve them. This often necessitates bringing a comprehensive and integrated research, teaching, and service scholarship portfolio to the situation. The public should be fully and equally engaged in this process.

KELLOGG COMMISSION SEVEN-PART TEST
The Kellogg Commission's seven-part test for an engaged university addresses issues that seem mainly directed at part 3 of the Doherty typology, but which have some applicability to all three types of public engagement.

Responsiveness: are we listening to the publics we serve and responding to their needs? Do we provide space and resources for community-university discussion of the public problems that need to be addressed?

• Respect for partners: Do we encourage joint definition of problems, solutions, and success? Do we respect the capabilities of our public partners?
• Academic neutrality: Do we maintain our role as neutral facilitator and source of factual information when public policy issues are discussed?
• Accessibility: Have we made access to our expertise and resources as simple and direct as possible? Is our expertise equally accessible to all constituencies?
• Integration: Do we bring multiple disciplines together to work toward solutions to interdisciplinary problems? Is there incentive for faculty and students to be involved in engagement activity?
• Coordination: Do we know what each other is doing?
• Resource partnerships: Do we commit adequate resources to engagement activity? Do we tap into various funding sources to get our work done? Do we help our partners get necessary funds?

PROPOSALS FOR OUTCOME AND ASSESSMENT MEASURES
With these general principles in mind, the Outcomes and Assessment Committee proposes three sets of outcomes and assessment measures, one for each of the three Doherty categories. Each of these sets of measures should assess involvement by learning and discovery by faculty, students, and community partners. Without involvement of all three groups in at least some of our efforts, public engagement in a university context is incomplete.

Type 1: Regular faculty teaching and research
Five-year vision: In five years, it will be widely recognized both inside and outside the University that public engagement, in its various manifestations, is an integral part of our teaching and scholarship, rather than the third of a three-part mission (teaching, research, outreach). Not everyone will be engaged equally or in the same way, but all will be cognizant of the ways in which their teaching and research serve public purposes.

Some desirable outcomes:

• All departments, colleges, and campuses will include public engagement in their mission statements. Effective public engagement that enriches teaching and scholarship will be be recognized, fostered, communicated, and rewarded.
• Faculty governance will be supportive of increased recognition of public engagement as properly integrated faculty work.
• Faculty efforts in public engagement, even when indirect, will be noted in annual reports and considered in merit reviews. The guidelines for reporting on and evaluating faculty research will include a category called "Impact," of which publication in refereed journals is one instance, and documented impact on community empowerment, public policy development, etc. are others.
• Discussion of the public implications of scholarly work will be common in departmental conversations, and an integral part of the acculturation of new faculty and graduate students.
• Invidious distinctions between basic and applied research will dissolve in recognition of a continuum of challenging problems and important solutions, as appropriate to each unit.
• The U will be recognized as taking a long-term, objective position on important but contentious issues. It will play a visible role in facilitating dialog, as a convener of groups with different interests.
• Our positive interactions with under-represented communities will result in the admission of more minority students into our undergraduate and graduate programs.
• The recognition that real social problems can rarely be solved along strict academic disciplinary lines will lead to the development of new interdisciplinary courses and research projects. Many of these might bring together scientific and humanistic perspectives.
• University policies and budgetary mechanisms will be seen as supportive of enhanced interdisciplinary activities.
• University Relations, in its role of communicating between the public and the academic units, will be more effective in telling the story of how fundamental research and scholarship serve important public purposes.

Some possible practical/quantifiable measures:
- teaching (classroom hours and credits, new interdisciplinary courses, service learning, ...)
- students (classes, dissertations, ...)
- research and scholarship (papers, books, reports, interdisciplinary projects, ...)
- funding (grants, gifts, contracts, ...)
- publicity (newspaper and magazine stories, TV and radio items, ...)
- contacts with publics (projects, lectures and performances, ...)
- cooperative interactions with other universities and colleges, ...
- increased recruitment of students from under-represented populations

Type 2: Traditional outreach and consulting
Five-year vision: In five years faculty and staff in all parts of the University will haveÄîto the extent they desireÄîinteractions with appropriate publics, through extension and consulting activities, that enrich their teaching and scholarship while providing benefits to constituents.

Some desirable outcomes:

• An updated vision for Extension will have been achieved, with a sustainable division of labor and financial support between university and MES assignments.
• The role of the College of Continuing Education in forwarding the public engagement agenda will be clarified.
• Consultation with the private sector and with state and local governmental entities will be recognized as an important part of civic engagement.

Some possible practical/quantifiable measures (need to be refined by consultation with Minnesota Extension Service)
- Constituents served
- Economic and social impact of services provided

Type 3: Community partnerships
Five-year vision: In five years, there will be numerous long-term community-university research partnerships in which the two are equal contributors, utilizing the strengths and insights of both partners.

Some desirable outcomes:

• There will be numerous formal and informal conversations between university and community members, including external advisory groups, leading to better understanding of the needs and priorities of both sides.
• Through web sites, newsletters, meetings, call-ins, and community-based staff, the resources of the University will be made better known and more accessible to the public.
• Faculty and students will learn the difficult but essential lessons of how to work patiently, respectfully and successfully with community partners.
• The strengths that communities bring to these partnerships will be broadly recognized.
• When possible, community members will be made part of research teams, and will be appropriately trained and compensated.
• Community outreach projects will be viewed as opportunities to recruit students from under-represented groups to the university.
• Service learning opportunities will be expandedÄîin ways that are not burdensome to community agenciesÄîby careful planning, training, and consultation; and will be as thoughtfully evaluated as other educational activities.
• Faculty peers will take into account, during merit review, the special challenges posed by properly collaborative community-based research.
• The complexities of real-world research will break down disciplinary barriers, producing new, rich interdisciplinary research and teaching opportunities. New sources of funding will be found to underwrite these ventures.
• There will be improved coordination among our various departments, centers, institutes, and other offices to facilitate interdisciplinary work with community partners.
• We will have better coordination with regional partners.

Some possible practical/quantifiable measures:

• Number of community-based projects, and number of people involved and served
• Number of community people who gain or enhance useful skills as a result of such projects
• Amount of new funding generated for these projects
• Number of under-represented minority students recruited to the University
• Number of students involved in service learning projects, and number of clients served thereby
• Number of new courses that are based on interdisciplinary, community-related themes, and the number of students taking these courses
• Number of papers and books published by faculty engaged in community-based research, and external funding generated thereby\] Successful promotion and tenure cases of faculty engaged in community-based research

A "SIMPLE" ALTERNATIVE: THE MICHIGAN STATE UNIVERSITY APPROACH OF MEASURING FACULTY EFFORT

Excerpted from the paper "Measuring Scholarly Outreach at Michigan State University - Definition, Challenges, Tools" by Robert L. Church, Diane L. Zimmerman, Burton A.
Bargerstock, Patricia A. Kenney. In Press, Journal of Higher Education Outreach and
Engagement, November 15, 2003, Revised January 23, 2003.

The Outreach Measurement Instrument

The Michigan State University Outreach Measurement Instrument (OMI) is divided into three sections. The first two sections require mostly quantitative data; the third section encourages respondents to describe in some detail their overall outreach work or some aspect of it.

The first section of the survey comprises a single question  total percentage of time spent on outreach work during the reporting year. In order to capture the sense that outreach is generally integrated with faculty research, teaching, and service, we acknowledge that this is a duplicative measure. The survey instructs respondents to count all work that has an outreach component, including that portion of your teaching, research, and service that is conducted for the direct and immediate benefit of audiences external to the academy. This same question is asked of faculty when they complete their annual effort reports. After they have divided 100% of their time among teaching, advising, new course development, research, consulting, university service, and general service, they are asked to estimate what part of that 100% effort included an outreach component. As faculty time is the universitys most valuable resource, determining how much of that time involves faculty in engagement with the external community is, we believe, the best single measure of institutional commitment to outreach.

The second section asks respondents to characterize the outreach activities or projects on which they spent the most time. The responses allow us to categorize outreach activity along several dimensions and thus contribute to better assessment and management of outreach as a campus-wide activity. The dimensions include:
- area of concern
- type of activity
- partners or groups involved
- location
- revenues

The primary dimension is area of concern. Faculty are asked to choose one or two areas from a 12-item menu that includes such areas as community and economic development, public safety and corrections, or preK-12 education. Although some facultys outreach activities touch on many different social issues, we ask them to limit their selections to no more than two of the areas of concern. Most, we expect, will choose the two on which they spend the most time, but they can report on any aspect of their activities even though it was not the most time consuming. We then ask what percentage of their outreach effort they devoted to each area of concern. Examination of the distribution of faculty time across the areas of concern allows us to judge whether individual faculty outreach efforts aggregate into an appropriate spread over the areas of concern  to insure that efforts aimed at strengthening preK-12 education are not leading to relative neglect of economic development, for example.

The remainder of the second section probes further into their work in the one or two identified areas. For each chosen area of concern, we ask whether their work has a particularly international or urban focus, where the work took place (from a menu ranging from specific Michigan counties to global), and what type of outreach work (outreach instruction, clinical service, etc.) they are doing. These responses reveal where and on what types of outreach university resources are being spent. Is instruction appropriately balanced with outreach research or clinical service, both in general and in specific areas of concern or location? How well is the University fulfilling its obligations to serve citizens in all parts of the state and to contributing to capacity building on a regional, national, and international basis?

The second section then asks faculty to characterize their collaborators in the external community  choosing one of 18 groups from a menu (educators, business and industry managers, labor advocacy and employment support personnel, etc.). Where this sections first question  on area of concern  identifies the target of their outreach work, this question focuses on their collaborators. Thus a faculty members work may target improvement in scientific understanding among the states K-12 students, but she might collaborate directly with some students, or with teachers, or with the association of school boards, or with state policy makers.

The second section of the survey concludes by asking for the number of people who attended or participated in the outreach work and for estimates of external funds (gifts, grants, contracts, or tuition revenue) coming to Michigan State to support the outreach activity and of external funds their outreach work helped generate for external collaborators. Earlier versions of the survey sought more detailed financial information, particularly estimates of resources the University and external partners contributed to the outreach work, but we found that faculty either resented such a heavy emphasis on monetizing activity that they consider socially rather than economically driven or were unable to estimate the costs of carrying on the activity. Thus, in the absence of detailed financial data, we rely on faculty time involved in outreach work as the primary measure of the Universitys investment in outreach.

The third section of the survey asks those faculty willing to do so to give more detailed descriptions of their outreach work or some aspect of it. In piloting earlier versions of this survey, we encountered an insistent demand from some faculty that efforts to quantify the outreach effort ignored, even suppressed, what is unique and important about outreach. Section three provides an opportunity for faculty to describe the importance of their work. A series of prompts asks them to describe the actions they took, for whom, with which collaborators (internal and external), and about what issue or problem. They are asked to specify how long the project has been underway and what plans, if any, there are for sustaining it, since the length of time a collaborative project is sustained is a good measure of outreach success. We are particularly interested in obtaining a sense of the outcomes of their outreach work  occurring both in the external community and in their own teaching or research  and in discovering whether any formal evaluation of the project occurred.

We do not expect to incorporate the data collected from this third section into the management analysis. Rather, we see the information gathered here as a rich storehouse of examples of how faculty expertise contributes to strengthening peoples capacity to achieve their goals. We plan to use this information as the basis for stories for the news media, for our presidents speeches and conversations across the state, for communication with the legislature and the alumni, and, perhaps most important, for communication to the faculty and staff about how their university enacts its outreach mission and how they might participate. We do not know how many faculty will avail themselves of the opportunity to give us this detail, but we expect that those who do will be the ones most involved in outreach and whose projects are most worthy of publicity.

Conclusion

Our data collection goal is to collect a rich array of information that we can use in a number of different ways. First, we will use the data as a management tool. We can examine changes in an academic units portfolio of outreach projects over time and survey the institutions overall portfolio of projects. Such information can be used in the planning and resource allocation process. Second, by having faculty document their outreach activities, we are better enabled to have outreach count in the allocation of rewards to faculty and units. Third, the information can be used as a major communication and publicity tool. Finally, we hope that what we learn can help provide a more accurate classification and definition of outreach/engagement for national forms of data collection and reporting.