Archival note: Although information contained in OIT Newsletter articles was current at the time of publication, some details may no longer reflect the present state of technology and the Office of Information Technology.
December 2003 Contents |
Information Technology Newsletter |
Cumulative Index |
| |
||
DMC Spotlight IssueTEL Evaluation Methods
This month, we highlight information about TEL evaluation methods discussed in recent TEL short courses and at DMC Faculty Fellowship and TEL Grants Program meetings. An online guide, a bibliography, information about campus projects, and links to other resources related to TEL evaluation methods are available on the DMC site at http://dmc.umn.edu/spotlight/. An excerpt appears below. DescriptionAs educational uses of technology have become more common, faculty members, administrators, and staff members have begun to investigate the impact that technology is having on students' learning environments. Are students able to integrate the use of technology into their study habits? Does it permit them to work effectively with their peers? Have their learning outcomes improved? A well-conceived evaluation project can produce very illuminating answers to these questions. For instance, a recent technology survey of students at the University of Minnesota revealed that:
Conducting a TEL evaluation is not very different from performing a standard educational evaluation. In essence, it's a matter of determining whether you have reached the goals you hoped to reach through your use of educational technology. Online guideAn online TEL Evaluation Methods guide is available on the DMC web site at http://dmc.umn.edu/evaluation/. Highlights include the following. Determine purpose and goals You need to determine the purpose of the evaluation. Your evaluation could be formative or directed at revision and improvement of a TEL project; or it could be summative, directed at determining whether a TEL program continues, is expanded, or is reduced. You also need to set out with some specificity the goals of your TEL project. These will often need to be clarified and operationalized in order to be amenable to evaluation. Example: Select strategies and tools Once you've laid your goals out in this way, you're ready to think about what evaluation strategies to use. Some strategies produce different types of data and make possible different sorts of inferences. Quantitative strategies focus on research that produces numbers, often attempting to quantify changes in learning outcomes; qualitative strategies focus on research into phenomena that are not amenable to easy quantification, like understanding and analytic ability. The distinction between quantitative and qualitative data often polarizes people in the evaluation field. The best approach probably is to use both approaches in complementary fashion. Quantitative evaluation is good at telling you what has occurred, while qualitative evaluation is good at telling you why those things occurred as they did. Quantitative methods provide data that is comparable and amenable to powerful analytic techniques; qualitative methods provide richness of context and depth of understanding. You also need to determine which tools you will use to collect data. Each has its advantages and disadvantages, and is appropriate for evaluating certain sorts of goals. It's often a good idea to use several different tools, such as the following:
Example: Collect data Once you have selected your strategies and tools, you need to construct and distribute them to students. You'll need to consider when to deliver each tool, what questions to ask, how to phrase the questions, how to deliver the tool, how to record data, and how to report results to the students if possible. The exact structure of a given evaluation project depends heavily on the goals and objectives of the educational experience and on the particular research questions in which you are interested.
Analyze and share the results When your evaluation is complete, you will need to analyze the data you've collected and draw appropriate conclusions about your use of educational technology. The analysis you perform can either be qualitative or quantitative. Whatever sort of analysis you perform, others will be interested in your results, so you might want to consider publishing them. Qualitative analysis example: You also could cross-tabulate answers about the type of Internet connection they use with their answers to questions about the frequency with which they access a course Web site or with their attitudes about the usefulness of online course materials. You then might discover that students with high-speed Internet connections access online materials more frequently or feel more positive about such materials. You then could recode the variables to compare the answers given by freshman and upper-class students. If you obtained information about the students' year in school through a survey question, you could recode the variables to gather several of the answers to this question (e.g., by sophomores, juniors, and seniors) into a single variable, and then compare the answers given by freshmen and upper-class students. Other resourcesIf you are interested in evaluating a TEL project, you may want to use one of the campus and outside resources that will be highlighted on a Spotlight Issues TEL Evaluation Methods page at http://dmc.umn.edu/spotlight/, such as the following
Other spotlight issuesThe next DMC Spotlight Issues will be the topics of the spring sessions of the TEL Seminar Series.
Digital Media Center: adapted by Christina Goodland from materials by J. D. Walker |
Page: www.umn.edu/oit/newsletter/03/1203_itn/spotlight.html |