Introduction and Overview
Welcome to Evaluating Results, the fifth course in the Transforming Life After 50 (TLA50) Fellowship series, with instructor Cindy Mediavilla. For the webinar offered in this course, see Week 3.
Seems like only yesterday we were discussing how to conduct community assessments. You’ve learned a lot in the meantime and are now ready to tackle evaluation. As you’ll see, some of the evaluation techniques we’ll be sharing are the same ones you’d use to assess your community. The difference, of course, is that while assessment occurs before programs exist to help plan them, evaluation occurs after library programs already exist to help determine their impact and effectiveness.
As a result of this course, you will be able to:
- Understand how important it is to evaluate the effectiveness of library programs.
- Develop measurable objectives and outcomes as the first step to effectively evaluating a library program.
- Distinguish the advantages of various evaluation methods, including observation, focus groups, and counting outputs.
- Utilize appropriate techniques and create valid instruments to evaluate library programs.
- Tabulate and interpret the data collected through program evaluation.
- Present program evaluation data in a coherent and appealing manner to relevant stakeholders and potential funders.
WEEK 1: Importance of Evaluation
This week will revisit the terminology of assessment and evaluation. The advantages of evaluating library programs and services, using objectives as the first step to designing an evaluation plan and using outcomes to measure the benefits of a library program or service will also be discussed.
Reading 1: Developing Objectives and Outcomes (pdf)
Assignment 1A: Review Your Understanding of Outcomes and Objectives
Apply what you’ve learned about outcomes and objectives and decide which of the statements below is an outcome and which is an objective. (Answers appear at the end of Week 1 below)
1. “By August 1, recruit three retired teachers to work as tutors in the library’s after-school homework center.” Is this statement an objective or outcome?
2. “A majority of the mid-life adults who attend the volunteer training workshop will report that they now have the skills needed to volunteer in the library.” Is this statement an outcome or objective?
3. “After participating in the first week of the Living Healthy lecture series, 40% of attendees will report that they eliminated caffeine from their diet.” Is this statement an outcome or an objective?
4. “By July 1, 75% of the nutrition collection will have circulated as a result of being featured as part of the Living Healthy lecture series.” Is this statement an outcome or an objective?
5. “25% of mid-life adults using the library’s career center will interview for a job by May 1.” Is this statement an outcome or an objective?
Assignment 1B: Writing Outcomes and Objectives
For this assignment, use your own grant-funded TLA50 initiative or choose a grant-funded initiative from the 2008-2009 list of California libraries that received TLA50 grants. Develop two objectives and one outcome statement for the initiative you’ve selected, and respond to the questions below. Keep in mind, your outcome should be expressed in patron terms and not library terms.
- What challenges did you face in constructing your objectives and outcomes?
- What steps did you take to resolve these challenges?
Supplementary Materials – Week 1
Sample Survey Tally Sheet (pdf)
McNamara, Carter. Basic Guide to Program Evaluation (including many additional resources), Free Management Library.
Rudd, Peggy D. “Documenting the Difference: Documenting the value of libraries through outcome measurement (pdf),” in Perspectives on Outcome Based Evaluation for Libraries and Museums.
Usherwood, Bob. (March 2002). Accounting for outcomes: Demonstrating the impact of public Libraries, Australasian Public Libraries and Information Services, 15 (1).
(Answers for Assignment 1A: 1. Objective; 2. Outcome; 3. Outcome; 4. Objective; 5. Outcome)
WEEK 2: Evaluation Techniques
How surveys and focus group techniques can be used to evaluate programs and services will be addressed. Observation as a data collection technique and counting outputs by gathering and interpreting traditional library statistics will also be explored.
Reading 2: Methods for Evaluating Achievement of Objectives and Outcomes (pdf)
Assignment 2A: Evaluation Methods for Sample Outcomes and Objectives
1. Download the Sample Evaluation Methods form (doc) and follow its instructions.
2. Answer these questions:
- What challenges did you face in deciding which evaluation methods to use?
- What differences between measuring objectives and outcomes did you encounter?
Assignment 2B: Evaluation Methods for Your Outcomes and Objectives
1. Download the Your Evaluation Methods form (doc) and follow its instructions.
2. Answer these questions:
- What challenges did you face in deciding which evaluation methods to use?
- What differences between measuring objectives and outcomes did you encounter?
Supplementary Materials – Week 2
Matthews, Joseph R. Measuring for Results: The Dimensions of Public Library Effectiveness
(Libraries Unlimited, 2004).
McNamara, Carter. Basic Guide to Program Evaluation (including many additional resources), Free Management Library.
Van House, Nancy A., et al. Output Measures for Public Libraries: A Manual of Standardized
Procedures (American Library Association, 1987).
Westbrook, Lynn. Identifying and Analyzing User Needs: A Complete Handbook and Ready-to-Use Assessment Workbook (Neal-Schuman, 2001).
WEEK 3: Tabulating and Presenting Data
This week formative and summative evaluation, drawing conclusions and use of triangulation to provide context and validate conclusions will be explored. In addition, how to share findings and make data understandable to stakeholders will also be discussed.
Reading 3: Evaluating and Drawing Conclusions (pdf)
Assignment 3A: Analyze Sample Data
1. Use the Sample Data Sheet on the Music of the 1960s: Rock’s Golden Era film and lecture series (pdf). Review the data presented and draw conclusions regarding the success—or failure—of the series.
2. Based on your review, answer the following questions:
- What challenges did you encounter in interpreting the data? What questions do the data answer? What questions still remain after compiling and processing the data? Did the methods help achieve triangulation?
- Which findings would you share with your library’s stakeholders? How would you share these data? Based on these data, what recommendations might you make to your library’s stakeholders regarding future baby boomer programs in your community?
Assignment 3B: Showcasing Baby Boomer Programs
1. View We’ve Only Just Begun (five-minute video), produced by the Sunnyvale Public Library, showcasing their Baby Boomer programs.
2. What do you think about their approach? How would this approach work for your library and your community?
Watch: Archived Recording of Course Webinar
As you watch this webinar on Evaluating Results, please consider these questions:
- Which methods have you used to evaluate library services and programs?
- What problems have you encountered while conducting evaluations?
- What methods have you successfully used to share program results?
Webinar Handouts
Webinar PowerPoint Slides (ppt)
Supplementary Materials – Week 3
W.K. Kellogg Foundation. Evaluation Handbook (2004)
Roseville Public Library video Transforming Life after Fifty
What to Do Next: Congratulations! You’ve completed the work for this course. If you’re interested, check out the other IMLS Fellowship online courses.
