This week’s assignment involves selecting a published evaluation report related to our professional or academic interests and critically examining it using the evaluation practices and concepts we’ve learned throughout the course.
Finding the 2016 Vintage Senior Center Annual Evaluation Report was a welcome surprise, as it closely aligns with the program I am evaluating. This annual internal evaluation was conducted by Vintage, a senior center in Pittsburgh, Pennsylvania, to assess whether its programs support the organization’s mission and improve seniors’ quality of life (Vintage Senior Center, 2017).
My initial reaction was mixed but mostly positive. It was well organized, but I wanted more information in some areas. I used Fitzpatrick et al.’s (2011) Checklist for Good Evaluation Reports to guide my critique. See checklist below.
The organization uses a consistent format with a logical layout and accessible language appropriate for staff, administrators, and community partners. I found the charts and graphs helpful in illustrating participation trends and program growth, which I prefer to written descriptions of data. This aligns with Fitzpatrick et al.’s (2011) emphasis on effective communication, visual appearance, and tailoring reports to their audiences.
The report has a clear evaluation purpose and explains how findings will be used. Though some parts were vague to me, I suspect those preparing this annual report assume some information is common knowledge. However, I would argue that basic information bears repeating. The report details stakeholder involvement, including the Center Director, Executive Director, and House Council (Vintage Senior Center, 2017). It also draws from surveys and participation records. One area I found lacking was the limited explanation of sampling, data collection, and analysis methods. Fitzpatrick et al. (2011) identify evaluation procedures and supporting technical information as important elements of a good report. Including these details would make the findings easier to interpret.
One interesting observation was the emphasis on increased attendance and program utilization. For example, the report identifies a 459% increase in Enhance Fitness participation over three years (Vintage Senior Center, 2017). Sounds impressive! But does increased participation necessarily demonstrate program effectiveness? Attendance measures utilization, but does not tell us whether participants experienced meaningful outcomes. Fitzpatrick et al. (2011) emphasize specifying criteria for evaluative judgments. The report includes outcome findings from the Better Choices, Better Health program, but I would have liked clearer standards for judging other programs.
A few other areas could use attention. There is no explicit discussion of evaluation limitations, and details about survey sampling and response rates are limited. Its tone also emphasizes strengths more than weaknesses, raising questions about potential bias in an internal evaluation. This does not mean the findings are biased, but I wonder whether staff evaluating their own programs influences how results are presented. I also found Vintage’s 2019 evaluation report and noticed similarities in the structure and background sections (Vintage Senior Center, 2020). While consistency makes comparisons easier, could using the same format each year limit opportunities to question assumptions or explore new approaches?
Overall, I consider this evaluation report satisfactory. Most criteria on Fitzpatrick et al.’s (2011) checklist were satisfactory, several needed improvement, and only one was unsatisfactory. I recognize that many items on my wish list come from being an outsider, while the report’s intended audience appears to be people familiar with the center. Reviewing this evaluation has made me look forward to completing my own evaluation. I hope to build on this report’s strengths while addressing areas I felt could use improvement.
Using Fitzpatrick et al.’s (2011) Checklist for Good Evaluation Reports, the following characters were used to review the evaluation report: (+) for satisfactory, (−/+) for needs improvement, (−) for unsatisfactory or poor, and (o) for cannot be determined or not applicable (Fitzpatrick et al., 2011, p. 479).
| A Checklist for Good Evaluation Reports | |
| o | Interim and final reports provided in time to be most useful |
| + | Report content, format, and style tailored to the audience(s) |
| o | Involvement of audiences in determining the format and style of the report |
| −/+ | An executive summary |
| + | An adequate introduction to set the stage |
| − | Mention of limitations of the study |
| −/+ | Adequate presentation of evaluation plan and procedures (primarily in appendices) |
| + | Effectively organized presentation of results |
| −/+ | All necessary technical information provided (preferably in appendices) |
| −/+ | Specification of standards and criteria for evaluative judgments |
| −/+ | Evaluative judgments |
| −/+ | Lists of identified strengths and weaknesses |
| + | Recommendations for action |
| o | Protection of clients’ and stakeholders’ interests |
| + | Sensitivity to those affected by the evaluation findings |
| o | Provision for minority reports or rejoinders |
| −/+ | Accurate and unbiased presentation |
| + | Effective communication and persuasion through telling the story |
| −/+ | Appropriate level of detail |
| + | Lack of technical jargon |
| + | Use of correct, uncomplicated, and interesting language |
| + | Use of examples and illustrations |
| + | Attention to visual appearance and eye appeal |
References
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Pearson.
Vintage Senior Center. (2016). Vintage Senior Center annual evaluation report: Calendar year 2016. https://vintageseniorservices.org/documents/Evaluation2016.pdf
Vintage Senior Center. (2020). Center for Active Adults: 2019 evaluation report. https://vintageseniorservices.org/documents/Evaluation.2019.FINAL.pdf