Evaluation Toolkit
The purpose of the local evaluation, in the words of the program announcement, is to “provide timely information for creating strategic plans, measuring progress, and keeping the project focused on the overall objective of the SS/HS initiative—promoting healthy childhood development and preventing violence and substance abuse.” Every SS/HS grantee is required to conduct a local evaluation. SS/HS grantees were required to include an evaluation design in their applications. This chapter will provide guidance in implementing your local evaluation as well as complying with other Federal evaluation requirements – notably the cross-site National Evaluation and the requirements of the Government Performance Results Act (GPRA). Additional tools, information, and resources on the topics discussed in this chapter can be found throughout the SS/HS Evaluation Toolkit.
Why Evaluate Your SS/HS Initiative?
Evaluating your SS/HS initiative provides timely information for managing both the day-to-day and long-term aspects of your project, measuring progress toward goals and objectives, and understanding if the initiative is, in fact, promoting healthy childhood development and preventing violence and substance abuse. Positive evaluation results can also gain the support needed to sustain components of the initiative after Federal funding ends. Evaluation is important for a number of reasons:
- Evaluation demonstrates to your local partners, stakeholders, and the community that your initiative (and the SS/HS initiative as a whole) is effective and worth sustaining. The Substance Abuse and Mental Health Services Administration, Department of Education, and Office of Juvenile Justice and Delinquency Prevention also use evaluation information to justify requests to Congress for monies to support prevention and intervention programs.
- Evaluation helps a project respond to changes that may impact its effectiveness by documenting implementation and how implementation interacts with the institutional and social environment in which the project takes place. Knowing this can help you decide where best to channel your resources and what parts of the initiative to institutionalize and sustain.
- Evaluation can improve program efficiency by revealing opportunities to streamline program delivery or enhance coordination between program components. Increased efficiency can help reduce cost or allow you to provide more services to a larger audience at the same cost.
Participatory Evaluation
In the participatory approach to evaluation, the evaluator works closely with the program, advising (rather than mandating) decisions related to the evaluation. Many SS/HS projects find that participatory evaluation is especially effective, as it reflects the inherent collaborative nature of the SS/HS initiative. The participatory approach relies on an Evaluation Team composed of one or more individuals trained in evaluation, program staff, and representatives of other stakeholders (such as students who are receiving the program or their parents). The members of the evaluation team work together to plan and implement all evaluation tasks. If you would like to undertake a participatory evaluation of your program, make sure to discuss this approach with candidates during the screening process.
Participatory evaluation is frequently recommended for a number of reasons, including the following:
- When an evaluation is planned and developed collaboratively between program staff and an evaluator, there is less potential for misunderstanding and an increased likelihood that program staff and other stakeholders will understand the benefits of evaluation. In being active participants in the process, program staff are also more likely to accept and use the findings from the evaluation.
- Since front-line staff are often more knowledgeable than an evaluator about the unique needs, culture, and circumstances of the target population, their input can make the evaluation more relevant, appropriate, and sensitive – this creates a more accurate estimate of why and how the program affected these people.
The participatory approach has also some potential disadvantages. It requires more time on the part of both the program staff and the evaluator. This includes meeting time, consulting time, and time to share and resolve differences of opinions. It also increases the potential for bias, since evaluators working closely with program staff may want to show project success as it reflects work done by their colleagues.
The project director is responsible for managing the evaluator regardless of whether a participatory approach is used. Guidance on this relationship can be found in Managing An Evaluator. The participatory approach entails additional responsibilities for the project director, including the following:
- Working with the evaluator to make certain the evaluation design responds to the project’s needs, is consistent with available resources and the initiative’s timeline, and fulfills all federal requirements
- Identifying key individuals to sit on an Evaluation Team (usually consisting of the project director, evaluators, and other stakeholders interested in the evaluation)
- Monitoring the evaluation to ensure that it is being carried out efficiently and on time
- Providing support to those carrying out the evaluation (including the Evaluation Team and those in the field who are engaged in activities such as administering surveys or abstracting and submitting data)
- Using information from the process evaluation to monitor and manage the initiative and make modifications to programming as needed
- Keeping partners and stakeholders informed about the evaluation and seeking their input and support
- Using evaluation data to fulfill Federal reporting requirements.
