Measures and Instruments
As part of the evaluation of your SS/HS initiative, you and your evaluator will be collecting data on the process and outcome measures you have identified in your logic model.For more information on this topic, please see the Measures and Instruments Resource Page.
Questionnaires are an efficient method of collecting information on relatively large groups of people. Questionnaires consist of measures or sets of measures. Adopting existing measures can save evaluation resources, increase your confidence in the results (provided that the measures you select are valid and reliable), and increase the ability to compare your data to that from other communities, your state, or the nation (depending on how widely the measure is used).
As part of the evaluation of your SS/HS initiative, you and your evaluator will be collecting data on the process and outcome measures you have identified in your logic model. Adopting existing measures rather than attempting to create your own measures can save evaluation resources, increase confidence in the results (provided that the measures you select are valid and reliable), and increase your ability to compare these data to those from other communities, your state, or the nation (depending on how widely the measure is used). Having a good understanding of the kind of questions that you want to answer will help you to identify appropriate measures and instruments.
Process and Outcome Measures
Process measures document implementation of the programs, policies, and practices that are put into place in your community with SS/HS funding. Process measures include counts of program participants, trainings, sessions, or meeting attendees; assessments, spot checks, or interviews to assess whether students are utilizing the program; and questionnaires, checklists, or observations of program staff to indicate whether or not they are implementing the program with fidelity. When analyzing your data, process measures can also be used to explore possible moderators of program outcomes – variables that can change the strength of a relationship. For example, schools that implement a program with greater fidelity may show better outcomes than those that do not. When identifying process measures, keep in mind what measures will generate useful information for your site. What measures will allow you to better understand quality of implementation? What measures will allow you to explore potential moderators of program effects? In some cases, evidence-based interventions will supply sources of process/fidelity assessment data (e.g., teacher check-lists, student satisfaction and utility surveys). In many cases, process measures will be unique to each individual implementation site.
Outcome measures are used to assess the intermediate and long-term effects of program services. Many evidence-based interventions come with pre-packaged evaluation instruments that assess the intermediate and long-term outcomes of that specific intervention. It is almost always prudent to use or adopt developer-supplied measures since, in most cases, these measures were created or chosen based on the intervention’s theory of change.
State and national surveillance instruments such as the National Institute on Drug Abuse’s Monitoring the Future Survey, the Centers for Disease Control and Prevention’s Youth Risk Behavior Survey, and the California Health Kids Survey can also serve as a source of outcome data. In most cases, communities will choose to adopt a selected number of questions or scales from these instruments rather than attempting to implement the entire survey. The primary advantage of this approach is that these measures have been validated and tested with large numbers of diverse audiences – resulting in increased confidence in the data generated from these questions. Another advantage of adopting measures from these surveys is that it facilitates comparisons to similar communities, your state, or the nation as a whole. Several of these surveys also contain items that you can use to collect data on your required GPRA measures. For example, many of these surveys contain items measuring 30-day AOD use, physical aggression, and perception of safety. The National Evaluation Team has also created several surveys that you may choose to use or adapt to measure outcomes such as school climate (see the National Evaluation Page for additional information).
Collaboration/Partnership Measures
Partnerships are a key component of SS/HS initiatives. They are intended to increase service efficiency and utilization by providing enhanced service coordination and eliminating system fragmentation and service silos. Ideally, SS/HS partnerships should continue beyond the period of Federal funding and continue to oversee coordination among the schools, mental health, law enforcement, and juvenile justice systems. Measuring partnership functioning through specific process and outcome measures can provide you with information on the health of this entity.
In general, partnership and collaboration measures focus on factors such as the characteristics and perceptions of members, characteristics of the participating individuals and organizations, group functioning (including structure, bylaws, climate), and perceptions of partnership impacts or outcomes. Some surveys also assess satisfaction with the partnership/collaboration, satisfaction with the leadership, perceptions that accomplishments are being made, and willingness/likelihood to continue participating in the group. Several of the items contained in the National Evaluation surveys are intended to measure some of these different aspects of partnership functioning. There is also a very rich research literature on coalitions and partnerships, which includes different measures of partnership functioning.
Choosing Appropriate Measures and Instruments
When choosing measures for your SS/HS evaluation, think about the data collection needs for your site that will help you to document what you have done and the fidelity of program implementation, make program improvements, help with program sustainability and marketing, and meet the federal reporting requirements. Consider collecting pre-existing sources of data, such as school truancy records or items from surveys that the district already requires students to take, that will meet your evaluation needs. You may even be able to add an additional administration of the survey to the existing schedule to meet your data reporting needs. This is often the case where school districts have existing surveillance systems in place (e.g., student surveys) that are collected every two years from the entire student body. In order to be in compliance with GPRA requirements and to collect more frequent information, you may need to implement your surveys (or a subset of questions in these surveys) on an annual basis.
When the data you need cannot be collected via existing sources, look at existing instruments that may be measuring the same concepts that you are looking to measure. These instruments may be effective as written, you may simply need to make a small tweak, or you may only need to adopt a few items from the entire instrument. It may even be possible to add these instruments or items to existing surveys currently being administered in your school or district. Be creative and you may find ways to get the data you need more cost-effectively and faster.
