Managing An Evaluator

It can be difficult to assess the work of an evaluator, especially if you do not have experience working with evaluators or understand evaluation methods and theories. It is helpful to lay out your expectations for how you would like to work with your evaluator during the recruitment process. (For more information on these issues, see “Hiring an Evaluator,” another NCMHPYVP Evaluation Brief). This publication outlines some strategies you can use to help monitor and assess an evaluator’s work.
 

Come to an agreement on the scope of work.

You and your evaluator should be in agreement on the specific tasks and subtasks that will be performed before work begins. A list of tasks and subtasks should be prepared, including any deliverables, such as reports, meeting presentations, or other products that are associated with each task. These tasks and deliverables can be changed over time, but the evaluator should not work on activities outside the established scope of work without your agreement.
 

Create an Evaluation Plan.

It’s always a good idea to have candidates for an evaluation position prepare a written plan for the evaluation before you hire them. Though they may not be able to specify all the details of the study in advance (e.g., which questionnaires they will use to collect data), they should be able to clearly identify areas of uncertainty and indicate how they will be resolved. The plan will be finalized in the first stages of the evaluation process. Evaluation plans typically include:

• Evaluation questions about process and outcomes that the study will answer.
• Evaluation design that determine whether the evaluation will be costly and rigorous or less costly and less rigorous.
• Types of data to be collected, including whether the evaluation will use a quantitative, qualitative, or mixed method approach.
• How these data will be collected, including choices of survey methodologies, pretests and posttests, or using existing sources of data.
• Sampling plan, including who will be sampled, how the evaluator will draw the sample, the extent to which the sample represents some larger group, and reasons why the evaluator is proposing a certain size sample.
• Data analysis plan, including the specific statistical or other procedures the evaluator will use to analyze the data.
• Protection of human subjects, including informed consent and confidentiality issues.
• Reporting procedures, including when reports will be submitted, what they will contain, and how they will be delivered.
• Staffing, including the percentage of time each person will devote to evaluation activities.
• Timetable for major evaluation activities and due dates for all deliverables.
• Budget for the evaluation.

Establish a timeline for reports and other products.

Due dates should be established for each deliverable listed in the scope of work. Time should be built into the schedule so reports can be reviewed and revised before submitted to funders and others.

Write a contract.

An important part of managing an evaluator is preparing a contract for the evaluation work. A contract should include the following:

• Scope of work. Include the list of evaluation tasks, all evaluation deliverables, and due dates for the completion of tasks and the submission of deliverables.
• Evaluation personnel and task responsibilities. Identify who will do the work. Evaluations often involve a team on which there is a senior evaluator who heads the team and junior staff who do much of the work. You should know which tasks the head evaluator will perform and which tasks will be carried out by assistants. The contract should also specify any evaluation tasks that will be the responsibility of your program staff.
• Ownership of data. Specify who owns the information and data collected by the evaluator and who has publication rights. The program or its parent agency usually owns this information. Authorized agency staff should always clear the release of information to outside parties. This includes the publication of data in professional papers and journal articles.
• Expected contacts with the program. The evaluator needs to keep the program staff informed about the status of the evaluation. The contract should specify any required progress reports, meetings with the program staff, and attendance at other meetings.

Meet regularly.

Regular face-to-face interaction provides you with the opportunity to monitor the progress of each task in the scope of work. During these meetings, ask the evaluator to brief you on the status of each task and identify any problems that have been encountered. Use these meetings to make any changes formally to the evaluation design. Monthly meetings with the evaluator are not too much to expect. If regular in-person meetings are not possible, regular conference calls can suffice.
 

Ask questions.

Any time you are unsure about the work that is being done, ask questions. Continue to ask questions until you understand what is going on. Don’t be afraid to ask for explanations of how work activities relate to the tasks in the scope of work.

Require progress reports. Brief monthly or quarterly progress reports are helpful. It is important that these reports should cover at least three areas:

• activities that were undertaken during the reporting period,
• any problems or issues that surfaced and how they were addressed
• activities that are expected to be carried out during the next reporting period.

Progress reports should be kept brief.
 

Adhere to the schedule of deliverables.

Make sure that all reports and other products are delivered according to schedule. Don’t be afraid to ask in advance if the next scheduled deliverable will be on time. Delays in the delivery of one product may have a ripple effect on the delivery of subsequent products. If the delivery schedule is too tight or unrealistic, modify the schedule to avoid future problems.
 

Adhere to the agreed upon payment schedule.

Evaluation consultants can be paid in a variety of ways. This is something you need to negotiate with your consultant before a contract is signed. Small consulting contracts are sometimes paid in one lump sum at the end of a contract or when the final evaluation report is submitted. Larger contracts are often paid in monthly installments upon the consultant’s submission of a detailed time log.

Perhaps one of the most important things to remember is to deal with issues or concerns as they arise. Trust your instincts—don’t sweep problems under the rug or ignore nagging worries just because you don’t know all the subtleties of evaluation design. Treat your evaluator as you would any other staff member. Remember, he or she is not on board to judge you or your program but to work with you to improve the quality of the services you provide.