This article was written by Christy Sullenberger, MS, Director of Member Services, NCHN & Rebecca J. Davis, Ph.D., Executive Director, NCHN (Adapted from the Field Guide to Nonprofit Program Design, Marketing and Evaluation) for the “Networking
News” monthly newsletter. The Network Technical Assistance
Project is funded by the Federal Office of Rural Health Policy, Health
Resources and Services Administration, U.S. Department of Health and Human
Services through a contract to Rural Health Innovations, LLC, a subsidiary of
the National Rural Health Resource Center.
Evaluating the network organization or a specific network program is an essential element of performance and process improvement, as well as overall assessment of effectiveness. Evaluations are used to improve programs, build organizational capacity, demonstrate value, and provide a basis for decision-making. Each element of a program evaluation provides insight into a different aspect of the network organization. Taken together, evaluation is a useful tool for the network leader, Board, stakeholders, and present and future funders.
The evaluation approach you choose guides you in the collection and organization of data, so it is important to develop an evaluation plan early in the process of implementation. In addition, as you begin collecting data, you can conduct an ongoing evaluation, which determines if implementation is going as expected. You may recognize this as part of the Plan Do Study Act cycle of process or performance improvement.
An evaluation may be goals-based, process-based (formative), outcomes-based (summative), or a combination of these. A goals-based evaluation determines whether you are meeting your overall objectives. A process-based evaluation addresses how your program works and highlights operational strengths and weaknesses. An outcomes-based evaluation addresses the benefits of your program to network members and/or the community. A final evaluation will often include pieces of all of these approaches.
Evaluation questions of these three approaches may be:
The evaluation approach you choose guides you in the collection and organization of data, so it is important to develop an evaluation plan early in the process of implementation. In addition, as you begin collecting data, you can conduct an ongoing evaluation, which determines if implementation is going as expected. You may recognize this as part of the Plan Do Study Act cycle of process or performance improvement.
An evaluation may be goals-based, process-based (formative), outcomes-based (summative), or a combination of these. A goals-based evaluation determines whether you are meeting your overall objectives. A process-based evaluation addresses how your program works and highlights operational strengths and weaknesses. An outcomes-based evaluation addresses the benefits of your program to network members and/or the community. A final evaluation will often include pieces of all of these approaches.
Evaluation questions of these three approaches may be:
- What do I need to know to make program decisions and adaptations?
- What is working well and what is not?
- How well does the program deliver value to members and stakeholders?
Evaluation Process Starts with Measureable Objectives
The evaluation process begins with determining clearly stated and measurable objectives for the program and then moves on to defining measurements of those objectives, collecting data, analyzing the data against program goals and objectives, and then illustrating the connection of the program's outcomes to the network's value.
To determine measureable objectives it is essential to sit down and consider your proposed activities and objectives. One common guideline for developing objectives is the SMART acronym. All objectives should be Specific, Measurable, Attainable, Realistic, and Time bound. Ensure that you have a strong list of realistic goals, achievable objectives, and appropriate activities that link directly to the desired results.
Collecting Data Comes is Many Shapes and Sizes
There are a variety of evaluation methods and models. Each method has strengthens and weaknesses. The important question to answer, is which method will best provide actual data that can be used to determine the effectiveness of the proposed project. An evaluation plan for the proposed project may need to incorporate different approaches. Some basic methods of program evaluation include the following:
The evaluation process begins with determining clearly stated and measurable objectives for the program and then moves on to defining measurements of those objectives, collecting data, analyzing the data against program goals and objectives, and then illustrating the connection of the program's outcomes to the network's value.
To determine measureable objectives it is essential to sit down and consider your proposed activities and objectives. One common guideline for developing objectives is the SMART acronym. All objectives should be Specific, Measurable, Attainable, Realistic, and Time bound. Ensure that you have a strong list of realistic goals, achievable objectives, and appropriate activities that link directly to the desired results.
Collecting Data Comes is Many Shapes and Sizes
There are a variety of evaluation methods and models. Each method has strengthens and weaknesses. The important question to answer, is which method will best provide actual data that can be used to determine the effectiveness of the proposed project. An evaluation plan for the proposed project may need to incorporate different approaches. Some basic methods of program evaluation include the following:
- Questionnaires and surveys: can be analyzed and presented numerically/quantitatively
- Interviews: provide primarily qualitative outcomes and can be conducted in person or on the phone and should be targeted and clear
- Documentation review: can be inexpensive, but may not provide a complete picture
- Focus groups: can provide a range of feedback, but may be slightly difficult to present analytically
- Case studies: can provide an in-depth look at a program and many variables
As you embark on your program evaluation keep in mind that the most important element in the evaluation process is that you start early in the process of implementation and that you are consistent in your collection methods.
Evaluation Resources
- Basic Guide to Program Evaluation (Including Outcomes Evaluation), Free Management Library
- Critical Components of Evaluation by Alana Knudson, Ph.D., National Rural Health Resource Center Evaluation Workshop (August 6, 2014)
- Tools and Strategies for Managing Health Networks: Network Evaluation, NCHN
- Program Evaluation, Rural Assistance Center Online Library
- Aim for Impact and Sustainability, National Rural Health Resource Center/Rural Health Innovations
- Flex Program Evaluation Toolkit, Technical Assistance Service Center (TASC), National Rural Health Resource Center
