Why Measurement Matters in Program Evaluation

Good measurement gives organizations confidence in their evaluation findings. It helps them understand whether a program is achieving its intended outcomes, where improvements may be needed, and which decisions the results can reasonably support.

Before evaluators calculate percentages, compare groups, or examine changes over time, they must determine whether the measures used to collect the data are well designed. Statisticians often describe this principle with the expression “garbage in, garbage out.” The results of any statistical analysis are only as good as the information being analyzed.

Even sophisticated analyses cannot compensate for questions that are unclear, response choices that do not fit the question, or survey logic that sends participants through the wrong set of items. Measurement is not simply a preliminary step in an evaluation. It is the foundation on which the evaluation rests.

Turning Outcomes Into Measurable Information

Measurement is the process of translating an outcome into information that can be observed and analyzed.

Some program outcomes are relatively straightforward. Attendance, graduation, employment, and completion of a certification can often be documented directly. Other outcomes—such as confidence, leadership, financial literacy, workplace readiness, self-advocacy, and quality of life—are more complex.

Evaluators must define what each outcome means within the context of the program and determine how it can be measured accurately.

For example, a program may be designed to increase participants’ financial literacy. Asking participants whether they feel more knowledgeable provides information about their perceptions, but it does not necessarily demonstrate that they can create a budget, compare borrowing costs, recognize financial risks, or make sound financial decisions.

An evaluation may need to measure both perceived growth and demonstrated knowledge or skills. The important point is that the questions must represent the outcomes the program is intended to influence. Otherwise, the evaluation may report improvement without clearly demonstrating what participants learned or can now do.

A Survey Is Not Automatically a Good Measure

Online survey platforms have made it easy to create questionnaires and collect large amounts of data. However, the ability to administer a survey is not the same as the ability to design a sound measurement instrument.

Small problems in survey questions can substantially affect the answers participants provide. Questions may be vague, overly complicated, leading, or open to multiple interpretations. A single question may also ask about two different ideas.

Consider the statement:

“I understand financial planning and feel confident managing my money.”

A participant might understand financial planning but lack confidence. Another might feel confident despite having limited financial knowledge. Because the statement combines two different ideas, the evaluator cannot determine which idea the participant considered when answering.

Every survey question should have a clear purpose. Evaluators should ask whether the wording is understandable, whether the question measures one idea at a time, whether it is appropriate for the people completing the survey, and whether the answer will provide information connected to the evaluation questions.

Survey Logic Determines Which Questions Participants Answer

Survey logic controls how participants move through a questionnaire. It determines which questions they receive based on their previous answers.

For example, participants who indicate that they are employed may be asked follow-up questions about their hours, wages, job responsibilities, and workplace experiences. Participants who are not employed should bypass those questions and receive questions about job seeking, training, or barriers to employment.

When survey logic is designed correctly, participants receive questions that are relevant to their experiences. The survey takes less time to complete and is less likely to frustrate or confuse respondents.

Poorly constructed survey logic can create serious data-quality problems. Participants may be asked to describe services they never received or experiences they did not have. Others may inadvertently skip questions that are essential to the evaluation. If a response is required, participants may select an inaccurate answer simply so they can continue through the survey.

These problems can be difficult to identify after data collection has ended. The dataset may appear complete even though some responses were created by an incorrect path through the questionnaire.

Survey logic must therefore be planned, programmed, and tested carefully. Evaluators should test every possible path before administering the survey and confirm that participants receive the correct questions based on their responses.

Response Scales Shape the Meaning of the Answers

The response scale is just as important as the wording of the question. It provides the options participants use to communicate their answers and determines how those answers can be interpreted.

The scale must match what the question asks. Questions about frequency require response options such as “never,” “rarely,” “sometimes,” “often,” and “always.” Questions about confidence require choices ranging from low to high confidence. Questions about agreement require clearly ordered levels of agreement.

Problems arise when the question and the response scale do not align. Asking how often someone performs a behavior and then providing choices from “strongly disagree” to “strongly agree” forces the participant to translate a frequency judgment into an agreement judgment. Different participants may make that translation differently.

Several other decisions affect the quality of a response scale:

• How many response options participants receive
• Whether each option is clearly labeled
• Whether a midpoint is appropriate
• Whether participants need a “not applicable” or “don’t know” option
• Whether the choices represent the full range of likely experiences
• Whether similar questions use scales consistently

More choices do not automatically produce more precise data. Participants must be able to distinguish meaningfully among the options. A ten-point scale may create the appearance of precision without ensuring that respondents interpret the difference between a six and a seven in the same way.

The wording of the choices also matters. A scale that offers mostly positive options can steer participants toward favorable responses. A scale without a legitimate “not applicable” choice may force them to provide an answer that does not represent their experience.

Well-designed response scales make it easier for participants to answer accurately and for evaluators to interpret the results responsibly.

Measurement Quality Must Be Established Before Data Collection

Strong measurement requires more than writing a collection of reasonable-sounding questions. The outcomes must be clearly defined, the questions must align with those outcomes, the response scales must fit the questions, and the survey logic must function as intended.

Measures should also be reviewed and tested with people similar to those who will complete them. This process can reveal unclear wording, missing response choices, unexpected interpretations, and problems with the survey pathway before those issues affect the evaluation data.

Reliability and validity provide more formal ways to examine whether a measure produces sufficiently consistent information and supports its intended interpretations. Because these concepts deserve careful attention, they will be explored in the next article in this series.

Better Measurement Leads to Better Decisions

Organizations invest in evaluation because they want useful answers. They want to know whether programs are working, how they can be improved, which participants benefit, and where resources should be directed.

Those answers are only as credible as the measures on which they are based.

A large dataset does not compensate for weak measurement. Neither do sophisticated analyses, polished visualizations, or professionally written reports. Trustworthy conclusions begin with clearly defined outcomes, carefully written questions, appropriate response scales, and accurate survey logic.

Before asking what the data show, organizations should first ask whether they collected the right data—and whether those data are strong enough to support the decisions that follow.

Need Help With Program Evaluation and Measurement?

Research Analytics Consulting helps organizations strengthen their program evaluation through thoughtful measurement strategies, survey development, and data collection tools designed to produce meaningful results. Whether you need program evaluation support in Florida, Arizona, Wisconsin, New York or anywhere in the United States, our consultants can help you develop reliable measures that align with your program goals and provide useful information for decision-making. Contact Research Analytics Consulting to discuss your program evaluation or survey development needs.