Can You Trust Your Results? Why Reliability and Validity Matter

Organizations use data to make important decisions every day. They use surveys to understand employees, clients, and program participants; assessments to evaluate professional knowledge; and research findings to decide whether an initiative should continue or expand.

But a polished report and a large dataset do not necessarily mean the conclusions are trustworthy. Before acting on the findings, organizations should ask two questions:

Are we measuring the right thing?

Are we measuring it consistently?

These practical questions are at the heart of validity and reliability.

Validity: Are We Measuring the Right Thing?

Validity concerns whether the available evidence supports the way results are interpreted and used. It is often described more simply as whether a measure captures what it is supposed to measure.

Consider an organization that offers a leadership-development program. At the end of the program, participants are asked whether they enjoyed the training, liked the facilitator, and would recommend the program to a colleague.

Those questions may provide useful information about participant satisfaction, but they do not measure leadership development. Participants could enjoy the program without improving their ability to resolve conflict, communicate expectations, provide constructive feedback, or lead a team.

If the organization interprets favorable satisfaction ratings as proof that leadership skills improved, it is drawing a conclusion the survey was not designed to support.

To measure leadership development, the organization would need questions or activities connected to the specific knowledge, skills, or behaviors the program was intended to strengthen. It might still measure participant satisfaction, but it should not treat satisfaction and learning as if they are the same outcome.

This distinction applies across social science research. A client survey may measure satisfaction with services without determining whether those services improved the client's circumstances. A professional assessment may emphasize memorization when the intended outcome is the ability to apply knowledge. A study of a workforce program may document employment without examining whether participants obtained stable jobs, adequate hours, or opportunities for advancement.

Validity begins with defining the intended outcome clearly and ensuring that the measure adequately represents it.

Reliability: Are We Measuring It Consistently?

Once we know that a measure addresses the intended outcome, we must ask whether it does so consistently.

Reliability refers to the consistency or dependability of measurement. Results should not fluctuate unpredictably because questions are confusing, instructions change, scoring criteria are applied differently, or the measure contains too much error.

Suppose two supervisors use the same rubric to evaluate an employee's performance but assign dramatically different scores to the same work. The difference may reflect unclear scoring criteria rather than a genuine difference in performance.

The same problem can occur in surveys and assessments. Participants may interpret vague questions differently. Items intended to measure the same concept may produce contradictory responses. Scores may vary substantially even when the underlying knowledge, attitude, or behavior has not changed.

Reliability does not mean that scores should never change. People learn, develop, and have new experiences. The goal is to ensure that changes in the results reflect genuine differences—not instability in the measure.

Why Consistency Alone Is Not Enough

A measure can be reliable without supporting a valid conclusion, but it cannot support a valid conclusion without being sufficiently reliable.

A bathroom scale provides a simple example. Imagine that the scale consistently reports a person's weight as five pounds heavier than it actually is. The result is dependable: the scale produces the same error every time. But interpreting that number as the person's actual weight would be incorrect.

Surveys and assessments can create the same problem. A series of questions may produce highly consistent responses while measuring only part of the intended outcome—or something different altogether.

An assessment intended to measure employees' ability to apply a new policy, for example, might consistently measure how well they memorized its wording. The scores could be reliable while failing to provide the information the organization actually needs.

Consistency matters, but consistent measurement of the wrong thing does not produce useful results.

Are the Results Appropriate for This Purpose?

Validity and reliability do not exist independently of context. A measure is not appropriate for every population, setting, and purpose simply because it performed well in one study.

A survey developed for senior executives may not function in the same way with direct-service staff. A measure developed for the general population may not be accessible to people with intellectual or developmental disabilities. An assessment designed to measure knowledge may not be suitable for evaluating performance in real-world situations.

The same measure may also be adequate for one decision but inadequate for another. A brief questionnaire might help identify topics for an upcoming training program while providing too little information to determine whether participants have mastered the content.

The evidence needed should reflect how the results will be used. The more consequential the decision, the more confidence an organization should require in the measure and its scores.

What Should Organizations Ask Before Acting?

Organizations do not need to become measurement experts, but they should ask informed questions before making decisions based on research findings:

• What was this instrument designed to measure?
• Do the questions adequately represent that outcome?
• What evidence shows that the scores are sufficiently consistent?
• Is the measure appropriate for these participants and this purpose?
• Do the results support the conclusion we want to draw?

Reliability and validity are not simply technical terms to include in a research report. They determine whether the findings mean what an organization believes they mean—and whether those findings are dependable enough to guide action.

Before asking what the results say, first ask whether the results can be trusted.

Looking for Psychometric Support You Can Rely On?

Whether you're evaluating a program, reviewing an existing evaluation tool, or developing a reliable and valid measurement instrument, such as a survey or assessment, Research Analytics Consulting can help determine whether your measures are capturing the right outcomes and producing results you can rely on. If your organization is located in Florida, Arizona, Wisconsin, New York, or anywhere in the United States, contact our team to talk through your project and the decisions your data needs to support.