Research Analytics Consulting, LLC

One Question, Many Lenses: The Value of Multidisciplinary Research

Wed, 14 Oct 2026 16:54:04 GMT

The questions organizations bring to researchers and evaluators are rarely simple. They may want to know whether a program is working, why some participants benefit more than others, how services can be improved, or whether an initiative should be expanded.

Answering these questions requires more than collecting data and producing a report. It requires an understanding of research design, measurement, statistics, lived experience, program implementation, and the environment in which the work occurs.

No single professional is likely to bring equal depth in all these areas. Multidisciplinary teams combine complementary perspectives so that research and evaluation can be more rigorous, relevant, and useful.

Complex Questions Require Different Types of Expertise

Every research or evaluation project involves a series of interconnected decisions.

Researchers must determine which questions can realistically be answered, select an appropriate design, define the outcomes, develop or select measures, collect information from the right people, analyze the data, and interpret the findings within the context of the program or organization.

Each decision affects the next. A strong statistical analysis cannot correct a poorly designed measure. A well-designed survey may still provide an incomplete picture if it overlooks participants' experiences. A technically sound study may have little practical value if the research team does not understand how the program operates.

A multidisciplinary team brings together professionals who view these decisions from different perspectives. Depending on the project, the team might include specialists in research design, psychometrics , statistics, qualitative research, program implementation, content development, or knowledge translation. It may also include practitioners, clients, program participants, or community members who understand the issue through direct experience.

The goal is not to add people for the sake of creating a larger team. It is to bring together the expertise and perspectives necessary to answer the questions responsibly.

Different Perspectives Reveal Different Problems

Professionals are trained to notice different things.

A statistician may recognize that the available sample is too small to support a proposed analysis. A psychometrician may identify questions that do not adequately represent the intended outcome. A qualitative researcher may recognize that the survey cannot explain an unexpected finding. A practitioner may point out that a proposed data-collection process will not work in the real-world setting.

People with lived experience may identify problems that are invisible to the research team. They can explain how language may be interpreted, why participants may hesitate to answer a question honestly, or how the design of a service differs from the way it is experienced.

These perspectives do more than add detail. They can prevent the team from pursuing the wrong question, using an inappropriate measure, or interpreting findings without sufficient context.

Constructive disagreement is also valuable. When team members examine assumptions from different professional viewpoints, they are more likely to identify weaknesses before those weaknesses affect the project.

Better Research Begins with Better Questions

A multidisciplinary team can improve a project before any data are collected.

Consider an organization evaluating a workforce-development program. A researcher might initially ask whether participants obtained employment after completing the program. That is an important outcome, but it may not be sufficient.

A workforce specialist might recommend examining whether the jobs align with participants' skills and career goals. A statistician might suggest comparing outcomes for participants with different characteristics or levels of program participation. A qualitative researcher might explore why some participants secured employment while others did not. Participants might explain that transportation, caregiving responsibilities, workplace accommodations, or unpredictable scheduling affected whether they could accept or retain a position.

Together, these perspectives create a more meaningful understanding of program success. Employment remains important, but it is considered alongside job quality, stability, access, and participants' actual experiences.

Without this broader perspective, the evaluation might produce a clear answer to a question that was too narrow.

Collaboration Strengthens Interpretation

Research findings do not speak for themselves. They must be interpreted carefully.

A change in scores may reflect genuine improvement, differences in who responded, changes in how services were delivered, or limitations in the measure. An apparent lack of progress may mean that the program was ineffective, but it could also mean that the intended changes require more time or that the selected outcomes did not capture the program's most important effects.

Multidisciplinary interpretation reduces the likelihood that one explanation will be accepted too quickly. Team members can examine whether the statistical findings align with participant experiences, whether the conclusions make sense in the program context, and whether alternative explanations should be considered.

This collaborative process does not weaken the findings by making them more complicated. It strengthens them by ensuring that the conclusions reflect the full body of available information.

Impact Requires More Than Technical Quality

A study can be methodologically sound and still have little influence. Findings become impactful when the people who need them can understand and use them.

Different team members often understand different audiences. Researchers may focus on methodological precision. Practitioners understand what can realistically be changed. Organizational leaders know which decisions must be made. Clients and participants can identify which findings matter most to the people affected by those decisions.

When these perspectives are incorporated throughout a project, the final report is more likely to answer useful questions, explain the findings clearly, and provide recommendations that can be implemented.

The Strength Is in the Combination

Multidisciplinary work does not mean that every team member performs every task. It means that each person contributes specialized knowledge while remaining engaged with the work of the larger team.

The strongest teams establish clear roles, communicate across disciplines, and create opportunities for assumptions and interpretations to be questioned. They value technical expertise without overlooking practical knowledge or lived experience.

Research and evaluation have the greatest impact when they do more than produce accurate findings. They must ask the right questions, examine the issue from multiple perspectives, and translate what is learned into meaningful action.

A multidisciplinary team makes that possible—not because more people automatically produce better work, but because the right combination

Program Evaluation Services for Organizations Across the United States

Effective program evaluation requires more than collecting data—it requires the right questions, appropriate methodologies, and a clear understanding of the people and communities a program serves. Research Analytics Consulting provides program evaluation services for organizations in Florida , New York , Arizona , Massachusetts , and throughout the United States. Our multidisciplinary approach combines research design, data analysis, measurement expertise, and stakeholder perspectives to help organizations understand program outcomes, identify areas for improvement, and make informed decisions based on meaningful evidence.

Contact Research Analytics Consulting today to discuss how our program evaluation services can support your organization's goals.

Can You Trust Your Results? Why Reliability and Validity Matter

Sat, 26 Sep 2026 00:13:53 GMT

Organizations use data to make important decisions every day. They use surveys to understand employees, clients, and program participants; assessments to evaluate professional knowledge; and research findings to decide whether an initiative should continue or expand.

But a polished report and a large dataset do not necessarily mean the conclusions are trustworthy. Before acting on the findings, organizations should ask two questions:

Are we measuring the right thing?

Are we measuring it consistently?

These practical questions are at the heart of validity and reliability.

Validity: Are We Measuring the Right Thing?

Validity concerns whether the available evidence supports the way results are interpreted and used. It is often described more simply as whether a measure captures what it is supposed to measure.

Consider an organization that offers a leadership-development program. At the end of the program, participants are asked whether they enjoyed the training, liked the facilitator, and would recommend the program to a colleague.

Those questions may provide useful information about participant satisfaction, but they do not measure leadership development. Participants could enjoy the program without improving their ability to resolve conflict, communicate expectations, provide constructive feedback, or lead a team.

If the organization interprets favorable satisfaction ratings as proof that leadership skills improved, it is drawing a conclusion the survey was not designed to support.

To measure leadership development, the organization would need questions or activities connected to the specific knowledge, skills, or behaviors the program was intended to strengthen. It might still measure participant satisfaction, but it should not treat satisfaction and learning as if they are the same outcome.

This distinction applies across social science research. A client survey may measure satisfaction with services without determining whether those services improved the client's circumstances. A professional assessment may emphasize memorization when the intended outcome is the ability to apply knowledge. A study of a workforce program may document employment without examining whether participants obtained stable jobs, adequate hours, or opportunities for advancement.

Validity begins with defining the intended outcome clearly and ensuring that the measure adequately represents it.

Reliability: Are We Measuring It Consistently?

Once we know that a measure addresses the intended outcome, we must ask whether it does so consistently.

Reliability refers to the consistency or dependability of measurement. Results should not fluctuate unpredictably because questions are confusing, instructions change, scoring criteria are applied differently, or the measure contains too much error.

Suppose two supervisors use the same rubric to evaluate an employee's performance but assign dramatically different scores to the same work. The difference may reflect unclear scoring criteria rather than a genuine difference in performance.

The same problem can occur in surveys and assessments. Participants may interpret vague questions differently. Items intended to measure the same concept may produce contradictory responses. Scores may vary substantially even when the underlying knowledge, attitude, or behavior has not changed.

Reliability does not mean that scores should never change. People learn, develop, and have new experiences. The goal is to ensure that changes in the results reflect genuine differences—not instability in the measure.

Why Consistency Alone Is Not Enough

A measure can be reliable without supporting a valid conclusion, but it cannot support a valid conclusion without being sufficiently reliable.

A bathroom scale provides a simple example. Imagine that the scale consistently reports a person's weight as five pounds heavier than it actually is. The result is dependable: the scale produces the same error every time. But interpreting that number as the person's actual weight would be incorrect.

Surveys and assessments can create the same problem. A series of questions may produce highly consistent responses while measuring only part of the intended outcome—or something different altogether.

An assessment intended to measure employees' ability to apply a new policy, for example, might consistently measure how well they memorized its wording. The scores could be reliable while failing to provide the information the organization actually needs.

Consistency matters, but consistent measurement of the wrong thing does not produce useful results.

Are the Results Appropriate for This Purpose?

Validity and reliability do not exist independently of context. A measure is not appropriate for every population, setting, and purpose simply because it performed well in one study.

A survey developed for senior executives may not function in the same way with direct-service staff. A measure developed for the general population may not be accessible to people with intellectual or developmental disabilities. An assessment designed to measure knowledge may not be suitable for evaluating performance in real-world situations.

The same measure may also be adequate for one decision but inadequate for another. A brief questionnaire might help identify topics for an upcoming training program while providing too little information to determine whether participants have mastered the content.

The evidence needed should reflect how the results will be used. The more consequential the decision, the more confidence an organization should require in the measure and its scores.

What Should Organizations Ask Before Acting?

Organizations do not need to become measurement experts, but they should ask informed questions before making decisions based on research findings:

What was this instrument designed to measure?
Do the questions adequately represent that outcome?
What evidence shows that the scores are sufficiently consistent?
Is the measure appropriate for these participants and this purpose?
Do the results support the conclusion we want to draw?

Reliability and validity are not simply technical terms to include in a research report. They determine whether the findings mean what an organization believes they mean—and whether those findings are dependable enough to guide action.

Before asking what the results say, first ask whether the results can be trusted.

Looking for Psychometric Support You Can Rely On?

Whether you're evaluating a program, reviewing an existing evaluation tool, or developing a reliable and valid measurement instrument, such as a survey or assessment , Research Analytics Consulting can help determine whether your measures are capturing the right outcomes and producing results you can rely on. If your organization is located in Florida , Arizona , Wisconsin , New York , or anywhere in the United States, contact our team to talk through your project and the decisions your data needs to support.

Why Measurement Matters in Program Evaluation

Fri, 11 Sep 2026 22:02:00 GMT

Good measurement gives organizations confidence in their evaluation findings. It helps them understand whether a program is achieving its intended outcomes, where improvements may be needed, and which decisions the results can reasonably support.

Before evaluators calculate percentages, compare groups, or examine changes over time, they must determine whether the measures used to collect the data are well designed. Statisticians often describe this principle with the expression “garbage in, garbage out.” The results of any statistical analysis are only as good as the information being analyzed.

Even sophisticated analyses cannot compensate for questions that are unclear, response choices that do not fit the question, or survey logic that sends participants through the wrong set of items. Measurement is not simply a preliminary step in an evaluation. It is the foundation on which the evaluation rests.

Turning Outcomes Into Measurable Information

Measurement is the process of translating an outcome into information that can be observed and analyzed.

Some program outcomes are relatively straightforward. Attendance, graduation, employment, and completion of a certification can often be documented directly. Other outcomes—such as confidence, leadership, financial literacy, workplace readiness, self-advocacy, and quality of life—are more complex.

Evaluators must define what each outcome means within the context of the program and determine how it can be measured accurately.

For example, a program may be designed to increase participants’ financial literacy. Asking participants whether they feel more knowledgeable provides information about their perceptions, but it does not necessarily demonstrate that they can create a budget, compare borrowing costs, recognize financial risks, or make sound financial decisions.

An evaluation may need to measure both perceived growth and demonstrated knowledge or skills. The important point is that the questions must represent the outcomes the program is intended to influence. Otherwise, the evaluation may report improvement without clearly demonstrating what participants learned or can now do.

A Survey Is Not Automatically a Good Measure

Online survey platforms have made it easy to create questionnaires and collect large amounts of data. However, the ability to administer a survey is not the same as the ability to design a sound measurement instrument.

Small problems in survey questions can substantially affect the answers participants provide. Questions may be vague, overly complicated, leading, or open to multiple interpretations. A single question may also ask about two different ideas.

Consider the statement:

“I understand financial planning and feel confident managing my money.”

A participant might understand financial planning but lack confidence. Another might feel confident despite having limited financial knowledge. Because the statement combines two different ideas, the evaluator cannot determine which idea the participant considered when answering.

Every survey question should have a clear purpose. Evaluators should ask whether the wording is understandable, whether the question measures one idea at a time, whether it is appropriate for the people completing the survey, and whether the answer will provide information connected to the evaluation questions.

Survey Logic Determines Which Questions Participants Answer

Survey logic controls how participants move through a questionnaire. It determines which questions they receive based on their previous answers.

For example, participants who indicate that they are employed may be asked follow-up questions about their hours, wages, job responsibilities, and workplace experiences. Participants who are not employed should bypass those questions and receive questions about job seeking, training, or barriers to employment.

When survey logic is designed correctly, participants receive questions that are relevant to their experiences. The survey takes less time to complete and is less likely to frustrate or confuse respondents.

Poorly constructed survey logic can create serious data-quality problems. Participants may be asked to describe services they never received or experiences they did not have. Others may inadvertently skip questions that are essential to the evaluation. If a response is required, participants may select an inaccurate answer simply so they can continue through the survey.

These problems can be difficult to identify after data collection has ended. The dataset may appear complete even though some responses were created by an incorrect path through the questionnaire.

Survey logic must therefore be planned, programmed, and tested carefully. Evaluators should test every possible path before administering the survey and confirm that participants receive the correct questions based on their responses.

Response Scales Shape the Meaning of the Answers

The response scale is just as important as the wording of the question. It provides the options participants use to communicate their answers and determines how those answers can be interpreted.

The scale must match what the question asks. Questions about frequency require response options such as “never,” “rarely,” “sometimes,” “often,” and “always.” Questions about confidence require choices ranging from low to high confidence. Questions about agreement require clearly ordered levels of agreement.

Problems arise when the question and the response scale do not align. Asking how often someone performs a behavior and then providing choices from “strongly disagree” to “strongly agree” forces the participant to translate a frequency judgment into an agreement judgment. Different participants may make that translation differently.

Several other decisions affect the quality of a response scale:

How many response options participants receive
Whether each option is clearly labeled
Whether a midpoint is appropriate
Whether participants need a “not applicable” or “don’t know” option
Whether the choices represent the full range of likely experiences
Whether similar questions use scales consistently

More choices do not automatically produce more precise data. Participants must be able to distinguish meaningfully among the options. A ten-point scale may create the appearance of precision without ensuring that respondents interpret the difference between a six and a seven in the same way.

The wording of the choices also matters. A scale that offers mostly positive options can steer participants toward favorable responses. A scale without a legitimate “not applicable” choice may force them to provide an answer that does not represent their experience.

Well-designed response scales make it easier for participants to answer accurately and for evaluators to interpret the results responsibly.

Measurement Quality Must Be Established Before Data Collection

Strong measurement requires more than writing a collection of reasonable-sounding questions. The outcomes must be clearly defined, the questions must align with those outcomes, the response scales must fit the questions, and the survey logic must function as intended.

Measures should also be reviewed and tested with people similar to those who will complete them. This process can reveal unclear wording, missing response choices, unexpected interpretations, and problems with the survey pathway before those issues affect the evaluation data.

Reliability and validity provide more formal ways to examine whether a measure produces sufficiently consistent information and supports its intended interpretations. Because these concepts deserve careful attention, they will be explored in the next article in this series.

Better Measurement Leads to Better Decisions

Organizations invest in evaluation because they want useful answers. They want to know whether programs are working, how they can be improved, which participants benefit, and where resources should be directed.

Those answers are only as credible as the measures on which they are based.

A large dataset does not compensate for weak measurement. Neither do sophisticated analyses, polished visualizations, or professionally written reports. Trustworthy conclusions begin with clearly defined outcomes, carefully written questions, appropriate response scales, and accurate survey logic.

Before asking what the data show, organizations should first ask whether they collected the right data—and whether those data are strong enough to support the decisions that follow.

Need Help With Program Evaluation and Measurement?

Research Analytics Consulting helps organizations strengthen their program evaluation through thoughtful measurement strategies, survey development, and data collection tools designed to produce meaningful results. Whether you need program evaluation support in Florida , Arizona , Wisconsin , New York or anywhere in the United States, our consultants can help you develop reliable measures that align with your program goals and provide useful information for decision-making. Contact Research Analytics Consulting to discuss your program evaluation or survey development needs.

Not All Program Evaluators Are Created Equal

Sat, 01 Aug 2026 01:25:46 GMT

What Sets a Rigorous Program Evaluator Apart?

Organizations increasingly recognize the importance of program evaluation for demonstrating accountability, informing decision-making, and supporting continuous quality improvement. Yet a common misconception persists: that program evaluation is simply a matter of collecting data, administering surveys, analyzing results, and producing a final report.

In reality, program evaluation is a professional discipline grounded in the scientific principles of research, measurement, and evaluation methodology. It draws upon evaluation theory, research design, educational psychology, psychometrics, quantitative and qualitative methods, implementation science, and project management to produce evidence that is credible, meaningful, and actionable. The quality of an evaluation depends not only on the data collected, but on the expertise of the professionals responsible for designing the evaluation, developing sound measures, interpreting findings, and translating evidence into recommendations that improve programs.

Every phase of an evaluation—from defining evaluation questions and selecting an appropriate design to developing valid measures, analyzing data, and interpreting findings—requires specialized expertise. Decisions made throughout this process influence the credibility of the evidence and, ultimately, the confidence that stakeholders can place in the conclusions drawn from the evaluation.

Rigorous Evaluation Begins with Sound Measurement

At the heart of every robust evaluation is the quality of its measurement. Whether evaluating changes in knowledge, skills, attitudes, behaviors, implementation fidelity, or program outcomes, the conclusions drawn from an evaluation are only as sound as the measures used to collect the evidence.

Developing high-quality measures is a specialized area of expertise grounded in educational and psychological measurement, commonly referred to as psychometrics . Reliable and valid measures provide the foundation for credible evidence and meaningful conclusions. Without sound measurement, even sophisticated analyses cannot compensate for flawed data.

Rigorous Evaluation Requires an Appropriate Research Design

Sound measurement alone is not enough. A rigorous evaluation also requires a research design that aligns with the evaluation questions, program context, and intended use of the findings. Selecting appropriate methodologies, sampling strategies, comparison groups (when appropriate) and data collection procedures ensures that the evidence generated is both credible and useful for decision-making.

Rigorous Evaluation Integrates Quantitative and Qualitative Evidence

Programs are complex, and no single source of evidence tells the complete story. Quantitative methods provide objective measures of outcomes and trends, while qualitative methods offer insight into participant experiences, implementation challenges, and contextual factors. Integrating both forms of evidence provides a more comprehensive understanding of program effectiveness and opportunities for improvement.

Rigorous Evaluation Benefits from Multidisciplinary Expertise

Modern program evaluation draws upon expertise from multiple disciplines, including evaluation methodology, educational psychology, psychometrics, statistics, qualitative research, implementation science, and project management. A multidisciplinary approach strengthens every phase of the evaluation process—from planning and instrument development to data interpretation and communicating findings to stakeholders.

Rigorous Evaluation Depends on Effective Project Management

Even the most technically sound evaluation can fall short without effective project management. Successful evaluations require thoughtful planning, clear communication, stakeholder engagement, adherence to timelines, and careful coordination of data collection and reporting activities. Strong project management ensures that rigorous methodology is translated into practical, actionable results.

Conclusion

Rigorous program evaluation is far more than the collection and analysis of data. It is a professional discipline grounded in scientific principles and informed by nationally recognized standards of practice. Organizations that understand the hallmarks of rigorous evaluation are better equipped to generate credible evidence, strengthen programs, and make informed decisions that lead to meaningful and lasting impact.

Organizations across the country rely on Research Analytics Consulting for scientifically rigorous program evaluation services . From Florida to Georgia , Texas , and beyond, our team helps organizations design evaluations, analyze evidence, and apply findings to improve programs and support informed decision-making.

About the Author

Cindy M. Walker, PhD, is the Founder and President of Research Analytics Consulting, a woman-owned small business specializing in program evaluation , educational measurement, psychometrics , survey and instrument development, and research methodology. A nationally recognized expert in educational psychology, measurement, and evaluation, Dr. Walker has more than 30 years of experience leading evaluation and research initiatives for federal agencies, state agencies, institutions of higher education, nonprofit organizations, and private-sector clients.

Dr. Walker is a former Dean and Professor of Educational Psychology whose career has focused on advancing the science and practice of program evaluation. She has held national leadership positions with the American Educational Research Association and the National Council on Measurement in Education and has published extensively in the areas of educational measurement, psychometrics, research methodology, and program evaluation. Through Research Analytics Consulting, she leads a multidisciplinary team dedicated to producing scientifically rigorous evaluations that generate credible evidence and support meaningful organizational improvement.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.testingstandards.net

American Evaluation Association. (2018). AEA evaluator competencies. https://www.eval.org/About/Competencies-Standards

Centers for Disease Control and Prevention. (2024). CDC framework for program evaluation in public health. U.S. Department of Health and Human Services. https://www.cdc.gov/evaluation/framework/index.html

Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage Publications.

Patton, M. Q. (2022). Qualitative research & evaluation methods: Integrating theory and practice (5th ed.). Sage Publications.

Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed.). Sage Publications.

What Drives Assessment Reliability? Findings from a Large-Scale Assessment Analysis

Mon, 13 Jul 2026 19:53:02 GMT

Assessment teams often ask a familiar question: What can we do to improve reliability?

The traditional answer is straightforward—add more items. While test length is known to influence reliability, assessment programs frequently face practical constraints such as limited seat time, development costs, and learner fatigue.

To better understand what drives reliability in operational assessments, our team conducted a research study examining more than 80 assessments administered within a professional learning environment.

Our goal was to identify which assessment characteristics were most strongly associated with reliability and to explore how quantitative and qualitative indicators of assessment quality work together.

The Study

For each assessment, we collected a variety of psychometric and content-related indicators, including:

Reliability (Cronbach's alpha)
Average item discrimination
Percentage of items with low discrimination
Average item difficulty
Average guessing parameter
Number of items on the assessment
Score variability
Qualitative item review scores
Differential item functioning (DIF) indicators

The qualitative review process evaluated items against established item-writing principles, including clarity of the stem, response option quality, ambiguity, parallelism, clueing, and other characteristics that can affect the testing experience.

We then conducted correlation and regression analyses to identify which variables were most strongly related to reliability.

What We Found

Several expected patterns emerged.

Reliability increased as assessments contained more items and as score variability increased. These findings are consistent with well-established psychometric theory.

However, one result stood out above all others.

Item Discrimination Was the Strongest Predictor of Reliability

Average item discrimination demonstrated an exceptionally strong relationship with reliability.

Across the assessments included in the study, the correlation between average item discrimination and reliability was approximately 0.93.

To further investigate this relationship, we fit a regression model using average item discrimination as the sole predictor of reliability.

The results showed that average item discrimination alone explained approximately 86% of the variability in reliability across assessments.

In practical terms, assessments containing items that effectively differentiate between more knowledgeable and less knowledgeable learners were substantially more likely to produce reliable scores.

Test Length Remains Important

We then expanded the model to include both average item discrimination and the total number of items on the assessment.

Together, these two variables explained approximately 97% of the variability in reliability.

This finding reinforces two important principles:

Longer assessments generally produce higher reliability.
Item quality, as reflected by discrimination statistics, remains critically important even after accounting for assessment length.

What About Qualitative Item Reviews?

While statistical indicators emerged as the strongest predictors of reliability in our analysis, the findings also reinforced the important role of qualitative item reviews in the assessment development process.

Our review framework evaluated items against established item-writing principles, including clarity of the stem, response option quality, ambiguity, parallelism, unnecessary complexity, and potential clues to the correct answer. These reviews provide valuable evidence that cannot be obtained from psychometric statistics alone.

Reliability tells us how consistently an assessment measures performance. Qualitative reviews help ensure that items are clear, fair, and aligned with the intended learning objectives. In other words, statistical analyses help us understand how well items perform, while qualitative reviews help us understand why they may perform the way they do.

Although qualitative review scores were not among the strongest predictors of reliability in this study, they remain a critical component of a comprehensive quality assurance process. By identifying potential item-writing issues before administration, qualitative reviews help support content validity, improve the learner experience, and strengthen the overall defensibility of assessment results.

The findings suggest that the most effective assessment programs leverage both approaches. Psychometric analyses provide evidence of item and test performance, while qualitative reviews provide expert insight into content quality and opportunities for improvement. Together, these complementary sources of evidence support the development of assessments that are both technically sound and instructionally meaningful.

Implications for Assessment Programs

The findings suggest several practical recommendations for assessment programs:

Monitor item discrimination statistics routinely.
Revise or replace items that consistently show weak discrimination.
Continue conducting structured qualitative item reviews.
Use statistical evidence and content review findings together when making assessment decisions.
Consider item quality alongside test length when evaluating reliability.

Most importantly, organizations should recognize that reliability is only one dimension of assessment quality. High-quality assessments require both strong psychometric performance and sound content design.

What These Findings Mean for Assessment Programs

Our analysis demonstrated that item discrimination is one of the strongest drivers of assessment reliability, while assessment length continues to play an important supporting role.

At the same time, the study reinforced the value of qualitative item reviews. Although qualitative review results were not the strongest predictors of reliability, they provide essential evidence about clarity, fairness, and content quality that cannot be captured through statistics alone.

The most effective assessment programs do not rely on a single source of evidence. They combine expert review, psychometric analysis, and continuous improvement to build assessments that are both reliable and defensible.

Data Security and Privacy in Modern Assessments

Wed, 29 Apr 2026 02:07:47 GMT

How organizations can navigate the evolving landscape of standards, authentication, and data protection in high-stakes testing environments:

Whether it's a certification exam for healthcare professionals, a corporate compliance assessment, a statewide student evaluation, or a research survey collecting sensitive demographic information, modern assessments generate and depend on data that demands serious protection. Test content represents significant intellectual property. Examinee records often contain personally identifiable information; and in many cases, that information belongs to minors, pertains to employment consequences, or intersects with health-related contexts.

Yet in our work across corporate, government, and educational assessment programs, we consistently find that data security and privacy are treated as afterthoughts or bolt-on concerns addressed late in the development cycle rather than principles embedded from the start. This post outlines the core security fundamentals that assessment programs should address, surveys the standards landscape organizations must navigate, and highlights recent shifts in authentication guidance that present both an opportunity and a challenge.

The Fundamentals: What Assessment Data Security Actually Requires

At its foundation, protecting assessment data involves four interrelated practices.

Encryption in transit and at rest. Any assessment platform transmitting examinee responses, scores, or personal information over a network should use TLS (Transport Layer Security) to protect data in transit. Equally important, stored data — whether in a cloud database, a file server, or a backup archive — should be encrypted using robust standards such as AES-256. This is not merely a best practice; it is a baseline expectation. Assessment content is high-value intellectual property, and a breach of live test items can invalidate an entire exam form, resulting in costs that extend far beyond the data itself.

Authentication and access rights management. Assessment programs involve a range of roles with very different data needs: test developers, psychometricians , proctors, administrators, and examinees. A well-designed system enforces role-based access control (RBAC) so that each user can access only the data and functions relevant to their role. A proctor, for instance, needs to verify examinee identity and manage session logistics but should never have access to

item-level scoring algorithms or raw psychometric data. Applying the principle of least privilege reduces the surface area available to both external attackers and inadvertent internal mishandling.

Data minimization. Not all data that can be collected should be collected. Organizations should clearly distinguish between data necessary for scoring and reporting, data required for psychometric research (item analysis, differential item functioning studies, norming, equating), and data that is simply convenient to have. Collecting more than what is needed increases both regulatory exposure and the potential impact of a breach. This distinction matters especially when assessments involve minors or when government-administered programs impose specific consent and transparency requirements.

Anonymization and de-identification for research. Much of the analytic work that follows an assessment — item calibration, bias analysis, reliability studies — can and often should be conducted on de-identified datasets. Techniques such as pseudonymization and aggregation thresholds allow psychometricians to perform rigorous analysis without retaining linkages to individual examinees. The key tension here is between analytic granularity and re-identification risk: subgroup analyses that are fine-grained enough to detect bias may also be fine-grained enough to identify individuals in small populations. Responsible programs address this tension explicitly rather than assuming that removing names and ID numbers is sufficient.

The Standards Landscape: A Patchwork With Real Consequences

Which standards apply to a given assessment program depends heavily on the type of organization, the industry it serves, and the populations it assesses. This is where things get complicated, and where we frequently see organizations struggling.

SOC 2 Type 2 audits evaluate an organization's controls against the Trust Service Criteria for security, availability, processing integrity, confidentiality, and privacy. For assessment platforms operating as SaaS products, which is increasingly the norm, a SOC 2 Type 2 report has become a de facto prerequisite for enterprise procurement. It demonstrates not just that controls exist on paper, but that they have been tested and verified over a sustained period.

HITRUST and HITECH become relevant where assessments intersect with healthcare: continuing medical education, clinical competency evaluations, employee wellness surveys, or any context where health-related data may be collected. The HITRUST Common Security Framework (CSF) is notable because it attempts to map and harmonize requirements across multiple regulations, but achieving and maintaining HITRUST certification is a substantial undertaking.

FERPA (the Family Educational Rights and Privacy Act) governs educational records in institutions receiving federal funding, making it directly relevant to K–12 and higher education assessment programs. FERPA imposes specific requirements around consent, access, and disclosure that shape how student assessment data can be stored, shared, and used for research.

Emerging state and international privacy laws — CCPA, GDPR, and a growing patchwork of state-level regulations — add further layers, particularly for assessment programs that operate across jurisdictions or involve international examinee populations.

In practice, these overlapping frameworks frequently lead to a fragmented implementation landscape. Organizations operating under multiple standards simultaneously may find that compliance requirements conflict, create redundant controls that add friction without proportionate security benefit, or encourage a "checkbox compliance" mentality where policies exist on paper but do not translate into genuinely robust security practices. We regularly encounter systems where encryption is applied inconsistently across legacy and modern components, where access controls are nominally in place but practically unenforced, or where data retention policies exist but are never audited.

The practical challenge is implementing the applicable standards coherently across an organization's actual technology stack and operational workflows.

Modern Authentication: What NIST's Updated Guidance Means for Assessments

One area where the gap between current best practice and actual implementation is especially visible is authentication management. NIST's Special Publication 800-63B , the authoritative federal guideline for digital authentication, underwent a significant revision with Revision 4, finalized in July 2025. The changes are worth understanding because they directly contradict policies still in place at many organizations.

The headline shifts include the elimination of mandatory periodic password rotation — a practice that NIST's own research found leads to weaker passwords as users resort to predictable incremental changes. Revision 4 now states that organizations should not require password changes unless there is evidence of compromise. The updated guidance also explicitly prohibits arbitrary complexity composition rules (requiring special characters, mixed case, etc.), instead emphasizing password length as the primary factor in strength and mandating that passwords be screened against databases of known compromised credentials.

Critically for assessment platforms, the revised standard requires that systems allow the use of password managers and autofill functionality, and recommends supporting paste functionality in password fields. Multi-factor authentication is strongly encouraged, with an emphasis on phishing-resistant methods. Time-based one-time password (TOTP) apps, such as Google Authenticator or Microsoft Authenticator, and single sign-on (SSO) integration represent the practical implementation of these recommendations for most organizations.

This matters for assessment programs because they often serve user populations with widely varying technical sophistication: a corporate employee taking a compliance quiz, a 16-year-old sitting for a state exam, a nurse completing a continuing education module. Legacy assessment platforms frequently still enforce the very practices NIST now considers counterproductive: forced 90-day password resets, complexity rules that encourage "Password1!" patterns, and disabled paste functionality that actively undermines password manager use.

We see opportunities for improvement here. Implementing SSO and modern MFA on assessment platforms can simultaneously improve security and reduce friction for examinees — fewer login barriers, fewer forgotten-password support tickets, and stronger protection against credential-based attacks. But many organizations' internal security policies have not yet caught up with the current NIST recommendations, and vendor platforms — particularly those serving regulated industries where older compliance checklists remain in effect — may lag as well.

A Note on Paper-Based Assessments

It is worth briefly acknowledging that paper-based assessment forms have seen a resurgence in some educational settings, driven by concerns about the integrity of electronic testing, equity of device access, and screen fatigue among young learners. However, returning to paper does not eliminate data security concerns, instead it transforms them. Chain-of-custody protocols, secure physical storage and destruction, and controlled data-entry processes for digitization all introduce their own risks and require their own discipline. Organizations that move assessment modalities should ensure their security planning follows.

Building Security Into the Assessment Lifecycle

The common thread across each of these areas is that data security and privacy work best when they are built into the assessment design process from the beginning rather than retrofitted after a platform has been selected or a program is already operational. This means asking the right questions early: What data do we actually need? Who will have access, and under what controls? Which standards apply to our specific context, and how do we implement them as a coherent system rather than a collection of disconnected compliance checkboxes?

Is Your Assessment Program's Data Security Up to Standard?

These are the kinds of questions that benefit from experience working across sectors and experiencing how the same frameworks play out differently in a corporate training environment than in a state education agency or a clinical credentialing body. At Research Analytics Consulting, this cross-sector perspective informs how we approach assessment design, platform evaluation, and data governance planning for our clients. If your organization is developing, procuring, or modernizing an assessment program, we welcome the conversation. Schedule a consultation today .

Quality of Care for IDD Adults Living in Florida: A Pilot Study

Thu, 13 Nov 2025 03:47:00 GMT

When Caring Isn’t Enough: Rethinking “Quality of Care” for Adults with IDD

By Cindy M. Walker, PhD
Founder & CEO, Research Analytics Consulting
Co-authored by Jacqueline Gosz, MS and Sue Gottesman, MBA

When I agreed to help design a survey on the quality of care for adults with intellectual and developmental disabilities (IDD) in Florida, it wasn’t a business decision — it was a personal one.

As a Guardian Advocate and Representative Payee for a 46-year-old woman with an IDD — someone I love like family — I’ve seen firsthand both the dedication and the limitations within our system of support. She lives independently now, with a live-in caregiver, a coach, and family who care deeply for her. But like so many others, she depends on a network that is good-hearted yet fragmented — one that often provides comfort, but not always growth. This pilot study, which gathered the voices of 157 adults with IDD and their families across Florida, was an effort to listen — truly listen — to how those receiving care experience their lives.

What We Heard

Respondents were asked about their caregivers, job coaches, roommates, Supported Living Coaches, and Adult Day Training programs — the web of supports that sustain everyday life for adults with IDD.

When we analyzed the data, a clear pattern emerged:

● The highest scores reflected emotional care — feeling loved, respected, and supported.
● The lowest scores reflected growth — learning new things, gaining independence, becoming more self-sufficient.

In other words, people feel cared for, but they aren’t always learning how to care for themselves.

As one family shared, “We love the staff — but my son wants to work, not just go to a day program. No one helps him with this.”

This is a crucial finding. It suggests that while our systems of care succeed in meeting emotional needs, they often fall short in nurturing autonomy — the very foundation of empowerment and dignity.

A System of Comfort, Not Empowerment

Across the data, one theme echoes: adults with IDD are surrounded by care, but rarely by opportunity. Only 17% of respondents reported working, and less than 25% worked with a Supported Living Coach — the very professionals meant to teach independence.

Many programs focus on safety, consistency, and affection — all vital. But without a parallel focus on skill-building and choice, we risk creating a system of learned helplessness rather than one of self-determination.

As researchers, we see the numbers.
As families, we feel the consequences.

Where We Go From Here

This pilot study doesn’t claim to represent every community or every story. But it raises an urgent question for all of us — families, providers, policymakers, and advocates alike:

Are we helping adults with IDD live independently, or simply keeping them comfortable?

The answer calls for a renewed vision — one that values both care and capability. A system where teaching life skills, encouraging employment, and honoring choice are not add-ons but essentials.

Because true “quality of care” isn’t only about feeling cared for — it’s about becoming capable of caring for oneself.

At Research Analytics Consulting, we believe data should reveal not just what is measurable, but what is meaningful.

This report — Quality of Care for IDD Adults Living in Florida: A Pilot Study — is both a mirror and a call to action.

Read the full report here

The Content, Pedagogy, Implementation, and Context Components (CPIC) Study

Mon, 10 Nov 2025 02:49:41 GMT

Challenge

The Content, Pedagogy, Implementation, and Context Components (CPIC) Study aims to better understand Evidence Based Programs (EBP) for Teen Pregnancy Prevention (TPP). Our challenge is to determine the relationship between the intended and the implemented core components using empirical data, collected over multiple years of APP evaluation work conducted by AMTC & Associates; and, to determine which implemented core components are most essential for producing desired outcomes.

Solution

In Phase I: assemble a cleaned, merged data set comprising 11 years of survey, implementation, and attendance data for TPP programs utilizing 6 curricula. In the upcoming Phase II: propose hypotheses and provide categorical data analysis.

Impact

Overall, this study will aid implementers in adapting program components for target populations, researchers in testing the effects of individual components on participant outcomes, and policymakers and funders in identifying and prioritizing interventions with promising components. Furthermore, this study will provide TPP researchers with multiple examples of how to conduct evaluation studies that focus on the core components of the program implemented.

Methodology

We extracted curriculum information from AMTC’s implementation database (OPTS) to aid in the qualitative analysis of implemented core components. We linked the OPTS database to a database, obtained through local evaluation efforts, that consists of responses to pre- and post-surveys, from youth that participated in the implementation of the six EBPs studied in Phase I. This newly created database will be used to determine the relationship between various core components and the intended outcomes of program participation.

Results

We found sufficient data to analyze five of the six curricula. We detailed multiple versions of each curriculum to help identify core components for delivered classes. We assembled disparate survey data sets measuring outcomes across programs spanning 11 years and matched participants with their corresponding attendance records.

Tools Used:

Python and SQL for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, knitr, dplyr, psych, caret).

A Psychometric Approach to Accreditation

Sat, 01 Feb 2025 03:14:02 GMT

Challenge

One of the big five accounting firms provides continuing education, with the requisite CPE credit, to their professionals. Their programs are all accredited by the AACSB and are large-scale high-stakes assessments. As such, the item and test analysis must be comprehensive, precise and psychometrically rigorous. Additionally, annual reports must be provided for continuous quality improvement of the assessments associated with courses. Previous psychometric analysis for the reports only used Classical Test Theory (CTT) approaches. Even though high reliability of these assessments is desired for consistent measurement of learner knowledge and ability, leveraging only CTT can result in unstable statistical estimates, because the statistics associated with this method are sample and test dependent. Modern test theory utilizing item response theory (IRT) is a more appropriate and rigorous approach.

Solution

In this project we have evaluated over 100 learning assessments, used to measure learning outcomes of continuing education courses for accounting professionals using both CTT and IRT to optimize their learning assessments. This allows us to quantify item performance on each of the final assessments using psychometric metrics. We have helped to identify opportunities for improvement while ensuring compliance with ongoing board certification requirements. These quantitative analyses are followed by expert qualitative analyses to recommend next steps for achieving more reliable and valid measurement of knowledge of learning objectives.

Impact

The quantitative analyses are used to identify problematic items that may be improved to more validly measure learner ability with respect to the intended learning objectives. Using the results of the qualitative analysis, item writers are able to further adapt future assessments to improve reliability and better ascertain learner proficiency. Items identified as problematic can also be opportunities to clarify educational content presented to learners.

Methodology

We used IRT, CTT, DIF analyses, and modern data visualizations to quantify test item performance and identify opportunities for improvement of assessment. This included the full scope of the item from the question to the response options, with visualization of the various item characteristic curves to ascertain how items were performing across examinees. Once items were flagged as problematic, a qualitative review was conducted, using a standardized approach to make recommendations to item writers with respect to how to write items of higher quality.

Results

This is a year-to-year ongoing project. In year one we worked to evaluate the test and assessment process and write the initial base code so the results are accurate and repeatable. In year two we repeated the analysis and worked to semi-automate the process due to the large number of assessments. We have begun working on qualitative psychometric aspects around item writing, which includes rigorous evaluation of the assessment constructs in the context of the goals and objectives. This ensures the items are measuring what is intended and helps to increase the statistical validity and reliability. In the next several years, we will continue automation utilizing machine learning and artificial intelligence. We will also explore predictive validity to ensure the assessments are useful for important outcomes.

Tools Used:

Python for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, dplyr, CTT, difR, psych, car, flextable, knitr), FlexMIRT and R (mirt, ggmirt) for IRT analysis.

The George W. Bush Institute Case Study

Fri, 31 Jan 2025 01:13:28 GMT

Challenge

Determine the impact of the School Leadership Initiative (SLI) on principals’ perceptions of several key variables of interest to the GWBI.

Solution

Clean and organize data collected by the GWBI from school districts and survey participants, combine with publicly available data, and perform a variety of statistical analyses to answer poignant research questions posed by the GWBI.

Impact

Retention of leadership talent and facilitation of their best performance strengthens educational opportunities for district children. The data collected by the GWBI is now supporting future efforts like the SLI.

Methodology

Research Analytics Consulting, LLC (RAC) embarked upon a pilot study to determine the impact of the George W. Bush Institute’s (GWBI) School Leadership Initiative (SLI) on principals’ perceptions of several key variables of interest to the GWBI. We constructed four variables based on an Exploratory Factor Analysis, and quantified the improvement in these variables associated with participation in the SLI. Then we identified the impact of each of these variables on school principal retention within the Fort Worth district.

Based on the results of the pilot study, RAC was contracted to conduct an expanded study using eight additional variables hypothesized to be impacted by the SLI, while focusing on three school districts that participated in the GWBI SLI from 2018 through 2021. We conducted reliability analysis, factor analysis and multilevel modeling.

Results

In the pilot study, four variables were uncovered fitting a factor analysis model to the SLI Principal Survey. There was some level of disagreement found on four of the 12 identical survey items that were administered to both district level personnel and principals. In most cases, district level personnel perceived district-level practices more positively than principals perceived. Approximately 12% of the variability in STARR Reading Achievement scores, for 2019 fifth grade students in the Fort Worth School District was explained by Job Satisfaction and School Climate. Fitting the same model, across all grade levels, yielded comparable results and explained 15% of the variability in STARR Reading Test Scores.

In the broader subsequent analysis, principal scores tended to increase over time, which implies that principals’ perceptions about their jobs and district level processes and procedures improved over time, as the GWBI SLI project matured. Job Satisfaction, School Climate, Compensation and Incentives, Working Environment, Job Embeddedness, and School Culture were found to be statistically significant as predictors of principal attrition.

Testing for a mediating relationship suggested that if principals feel more embedded in their jobs then they are more likely to stay in the district, even if they are not that satisfied with their compensation and incentives, as measured by level of perceived competitiveness of pay and non-monetary compensation for principals within the district.

Tools Used:

Python for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, knitr, dplyr, psych, caret, mlbench).

Determining the Impact of a Unique Public Health Initiative

Sun, 05 May 2024 02:56:08 GMT

Challenge

According to the Centers for Disease Control many young persons in the United States engage in risky sexual behavior and experience negative reproductive health outcomes NJPAG has created a curriculum for youth to help them avoid risk and make healthy choices using a three-pronged approach: the right information, a realistic application to one’s life, and sincere encouragement and support. In other words, NJPAG educators strive to give young people the facts and help them believe that they can make difficult, yet wise, choices in their own best interest. Their curriculum is currently used in over 250 schools throughout New Jersey and 8 additional states. They were recently awarded a PREIS grant to conduct a Randomized Controlled Trial (RCT) to determine the impact of their curriculum.

Solution

In this project we work closely with NJPAG as their external evaluator. We have created all of the processes and procedures associated with data collection and analysis. We have conducted site visits to ensure these processes and procedures are implemented with fidelity. Specifically, we are responsible for registering the study and obtaining IRB approval, creating valid and reliable measures, developing scripts for data collection processes, conducting power analyses and tests of baseline equivalence, and tracking attrition of our treatment and control groups to ensure we do not have differential attrition. The goal of this study is to see if the NJPAG curriculum can be added to the evidence based list for promising programs for reducing sexual risk.

Impact

The impact of this study is that if found successful the curriculum will be included on the evidence abased list.

Methodology

We are using standard methods, associated with evaluation; however, as this study is funded by the Department of Health and Human Services (DHHS) we are held to the highest standards of rigor. The results of this study will only be reviewed for inclusion on the evidence based list if we have adequate power, baseline equivalence, and the lack of differential attrition. We are only in the first year of the actua study. However, preliminary analyses have suggested that our study does meet these criteria.

Results

This is an ongoing project.

Tools Used:

This is an ongoing project.