Research Analytics Consulting, LLC

Not All Program Evaluators Are Created Equal

Sat, 01 Aug 2026 01:25:46 GMT

What Sets a Rigorous Program Evaluator Apart?

Organizations increasingly recognize the importance of program evaluation for demonstrating accountability, informing decision-making, and supporting continuous quality improvement. Yet a common misconception persists: that program evaluation is simply a matter of collecting data, administering surveys, analyzing results, and producing a final report.

In reality, program evaluation is a professional discipline grounded in the scientific principles of research, measurement, and evaluation methodology. It draws upon evaluation theory, research design, educational psychology, psychometrics, quantitative and qualitative methods, implementation science, and project management to produce evidence that is credible, meaningful, and actionable. The quality of an evaluation depends not only on the data collected, but on the expertise of the professionals responsible for designing the evaluation, developing sound measures, interpreting findings, and translating evidence into recommendations that improve programs.

Every phase of an evaluation—from defining evaluation questions and selecting an appropriate design to developing valid measures, analyzing data, and interpreting findings—requires specialized expertise. Decisions made throughout this process influence the credibility of the evidence and, ultimately, the confidence that stakeholders can place in the conclusions drawn from the evaluation.

Rigorous Evaluation Begins with Sound Measurement

At the heart of every robust evaluation is the quality of its measurement. Whether evaluating changes in knowledge, skills, attitudes, behaviors, implementation fidelity, or program outcomes, the conclusions drawn from an evaluation are only as sound as the measures used to collect the evidence.

Developing high-quality measures is a specialized area of expertise grounded in educational and psychological measurement, commonly referred to as psychometrics . Reliable and valid measures provide the foundation for credible evidence and meaningful conclusions. Without sound measurement, even sophisticated analyses cannot compensate for flawed data.

Rigorous Evaluation Requires an Appropriate Research Design

Sound measurement alone is not enough. A rigorous evaluation also requires a research design that aligns with the evaluation questions, program context, and intended use of the findings. Selecting appropriate methodologies, sampling strategies, comparison groups (when appropriate) and data collection procedures ensures that the evidence generated is both credible and useful for decision-making.

Rigorous Evaluation Integrates Quantitative and Qualitative Evidence

Programs are complex, and no single source of evidence tells the complete story. Quantitative methods provide objective measures of outcomes and trends, while qualitative methods offer insight into participant experiences, implementation challenges, and contextual factors. Integrating both forms of evidence provides a more comprehensive understanding of program effectiveness and opportunities for improvement.

Rigorous Evaluation Benefits from Multidisciplinary Expertise

Modern program evaluation draws upon expertise from multiple disciplines, including evaluation methodology, educational psychology, psychometrics, statistics, qualitative research, implementation science, and project management. A multidisciplinary approach strengthens every phase of the evaluation process—from planning and instrument development to data interpretation and communicating findings to stakeholders.

Rigorous Evaluation Depends on Effective Project Management

Even the most technically sound evaluation can fall short without effective project management. Successful evaluations require thoughtful planning, clear communication, stakeholder engagement, adherence to timelines, and careful coordination of data collection and reporting activities. Strong project management ensures that rigorous methodology is translated into practical, actionable results.

Conclusion

Rigorous program evaluation is far more than the collection and analysis of data. It is a professional discipline grounded in scientific principles and informed by nationally recognized standards of practice. Organizations that understand the hallmarks of rigorous evaluation are better equipped to generate credible evidence, strengthen programs, and make informed decisions that lead to meaningful and lasting impact.

Organizations across the country rely on Research Analytics Consulting for scientifically rigorous program evaluation services . From Florida to Georgia , Texas , and beyond, our team helps organizations design evaluations, analyze evidence, and apply findings to improve programs and support informed decision-making.

About the Author

Cindy M. Walker, PhD, is the Founder and President of Research Analytics Consulting, a woman-owned small business specializing in program evaluation, educational measurement, psychometrics, survey and instrument development, and research methodology. A nationally recognized expert in educational psychology, measurement, and evaluation, Dr. Walker has more than 30 years of experience leading evaluation and research initiatives for federal agencies, state agencies, institutions of higher education, nonprofit organizations, and private-sector clients.

Dr. Walker is a former Dean and Professor of Educational Psychology whose career has focused on advancing the science and practice of program evaluation. She has held national leadership positions with the American Educational Research Association and the National Council on Measurement in Education and has published extensively in the areas of educational measurement, psychometrics, research methodology, and program evaluation. Through Research Analytics Consulting, she leads a multidisciplinary team dedicated to producing scientifically rigorous evaluations that generate credible evidence and support meaningful organizational improvement.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.testingstandards.net

American Evaluation Association. (2018). AEA evaluator competencies. https://www.eval.org/About/Competencies-Standards

Centers for Disease Control and Prevention. (2024). CDC framework for program evaluation in public health. U.S. Department of Health and Human Services. https://www.cdc.gov/evaluation/framework/index.html

Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage Publications.

Patton, M. Q. (2022). Qualitative research & evaluation methods: Integrating theory and practice (5th ed.). Sage Publications.

Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed.). Sage Publications.

What Drives Assessment Reliability? Findings from a Large-Scale Assessment Analysis

Mon, 13 Jul 2026 19:53:02 GMT

Assessment teams often ask a familiar question: What can we do to improve reliability?

The traditional answer is straightforward—add more items. While test length is known to influence reliability, assessment programs frequently face practical constraints such as limited seat time, development costs, and learner fatigue.

To better understand what drives reliability in operational assessments, our team conducted a research study examining more than 80 assessments administered within a professional learning environment.

Our goal was to identify which assessment characteristics were most strongly associated with reliability and to explore how quantitative and qualitative indicators of assessment quality work together.

The Study

For each assessment, we collected a variety of psychometric and content-related indicators, including:

Reliability (Cronbach's alpha)
Average item discrimination
Percentage of items with low discrimination
Average item difficulty
Average guessing parameter
Number of items on the assessment
Score variability
Qualitative item review scores
Differential item functioning (DIF) indicators

The qualitative review process evaluated items against established item-writing principles, including clarity of the stem, response option quality, ambiguity, parallelism, clueing, and other characteristics that can affect the testing experience.

We then conducted correlation and regression analyses to identify which variables were most strongly related to reliability.

What We Found

Several expected patterns emerged.

Reliability increased as assessments contained more items and as score variability increased. These findings are consistent with well-established psychometric theory.

However, one result stood out above all others.

Item Discrimination Was the Strongest Predictor of Reliability

Average item discrimination demonstrated an exceptionally strong relationship with reliability.

Across the assessments included in the study, the correlation between average item discrimination and reliability was approximately 0.93.

To further investigate this relationship, we fit a regression model using average item discrimination as the sole predictor of reliability.

The results showed that average item discrimination alone explained approximately 86% of the variability in reliability across assessments.

In practical terms, assessments containing items that effectively differentiate between more knowledgeable and less knowledgeable learners were substantially more likely to produce reliable scores.

Test Length Remains Important

We then expanded the model to include both average item discrimination and the total number of items on the assessment.

Together, these two variables explained approximately 97% of the variability in reliability.

This finding reinforces two important principles:

Longer assessments generally produce higher reliability.
Item quality, as reflected by discrimination statistics, remains critically important even after accounting for assessment length.

What About Qualitative Item Reviews?

While statistical indicators emerged as the strongest predictors of reliability in our analysis, the findings also reinforced the important role of qualitative item reviews in the assessment development process.

Our review framework evaluated items against established item-writing principles, including clarity of the stem, response option quality, ambiguity, parallelism, unnecessary complexity, and potential clues to the correct answer. These reviews provide valuable evidence that cannot be obtained from psychometric statistics alone.

Reliability tells us how consistently an assessment measures performance. Qualitative reviews help ensure that items are clear, fair, and aligned with the intended learning objectives. In other words, statistical analyses help us understand how well items perform, while qualitative reviews help us understand why they may perform the way they do.

Although qualitative review scores were not among the strongest predictors of reliability in this study, they remain a critical component of a comprehensive quality assurance process. By identifying potential item-writing issues before administration, qualitative reviews help support content validity, improve the learner experience, and strengthen the overall defensibility of assessment results.

The findings suggest that the most effective assessment programs leverage both approaches. Psychometric analyses provide evidence of item and test performance, while qualitative reviews provide expert insight into content quality and opportunities for improvement. Together, these complementary sources of evidence support the development of assessments that are both technically sound and instructionally meaningful.

Implications for Assessment Programs

The findings suggest several practical recommendations for assessment programs:

Monitor item discrimination statistics routinely.
Revise or replace items that consistently show weak discrimination.
Continue conducting structured qualitative item reviews.
Use statistical evidence and content review findings together when making assessment decisions.
Consider item quality alongside test length when evaluating reliability.

Most importantly, organizations should recognize that reliability is only one dimension of assessment quality. High-quality assessments require both strong psychometric performance and sound content design.

What These Findings Mean for Assessment Programs

Our analysis demonstrated that item discrimination is one of the strongest drivers of assessment reliability, while assessment length continues to play an important supporting role.

At the same time, the study reinforced the value of qualitative item reviews. Although qualitative review results were not the strongest predictors of reliability, they provide essential evidence about clarity, fairness, and content quality that cannot be captured through statistics alone.

The most effective assessment programs do not rely on a single source of evidence. They combine expert review, psychometric analysis, and continuous improvement to build assessments that are both reliable and defensible.

Data Security and Privacy in Modern Assessments

Wed, 29 Apr 2026 02:07:47 GMT

How organizations can navigate the evolving landscape of standards, authentication, and data protection in high-stakes testing environments:

Whether it's a certification exam for healthcare professionals, a corporate compliance assessment, a statewide student evaluation, or a research survey collecting sensitive demographic information, modern assessments generate and depend on data that demands serious protection. Test content represents significant intellectual property. Examinee records often contain personally identifiable information; and in many cases, that information belongs to minors, pertains to employment consequences, or intersects with health-related contexts.

Yet in our work across corporate, government, and educational assessment programs, we consistently find that data security and privacy are treated as afterthoughts or bolt-on concerns addressed late in the development cycle rather than principles embedded from the start. This post outlines the core security fundamentals that assessment programs should address, surveys the standards landscape organizations must navigate, and highlights recent shifts in authentication guidance that present both an opportunity and a challenge.

The Fundamentals: What Assessment Data Security Actually Requires

At its foundation, protecting assessment data involves four interrelated practices.

Encryption in transit and at rest. Any assessment platform transmitting examinee responses, scores, or personal information over a network should use TLS (Transport Layer Security) to protect data in transit. Equally important, stored data — whether in a cloud database, a file server, or a backup archive — should be encrypted using robust standards such as AES-256. This is not merely a best practice; it is a baseline expectation. Assessment content is high-value intellectual property, and a breach of live test items can invalidate an entire exam form, resulting in costs that extend far beyond the data itself.

Authentication and access rights management. Assessment programs involve a range of roles with very different data needs: test developers, psychometricians , proctors, administrators, and examinees. A well-designed system enforces role-based access control (RBAC) so that each user can access only the data and functions relevant to their role. A proctor, for instance, needs to verify examinee identity and manage session logistics but should never have access to

item-level scoring algorithms or raw psychometric data. Applying the principle of least privilege reduces the surface area available to both external attackers and inadvertent internal mishandling.

Data minimization. Not all data that can be collected should be collected. Organizations should clearly distinguish between data necessary for scoring and reporting, data required for psychometric research (item analysis, differential item functioning studies, norming, equating), and data that is simply convenient to have. Collecting more than what is needed increases both regulatory exposure and the potential impact of a breach. This distinction matters especially when assessments involve minors or when government-administered programs impose specific consent and transparency requirements.

Anonymization and de-identification for research. Much of the analytic work that follows an assessment — item calibration, bias analysis, reliability studies — can and often should be conducted on de-identified datasets. Techniques such as pseudonymization and aggregation thresholds allow psychometricians to perform rigorous analysis without retaining linkages to individual examinees. The key tension here is between analytic granularity and re-identification risk: subgroup analyses that are fine-grained enough to detect bias may also be fine-grained enough to identify individuals in small populations. Responsible programs address this tension explicitly rather than assuming that removing names and ID numbers is sufficient.

The Standards Landscape: A Patchwork With Real Consequences

Which standards apply to a given assessment program depends heavily on the type of organization, the industry it serves, and the populations it assesses. This is where things get complicated, and where we frequently see organizations struggling.

SOC 2 Type 2 audits evaluate an organization's controls against the Trust Service Criteria for security, availability, processing integrity, confidentiality, and privacy. For assessment platforms operating as SaaS products, which is increasingly the norm, a SOC 2 Type 2 report has become a de facto prerequisite for enterprise procurement. It demonstrates not just that controls exist on paper, but that they have been tested and verified over a sustained period.

HITRUST and HITECH become relevant where assessments intersect with healthcare: continuing medical education, clinical competency evaluations, employee wellness surveys, or any context where health-related data may be collected. The HITRUST Common Security Framework (CSF) is notable because it attempts to map and harmonize requirements across multiple regulations, but achieving and maintaining HITRUST certification is a substantial undertaking.

FERPA (the Family Educational Rights and Privacy Act) governs educational records in institutions receiving federal funding, making it directly relevant to K–12 and higher education assessment programs. FERPA imposes specific requirements around consent, access, and disclosure that shape how student assessment data can be stored, shared, and used for research.

Emerging state and international privacy laws — CCPA, GDPR, and a growing patchwork of state-level regulations — add further layers, particularly for assessment programs that operate across jurisdictions or involve international examinee populations.

In practice, these overlapping frameworks frequently lead to a fragmented implementation landscape. Organizations operating under multiple standards simultaneously may find that compliance requirements conflict, create redundant controls that add friction without proportionate security benefit, or encourage a "checkbox compliance" mentality where policies exist on paper but do not translate into genuinely robust security practices. We regularly encounter systems where encryption is applied inconsistently across legacy and modern components, where access controls are nominally in place but practically unenforced, or where data retention policies exist but are never audited.

The practical challenge is implementing the applicable standards coherently across an organization's actual technology stack and operational workflows.

Modern Authentication: What NIST's Updated Guidance Means for Assessments

One area where the gap between current best practice and actual implementation is especially visible is authentication management. NIST's Special Publication 800-63B , the authoritative federal guideline for digital authentication, underwent a significant revision with Revision 4, finalized in July 2025. The changes are worth understanding because they directly contradict policies still in place at many organizations.

The headline shifts include the elimination of mandatory periodic password rotation — a practice that NIST's own research found leads to weaker passwords as users resort to predictable incremental changes. Revision 4 now states that organizations should not require password changes unless there is evidence of compromise. The updated guidance also explicitly prohibits arbitrary complexity composition rules (requiring special characters, mixed case, etc.), instead emphasizing password length as the primary factor in strength and mandating that passwords be screened against databases of known compromised credentials.

Critically for assessment platforms, the revised standard requires that systems allow the use of password managers and autofill functionality, and recommends supporting paste functionality in password fields. Multi-factor authentication is strongly encouraged, with an emphasis on phishing-resistant methods. Time-based one-time password (TOTP) apps, such as Google Authenticator or Microsoft Authenticator, and single sign-on (SSO) integration represent the practical implementation of these recommendations for most organizations.

This matters for assessment programs because they often serve user populations with widely varying technical sophistication: a corporate employee taking a compliance quiz, a 16-year-old sitting for a state exam, a nurse completing a continuing education module. Legacy assessment platforms frequently still enforce the very practices NIST now considers counterproductive: forced 90-day password resets, complexity rules that encourage "Password1!" patterns, and disabled paste functionality that actively undermines password manager use.

We see opportunities for improvement here. Implementing SSO and modern MFA on assessment platforms can simultaneously improve security and reduce friction for examinees — fewer login barriers, fewer forgotten-password support tickets, and stronger protection against credential-based attacks. But many organizations' internal security policies have not yet caught up with the current NIST recommendations, and vendor platforms — particularly those serving regulated industries where older compliance checklists remain in effect — may lag as well.

A Note on Paper-Based Assessments

It is worth briefly acknowledging that paper-based assessment forms have seen a resurgence in some educational settings, driven by concerns about the integrity of electronic testing, equity of device access, and screen fatigue among young learners. However, returning to paper does not eliminate data security concerns, instead it transforms them. Chain-of-custody protocols, secure physical storage and destruction, and controlled data-entry processes for digitization all introduce their own risks and require their own discipline. Organizations that move assessment modalities should ensure their security planning follows.

Building Security Into the Assessment Lifecycle

The common thread across each of these areas is that data security and privacy work best when they are built into the assessment design process from the beginning rather than retrofitted after a platform has been selected or a program is already operational. This means asking the right questions early: What data do we actually need? Who will have access, and under what controls? Which standards apply to our specific context, and how do we implement them as a coherent system rather than a collection of disconnected compliance checkboxes?

Is Your Assessment Program's Data Security Up to Standard?

These are the kinds of questions that benefit from experience working across sectors and experiencing how the same frameworks play out differently in a corporate training environment than in a state education agency or a clinical credentialing body. At Research Analytics Consulting, this cross-sector perspective informs how we approach assessment design, platform evaluation, and data governance planning for our clients. If your organization is developing, procuring, or modernizing an assessment program, we welcome the conversation. Schedule a consultation today .

Quality of Care for IDD Adults Living in Florida: A Pilot Study

Thu, 13 Nov 2025 03:47:00 GMT

When Caring Isn’t Enough: Rethinking “Quality of Care” for Adults with IDD

By Cindy M. Walker, PhD
Founder & CEO, Research Analytics Consulting
Co-authored by Jacqueline Gosz, MS and Sue Gottesman, MBA

When I agreed to help design a survey on the quality of care for adults with intellectual and developmental disabilities (IDD) in Florida, it wasn’t a business decision — it was a personal one.

As a Guardian Advocate and Representative Payee for a 46-year-old woman with an IDD — someone I love like family — I’ve seen firsthand both the dedication and the limitations within our system of support. She lives independently now, with a live-in caregiver, a coach, and family who care deeply for her. But like so many others, she depends on a network that is good-hearted yet fragmented — one that often provides comfort, but not always growth. This pilot study, which gathered the voices of 157 adults with IDD and their families across Florida, was an effort to listen — truly listen — to how those receiving care experience their lives.

What We Heard

Respondents were asked about their caregivers, job coaches, roommates, Supported Living Coaches, and Adult Day Training programs — the web of supports that sustain everyday life for adults with IDD.

When we analyzed the data, a clear pattern emerged:

● The highest scores reflected emotional care — feeling loved, respected, and supported.
● The lowest scores reflected growth — learning new things, gaining independence, becoming more self-sufficient.

In other words, people feel cared for, but they aren’t always learning how to care for themselves.

As one family shared, “We love the staff — but my son wants to work, not just go to a day program. No one helps him with this.”

This is a crucial finding. It suggests that while our systems of care succeed in meeting emotional needs, they often fall short in nurturing autonomy — the very foundation of empowerment and dignity.

A System of Comfort, Not Empowerment

Across the data, one theme echoes: adults with IDD are surrounded by care, but rarely by opportunity. Only 17% of respondents reported working, and less than 25% worked with a Supported Living Coach — the very professionals meant to teach independence.

Many programs focus on safety, consistency, and affection — all vital. But without a parallel focus on skill-building and choice, we risk creating a system of learned helplessness rather than one of self-determination.

As researchers, we see the numbers.
As families, we feel the consequences.

Where We Go From Here

This pilot study doesn’t claim to represent every community or every story. But it raises an urgent question for all of us — families, providers, policymakers, and advocates alike:

Are we helping adults with IDD live independently, or simply keeping them comfortable?

The answer calls for a renewed vision — one that values both care and capability. A system where teaching life skills, encouraging employment, and honoring choice are not add-ons but essentials.

Because true “quality of care” isn’t only about feeling cared for — it’s about becoming capable of caring for oneself.

At Research Analytics Consulting, we believe data should reveal not just what is measurable, but what is meaningful.

This report — Quality of Care for IDD Adults Living in Florida: A Pilot Study — is both a mirror and a call to action.

Read the full report here

The Content, Pedagogy, Implementation, and Context Components (CPIC) Study

Mon, 10 Nov 2025 02:49:41 GMT

Challenge

The Content, Pedagogy, Implementation, and Context Components (CPIC) Study aims to better understand Evidence Based Programs (EBP) for Teen Pregnancy Prevention (TPP). Our challenge is to determine the relationship between the intended and the implemented core components using empirical data, collected over multiple years of APP evaluation work conducted by AMTC & Associates; and, to determine which implemented core components are most essential for producing desired outcomes.

Solution

In Phase I: assemble a cleaned, merged data set comprising 11 years of survey, implementation, and attendance data for TPP programs utilizing 6 curricula. In the upcoming Phase II: propose hypotheses and provide categorical data analysis.

Impact

Overall, this study will aid implementers in adapting program components for target populations, researchers in testing the effects of individual components on participant outcomes, and policymakers and funders in identifying and prioritizing interventions with promising components. Furthermore, this study will provide TPP researchers with multiple examples of how to conduct evaluation studies that focus on the core components of the program implemented.

Methodology

We extracted curriculum information from AMTC’s implementation database (OPTS) to aid in the qualitative analysis of implemented core components. We linked the OPTS database to a database, obtained through local evaluation efforts, that consists of responses to pre- and post-surveys, from youth that participated in the implementation of the six EBPs studied in Phase I. This newly created database will be used to determine the relationship between various core components and the intended outcomes of program participation.

Results

We found sufficient data to analyze five of the six curricula. We detailed multiple versions of each curriculum to help identify core components for delivered classes. We assembled disparate survey data sets measuring outcomes across programs spanning 11 years and matched participants with their corresponding attendance records.

Tools Used:

Python and SQL for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, knitr, dplyr, psych, caret).

A Psychometric Approach to Accreditation

Sat, 01 Feb 2025 03:14:02 GMT

Challenge

One of the big five accounting firms provides continuing education, with the requisite CPE credit, to their professionals. Their programs are all accredited by the AACSB and are large-scale high-stakes assessments. As such, the item and test analysis must be comprehensive, precise and psychometrically rigorous. Additionally, annual reports must be provided for continuous quality improvement of the assessments associated with courses. Previous psychometric analysis for the reports only used Classical Test Theory (CTT) approaches. Even though high reliability of these assessments is desired for consistent measurement of learner knowledge and ability, leveraging only CTT can result in unstable statistical estimates, because the statistics associated with this method are sample and test dependent. Modern test theory utilizing item response theory (IRT) is a more appropriate and rigorous approach.

Solution

In this project we have evaluated over 100 learning assessments, used to measure learning outcomes of continuing education courses for accounting professionals using both CTT and IRT to optimize their learning assessments. This allows us to quantify item performance on each of the final assessments using psychometric metrics. We have helped to identify opportunities for improvement while ensuring compliance with ongoing board certification requirements. These quantitative analyses are followed by expert qualitative analyses to recommend next steps for achieving more reliable and valid measurement of knowledge of learning objectives.

Impact

The quantitative analyses are used to identify problematic items that may be improved to more validly measure learner ability with respect to the intended learning objectives. Using the results of the qualitative analysis, item writers are able to further adapt future assessments to improve reliability and better ascertain learner proficiency. Items identified as problematic can also be opportunities to clarify educational content presented to learners.

Methodology

We used IRT, CTT, DIF analyses, and modern data visualizations to quantify test item performance and identify opportunities for improvement of assessment. This included the full scope of the item from the question to the response options, with visualization of the various item characteristic curves to ascertain how items were performing across examinees. Once items were flagged as problematic, a qualitative review was conducted, using a standardized approach to make recommendations to item writers with respect to how to write items of higher quality.

Results

This is a year-to-year ongoing project. In year one we worked to evaluate the test and assessment process and write the initial base code so the results are accurate and repeatable. In year two we repeated the analysis and worked to semi-automate the process due to the large number of assessments. We have begun working on qualitative psychometric aspects around item writing, which includes rigorous evaluation of the assessment constructs in the context of the goals and objectives. This ensures the items are measuring what is intended and helps to increase the statistical validity and reliability. In the next several years, we will continue automation utilizing machine learning and artificial intelligence. We will also explore predictive validity to ensure the assessments are useful for important outcomes.

Tools Used:

Python for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, dplyr, CTT, difR, psych, car, flextable, knitr), FlexMIRT and R (mirt, ggmirt) for IRT analysis.

The George W. Bush Institute Case Study

Fri, 31 Jan 2025 01:13:28 GMT

Challenge

Determine the impact of the School Leadership Initiative (SLI) on principals’ perceptions of several key variables of interest to the GWBI.

Solution

Clean and organize data collected by the GWBI from school districts and survey participants, combine with publicly available data, and perform a variety of statistical analyses to answer poignant research questions posed by the GWBI.

Impact

Retention of leadership talent and facilitation of their best performance strengthens educational opportunities for district children. The data collected by the GWBI is now supporting future efforts like the SLI.

Methodology

Research Analytics Consulting, LLC (RAC) embarked upon a pilot study to determine the impact of the George W. Bush Institute’s (GWBI) School Leadership Initiative (SLI) on principals’ perceptions of several key variables of interest to the GWBI. We constructed four variables based on an Exploratory Factor Analysis, and quantified the improvement in these variables associated with participation in the SLI. Then we identified the impact of each of these variables on school principal retention within the Fort Worth district.

Based on the results of the pilot study, RAC was contracted to conduct an expanded study using eight additional variables hypothesized to be impacted by the SLI, while focusing on three school districts that participated in the GWBI SLI from 2018 through 2021. We conducted reliability analysis, factor analysis and multilevel modeling.

Results

In the pilot study, four variables were uncovered fitting a factor analysis model to the SLI Principal Survey. There was some level of disagreement found on four of the 12 identical survey items that were administered to both district level personnel and principals. In most cases, district level personnel perceived district-level practices more positively than principals perceived. Approximately 12% of the variability in STARR Reading Achievement scores, for 2019 fifth grade students in the Fort Worth School District was explained by Job Satisfaction and School Climate. Fitting the same model, across all grade levels, yielded comparable results and explained 15% of the variability in STARR Reading Test Scores.

In the broader subsequent analysis, principal scores tended to increase over time, which implies that principals’ perceptions about their jobs and district level processes and procedures improved over time, as the GWBI SLI project matured. Job Satisfaction, School Climate, Compensation and Incentives, Working Environment, Job Embeddedness, and School Culture were found to be statistically significant as predictors of principal attrition.

Testing for a mediating relationship suggested that if principals feel more embedded in their jobs then they are more likely to stay in the district, even if they are not that satisfied with their compensation and incentives, as measured by level of perceived competitiveness of pay and non-monetary compensation for principals within the district.

Tools Used:

Python for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, knitr, dplyr, psych, caret, mlbench).

Determining the Impact of a Unique Public Health Initiative

Sun, 05 May 2024 02:56:08 GMT

Challenge

According to the Centers for Disease Control many young persons in the United States engage in risky sexual behavior and experience negative reproductive health outcomes NJPAG has created a curriculum for youth to help them avoid risk and make healthy choices using a three-pronged approach: the right information, a realistic application to one’s life, and sincere encouragement and support. In other words, NJPAG educators strive to give young people the facts and help them believe that they can make difficult, yet wise, choices in their own best interest. Their curriculum is currently used in over 250 schools throughout New Jersey and 8 additional states. They were recently awarded a PREIS grant to conduct a Randomized Controlled Trial (RCT) to determine the impact of their curriculum.

Solution

In this project we work closely with NJPAG as their external evaluator. We have created all of the processes and procedures associated with data collection and analysis. We have conducted site visits to ensure these processes and procedures are implemented with fidelity. Specifically, we are responsible for registering the study and obtaining IRB approval, creating valid and reliable measures, developing scripts for data collection processes, conducting power analyses and tests of baseline equivalence, and tracking attrition of our treatment and control groups to ensure we do not have differential attrition. The goal of this study is to see if the NJPAG curriculum can be added to the evidence based list for promising programs for reducing sexual risk.

Impact

The impact of this study is that if found successful the curriculum will be included on the evidence abased list.

Methodology

We are using standard methods, associated with evaluation; however, as this study is funded by the Department of Health and Human Services (DHHS) we are held to the highest standards of rigor. The results of this study will only be reviewed for inclusion on the evidence based list if we have adequate power, baseline equivalence, and the lack of differential attrition. We are only in the first year of the actua study. However, preliminary analyses have suggested that our study does meet these criteria.

Results

This is an ongoing project.

Tools Used:

This is an ongoing project.