Research Analytics Consulting, LLC

Data Security and Privacy in Modern Assessments

Wed, 29 Apr 2026 02:07:47 GMT

How organizations can navigate the evolving landscape of standards, authentication, and data protection in high-stakes testing environments:

Whether it's a certification exam for healthcare professionals, a corporate compliance assessment, a statewide student evaluation, or a research survey collecting sensitive demographic information, modern assessments generate and depend on data that demands serious protection. Test content represents significant intellectual property. Examinee records often contain personally identifiable information; and in many cases, that information belongs to minors, pertains to employment consequences, or intersects with health-related contexts.

Yet in our work across corporate, government, and educational assessment programs, we consistently find that data security and privacy are treated as afterthoughts or bolt-on concerns addressed late in the development cycle rather than principles embedded from the start. This post outlines the core security fundamentals that assessment programs should address, surveys the standards landscape organizations must navigate, and highlights recent shifts in authentication guidance that present both an opportunity and a challenge.

The Fundamentals: What Assessment Data Security Actually Requires

At its foundation, protecting assessment data involves four interrelated practices.

Encryption in transit and at rest. Any assessment platform transmitting examinee responses, scores, or personal information over a network should use TLS (Transport Layer Security) to protect data in transit. Equally important, stored data — whether in a cloud database, a file server, or a backup archive — should be encrypted using robust standards such as AES-256. This is not merely a best practice; it is a baseline expectation. Assessment content is high-value intellectual property, and a breach of live test items can invalidate an entire exam form, resulting in costs that extend far beyond the data itself.

Authentication and access rights management. Assessment programs involve a range of roles with very different data needs: test developers, psychometricians , proctors, administrators, and examinees. A well-designed system enforces role-based access control (RBAC) so that each user can access only the data and functions relevant to their role. A proctor, for instance, needs to verify examinee identity and manage session logistics but should never have access to

item-level scoring algorithms or raw psychometric data. Applying the principle of least privilege reduces the surface area available to both external attackers and inadvertent internal mishandling.

Data minimization. Not all data that can be collected should be collected. Organizations should clearly distinguish between data necessary for scoring and reporting, data required for psychometric research (item analysis, differential item functioning studies, norming, equating), and data that is simply convenient to have. Collecting more than what is needed increases both regulatory exposure and the potential impact of a breach. This distinction matters especially when assessments involve minors or when government-administered programs impose specific consent and transparency requirements.

Anonymization and de-identification for research. Much of the analytic work that follows an assessment — item calibration, bias analysis, reliability studies — can and often should be conducted on de-identified datasets. Techniques such as pseudonymization and aggregation thresholds allow psychometricians to perform rigorous analysis without retaining linkages to individual examinees. The key tension here is between analytic granularity and re-identification risk: subgroup analyses that are fine-grained enough to detect bias may also be fine-grained enough to identify individuals in small populations. Responsible programs address this tension explicitly rather than assuming that removing names and ID numbers is sufficient.

The Standards Landscape: A Patchwork With Real Consequences

Which standards apply to a given assessment program depends heavily on the type of organization, the industry it serves, and the populations it assesses. This is where things get complicated, and where we frequently see organizations struggling.

SOC 2 Type 2 audits evaluate an organization's controls against the Trust Service Criteria for security, availability, processing integrity, confidentiality, and privacy. For assessment platforms operating as SaaS products, which is increasingly the norm, a SOC 2 Type 2 report has become a de facto prerequisite for enterprise procurement. It demonstrates not just that controls exist on paper, but that they have been tested and verified over a sustained period.

HITRUST and HITECH become relevant where assessments intersect with healthcare: continuing medical education, clinical competency evaluations, employee wellness surveys, or any context where health-related data may be collected. The HITRUST Common Security Framework (CSF) is notable because it attempts to map and harmonize requirements across multiple regulations, but achieving and maintaining HITRUST certification is a substantial undertaking.

FERPA (the Family Educational Rights and Privacy Act) governs educational records in institutions receiving federal funding, making it directly relevant to K–12 and higher education assessment programs. FERPA imposes specific requirements around consent, access, and disclosure that shape how student assessment data can be stored, shared, and used for research.

Emerging state and international privacy laws — CCPA, GDPR, and a growing patchwork of state-level regulations — add further layers, particularly for assessment programs that operate across jurisdictions or involve international examinee populations.

In practice, these overlapping frameworks frequently lead to a fragmented implementation landscape. Organizations operating under multiple standards simultaneously may find that compliance requirements conflict, create redundant controls that add friction without proportionate security benefit, or encourage a "checkbox compliance" mentality where policies exist on paper but do not translate into genuinely robust security practices. We regularly encounter systems where encryption is applied inconsistently across legacy and modern components, where access controls are nominally in place but practically unenforced, or where data retention policies exist but are never audited.

The practical challenge is implementing the applicable standards coherently across an organization's actual technology stack and operational workflows.

Modern Authentication: What NIST's Updated Guidance Means for Assessments

One area where the gap between current best practice and actual implementation is especially visible is authentication management. NIST's Special Publication 800-63B , the authoritative federal guideline for digital authentication, underwent a significant revision with Revision 4, finalized in July 2025. The changes are worth understanding because they directly contradict policies still in place at many organizations.

The headline shifts include the elimination of mandatory periodic password rotation — a practice that NIST's own research found leads to weaker passwords as users resort to predictable incremental changes. Revision 4 now states that organizations should not require password changes unless there is evidence of compromise. The updated guidance also explicitly prohibits arbitrary complexity composition rules (requiring special characters, mixed case, etc.), instead emphasizing password length as the primary factor in strength and mandating that passwords be screened against databases of known compromised credentials.

Critically for assessment platforms, the revised standard requires that systems allow the use of password managers and autofill functionality, and recommends supporting paste functionality in password fields. Multi-factor authentication is strongly encouraged, with an emphasis on phishing-resistant methods. Time-based one-time password (TOTP) apps, such as Google Authenticator or Microsoft Authenticator, and single sign-on (SSO) integration represent the practical implementation of these recommendations for most organizations.

This matters for assessment programs because they often serve user populations with widely varying technical sophistication: a corporate employee taking a compliance quiz, a 16-year-old sitting for a state exam, a nurse completing a continuing education module. Legacy assessment platforms frequently still enforce the very practices NIST now considers counterproductive: forced 90-day password resets, complexity rules that encourage "Password1!" patterns, and disabled paste functionality that actively undermines password manager use.

We see opportunities for improvement here. Implementing SSO and modern MFA on assessment platforms can simultaneously improve security and reduce friction for examinees — fewer login barriers, fewer forgotten-password support tickets, and stronger protection against credential-based attacks. But many organizations' internal security policies have not yet caught up with the current NIST recommendations, and vendor platforms — particularly those serving regulated industries where older compliance checklists remain in effect — may lag as well.

A Note on Paper-Based Assessments

It is worth briefly acknowledging that paper-based assessment forms have seen a resurgence in some educational settings, driven by concerns about the integrity of electronic testing, equity of device access, and screen fatigue among young learners. However, returning to paper does not eliminate data security concerns, instead it transforms them. Chain-of-custody protocols, secure physical storage and destruction, and controlled data-entry processes for digitization all introduce their own risks and require their own discipline. Organizations that move assessment modalities should ensure their security planning follows.

Building Security Into the Assessment Lifecycle

The common thread across each of these areas is that data security and privacy work best when they are built into the assessment design process from the beginning rather than retrofitted after a platform has been selected or a program is already operational. This means asking the right questions early: What data do we actually need? Who will have access, and under what controls? Which standards apply to our specific context, and how do we implement them as a coherent system rather than a collection of disconnected compliance checkboxes?

Is Your Assessment Program's Data Security Up to Standard?

These are the kinds of questions that benefit from experience working across sectors and experiencing how the same frameworks play out differently in a corporate training environment than in a state education agency or a clinical credentialing body. At Research Analytics Consulting, this cross-sector perspective informs how we approach assessment design, platform evaluation, and data governance planning for our clients. If your organization is developing, procuring, or modernizing an assessment program, we welcome the conversation. Schedule a consultation today .

Quality of Care for IDD Adults Living in Florida: A Pilot Study

Thu, 13 Nov 2025 03:47:00 GMT

When Caring Isn’t Enough: Rethinking “Quality of Care” for Adults with IDD

By Cindy M. Walker, PhD
Founder & CEO, Research Analytics Consulting
Co-authored by Jacqueline Gosz, MS and Sue Gottesman, MBA

When I agreed to help design a survey on the quality of care for adults with intellectual and developmental disabilities (IDD) in Florida, it wasn’t a business decision — it was a personal one.

As a Guardian Advocate and Representative Payee for a 46-year-old woman with an IDD — someone I love like family — I’ve seen firsthand both the dedication and the limitations within our system of support. She lives independently now, with a live-in caregiver, a coach, and family who care deeply for her. But like so many others, she depends on a network that is good-hearted yet fragmented — one that often provides comfort, but not always growth. This pilot study, which gathered the voices of 157 adults with IDD and their families across Florida, was an effort to listen — truly listen — to how those receiving care experience their lives.

What We Heard

Respondents were asked about their caregivers, job coaches, roommates, Supported Living Coaches, and Adult Day Training programs — the web of supports that sustain everyday life for adults with IDD.

When we analyzed the data, a clear pattern emerged:

● The highest scores reflected emotional care — feeling loved, respected, and supported.
● The lowest scores reflected growth — learning new things, gaining independence, becoming more self-sufficient.

In other words, people feel cared for, but they aren’t always learning how to care for themselves.

As one family shared, “We love the staff — but my son wants to work, not just go to a day program. No one helps him with this.”

This is a crucial finding. It suggests that while our systems of care succeed in meeting emotional needs, they often fall short in nurturing autonomy — the very foundation of empowerment and dignity.

A System of Comfort, Not Empowerment

Across the data, one theme echoes: adults with IDD are surrounded by care, but rarely by opportunity. Only 17% of respondents reported working, and less than 25% worked with a Supported Living Coach — the very professionals meant to teach independence.

Many programs focus on safety, consistency, and affection — all vital. But without a parallel focus on skill-building and choice, we risk creating a system of learned helplessness rather than one of self-determination.

As researchers, we see the numbers.
As families, we feel the consequences.

Where We Go From Here

This pilot study doesn’t claim to represent every community or every story. But it raises an urgent question for all of us — families, providers, policymakers, and advocates alike:

Are we helping adults with IDD live independently, or simply keeping them comfortable?

The answer calls for a renewed vision — one that values both care and capability. A system where teaching life skills, encouraging employment, and honoring choice are not add-ons but essentials.

Because true “quality of care” isn’t only about feeling cared for — it’s about becoming capable of caring for oneself.

At Research Analytics Consulting, we believe data should reveal not just what is measurable, but what is meaningful.

This report — Quality of Care for IDD Adults Living in Florida: A Pilot Study — is both a mirror and a call to action.

Read the full report here

The Content, Pedagogy, Implementation, and Context Components (CPIC) Study

Mon, 10 Nov 2025 02:49:41 GMT

Challenge

The Content, Pedagogy, Implementation, and Context Components (CPIC) Study aims to better understand Evidence Based Programs (EBP) for Teen Pregnancy Prevention (TPP). Our challenge is to determine the relationship between the intended and the implemented core components using empirical data, collected over multiple years of APP evaluation work conducted by AMTC & Associates; and, to determine which implemented core components are most essential for producing desired outcomes.

Solution

In Phase I: assemble a cleaned, merged data set comprising 11 years of survey, implementation, and attendance data for TPP programs utilizing 6 curricula. In the upcoming Phase II: propose hypotheses and provide categorical data analysis.

Impact

Overall, this study will aid implementers in adapting program components for target populations, researchers in testing the effects of individual components on participant outcomes, and policymakers and funders in identifying and prioritizing interventions with promising components. Furthermore, this study will provide TPP researchers with multiple examples of how to conduct evaluation studies that focus on the core components of the program implemented.

Methodology

We extracted curriculum information from AMTC’s implementation database (OPTS) to aid in the qualitative analysis of implemented core components. We linked the OPTS database to a database, obtained through local evaluation efforts, that consists of responses to pre- and post-surveys, from youth that participated in the implementation of the six EBPs studied in Phase I. This newly created database will be used to determine the relationship between various core components and the intended outcomes of program participation.

Results

We found sufficient data to analyze five of the six curricula. We detailed multiple versions of each curriculum to help identify core components for delivered classes. We assembled disparate survey data sets measuring outcomes across programs spanning 11 years and matched participants with their corresponding attendance records.

Tools Used:

Python and SQL for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, knitr, dplyr, psych, caret).

A Psychometric Approach to Accreditation

Sat, 01 Feb 2025 03:14:02 GMT

Challenge

One of the big five accounting firms provides continuing education, with the requisite CPE credit, to their professionals. Their programs are all accredited by the AACSB and are large-scale high-stakes assessments. As such, the item and test analysis must be comprehensive, precise and psychometrically rigorous. Additionally, annual reports must be provided for continuous quality improvement of the assessments associated with courses. Previous psychometric analysis for the reports only used Classical Test Theory (CTT) approaches. Even though high reliability of these assessments is desired for consistent measurement of learner knowledge and ability, leveraging only CTT can result in unstable statistical estimates, because the statistics associated with this method are sample and test dependent. Modern test theory utilizing item response theory (IRT) is a more appropriate and rigorous approach.

Solution

In this project we have evaluated over 100 learning assessments, used to measure learning outcomes of continuing education courses for accounting professionals using both CTT and IRT to optimize their learning assessments. This allows us to quantify item performance on each of the final assessments using psychometric metrics. We have helped to identify opportunities for improvement while ensuring compliance with ongoing board certification requirements. These quantitative analyses are followed by expert qualitative analyses to recommend next steps for achieving more reliable and valid measurement of knowledge of learning objectives.

Impact

The quantitative analyses are used to identify problematic items that may be improved to more validly measure learner ability with respect to the intended learning objectives. Using the results of the qualitative analysis, item writers are able to further adapt future assessments to improve reliability and better ascertain learner proficiency. Items identified as problematic can also be opportunities to clarify educational content presented to learners.

Methodology

We used IRT, CTT, DIF analyses, and modern data visualizations to quantify test item performance and identify opportunities for improvement of assessment. This included the full scope of the item from the question to the response options, with visualization of the various item characteristic curves to ascertain how items were performing across examinees. Once items were flagged as problematic, a qualitative review was conducted, using a standardized approach to make recommendations to item writers with respect to how to write items of higher quality.

Results

This is a year-to-year ongoing project. In year one we worked to evaluate the test and assessment process and write the initial base code so the results are accurate and repeatable. In year two we repeated the analysis and worked to semi-automate the process due to the large number of assessments. We have begun working on qualitative psychometric aspects around item writing, which includes rigorous evaluation of the assessment constructs in the context of the goals and objectives. This ensures the items are measuring what is intended and helps to increase the statistical validity and reliability. In the next several years, we will continue automation utilizing machine learning and artificial intelligence. We will also explore predictive validity to ensure the assessments are useful for important outcomes.

Tools Used:

Python for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, dplyr, CTT, difR, psych, car, flextable, knitr), FlexMIRT and R (mirt, ggmirt) for IRT analysis.

The George W. Bush Institute Case Study

Fri, 31 Jan 2025 01:13:28 GMT

Challenge

Determine the impact of the School Leadership Initiative (SLI) on principals’ perceptions of several key variables of interest to the GWBI.

Solution

Clean and organize data collected by the GWBI from school districts and survey participants, combine with publicly available data, and perform a variety of statistical analyses to answer poignant research questions posed by the GWBI.

Impact

Retention of leadership talent and facilitation of their best performance strengthens educational opportunities for district children. The data collected by the GWBI is now supporting future efforts like the SLI.

Methodology

Research Analytics Consulting, LLC (RAC) embarked upon a pilot study to determine the impact of the George W. Bush Institute’s (GWBI) School Leadership Initiative (SLI) on principals’ perceptions of several key variables of interest to the GWBI. We constructed four variables based on an Exploratory Factor Analysis, and quantified the improvement in these variables associated with participation in the SLI. Then we identified the impact of each of these variables on school principal retention within the Fort Worth district.

Based on the results of the pilot study, RAC was contracted to conduct an expanded study using eight additional variables hypothesized to be impacted by the SLI, while focusing on three school districts that participated in the GWBI SLI from 2018 through 2021. We conducted reliability analysis, factor analysis and multilevel modeling.

Results

In the pilot study, four variables were uncovered fitting a factor analysis model to the SLI Principal Survey. There was some level of disagreement found on four of the 12 identical survey items that were administered to both district level personnel and principals. In most cases, district level personnel perceived district-level practices more positively than principals perceived. Approximately 12% of the variability in STARR Reading Achievement scores, for 2019 fifth grade students in the Fort Worth School District was explained by Job Satisfaction and School Climate. Fitting the same model, across all grade levels, yielded comparable results and explained 15% of the variability in STARR Reading Test Scores.

In the broader subsequent analysis, principal scores tended to increase over time, which implies that principals’ perceptions about their jobs and district level processes and procedures improved over time, as the GWBI SLI project matured. Job Satisfaction, School Climate, Compensation and Incentives, Working Environment, Job Embeddedness, and School Culture were found to be statistically significant as predictors of principal attrition.

Testing for a mediating relationship suggested that if principals feel more embedded in their jobs then they are more likely to stay in the district, even if they are not that satisfied with their compensation and incentives, as measured by level of perceived competitiveness of pay and non-monetary compensation for principals within the district.

Tools Used:

Python for data extraction, cleaning and pre-processing; R for statistical analysis (tidyverse, knitr, dplyr, psych, caret, mlbench).

Determining the Impact of a Unique Public Health Initiative

Sun, 05 May 2024 02:56:08 GMT

Challenge

According to the Centers for Disease Control many young persons in the United States engage in risky sexual behavior and experience negative reproductive health outcomes NJPAG has created a curriculum for youth to help them avoid risk and make healthy choices using a three-pronged approach: the right information, a realistic application to one’s life, and sincere encouragement and support. In other words, NJPAG educators strive to give young people the facts and help them believe that they can make difficult, yet wise, choices in their own best interest. Their curriculum is currently used in over 250 schools throughout New Jersey and 8 additional states. They were recently awarded a PREIS grant to conduct a Randomized Controlled Trial (RCT) to determine the impact of their curriculum.

Solution

In this project we work closely with NJPAG as their external evaluator. We have created all of the processes and procedures associated with data collection and analysis. We have conducted site visits to ensure these processes and procedures are implemented with fidelity. Specifically, we are responsible for registering the study and obtaining IRB approval, creating valid and reliable measures, developing scripts for data collection processes, conducting power analyses and tests of baseline equivalence, and tracking attrition of our treatment and control groups to ensure we do not have differential attrition. The goal of this study is to see if the NJPAG curriculum can be added to the evidence based list for promising programs for reducing sexual risk.

Impact

The impact of this study is that if found successful the curriculum will be included on the evidence abased list.

Methodology

We are using standard methods, associated with evaluation; however, as this study is funded by the Department of Health and Human Services (DHHS) we are held to the highest standards of rigor. The results of this study will only be reviewed for inclusion on the evidence based list if we have adequate power, baseline equivalence, and the lack of differential attrition. We are only in the first year of the actua study. However, preliminary analyses have suggested that our study does meet these criteria.

Results

This is an ongoing project.

Tools Used:

This is an ongoing project.