Research

Throughout my scientific career, my research has spanned a wide range of fields, from physics and astronomy to educational measurement. My current work focuses on advancing the assessment of complex skills through innovations in digital technology, AI, data science, and psychometrics. Specific areas of focus include collaborative problem solving, communication, automated scoring and coding, test security in remote testing environments, and game- and simulation-based assessments. Below are selected projects that I have been leading and contributing to.

Assessment of Complex Skills [2013 - present]

Since 2013, I have led and co-led a sustained portfolio of research projects focused on the assessment of complex skills using technology-enabled and performance-based approaches. Key projects include:

• AI-enabled Assessment of Communication Skills [2025–Present]
  Technology lead for the development of an AI-enabled assessment prototype targeting communication skills.
• Assessment of Self-Regulated Learning in AI-Assisted Environments [2025–Present]
  Initiated and guided a project, providing oversight for the development of an assessment prototype for self-regulated learning in AI-assisted environments.
• Developing Middle Grade Students’ Social-Emotional Learning Skills through Technology-Enhanced Collaborative Learning [2024–2028]
  Co-Principal Investigator on a U.S. Department of Education–funded Education Innovation and Research (EIR) project focused on technology-enhanced assessment and instruction of SEL skills. Selected publication
• ETS Platform for Collaborative Assessment and Learning (EPCAL) [2015–Present]
  Led the design and development of an online research platform for assessing and supporting human–human interaction, including collaboration, communication, and social-emotional learning. EPCAL serves as a core infrastructure for studying interpersonal skills and has supported over 20 projects funded by ETS and external agencies, including NSF, IES, ARI, and the Gates Foundation. Selected publication. Some federally funded projects include:
  • Kyllonen, P. C. (PI), Hao, J. (Co-PI), Andrews, J., Elliott, S. N., & DiPerna, J. C. (2024–2027). Developing middle school students’ social-emotional learning skill applications through technology-enhanced collaborative learning. U.S. Department of Education, Office of Elementary and Secondary Education, Education Innovation and Research (EIR) Program. Funding amount: $4,329,001.
  • Kyllonen, P. C. (PI) & Hao, J. (Co-PI). (2019–2022). Collaborative problem solving (CPS) skill: Estimating an individual’s contribution to small group performance. U.S. Army Research Office (Award No. W911NF-19-1-0106). Funding amount: $993,345.
  • Song, Y. (PI), Sabatini, J. P. (Co-PI), & Ferretti, R. P. (Co-PI). (2019–2024). Exploring and assessing the development of students’ argumentation skills. Institute of Education Sciences. Funding amount: $1,399,578.
  • Andrews Todd, J. (PI), D’Mello, S., Chung, G., & Jackson, T. (Co-PIs). (2017–2022). A theory- and data-driven approach for identifying evidence of collaborative problem-solving skills. Institute of Education Sciences (Grant No. R305A170432). Funding amount: $1,399,250.
  • Graf, A. (PI), Crombie, B., Lai, Y., Lizano, C., & Deleon-Cuevas, C. (Co-PIs), Andrews Todd, J., & van Rijn, P. (Co-Is). (2021–2025). Investigating the role of collaboration on the development of student ideas using a learning progression for the function concept. National Science Foundation (DRL-2101393). Funding amount: $3,063,630.
  • Glogger-Frey, I. (PI), Abele, S. (co-PI), & Gschwendtner, T. (co-PI). (2019–2023). Digitale Diagnostik und Intervention im Kfz-Wesen (DigiDIn-Kfz). Bundesministerium für Bildung und Forschung, Förderinitiative ASCOT+ (German Federal Ministry of Education and Research, ASCOT+). Development of a digital video-based learning instrument to support diagnostic strategy acquisition among automotive mechatronics apprentices.
• Improving Learning and Assessment through Human–Human Interaction [2020–2025]
  Led a multi-year ETS Research Allocation project examining how evidence from human–human interaction can inform learning and assessment design.
• ETS Platform for Collaborative Human Annotation (EPCHA) [2020–Present]
  Led the development of a distributed platform to support scalable and efficient human annotation of communication and interaction data from collaborative tasks.
• Collaborative Problem Solving Skills: Estimating an Individual’s Contribution to Small Group Performance [2019–2022]
  Co-Principal Investigator on an Army Research Institute–funded project developing methods to measure individual contributions within team-based problem-solving contexts.
• Collaborative Science Assessment Prototype (ECSAP) [2013–2017]
  Led the development of an assessment prototype for collaborative problem solving in science. The prototype was highlighted by NCES Commissioner Peggy Carr as an example of next-generation assessments in a 2014 White House presentation. Selected publication

Large Language Model and Responsible Use of AI [2023 - now]

Since the release of ChatGPT in November 2022, I initiated and led a team of scientists to explore its impact on assessment and learning. The projects include:

• Characterizing LLM-Assisted Writing and Automated Scoring: Led research to identify distinguishing features of LLM-assisted essays and develop automated scoring and detecting methods based on LLMs. Selected publication
• Automated Coding of Communication Data Using LLM: Led a series of projects to investigate the accuracy and bias of LLM-based automated coding of communication data for assessing complex skills. Selected publication
• Responsible Detection of AI-generated Texts: Led the development of a suite of machine learning-powered detectors to identify AI-generated essays and investigate its responsible use across subgroups. Selected publication
• Impacts and implications of LLMs and GenAI on Assessment: Led a community-wide effort with researchers from leading assessment organizations (ETS, Duolingo, NBME, Cambium, Boston College, and the University of Iowa) to develop a collaborative paper discussing the implications of LLMs and Generative AI in assessment and introducing guidelines for their responsible and ethical use. Selected publication
• Generative AI-powered Tools: Led the development of several LLM- and GenAI-powered applications for audio creation, text translation, and automated coding, enhancing the efficiency and accessibility of AI-supported assessment tools.
• Generative AI-enabled Assessment: Led the design and prototyping of GenAI-driven assessment systems to evaluate complex skills such as communication, collaboration, argumentation, and social-emotional competencies, advancing next-generation approaches to performance-based assessment.

Data Analytics and AI to Support Test Security [2020 - present]

Since 2020, I have led multiple AI and data analytics-driven projects funded by the ETS Test Security Initiative to advance analytics and AI methods supporting test security, particularly for remotely administered tests. These projects have resulted in several impactful systems that contribute to over 60% of cheating detection at the peak time. Listed below are some of the key tools:

• AutoESD - Automated Essay Similarity Detection: A system that identifies highly similar essays and presents them to human experts through an AI-assisted analytics dashboard for final adjudication. Selected publication
• AutoSSD - Automated Speech Similarity Detection: A system that detects similar speech responses and presents them to human experts via an AI-assisted dashboard for review and decision-making. Selected publication
• Keystroke Analytics and Biometrics: A suite of tools designed to identify imposters, copywriting incidents, and AI-generated essays through typing behavior and biometric patterns. Selected publication
• Clickstream Data-Based Suspicious Behavior Detection: A set of analytics tools that detect anomalous behaviors during test-taking to flag potentially suspicious activity. Selected publication
• Overall Summary Post.

Development of Core Data and Software Infrastructure for Game/Simulation-Based (Virtual Performance) Assessments [2014 - 2021]

I led a series of projects dedicated to developing data analytics solutions for telemetry logs from game-based assessments and digital learning environments. Key projects include:

• GlassPy: Game Log Analysis in Python [2014 - 2019]. Led the development of a Python package for the analysis of log data from game-based assessments. An early publication from this project is available here, and a more recent presentation can be viewed at this YouTube Video.
• Data Model for VPAs and Evidence Trace File [2014 - 2018]. Led the development of ETS’s data model for virtual performance assessments, introducing the Evidence Trace File (ETF) as a replacement for traditional log files. Designed the Evidence Identification-Centered Data Design (EICDD) framework to bridge evidence-centered design (ECD) and data engineering. Selected publication.
• EPCAL Analytics [2019 - 2021]. Led the development of analytics dashboards for data generated from collaborative tasks deployed on the EPCAL platform, enabling real-time visualization and insight into learner interaction processes.

Application of Data Science, Machine Learning/AI, and Natural Language Processing to Improve Learning and Assessment [2013 - present]

I have led, co-led, and contributed to a wide range of projects aimed at enhancing learning and assessment through the analysis of response process data. These initiatives employ innovative data science and AI approaches, as outlined below:

• CPS-Rater: Developed automated annotation methods for analyzing communication in collaborative problem-solving activities, streamlining the evaluation of group interactions and teamwork dynamics. Selected publication
• Writing Analytics and Biometrics: Leveraged keystroke dynamics from the writing process to gain deeper insights into writing fluency, cognitive load, and individual writing strategies. Selected publication
• Learner Behavioral Analytics in Writing Mentor: Analyzed learner interactions within the online writing platform Writing Mentor to uncover behavioral patterns and inform personalized feedback and instructional support. Selected publication
• Winsight Process Data Analytics: Applied advanced process data analytics to strengthen psychometric modeling, supporting research on student performance, engagement, and cognitive strategies.
• Analysis and Modeling of Process Data from the National Assessment of Educational Progress (NAEP): Conducted large-scale analyses of NAEP process data to uncover insights into learning behaviors, item interaction patterns, and trends in educational progress.

Astrophysics and Cosmology (Before 2013)

I conducted research in astrophysics and cosmology before I switched to educational measurement. Key research topics include:

• Error-corrected Gaussian mixture models for red-sequence evolution
• GMBCG algorithm and large-scale optical galaxy-cluster catalogs
• Dark Energy Survey galaxy-cluster data management
• DECam focal-plane flatness measurement
• CCD image reduction and image-quality pipelines
• Precision measurements of galaxy alignment using SDSS data
• Dynamical system of dark energy
• Stability of Black hole; Solitons