The problem
How do you teach a room of strangers to program?
Teaching introductory computer science has always posed a stubborn challenge: how do you effectively teach students with vastly different backgrounds in a single class?
That question contains a dozen smaller ones, and one of the most interesting is the question of diagnosis — how do you even figure out what your students are struggling with, before you can begin to help?
One of the tools educators reach for is the concept inventory (CI) — an assessment instrument specifically designed to surface common misconceptions, not just measure right and wrong answers. CIs have quietly been doing this work across STEM for decades. In computer science, they’ve been a burgeoning field for close to ten years.
This paper was a stocktake.
The study
A literature review, systematically.
72
Academic papers
7
Researchers
4
Months
1
ACM ICER acceptance
- Who: Murtaza, Sourojit, Prerna, Raveena, Sophia, Alix, Mengqi — with Dr. Sayamindu Dasgupta
- Where: HCDE, University of Washington
- When: January — April 2023 (ongoing follow-up)
- Role: researcher
- Methods: systematic literature review, secondary research, thematic analysis, open-coding, quantitative analysis, academic writing
- Tools: R, Google Sheets, LaTeX, Zotero, UW Libraries
We conducted a systematic literature review of concept-inventory work in computer science, paying special attention to the field’s immense growth as well as the challenges researchers have been running into along the way. Out of that work emerged a map of what the field has done well, where it’s stuck, and what it could do next.
The argument
An era that has only just begun.
Should the community embrace this challenge and develop CIs for a range of computer science courses, these CIs could usher in a new era of curricular and pedagogical innovation and evaluation.
Dare we say, this era has begun.
Publication & follow-up
What came next.
The paper — Taking Stock of Concept Inventories in Computing Education: A Systematic Literature Review — was accepted at ACM ICER 2023.
The work didn’t end there. I’m now part of a follow-up project with Benji Xie and Murtaza Ali: a benchmarking tool for concept inventories, which was later presented at ACM ICER 2024 as Using Benchmarking Infrastructure to Evaluate LLM Performance on CS Concept Inventories.