What can we learn from scholarship of teaching and learning (SoTL) project failures to improve programmatic support for future SoTL inquiry? In this episode, Jessie shares an open-access article by a team of educational developers at Carnegie Mellon who reflected on project failures to learn how they could better support SoTL scholars working on technology-enhanced learning interventions.
See our full episode notes at https://www.centerforengagedlearning.org/learning-from-sotl-failure/
What can we learn from scholarship of teaching and learning (SoTL) project failures to improve programmatic support for future SoTL inquiry?
In this episode, Jessie shares an open-access article by a team of educational developers at Carnegie Mellon who reflected on project failures to learn how they could better support SoTL scholars working on technology-enhanced learning interventions:
Hershock, Chad, Laura Ochs Pottmeyer, Jacqueline Pincus, H. Elisabeth Ellington, and Zach Mineroff. 2026. “Unpacking SoTL ‘Failure’: Lessons from a Fellowship Program Investigating Generative AI's Impacts.” Teaching & Learning Inquiry 14.
This episode was hosted, edited, and produced by Jessie L. Moore, Director of the Center for Engaged Learning and Professor of Professional Writing & Rhetoric.
60-Second SoTL is produced by the Center for Engaged Learning at Elon University.
Music: “Cryptic” by AudioCoffee.
Image in show art is by johnstocker on Magnific
(Music)
0:10
Jessie L. Moore:
What can we learn from SoTL project failures to improve programmatic support for future SoTL inquiry? That’s the focus of this week’s 60-second SoTL from Elon University’s Center for Engaged Learning. I’m Jessie Moore.
(Music)
0:30
In "Unpacking SoTL 'Failure': Lessons from a Fellowship Program Investigating Generative AI's Impacts,” Chad Hershock, Laura Ochs Pottmeyer, Jacqueline Pincus, Elisabeth Ellington, and Zach Mineroff argue that failure isn't simply a project that finds no significant results. Instead, failure occurs when researchers cannot meaningfully interpret or share what they learned because of problems with study design, implementation, or circulation. Their article appears in the open-access journal, Teaching & Learning Inquiry.
1:03
Drawing lessons from 32 projects in Carnegie Mellon's Generative AI Teaching as Research—or GAITAR—Fellowship Program, the authors identify potential failure points in scholarship of teaching and learning projects focused on technology-enhanced learning (TEL) interventions, and they suggest implications for educational developers supporting future SoTL efforts.
1:23
The authors note that success in their context entails educators collecting at least one measure of student performance in the studied course, analyzing and interpreting that data with attention to implications for future courses, and circulating the findings via presentation or publication. With that framing in mind, they suggest ways that educational developers can help educators minimize the potential of failure.
1:48
First, the authors advocate a project proposal process with iterative feedback cycles focused on clearly articulating a research question and viable study design. Because the projects in their fellowship program focus on learning interventions, the authors also note that they require participants to include a comparison group and they offer resources on comparison groups. The GAITAR fellowship does not accept proposals with comparison groups formed by student self-selection, and fellowship participants are encouraged to use within-subjects designs in which students experience multiple cycles of conditions.
2:23
Second, Hershock and colleagues moved to limiting proposals for the GAITAR fellowship to projects that use fully developed and tested tools with repeated piloting prior to the study. In essence, this requirement strives to reduce failure due to tool performance in studies of learning interventions.
2:41
Third, the authors advocate helping educators ensure that their course design includes individual learner deliverables that measure individual students’ learning. In projects involving GenAI, they also encourage fellowship participants to include a transfer task, completed after a delay, to explore whether improvement in student performance is an outcome of learning or associated with access to GenAI.
3:05
Fourth, the GAITAR fellowship incentivizes circulation of findings, even when the study leads to null results. The consultation team shares examples of peer-reviewed publications of studies with null results and facilitates writing accountability groups.
3:21
The strategies shared by the authors can strengthen both individual projects and the broader culture of SoTL. However, it’s worth noting that the strategies align with very specific disciplinary lenses. The focus on quasi-experimental designs with null hypotheses and the authors’ references to writing templates that follow an Intro-Methods-Results-Discussion format may result in missing the richness provided by other SoTL methods, such as those grounded in arts and humanities traditions. For that reason, I’d love to read parallel articles exploring what might constitute SoTL failure—and strategies to avoid it—from other big-tent SoTL perspectives.
To learn more about these strategies, visit our show notes for a link to the open access article.
4:04
(Music)
4:08
Jessie L. Moore:
Join us for our next episode of 60-second SoTL from Elon University’s Center for Engaged Learning for another snapshot of recent scholarship of teaching and learning. Learn more about the Center at www.CenterForEngagedLearning.org.
(Music)