Identifying Critical Prompt Elements for Zero-Shot Detection of Notice and Wonder Routines: An Ablation Study of Large Language Model Biases in Classroom Discourse Analysis

The Notice and Wonder (N&W) routine is a valuable instructional approach that helps initiate mathematics lessons by encouraging students to make observations and pose questions about mathematical scenarios. While high-quality instructional materials like N&W routines benefit learning outcomes, evaluating their implementation typically requires labor-intensive classroom observations.

In this research collaboration with TeachFX, an educational technology company focused on classroom discourse analytics, we investigated whether large language models (LLMs) can detect these routines through zero-shot prompting, without domain-specific training.

Dataset Characteristics

Research Methodology

We employed what we termed an “inverse cumulative ablation” methodology to systematically test four prompt dimensions:

• Component Definition (descriptions of routine components)
• Component Recognition (linguistic markers and functional purposes)
• Structural Flexibility (guidance on order flexibility)
• Contextual Complexity (instructions for handling complex transcripts)

This approach allowed us to identify which elements most significantly contribute to model performance and what they reveal about LLM biases in classroom discourse analysis.

Key Findings

1. Our zero-shot prompting approach achieved promising detection capability (F1=0.864), though performance varied across routine components.

   	Time-Weighted Precision	Time-Weighted Recall	Time-Weighted F1
   Overall N&W Routine	0.967 ± 0.114	0.813 ± 0.198	0.864 ± 0.139
   Task Launch	0.707 ± 0.358	0.645 ± 0.312	0.589 ± 0.316
   Independent Think Time	0.829 ± 0.354	0.547 ± 0.493	0.546 ± 0.469
   Small Group Discussion	0.889 ± 0.133	0.622 ± 0.403	0.636 ± 0.383
   Whole Class Discussion	0.843 ± 0.249	0.736 ± 0.305	0.747 ± 0.284

   

2. We identified two critical biases in how LLMs process classroom discourse:

   • Utterance Dominance Bias: Models favor teacher-led discourse over student-centered activities

     

   • Form-over-Function Bias: Models prioritize linguistic patterns over pedagogical intentions

     

3. Component Recognition elements yielded the largest performance gains (+6.7%), suggesting that LLMs require explicit guidance on both linguistic markers and their instructional purposes to effectively analyze classroom discourse.

   

Implications

These findings demonstrate that while LLMs can effectively detect instructional routines, they have fundamental misalignment with classroom realities. For educational technology developers, this emphasizes the necessity of balancing technical performance with preserving authentic instructional flexibility.

The study highlights that effective automated feedback systems require expertise in both prompt engineering and pedagogical theory to avoid inadvertently standardizing teaching practices. By understanding these biases, we can develop more nuanced approaches that support teachers in implementing high-quality instructional materials while respecting the diversity of effective teaching approaches.