A Bigger Catch: Fine-Grained Curriculum Standards Alignment on the MathFish Benchmark

Problem & Motivation

Most math benchmarks for LLMs (GSM8K, MATH) ask only one question: can the model get the right answer? In K-12 classrooms, the categorization of problems by curriculum standard is just as critical. Professional reviewers spend months mapping problems to fine-grained standards, and as LLMs get deployed into classroom tools, it matters whether they grasp this pedagogical structure and not just produce correct answers.

Tagging curriculum standards is hard to automate because standards sharing dense surface vocabulary can still encode subtle pedagogical distinctions that off-the-shelf embeddings can't separate.

Can a retrieval-and-reasoning pipeline align math problems to fine-grained Common Core standards anywhere near the precision human curriculum reviewers apply?

Solution

We evaluate a three-stage pipeline, plus a training-free alternative, on MathFish (Lucy et al., 2024), which frames curriculum alignment as multi-label prediction over 385 Common Core State Standards (CCSS) for Mathematics, organized in a four-level hierarchy (grade → domain → cluster → standard) with 1,040 conceptual connections.

Conference poster for A Bigger Catch: Fine-Grained Curriculum Standards Alignment on the MathFish Benchmark, summarizing the problem, four-stage pipeline (M1–M3, A1), results table, and analysis.

pipeline breakdown

M1: Contrastive Bi-Encoder with Hard Negatives

A bge-large-en-v1.5 bi-encoder trained with InfoNCE loss plus curriculum-informed hard negatives (40% same-cluster siblings, 30% ATC conceptual neighbors, 20% grade-adjacent, 10% random), sampled to target the confusion patterns MathFish's own error analysis reveals, retrieves the top-20 candidate standards per problem from a shared vector space.

M2: Cross-Encoder Re-Ranking

A roberta-base cross-encoder jointly attends over each (problem, standard) pair, re-scoring and re-ranking the bi-encoder's top-20 down to a top-5 final candidate set.

M3: ReAct Agent with LLM-as-Judge Critic

The cross-encoder's top-5 candidates are passed to a gemini-2.0-flash-001 ReAct agent, which iteratively reasons over the set via tool calls (fetching full standard descriptions, exploring sibling/conceptual neighbors on the ATC curriculum graph, or issuing new lexical queries) before an LLM-as-judge critic prunes to the final prediction.

A1: Training-Free Hybrid Retrieval + Graph Rerank

Replaces the trained bi-encoder and cross-encoder with a training-free hybrid retriever (BM25 + dense embeddings) reranked via GraphRAG over the ATC curriculum graph, reusing the same M3 agent and critic as a training-free comparison point.

Key Results

M3 achieves 31.3% exact match, roughly 6.5× the three-shot GPT-4-Turbo baseline (4.8%), on the full Addressing/Alignment-filtered MathFish development set (1,942 problems).

Method	Exact Match	Weak Acc	GraphDist ↓	Avg Pred
Three-shot GPT-4-Turbo baseline	0.048	0.502	1.90	3.05
M1: Bi-Encoder	0.000	0.728	4.24	20.00
M2: + Cross-Encoder Rerank	0.000	0.688	2.42	5.00
M3: + ReAct	0.313	0.589	0.93	1.37
A1: Hybrid + Graph Rerank + ReAct	0.275	0.541	1.047	1.438

Key Insight: Retrieval and reranking alone are necessary but not sufficient. M1 and M2 both score 0.000 exact match despite strong weak accuracy (≥0.688) and Recall@20 of 0.632, because they return fixed-size candidate sets (20 and 5) against a gold average of just 1.47 standards per problem. Attaching the ReAct agent and critic is what converts a ranked list into a precise, variable-size prediction set, cutting average predictions per problem from 5.00 to 1.37 and driving the leap from 0.000 to 0.313 exact match. A single-pass critic alone (no ReAct loop) already reaches 28.4%; the agentic reasoning loop adds a further +2.9 points on top.

Error Analysis

Even the best system tops out at 31.3% exact match. Three systematic failure modes explain most of the remaining gap:

Over half of M3's errors still register weak accuracy, and graph distance improves from 1.90 (GPT-4) to 0.93. The system is frequently in the right neighborhood even when it misses the exact gold label.

Implications

For practitioners, the pipeline supports two operating points: M2's top-5 ranked list (weak accuracy 0.688) enables a suggest-and-verify workflow where a reviewer scans a short candidate list rather than searching across all 385 standards; M3's variable-size output (exact match 0.313, graph distance 0.93) enables a draft-tagging workflow where the system proposes an initial alignment for a human to audit, since even incorrect M3 predictions typically land within the same or an adjacent curriculum cluster.

Curriculum alignment is a fundamentally hard reasoning problem, harder in some ways than the math itself. It requires inferring the pedagogical intent behind a problem and how that intent relates to structurally adjacent standards, a distinction that persists even for our best agentic system. We argue this calls for evaluation protocols beyond binary exact match, such as partial credit weighted by curriculum-graph proximity.