Teaching Models to Read and Respond to a Learner's Reasoning

Abstract

When a learner makes a mistake with fractions or decimals, the reasoning they write down usually shows why. We ask whether a language model uses that evidence in its first reply: does it name the likely error, present it as a likely explanation rather than a certainty, credit what is correct, explain the idea and end with a next step? SENSUS Bench v1.0 measures this on a model's first and only reply to a learner's written work under a one-line tutor instruction, the setting a student or teacher meets when they bring a confused learner's work to a general-purpose model. It uses a sealed set of grade 4 and 5 cases that no model was trained on, each graded line by line against its own checklist by two model graders from different providers. We release SENSUS SFT, 1,015 synthetic training conversations, and use it to fine-tune low-rank adapters on three open models. Qwen3.8-27B trained on SENSUS scores 0.825 on average over two training runs, against 0.290 for the same base before training and 0.525 for GPT-6 Sol, the highest-scoring general-purpose model. A control trained on generic tutor replies to the same prompts gains far less, so the SENSUS replies as a package, not tutor-style fine-tuning in general, account for the improvement. Most of the fine-tuned model's lead lies in the targeted behaviours, which the general-purpose models' default replies rarely show; with the length, hedging and certainty rules set aside, it is level with Opus 5.5 and still ahead of GPT-6 Sol. At 27B the fine-tuned model also makes fewer false statements than its untuned base and almost no serious errors on these cases (none in the second training run, one in the first), while the smaller fine-tuned models make false statements more often than any general-purpose model.

1. Introduction

Ask a capable language model to help a ten-year-old who has decided that 4/10 is bigger than 3/6 because 4 is bigger than 3, and it will usually explain how to compare fractions correctly. It is less likely to say what went wrong in the child's own method. Feedback research suggests that this second step matters. Feedback works best when it addresses the gap between what the learner did and the goal (Hattie and Timperley, 2007), and it should come in pieces small enough to act on (Shute, 2008). Errors in elementary fractions and decimals are often systematic. Learners carry whole-number rules into fractions (Ni and Zhou, 2005) or judge a decimal by how many digits it has (Resnick et al., 1989; Steinle and Stacey, 2004). Learning research has long treated such errors as knowledge in transition (Smith et al., 1993).

SENSUS targets two specific failures. When a learner states an incorrect way of working, a model often does not name the learner's likely error, or it states its diagnosis as certain when it can only be likely. We define five behaviours for a reply to a learner's written reasoning. The first two address those failures directly, and the other three make the reply useful.

1. B1. Name a likely error tied to what the learner wrote.
2. B2. Hedge the diagnosis. The cause of an error is an inference about someone else's thinking.
3. B3. Credit what is correct.
4. B4. Explain or model the idea the error concerns.
5. B5. End with a next step the learner can take, usually a question.

A reply should also be correct and short. We drew these behaviours from research on feedback, misconceptions and responsive teaching (Section 3.3). SENSUS Bench turns them into case-specific checklist lines that measure observable properties of a single reply, and SENSUS SFT gives 1,015 examples of replies that show them. Figure 1 shows one case from start to finish.

SENSUS measures the first reply. Every model, fine-tuned or not, receives the same one-line tutor instruction and the learner's work, and answers once; no model sees the checklist or is told about B1 to B5. This is the setting that matters in practice. A student who pastes their working into a general-purpose model, or a teacher who tries one with a class, gets that model's default reply, and the first reply to a confused learner is where a diagnosis is offered or missed. The benchmark therefore asks three questions: whether general-purpose models show B1 to B5 by default, whether about a thousand examples teach an open model to show them, and what the resulting gain consists of.

We fine-tune three open models on SENSUS SFT and evaluate them, their untuned bases, matched generic controls and four general-purpose models on SENSUS Bench. Five findings stand out.

1. About a thousand examples teach the behaviour. Fine-tuning Qwen3.8-27B on SENSUS SFT nearly triples its benchmark score, and a control trained on generic tutor replies to the same prompts gains far less (Section 6.2).
2. A 27B open model outscores the default replies of every general-purpose model we tested. It leads on all 12 standards and beats GPT-6 Sol on 68 / 3 / 5 cases (wins / losses / ties) under the same one-line tutor instruction (Figure 2).
3. The 27B model also became more accurate. After fine-tuning, more of its replies are free of false statements, and replies with a listed serious error fall from 26 of 76 to one or none (Section 6.2).
4. The gain lies in the targeted behaviours. Fine-tuning moves naming, hedging and the next step (Figure 4). Rescoring without the length, hedging and certainty rules puts Opus 5.5 level with the fine-tuned 27B model and leaves GPT-6 Sol behind (Section 7).
5. Style can outrun accuracy in smaller models. At 8B and 14B the fine-tuned models adopt the reply style fully but state something false in 51% and 30% of replies.

Because no model was told about B1 to B5, these comparisons describe default behaviour: what a learner gets from each model as it ships, under a short tutor instruction. Prompting alone did little in the one prompting test we ran (Section 6.3).

The paper makes four contributions.

1. SENSUS Bench v1.0. 76 cases over 12 Common Core standards (NGA and CCSSO, 2010), with 549 checklist lines and 218 serious errors. The two graders agree on 94.7% of line judgments (κ = 0.89).
2. SENSUS SFT. 1,015 conversations under CC BY 4.0. No question is shared with the benchmark, and the release includes checksums and an offline verifier.
3. Training results. Fine-tuning results at 8B, 14B and 27B with matched generic controls and two training runs per recipe, and the change in false statements and serious errors at each size (Section 6.2).
4. A breakdown of the gain. Rescoring the recorded marks under seven rules separates the behavioural gain from mathematical content and shows what fine-tuning cost in accuracy.

2. Related Work

Tutoring benchmarks. MathDial collects teacher moves in dialogues with a model that simulates student errors (Macina et al., 2023). Bridge records the decisions expert tutors make when a student errs, and shows that models improve when given those decisions (Wang et al., 2024b). The taxonomy of Maurya et al. (2025), which includes mistake identification and providing guidance, underlies the BEA 2025 shared task (Kochmar et al., 2025). MathTutorBench (Macina et al., 2025), stepwise verification before remediation (Daheim et al., 2024) and evaluation programs for educational assistants (Jurenka et al., 2024) cover broader tutoring skills, and TutorBench scores explanations, feedback and hints against rubrics written for each sample (Srinivasa et al., 2025). SENSUS is narrower. Its replies are single turns, and its target is a hedged diagnosis drawn from the learner's own words. Case-specific rubrics are not new. Our contribution is to pair them with this reply policy, an open training set, same-base training comparisons and an analysis of how the scoring rules shape the results.

Misconception diagnosis. Work on diagnosing misconceptions with language models has mostly addressed algebra (Otero et al., 2025), and models have been used to simulate students who hold specific misconceptions (Sonkar et al., 2024). We study fractions and decimals, and we ask the model to act on its diagnosis in the reply.

Fine-tuning tutors on synthetic data. TutorChat and SocraticLM fine-tune on large synthetic tutoring corpora (Chevalier et al., 2024; Liu et al., 2024), and Petukhova et al. (2026) add preference optimization for mistake remediation. Synthetic instruction data is now common practice (Wang et al., 2023; Long et al., 2024). LIMA showed that around a thousand examples can set a model's response style (Zhou et al., 2023), and imitation data can change style more than capability (Gudibande et al., 2024). Our results are consistent with both. We train low-rank adapters (Hu et al., 2022), which tend to forget less of the base model than full fine-tuning does (Biderman et al., 2024).

Feedback and teaching research. Research on feedback (Hattie and Timperley, 2007; Shute, 2008; Kluger and DeNisi, 1996), formative assessment (Black and Wiliam, 1998), student errors (Brown and Burton, 1978) and teacher noticing (Jacobs et al., 2010) shaped the reply policy. Section 3.3 sets out how each behaviour and checklist line follows from it.

Model graders. Model judges approximate human preferences in many settings (Zheng et al., 2023; Chiang and Lee, 2023). They are also sensitive to answer order (Wang et al., 2024a), favour their own outputs (Panickssery et al., 2024), and respond to length and style (Dubois et al., 2024; Feuer et al., 2025). Binary checklists make judgments more reliable (Lee et al., 2025; Ye et al., 2024), and conversation-specific rubrics with penalized criteria have been used at scale in health (Arora et al., 2025). We use both ideas, and Sections 7 and 8 measure how far our results depend on style.

3. SENSUS Bench v1.0

3.1 Cases

Each case is one teaching moment. It has a mathematics question labelled with its grade band and Common Core standard, a short description of what a synthetic learner did and said, the decision the reply must make, one reference reply, and a checklist of yes-or-no lines that define a good reply. The 76 cases cover 12 standards in grade 4 fractions and grade 5 place value and decimals (Tables 1 and 7). They were sealed before the first training run and excluded from training (Section 10). The model sees the system message “You are a mathematics tutor. A student is working on the question below. Reply to the student directly, as you would in the conversation.”, then a user message with the question and the learner's work, and replies once. The benchmark scores that single reply; there is no second turn. Appendix B gives two cases in full.

3.2 Checklists and Serious Errors

Every case has 7 to 8 binary checklist lines. The first line requires that the reply say nothing false. One line asks for a hedged diagnosis (B2), one for a next step (B5), and one for about 120 words or fewer, following the advice to give feedback in manageable units (Shute, 2008). The remaining lines are about the case itself. A line may require naming the error of comparing numerators and denominators as separate whole numbers, for example, or showing that 4/10 is less than 3/6. Each case also lists 2 to 3 serious errors. Every case counts a diagnosis stated as certain as a serious error, and about two thirds also list endorsing a wrong answer. The word limit and the certainty rule are design choices that encode brevity and B2, and Section 7 reports results without them. A listed serious error and a false statement are scored differently. A reply can state something false without triggering any listed serious error; it then loses the correctness line but keeps partial credit, as Case 1 in Appendix B shows. We therefore report correctness separately and test a stricter rule in Section 7. Table 2 summarises what the checklist rewards.

3.3 Why the Checklist Asks for These Behaviours

We wrote the behaviours, and the checklist lines that test them, from research on feedback, mathematical misconceptions and responsive teaching. Mathematics education describes a teacher's response to a student's work as professional noticing: attending to the student's strategy, interpreting what it shows about their understanding, and deciding how to respond (Jacobs et al., 2010). B1 and B2 are the interpreting step, and B3 to B5 are the response. The checklist keeps only what can be seen in one written reply. Table 2 lists the main sources for each element.

Naming the likely error (B1). Errors in fractions and decimals tend to be systematic (Section 1). Brown and Burton (1978) modelled such errors as consistent faults, or bugs, in an otherwise working procedure, which is why the cause of an error can often be read from the work. Tutors do not always do this: in two samples of one-to-one sessions with non-expert tutors, the tutors worked from a rough sense of what the student knew and rarely diagnosed specific errors (Graesser et al., 1995). Research also suggests that studying and comparing incorrect and correct examples can support learning: students who compared incorrect and correct decimal examples, with prompts, gained more understanding and kept fewer misconceptions than students who studied correct examples only (Durkin and Rittle-Johnson, 2012). That study tested worked examples, not a tutor naming a learner's error; B1 draws on that motivation by asking the reply to identify the specific likely error, tied to what the learner wrote, rather than restate the correct rule.

Hedging the diagnosis (B2). A wrong answer does not settle its cause, since different faulty procedures can produce the same answer (Brown and Burton, 1978). A diagnosis drawn from one written explanation is an inference, and B2 asks the reply to present it as one (“it looks like you may be…”). Stating the cause tentatively lets the learner confirm or correct it, which keeps the exchange formative (Black and Wiliam, 1998). It also keeps the feedback on the task rather than on the learner; in a large review of feedback studies, feedback that drew attention to the self tended to lower performance (Kluger and DeNisi, 1996). B2 is our design choice built on these findings, and Section 7 reports scores without it.

Crediting what is correct (B3). Productive teaching builds on what the student already thinks rather than replacing it (Leatham et al., 2015), and misconceptions usually grow out of knowledge that is partly right (Smith et al., 1993). Feedback is most useful when it addresses the task and the process the student used (Hattie and Timperley, 2007). B3 lines therefore require a specific correct step, such as the learner's correct renaming of a fraction; a general “Good job” is marked no.

Explaining the idea (B4). Conceptual understanding and procedural skill develop together, each supporting the other, as shown with grade 5 and 6 students learning decimals (Rittle-Johnson et al., 2001). The U.S. Department of Education's practice guide on fractions recommends measurement representations such as number lines and explaining why procedures work (Siegler et al., 2010). B4 lines ask for a reason a grade 4 or 5 learner can check, such as comparing to one half, a drawing or a common denominator. Stating which number is bigger does not count.

Ending with a next step (B5) and keeping it short. Feedback should tell the learner where to go next (Hattie and Timperley, 2007), and prompting learners to explain their own reasoning improves understanding (Chi et al., 1994). B5 asks the reply to end with a question or task the learner can answer; a trailing “Does that help?” alone does not count. The length line follows the advice to give feedback in manageable units (Shute, 2008).

Correctness and serious errors. Feedback does not help automatically. In Kluger and DeNisi's (1996) review, more than a third of feedback interventions lowered performance, which is why the checklist treats some replies as failures rather than weaker successes. A reply that says something false fails the correctness line. A reply that endorses a wrong answer or states the diagnosis as certain commits a serious error and scores zero. The zero is our scoring choice: we treat confirming a learner's error and asserting a guess about their thinking as the two moves most likely to mislead them. Section 7 rescores the replies without the certainty rule and with a stricter correctness rule.

The checklist measures properties of a single written reply. Within a case, its lines count equally when no serious error is flagged, and a serious-error flag overrides them and sets the score to zero.

3.4 Metrics

For a reply to case c with n_c checklist lines, let m_c be the number of lines that both graders mark as met. The reply's score is

score_c = 0 if either grader marks any serious error, and m_c / n_c otherwise.

We report four quantities for each model. Mean score is the mean of score_c over the 76 cases, so every case counts equally whether it has seven or eight lines. Full passes counts cases with score_c = 1, meaning every line met and no serious error. Serious errors counts replies in which either grader flags at least one listed serious error. No false statement is the share of replies whose correctness line both graders mark as met. For pairwise preference judgments, net preference is (wins − losses) / resolved comparisons, where ties count as resolved. The reference replies score 0.991 on average, which shows that the checklists can be satisfied.

4. Training Data and Fine-Tuning

4.1 SENSUS SFT

SENSUS SFT v1.1 has 1,015 synthetic conversations over 884 distinct questions. Each conversation has the benchmark's system message, a user message with the question and the learner's work, and the reference reply as the assistant turn. 98% of the replies contain a hedging phrase, 97% end with a question, and none is longer than 109 words (Table 13). Cases are kept in groups by question, and each group goes wholly to training or wholly to evaluation. No evaluation question matches a training question exactly (Section 4.3). The benchmark and the training set share their format, standards and kinds of misconception by design, so SENSUS measures in-distribution learning of a behaviour.

4.2 Generic Control

A control set answers 1,012 of the same prompts (three prompts have no generic reply) with correct, clear, generic tutoring replies that never diagnose the learner. Its replies are short like the SENSUS replies, so training on it shows what fine-tuning on tutor-shaped text achieves without the diagnosis. Because it leaves out B1 and B2, generic training also pushes those behaviours below the untuned base (Figure 4). The control differs from SENSUS in several behaviours at once, so the comparison shows the effect of the whole SENSUS policy and not of any single behaviour. We therefore report the comparison with the untuned base alongside the comparison with the control.

4.3 Overlap with the Benchmark

No benchmark question appears in the training file (0 exact matches after normalising case and punctuation). Questions in this domain share stock phrasing. 26 of 76 benchmark questions share at least one word 8-gram with some training question, and the largest share of one question's 8-grams found in a single training question is 33%. We compared questions only (Section 10).

4.4 Recipe

The base models are Qwen3-8B and Qwen3-14B (Yang et al., 2025) and Qwen3.8-27B, a newer generation of the same family (Table 3). Each model is trained with supervised fine-tuning on the conversations. It reads the instruction, the question and the learner's work, and learns to produce the reference reply. Training updates a rank-32 LoRA adapter on the frozen base for eight epochs at a learning rate of 0.0003 (Table 3).

Eight epochs on about a thousand rows is an aggressive setting, and the fine-tuned replies are formulaic as a result: 97% of the 27B model's replies contain a hedging phrase, and a single two-word opening (“You correctly”) begins 24% of them. This is intended; Section 7 measures how much of the score that shared style accounts for.

5. Evaluation Setup

5.1 Models

We evaluate the fine-tuned models, the untuned bases, the generic controls and four general-purpose models (Table 4). Every model receives the same system and user messages. None sees the checklist, and none is told about B1 to B5, so every score describes a first reply under the same minimal instruction (Section 1). Each general-purpose model was evaluated once, with the inference settings in Table 4. Unless stated otherwise, fine-tuned results use the second (most recent) run of each recipe.

5.2 Graders

Grader 1 is Opus 5.5 and grader 2 is GPT-6.1 Sol at medium reasoning effort. Each grader works alone. It sees the case as the model saw it, the checklist, the serious errors and one reply, with no indication of which model wrote it, and marks every line true or false (Appendix A). Each reply was graded once. Grader 2 shares a provider with GPT-6 Sol, and for replies written by Opus 5.5 an alternate first grader from its provider replaced Opus 5.5; Section 8.2 reports each grader separately.

5.3 Statistics

Intervals on means are 95% percentile bootstrap intervals with 10,000 resamples (Efron and Tibshirani, 1993), and intervals on pass rates are Wilson intervals (Wilson, 1927). Differences between models resample cases jointly. Because cases cluster by standard, Table 6 also resamples whole standards. Case-level wins and losses use a two-sided exact sign test that excludes ties, with Holm correction across the comparisons in that table (Holm, 1979; Dror et al., 2018). Grader agreement is reported as raw agreement and Cohen's kappa (Cohen, 1960). The intervals cover sampling over cases only; variation between the two runs of each recipe was 0.014 to 0.027, and with 76 cases only large differences are reliable (Card et al., 2020). The analysis plan of each run, including 27B against 14B as the primary comparison, was fixed before the run started.

6. Results

6.1 Main Results

All three fine-tuned models score above every general-purpose model (Table 5 and Figure 2). Qwen3.8-27B+SENSUS fully passes 24 of 76 cases and has no serious error in the second run (one in the first), while no general-purpose model fully passes any case. GPT-6 Sol writes the shortest general-purpose replies (54 words on average) and makes no false statements. Opus 5.5, Nemotron 3 Ultra and Kimi K3 average 178 to 225 words, and 18 to 28 of each model's 76 replies are flagged for a serious error, almost all for a diagnosis stated as certain.

Figure 3 shows the score of every case, not only the means.

Against GPT-6 Sol, the fine-tuned 27B model wins 68 / 3 / 5 cases (wins, losses, ties), a mean difference of +0.313 [+0.263, +0.363]. Every comparison in Table 6 holds after Holm correction. The model scores above every general-purpose model on each of the 12 standards (Table 7) and in each third of the cases ranked by grader agreement (Table 8).

6.2 Training Effects

Fine-tuning on SENSUS raises Qwen3.8-27B from 0.290 to 0.825 on average over two runs (0.811 and 0.838), a pooled gain of +0.534 [+0.476, +0.591]; tables and figures show the second run (0.838) unless stated otherwise. The generic control scores 0.445, and against it the pooled gain is +0.379 [+0.338, +0.421] at 27B and +0.323 [+0.272, +0.374] at 14B, so the SENSUS replies as a whole teach far more than generic tutor replies do (Tables 9 and 10). The 27B model also became more accurate on these cases. The share of replies both graders marked free of false statements rose from 75% untuned to 91% for the second SENSUS run, and replies with a listed serious error fell from 26 of 76 to 0 in the second run and one in the first (Table 10). The generic control also removes most serious errors (1 of 76), so part of that drop comes from fine-tuning on short tutor replies in general. The pre-specified primary comparison, 27B against 14B, is +0.111 [+0.070, +0.152]. The 27B base is also a newer generation, so this difference reflects the base model as a whole and not only its size.

Figure 4 shows which behaviours changed. At 27B the largest gains are in naming the error (B1, 0.564 to 0.846) and hedging it (B2, 0.304 to 0.913), the two failures SENSUS targets. The figure also shows what the general-purpose models do well. Opus 5.5 and Kimi K3 credit the learner's work and explain the idea more often than the fine-tuned model, and fall behind on naming the error, hedging it and ending with a next step.

6.3 Development History

The released recipe was settled over several development rounds on Qwen3-8B, and each round was scored on the same 76 cases (Table 11). The first fine-tuned model, trained for three epochs at rank 16, scored 0.280. The final settings raised the same 502 rows to 0.580. Adding rows helped up to the released 1,015 but not beyond (Figure 5). The 884-row set scored 0.578, and an extended 1,232-row set scored 0.643 at 8B and 0.814 and 0.809 at 27B, no better than the released set. Five SENSUS examples in the prompt, with no training, scored 0.312, a small gain over the untuned 0.239 compared with fine-tuning. Two further attempts did not help and are not part of the released recipe: direct preference optimization on the fine-tuned 8B model (Rafailov et al., 2023), tried twice, scored −0.004 [−0.052, +0.043] against the SFT model it started from, and two 27B runs with 345 further training families scored no better than the released set, with differences inside the run-to-run gaps. The cases were sealed before the first run and never trained on, and each experiment's decision rule was fixed before the run; but every round was scored on them, eight rounds by the final 27B run, and the released data size and recipe follow from those scores. The main results are therefore results on a sealed set that was examined repeatedly, not on a fresh test set (Section 10).

7. What the Gain Consists Of

The training replies and the checklist share a style that the general-purpose models' default replies lack. To separate that style from mathematical content, we rescored every recorded mark under the rules in Table 12. No reply was generated or graded again. Line kinds come from each line's wording. Every case has one correctness line, one word-count line, one hedging line and one next-step line, and the other 245 lines concern the content of the case.

Removing any one style rule leaves the fine-tuned 27B model well ahead. Its lead over Opus 5.5 is +0.279 [+0.198, +0.363] without the word-count line, +0.295 without the hedging lines and +0.245 without the certainty rule. Removing all three together brings Opus 5.5 level with the fine-tuned model (−0.021 [−0.072, +0.029]), while the 27B model stays ahead of GPT-6 Sol by +0.208 [+0.155, +0.261] (the GPT-6 Sol differences are not shown in Table 12). Also removing the next-step line leaves mostly correctness and case content. On those lines Opus 5.5 leads (difference −0.098 [−0.151, −0.045]), Kimi K3 scores 0.814 against 0.797, and the 27B model still leads GPT-6 Sol by +0.092 [+0.028, +0.155].

Most of the gap between the fine-tuned and general-purpose models therefore lies in B2 (the hedging line and the certainty rule) and brevity, and to a lesser extent in B5. The benchmark targets these on purpose; they are what a default reply most often lacks. On content lines a 27B open model trained on about a thousand examples trails Opus 5.5, which explains more and credits more of the learner's work (Figure 4), and stays ahead of GPT-6 Sol. This fits the view that small supervised datasets mainly teach a model how to respond (Zhou et al., 2023; Gudibande et al., 2024). Table 13 shows the style difference directly.

8. Measurement Checks

8.1 Agreement Between Graders

Before marks are combined, the graders agree on 94.7% of 4,392 line judgments (κ = 0.89) and on 97.2% of serious-error judgments (κ = 0.72, lower because serious errors are rare). Agreement varies by line kind (Table 14). It is highest on the word-count line, as expected. Among the checklist lines, agreement beyond chance is lowest on the correctness line (κ = 0.79), the line that matters most for learners. Table 15 gives the full confusion matrix. Taking grader 1 as the reference, grader 2 marks a line as met with 95.5% precision and 96.2% recall.

8.2 Each Grader Alone

The fine-tuned models rank above GPT-6 Sol under either grader (Table 16). Requiring both graders to agree costs the long-reply models more. It lowers Opus 5.5 by 0.064 and Kimi K3 by 0.068, against 0.011 for the fine-tuned 27B model. Grader 1 is more generous than grader 2 to Opus 5.5 (0.575 against 0.474). For Opus 5.5, grader 1 is the alternate grader from its own provider, so the gap may reflect same-provider leniency. Grader 1 is also more generous to Kimi K3 and Nemotron 3 Ultra but not to the untuned 27B model, so the pattern is mixed. GPT-6 Sol scores about the same under the grader from its own provider (0.565) as under the other (0.552).

8.3 Open Preference

A checklist measures named properties, so we also compared the fine-tuned 27B model with each general-purpose model in blind pairwise judgments without a checklist. These used 142 additional cases that are not in the benchmark, and the first 27B training run (Table 17). Judges preferred the fine-tuned model over GPT-6 Sol and Nemotron 3 Ultra. Against Kimi K3 the two judges disagreed in sign, and the combined result is near zero. Judges preferred Opus 5.5 overall, but the preference came mostly from the judge from the same provider as Opus 5.5 (−0.718 for the fine-tuned model), while GPT-6.1 Sol was close to even (+0.049). The independent third judge also preferred Opus 5.5. On these cases 13 Opus 5.5 replies failed the automatic code checks, against 4 for the fine-tuned model. On the benchmark cases, giving the judges the checklist reverses the Opus 5.5 comparison in favour of the fine-tuned model (Table 18). When Opus 5.5 was told to keep to 120 words, open preference for it became stronger, so reply length does not explain why judges favour it. An open judgment and the SENSUS checklist reward different things, and we do not yet know why they diverge.

8.4 Targeted Edits

We edited Opus 5.5 replies to reveal the full answer, or to state a certain diagnosis without a closing question, and asked three model judges to compare each edit with the original. They preferred the original over the answer-revealing edit in 66, 67 and 70 of 76 cases, and over the certain, question-free edit in 69, 71 and 71 cases. The edits change several features at once and are longer than the originals, so this shows that the judges notice the edits, not which single property they respond to.

9. Failure Analysis

Table 19 counts how each model's replies fail. The fine-tuned 27B model's most common failure is missing a content line (55% of replies), then missing the next step (26%) and making a false statement (9%). In the second run it never states a diagnosis as certain. The general-purpose models fail in a different pattern. GPT-6 Sol misses the next step in 96% of replies, and Opus 5.5 and Kimi K3 state a diagnosis as certain in 24% and 36% of replies.

Headroom. Table 20 lists the checklist lines that the fewest models meet. Most ask the reply to end with a specific next task tied to the case rather than a general question, which is where even the fine-tuned model still falls short.

Correctness. Both graders mark the reply free of false statements for 91% of the fine-tuned 27B model's replies, 70% at 14B and 49% at 8B, against 100% for GPT-6 Sol and 96% for Opus 5.5. Under the published rule a false statement costs one checklist line. If any false statement instead counts as a serious error (Table 12, second row), the 27B model scores 0.783 and stays first. The 14B model falls to 0.558, close to GPT-6 Sol, and the 8B model to 0.400, below it. At 27B, fine-tuning reduced false statements relative to the untuned base (from 75% of replies free of false statements before training to 91% after; Section 6.2). The smaller models learned the shape of a good reply but often got its mathematics wrong. Shorter replies may also give fewer chances to make a false statement. Case 1 in Appendix B shows a 27B reply that is diagnostic in form but states that 4/10 is one half.

10. Limitations

• Default replies only. By design, SENSUS scores each model's first reply under a one-line tutor instruction, and no model is told about B1 to B5. Section 7 shows that most of the general-purpose models' gap lies in those behaviours. Whether a general-purpose model prompted with B1 to B5 would show them, and how reliably, is a separate question that this paper does not answer.
• Model-only, synthetic evaluation. The cases, learners and training data are synthetic, and every judgment is made by a model. The benchmark measures specified reply properties and not learning outcomes: this study does not test whether a reply that meets the checklist helps a learner learn more. The research the behaviours draw on comes mostly from human teachers working with students over time, and we know of no study showing that hedged wording itself improves learning. The agreement figures in Section 8.1 measure agreement between two models, not accuracy, and the line judgments are nested within replies and cases, so they are not independent observations; among the checklist lines, agreement beyond chance is lowest on correctness.
• Scoring rules. A confident diagnosis scores zero while some false mathematical statements keep partial credit, and the relative severity of the two has not been validated. The reference replies were prepared alongside the checklists, so their near-perfect score is not independent evidence that the checklists are valid.
• A sealed set examined repeatedly. The 76 cases were sealed before the first training run, excluded from training and graded blind, and each experiment's decision rule was fixed before the run; a separate set of 30 development cases, graded by one grader, served as an advisory check of fit. The sealed cases were nevertheless scored after every development round, and the released data size and recipe follow from those scores (Section 6.3). The cases were excluded from training but examined repeatedly, so they now serve as historical evaluation cases rather than a fresh test set, and a fresh set is needed to confirm the headline results. The analysis plan was fixed before each run but not publicly pre-registered, and the bootstrap intervals do not account for the repeated examination.
• In-distribution test. The benchmark shares its format, standards and kinds of misconception with the training data. The overlap check compared questions only; learner descriptions, replies and semantic similarity were not checked, so near-duplicates in the full examples are not ruled out, which falls short of the contamination checks Sainz et al. (2023) recommend.
• Accuracy and general ability. The fine-tuned models make false statements more often than GPT-6 Sol and Opus 5.5, most of all at 8B and 14B, and ability on other tasks was not measured.
• Training records. Several training settings were service defaults that were not recorded, and the general-purpose models ran with different provider defaults for reasoning. Inference settings are not identical across model families, and those differences, together with differences in model generation, limit comparisons across families. Two runs per recipe and one per control limit what can be said about variance.
• Grader overlap. One grader shares a provider with GPT-6 Sol, and Opus 5.5 was graded by an alternate grader from its own provider.

11. Ethics and Broader Impact

SENSUS concerns software that responds to children's mathematical work. Every learner and situation in it is synthetic, and no personal data is involved. A diagnostic tone can make a wrong reply more convincing, and Section 9 shows that the fine-tuned models are not free of false statements. Hedging expresses uncertainty about a diagnosis; it does not make the diagnosis correct. The dataset card therefore rules out using the data as a sole source of instruction for children without independent review. All evaluation is by models, so the results should not be presented as evidence of learning gains. Showing that would need a study with learners.

12. Release and Reproducibility

SENSUS SFT v1.1 is released under CC BY 4.0 with a dataset card, SHA-256 checksums and an offline verifier. The evaluation kit contains the 76 cases, a sampling script, the grading script with the instruction in Appendix A, and the recorded replies of the eight leaderboard models with both graders' marks. Its grading script recomputes those eight scores offline. The training recipe and the aggregate results of every run are released too, and every statistic for the eight leaderboard models is computed from these public files with a fixed random seed. Individual replies are released only for the eight leaderboard models. The first 27B run, the generic controls, the untuned 8B and 14B models, the development rounds and the 142-case preference test are released as aggregate results, so those numbers can be checked against the published results but not recomputed reply by reply. Recomputing scores from the recorded marks is not the same as regenerating replies or retraining the adapters. Some training settings were service defaults (Table 3), and the generic control set is not released, so the control comparison cannot be retrained from public files. The training data and the evaluation kit are available at huggingface.co/datasets/Lorm-Labs/sensus-bench-v1 and from the Lorm website.

Dataset and training guideBrowse the evaluation cases

13. Conclusion and Future Work

About a thousand open training examples teach a 27B model to reply to a learner's reasoning in a way that strong general-purpose models seldom do by default. The fine-tuned model names the likely error, hedges it, keeps the reply short and asks the next question, and a generic control trained on the same prompts does not learn this. On the content lines of each case it trails Opus 5.5 and stays ahead of GPT-6 Sol. At 27B, fine-tuning also made the replies more accurate on these cases, though the smaller models paid for the new style with more false statements. Our next experiments will prompt general-purpose models with B1 to B5, to learn whether the behaviours are absent or only not elicited, and test on fresh cases from a separate source. We will also adopt the stricter correctness rule of Table 12 for the next version of the benchmark, measure general capability after fine-tuning and add a grader from a third provider.

References

1. Arora, R. K., Wei, J., Hicks, R. S., et al. (2025). HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint. https://arxiv.org/abs/2505.08775
2. Biderman, D., Portes, J., Gonzalez Ortiz, J. J., et al. (2024). LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research. https://arxiv.org/abs/2405.09673
3. Black, P., and Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
4. Brown, J. S., and Burton, R. R. (1978). Diagnostic Models for Procedural Bugs in Basic Mathematical Skills. Cognitive Science, 2(2), 155–192. https://doi.org/10.1207/s15516709cog0202_4
5. Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., and Jurafsky, D. (2020). With Little Power Comes Great Responsibility. Proceedings of EMNLP 2020, pp. 9263–9274. https://doi.org/10.18653/v1/2020.emnlp-main.745
6. Chevalier, A., Geng, J., Wettig, A., et al. (2024). Language Models as Science Tutors. Proceedings of ICML 2024, PMLR 235, pp. 8310–8335. https://arxiv.org/abs/2402.11111
7. Chi, M. T. H., de Leeuw, N., Chiu, M.-H., and LaVancher, C. (1994). Eliciting Self-Explanations Improves Understanding. Cognitive Science, 18(3), 439–477. https://doi.org/10.1207/s15516709cog1803_3
8. Chiang, C.-H., and Lee, H.-y. (2023). Can Large Language Models Be an Alternative to Human Evaluations? Proceedings of ACL 2023, pp. 15607–15631. https://doi.org/10.18653/v1/2023.acl-long.870
9. Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
10. Daheim, N., Macina, J., Kapur, M., Gurevych, I., and Sachan, M. (2024). Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors. Proceedings of EMNLP 2024, pp. 8386–8411. https://doi.org/10.18653/v1/2024.emnlp-main.478
11. Dror, R., Baumer, G., Shlomov, S., and Reichart, R. (2018). The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. Proceedings of ACL 2018, pp. 1383–1392. https://doi.org/10.18653/v1/P18-1128
12. Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. Conference on Language Modeling (COLM 2024). https://arxiv.org/abs/2404.04475
13. Durkin, K., and Rittle-Johnson, B. (2012). The Effectiveness of Using Incorrect Examples to Support Learning about Decimal Magnitude. Learning and Instruction, 22(3), 206–214. https://doi.org/10.1016/j.learninstruc.2011.11.001
14. Efron, B., and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC. https://doi.org/10.1201/9780429246593
15. Feuer, B., Goldblum, M., Datta, T., et al. (2025). Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking. International Conference on Learning Representations (ICLR 2025). https://arxiv.org/abs/2409.15268
16. Gebru, T., Morgenstern, J., Vecchione, B., et al. (2021). Datasheets for Datasets. Communications of the ACM, 64(12), 86–92. https://doi.org/10.1145/3458723
17. Graesser, A. C., Person, N. K., and Magliano, J. P. (1995). Collaborative Dialogue Patterns in Naturalistic One-to-One Tutoring. Applied Cognitive Psychology, 9(6), 495–522. https://doi.org/10.1002/acp.2350090604
18. Gudibande, A., Wallace, E., Snell, C., et al. (2024). The False Promise of Imitating Proprietary Language Models. International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2305.15717
19. Hattie, J., and Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
20. Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
21. Hu, E. J., Shen, Y., Wallis, P., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685
22. Jacobs, V. R., Lamb, L. L. C., and Philipp, R. A. (2010). Professional Noticing of Children’s Mathematical Thinking. Journal for Research in Mathematics Education, 41(2), 169–202. https://doi.org/10.5951/jresematheduc.41.2.0169
23. Jurenka, I., Kunesch, M., McKee, K. R., et al. (2024). Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach. arXiv preprint. https://arxiv.org/abs/2407.12687
24. Kluger, A. N., and DeNisi, A. (1996). The Effects of Feedback Interventions on Performance: A Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory. Psychological Bulletin, 119(2), 254–284. https://doi.org/10.1037/0033-2909.119.2.254
25. Kochmar, E., Maurya, K. K., Petukhova, K., Srivatsa, K. A., Tack, A., and Vasselli, J. (2025). Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors. Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 1011–1033. https://doi.org/10.18653/v1/2025.bea-1.77
26. Leatham, K. R., Peterson, B. E., Stockero, S. L., and Van Zoest, L. R. (2015). Conceptualizing Mathematically Significant Pedagogical Opportunities to Build on Student Thinking. Journal for Research in Mathematics Education, 46(1), 88–124. https://doi.org/10.5951/jresematheduc.46.1.0088
27. Lee, Y., Kim, J., Kim, J., et al. (2025). CheckEval: A Reliable LLM-as-a-Judge Framework for Evaluating Text Generation Using Checklists. Proceedings of EMNLP 2025, pp. 15771–15798. https://doi.org/10.18653/v1/2025.emnlp-main.796
28. Liu, J., Huang, Z., Xiao, T., et al. (2024). SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). https://proceedings.neurips.cc/paper_files/paper/2024/hash/9bae399d1f34b8650351c1bd3692aeae-Abstract-Conference.html
29. Long, L., Wang, R., Xiao, R., et al. (2024). On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. Findings of ACL 2024, pp. 11065–11082. https://doi.org/10.18653/v1/2024.findings-acl.658
30. Macina, J., Daheim, N., Chowdhury, S. P., et al. (2023). MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems. Findings of EMNLP 2023, pp. 5602–5621. https://doi.org/10.18653/v1/2023.findings-emnlp.372
31. Macina, J., Daheim, N., Hakimi, I., Kapur, M., Gurevych, I., and Sachan, M. (2025). MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors. Proceedings of EMNLP 2025, pp. 204–221. https://doi.org/10.18653/v1/2025.emnlp-main.11
32. Maurya, K. K., Srivatsa, K. A., Petukhova, K., and Kochmar, E. (2025). Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors. Proceedings of NAACL 2025, pp. 1234–1251. https://doi.org/10.18653/v1/2025.naacl-long.57
33. NGA Center and CCSSO (National Governors Association Center for Best Practices and Council of Chief State School Officers) (2010). Common Core State Standards for Mathematics. Washington, DC. https://www.thecorestandards.org/Math/
34. Ni, Y., and Zhou, Y.-D. (2005). Teaching and Learning Fraction and Rational Numbers: The Origins and Implications of Whole Number Bias. Educational Psychologist, 40(1), 27–52. https://doi.org/10.1207/s15326985ep4001_3
35. Otero, N., Druga, S., and Lan, A. (2025). A Benchmark for Math Misconceptions: Bridging Gaps in Middle School Algebra with AI-Supported Instruction. Discover Education, 4, Article 277. https://doi.org/10.1007/s44217-025-00742-w
36. Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). https://arxiv.org/abs/2404.13076
37. Petukhova, K., Nguyen, T. D., and Kochmar, E. (2026). Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation. arXiv preprint. https://arxiv.org/abs/2606.21502
38. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2023). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. Advances in Neural Information Processing Systems 36. https://arxiv.org/abs/2305.18290
39. Resnick, L. B., Nesher, P., Leonard, F., Magone, M., Omanson, S., and Peled, I. (1989). Conceptual Bases of Arithmetic Errors: The Case of Decimal Fractions. Journal for Research in Mathematics Education, 20(1), 8–27. https://doi.org/10.5951/jresematheduc.20.1.0008
40. Rittle-Johnson, B., Siegler, R. S., and Alibali, M. W. (2001). Developing Conceptual Understanding and Procedural Skill in Mathematics: An Iterative Process. Journal of Educational Psychology, 93(2), 346–362. https://doi.org/10.1037/0022-0663.93.2.346
41. Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., and Agirre, E. (2023). NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark. Findings of EMNLP 2023, pp. 10776–10787. https://doi.org/10.18653/v1/2023.findings-emnlp.722
42. Shute, V. J. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
43. Siegler, R., Carpenter, T., Fennell, F., Geary, D., Lewis, J., Okamoto, Y., Thompson, L., and Wray, J. (2010). Developing Effective Fractions Instruction for Kindergarten Through 8th Grade (NCEE 2010-4039). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/docs/practiceguide/fractions_pg_093010.pdf
44. Smith, J. P., III, diSessa, A. A., and Roschelle, J. (1993). Misconceptions Reconceived: A Constructivist Analysis of Knowledge in Transition. Journal of the Learning Sciences, 3(2), 115–163. https://doi.org/10.1207/s15327809jls0302_1
45. Sonkar, S., Chen, X., Liu, N., Baraniuk, R. G., and Sachan, M. (2024). LLM-based Cognitive Models of Students with Misconceptions. arXiv preprint. https://arxiv.org/abs/2410.12294
46. Srinivasa, R. S., Che, Z., Zhang, C. B. C., et al. (2025). TutorBench: A Benchmark to Assess Tutoring Capabilities of Large Language Models. arXiv preprint. https://arxiv.org/abs/2510.02663
47. Steinle, V., and Stacey, K. (2004). A Longitudinal Study of Students’ Understanding of Decimal Notation: An Overview and Refined Results. Proceedings of the 27th Annual Conference of the Mathematics Education Research Group of Australasia, Vol. 2, pp. 541–548.
48. Wang, P., Li, L., Chen, L., et al. (2024a). Large Language Models are not Fair Evaluators. Proceedings of ACL 2024, pp. 9440–9450. https://doi.org/10.18653/v1/2024.acl-long.511
49. Wang, R. E., Zhang, Q., Robinson, C., Loeb, S., and Demszky, D. (2024b). Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes. Proceedings of NAACL 2024, pp. 2174–2199. https://doi.org/10.18653/v1/2024.naacl-long.120
50. Wang, Y., Kordi, Y., Mishra, S., et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. Proceedings of ACL 2023, pp. 13484–13508. https://doi.org/10.18653/v1/2023.acl-long.754
51. Wilson, E. B. (1927). Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association, 22(158), 209–212. https://doi.org/10.1080/01621459.1927.10502953
52. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., and others (2025). Qwen3 Technical Report. arXiv preprint. https://arxiv.org/abs/2505.09388
53. Ye, S., Kim, D., Kim, S., et al. (2024). FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets. International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2307.10928
54. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685
55. Zhou, C., Liu, P., Xu, P., et al. (2023). LIMA: Less Is More for Alignment. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://arxiv.org/abs/2305.11206

Appendix A. Grader Instruction

Each grader receives the instruction below. The four slots in braces are filled in for every reply: {view} with the system and user messages the model was given, {criteria} and {critical} with the case's numbered checklist lines and serious errors, and {reply} with the model's reply. In the instruction, “criteria” are checklist lines and “critical” lines are serious errors. The correctness line carries a “(critical)” tag in the kit; under the published rule it is scored as an ordinary checklist line (Section 3.2).

You grade one tutor reply to a student, line by line, against a checklist. Work from the checklist lines exactly as written; do not add your own.

Tutor's view (what the tutor was shown):
{view}

Checklist lines the reply should meet ("criteria", numbered from 0):
{criteria}

Serious-error lines ("critical", numbered from 0): mark yes only if the reply commits the error.
{critical}

The tutor's reply:
<reply>
{reply}
</reply>

Return only JSON: {"criteria": [true|false, ...one per criteria line...], "critical": [true|false, ...one per critical line...], "reason": "one short sentence"}

The same instruction filled in for Case 1 of Appendix B and the reply of Qwen3.8-27B+SENSUS (second run), exactly as a grader received it:

You grade one tutor reply to a student, line by line, against a checklist. Work from the checklist lines exactly as written; do not add your own.

Tutor's view (what the tutor was shown):
You are a mathematics tutor. A student is working on the question below. Reply to the student directly, as you would in the conversation.

Question (grades 4 and 5; 4.NF.A.1):
Priya wrote 3/6 = 4/10, but this is not true. Change exactly one of the four numbers so that the equation is true.

What the student did and said:
The learner is working on making 3/6 = 4/10 true by changing one number. The learner writes: "4/10 is bigger than 3/6, because 4 is bigger than 3 and 10 is bigger than 6. So I need to make 4/10 smaller. I changed the 4 to a 3: 3/6 = 3/10."

Write your reply to the student.

Checklist lines the reply should meet ("criteria", numbered from 0):
0. (critical) Nothing false (for example, saying 4/10 is greater than 3/6, or saying 3/6 = 3/10 is true).
1. Names a likely error tied to the learner's "4 is bigger than 3 and 10 is bigger than 6": judging which fraction is bigger by comparing the top numbers and the bottom numbers as whole numbers.
2. Says the error as likely, not certain, using words such as "looks like", "might", "maybe" or "it seems". A reply that names no reason for the error is marked no.
3. States that 4/10 is less than 3/6.
4. Gives a reason a grade 4 learner can check for why 4/10 is less than 3/6. Comparing each to one half, a drawing or renaming both with the same denominator counts. Only stating which is bigger does not count.
5. Names a specific thing the learner did correctly, such as changing only one number or comparing before changing. "Good job" alone is marked no.
6. Ends with a question or task the learner can answer next. A trailing "Does that help?" alone does not count.
7. Is about 120 words or fewer.

Serious-error lines ("critical", numbered from 0): mark yes only if the reply commits the error.
0. States as certain the reason the learner thought 4/10 is bigger, for example "You thought bigger numbers make a bigger fraction" with no word such as "maybe" or "looks like". Repeating what the learner wrote does not count.
1. Says or agrees that 4/10 is greater than 3/6.
2. Says or agrees that 3/6 = 3/10 is true.

The tutor's reply:
<reply>
You compared the top numbers and the bottom numbers separately. It sounds like you may be treating bigger numbers on top and bottom as meaning a bigger fraction. Instead, let's use equal-sized rectangles divided into sixths and tenths. Three sixth-sized pieces cover half a rectangle; four tenth-sized pieces also cover half. So 3/6 and 4/10 are already equal. Your change to 3/10 makes the right side smaller than the left. If you want to change just one number, which number could you change to make the two sides equal again?
</reply>

Return only JSON: {"criteria": [true|false, ...one per criteria line...], "critical": [true|false, ...one per critical line...], "reason": "one short sentence"}

Appendix B. Two Cases in Full

Case 1 is the first case in file order. Case 2 is the first case in file order that the fine-tuned 27B model passes fully while GPT-6 Sol scores below 0.6. The replies are from Qwen3.8-27B+SENSUS (second run) and GPT-6 Sol as recorded, and each mark is shown as grader 1 / grader 2. All 76 cases, with every model's reply, can be browsed on the evaluation cases page.

Case 1: Equivalent Fractions (4.NF.A.1)

Question. Priya wrote 3/6 = 4/10, but this is not true. Change exactly one of the four numbers so that the equation is true.

What the learner did and said. The learner is working on making 3/6 = 4/10 true by changing one number. The learner writes: "4/10 is bigger than 3/6, because 4 is bigger than 3 and 10 is bigger than 6. So I need to make 4/10 smaller. I changed the 4 to a 3: 3/6 = 3/10."

Case 2: Comparing Fractions (4.NF.A.2)

Question. Write the whole number that goes in the blank to make the statement true: 3/5 < 6/___ < 3/4

What the learner did and said. The learner is filling the blank in 3/5 < 6/___ < 3/4. The learner writes: "3/5 is 6/10 and 3/4 is 6/8. The middle has to be bigger than 6/10, so I put 11, because 11 is bigger than 10. My answer is 6/11."

Appendix C. Datasheet