We started with zero AI and worked our way up

A model-last approach to maths tools for teachers. From Jin at Spirian.

Abstract

The public debate about AI in education asks how much of teaching AI should take over. The evidence points to a different variable: what the tool is built to do. We describe a model-last approach to building maths tools for teachers. We start from zero AI and add a language model only where ordinary code cannot do the job, we gate every model output with a check, and we keep the tutor as the author of what students see. The result is a pipeline in which a model is needed for three bounded jobs, is a fallback in six further stages, and is absent from the rest, at about 3 cents per certified template. We are in internal testing, and we set out where we are taking it next.

1. The wrong question

I was talking with some other tutors recently and the conversation turned to AI and ethics. One of them said they didn't think replacing humans with AI could ever work. Learning through mentorship, they said, is a human thing, and taking that away would ruin education.

I agreed. But what stayed with me was that everyone at the table, me included, was answering the same question: how much of teaching should AI take over? It is the question in almost every headline, and it comes with two camps, all of it or none of it. Both start in the wrong place. The question underneath is who is accountable when the tool is wrong. If a student is handed a wrong answer, someone has to own that, and it cannot be a machine.

Figure 1. Two starting points. A common approach begins with AI doing the whole job and adds guardrails. Ours begins with ordinary code and admits a model only where code cannot do the job.

2. What the field shows

AI is already in Australian classrooms. The ACER analysis of the Teaching and Learning International Survey found about two-thirds of lower secondary teachers using it, and more than 80% of them worried that it can make recommendations that are not appropriate or correct. In a 2025 Learning First survey of New South Wales schools, 80% of teachers and leaders whose students use AI said they were worried about its educational impact.

The research explains why design matters more than adoption. In a study of nearly a thousand high school maths students, a standard chatbot lifted practice grades by 48%, but those students then scored 17% lower on the exam once it was taken away, compared with students who never had it. A version that gave teacher-designed hints mitigated that harm (Figure 2). At Harvard, a carefully designed AI tutor helped 194 physics students learn more than twice as much in less time as an in-class active learning session. And a University of Sydney and Barker Institute review of 271 studies found that outcomes were strongest when AI was combined with professional judgement and human relationships. Four of those 271 studies were Australian.

Figure 2. The same tool, designed two ways. Grade changes are relative to a group with no AI. Source: Bastani et al., PNAS, 2025.

Maddy Sims of the Center on Reinventing Public Education put the stakes plainly: the real risk "isn't that AI will replace human relationships in schools. It's that education will fail to define and protect what is most human." Across the evidence the pattern is consistent. Design, not the mere presence of AI, decides the outcome.

3. What we set out to protect

A student of mine once finished every practice question I had in two days and asked for more. I did not have more. The thing standing between her and the next question was not knowledge or patience. It was the hour I would have to spend making paper. It felt like being about 80% of the way through a lesson when a student runs out of everything you brought, and you suddenly need a pile more resources you do not have.

That hour is the part of tutoring nobody sees. One tutor told me a one hour session is really two or three once you count the preparing and the marking. None of that is teaching.

So we set out to keep the human part of tutoring more human than ever, and to replace the back end: the retyping, renumbering and re-checking around it. The front end, noticing that a student is stuck rather than tired and choosing the next question because of who is in front of you, stays with the tutor.

4. Why volume cannot come from a model

The obvious way to produce volume is to ask a language model for it. That fails, and not because of prompting. A model predicts what a maths question looks like, and each variation is an independent guess, so 10,000 variations are 10,000 chances to be wrong with no practical way to read them. Trustworthy volume has to come from something that can be checked mechanically.

The failure is specific. Take 5w² + 34w − 7 = 0. It factorises because the discriminant, 34² − 4(5)(−7) = 1296, is a perfect square (36²). Change the 34 to a 35 and it becomes 1365, which is not. The question is still a valid quadratic and still produces an answer, but the method it was written to test no longer works. A model asked for "another one like this" cannot know that rule. Our pipeline derives it.

5. Our approach: model last

So we began from the other end. Every stage of the pipeline started as a question: can ordinary code do this? A model was added only where the answer was no. Three tests decide it. Can code do the job? Can code check the result? If the model is wrong, will we know?

Figure 3 follows a question through the pipeline. A model is needed for three bounded jobs: deciding which numbers in a question are free to change (in the example above the 5, 34 and 7 can, and the 2 in w² cannot, or it stops being a quadratic), stating the relation the question is built on, and checking that the finished wording claims nothing the maths does not support. In six further stages a model appears only if the mechanical route fails, and in the rest it never does. The worked solution a student sees is the tutor's own working, slotted in step by step, so the tutor's voice is preserved.

Figure 3. Where a model can appear. Amber marks a stage that can use one, green marks a stage that tries code first, and teal marks a stage that never does. Drawing a question for a student afterwards involves no model.

Figure 4 shows the rule at every stage. Code goes first. A model is called only if the check fails, and it is handed the failed attempt to work from. Its answer must pass the same check, and if it does not, the item is declined and flagged for a person rather than shipped.

Figure 4. Code first, model last, and every answer checked.

6. What this gives us

• A bounded model. It does the jobs that need judgement about maths and nothing else, and it never does the arithmetic. Every model output is gated by a check, and almost all of those checks are mechanical.
• Low cost. About 3 cents per certified template, because the model is rationed to the cheapest tier that does each job.
• The tutor's voice. The question and the working students see are the tutor's own.
• Original source material. Our seed questions are written by hand from the syllabus, not lifted from past papers.
• Failures caught early. The design favours catching a bad template before it reaches a student over flattering a headline number. Anything we cannot verify is declined and flagged for a person.

One boundary is worth stating. Verification shows that a template is internally consistent. It does not by itself show that a question is a good one, so a person reviews what the system flags.

7. Where we are taking it

We are in internal testing, with no classroom evidence yet. Three directions guide the work (Figure 5): use the model less by replacing more model calls with code, firm up the one check that still relies on a model, and measure and share, tracking the model-free share of each stage and building the classroom evidence that the Sydney review says Australia lacks.

Figure 5. Today, and the three directions for the work.

If you teach, tutor, build in this space or research it, what would you want us to measure before you trusted a tool like this?