FeaturesTry itAboutBlogJoin the waitlist now
← All writing

The template is the bottleneck

Jin Choi11 min read

Post one of a series on building automated template extraction for Australian senior secondary maths.

I gave one of my students every practice question I had on a topic. She had a trial exam coming up. She finished the lot in two days and asked for more, and I did not have more.

That is the part of tutoring that nobody outside it sees. The teaching is the easy part. The bottleneck is material. Past papers are finite, question banks are finite, and every tutor and every firm eventually hits the same wall. The honest options at that point are all bad: reuse questions the student has already seen, buy another bank that runs out more slowly, or write more by hand.

So I did what everyone does now, and asked a language model to write more.

Why that does not work, and why it is not a prompting problem

The questions came back wrong. Not obviously wrong, which would have been fine. Plausibly wrong. Answers that did not follow from the setup. A triangle with sides that cannot close. Working that looked correct until you actually did it.

The instinct is to blame the prompt. That is the wrong diagnosis. A language model is not calculating anything. It is predicting what a maths question looks like, based on the very large number of maths questions it has seen. Most of the time that is close enough to survive a glance, which is precisely what makes it dangerous, because the failures are invisible until a student hands one back.

Checking the output takes longer than writing from scratch. That is the real reason most tutoring businesses we have spoken to have quietly given up on AI for this, and they were right to.

What already exists, which is more than most people assume

We expected to find very little here. We found a mature field instead.

Parameterised question generation is decades old. WeBWorK has been running in universities since the 1990s. STACK, built by Chris Sangwin at Edinburgh and backed by the Maxima computer algebra system, runs inside Moodle. Numbas, Maple T.A., ALEKS and the Khan exercise framework all do versions of it. The psychometric literature calls this Automated Item Generation, and Mark Gierl and Hollis Lai have been publishing on it since well before the current wave of AI.

The mechanism is the same in every one of them. A human writes an item model: the question with slots where the numbers go, plus the rules governing what can go in those slots. Software then generates as many instances as you like. It works, it has worked for thirty years, and the questions it produces are correct because a person guaranteed they would be.

Commercially, the best current example is Cerebry in Singapore, founded 2016, which generates questions by deductive reasoning over a knowledge base encoded in first-order logic. They are profitable, they serve Common Core, IB, Cambridge, Singapore, CBSE and more. They do not serve any Australian curriculum.

Anyone claiming nobody has done AI question generation has not looked. We nearly made that claim ourselves.

The two poles, and the gap between them

Once you have read enough of it, the field splits cleanly into two.

At one end, hand-authored templates. Reliable, high quality, and expensive. Every model is skilled human work, done once per template, and that cost is why coverage is always narrow and always tied to one curriculum. Rudner's estimate, cited across the AIG literature and repeated by Kosh and colleagues in 2019, puts a single high-stakes multiple-choice item at $1,500 to $2,000 US dollars once editing, field testing and calibration are counted. School-level questions are far cheaper than that, but the shape is the same: a person does the work, once, per template.

Kosh and colleagues put a number on where that stops being worth it. Working through K-12 mathematics, they calculated that automated item generation only becomes cheaper than writing items by hand once you need somewhere between 173 and 247 items within a single fine-grained content area. Below that, the upfront cost of building the item model is not recovered.

That break-even exists entirely because a person has to author the model first. Automate that step and it collapses toward one.

At the other end, full language model generation. Cheap, instant, and unreliable for exactly the reasons above.

The gap in the middle is the interesting part. What if a model never generated a question at all, and instead read one question you already have, worked out its structure, and handed that structure to software that builds the template? The model does the reading. Engineering does the parameterising. A computer algebra system proves the result before anything reaches a student.

That is the thing we are building, and we have started calling it Automated Template Extraction: deriving a verified, executable question template directly from a single existing question, without a human authoring the template by hand.

The distinction from what came before is one sentence. Automated Item Generation runs model to items. Automated Template Extraction runs item to model, and then to items. The item model has always been the human's output. We are trying to make it the machine's.

Recent work that narrows the claim

However, this gap is not untouched.

In February 2026, Yeo and colleagues published Adaptive Problem Generation via Symbolic Representations. It takes a seed word problem, converts it into a symbolic representation in SymPy, modifies the structure, translates it back into natural language, and verifies with a symbolic solver. That is the same computer algebra system and broadly the same shape as what we do.

TemplateGSM, presented at ICLR 2025, produced over seven million verified grade-school problems from 7,473 parameterised templates, and those templates were authored by GPT-4 rather than by people.

And in July 2026, a paper in Mathematics tested exactly this idea against the real thing: AI-assisted authoring of STACK questions, validated by a computer algebra system, benchmarked against five mathematics teachers with STACK authoring experience doing the same ninety problems by hand. It reported a 79% success rate and a 35.7% reduction in authoring time. Its authors were careful to say the output still needs review before use.

So what is actually left?

The distinction that survives is about the acceptance test, not the technique. Those systems ask whether a generated problem is well posed and solvable. That is the right bar for producing training data for a language model, which is what most of this work is for.

It is not the bar for a question you hand a Year 12 student the week before a trial exam. That question has to still be inside the syllabus at every set of values it can take. It has to still test the skill the original was written to test. The worked solution has to stay readable and the marking rubric has to still describe it. None of that is visible to a computer algebra system, and none of those systems were built to look for it.

That is a much smaller claim than the one we started with. It is also the one we can defend.

Why Australia, and why Years 11 and 12

The first reason is that the work is unevenly distributed. Cerebry has encoded seven curricula over ten years and none of them is Australian. The AI worksheet tools that do serve the Australian curriculum concentrate on primary and junior secondary. Senior secondary maths, where the questions are hardest and the stakes are highest, is where the least has been built.

Automated authoring is what makes a market that size economic. Encoding NSW Stage 6 by hand is a large fixed cost for a few hundred thousand students, which is exactly why nobody has done it. If the authoring is automated, the arithmetic changes.

The second reason is that the need is measured, not assumed. The Grattan Institute surveyed 5,442 Australian teachers and school leaders in 2021. Australian lower-secondary teachers spend 8.4 hours a week planning and preparing, above the OECD average of 7.4. 86% say they do not have enough time to do it well. Around 90% agreed that good shared resources would give them back roughly three hours a week. Australia's private tutoring sector has grown to about $2.2 billion, and a tutor's hour is worth $60 to $120 of billable time, every hour of which is lost when it goes into building material by hand.

There is a third reason we are less able to evidence, so we state it as an opinion. Australian education is unusually sceptical of AI, and we think that scepticism is well earned. A tool that cannot show you why its output is correct deserves it. That is the standard we are trying to build to, rather than argue against.

Who we are

Two undergraduates. I am in Sydney, studying actuarial studies and commerce, and I tutor senior maths on the side. Leo is a computer science student in Perth, 3,300 kilometres away.

Nobody is funding this and nobody is waiting on it, which is worth saying because it explains the choices. We can withhold a question instead of shipping it. We can spend a fortnight on a failure that affects a handful of questions, because there is no quarter to get through. We would rather be slow and right than quick and plausible, and at the moment we can afford to be.

The pipeline takes a question in, and returns a template with a general method, a valid range of values, a worked solution and a marking rubric. Our first working design produced a usable template from about 15% of the selected set of questions we fed it. After a redesign, about 80% now reach a verified general method. The stages after that, prose and rubric generation, are not yet measured end to end, so that 80% is not an end-to-end figure and we are not presenting it as one. The July 2026 result above is end to end. Ours is not yet comparable to it, and making it comparable is the next thing on the list.

We plan to share our findings and our journey through these posts.

The wider reason we are doing this is simpler than the engine. Most of what gets built under the AI label aims at the interesting part of someone's job. We think the useful direction is the opposite: find the part that is genuinely mechanical, prove it is correct, and hand the hours back. A tutor is paid to teach. Writing six versions of a question late on a Tuesday is not teaching, it is the tax on teaching. That tax exists in most professions.

The condition, and it is the reason for everything above, is that whatever does the work has to be able to show that it is right. Otherwise it has not given anyone time back, it has only moved the work from doing to checking. We would like Australia to be the place that insists on that, and we would like to be worth trusting when it does.

If you work in assessment, item generation or applied AI, and particularly if you have hand-authored parameterised questions yourself, we would like to know one thing: how long did one actually take you? That number does not appear anywhere in the literature we can find, and it is the number this whole argument rests on.

References

Automated item generation, the field

Systems that generate parameterised questions

Recent work on generating questions symbolically

  • Yeo, T. et al. (2026). Adaptive Problem Generation via Symbolic Representations. arXiv:2602.19187. https://arxiv.org/abs/2602.19187
  • Zhang, Y. (2025). Training and Evaluating Language Models with Template-based Data Generation. ICLR 2025. arXiv:2411.18104. https://arxiv.org/abs/2411.18104 Source of the 7,473 GPT-4-authored templates and 7 million problems.
  • Li, Z. et al. (2024). Neuro-Symbolic Data Generation for Math Reasoning. NeurIPS 2024. arXiv:2412.04857. https://arxiv.org/abs/2412.04857
  • A Proposal of a Mathematics Problem Generation Tool Using Generative AI for STACK Online Assessment System. (2026). Mathematics, 14(14), 2481, published 9 July 2026. https://doi.org/10.3390/math14142481 Source of the 79% success rate and 35.7% authoring time reduction against five experienced human authors.
  • Sonnleitner, P., Bernard, S., Michels, M. A., Inostroza-Fernandez, P., Keller, U., Gierl, M. J., Cardoso-Leite, P. and Hornung, C. (2025). Establishing cognitive item models for fair and theory-grounded automatic item generation. Applied Measurement in Education, 38(2), 95-117. https://doi.org/10.1080/08957347.2025.2563889

Australian context

Company background