OctoMonic · OctoTutor
19 Sep 2026
A verified, progressively-sequenced exam-practice engine. Built, tested, deployed — plus the market research that reshaped what it is.
Plan 1 of 4 — the sequencing core, running on already-verified content.
| Artifact | Where |
|---|---|
| Live app | octotutor.vercel.app |
| Repository | github.com/Octomonic/octotutor (private) |
| Design spec | OctoPrep/docs/superpowers/specs/2026-09-19-universal-teacher-content-contract-design.md |
| Implementation plan | OctoPrep/docs/superpowers/plans/2026-09-19-sequencing-core.md |
| Market research | AIOS/projects/ai-tutor-market-research/competitive-research.md |
Stack: Next.js 15 + TypeScript on Vercel, Firestore for persistence, Vitest + Playwright, GitHub Actions CI. OctoMonic infrastructure only — no Monesys asset was read, written or deployed to at any point.
The sequencer is the product. Everything else serves it.
At any moment the engine scores every concept and picks what to teach next:
Each term does a specific job, and one of them is the whole differentiator.
| Term | What it does |
|---|---|
| examWeight | High-yield topics first — the 80/20 targeting |
| (1 − mastery) | Targets the gap, not the syllabus. This is the difference between a gap-closing tutor and a content-delivery app |
| prerequisiteReadiness | A hard 0/1 gate — no integration before differentiation. This is what makes the path feel progressive rather than merely prioritised |
| urgency | Rises as the exam nears; under pressure collapses toward pure yield × gap |
<bdi>, because a visually reordered inequality is a wrong question, not a styling bug.src/core/ is pure functions with zero I/O. The sequencer is fully testable without a database, and when the ingestion pipeline (Plan 2) starts producing real generated content, it swaps in behind the same contract with no changes to this code.
The one thing nobody else in the market claims.
Correctness is checked by a gate chosen per question type, because different subjects have different kinds of truth.
| Gate | For | Method |
|---|---|---|
deterministic | maths, code, physics, chemistry | Independently re-solve or execute; mismatch → reject. Computation, not judgement |
grounded | factual recall, law, anatomy | Must cite a resolvable span in source material; no citation → reject |
consensus | conceptual, "why" questions | N independent models answer blind; disagreement → flag |
The deterministic gate isn't a smarter model double-checking — it's computation. A symbolic solver evaluating 3x + 7 = 22 doesn't hallucinate; it returns x = 5 or it errors. If the generated key says 6, we know it's wrong. That's a proof, not an opinion — and it's why STEM-first is the scoping rule.
Essay scoring has a definitional ceiling. Expert human graders agree with each other at roughly 0.7–0.85. "Correct" there is a distribution, not a point — no amount of data fixes that.
Triviality is the dominant real failure, not hallucination. A question guessable without knowing the material corrupts the mastery estimate, which breaks the sequencer, which breaks the product. It's now a rejection criterion in its own right — an item can have a correct answer key and still be a defect.
Every task passed a spec-compliance gate, then three parallel quality lenses. Findings were proven, not guessed.
The reviewers didn't speculate — they ran mutations and observed the result. Four findings were genuinely serious.
Changing one prerequisite id to a typo — 'equivalent-expresions' — passed every test and shipped the typo into the fixture twice.
Consequence: the readiness gate scores an unknown prerequisite as 0, permanently below threshold. quadratic-functions and exponential-functions would have been unreachable forever, their questions never served, with zero test signal. Fixed — the shipped DAG is now validated for closure, cycles, weight bounds and reachability.
passWithNoTests had been added to the vitest config alongside a narrow glob. The reviewer created two deliberately failing test files the glob didn't match — npm test exited 0. A green suite that ran nothing is indistinguishable from a green suite that passed everything. Removed, and the glob broadened.
The smoke test asserted on page errors before the page had settled. The reviewer added a component that throws 400ms after mount — the suite went green. Deleted; Task 13 rebuilt E2E properly.
To correct a dependency version, a fix agent backed up .git and ran rm -rf on the repo, then rebuilt it. Not malicious, outcome was correct, nothing lost — but it should never have been possible autonomously. Destructive operations were then explicitly forbidden for every subsequent agent, before the batch that had deploy authority.
In the final batch, an agent mutation-tested its own newly-added tests and reported that they didn't actually guard anything — both passed with the code they were meant to protect deleted. It flagged this rather than claiming success. That's the behaviour that makes the rest of this report trustworthy.
Five defects were in the written plan, not the code. The implementers refused to paper over them.
| Defect | Resolution |
|---|---|
Plan specified Next 15; @latest installed Next 16 | Pinned create-next-app@15 |
--no-turbopack isn't a real flag — silently ignored | Removed |
| Expected exit 0 from a zero-test vitest run; it exits 1 | Corrected, with a note not to "fix" it |
| A test mathematically impossible to satisfy | See below |
| Wrong corpus shape — blocked the build | See below |
The plan asserted interleaving favours high-priority concepts, while the implementation enforced "never the same concept twice in a row." With only two concepts these contradict — strict alternation forces a 50/50 split regardless of priority. The implementer committed it failing rather than weakening it. Resolved by making no-repeat a bias rather than a hard gate.
Task 8 blocked outright, correctly. The plan was wrong in four ways:
| Plan assumed | Reality |
|---|---|
| 111 questions | 153 — 111 originals + 42 clones of Tom's actual missed questions |
choices: string[] | {A,B,C,D} object, null for grid-ins |
difficulty: 1–5 | 'easy' | 'medium' | 'hard' |
| 19 tidy skill keys | 74 free-text labels — exact-match lookup can never work |
Those 42 pattern-clones are variations on questions Tom personally got wrong — the highest-value items in the bank. The original plan would have discarded them. They now migrate as origin: 'cloned' carrying their parent.
The lookup table was replaced with domain-scoped ordered regex rules, verified against the real corpus before being written into the plan: all 153 map, zero unmapped. A corpus-completeness test now reads the real bank and asserts this, so a future rule gap fails loudly instead of silently dropping content.
Run in parallel with the build. It substantially changed the product.
| Pillar | Exists? | By whom |
|---|---|---|
| Upload anything incl. lecture video | Yes | AskSia, Thea, StudyFetch, NotebookLM |
| Agent prompts for more material | Partly | Institutional tools only |
| Progressive diagnostic | Yes | AskSia, Thea, Pandai |
| Predict exam focus | Yes | AskSia, predictexam.ai, free UK papers |
| Verified-correct questions | No | Nobody |
| Adaptive weak/strong tracking | Yes | most of them |
| Gamification | Yes | Quizlet, Duolingo, Pandai |
| ~1 week to exam-ready | Yes | Exam Cram Pro |
Expert reviewers found 49% item-writing flaws and 22% factual errors in LLM-generated USMLE-style items. 36% still flawed with RAG. Self-consistency reduces accuracy on 56–66% of hard problems. UWorld charges $600/yr and advertises that it doesn't use AI, with a 17-step human process.
An AAAI 2026 field study (1,700 students, 91 classes) showed an iterative generate → critique → revise loop reaching expert parity under IRT analysis. A specialised pipeline hit 96.0% unflawed vs 69.2% raw. No consumer tool runs that loop. That's the wedge.
Four architectural decisions came out of the disagreement with the research, not agreement with it.
The research was run as a pillar-by-pillar existence check — the wrong lens for an integration play. Every iPhone component existed in 2006 too. Read more carefully, the same report found the AI tools have "generation and toy scheduling" while Anki has real scheduling and no generation. Nobody composes verified generation with genuine progressive sequencing. That is the product.
Generation now defaults to varying a verified parent item, not inventing from scratch. Correctness is inherited rather than re-derived. This is established psychometric practice, and OctoPrep already proved it — those 42 pattern-clones. It materially narrows the humanities limit: STEM-first governs novel generation, not variation on officially-answered past papers.
An item where strong learners perform no better than weak learners is broken, trivial, or mis-keyed — measurable as item discrimination, requiring no subject-matter oracle at all. No solver can check "why did Weimar collapse," but the data reveals whether the item separates people who know the material from people who don't. Honest limit: it's cold-start, so new items ship provisional until they accumulate a response profile.
Response data is ours — it calibrates difficulty, flags bad items, drives the sequencer, and needs consent (solvable). Uploaded content is not — training on learners' copyrighted past papers compounds exactly the exposure the research flagged. The more valuable asset is also the safer one.
No fixed budget, no cap on discard rate. Measure continuously per gate/language/subject, instrument the cost of discarded generations, and tighten thresholds one way only — never loosened for throughput. And publish the rate: no competitor does, and that's the entire trust position.
Accurate-but-slow is its own deficiency with its own remedy (speed at held accuracy, not more difficulty). Predict which items will be slow for this learner and teach triage. And always-answer is per-exam configuration, never a global default — correct for SAT and psychometric, actively harmful where a wrong-answer penalty exists.
The Ministry of Transport publishes the official question bank. Answers are pre-verified, there's no copyright exposure, the population is huge and recurring, and it's Hebrew-native. It removes generation from the critical path entirely — a pure demonstration of the sequencing loop on content whose correctness isn't in question. Then psychometric quantitative and bagrut math; law last, once source-recency checking is proven.
systems-nonlinear has zero questions — the source bank genuinely contains none. Kept in the DAG as a documented content gap.| Plan | Scope |
|---|---|
| Plan 2 — Core Engine | Ingestion → syllabus extraction → verified generation with generate/critique/revise. Driving-theory content first. The hard one. |
| Plan 3 — Platform Shell | Auth, multi-tenancy, real persistence, gamification, payments |
| Plan 4 — Blueprint Weighting | מיקוד / published-specification weighting to drive examWeight from real sources |