From AI Feedback to Verified SLS Interactives: Building a Guarded Codex Skill with Playwright

What happens when AI feedback is paired with an interactive mathematical model—and when the process of building that pairing is itself assisted by AI?

SLS Modules Run by Automation Scripts - Google Sheets

That question led me to build sls-finalize-community-module, a Codex skill supported by a deterministic Playwright automation package for Singapore Student Learning Space (SLS) authoring.

The aim was never to create an autonomous bot that clicks through SLS as quickly as possible.

The more interesting challenge was this:

Can AI exercise useful judgment while deterministic automation constrains what it is allowed to do, verifies what actually happened, and hands ambiguous decisions back to a human?

That distinction became central to the project.

The resulting workflow combines three different strengths:

AI for interpretation, curriculum reasoning and prompt construction;
Playwright for repeatable browser actions, guards and verification;
teachers for ambiguity, pedagogy and consequential decisions.

Along the way, some of the most useful lessons came not from successful runs, but from failures—including one memorable case where a picture of a parabola was confidently tagged as a matrix question.

Why combine AI feedback with an interactive?

The educational starting point was simpler than the engineering.

SLS already supports AI-enabled feedback. A learner can attempt a mathematics question and receive feedback that diagnoses an error or suggests how to proceed.

That is useful.

But feedback is usually expressed through language.

For some mathematical ideas, students also need to see what changes and what remains invariant.

An interactive model can complement feedback by allowing a learner to:

change quantities;
observe the effect immediately;
connect symbolic and visual representations;
test another case;
identify a general pattern.

The learning sequence I wanted was therefore:

attempt → feedback → visualise → vary → try again

I demonstrate this idea in the video:

What Happens When AI Feedback Meets an Interactive? Make Mathematics Visible

https://www.youtube.com/watch?v=GY5osFZ6ijA

The interesting question then became: if this pairing is pedagogically useful, can we make the authoring process easier to reproduce across many SLS resources?

The authoring problem is larger than generating an interactive

Generating an interactive is only one step.

Preparing an existing SLS module properly can involve:

inspecting every section, activity, page and question;
identifying the correct Subject, Level and Content Map;
assigning suitable curriculum learning outcomes;
ensuring assessed questions contribute to Learning Progress;
preserving existing questions and uploaded interactive ZIP files;
separating multiple questions with sensible page breaks;
generating an appropriate practice interactive;
saving changes;
reopening the activity;
checking that every change actually persisted.

A human author can do these things.

The difficulty is doing them consistently across many modules without turning automation into a source of new errors.

That led to the main architectural decision.

AI decides; Playwright executes

The project separates reasoning from execution.

Teacher request + SLS URL
          ↓
Codex skill
intent, routing, evidence rules,
safety boundaries and decisions
          ↓
Playwright runner
deterministic browser actions,
guards and verification
          ↓
SLS authoring tools
including Authoring Copilot

The Codex skill contains the operational knowledge.

Its SKILL.md describes things such as:

which workflow should be used;
what evidence is authoritative;
what conditions should stop a run;
which mutations are allowed;
when ambiguity should be escalated;
what must be reopened and verified before success can be claimed.

The Playwright package performs the actual browser operations.

It navigates SLS using maintained locators, waits for observable states, saves evidence and refuses actions when the state of the interface is uncertain.

The generative AI step is deliberately narrower.

For an ACP interactive, the automation builds a detailed prompt using the iwant2study SLS Prompt Generator and Interactive Prompt Library, then submits it to SLS Authoring Copilot's Interactive generator.

AI is therefore not improvising the entire browser workflow.

It is being used where generative capability is useful, while navigation and verification remain deterministic.

Read first, write second

One rule became fundamental very early:

Understand the target before changing it.

A normal run therefore starts by establishing identity and collecting evidence.

Conceptually, the workflow is:

Receive an SLS module, section or activity URL.
Preserve its exact module identifier.
Open the module and verify its visible title.
Inventory sections, activities, pages and questions.
Read existing curriculum tags.
Resolve eligible curriculum outcomes.
Analyse each question.
Propose changes only where evidence is sufficiently strong.
Apply the change.
Save.
Reload.
Reopen the affected item.
Verify the persisted state.
Record evidence.

That final verification matters.

A successful click is not proof.

A disappearing dialog is not proof.

Even a success toast is not proof.

The standard became:

inspect → mutate → observe save → reopen → verify persisted state

This simple principle prevented a surprising number of false successes.

Curriculum tagging is a constrained reasoning problem

Curriculum tagging turned out to be one of the most interesting parts of the project.

A naïve approach would be to read the question, search every curriculum outcome and choose the sentence with the greatest semantic similarity.

That is dangerous.

A beautifully matching outcome may belong to the wrong level, stream, subject or Content Map.

Instead, the workflow narrows the search space first.

It begins with saved SLS evidence:

read the module's existing tags;
determine Subject and Level;
identify the saved Content Map;
harvest the official outcomes available within that map;
rank only outcomes that are actually eligible.

The question is therefore not:

Which syllabus statement anywhere sounds most like this question?

It is:

Given what SLS already tells us about this module, which of the eligible official outcomes is best supported by the mathematical evidence in this question?

That difference is important.

Reading mathematics is harder than reading text

An SLS mathematics question is not necessarily represented by ordinary visible HTML text.

The evidence reader may need to combine:

normal DOM text;
content inside open shadow DOM;
WIRIS MathML embedded in SVG comments;
image alt and title metadata;
OCR from substantial diagrams;
shared stimuli from multipart questions;
suggested answers as supporting evidence.

OCR is performed locally using Tesseract.js rather than uploading SLS question images to an external OCR service.

Once the evidence has been collected, the system extracts mathematical features.

These include:

Operations

addition;
subtraction;
multiplication;
division.

Representations

fractions;
decimals;
percentages;
ratios;
integers;
mixed numbers.

Mathematical structures

equations;
quadratic functions;
fractional equations;
matrices;
graphs;
angle relationships;
measurement;
data interpretation.

Topic evidence receives greater weight than generic operations.

And that distinction became very important after one particular failure.

The day a parabola became a matrix

One Secondary Mathematics module concerned Quadratic and Fractional Equations.

Question 5 visibly contained a parabola.

Yet the automation selected an outcome about:

addition, subtraction and multiplication of matrices

At first glance, this looked absurd.

But the report made the reasoning failure understandable.

OCR had recovered mainly the axes and tick labels from the graph:

5 10 −5 −10 5 10 15 −5 0 0,0 x y

The negative numbers looked like subtraction.

The matrix outcome happened to contain words such as addition, subtraction and multiplication.

The early matcher rewarded those overlapping operations without requiring actual evidence of a matrix.

The answer looked confident.

The underlying reasoning was wrong.

This became one of the most useful failures in the project.

Fix the reasoning model, not the individual question

The wrong solution would have been:

if question == Q5:
    choose quadratic

That would fix one example while leaving the underlying weakness untouched.

Instead, the failure changed the model of mathematical evidence.

The revised system:

explicitly recognises matrix, quadratic, fractional equation and function graph structures;
identifies coordinate-graph OCR only when axes and sufficient numerical evidence are present;
requires genuine matrix evidence before a matrix-specific outcome can win;
allows strong question-level evidence to override a weaker activity-title prior;
penalises outcomes whose defining mathematical concept is absent.

Regression tests then check both directions.

A quadratic graph must not become a matrix question.

But:

Given matrices A and B...

must still be recognised as a matrix question even if it appears inside an activity with an unrelated title.

The broader lesson was:

Context should narrow plausible interpretations, but semantic prerequisites must block category errors.

That principle applies well beyond mathematics tagging.

Context is useful—but only as a bounded prior

Module and activity titles provide useful information.

If an activity is titled Quadratic and Fractional Equations, that should affect how a weak or incomplete question is interpreted.

But titles must not dictate the answer.

I therefore treat titles as a bounded prior.

They can help funnel ambiguous evidence towards an appropriate syllabus family.

They cannot:

invent mathematics that is absent from the question;
override strong contradictory evidence;
turn an empty stem into a valid classification.

This is one way of combining contextual AI reasoning with explicit safeguards.

Multipart questions need a pedagogical abstraction

Another failure came from FA Mathematics multipart questions.

A page clearly contained Question 1(a) and Question 1(b), yet an early scan reported that no suitable FA Mathematics question existed.

The problem was the abstraction.

The automation initially treated nested response parts as independent objects.

Pedagogically, however, the SLS card represented one coherent task containing:

a shared stimulus;
several response parts;
separate answer keys;
possibly randomized values;
one conceptual context.

The inventory was therefore changed to treat the multipart card as a single interactive candidate.

Its shared context, nested parts, answers and randomized parameters are collected together.

That evidence can then become one coherent prompt for one interactive rather than several disconnected mini-apps.

Generating an interactive safely

Once a page qualifies for an ACP interactive, the workflow collects the full question evidence and, where applicable, its randomized parameter constraints.

It then uses the iwant2study SLS Prompt Generator and Interactive Prompt Library:

https://iwant2study.moe.edu.sg/lookangejss/promptLibrary/ai-prompt-library.html

The generated prompt includes relevant expectations for an educational interactive, including considerations such as:

keyboard and touch interaction;
readable labels;
non-colour visual cues;
accessibility;
reduced-motion support where appropriate.

The prompt is then submitted to:

SLS Authoring Copilot → Interactive (Beta)

But even here, the difficult part is not simply clicking Generate.

It is knowing when generation is genuinely complete.

Why waiting for observable evidence matters

During development, one interactive preview took roughly 150 seconds to appear.

Earlier automation assumed that if a dialog disappeared, generation had probably completed.

That assumption was unsafe.

SLS may also enter another Generating your interactive state after the user selects ADD.

The runner was therefore changed to wait for observable evidence of completion.

By default, it can wait up to 200 seconds for the completed preview or ZIP.

It will not:

click an ADD button hidden beneath an active loading overlay;
assume that a closed modal means success;
declare completion before the ZIP exists;
trust the initial state without reopening the activity.

A controlled live trial generated an interactive ZIP, selected Done, reopened the activity and verified that the ZIP remained attached.

Because that run intentionally used a one-interactive limit, the report recorded exactly what had been proven:

one complete successful transaction, not completion of the entire module.

That distinction matters when claiming what automation has accomplished.

Page breaks are not just coordinates

Pages containing several top-level FA Mathematics questions need to be separated before ACP generation.

This produced another interesting automation problem.

A page break cannot simply be inserted every time the vertical coordinate changes.

Two questions may sit side by side and belong on the same visual row.

The page-break planner therefore examines rendered geometry:

side-by-side questions remain together;
a break is proposed before the first question in the next row;
SLS must show exactly one additional page after insertion;
ambiguous layouts are skipped rather than guessed.

This is a good example of how deterministic evidence and semantic caution can work together.

Geometry provides useful evidence.

It does not automatically tell us what the author intended.

Failures became regression tests

Much of the robustness of the project came from encountering real interface behaviour.

For example:

A navigation overlay intercepted a click

A section locator was correct, but an SLS header drawer remained over it.

Rather than force-clicking through the obstruction, the automation was changed to close the overlay and verify that it had actually disappeared.

A checkbox would not accept a programmatic state change

Learning Progress could not reliably be changed through a direct checkbox operation.

Clicking the associated visible label—the way a user would—worked.

The durable rule became:

interact through the user-visible control, then verify the actual state.

An ADD button existed before it was usable

The DOM already contained an ADD control while generation was still running.

The solution was not to click faster.

It was to require the completed preview and absence of the generating overlay.

One ambiguous page stopped an entire module

Originally, uncertainty in one page could abort the larger process.

The policy changed.

Ambiguity is now local.

Leave the uncertain page untouched, explain why, and continue processing independent candidates that are clear.

Each failure became a regression test.

That is much more useful than adding another comment saying “be careful.”

Evidence is part of the product

At the time of writing, the local working revision passed:

232 automated tests;
static/package checks across 88 JavaScript files;
52 JSON files;
32 SLS configurations;
15 Windows CMD launchers;
15 macOS launchers.

Every live run can produce an evidence folder containing:

report.json;
Playwright trace.zip;
screenshots;
persistence checks;
resumable checkpoints.

Playwright Trace Viewer is particularly useful because a failed browser run becomes an inspectable timeline containing browser actions, DOM states and network activity.

Instead of:

It didn't work.

we can ask:

What did the browser see, what action did it perform, what state changed, and what evidence existed afterward?

That changes debugging from guesswork into investigation.

Authentication is a boundary, not an obstacle

SLS authentication is completed manually in a visible browser.

Reusable Playwright authentication state is stored locally under .auth/ and excluded from source control.

The workflow deliberately:

does not request a password from the terminal;
treats /login and MIMS as hard-stop boundaries;
distinguishes an expired session from a locked browser profile;
allows only the appropriate authentication helper to use its dedicated profile.

If authentication expires, automation should not become more inventive.

It should stop and hand control back to the user.

This principle extends to other uncertainty:

When confidence falls below the mutation boundary, do less—not more.

Privacy also changes what can be published

The public repository contains the reusable engineering artifacts:

source code;
launchers;
tests;
taxonomies;
configurations;
documentation.

It deliberately excludes:

authenticated browser state;
local checkpoints;
screenshots and traces;
raw workflow recordings;
credentials;
restricted learner or teacher information.

This matters because a trace or screenshot can contain lesson or account-interface information even when the source code itself contains nothing sensitive.

Publication boundaries therefore need to cover evidence artifacts, not just code.

Reproducing the project

The source is available at:

https://github.com/lookang/codexSkill

The SLS finalization skill is located at:

https://github.com/lookang/codexSkill/tree/main/sls-finalize-community-module

The Playwright package includes setup instructions, smoke checks, dry runs and guarded write workflows for Windows and macOS.

A useful principle when trying the project is:

Start read-only.

Run the smoke check.

Inspect the report.

Review which questions and curriculum outcomes have been detected.

Dry-run the page-break or ACP workflows.

Only then perform a guarded write run.

The repository continues to evolve, so the current README and launcher instructions should be treated as the operational source of truth.

What this project changed for me

The most important lesson was not:

AI can automate SLS.

That statement is too broad to be useful.

A more accurate conclusion is:

AI is most useful when its judgment is enclosed by evidence, deterministic execution, explicit mutation boundaries and observable verification.

The same idea applies pedagogically.

The Mathematics Assistant and an interactive model are not substitutes for one another.

Feedback can diagnose the learner's current error.

An interactive can expose the underlying structure, allow quantities to vary and support generalisation.

Together, they create a stronger sequence:

attempt
   ↓
feedback
   ↓
visualise
   ↓
vary
   ↓
try again

The engineering architecture follows a similar logic.

Codex provides adaptable judgment.

Playwright provides repeatable execution.

Reports and tests provide accountability.

The teacher remains responsible for pedagogical and consequential decisions.

That is the kind of AI-assisted workflow I want to build.

Not automation that is merely faster.

Automation that is bounded, inspectable, reproducible and progressively improved by every real failure.

And perhaps that is one useful direction for agentic AI in education: not replacing professional judgment, but surrounding AI judgment with enough evidence and verification that teachers can actually trust—and improve—the workflow.

AI HTML5, Open Source Physics (Easy JavaScript Simulation and Tracker) and TagUI (AI-Singapore)

Pages

Thursday, August 27, 2026