From solution steps to skill evidence: why every line is not another independent success

A written line is not always a new action

One learner splits a transformation across lines; another combines several actions on one line. Measurement based on line count can reward formatting rather than mathematical capability. Analysis needs to concern the action, its meaning and its relationship to the task.

Even a valid action needs context. Rewriting a result supplied by a hint does not provide the same evidence as independently choosing a transformation. Observations must stay separate from interpretations of knowledge: an acceptable visible solution does not reveal the learner’s internal reason for writing it.

Dependencies change the interpretation

Later steps may depend on an earlier result, so they are not automatically independent tests of the same idea. An error may propagate while a later operation is valid relative to the input the learner used. Reducing the entire solution to success or failure hides that distinction.

Alternative methods need content coverage

CTAT’s tutorial demonstrates alternative solution paths with associated errors and hints. In that example, a correct path absent from the graph is not recognised until an author adds it. This illustrates the role of content coverage, rather than a judgement about all tutoring systems. Official tutorial.

For Warda, accepting different methods requires an explicit account of what is acceptable and why. A valid method should not be rejected for differing from a reference solution, nor should every transition be accepted because the final result matches. Verification, content authoring and decisions about support meet at this boundary.

Assisted evidence is different evidence

Warda’s approach distinguishes attempt context and assistance when interpreting answer evidence. These boundaries matter before making mastery inferences. A hint may support learning, but it changes the interpretation of the response that follows; its role should not disappear when reviewing performance evidence.

We therefore do not propose feeding each written step into a mastery model as another question response. The design first needs to define the evidence unit: which action is sufficient, how actions map to skills, and how ambiguity, assistance and carried-forward errors affect interpretation. These are measurement questions, not simply more detailed chat messages.

Mastery is an inference with conditions

BKT distinguishes an observed response from the knowledge state it seeks to estimate. A correct response can occur without stable knowledge, and a knowledgeable learner can make an error. Its result is an estimate based on assumptions and evidence, not direct access to understanding or an examination grade.

Warda’s design establishes evidence eligibility before using it in a mastery inference. A step copied from a hint, an independent attempt and a transition carrying an earlier error are not interchangeable observations. Skill mapping needs an explicit meaning for each observation and limits on the inference it supports.

This approach connects a decision with its rationale: why might another attempt help, and which idea warrants review? A useful statement about capability requires more than additional signals. It requires knowing what each signal represents and what it cannot establish.