AI feedback for coding learners needs evidence of learning beyond a fast response or a favorable survey. Before relying on a tutor, ask three separate questions. How long does the workflow take? Does the feedback match a defensible answer standard? Can you later solve a new task independently?
A new US university pilot offers a useful case for this audit. We reviewed the primary preprint, checked its denominators and built reusable evidence tools. We did not evaluate the platform, call a tutoring model or conduct a student experiment.
What the 36-student pilot reports
Moraish Kapoor and Jiawei Zhou submitted the Edhivo preprint on 27 October 2026. Its Spring 2026 deployment involved 36 graduate students in a Stony Brook University large-language-model course. The post-pilot survey received 30 responses.
All respondents agreed or strongly agreed on ease of use and the perceived benefit of instant feedback. That describes respondent opinion. The six nonrespondents’ views are unknown; nonresponse does not mean disagreement. The findings are not a representative estimate for US beginner programmers.
Separate three layers of evidence
Our proposed framework keeps operational speed, rubric agreement and retained learning apart. A quick result can still contain a mistaken explanation. A correct explanation can still leave a learner unable to use the idea later. Give each question its own evidence and limitations.
For speed, define the start and finish. Does the timer include preparing a question, waiting for an answer, checking the feedback and fixing errors? A generation-only timer cannot describe the whole workflow. Save raw observations rather than only a rounded average.
For agreement, write the expected answer and rubric before inspecting a tool’s response. Record disagreements instead of silently replacing them with a consensus score. For learning, use a later unseen exercise with a declared delay and an independent attempt. These are proposed checks, not results from this article.
Read preparation-time estimates in context
The paper compares an informally estimated 30-minute manual preparation baseline with a reported 10-second generation median plus five minutes of human review. The combined arithmetic is about 5.17 minutes, rounded to 5.2 in its table. This is not a controlled comparison across instructors.
Our chart uses a hatched outline for the manual estimate and separates generation from review. The visual labels matter: a reader should not mistake unlike evidence types for two equally measured experimental groups. We do not convert the comparison into a headline percentage saving.
If you evaluate your own workflow, define the same task and quality standard for both methods. Log revisions and failed attempts. Otherwise, a quicker draft can hide work that simply moved into checking. Keep uncertain or missing times blank, with an explanation.
Audit the learning claims attached to AI coding feedback
Formal human–AI grading reliability remains future work in the preprint. Its survey does not establish durable transfer to unseen tasks. The authors also flag possible linguistic bias. Their classroom assessments were formative, rather than contributions to official course grades.
Use an evidence ledger to separate reported observations, estimates, opinions and unresolved questions. A blank result is useful when the study does not supply one. Do not replace “not established here” with either “proven effective” or “proven ineffective.” Both would exceed the evidence.
| Your question | Evidence to request |
|---|---|
| Is the full workflow faster? | Comparable tasks, timing boundaries and human review time |
| Is the feedback defensible? | A versioned rubric, human review and visible disagreements |
| Does understanding last? | An independent later attempt on an unseen task |
Try a proposed four-answer tutor audit
Our original worksheet uses a simple prompt: explain an algorithm that sums a list of numbers, including empty-input behavior. We wrote four fictional answers. They contain correct reasoning, a subtle misconception, incomplete reasoning and fluent but irrelevant prose.
In the misconception case, the answer places the return inside the loop while claiming to sum a list of any length. A suitable review should catch the early return and explain a two-item counterexample. The incomplete answer calls for missing steps; it does not justify inventing details the writer never supplied.
These expected feedback notes form a proposed rubric, not an authoritative benchmark. All four cases are unexecuted. The observed-feedback and reviewer-disagreement fields remain blank. A teacher can revise the rubric before using it; no model has earned a pass rate from this worksheet.
Tech Consulting Tailored to Your Coding Journey
Get expert guidance in coding with a personalized consultation. Receive unique, actionable insights delivered in 1-3 business days.
Get StartedKeep a later independent attempt in your learning log
For private practice, record your first attempt, what the feedback changed and what you could explain afterward. Later, try a fresh problem without the tool. Write down any help used. This makes your own process easier to inspect without claiming that one exercise proves general educational effectiveness.
For a coding example, change the input boundary or ask for a counterexample. Our Sudoku uniqueness project shows how a found answer differs from a stronger claim. Our portfolio evidence audit helps attach specific checks to the claims you publish.
Keep private coursework and identifying information out of public examples. If you turn a practice log into research involving other people, obtain appropriate consent and review before collecting data. The downloadable templates contain no participant records and require none to understand the framework.
Download the evidence tools
Inspect our source evidence register (CSV), claim-to-evidence ledger (CSV), four unexecuted tutor fixtures (CSV) and blank learning log (CSV). The source register links exact paper sections. The other files contain original proposed checks with observation fields left empty.
Our arithmetic uses the reported counts directly: responses divided by invitations, and seconds converted to minutes before adding review time. We did not reconstruct individual survey responses or split agreement categories that the text does not enumerate. All figures were plotted from retained inputs and explicit classifications.
What remains uncertain
This is an evidence appraisal of one small, single-course preprint. It cannot establish unbiased automated grading, lasting coding gains or suitability for high-stakes assessment. No independent tool evaluation supports a recommendation here. Read the full study and its limitations before using its claims.
When choosing how to use feedback, keep the next decision concrete. Check a questionable explanation, preserve disagreements and attempt a fresh task independently. Those actions produce evidence you can examine; a quick reply and a reassuring score cannot answer every learning question.
