An AI-Assisted Deliverable Is Not Yet Evidence That Its Owner Can Maintain It
Assess AI-assisted workplace learning with a separate maintenance task. Use a practical debugging rubric while keeping delivery speed and competence distinct.
A junior analyst delivers a working importer with AI assistance. The demonstration passes, the deadline is met and the team is pleased. A week later, the input format changes. Can the analyst identify the failed assumption, repair the importer and explain the risk of rerunning it?
That maintenance question concerns a capability the successful demonstration did not necessarily measure. A team can benefit from AI-assisted production while still needing a separate process for developing people who can supervise the result.
Research available in October 2026 makes the distinction timely. The practical response is not to count prompts or ban useful tools. It is to define the independent decisions a role requires and create a small, observable check of those decisions.
Distinguish production support from an apprenticeship task
Production work aims to deliver an acceptable result under the organization's rules. An apprenticeship task also aims to change what the worker can understand and do. A single assignment can serve both goals, but completing the production goal does not automatically establish the learning goal.
Before assigning unfamiliar work, name the capability the person should acquire. “Use the new library” is vague. “Explain its cancellation behavior and diagnose an interrupted operation” gives the mentor something observable.
Also state which resources are permitted. An unaided reasoning check can still allow the language reference or a supplied API sheet if memory of syntax is not the target. The conditions should reflect the capability being evaluated, not an artificial test of typing from memory.
Read the coding evidence within its actual scope
Shen and Tamkin's How AI Impacts Skill Formation, first submitted to arXiv on January 28, 2026, studies learning an unfamiliar asynchronous Python library. Its randomized coding experiment reports weaker conceptual understanding, code reading and debugging under AI assistance, without a statistically significant average completion-time improvement. Qualitative interaction patterns differed: some involved more active understanding than others.
This is a bounded research result, not evidence that every AI workflow reduces every engineer's competence. The behavior categories were observed within the study; they were not separately randomized interventions that establish which particular prompt strategy causes better learning.
The practical inference for a team is narrower: measure the skill needed to supervise work rather than treating the completed artifact as its proxy.
Keep short-term withdrawal effects separate from lasting skill loss
A research version revised October 13, 2026, by Liu and colleagues reports randomized experiments with 1,222 participants across arithmetic and reading tasks. AI access improved immediate task performance, while subsequent unassisted performance and persistence were lower after access was removed.
Those brief experiments do not measure a year of workplace development or prove irreversible loss of skill. They help explain why an assessment performed entirely with an assistant can miss what happens when the person must reason independently. The arXiv version is cited here directly; no claim about a newer peer-review status is required.
For managers, this evidence supports separating observations. Delivery, independent maintenance and willingness to investigate a difficulty belong in different fields rather than one blended productivity score.
Build one realistic maintenance probe
Return to the fictional importer. The original task accepts a documented CSV layout and produces a validated result. For the maintenance probe, change one relevant condition: a required identifier is missing, a timestamp has an unexpected offset or the input contains a duplicate business record.
Supply a tiny synthetic fixture whose correct handling is known. Ask the analyst to predict the failure, locate the assumption and propose an appropriate next step before opening an AI assistant. Keep the probe narrow enough that the mentor can inspect the reasoning.
This is an original assessment design, not a reproduction of either study. Its value depends on how well it represents the employee's role. A puzzle unrelated to the production system can assess puzzle solving while missing the actual supervision responsibility.
Score decisions with visible evidence
A compact rubric helps mentors distinguish an unsupported guess from a defensible diagnosis. Keep the person's explanation and proposed test beside the score so another reviewer can inspect the judgment.
| Capability | Observable evidence in the fictional importer task | Incomplete evidence |
|---|---|---|
| Explain the contract | Names required fields and the output expectation | Restates that the script imports files |
| Diagnose a failure | Connects a changed input to a specific assumption | Pastes an error into a tool without interpreting it |
| Design a check | Gives a small input and expected behavior | Says that tests should be added |
| Choose a safe next action | Explains repair, rollback or escalation boundaries | Reruns against real data without resolving uncertainty |
The table is a discussion aid, not a validated proficiency scale. Use it to decide what needs practice, not to infer a person's entire professional value from one short task.
Allow repair without erasing the first observation
After the initial attempt, bring back the normal resources: documentation, the mentor and approved AI tools. Let the worker investigate, correct the implementation and explain the revised model.
Record the opening diagnosis separately from the final repaired result. Otherwise a successful collaborative repair can overwrite the evidence about what was independently available at the start. Equally, an opening failure should not erase the learning that occurred during the repair.
Ask a follow-up question that changes the condition again. If the person now predicts the consequence and designs a check, that is new evidence of understanding under those conditions. It is more informative than requesting the exact explanation they have just read.
Protect time for the capability the role needs
A mentor can choose one small maintenance capability per assignment instead of demanding that a junior worker independently recreate an entire application. Explain why the capability matters: the person may be responsible for investigating failures, reviewing changes or escalating uncertain outcomes.
Avoid turning every production deadline into an exam. Separate a planned learning probe from an urgent incident. During an incident, the team's immediate obligation is recovery; a later review can identify what training would make future supervision stronger.
Evaluate a small program before expanding it
For a pilot, keep three kinds of evidence: accepted deliverables under normal assistance, maintenance attempts under stated conditions and later follow-up attempts. Compare like tasks and note prior experience. Do not call a higher later score a causal effect of the training unless the evaluation design supports that inference.
If people deliver faster but struggle with diagnosis, add targeted practice in tracing assumptions. If independent diagnosis is strong but recovery procedures are unclear, improve the operational handoff. Different observations should lead to different changes.
For evaluating a completed artifact, see work-sample rubrics. For the receiving owner's responsibilities, see workplace handoff acceptance. The purpose of an apprenticeship is to develop someone who can take responsibility for the next change, as well as finish today's task.
Sources and limits
- Shen and Tamkin, 2026: How AI Impacts Skill Formation: primary research manuscript; coding-task methods, results and qualitative analysis reviewed.
- Liu and colleagues, revised October 13, 2026: AI Assistance Reduces Persistence and Hurts Independent Performance: primary research version; short arithmetic and reading experiments reviewed.
Checked October 28, 2026. The importer scenario, maintenance rubric and pilot design are original practical adaptations. Neither study directly validates this workplace program, establishes long-term effects for every profession or justifies treating AI use itself as evidence of poor competence.