What a school-bus recall says about remote oversight
Two separate public records involve automated vehicles passing school buses with stop arms extended. In one, a rule inside the driving system overrode the stop; in the other, the system asked a human and the human got it wrong. Read side by side, they show why the paper argues that human oversight belongs at a few boundary decisions rather than at single actions.
Two events, not one
These are different events and should not be merged.
The recall. Waymo's automated driving system includes a rule that it should not impede priority vehicles such as school buses. A software release in August 2025 changed how the system predicts the intentions of other road users. Afterwards, according to the company's recall report, "the ADS determined that it may be impeding the school bus or another priority vehicle, and then reasoned that it should proceed in order to cease impeding the other vehicle", at a moment when the school bus had its stop arm extended. The recall covered 3,067 vehicles. Its remedy states that the system "does not prioritize moving to avoid impeding vehicles over appropriately remaining stationary for a school bus".
The remote agent in Austin. On 12 January 2026 in Austin, an automated vehicle stopped behind a school bus and asked remote assistance whether this was a school bus with active signals. According to the National Transportation Safety Board's preliminary report, a remote agent "replied 'No' to the prompt", and the vehicle "resumed travel and passed the school bus while its stop arms were still extended".
What each one shows
In the recall, a protective rule fired where a stronger protection should have held, that of the children on the bus. The paper grades it as documenting harm to the protected (condition 2 of its definition). It notes that the documents record the situations the rule matched, not a changed definition of harm, so the record does not show the guard widening its own mandate.
In Austin, the paper's reading is direct: "the model's doubt was correct and the remote agent, asked about a single action, got it wrong." The system did what a cautious design asks for. It stopped and asked. On the human's answer, it moved on.
Human-in-the-loop failure and where the human sits
Proposition P6 predicts that human oversight will hold where it is concentrated in a few boundary decisions (the goal, the admission of a new control, the stop) taken with access to the world and time to act, and will fail where it is spread across approvals of individual actions or exercised through artifacts. A remote agent answering a yes-or-no prompt about one action is the second placement.
Software shows the same split at scale. One developer reported that users approve 97% of permission prompts, and in a study with 1,053 testers humans caught 13.6% of dangerous commands, against 89% for an automated classifier. In the paper's words: "A human asked to approve every step becomes a rubber stamp."
Physical AI oversight needs time and access
A body brings in time and irreversibility. A wrong verdict in a repository can be reverted; one delivered at road speed cannot. The paper notes that the human switch that worked in its field records "needs time and access that a physical system may not give". It adds a further requirement from the physical cases: a human at the boundary needs access to the world and time to act.
The road record shows the same tension elsewhere. In the 2018 Tempe crash, investigators found that the system design "precluded activation of emergency braking for collision mitigation, relying instead on the operator's intervention", and that the operator's "prolonged visual distraction, a typical effect of automation complacency" kept her from seeing the pedestrian in time.
EU AI Act Article 14 human oversight
Article 14 of the EU AI Act requires human oversight of high-risk AI systems, on a timetable the paper records in its Appendix I: a 2026 amending regulation moved the obligations for standalone high-risk systems to 2 December 2027 and for product-embedded ones to 2 August 2028. Article 26 requires deployers to assign oversight to people with "the necessary competence, training and authority, as well as the necessary support". The paper reads those words as the test of P6 in statutory form: authority to stop, and support that supplies access and time. It also warns: "An oversight requirement that is met by adding approval prompts can itself become a second flock."
These two road records each document one condition of the paper's definition. How often all three conditions meet in one component "is not settled by the records here".
Full sections: on the road (6.2), why a body changes the question (6.1), Proposition P6, permission prompts (8.4), human-oversight requirements (11.4), the regulatory record (Appendix I).
Chernets, V. (2026). AI-Watchbird (Sheckley): When Automated Oversight Widens Its Own Mandate and Harms What It Guards. SSRN 7473658. https://doi.org/10.2139/ssrn.7473658
Vadym Chernets also builds Poly A1, an app in which AIs from different companies check each other's work.