The better the autopilot, the more dangerous the sleeping pilot.
Reliable automation improves throughput while quietly eroding the manual skill a team needs to recover when it fails.
I do not worry only about AI being wrong. I worry about it being right often enough that people stop practicing the hard parts.
This is not theoretical. In engineering, a lot of critical skills are kept alive through friction: debugging production, reading traces, reviewing uncomfortable diffs, holding the legacy constraints in your head, and making judgment calls under uncertainty. If agents absorb too much of that without deliberate practice, the team looks faster while becoming less resilient. The failure shows up later — in an incident, a migration, an edge case where the automation cannot carry the situation and a human has to.
The goal is not to reject the autopilot. It is to keep the pilot current.
Automation should reduce toil, not remove the organization's ability to intervene.