Three days, three failures: is AI-driven overconfidence breaking our systems?
This week a global Microsoft outage, a halt of all departing air traffic in the US and an exploded Starship launcher came within 48 hours of each other. The official explanations point to network upgrades, old equipment and test-flight hardware. I expect the post-incident reports to show that overconfident use of AI in system development and testing, combined with fewer reviewers and schedule pressure, was a contributing factor in each.
In the meantime, Microsoft cites a network capacity upgrade and a later network change. The FAA says new technology overloaded antiquated parts of the system. And Musk blamed engine computer software for the Starship booster explosion. All of this examples of what rushed, automated software development will increasingly look like. Already there is evidence from different research studies:
- Higher AI adoption is associated with lower delivery stability, and instability persists in the latest DORA data.
- Experienced developers using AI tools were 19% slower while believing they were faster.
- LLM-generated code introduced serious vulnerabilities in 45% of test tasks.
- Microsoft says 20 to 30% of its code is written by software, while its 2025 layoffs hit software engineers hardest.
More generated code, fewer reviewers and faster releases make systems brittle.
I expect that soon we'll see that this week was no coincidence. When the reports on this week's events come, check who wrote and reviewed the change, how much was automated, what testing preceded deployment and how many experienced people were left to catch the error.
GenAI produces fluent output, and fluency is easily mistaken for competence. Confunding the two is leading to organisational decisions to cut on the people who do and check the work, will keep producing weeks like this one.
This is why governance matters. Aviation, medicine and nuclear power became safe through independent investigation, mandatory incident reporting, clear liability and professional standards, not through the goodwill of their builders. Digital infrastructure now carries comparable weight, yet is still governed by voluntary assurances and press statements after the fact. Responsible AI is about who is accountable when AI-assisted code and decisions enter critical systems. That requires: ➡️Accountability: a named responsible party for every automated change to critical systems.
➡️Transparency: records of where AI was used in design, coding, testing and deployment, open to auditors and investigators.
➡️Oversight capacity: enough qualified people, with time and authority, to challenge what the tools produce.
This is not anti-technology. It lets us benefit from AI without the public paying for it as the test environment. The question is not whether AI will be used in critical systems, but whether we demand regulatory oversight to know how, by whom and under what safeguards it is used. We cannot wait for the next failure to ask.
