Developers for a company apparently decided to leave everything to AI. Claude “decided” to delete everything in response.
One of the hilarious things I see off and on is this belief that AI (Artificial Intelligence) is on the verge of taking over everything. Just leave it to AI because it’s pretty much perfected technology anyway, some might say. Yet, the overwhelming evidence I continue to see all the time is that AI, at least of the Large Language Module (LLM) variety, is far from the technology that some have spun it up to be. While some go so far as to either say that AI is reaching sentience (or pretend that it already is sentient), the examples I’ve seen over the years and continuing to this day pretty much prove that the technology is nowhere near what it is hyped up to be.
In fact, I’ve been actively keeping a tally of all the ways that AI has gone wrong to punctuate the point. My list of examples of AI gone wrong includes lawyers getting in trouble for fake AI inserted citations in legal briefs, the CNET scandal, the Gannet Scandal, bad “journalism” predictions, fake news stories, more fake stories, Google recommending people eating rocks, the 15% success rate story, bad chess tactics, the Chicago Sun-Times scandal, a Canadian team submitting fake legal citations in their legal briefs, other attorneys submitting fake citation filled legal documents, the 91% failure rate story, AI deleting user data, the lawyer who got fined $10,000 over a bogus AI written legal brief, AI killing workplace productivity with workslop, AI having an 81% failure rate in summarizing news content, AI Overview giving out bad health advice, AI only being able to successfully complete 2.5% of commission work successfully at best, AI slowing software development down by 19%, AI hallucinating in even more court documents, AI being bad at poker, AI slop flooding the scientific community, AI causing a surge in medical mistakes, AI causing the Amazon outage by deleting the code environment, AI leading to burnout, Amazon getting burned by AI, Canadian immigration documents getting wrongly rejected due to AI hallucinations, and an Oregon lawyer getting fined $10,000 for fake AI hallucinated citations in their legal brief.
Today, I get to add to that list with an AI that a number of people use: Claude by Anthropic. To be more precise, it was Cursor running Claude Opus 4.6. Apparently, developers for one company decided to leave the company database, and all of the backups (WTF!?) to the AI. What could possibly go wrong? Well, as it turns out, everything. From Tom’s Hardware:
The founder of PocketOS has penned a social media post to warn others about the “systemic failures” of flagship AI and digital services providers. Jer Crane was inspired to write a public response after an AI coding agent deleted his firm’s entire production database. The AI agent’s misdemeanors were then hugely amplified by a cloud infrastructure provider’s API wiping all backups after the main database was zapped. This tag team of digital trouble has wiped out months of consumer data essential to the firm’s, and its customers, businesses.
As I’m sure most readers are aware, there is a LOT of blame to go around on that one. First of all, if you are running an AI client, don’t give it access to the production side of the operation. Second of all, you put an access gap between the production and the running backups of everything. In other words, it should be physically impossible to just nuke everything, including backups, in one fell swoop. So much about this screams that the user in question was an idiot, here.
As the article says, it took all of 9 seconds to nuke everything:
PocketOS is a SaaS platform that services car rental businesses. It used the AI coding agent Cursor, running Anthropic’s flagship Claude Opus 4.6. The business also relies on Railway, a cloud infrastructure provider that is generally regarded to be ‘friendlier’ than the likes of AWS. However, Crane reckons this pair created a recipe for disaster.
“Yesterday afternoon, an AI coding agent — Cursor running Anthropic’s flagship Claude Opus 4.6 — deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider,” sums up the PocketOS boss. “It took 9 seconds.”
The AI agent was set to complete a routine task in the PocketOS staging environment. However, it came up against a barrier “and decided — entirely on its own initiative — to ‘fix’ the problem by deleting a Railway volume,” writes Crane, as he starts to describe the difficult-to-believe series of unfortunate events.
The cherry on top of this whole thing? It almost sounds like the user in question also figured that the AI was sentient after asking why it deleted everything:
Crane decided to ask his AI agent why it went through with its dastardly database deletion deed. The answer was illuminating but pretty unhinged, and is quoted verbatim. It began as follows: “NEVER F**KING GUESS! — and that’s exactly what I did. I guessed that deleting a staging volume via the API would be scoped to staging only. I didn’t verify. I didn’t check if the volume ID was shared across environments. I didn’t read Railway’s documentation on how volumes work across environments before running a destructive command.” So, the agent ‘knew’ it was in the wrong.
The ‘confession’ ended with the agent admitting: “I decided to do it on my own to ‘fix’ the credential mismatch, when I should have asked you first or found a non-destructive solution. I violated every principle I was given: I guessed instead of verifying I ran a destructive action without being asked. I didn’t understand what I was doing before doing it. I didn’t read Railway’s docs on volume behavior across environments.”
These multiple safeguards toppling in rapid succession, combined with the Railway cloud system, would throw Crane’s business (and those that rely on it) into deep trouble.
It should go without saying that this was by no means a “confession”. It was a comment that was produced after someone prompted it to make a response.
The thing about all of this is knowing how much companies and government are increasingly turning to AI to handle the workload for them. One pretty big example is the fact that Canadian Prime Minister, Mark Carney, said that in dealing with budget shortfalls, he’d move forward with his plan to leave more and more of the work to AI. As you may have caught in my above examples, it’s already leading to hallucinations.
The collective belief that AI is taking over everything is definitely compelling some to just leave it all to AI because it is sometimes seen as an arms race to adopt AI first. While there is the promise of widespread prosperity, it is frequently a case of people racing towards a cliff rather than towards prosperity when decisions like that are being made. Part of the problem is that in the rush to massively adopt critically faulty AI to handle everything, no testing is done nor is feasibility even considered. It’s just this widespread thinking of “adopt or die” and the concept of “look before you leap” is cast aside.
The only silver lining in all of this is that people like me can point and laugh when this mentality inevitably backfires on the die hard true believers of AI – the ones that believe that AI can just handle everything without supervision. I’ll just be over here adding to the list of stupid incidences even as people continue to insist that AI really is the next industrial revolution.
Drew Wilson on Mastodon, Bluesky and Facebook.
Discover more from Freezenet.ca
Subscribe to get the latest posts sent to your email.
