November 4, 2026 • Systems Engineering

AI Is Infrastructure. Run It Like Good IT.

An IT technician listens to a smiling yellow AI box with a stethoscope while a server, a printer, a router and a laptop wait in line for their own check

On 30 September, Anthropic told developers using Claude Sonnet 4.5 that the model will be retired from its API on 30 November. That is about two months’ notice, which is in line with the vendor’s published commitment of at least 60 days, and after that date requests to the old model simply fail. If that model is sitting inside your claims triage, your document review or your client reporting, you now have a change to plan, test and release on a date the vendor set. Three weeks later the same vendor shipped an API version that rejects the temperature and sampling settings outright, so code that sets them gets an error back the day it moves to that version. Vendors retire services on their own schedules, and IT teams have long managed that. The same habits of keeping an inventory, pinning versions, testing changes, monitoring production and keeping a way back apply to AI models, and they leave your own experts deciding what a correct answer is.

A model is a service that changes on the vendor’s schedule

Most organisations first meet AI as a clever assistant, and the habits that come with that framing are personal ones. Someone finds a prompt that works, shares it, and the prompt quietly becomes part of how the business runs. The trouble is that the thing behind the prompt keeps changing, sometimes under the same name.

Researchers at Stanford and Berkeley showed this early. Chen, Zaharia and Zou compared the March 2023 and June 2023 versions of GPT-4 and GPT-3.5 on the same tasks. GPT-4’s accuracy at telling prime from composite numbers fell from 84% to 51%, partly because the June version followed the step-by-step reasoning instruction less reliably. On a set of 50 LeetCode problems, the share of GPT-4’s answers that passed LeetCode’s checker as written fell from 52% to 10%, and a big part of that was the newer version wrapping its code in Markdown formatting it had been told not to add. With that formatting stripped out, GPT-4 went from 52% in March to 70% in June. Some tasks got better over the same period. The authors’ conclusion was that the “same” service can change substantially in a short time, and that anyone relying on one inside a workflow should monitor it the way they did.

The vendors are candid about this too. In April 2025 OpenAI rolled back a GPT-4o update in ChatGPT after it made the model overly flattering and agreeable, and said the update had leaned too hard on short-term user feedback. In September 2025 Anthropic published a postmortem of three infrastructure bugs that had intermittently degraded Claude’s answers between August and early September. One was a routing error that, at its worst hour, affected 16% of Sonnet 4 requests, and another occasionally dropped stray Thai or Chinese characters into English replies. Anthropic says the problems came from infrastructure bugs alone. Anthropic’s own account is that its evaluations didn’t capture what users were seeing, and one of its fixes was to run quality evaluations continuously on its real production systems.

I read the OpenAI and Anthropic write-ups as good engineering behaviour by the vendors, and as a plain statement of the operating conditions. The model you call is a service with its own release train, its own incidents and its own retirement schedule. It pays to expect it to behave differently over time, and to set things up so you spot the change before your customers do.

What good IT already knows

Google’s Site Reliability Engineering book reports that roughly 70% of outages are due to changes in a live system, and that the answer is progressive rollouts, fast and accurate detection of problems, and safe rollback. The useful thing is that those habits carry across to AI well, mostly with small changes of vocabulary.

It starts with an inventory. A dependency becomes manageable once you know it exists, so you write down which processes call which models, which exact version each one calls, who owns that step, and what happens to the business if it misbehaves. It sounds basic, and it is the first thing worth checking, because every other habit depends on it.

Next comes change control. Where a vendor offers both pinned model IDs and shorter aliases, you call the pinned ID. Anthropic, for example, says it doesn’t change the model behind a pinned ID, and its aliases for older models point to the latest snapshot. You also treat a model upgrade, a prompt change or a new tool permission as a release. A release needs a test, and the test that matters is a set of your own cases with answers your experts agree are right. That set is the part only you can write, and it is the most useful score you will have, because it measures performance on your own work.

Monitoring is the third habit, and it is the one the postmortem above argues for most strongly. If the vendor’s own evaluations missed a degradation that its users noticed, catching the next one takes testing that keeps running long after launch week. A small slice of known cases run on a schedule against production, with someone who reads the results, catches drift before it becomes a complaint.

Access is the fourth. An agent that can read a mailbox, write to a ledger or send on someone’s behalf is a privileged account and deserves the treatment you give one, which means a named owner, the least access the job needs, and a log someone can review.

Then there is the incident and the way back from it. When an answer goes wrong, you want to be able to say which version, which prompt and which inputs produced it, and you want the previous working configuration still runnable so that rolling back means switching back to that configuration. For the steps that really matter, you also keep a manual path that people still know how to use.

The last habit is an exit plan. Every hosted model you depend on will eventually be retired. The Sonnet 4 snapshot dated May 2025 was retired in June 2026, roughly a year later, with the vendor recommending a replacement. If your regression cases already exist, a forced migration is a few days of testing and a release. Starting from scratch, you spend the notice period rediscovering what “working” meant.

APRA already thinks this way about critical operations

I’ve run CPS 230 and CPS 234 audits and uplift programs in financial services, and the regulator’s language maps neatly onto this problem. APRA’s operational risk standard, CPS 230, asks banks, insurers and super funds to maintain a register of their critical operations, keep those operations running within set tolerance levels through severe disruption, monitor the age and health of their information assets, and regularly test their controls for design and operating effectiveness. It also expects them to manage the service providers those operations rely on.

CPS 230 doesn’t name AI models, so what follows is my own reading of it. Still, if a model sits inside a critical operation, it is hard to see why it would escape the same expectations as the database or the payments gateway beside it. Even if you are nowhere near APRA’s remit, those expectations are a sensible checklist for any business that would be hurt by a quiet change in a supplier’s service.

Who decides what a good answer is

The domain expert decides what a good answer looks like, writes or approves the cases that prove it, and sets how much drift the business can tolerate. Our job at Pairti is to put the rails under that judgment, which means pinned versions, regression cases, monitoring, named access and a rollback path, so that when the next vendor notice arrives, you run your test cases against the replacement model, fix what fails and release it on a date you set. Your experts still decide what counts as a correct answer, and the model is managed like any other system it sits beside.

If you’d like to know which of your processes now depend on a model version nobody pinned, and what would happen on the day that model is retired, start with a Diagnostic, or ask me through the chat button on this page.

Sources