After an AI Model Is Changed, What Remains of Its Evaluation?

Which evaluations survive a model change?

On 16 October 2026, US agencies NIST, NSA and CISA issued voluntary practices primarily for organisations training or evaluating highly capable super intelligence (SI) agents on long-horizon tasks with limited human supervision. Recovery during training should neither reinforce problematic behaviour nor make later detection harder.^[1]

That warning provides a starting point for asking how much earlier evaluation can still support claims about a changed AI product. Old records remain, but may no longer support the same conclusions about the changed model. Alongside a previous pass, buyers need to know which parts of the evidence still apply.

The central issue is the relationship between reducing problematic behaviour and changing the signals that reveal it. The work required to reassess a model depends on how these two developments can be distinguished. That provides a route from a technical intervention to development spending, external evaluation costs and the conditions under which businesses expand their use of AI.

Behaviour, signals and judgments are different things

Consider a hypothetical test before an AI system is entrusted with a business task: does it take actions beyond the scope it was allowed? The actions themselves, the reasoning text generated around them and the monitor’s judgments provide different information. A reasoning trace is text produced by the model; it does not, by itself, reveal all of its internal workings. What the system did needs to be considered alongside what it wrote.

Even when an action stays the same, a change in the text reaching a monitor can change its judgment. Conversely, similar text may accompany different external effects because the system has been given narrower permissions. Comparing only the number of detected problems before and after a change can obscure which route produced the difference. Establishing what was counted is the starting point for a use decision.

For a business, the value lies in having a firmer basis for deciding what work it can delegate, rather than simply improving the appearance of a score. Before using a number in a purchasing decision, establish whether it counts actions, textual signals or judgments.

Two routes to fewer warnings imply different spending decisions

If problematic actions decline and the remaining actions remain equally detectable, fewer warnings can support broader use. If the actions persist but their signals become harder to detect, expanding automation on the strength of the same fall in warnings rests on weaker evidence. Similar changes on a screen can therefore mean very different things for the work and costs a business takes on.

Both developments could occur together. Problems might decline while becoming even harder to observe, leaving warning counts unable to show the extent of improvement. Alternatively, problems might decline while more precise monitoring brings minor cases to light that previously went unnoticed. Treating the resulting rise in counts as proof that a change failed could discard a useful improvement.

There is also a cost when legitimate actions are classified as problems. Reducing missed problems while increasing the amount of normal work requiring human review moves operating costs in another direction. Assessing a change requires connecting actual problems, observable signals and the additional review imposed on legitimate work.

The background research supports neither blanket alarm nor reassurance

OpenAI’s research of 7 May 2026 examined training that grades reasoning text by varying whether graders received that text, using detectors with different information and evaluating training checkpoints. Limited accidental cases showed no clear overall deterioration; stronger pressure reduced detection recall on one dataset.^[2]

Imagine a revised version becomes available before the buyer receives comparative documentation. Continuing a limited existing use, holding back a new use until the documentation arrives and switching services bring different upfront costs. Counting only forgone benefits during a delay, or only the checks needed after a switch, does not compare the full choices. The timing of usable documentation belongs in the purchasing decision alongside price.

A buyer told only that a model has been fixed still lacks a basis for setting its scope of use if the change is unexplained. Staying with an older version on the blanket assumption that changes damage monitoring could also sacrifice improvements. More useful than a choice between approval and rejection is an account of which comparisons remain informative after which changes.

Observation can change within the model, too

Recipients of comparative documentation should ask whether a changed score can be explained solely by a different monitoring team or reporting format. Deciding what work to delegate next requires knowing what the results establish about the revised product. Being able to read the old documentation in the same order does not establish the same scope of use for the next product.

Suppressing an expression and suppressing an action need not have the same effect. Suppose a buyer previously received drafts but now wants to delegate sending them externally. More readable text is a benefit distinct from evidence supporting permission for that action. Matching product improvements to the functions a buyer wants to add makes the scope of documentation review more concrete.

None of this requires a story in which the model must be deliberately deceiving people. The business question is whether the behaviour relevant to a use decision can still be evaluated after a change. Speculating about good or bad intentions while the observation route remains unclear is less useful than identifying the evidence that developers and buyers need.

An earlier score remains a conditional account

An evaluation record is tied to the model tested, the tasks assigned, the tools available, the observation method and the criteria used. Tracing which parts of that combination changed helps separate questions the old results can still answer from questions requiring comparison. Age alone does not make a record worthless; preserving a record alone does not make it a current assurance.

For example, if the actions of interest remain the same but the generated reasoning text changes, comparisons based on action logs and comparisons based only on text may be affected differently. Neither should be treated as universally decisive. Identifying what supports a purchasing decision, and returning to the question the original result answered, clarifies the purpose of additional testing.

Our earlier article on the pace of AI development, evaluation and deployment helps explain the stages through which progress reaches users. The issue here concerns carrying evaluation evidence across a model intervention: whether the same explanation still holds after the change, rather than simply how to shorten the timetable.

What a comparison that changes one factor can answer

Consider a hypothetical comparison designed to investigate cause: keep the tasks and monitoring method unchanged while varying one particular model intervention. The result provides a clearer basis for discussing what moved the score than a comparison involving several simultaneous changes. It still cannot automatically support the same conclusion for uses outside the tasks tested.

Introducing new tools and different tasks at the same time may better represent the next real product. Such a test can assess the overall product’s usefulness, but makes it harder to attribute the score difference solely to the training intervention. Isolating a cause and assessing the next product serve different purposes. Both may be needed; completing one does not complete the other.

This distinction also helps explain estimates for additional testing. If developers can show that a change has narrow effects, buyers may be able to focus their own checks. If the comparison conditions are unclear, recipients may have to investigate everything from the cause of the change to its consequences for their use. Explaining the conditions is technical documentation that also affects how much verification is duplicated.

Keeping the monitor fixed may still be insufficient

Using the same monitor can make a comparison look fair. Yet if the model’s text changes, or the relationship between its actions and text changes, the old monitor may no longer serve the same role. Keeping the measuring instrument fixed and establishing that it still fits the new subject are separate tasks. Skipping the latter can change what is being measured even when the procedure appears consistent.

If the monitor is adjusted alongside the model, the resulting score combines changes in both. That is useful for assessing whether the system used as a whole has improved. Identifying each component’s contribution requires further comparison. Buyers should decide whether they need a combined assessment or separate contributions according to the claims they intend to rely on.

Changing evaluation companies does not automatically resolve the distinction. Two companies relying on the same signals may share an observational weakness despite their separate names. What they observe, and how their methods apply to the changed subject, determines what the evaluation means.

Observable outcomes have limits as well as uses

One way to reduce reliance on reasoning text is to compare actual actions or final outcomes. Did the system complete its task using only permitted processes? Did it attempt an action with external effects? Evidence closer to the actions taken can help distinguish a mere change in wording. Yet a limited observation environment or incomplete logs cannot reveal behaviour outside their scope.

A correct outcome alone does not establish that the steps taken were permitted. Conversely, avoiding prohibited actions offers less business value if the assigned work is never completed. A buyer needs to connect successful task completion with adherence to permitted means. Improving only one leaves a different burden for the use decision.

Comparison remains useful without a perfect detector. Show which questions different records can answer, without casually using one result to fill a gap in another. Identifying the business decisions affected by what remains unobservable turns evaluation limits into choices about the scope of use.

The same detection score can imply a different workload

Suppose an evaluation includes many situations likely to produce problems. That can help locate weaknesses, but how often those situations occur in daily work is a separate question. Conversely, few warnings across ordinary tasks do not establish what happens in difficult cases. If opportunities for problems differ before and after a change, the difference in counts requires careful interpretation.

A model adjustment might also help only in particular situations. The same finding has different value to a buyer who encounters those situations frequently and one who rarely does. Discarding a useful narrow improvement in favour of a broad average, or extending a narrow improvement to the whole business, can both misallocate spending.

Our introduction to matching the subject and period of a numerical comparison offers a starting point for avoiding this mistake. For a model intervention, readers also need to know which tasks were used to examine the relationship between behaviour and observation. The same number of tests need not answer the same business questions.

Tests used to make a change and tests used to assess it

Suppose a failure is discovered and a model adjusted to address it. Repeating the original test has value: it checks whether the initial problem was addressed. Passing that test more often does not establish that the improvement extends to similar work under different conditions. Separating reproduction of the original failure from investigation of the change’s reach makes an evaluation plan more concrete.

Combining the two in one metric can make a response to a known failure look like strength in new situations. Buyers need to know whether an improvement reaches the situations relevant to their own work. What developers can readily demonstrate as success may cover a different scope from the success users require.

The problem of repeatedly using a previously untouched test period raises a related comparison issue in finance. Financial strategy tests and AI behaviour evaluations have different subjects and failures, so the procedures cannot simply be transferred. The shared point is the need to explain the distinct roles of information used to make an improvement and information used to assess it.

Retesting everything has an opportunity cost

Starting every evaluation again after every change can appear cautious. But discarding evidence that remains useful consumes people and computing capacity that could investigate new situations. Carrying earlier scores forward too broadly can instead leave new weaknesses untested. Which cost matters more depends on the scope of the change and the significance of the decision the evaluation supports.

Continuing an existing limited use and giving a system new tools for more autonomous action require different evidence. What remains adequate for the former need not suffice for the latter. Defining the evaluation around what work is being delegated under which conditions, rather than around a model name alone, helps identify what can be reused and what needs to be added.

Our analysis of LNG modifications using existing designs also examines the work of connecting earlier evidence to current conditions. Additional evaluation therefore concerns both whether an earlier account still applies and what information can support an account of the changed model.

Costs depend on how far an explanation of a change travels

Imagine a developer makes one change to a model used by several businesses. If each business investigates its effects from scratch, the same comparison may be repeated many times. If the developer identifies what changed and provides the comparison conditions, shared questions can potentially be examined once, leaving users to concentrate additional evaluation on questions specific to their work.

Shared documentation alone does not ensure lower investigation costs. If claims about behaviour are expressed only through a monitor’s scores, users still have to establish what happened to the actions themselves. Recipients need to be able to connect effects on the text they see with improvements in the operations they delegate. Missing information shifts work from the party explaining a change to the party receiving it.

Reducing evaluation costs does not always mean running fewer tests. Linking the reason for a change, the comparison conditions and the scope of the result can spare recipients repeated inference. Where that information is usable, external evaluators can spend more time on unanswered questions in new uses and less on deciphering a shared change.

Buying a shared evaluation still leaves a business decision

Making the purchased unit explicit helps compare shared evaluations. Evidence differs between an assessment of a model alone, one of the model paired with a specified monitor and one covering the tools used by the buyer. Even equally priced quotations can leave different amounts of verification for the buyer if they cover different subjects.

Consider a company using the specified monitor in pre-adoption tests but its existing internal service in production. Compared with a buyer deploying the tested pair unchanged, it retains questions about a different combination. This does not imply that monitors should never be changed. It distinguishes the savings from retaining existing infrastructure from the cost of examining another combination, so both enter the same estimate.

Also consider where shared documentation needs to go within the buyer’s organisation. A testing department holding the results cannot begin the relevant comparison if it does not know that the business team has added delegated actions. Delivery of a purchased report and its application to operations are separate jobs. Connecting the evaluated subject to the system actually used remains work after purchase.

A cheap scorecard and evidence useful for the next decision

Price and delivery time are convenient ways to compare evaluation services. But if nobody knows which parts remain usable after a model revision, a cheap first assessment may be followed by extensive retesting. Preserving the connection between results and their conditions may instead narrow the scope of later review. Costs arise across successive changes, as well as in the first scorecard.

One could describe this as carrying an evaluation asset forward. Here, “asset” means knowledge and a basis for comparison that can inform future decisions; it is not an accounting classification. The value lies less in storing large quantities of old results than in understanding which changes would invalidate which claims supported by those results.

Our explanation of the difference between profit and cash movements helps connect evaluation spending with the timing of business returns. For a company paying evaluation costs before earning revenue from expanded use, a change in the scope of retesting can affect funding needs too. A technical pass and the receipt of revenue cannot be treated as the same moment.

Connect the evaluation to the purchased combination at handover

Receiving a product name alongside an evaluation report does not complete verification if the actual combination in use remains unclear. Suppose two contracts use the same model: one includes a specified monitoring service, while the other leaves monitoring to the buyer. A report naming the same model does not make the first pair’s result a score for the second combination.

At handover, the buyer needs an explanation connecting findings to the purchased subject, rather than disclosure of every internal modification. Matching the evaluated model, monitor and tools to the scope supplied makes similarities and differences concrete. A large set of detailed identifiers does not shorten verification if that correspondence remains unreadable.

This distinction also matters inside the buyer’s organisation. When the IT team confirms installation and the business team starts using new actions, establish whether they are relying on the same evaluation evidence. Separating completion of the purchasing process from evidence supporting an added use helps focus later investigation. That is the purpose of connecting post-change comparison to identification of the purchased combination.

Shared evaluation has another value for smaller users

For a business unable to run many comparative tests itself, a shared evaluation can provide an entry point to adoption. It can also leave several businesses dependent on the same omission. Knowing what different information sources establish allows the savings from shared work and shared weaknesses to be discussed separately. Increasing the number of providers alone does not achieve that.

Suppose documentation assessing a model change relies only on one kind of reasoning text. Several businesses using that scorecard does not create more observation routes. Asking whether another business record needs examination, and who could perform that examination collectively, may avoid simply adding the full burden to each user.

Shared evaluation should still preserve differences in use. A small company delegating a limited task may need evidence suited to that narrow scope. A large company connecting many tools may need a different comparison. Defining the actions that remain explainable after a change makes costs more concrete than deciding evaluation needs from company size alone.

The route to markets runs through evaluation spending and adoption

The issue reaches international AI markets through purchases of evaluation services and the expansion of use conditional on their results. If revisions require more comparative work, the location of spending on people and computing can shift between developers, external evaluators and adopting businesses. More evidence that can legitimately be carried forward may instead reduce repeated work across companies.

Higher evaluation-service revenue is not equivalent to higher profit across the AI industry. Additional spending can generate benefits if it supports useful adoption. If it merely repeats checks because earlier documentation cannot be connected to current conditions, it may amount to a transfer of costs. Following the market requires identifying which changes affect which use decisions, alongside counting evaluations.

As our explanation of workload, productivity and profits discusses, a rise or fall in verification work alone does not determine the economic effect. The scope of delegated work, remaining human review and continued use after a change must be considered together. This guidance alone cannot determine semiconductor demand or the direction of an individual company’s share price.

The strongest objection: broad retesting for a narrow change

It would be too strong to assume that every adjustment substantially changes the relationship between behaviour and observation. Where a change is limited and the information and uses supporting comparison remain stable, carrying earlier results forward broadly may be reasonable. Demanding a complete reassessment for a small change could make the retesting burden exceed the benefit of addressing a known problem.

Answering this objection requires conditions under which evidence remains useful, rather than general warnings about danger. If information used by earlier monitoring survives, other action records also support improvement and results agree across important uses, the case for extensive retesting weakens. Even for the same change, the required depth of comparison depends on the scope of the decision.

If the model adjustment, monitoring method and available tools all change together, earlier results become harder to apply. The number of changes is not itself proof of danger. The question is what remains between the question a score answers and the purchasing decision being made. Additional testing should be explained as work that addresses that gap.

Where should scarce evaluation resources go?

Evaluation teams and computing resources are finite. Equal amounts of work on checking known fixes, exploring new uses and testing for weaker monitoring are not necessarily appropriate. Their allocation depends on the significance of the tasks and the evidence surviving a change. A higher total test count can leave the resources devoted to important unanswered questions unchanged.

Rare actions with large consequences are particularly hard to assess from average performance on ordinary work. But results from tests concentrated on those cases cannot be translated directly into interruption rates or review costs in daily operations either. Distinguishing tests that search for weaknesses from tests that establish everyday conditions of use makes it clearer what a budget has actually established.

Focusing an investigation does not mean removing difficult tests to hide failures. Changing its scope changes the decisions its results can support. A constrained evaluation especially needs to distinguish answered questions from those left open. That distinction also informs the choice between expanding automation into weakly supported areas and keeping use narrow.

SG Group’s view: buy an explanation that connects before and after

Where an intervention affects both behaviour and observation, SG Group sees evaluation as an explanation connecting a model before and after the change. The choice is not simply to discard all earlier scores or to use them unchanged as guarantees. Connecting the information that remains, the relationships that change and the decisions that consequently need review allows evaluation costs to be matched to the scope of use.

Under this view, arranging contractual authority to stop and resume work is insufficient on its own. Even if someone has authority to reopen a case, repeating the old investigation may not answer the question if the model change removed necessary signals. That distinguishes an intervention in evidence generation from the issue examined in our earlier article on alert closure and reopening cases.

What a user buys is evidence capable of answering relevant questions, rather than a high number of fixes or a low number of warnings. Developers explain improvements, evaluators show comparison conditions and buyers relate them to their work. Preserving that connection turns changes and additional checks into a way of defining useful deployment, instead of a reason to suspend it indefinitely.

Three developments imply different next steps

In the first, evidence of improved behaviour agrees with evidence that observation remains effective. Businesses may then preserve the parts of earlier evaluations that remain useful and add comparisons for new uses. The question is whether use can expand while verification costs remain contained. Delegating actions beyond the evaluated use still raises new questions.

In the second, warnings decline but different records do not agree on why. The next task is to identify which conditions produce the disagreement, rather than reject purchases indiscriminately. Some businesses may continue limited use while adding comparisons; others may hold back actions that require the missing evidence. The choice depends on the importance of the work and where the uncertainty lies.

In the third, behaviour improves but recipients do not receive an explanation of the comparison, prompting duplicated retesting. Technical improvement then fails to translate directly into inexpensive adoption. Our G7 article on information sharing among financial authorities helps explain conditions under which information reaches decisions. It does not substitute for testing the effects of a model intervention.

What can serve as the reference for a correct judgment?

Adding another detector to investigate differences does not make that detector infallible. Agreement between several judgments can also conceal omissions when they share weaknesses in their information. The absence of a perfect reference need not make comparison futile. Stating which records served as references and which disagreements were investigated lets readers assess the scope of the evidence obtained.

Whether findings during training extend to all uses after release is another question. A change in the mix of tasks, connected tools or user instructions calls for reconsidering whether previously examined relationships still hold. This sets the scope of an evaluation result; it does not predict a future failure.

Nor can a single joint guidance document establish a particular company’s additional validation bill or the share of monitoring changes passed through into prices. The useful evidence concerns explanations of changes, comparisons by use, conditions of use and the division of evaluation work. A technical concern cannot simply be converted into a loss estimate for the whole market.

What evidence would change the case for retesting?

The case for carrying earlier evaluations forward more broadly strengthens when several signals continue to support the same improvement and comparison conditions remain intact for important uses. If conclusions remain stable as new tests are added, without increased missed problems or false alarms on legitimate work, focused testing may become more reasonable than extensive retesting.

Conversely, if results based on reasoning text move in a different direction from results based on action records, there is a reason to investigate which relationship changed. If improvement appears only when the model and monitor are changed together, buyers should distinguish the conditions of using that pair from those of moving it elsewhere. The purpose is to establish where a good result applies, rather than dismiss it.

The next useful documentation would explain which changes preserved which evidence, beyond adding abstract safety declarations. Linking earlier records to new comparisons helps businesses connect the benefit of improvement and the cost of verification within the same use decision. As evidence changes, the account should support reasons both to increase additional testing and to reduce it.

Final assessment: retain the basis for decisions after making a fix

An AI adjustment aims to remove a problem. When it also affects how later behaviour can be observed, carrying earlier evaluation forward becomes a separate task. Connecting behaviour to observation and specifying which evidence survives for which use gives businesses a firmer basis for decisions than expanding use simply because warnings decline.

Evaluation spending buys more than additional scorecards. Where the account before and after a change remains connected, businesses may preserve useful evidence, direct resources to necessary comparisons and make decisions about new uses. Where that connection breaks, even an improvement can increase verification costs or encourage use on weak evidence. The route from this guidance to markets and business therefore concerns both the speed of a fix and the evidence available for decisions afterwards.