Five Decisions Your AI Pilot Needs Before Procurement Review

A year ago, a buyer's security team opened a vendor file, confirmed a SOC 2 Type II was in the folder, skimmed the model card, and moved on. The AI section of the questionnaire ran two or three questions long, and most of them were about data retention. That's over.

The same reviewers are now opening the file with a separate AI addendum. They want to know who sees the prompts, how the system detects an injection attempt, what happens when a human disagrees with an agent's action, and what the vendor will do when the base model provider pushes an update nobody downstream asked for.

The pilot still demos well. The paperwork is where it stalls. Teams that ship through procurement in a reasonable window have stopped waiting to be asked and started preparing an AI fact sheet of their own. For a walk-through of the specific questions a SOC 2 doesn't cover, see The Digital Reader on AI procurement questionnaires, a useful mirror for anyone staring at question seven of a 140-question list.

Decide Who Owns the Version You're Running

The first decision is whether you or the provider controls the model sitting under your product. If you call a hosted API, the provider does, and that includes the right to deprecate, swap, or retune the weights on a timeline that isn't yours.

The written policies look reassuring until you read them closely. OpenAI's deprecation page commits to at least six months of notice for generally available models, three months for specialized variants, and as little as two weeks for preview models. Anthropic's model deprecations page commits to at least sixty days for publicly released models.

Those windows set the floor for your migration plan, not the ceiling. The trade-off is real. Hosted APIs give you capability you can't build yourself, along with a dependency whose retirement schedule you don't set. Running an open-weight model on infrastructure you control flips that around: slower to adopt the newest frontier capability, but you decide when the version changes.

Decide What Counts as a Material Change

The second decision is contractual. The harder case is a provider update that keeps the model name the same while shifting behavior underneath, because your SLA probably doesn't define that as a change at all. Buyers' procurement teams are starting to write their own definition into the agreement, and your pilot needs an answer ready.

Decide How You'll Detect Drift Before a Customer Does

A model under the same name can behave meaningfully differently from one month to the next. Whether that specific task matters to your product or not, the pattern does: silent updates break assumptions you baked into prompts and evals months ago.

The decision here is how often you re-run a regression suite against the live endpoint, and what you do when the numbers move. A weekly eval against a frozen test set is cheap. Catching a regression a week after it ships costs far less than catching it from a support ticket. Keep the suite small enough that it actually runs, and specific enough that a drop points at a real workflow.

Decide Where a Human Stands in the Loop

Buyers are no longer satisfied by a yes-or-no on human review. They want to know which actions require sign-off, which get logged for after-the-fact review, and which the agent executes on its own. The decision isn't whether to have humans in the loop; it's where to put them so the system stays useful and the audit trail stays honest.

Pair that with a plain answer on prompt-injection detection. Say what you scan for, what you block, and what you log when a suspicious input gets through. Model upgrades can shift the picture, because a newer model may follow embedded instructions more literally than the one it replaced. Fold injection resistance into the regression suite rather than testing it once at launch.

Prepare the Fact Sheet Before Procurement Asks for It

The last decision is when to write all of this down. The teams moving fastest through security review are the ones who stopped waiting for the questionnaire and started shipping a short AI fact sheet alongside the SOC 2. It isn't a model card. It's your answer to the questions a model card doesn't touch: prompt retention and redaction, where inference runs, what logs the buyer's SIEM can pull, what the human review surface looks like, how injection attempts get detected, and what happens the day the base model provider announces a retirement.

Write it before you need it. The fact sheet shortens the questionnaire because reviewers can cross off two-thirds of the AI section without a meeting, and it forces your team to name the things you haven't decided yet. That's the useful part.

The pilot that can't answer question seven today is the one that will answer it in the contract, under terms the buyer wrote.