How to Test an AI API Relay for Model Swaps, Truncation, and Silent Feature Loss (2026)
A model feeling worse does not prove that a relay swapped it. The same symptoms can come from a wrong model ID, truncated context, rewritten parameters, protocol conversion, client defaults, or an upstream incident. This guide turns those possibilities into six repeatable tests.
1. Define the failure before testing it
People use degradation to describe several unrelated problems. A true route change means the request reached a different model or deployment. Context truncation means the intended input never reached the model in full. Parameter rewriting changes limits, temperature, reasoning effort, or tool behavior. Protocol loss occurs when a gateway translates between formats and cannot represent every source field. Upstream capacity or safety changes can alter behavior even when the model name stays the same. Client bugs create the same symptoms by sending a different request than the UI suggests. Write down the observed symptom first: shorter output, forgotten middle context, missing tools, different token counts, a changed response model, or higher error rate. A precise symptom determines the probe.
2. Build a baseline that can be reproduced
A useful baseline contains more than the answer text. Save the endpoint, model ID, timestamp, SDK and client versions, complete redacted request body, HTTP status, response model, provider request ID when available, finish reason, usage object, time to first token, and total latency. Keep the prompt deterministic and remove timestamps or generated IDs unless they are part of the test. Use a low-value test key and synthetic content rather than customer data. Run the same test several times while the route is healthy and retain the distribution, not just the best response. Without this reference, a later result can look different simply because the task, context, client, or model settings changed.
3. Probe one: inspect the transport and response envelope
Start with a short non-streaming request and preserve the raw response headers and JSON. Confirm that the requested model ID is valid in the provider's current model list, the returned model field is compatible with that request, and the endpoint is the intended one. Capture a provider request ID when the service exposes one; otherwise record your own client trace ID and note that limitation. Repeat through every route you actually use: an OpenAI-compatible endpoint, an Anthropic-compatible endpoint, or a local router can behave differently. This catches wrong aliases, accidental fallbacks, stale configuration, and obvious response rewriting. It is only a first filter. A model field is metadata supplied by the service and can be normalized or rewritten, so it is not cryptographic proof of the underlying model.
4. Probe two: audit usage without over-interpreting it
Send the exact same serialized request through the same endpoint and model several times. Ordinary input-token counts should establish a stable baseline, subject to the endpoint's documented usage semantics. A change can indicate that the client added messages, a router changed templates, a model tokenizer changed, or the provider altered accounting. It does not by itself prove deliberate prompt injection. Compare the complete usage object rather than one total: prompt or input tokens, cached input, cache creation or writes, output tokens, and reasoning tokens where available. Also verify whether streaming requires an explicit include-usage option. If two clients disagree, capture what each client actually sent before blaming the relay.
5. Probe three: test context integrity at several positions
Create a synthetic long document and insert unrelated random facts at roughly 20%, 45%, and 75% of the input. Ask for all three values at the end. Use facts that cannot be guessed, such as randomly generated labels and numbers. Repeat the test, then rerun with a much shorter document. If the short control succeeds while the same middle positions repeatedly fail in the long version, investigate client contextLength settings, gateway limits, tokenizer estimates, and server-side truncation. Do not treat one wrong answer as proof: long-context retrieval is not perfect even when every token arrives. Stronger evidence combines repeated positional failures with usage totals that are materially below the expected input size.
6. Probe four: exercise the complete tool-call round trip
Test tools in layers. First define one function with two required scalar fields and verify that the response contains the expected tool name and valid JSON arguments. Next request two independent tools in the same turn and observe whether the route preserves multiple calls when the selected model and endpoint support them. Then use a nested schema with required properties, arrays, and enums. Finally return the tool result using the exact call ID and role required by the protocol, and verify that the model continues rather than repeating the call. Record both directions. Many compatibility failures appear only when tool results travel back to the model, not when the initial tool call is generated.
7. Probe five: verify caching with creation followed by reads
Do not compare input_tokens between a request with cache_control and one without it. On Claude-style usage, input_tokens excludes tokens reported as cache creation or cache reads, so a change can be the expected accounting split rather than evidence that a field survived or disappeared. The valid test uses a stable prefix that meets the selected model and platform's minimum cacheable length. Send the requests sequentially. On the first eligible request, inspect cache_creation_input_tokens. On later requests within the applicable lifetime, inspect cache_read_input_tokens. Both staying zero can mean the prefix was too short, the route or model did not support caching, the cache boundary was lost, or the serialized prefix changed. Reproduce before assigning a cause.
import json, os, urllib.request
URL = os.environ["ANTHROPIC_MESSAGES_URL"]
KEY = os.environ["TEST_API_KEY"]
MODEL = os.environ["TEST_MODEL_ID"]
STABLE = open("stable-prefix.txt", "r", encoding="utf-8").read()
payload = {
"model": MODEL,
"max_tokens": 32,
"system": [{
"type": "text",
"text": STABLE,
"cache_control": {"type": "ephemeral"}
}],
"messages": [{"role": "user", "content": "Reply with only: verified"}]
}
for attempt in range(1, 4):
request = urllib.request.Request(
URL,
data=json.dumps(payload).encode(),
headers={
"x-api-key": KEY,
"anthropic-version": "2023-06-01",
"content-type": "application/json"
}
)
with urllib.request.urlopen(request, timeout=120) as response:
usage = json.load(response).get("usage", {})
print(attempt, {
"input": usage.get("input_tokens"),
"cache_create": usage.get("cache_creation_input_tokens"),
"cache_read": usage.get("cache_read_input_tokens"),
"output": usage.get("output_tokens")
})8. Probe six, controls, and how to report a finding
Finish with a repeated-run baseline using tasks that have externally checkable answers or strict output structure. Temperature zero can reduce variation but does not make all inference deterministic, so compare rates and distributions rather than demanding byte-identical prose. When a probe fails, repeat it at another time, reduce the request to the smallest reproduction, and run a documented control route if available. Report the exact endpoint, model, Beijing-time timestamp, provider request IDs when available, redacted request hash, and raw usage fields. Ask the provider which layer handled the request before alleging a model swap. The same standard applies to cocodot: its OpenAI-compatible and Anthropic-compatible routes can be tested with this suite, and probe.cocodot.co can help organize repeatable checks. Use a disposable limited-value key for any third-party diagnostic.
Six probes and what each one can actually establish
| Probe | Pass signal | What a failure suggests | What it does not prove |
|---|---|---|---|
| Response envelope | Requested model, response model, and endpoint are consistent; capture a request ID when exposed | Wrong alias, route, or response rewriting | The model field alone cannot prove model identity |
| Usage repeatability | Identical serialized requests produce a stable input-token baseline | Hidden request changes, client differences, or accounting changes | A malicious prompt injection is not the only explanation |
| Context integrity | Random facts at several positions are retrieved across repeated runs | Client or gateway truncation, or weak long-context retrieval | One missed fact does not prove the context was removed |
| Tool and schema fidelity | Tool names, arguments, IDs, enums, and nested fields survive round trips | Protocol conversion or client compatibility loss | A stochastic tool-choice miss is not automatically field loss |
| Prompt-cache preservation | First eligible call reports cache creation; later sequential calls report cache reads | Cache fields were ignored, prefix changed, route is unsupported, or threshold was not met | Comparing input_tokens with and without cache_control is not a valid test |
| Repeated baseline | Accuracy and structure remain within the saved reference distribution | A reproducible behavioral or infrastructure change | Temperature zero does not guarantee identical output |