Start Agent Evaluation Run
Start an agent evaluation run to queue test conversations against a saved draft or published version.
Start Agent Evaluation Run
Save tests with Create Agent Evaluation Test. Every selected test must belong to this agent, and at least one must be enabled. The selected target must have a saved version. Only one queued or running run may be active for an agent.Minimal request
Send non-emptytest_ids, repeat from 1 to 20, and target set to draft or published. draft selects the newest saved version. prompt_override applies only to draft runs.
k7x2m9q4wz1abc with an actual saved test ID.
Evaluation run settings and side effects
modedefaults totext, which skips speech recognition and synthesis.voiceruns each conversation as a browser-style test call; no phone number is dialled. Evaluation test calls are not charged to the workspace wallet.llmdefaults toprimary, keeping the version configuration and its fallback.secondaryuses the enabled fallback model alone.bothqueues each test against both models and doubles the conversation count for the same repeat setting. Secondary and both require an enabled fallback model.- Disabled tests are excluded. The returned
totalis the number of queued conversations after model selection and repeats. - Simulations can cover the primary and secondary languages in one conversation.
primary_language_onlyin the saved test configuration limits a simulation to the primary language. - Tools default to mocked execution: every function tool is mocked, and a tool with no mock response returns an error instead of running.
mock_strategy: noneruns all tools for real,selectedruns the tools not selected for mocking for real, andmock_fallback: realruns tools that have no mock response. Real execution performs the external action, so reviewmock_strategy,mock_fallback, andmocksbefore running. triggerdefaults tomanual.pre_publishlabels the run; it does not publish the agent or impose a publish gate.
Verify the result
A201 response contains the queued run in data. Store data.id and poll Get Agent Evaluation Run until it reaches a terminal status. Check failed, errored, and conversation outcomes before treating it as a pass.
Errors and recovery
400 means a selected test is missing or belongs to another agent, every selected test is disabled, the target has no version, or secondary-model evaluation is unavailable. 404 means the agent is unavailable. 409 means an active run already exists: poll that run instead. A lost response may still have queued work. List runs before retrying; another successful request creates another run and repeats its test work and any configured real tool actions.
Missing, expired, or invalid API credentials return 401. See API authentication.
Related endpoints
- List Agent Evaluation Tests
- Create Agent Evaluation Test
- Get Agent Evaluation Run
- Get Agent Evaluation Conversation
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Path Parameters
Agent ID from List Agents.
Body
Saved test IDs belonging to this agent. Disabled tests are excluded.
11 <= x <= 20draft selects the newest saved version; published selects the published version.
draft, published Optional editor prompt for draft runs. Ignored for a published target.
Run label. pre_publish does not publish the agent or block publishing.
manual, pre_publish Text skips speech recognition and synthesis. Voice uses the audio test-call path.
text, voice Primary keeps the version configuration, including fallback. Secondary uses the enabled fallback model alone. Both queues each test for each model; requires an enabled fallback model. Other settings stay the same.
primary, secondary, both