Agent quality

Cypress's own team app, Grove, runs an agent, Cedar. To know how well Cedar answers, and to make it better, Grove does two things and Cypress closes the loop:

1. Grove sends each benchmark run. Ten to twenty real questions per model setting, each answered by Cedar and graded by a judge model (faithful to what it read, correct, useful), with the time each answer took.
2. Grove sends the answers it flagged. A live check failed, a person corrected Cedar, the judge scored an answer low, or it was picked at random.
3. Cypress staff judge them on the Cedar quality page (Admin, staff only): right, wrong (with what the right answer was) or dismissed, and whether to teach that organization or add the question to the benchmark.
4. Grove reads the verdicts back for the organization and turns them into lessons for that organization and new benchmark questions.

Since 1.9.0.

Who may call

These three endpoints are for an app Cypress made (a first-party app, such as Grove), on its own Connect Cypress token, acting for a person of the organization or for Cypress staff working for it. An API key cannot hold the scope, a partner app cannot be registered for it, and the endpoints refuse an API key, a partner app, an agent's token and a CM token with 403 POLICY_DENIED, whatever they hold. A first-party token without the scope gets 403 SCOPE_MISSING.

The organization is always the token's. A benchmark usually runs in a sandbox, so it lands in the sandbox; Cypress staff see every organization's runs.

The staff reads and the verdict are Cypress staff's: a CM token, or the Cedar quality page.

Every write takes ?dryRun=true (it checks the caller and the body, and writes nothing) and an Idempotency-Key.

Send a benchmark run

curl -X POST "$BASE/api/v1/agent-quality/benchmark-runs" \
  -H "authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d @run.json

{
  "externalId": "bench-2026-10-28-flash-off",
  "agent": "grove-cedar",
  "label": "flash, no thinking",
  "model": "gemini-2.5-flash",
  "thinking": "off",
  "configVersion": "cedar-v7",
  "startedAt": "2026-10-28T10:00:00Z",
  "finishedAt": "2026-10-28T10:04:12Z",
  "passed": true,
  "gates": [
    { "name": "right", "passed": true, "value": 0.9, "threshold": 0.85 },
    { "name": "slowest tenth", "passed": true, "value": 2.6, "threshold": 4, "detail": "seconds" }
  ],
  "summary": {
    "items": 20, "medianMs": 1800, "p90Ms": 2600,
    "right": 0.9, "faithful": 0.95, "useful": 0.8, "judgment": 0.75, "thinkingTokens": null
  },
  "items": [
    {
      "questionKey": "seller-open-offers",
      "side": "seller",
      "kind": "lookup",
      "question": "How many offers are open on the Camry?",
      "answer": "Two. The best is $24,100 from Lone Star Auto Group.",
      "ms": 1650,
      "marks": { "faithful": 1, "correct": 1, "useful": 0.8, "expected": null, "speed": 1 },
      "errors": [],
      "traceId": "trace-81f2"
    }
  ]
}

A new run answers 201, one that replaced an earlier post answers 200:

{ "id": "clx…", "replaced": false }

Send flagged answers

curl -X POST "$BASE/api/v1/agent-quality/audit-items" \
  -H "authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d '{ "items": [ {
    "externalId": "answer-7f3a",
    "agent": "grove-cedar",
    "question": "How many offers are open on the Camry?",
    "answer": "Two offers, the best is $24,100.",
    "reads": [ { "label": "Open offers on the Camry", "n": 2 }, { "label": "The Camry'"'"'s floor", "detail": "Set by Dana" } ],
    "flags": [ "check:grounded", "corrected" ],
    "model": "gemini-2.5-flash",
    "answeredAt": "2026-10-28T09:00:00Z",
    "link": "https://grove.getcypress.ai/rooms/clx…"
  } ] }'

At most 100 per call. reads is what the agent read, in plain words (at most 50: a label, and optionally a detail and a count n). flags says why it is up for review, lowercase: check:concise, check:grounded, check:no-claims, corrected, judge:low, sampled, or your own. link is where the answer is in your app, http or https only.

{ "created": 1, "updated": 0, "skipped": 0 }

Keyed by agent and externalId within the organization. Sending one again only adds the flags it did not have (updated); the question, answer and reads stay as first sent. One with nothing new, or one Cypress already decided, is skipped: a decided item is never reopened or changed.

Read the verdicts back

curl "$BASE/api/v1/agent-quality/verdicts?since=2026-10-28T12:00:00.000Z&limit=100" -H "authorization: Bearer $TOKEN"

{
  "verdicts": [
    {
      "externalId": "answer-7f3a",
      "agent": "grove-cedar",
      "verdict": "WRONG",
      "note": "Three offers: one came in at 9:05.",
      "lessonForOrg": true,
      "makeTestCase": true,
      "decidedAt": "2026-10-28T14:02:11.204Z",
      "decidedBy": { "name": "Cyd Manager" }
    }
  ],
  "nextSince": "2026-10-28T14:02:11.204Z"
}

The organization's decided items after since, oldest first (every verdict when since is left out), at most limit (100 when left out, 200 at most), filtered by agent if you send one. Keep nextSince and send it as the next call's since: a page never splits verdicts that share a time, so nothing is skipped, and with nothing new nextSince is the since you sent.

A verdict is final: each item comes back once.

Cypress staff

With a CM token (cyp_cm_…), or on the Cedar quality page:

• GET /api/v1/agent-quality/benchmark-runs?agent=&label=&orgId=&limit=: every organization's runs, newest first, each with its organization, setting, summary and gates. The organization named in Cypress-Organization narrows it when orgId is left out.
• GET /api/v1/agent-quality/benchmark-runs/{id}: one run with its questions, in the order sent.
• GET /api/v1/agent-quality/audit-items?status=&orgId=&flag=&limit=&cursor=: status is OPEN (the default, newest answer first), DECIDED (newest decision first), RIGHT, WRONG or DISMISSED. Answers items, nextCursor, counts by status and facets (the open queue by flag and by organization).
• POST /api/v1/agent-quality/audit-items/{id}/verdict with { "verdict": "WRONG", "note": "…", "lessonForOrg": true, "makeTestCase": false }: WRONG needs a note (422 INVALID without one); an item already decided answers 409 CONFLICT. Recorded in the organization's audit trail.