AI generated
Disclosure: I'm the author. Attest is an Apache-2.0 testing framework for agents (Go engine, Python and TypeScript SDKs). Posting here because most of it runs with no hosted model at all. Replay instead of inference in CI. attest capture -- python my_agent.py runs your agent behind a local recording proxy that forwards to whatever OPENAI_BASE_URL points at and records every model call. In CI, the agent's current code runs against that recording: attest replay: 4 replayed · 0 live model calls · $0.00 The CI runner never loads a model. If you change the prompt, replay misses and tells you what changed instead of guessing. Deterministic checks first. Schema, cost/token/latency limits, which tools ran, in what order, how many times per order id, and what the run did to a simulated backend. None of that needs a model. Local embeddings. Similarity checks can use a bundled ONNX all-MiniLM-L6-v2 instead of an embeddings API. Local judge. When a check really needs a judge, set ATTEST_JUDGE_BASE_URL to any OpenAI-compatible server (vLLM, Ollama's /v1, LiteLLM). There's also a beta Ollama adapter for recording traces. Small judges break JSON. A malformed judge reply is reported as "not evaluated", never as a pass, and your policy decides whether that warns or blocks. 1.1 (released November 10) is a correctness release: it fixes four ways 1.0 could report a failing assertion as passing. The demo is a synthetic support agent that refunds a customer twice while its reply stays polite and correct. make setup walkthrough runs the whole story, no API key, about two minutes after setup. Limits, honestly: capture is tested against a scripted OpenAI-compatible server and the hosted APIs, not against every local server's streaming quirks. If your server's stream trips the proxy, I'd like to see the trace. Also, most of the code was written by coding agents (Claude Code and Codex) from plans I wrote and reviewed. - Repo: https://github.com/attest-framework/attest - Web Page: https://attest-framework.github.io/ - Demo: https://github.com/attest-framework/refund-desk - Judge setup: https://github.com/attest-framework/attest/blob/main/docs/guides/judges.md