Show HN: Coder Eval – A Framework for Evals
Test that your Claude Code skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, activation checks, A/B experiments, CI gates. - UiPath/coder_eval
Summary not yet generated for this article.
Original Text
We needed a framework to write our evals in and easily update them, run A/B tests, set all kinds of constraints (ex: timeouts, number of turns), and configure the execution environment (ex: sandboxes, dependencies, how to handle AskQuestion). We also have an agent judge.
It’s all in a YAML file now.
We seem to be moving toward a world where companies rely more and more on evals to decide what to ship, so this should be super helpful for normalizing evals across teams.
Comments (1 of ) ⌛