Why this matters now
If you've built a Copilot skill, a custom agent or an MCP tool, you'll know the problem. You tweak a prompt or a tool description, try it once, it looks fine, and you ship it. A week later someone finds it now skips a step it used to handle. Language models don't give the same answer every time, so one good run tells you very little.
On 17 November 2026, GitHub opened a public preview of GitHub Copilot Evals to deal with this. It's for anyone on Copilot Individual, Copilot Business or Copilot Enterprise. GitHub calls an eval "a repeatable test for AI behavior": you give an agent a realistic task, write down what a good result looks like, and check the final output and the steps the agent took along the way. After each change to your customisation you run the same eval again and compare the results.
Where it runs and how you get it
Copilot Evals ships as a plugin and a VS Code extension, with the source in the github/evals repository. You can use it in four places:
- Copilot CLI: add the marketplace with
copilot plugin marketplace add github/evals, then install withcopilot plugin install eval@github-copilot-evals. The/evalslash commands and the visual Canvas only appear once you've switched on experimental mode (/settings experimental on). Without it, you can still ask for the same things in plain language. - VS Code: install the
GitHub.copilot-evalsextension from the Visual Studio Marketplace. It adds a sidebar, the Canvas and native tools. - VS Code Agents: add
github/evalsas a marketplace under Customizations, install the eval plugin and start a fresh session. - GitHub Copilot app: open Customize, add the same marketplace and install from there.
The minimum versions are fairly recent. The CLI plugin wants Node.js 24.0.0 (the VS Code route accepts 22.14.0), plus Git 2.28.0 or later, Copilot CLI 1.0.85 or later and the @microsoft/vally-cli package at 0.19.0 or above. You'll also need to be signed in to Copilot and to mark the project folder as trusted. The first tagged build, plugin 0.24.53, went out on 17 November as a pre-release. For offline installs, its release archives include a local marketplace descriptor.
What's in the box
The plugin is built around six skills: create, review, improve, run, compare and setup. Each covers one stage of the testing loop:
- Create drafts an eval for a chosen skill, agent or tool. GitHub's walkthrough produces a YAML file (
pr-description.eval.yaml) with test cases and the checks each case must pass. The tool also suggests cases worth covering, such as deferred work being passed off as shipped, missing verification, or instructions hidden inside change notes. - Run executes saved evals. You approve each run before it starts, and the approval prompt shows how many evals, cases, trials and total executions you're about to spend.
- Review reads a run report and helps you work out why a check failed.
- Improve revises an eval and keeps the original.
- Compare puts a new result next to an earlier one so you can see whether your change helped.
- Setup picks eval folders, groups them into suites and wires up CI checks.
There's also a bundled MCP server called copilot-evals with 15 tools, three automation blueprints for VS Code Agents (a health review, a smoke run and a skill comparison), and the Evals Canvas, a side panel where you can browse project assets, edit evals as an outline or as raw YAML, and read saved reports.
The engine underneath
GitHub didn't build the scoring engine itself. Copilot Evals runs on Vally, an eval platform from Microsoft that GitHub describes as an extensible local tool. Each Vally run follows one pipeline. A stimulus (a prompt or task) kicks it off, an executor drives the agent or tool under test, and a trajectory records every turn, tool call and token. Graders then score that trajectory, and the final score covers pass or fail, quality, speed and cost.
That trajectory is the useful bit for agent builders. You can check whether the agent called the right tool, in the right order, without touching files it shouldn't. A final answer on its own can't show you that. Vally ships built-in graders and lets you write your own, and the Copilot plugin adds AI judges on top. Vally's setup guide says to authenticate with a fine-grained personal access token that has the Copilot Requests permission, and warns against putting that token in eval.yaml or committing it.
Costs
Writing an eval, validating it, reading saved results and comparing old runs don't call a model, so they cost nothing. Running an eval and testing graders do call models, and GitHub says those calls can use tokens or credits on your normal Copilot allowance. The documentation says costs are uncapped. The approval screen shows execution counts and cost estimates before each run, but nothing stops a large suite with many trials from eating through credits. Keep trial counts low until you know what a typical run costs.
Limits and risks
We'd treat this as a power tool that needs care:
- No sandbox. The docs say runs have no operating system or network sandbox. Tools can change files and reach the network, and your credentials are passed down to child processes. Setup scripts, custom graders and AI judges all run code with your local permissions.
- Cancelling doesn't roll back. Stopping a run won't undo setup steps or anything already written to remote systems.
- Results can leak. Reports are saved under
.copilot/evals/resultsin your project and may contain sensitive material. Check them before you share or commit them. - Passing isn't proof. GitHub itself says a passing check doesn't guarantee the agent will behave correctly every time.
- Telemetry is on by default in the standalone plugin. It collects usage events, approval decisions, outcomes, durations and pseudonymous device and session IDs. GitHub says it doesn't collect source code, prompts, model outputs, paths or repository names. You can turn it off with
COPILOT_EVALS_TELEMETRY=off,DO_NOT_TRACK=1orOTEL_SDK_DISABLED=true. The VS Code extension follows your editor's telemetry setting. - The licence isn't open source. The code is published under the GitHub Limited Use Source License. You can read it, contribute and use it internally, but you can't resell it or offer it as a service.
- It's a preview. Commands, file formats and behaviour may change.
What builders can do with it
For a team running a few in-house Copilot customisations, here's how we'd pilot it:
- Pick one skill that matters and has broken before. Write three to five cases that capture the behaviour you rely on, including at least one tricky input, such as an instruction smuggled into the data.
- Run it on a throwaway branch or in a disposable container, with credentials scoped to what the test needs. Without a sandbox, isolation is your job.
- Save a baseline, change the skill, rerun with a couple of trials and use compare to see what moved.
- Once results look stable, use the setup skill to add the suite to CI, and gitignore the results folder unless you've reviewed it.
- Decide on telemetry before anyone installs it, and check the licence with whoever handles your software approvals.
The main benefit is a habit: agent customisations get a test suite like the rest of your code. It's early and it has sharp edges, but for teams already shipping MCP tools and custom agents, it gives you a structured way to catch regressions before your users do.
Sources
- GitHub Copilot Evals is now in public preview (GitHub Changelog)
- github/evals repository
- GitHub Copilot Evals documentation
- GitHub Copilot Evals (Preview) Plugin 0.24.53 release
- Vally
- Vally runtime setup
Build with people working on the same things
Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.