Original announcement: GitHub · 2026-11-17. Read the official announcement.
Summary of the GitHub Copilot Evals Public Preview Launch
On November 17, 2026, GitHub announced the public preview release of GitHub Copilot Evals, a feature designed to help developers optimize and verify AI behavior within their GitHub Copilot setups. This tool allows users to create customizable, repeatable tests (evals) to assess how well Copilot performs on specific tasks, providing insights to improve skills, agents, and managed code processing (MCP) tools integrated with Copilot.
What Is GitHub Copilot Evals and Who Should Care?
GitHub Copilot Evals is essentially a testing framework for AI interactions in your development workflow. It:
- Enables developers to define realistic programming tasks as tests.
- Describes criteria for what constitutes a successful output.
- Evaluates both the outcome and the reasoning process of the AI.
- Allows re-running tests after changes to detect regressions or improvements.
This new preview is relevant for users of GitHub Copilot across all individual, business, and enterprise plans, particularly those who customize their Copilot environment with specific skills, agents, or tools and want to verify the continued effectiveness of these components.
Key Features and Workflow of Copilot Evals
- Eval Creation: Developers can create evals targeting specific skills, custom agents, or MCP tools.
- Run and Review: Each eval run requires user approval, then displays pass/fail statuses for individual checks.
- Result Analysis: Access detailed information on failures to inform adjustments within the Copilot setup.
- Comparison: Compare evaluation results before and after a customization to measure improvement.
- Evaluation Engine: Powered by Vally, an extensible local tool for eval execution.
Getting Started and Next Steps for Developers
GitHub Copilot Evals is integrated with popular Copilot interfaces, making it accessible within existing development environments:
- Visual Studio Code: Install the GitHub Copilot Evals extension from Visual Studio Marketplace.
- Copilot CLI: Add the GitHub Copilot Evals plugin using the provided installation instructions.
- GitHub Copilot App: Customize your setup by adding the
github/evalsmarketplace and installing the plugin.
After installation, developers should:
- Create an eval reflecting a typical task or workflow relevant to their projects.
- Run the eval to establish a performance baseline.
- Modify Copilot skills, agents, or tools as needed.
- Re-run evals to validate that changes are beneficial and do not introduce regressions.
- Use eval results to iteratively improve AI assisted coding experience.
Practical Checklist for Developers
- Confirm you use GitHub Copilot with an eligible plan (Individual, Business, or Enterprise).
- Install the Copilot Evals plugin/extension appropriate to your environment.
- Create repeatable evals that represent your regular coding tasks or AI interactions.
- Run and approve eval runs, review results carefully.
- Use findings to fine-tune your Copilot skills or agents.
- Regularly re-evaluate after any customization to catch regressions early.
Note: The announcement does not specify detailed limits or pricing changes during the public preview, so check official GitHub documentation for updates.


