AI coding agents now choose SDKs, follow documentation, and call tools on a developer's behalf. Yet there has been no open way to check whether they do any of that well. Microsoft Scope is an open source platform that runs agents such as GitHub Copilot and Claude Code through the same coding tasks. It scores what they produce against criteria you define, and it shows you the evidence behind every score (microsoft/scope, GitHub, retrieved 2026-10-23). Microsoft Developer Relations announced Scope on October 19, 2026 and released it under the MIT License. The question it answers is the one every SDK, CLI, and MCP server team now faces: can an agent actually use what we ship?
Key Takeaways
Scope is Microsoft's open source evaluation platform for AI coding agents, announced on October 19, 2026 and released under the MIT License.
It drives existing agents, including GitHub Copilot, Claude Code, and VS Code with Copilot, through tasks you define, instead of testing one model in isolation.
Success is defined as criteria in a dependency graph and scored by an LLM-based Judge service, with full run traces kept for human review.
Inside Microsoft, the Azure Cosmos DB team used Scope to surface issues that led to 20 Agent Kit fixes and 27 documentation fixes.
Scope is self-hosted software in research preview, not a hosted service. You run it with Docker Compose locally or on Kubernetes, and you bring your own agent and model access.
Scope did not arrive alone. In the two weeks before launch, Microsoft published its definition of Agent Experience on October 6. It followed with the AX Practitioner Playbook on October 9, a step by step method for measuring how agents experience a product, according to TrendinTech's launch coverage. Scope is the software that puts that playbook into practice. DevAx (Developer Agent Experience) is Microsoft's name for the discipline behind both: understanding how agents interact with your software, then using evidence to improve the outcomes they achieve.
What Is Microsoft Scope?
Microsoft Scope is a self-service platform for measuring the agentic coding experience of your product surfaces, skills, MCP servers, and extensions, across agents, at scale. That is how its official documentation describes it, and two clarifications in that sentence matter. First, Scope is not a coding agent. It does not write your code. It drives agents that already exist and measures what they do. Second, it is not a leaderboard. The goal is not just whether the code worked. It is how the agent behaved on the way there: what it asked for, what tools it reached for, and where it got stuck. Scope also shows how that behavior changes when you swap the agent, the model, or the surrounding tools.
Microsoft Developer Relations began building Scope more than a year before the public release, to automate evaluations its own product teams were running by hand. The repository went public on GitHub on July 28, 2026 and, at the time of writing, carries a research preview label, 651 commits, 51 stars, and 26 forks, with no numbered release yet. This is an early, methodology-first release, and Microsoft presents it that way.
Aspect | Microsoft Scope |
|---|---|
Announced | October 19, 2026 |
License | MIT License |
Built by | Microsoft Developer Relations |
Agents it drives | GitHub Copilot, Claude Code, VS Code with Copilot |
Deployment | Self-hosted only. Docker Compose for a local stack, built on Kubernetes |
Status | Research preview. No numbered release yet |
How Scope Works
The documentation breaks a Scope evaluation into four steps:
Define. You write a representative task, the evaluation criteria that define success, and the agent configuration to test.
Submit. You send the run through the Portal or the CLI. The API stores the request in MongoDB, and the scheduler dispatches it to agent workers through storage queues.
Execute and evaluate. The worker runs the agent on the task and asks the Judge service to evaluate the output. Runs can include multiple feedback iterations, so an agent can respond to what the Judge found.
Inspect and compare. You review logs, workspace snapshots, and criteria results side by side, across agents, models, and configurations.
The criteria system is the heart of it. Criteria live in a directed acyclic graph, so one check can be gated on another passing. Microsoft's announcement gives the shape of it: a runtime check, for example, can depend on a successful build. It's FOSS illustrates the same idea with a smaller example: ask an agent for a Node.js server, then check for a GET /health route that returns JSON. The Judge is an LLM-based evaluation service. It scores each run against those criteria.
Three supporting systems make the results comparable instead of anecdotal. Versioned profiles bundle the worker, model, agent version, MCP servers, skills, and extensions behind a run, so you can change exactly one variable and rerun. Configurable personas vary the simulated user's perspective and communication style. And prompt features are characteristics Scope detects in a task prompt, such as "asks for an API" or "asks for TypeScript". They let you ask questions like how Claude Code does on database tasks versus Copilot, without tagging every run by hand.
Evidence is a first-class output. Scope streams activity while the agent works and retains the interactions, generated files, and workspace snapshots afterwards, with Redis relaying live logs and Blob Storage holding the larger artifacts. When a criterion fails, your team can trace the failure back through the exact steps that preceded it.
The Evidence: What Scope Found at Cosmos DB
Launch posts usually promise benefits. Microsoft's announcement instead publishes a worked example. The Azure Cosmos DB team compared bare models with its Agent Kit across end-to-end scenarios, then used the findings to fix both the kit and the documentation. The evaluations led to 20 Agent Kit issues and 27 documentation issues resolved. After one targeted Agent Kit update, hierarchical partition-key adoption rose from 47% to 100%, while OFFSET/LIMIT usage fell from 47% to 0% (Microsoft's announcement). Jay Gordon, the team's Senior Program Manager, put it this way:
"Using SCOPE, we compared bare models with our Agent Kit across eight end-to-end Azure Cosmos DB scenarios in five languages. The evaluations led to 20 Agent Kit issues and 27 documentation issues resolved. After one targeted Agent Kit update, hierarchical partition-key adoption rose from 47% to 100%, while OFFSET/LIMIT usage fell from 47% to 0%."
That restraint is the point of the tool. Microsoft is careful to bound the claim: the gains describe the Cosmos DB scenarios evaluated, rather than a general ranking of agents. Scope connects evaluation to decisions a product team can act on. Those decisions are concrete: which documentation needs clarification, where the tooling needs a fix, and whether an update actually changes agent behavior when you rerun the same scenarios.
How to Run Scope
Scope is software you deploy, not a hosted service included with the release. The quickest route is the local stack: the repository includes the application services, agent workers, evaluation service, and documentation, and you can start a local stack with Docker Compose using the quick-start guide. Local development uses MongoDB, Redis, Azurite (the Azure Storage emulator), and Lowkey Vault. For shared deployments the platform is built on Kubernetes. If Kubernetes itself is new to you, our Kubernetes explainer covers what it is and why it runs the cloud.
Self-hosting comes with the responsibilities you would expect. You choose the connected services and manage access to experiment data and credentials. Running agents and model-based evaluation requires access to the corresponding providers, so a Scope run consumes whatever GitHub Copilot, Claude Code, or model access your configuration points at. There is no Scope bill, but there is no free compute either.
What Independent Coverage Says
Independent coverage has been positive about the idea and consistent about the caveats. It's FOSS describes Scope as an MIT-licensed, self-hosted tool aimed at teams that maintain SDKs, documentation, CLIs, or MCP servers and want a repeatable way to see where coding agents get stuck. Its walkthrough of the mechanics ends with the same warning Microsoft repeats: the LLM-based Judge can make mistakes, and important conclusions still warrant human review.
TrendinTech's coverage stresses the timing. As of October 20, 2026, it notes, developers choosing between AI coding agents face more options and less certainty than ever. GitHub alone rotated six coding models in and out of Copilot within a single week. A repeatable way to compare agents on your own tasks, rather than on a vendor's benchmark, is the gap Scope is aimed at.
Who Should Use Scope, and Who Should Not
The documentation lists four situations where Scope is the right tool:
Comparing the agentic behavior of two or more coding agents on the same task.
Tracking how a single agent's behavior changes across versions or model swaps.
Evaluating the impact of MCP servers, skills, or extensions on coding outcomes. The skills an agent carries change its results, which is exactly the territory of agent skill libraries such as the one in our Scientific Agent Skills article.
Building a shared, reproducible measurement suite that your team can extend over time.
Its named audiences are product managers evaluating how agents use their software, engineers designing task prompts and criteria, researchers comparing agent trajectories, and pipelines that submit runs through the REST API. Skip Scope if you want a hosted service, a universal agent ranking, or an agent runtime to embed in your product. The documentation is explicit that it is none of those: results describe the tasks and configurations you tested, not a universal verdict.
Frequently Asked Questions
Is Microsoft Scope free? The software is free and open source under the MIT License. Your costs are the infrastructure you run it on and the agent and model access your evaluations consume, since running agents and model-based evaluation requires access to the corresponding providers.
Is Scope itself a coding agent? No. Scope drives existing coding agents, including GitHub Copilot, Claude Code, and VS Code with Copilot, and evaluates what they produce. It is a measurement platform, not an agent runtime and not a hosted IDE.
How does Scope decide whether an agent passed? You define the criteria. Criteria form a dependency graph, so a check can require a prerequisite to pass first, and an LLM-based Judge service scores the agent's output against them. Because automated judges can make mistakes, Microsoft advises representative tasks, repeated runs, and human review of the evidence.
Can I run Scope on my own machine? Yes. The quick-start guide starts a local stack with Docker Compose. Local development uses MongoDB, Redis, Azurite, and Lowkey Vault, and the same scenarios can later run on a Kubernetes deployment.
Does Scope rank AI coding agents? No. Scope compares agents on your tasks and your criteria. Microsoft and the documentation both stress that results describe the tasks and configurations tested, rather than a general ranking of agents.
The Bottom Line
Scope turns "agents seem to like our product" into an experiment you can rerun. The Cosmos DB numbers show the loop working inside Microsoft: define a task, watch where agents fail, fix the docs or the tooling, and measure again. The MIT License and the self-hosted design mean any team can run the same loop against its own SDK, CLI, or MCP server. It is early software, the Judge needs human oversight, and DevAx is a young discipline. But the direction is hard to argue with: agents are users now, and Scope is the first open platform built to measure their experience. Start with the GitHub repository and the official documentation, or read the Indonesian version of this article.