Quality scoring for AI coding-assistant sessions. Copilot, Claude Code and Cursor already emit OpenTelemetry. Point them here and get more than a token counter: a session quality score, the turn where the conversation stalled, and whether the accepted edits survived. Runs on your machine — no SDK, no account, nothing leaves the box.
What it is
CopilotScope captures GitHub Copilot chat sessions as OpenTelemetry spans and computes a composite quality score (0–100) in real time. No extra SDK, no data leaving your machine — just one OTLP endpoint pointed at the collector.
The six-component engine weighs reliability, acceptance, friction, latency, feedback, and efficiency. Only components with actual data enter the composite — no sessions are pinned near a neutral prior when telemetry is sparse.
Sibling project of AgentHelm: Scope observes (what was that session worth?), Helm steers (run the session, approve the tools, keep the record).
What we measure
Six weighted components: reliability 0.25, acceptance 0.20, friction 0.20, latency 0.15, feedback 0.10, efficiency 0.10. Weights renormalize to data that actually arrived — no phantom penalty for missing telemetry.
Every invoke_agent trace is one turn, scored for friction — LLM/tool errors, latency vs. this session's median TTFT, and repair loops (tool-call bursts with failures). Best and worst turns surfaced with reasons.
Four-gram matching plus a no-revert split (0.4/0.6 weight). Measures whether edits the agent suggested actually survived downstream — the truest signal of acceptance quality.
Per-model cost with configurable pricing (CopilotScope:Pricing), cache hit savings, cost per turn, and cost per accepted edit. Understand what each session actually spent.
Per-sample utility curve with >2 s and >8 s risk buckets. Translates raw TTFT numbers into an actionable quality signal — slow responses hurt differently at different thresholds.
EN/PL lexicon, Jaccard-distance rephrasing detection, and typography signals (caps lock, repeated punctuation). Local heuristic today; cloud deep-classifier via the judge agent is planned.
Write-behind upsert every second — telemetry bursts never cause write storms. Rehydrates sessions on startup. A Postgres outage degrades gracefully to in-memory; ingest never blocks.
Cross-session summary: total input/output/cache tokens, tokens per day, calls per model, top sessions by token burn, and average quality. Everything you burned across all chats in one view.
Quality engine
v2 of the quality engine renormalizes weights across components that have data. A session without feedback telemetry isn't penalized — the remaining five components absorb the weight and score what actually happened.
Algorithm status
Seven algorithms run entirely on your machine. Three requiring LLM access (G-Eval, SPUR, RAGAS) are planned for the cloud deployment via Azure AI Foundry — they register as "no-data" locally, never as an error.
Every algorithm below is also written up as a standalone thesis topic — one A4 page each, with scope, methodology and acceptance criteria — in Thesis Topics (PDF).
| Algorithm | Local | Cloud | Status |
|---|---|---|---|
| Edit Survival Analysis | ✓ full | ✓ full | four-gram + no-revert split |
| Acceptance-weighted throughput | ✓ full | ✓ full | LOC/turn, rejection-discounted |
| Turn Friction & Repair (TFRA) | ✓ full | ✓ full | error rate, latency, repair loops |
| Latency-utility model | ✓ full | ✓ full | utility curve, risk buckets |
| Token & cache economics | ✓ full | ✓ full | per-model cost, cache savings |
| Frustration detection | heuristic | 🔜 deep | lexicon + rephrasing + typography |
| LLM-as-a-Judge (G-Eval) | — | planned | needs Azure AI Foundry judge agent |
| SPUR / RAGAS | — | planned | rubric + retrieval metrics via judge |
Quick start
Prerequisite: none. One self-contained program for Windows, macOS and Linux (x64 and arm64): no Docker, no .NET, no key to generate, no environment variable to export, no JSON to hand-edit. It binds to 127.0.0.1, keeps sessions under ~/.copilotscope/data, and scores the Claude Code history already on your disk without any telemetry configured. For a team or a shared server the Docker Compose stack is still there, behind --docker.
The installer downloads the newest release, checks it against the release's SHA256SUMS and refuses it on any mismatch, installs it under ~/.copilotscope, and offers to point each assistant it finds at it — showing every change first. The path names a release on purpose: the installer you run is the one that release shipped and was tested with. Prefer no script at all? Download the archive for your platform, check the sum yourself, extract it and run copilotscope from the extracted folder. Team or shared server: add --docker (-Docker on Windows) to install the Compose stack instead.
Learning it properly: four step-by-step tutorials in English and Polish live in docs/tutorials, and the whole system — architecture, every signal, the scoring arithmetic and the honest limits — is one PDF in either language, built from docs/papers.
Why a settings file, not export: an exported variable lives in one terminal, which is the single most common reason a correctly-configured assistant produces an empty dashboard. Claude Code and VS Code both read a file at startup; only Copilot CLI is env-var-only. copilotscope disconnect removes exactly the keys it added.
Requires the .NET 10 SDK; Aspire ships as a NuGet MSBuild SDK, so there is no workload to install. F5 on CopilotScope.AppHost does the same with the debugger attached to everything. Run dotnet test for the full suite (decoder, quality, persistence round-trips).
Try it without a Copilot subscription
dotnet run --project tools/CopilotScope.TelemetryGen -- http://localhost:4318 plays one realistic session over the real OTLP path; the Docker stack also has copilotscope demo for a whole fabricated datasetcopilotscope demo dataset is badged demo, so a screenshot is never mistaken for evidenceAlready use Claude Code? Its history on disk is read on every start, and copilotscope scan reads it now, with no telemetry configuration at all. From a clone with the SDK, the same three tools run as dotnet run --project tools/CopilotScope.TelemetryGen and friends.