# Quality gates that demand evidence Every change to a project is scored by an aggregate quality gate before it ships, not waved through on a green checkmark alone. Tests, coverage, security, code health, and live preview quality are each evaluated independently and posted as their own GitHub commit status, separate from the ordinary pull request review status. 5-part quality gate10-language test engineVerified network isolation [See it inside HotFix](https://crewwork.ai/sentry/) ## One aggregate gate, five fixed components Every project can enforce a canonical quality gate that aggregates five evidence categories, each with its own passed, failed, missing, or disabled status. The gate posts as a distinct GitHub commit status, “CrewWork / quality gate,” reported separately from the ordinary PR review status, so a failure is visible as its own check and blocks merge wherever your branch protection requires it. Default policy: 80 percent minimum line coverage, a health score of 70 or higher, and zero tolerance for new critical or high severity findings. The workbench’s Quality & Security surface starts wide, with a portfolio view of every project you can read before you open any one project. - Each project shows its latest health score, a security rollup with new critical and high counts, the exact scan revision behind that rollup, and what it is currently working on. - Health is tracked over time, with each changed sample tagged improving, declining, or stable, and any previous scan can be opened to see its run provenance and a bounded page of its findings in their current status. - Opening a project surfaces its findings as a filterable triage queue (action needed, reviewed, or scanner diagnostics; faceted by severity and family; with search) alongside a single finding inspector where every review action, including checking fix eligibility and the active remediation attempt, happens. ### Tests Every language stack in the run must pass. A failing test suite fails the component outright, independent of the other four. ### Coverage Minimum 80 percent line coverage by default; measured for Python and JS/TS only (details below). ### Security Zero new critical or high severity findings by default, evaluated against the diff, not the whole repository history. ### Code health Minimum health score of 70, out of 100, from static analysis of the changed code. ### Live preview quality Accessibility, Core Web Vitals, bundle size, and console cleanliness measured against the exact committed revision running in preview, not a synthetic or stale snapshot. ## Ten language stacks, one execution engine The test-run engine runs your project’s own test command inside an ephemeral, network-isolated container on an allowlisted, digest-pinned base image, one per language. Any image outside the allowlist is rejected outright. Runs default to 2 concurrent test runs and a 30-minute timeout, with a hard 8MB ceiling on artifact output. Language Test command Pinned image Python pytest python:3.12-slim JavaScript / TypeScript npm test node:20-alpine Go go test -json golang:1.22-alpine Rust cargo test --offline --locked rust:1.75-slim Ruby bundle exec rake test ruby:3.3-bookworm PHP phpunit composer:2 .NET dotnet test mcr.microsoft.com/dotnet/sdk:8.0 Java (Maven) mvn --offline test maven:3.9-eclipse-temurin-21-alpine Java (Gradle) gradle --offline test gradle:8-jdk21-alpine C / C++ cmake -G Ninja, then ctest crewwork-native-test-runtime C and C++ share a dedicated crewwork-native-test-runtime image (build-essential, CMake, Ninja, running as a non-root user) rather than installing a toolchain per run. Every other stack runs on a fixed, pinned public base image, so results reflect a known toolchain version, not whatever happened to be installed. ### Every test command runs after network isolation is verified Dependency installation for a test run happens in a separate, network-enabled phase against an allowlisted package index. The container is then disconnected from every network, and that isolation is verified before any test command runs, not assumed. This runs on the same gVisor-isolated foundation used across the platform; see [Runtime Isolation](https://crewwork.ai/isolation/) for the full sandboxing model and resource caps. ## Coverage comes from the tools themselves Coverage numbers come from tool-native reports parsed directly off the test run: Python from coverage.py’s own JSON summary and JavaScript/TypeScript from an Istanbul/c8 JSON summary, each tagged with the exact tool version that produced it. Measured coverage is available for Python and JS/TS only; the other eight stacks report pass/fail counts but do not yet feed a coverage percentage into the gate. Coverage is also retained at the file level, covered and total lines per file straight from that same report, and files below an 80 percent coverage floor surface as low-coverage targets in the Quality & Security surface. Anyone with write access can open a target, review its exact measured baseline, choose a goal percentage, and launch a reviewed run scoped to lifting that one file. The run passes through the same plan-approval pause and quality gate as any other, and the improvement is only ever reported from a later measured test result, never estimated. ### On every PR - Line-coverage percentage-point delta from base to head, broken out by language - Dependency delta: added, removed, and updated package counts plus risk-tier counts - Both computed from a real base/head comparison, never an estimate ## Flagged automatically, quarantined by a human The engine persists per-testcase pass/fail history and flags any test whose outcome varies across its last 20 runs. Flagging is automatic. Quarantine is not: a human reviews the flagged test from the workbench or through the API and quarantines it with a mandatory reason and an expiry capped at 90 days, and can release it early at any time. - 20-run history window: a test is flagged when its outcome varies across its last 20 recorded occurrences on the same identity - Human-reviewed quarantine: a flagged test is quarantined from the workbench or through the API with a mandatory reason, never silently or automatically. Reviewing a flagged test also opens its outcome and duration history: the most recent recorded runs with pass, fail, or error, duration, branch, and commit - 90-day maximum expiry: every quarantine carries a timezone-aware expiry capped at 90 days, and can be released early once fixed ## Fail-safe predictive test selection Before the full suite runs, the engine ranks and runs the test files most relevant to the changed files in that run first, so feedback on likely-affected code arrives sooner. The full suite still runs and remains the authoritative result: this is an ordering optimization, not a substitute for coverage. If ranking fails for any reason, including a database error or a type failure, the engine fails safe and runs the full suite it would have run anyway. The one shortcut is honest too: if the preflight turns up a real failure, the run is reported failed immediately, since the full suite could not change that outcome. A passing verdict is only ever possible with the full suite executed. ## Budgets measured on the exact committed revision The preview quality component runs a real headless-browser measurement against a running preview, not a synthetic score. It only accepts evidence gated on a clean committed git revision, with an exact HEAD-and-tree match and an exact runner and app container identity, so a passing result can only describe the code actually under review. You can also trigger this exact measurement on demand from the Preview surface and see the same scoring the quality gate uses, before it ever runs as part of a PR. - Accessibility violations, via an automated axe-core scan - Core Web Vitals: Largest Contentful Paint (LCP) and Cumulative Layout Shift (CLS) - Bundle size, in bytes - Navigation latency (DOMContentLoaded) - Console error count CrewWork measures 1 to 8 declared same-origin paths per project. Every threshold is configurable per project. Once any threshold is enabled, preview quality becomes a blocking input to the aggregate quality gate. With none enabled, it still runs and reports as non-blocking enrichment. ### Exact API contract diffing gates the PR PR review diffs your base and head OpenAPI, GraphQL, and Protobuf contracts and classifies every change as compatible, dangerous, breaking, or an outright error. Any breaking or error-classified change blocks the pull request, which fails the quality gate. ## Passive scanning by default, active scanning opt-in A passive baseline scan runs against the exact running preview build with no extra authorization required. Two bounded active modes, web and API, are also available, but only with explicit per-project authorization: both run rate-limited and on an ephemeral, scanner-only network alongside the exact preview app, torn down after the scan completes. See [Runtime Isolation](https://crewwork.ai/isolation/) and [Security](https://crewwork.ai/security/) for how scanning fits the rest of the isolation and security model. ## Common questions ## Does the quality gate apply to platform self-repair too? Platform self-repair runs its own dedicated validation pipeline with tiered matrix and shadow validation, described on the [Platform Self-Repair](https://crewwork.ai/self-modification/) page. The five-part quality gate described here is the canonical per-project gate that governs your application repositories. ## What happens when a HotFix run fails the gate? The review still publishes with its usual comment. The quality gate posts its own separate “CrewWork / quality gate” GitHub status reporting the failure, which blocks merge when your branch protection requires that check to pass. See [Sentry & HotFix](https://crewwork.ai/sentry/) for how a fix reaches this stage in the first place. ## Can I connect my own CI instead? CrewWork’s test-run engine is the substrate that both HotFix and your own triggered runs use. Source provider connections, including GitHub, GitLab.com, Bitbucket Cloud, and Jira Cloud, are covered on [Source Control & Integrations](https://crewwork.ai/integrations/). ## Are quality gate thresholds configurable per project? Yes. Minimum coverage, minimum health score, and the critical/high finding tolerance are all per-project policy values. The numbers on this page are the shipped defaults, not fixed limits. ## Bring your next piece of work. Discuss what you want to build, repair, or improve, along with your deployment needs and access to CrewWork. [Discuss your project](https://crewwork.ai/about/#access) [Truth & Trust Engineering](https://crewwork.ai/trust-engineering/) Or [view the architecture](https://crewwork.ai/architecture/) for the full technical picture. --- Canonical page: https://crewwork.ai/quality/