Loading...
Testing & Quality Gates
Every change to a project is scored by an aggregate quality gate before it ships, not waved through on a green checkmark alone. Tests, coverage, security, code health, and live preview quality are each evaluated independently and posted as their own GitHub commit status, separate from the ordinary pull request review status.
The Gate
Every project can enforce a canonical quality gate that aggregates five evidence categories, each with its own passed, failed, missing, or disabled status. The gate posts as a distinct GitHub commit status, “CrewWork / quality gate,” reported separately from the ordinary PR review status, so a failure is visible as its own check and blocks merge wherever your branch protection requires it. Default policy: 80 percent minimum line coverage, a health score of 70 or higher, and zero tolerance for new critical or high severity findings.
The workbench’s Quality & Security surface starts wide: open it without picking a project and you get a portfolio view of every project you can read, each with its latest health score, security rollup with new critical and high counts, the exact scan revision behind that rollup, and what it is currently working on. Health is tracked over time too, with each changed sample tagged improving, declining, or stable, and any previous scan can be opened to see its run provenance and a bounded page of its findings in their current status. Opening a project surfaces its findings as a filterable triage queue (action needed, reviewed, or scanner diagnostics, faceted by severity and family, with search) alongside a single finding inspector where every review action, including checking fix eligibility and the active remediation attempt, happens.
Every language stack in the run must pass. A failing test suite fails the component outright, independent of the other four.
Minimum 80 percent line coverage by default, measured from tool-native reports for Python and JS/TS. Other stacks report pass/fail counts but do not yet feed a coverage number into this component.
Zero new critical or high severity findings by default, evaluated against the diff, not the whole repository history.
Minimum health score of 70, out of 100, from static analysis of the changed code.
Accessibility, Core Web Vitals, bundle size, and console cleanliness measured against the exact committed revision running in preview, not a synthetic or stale snapshot.
Test Engine
The test-run engine runs your project’s own test command inside an ephemeral, network-isolated container on an allowlisted, digest-pinned base image, one per language. Any image outside the allowlist is rejected outright. Runs default to 2 concurrent test runs and a 30-minute timeout, with a hard 8MB ceiling on artifact output.
| Language | Test command | Pinned image |
|---|---|---|
| Python | pytest | python:3.12-slim |
| JavaScript / TypeScript | npm test | node:20-alpine |
| Go | go test -json | golang:1.22-alpine |
| Rust | cargo test --offline --locked | rust:1.75-slim |
| Ruby | bundle exec rake test | ruby:3.3-bookworm |
| PHP | phpunit | composer:2 |
| .NET / C# | dotnet test | mcr.microsoft.com/dotnet/sdk:8.0 |
| Java (Maven) | mvn --offline test | maven:3.9-eclipse-temurin-21-alpine |
| Java (Gradle) | gradle --offline test | gradle:8-jdk21-alpine |
| C / C++ | cmake -G Ninja, then ctest | crewwork-native-test-runtime |
C and C++ share a dedicated crewwork-native-test-runtime image (build-essential, CMake, Ninja, running as a non-root user) rather than installing a toolchain per run. Every other stack runs on a fixed, pinned public base image, so results reflect a known toolchain version, not whatever happened to be installed.
Isolation
Test containers get no Docker socket and no mounted volumes, and only lose network access after that disconnection is programmatically verified, not assumed. This runs on the same gVisor-isolated foundation used across the platform; see Runtime Isolation for the full sandboxing model and resource caps.
Coverage
Coverage numbers come from tool-native reports parsed directly off the test run, not a heuristic guess. Python coverage is read from coverage.py’s own JSON summary, and JavaScript/TypeScript coverage from an Istanbul/c8 JSON summary, each tagged with the exact tool version that produced it.
Coverage measurement is honest about its own reach today: it covers Python and JS/TS. The other eight stacks in the test engine run and report pass/fail counts, but do not yet feed a measured coverage percentage into the gate.
Coverage is also retained at the file level, covered and total lines per file straight from that same report, and files below an 80 percent coverage floor surface as low-coverage targets in the Quality & Security surface. Anyone with write access can open a target, review its exact measured baseline, choose a goal percentage, and launch a reviewed run scoped to lifting that one file. The run passes through the same plan-approval pause and quality gate as any other, and the improvement is only ever reported from a later measured test result, never estimated.
Flaky Tests
The engine persists per-testcase pass/fail history and flags any test whose outcome varies across its last 20 runs. Flagging is automatic. Quarantine is not: a human reviews the flagged test from the workbench or through the API and quarantines it with a mandatory reason and an expiry capped at 90 days, and can release it early at any time.
Speed
Before the full suite runs, the engine ranks and runs the test files most relevant to the changed files in that run first, so feedback on likely-affected code arrives sooner. The full suite still runs and remains the authoritative result: this is an ordering optimization, not a substitute for coverage. If ranking fails for any reason, including a database error or a type failure, the engine does not skip tests. It fails safe, running the full suite it would have run anyway. The one shortcut is honest too: if the preflight turns up a real failure, the run is reported failed immediately, since the full suite could not change that outcome. A passing verdict is only ever possible with the full suite executed.
Contracts
PR review diffs your base and head OpenAPI, GraphQL, and Protobuf contracts and classifies every change as compatible, dangerous, breaking, or an outright error. Any breaking or error-classified change blocks the pull request, which fails the quality gate.
Preview Quality
The preview quality component runs a real headless-browser measurement against a running preview, not a synthetic score. It only accepts evidence gated on a clean committed git revision, with an exact HEAD-and-tree match and an exact runner and app container identity, so a passing result can only describe the code actually under review. You can also trigger this exact measurement on demand from the Preview surface and see the same scoring the quality gate uses, before it ever runs as part of a PR.
CrewWork measures 1 to 8 declared same-origin paths per project. Every threshold is configurable per project. Once any threshold is enabled, preview quality becomes a blocking input to the aggregate quality gate. With none enabled, it still runs and reports as non-blocking enrichment.
Dynamic Scanning
A passive baseline scan runs against the exact running preview build with no extra authorization required. Two bounded active modes, web and API, are also available, but only with explicit per-project authorization: both run rate-limited and on an ephemeral, scanner-only network alongside the exact preview app, torn down after the scan completes. See Runtime Isolation and Security for how scanning fits the rest of the isolation and security model.
See exactly how CrewWork keeps a model honest, explore the architecture, or reach out to talk it through.
Or view the architecture for the full technical picture.