Testing Policy¶

This page states the team's rules about testing: what must be tested, what may be skipped, how failures are handled, and what is allowed to be merged. The policy applies to the Toodle (frontend), toodle-api (backend) and toodle-docs (documentation) repositories equally, except where a repository does not have the relevant tooling.

The policy is enforced in three places: local Vitest coverage thresholds, CI status checks, and pull-request review. Reviewers reject work that does not follow the policy even when CI is green.

Core rules¶

#	Rule	Enforcement
P1	Feature changes that affect workflows must include useful tests for success, failure and role-based cases	Code review
P2	Bug fixes must include a regression test that reproduces the bug where practical	Code review
P3	CI must fail on test, lint, format or build failures; a broken pipeline blocks merge	CI status checks
P4	Coverage must not fall below the configured targets (see Coverage requirements)	Vitest thresholds + Codecov
P5	Tests may only be skipped behind an explicit, documented condition — never silently	Code review; `describe.skipIf` pattern
P6	Tests must never use real user data, real credentials or the school's shared database	Code review
P7	Tests must be deterministic: no flaky sleeps, no dependence on network services or real time unless explicitly controlled	Code review
P8	Changes that alter behaviour must update the affected documentation and tests in the same change set	Code review
P9	A deployment is not accepted until the public URL, authentication, role journeys and database migration history have been checked	Release checklist
P10	Skipped tests and real logs must be recorded when a run is used as evidence	Verification records

What must be tested¶

A feature is not complete until its tests exist. For a typical API feature this means:

success case — a valid request produces the documented response;
failure cases — invalid input, missing fields and unsatisfied business rules are rejected with the documented error;
role cases — the route enforces its role and ownership boundaries (who may call it, and whose data it may touch); and
state changes — created/updated records are persisted as described (via mocks in unit/route tests, or real PostgreSQL where applicable).

For a frontend feature:

the component renders for each relevant role state;
form input and submission produce the expected API payload;
loading, success and error states are displayed; and
role-gated controls are shown or hidden correctly.

The existing suites demonstrate this pattern — for example, the course application tests in toodle-api/tests/ cover approval, rejection, verification and duplicate/closed application behaviour, and Toodle/tests/course-applications.test.jsx covers the corresponding UI.

Bug fixes¶

Every bug fix ships with a regression test that reproduces the reported behaviour. The test must fail against the pre-fix code and pass after the fix. If a test genuinely cannot be written (for example, an environment-only issue), the pull request must state why and link the manual verification evidence instead.

Bugs are tracked through Trello with reproduction steps, severity, owner, fix and retest evidence, as described in Team & Methodology.

Coverage requirements¶

Coverage policy is defined identically in both application repositories via codecov.yml and enforced locally by Vitest:

Metric	Target
Project coverage (lines, functions, statements)	70%
Branch coverage	60%
Patch coverage (new/changed lines)	60%
Tolerance before a project check fails	2%

Interpretation:

Project target — the suite as a whole must stay at or above 70%. A drop of more than 2% below target fails.
Patch target — at least 60% of lines added or changed in a pull request must be exercised by tests. Trivial wiring code is the exception, not the rule.

Coverage numbers are a floor, not a goal. A change that meets 60% patch coverage but leaves the risky branch untested still fails review.

Skipping tests¶

Skipping is allowed only when a test depends on infrastructure that is deliberately absent from the environment. The only sanctioned pattern is the conditional suite used by the PostgreSQL tests in toodle-api:

describe.skipIf(!process.env.B5_TEST_DATABASE_URL)('B5 PostgreSQL transaction integration', () => {
  // real-database tests, guarded against the school's database in beforeEach
});

Requirements for any skip:

the condition must be explicit and inspectable (skipIf + environment variable, or equivalent);
the file must document the opt-in variable and how to enable it;
the reason for the skip must be visible in the test report, not hidden in a comment; and
skipped suites must be run and recorded before their behaviour is claimed as verified.

it.skip / describe.skip used to temporarily disable a failing test is not acceptable in a pull request: the test must be fixed or the change must not merge.

Test data and security¶

Tests use fabricated fixtures (for example the fixed UUIDs and names in toodle-api/tests/helpers/swap-database.js) and test identities such as auth0|integration-tutor.
Tests never use real student marks, names, emails or transcripts; recorded evidence (screenshots, logs) is redacted.
Test database connections must point at a disposable local database (b5_test); the PostgreSQL suites refuse to run against any other database.
Secrets, API keys and .env values are never committed to test fixtures or CI logs. CI uses test-only placeholders (for example test-tenant.auth0.com).
Coverage and CI configuration files must not embed live tokens.

Pre-merge checklist¶

Before opening a pull request, the contributor runs, in the affected repository:

npm run lint (and npm run format:check) — style and static checks;
npm test or npm run test:coverage — the automated suite, including thresholds;
npm run build — the production build or Prisma client generation;
a manual check of the affected feature; and
python -m mkdocs build --strict — for documentation changes.

The pull request is opened only after all of these pass locally. Reviewers re-verify the CI status and confirm the policy cases (P1, P2, P5, P6) are covered.

For the documentation repository, the validation command is:

python -m mkdocs build --strict

Merge and release gates¶

A pull request cannot be merged while any CI check fails.
A deployment to production happens only from main, and only after the pipeline (including tests) passes — the API's deploy step runs only on main and only after the lint-and-test job.
A release is accepted only when the deployed public URL, authentication flow, role journeys and database migration history have been verified against the live system.

Exceptions¶

If a rule cannot be followed, the pull request must record:

which rule is being excepted;
why it cannot be followed now;
what compensating evidence is provided (for example, manual verification records); and
a follow-up task in the tracker to close the gap.

Exceptions are decided by the team in review — not silently by the author. Repeated exceptions for the same area indicate the policy or the design is wrong and should be revisited at the next retrospective.

Where each rule lives¶

Rule	Source of truth
Coverage targets	`codecov.yml` and `vitest.config.mjs` (`toodle-api`) / `vite.config.js` test block (`Toodle`)
CI stages and gates	`.gitea/workflows/` and `.github/workflows/` in each repository
Suite scope and skip conditions	`toodle-api/tests/setup.js`, `tests/integration/.postgres.test.js`, `Toodle/tests/`
Bug fix and feedback loop	User Feedback Process and Team & Methodology
Pre-merge and Definition of Done	Team & Methodology
Evidence standard	Sprint 2 audit