Skip to content
Impact Gate Impact Gate Impact Gate

AI Guardrails

Guarded generation

Impact Gate does not treat generated code as trustworthy just because a model produced it. The AI path is designed to reduce hallucinations before generation and to block suspicious output after generation.

Guardrail 1: Deterministic First

01 deterministic first

The strongest workflow does not depend on a model

Diff analysis, impact mapping, coverage planning, release-diff planning, and threshold gating already exist before any prompt is sent.

02 local grounding

Generation is anchored to the repo’s API surface

The tool discovers page objects, helper methods, inherited methods, and method signatures before it asks the model to write anything.

The strongest workflow does not depend on an LLM:

  • diff analysis
  • impact mapping
  • coverage planning
  • release-diff planning
  • threshold gating

AI is layered on after the deterministic picture already exists.

Guardrail 2: Local API-Surface Grounding

Generation is grounded in the actual repository, not a generic mental model of Playwright tests.

The tool extracts:

  • page objects
  • helper methods
  • inherited methods
  • method signatures

That discovered API surface is then injected into generation prompts so the model works from real project methods.

Guardrail 3: Prompt Constraints

03 prompt constraints

The prompt narrows the space where hallucinations appear

  • use only known methods
  • do not invent project-specific helpers
  • fall back to raw Playwright selectors if a helper does not exist
  • return code only
04 prompt sanitization

Inputs are cleaned before they reach the model

User-action strings, evidence, and flow names are sanitized to reduce prompt pollution and keep upstream text from contaminating the output.

Generation prompts explicitly say:

  • use only known methods
  • do not invent project-specific helpers
  • fall back to raw Playwright selectors if a helper does not exist
  • return code only

This sharply narrows the space where hallucinations usually creep in.

Guardrail 4: Prompt Sanitization

User-action strings, evidence, and flow names are sanitized before they go into prompts. That reduces prompt pollution and lowers the chance that upstream text contaminates the generated output.

Guardrail 5: Hallucination Detection

05 hallucination detection

Suspicious method calls are scanned after generation

Invented page-object methods, fabricated helpers, and fake wrapper calls are detected against the discovered API surface.

06 quarantine

Suspicious specs are moved into a review queue

Blocked specs are written to generated-needs-review/ so they stay visible without silently entering the trusted suite.

After generation, the code is scanned for method calls that do not exist in the discovered API surface.

Examples of suspicious output:

  • invented page-object methods
  • fabricated helpers
  • project-specific wrapper calls that do not actually exist

By default, suspicious specs are blocked.

Guardrail 6: Needs-Review Quarantine

Blocked specs are written to:

generated-needs-review/

That keeps suspicious code out of the main trusted test tree while still letting the team inspect what the model attempted.

Guardrail 7: Compile And Smoke Verification

07 verification

Written specs still have to earn trust

  • compile check first
  • smoke run when Playwright is available
  • failed specs move out of the trusted path

Specs that are written into the main path are still verified:

  • compile check first
  • smoke run when Playwright is available

Specs that fail verification are moved out of the trusted path.

Practical Takeaway

Practical takeaway

This is not “generate and pray”

01 build evidence 02 ground the prompt 03 constrain the output 04 detect suspicious calls 05 quarantine risky specs 06 verify written specs

That is a much stronger story for teams evaluating the product than generic “AI-powered testing.”