← Back to home
How it works

A closer look at one run

One change, followed from merge to report, using a checkout test as the example. It's simplified, not from a client project.

  1. Step 1

    Read the change

    A pull request that changed the payment step just got merged. The system reads the code change and the ticket behind it, and works out which tests it could affect. Here, that means the checkout tests. Everything else is left alone, so runs stay quick.

  2. Step 2

    Update the tests

    Each affected test is rewritten in plain English, with a list of things the agent must prove it saw. Then it's turned into code. If the code and the English don't match, it gets flagged for me instead of guessed at.

    example · checkout test
    test: Guest checks out with a saved card
    updated: after PR #482 changed the payment step
     
    steps:
    1. Open the cart and start checkout as a guest
    2. Enter buyer@example.com as the email
    3. Pick "Use saved card" and place the order
     
    evidence the agent has to show:
    - the saved card list finished loading
    - an "Order confirmed" heading on screen
    - the URL moved to /orders/<id>
  3. Step 3

    Run and explore

    The tests run against the new build. A test only passes if the agent shows it saw everything on its list: an element on screen, a screenshot, or a behavior. Meanwhile, a second agent tries unusual paths, like logging out halfway through checkout, to find bugs nobody wrote a test for.

  4. Step 4

    Report

    Failures are rerun to rule out flaky tests, and a real failure gets its own check before it's reported. Missing proof doesn't count as a failure. It usually means the agent got lost, and I sort that out. Your key flows are also timed, so slowdowns get flagged. I read the report before you see it.

    run · main @ 4f2a9c1 (PR #482)
    env: Chromium · Firefox · WebKitrun: 2026-06-09 09:41 UTC 2m 47s
    58
    passed
    0
    failed
    1
    flaky
    • checkout › guest with saved card (3 of 3 proven)5.9s
    • checkout › declined card6.0s
    • cart › add, edit, remove4.6s
    • search › filters + sort8.2s
    • explore › log out mid-checkout, come back11.4s
    Evidence is attached for every pass. The search test failed once and passed on rerun, so it's marked flaky, not failed. Exploring between flows turned up nothing new this time.

Want to see it on your app?

The first audit is free. Setup takes a few days, then a monthly plan keeps it running.

Get a free audit Book a 20-minute call

See pricing