Scenarios & Regression Testing

Describe a failure pattern once in plain words. Alloy searches your recent runs for it, checks every new run, and lists each match with the evidence, so you can see whether a fix worked.

What it is

Describing a failure pattern once, in plain words, and having Alloy check every run for it: your recent history and every new run.

How Alloy does it

Describe it in plain words. On the Scenarios page, or in chat. The agent works with you on a precise definition, then creates the check.
It searches your history. The previous 7 days by default, and up to 90 days back.
It checks every new run. Runs uploaded later are checked as they arrive. You do not re-run anything.
Every match shows its evidence. The match reason, the time window, the run, and images when the run has them. Filter matches by robot, location, date or release.

Ask in the Alloy app, in Slack, on a Jira ticket, or from your coding agent.

Example

Fictional example: the robots, the warehouse and the numbers are invented.

A team runs 25 warehouse robots. Release 4.1 shipped twelve days ago. An operator reports that AMR-09 failed to dock twice last night.

The fleet engineer opens Scenarios, clicks Create, and describes the pattern to the agent:

"Find times when a robot came within 1 m of a charger, stopped, and retried docking more than twice."

The agent works with her on a precise definition, then creates the scenario. She sets the Initial scan range to Previous 3 months, so the search reaches back before 4.1.

When the status reads Up to date, the scenario has 11 matches in 8 missions. She filters them by the release label:

Release 4.0: 0 matches.
Release 4.1: 11 matches, on 6 robots.

One match is a robot that a person stopped by hand. She removes it with Remove match…. That leaves 10.

The navigation team fixes the docking approach in release 4.1.1. The scenario checks each new run. When she filters by release again, 4.1.1 has no matches so far. She keeps the scenario active, so a return of the pattern shows up as a new match.

How to start

Ready to start. Talk to us. Alloy sets up your workspace with you, and the patterns your team already watches for become scenarios that check every new run.

Want proof first. Describe a failure you have already fixed and scan back as far as you can. You see where else it happened, and whether the fix held.

Just looking. See the agent work on sample robots in Try Alloy.

What we need from you: access to your robot data, and a release or version label on your runs.

Outcomes for your team

A regression shows up on the first bad run, not at the first complaint. Measure the time from a release to the first detection of a known problem.

You see whether a fix worked. Filter the matches by release: before the fix, after the fix.

Known problems stop costing investigation time. Patterns you have described are found for you, so engineers spend their time on the new ones.

"When a bug is truly novel, I can spend hours or days hunting it down rather than having to go classify everything"

Tom Cameron, Tech Lead, AI Engineering. How Andromeda uses Alloy

Root Cause Analysis: find the cause before you write the check.
Fleet Monitoring: turn match counts into a report your team shares.
Validation & Testing: check a new build before it ships.
Scenarios reference: every column, filter and setting.