Validation & Testing

Know whether a build is ready the day the test ends. Your validation steps run on every test run, builds are compared across runs, and the result is a readiness report your lead can sign off.

What it is

Running your team's validation analysis on every test run and comparing builds across runs, so you know whether a release is ready.

How Alloy does it

Your steps run on every test run. Write your validation steps once, in plain words, as a recurring report. It runs when each run finishes processing, and its first step can check that every log you need is there.
It computes against your criteria. The agent queries the recorded signals and runs its own analysis code: hold error, saturation events, whatever your steps define. It can draw on your team's scripts in the scripts library.
It compares builds. It groups runs by version, vehicle or test case and reports the differences, with the charts and the moments behind them.
The result is a report you can share. A readiness report with links to every run, and a version history so a reviewer sees what changed.

Ask in the Alloy app, in Slack, on a Jira ticket, or from your coding agent.

Example

Fictional example: the team, the drones and the numbers are invented.

A drone team tests flight-controller release 2.6. On Tuesday they fly 9 position-hold runs in gusty conditions: 6 on quadcopters and 3 on a heavier hexacopter. Each run uploads when the drone lands.

After each run processes. A recurring report runs the team's own steps on each run: compare the position estimate with the survey-grade reference track, compute horizontal hold error, and count motor-saturation events. Its first step is a data check. On run 7 it flags that the reference log is missing, so the run cannot be scored. The test lead sees it before the team packs up and re-flies the run.

At the end of the day. The validation engineer asks the agent:

"Compare 2.6 with 2.5 across all position-hold runs from the last two campaigns. Report hold error in gusts above 8 m/s, by airframe."

The agent answers with a report:

Quadcopter: no change. Mean hold error 0.42 m on 2.6 (14 runs) against 0.40 m on 2.5 (17 runs).

Hexacopter: worse. Mean hold error 1.15 m on 2.6 (3 runs) against 0.44 m on 2.5 (5 runs). In all three 2.6 runs the error grows after the motors saturate during a gust, which does not happen on 2.5. [Chart] [Moments in Replay]

Excluded: run 7 (original), no reference log.

Not confirmed: three hexacopter runs is a small sample. A cause is not identified; the pattern matches a change in how 2.6 limits thrust on heavier airframes.

Readiness: 2.6 meets the quadcopter hold criterion. 2.6 does not meet the hexacopter hold criterion.

The lead shares the report link with the flight-control team the same evening. They get the three runs and the moments, not a summary in a chat message.

How to start

Ready to start. Talk to us. Alloy sets up your workspace with you, writes your validation steps as a report, and your next field test gets the analysis as it uploads.

Want proof first. Send the logs from a campaign you have already analysed, with your team's results. Alloy runs the same analysis so you can compare the numbers line by line.

Just looking. See the agent work on sample robots in Try Alloy.

What we need from you: access to your test logs, your validation steps or scripts, and the metadata you use: vehicle, version, test case.

Outcomes for your team

Analysis takes minutes, not a day. The readiness question is answered the day the test ends.

"We used to spend a full day after every field test just doing analysis. Now with Alloy, that's just ten minutes, and the conversation has completely shifted."

Jai Castle, Product Validation Manager. How Advanced Navigation cut validation analysis

Bad runs are caught before the team packs up. A data check on each run flags a missing or broken log while the vehicle is still on site.

Your engineers stop rewriting one-off analysis scripts. The same steps run on every run, so analysis effort goes into the questions only your team can answer.

"I definitely don't miss writing all the NumPy and Matplotlib Python scripts"

Mitch Galea, Senior Autonomy Engineer. How SwarmFarm is scaling its robot fleet

Scenarios & Regression Testing: check every new run for a regression you have seen before.
Fleet Monitoring: the same recurring report, for every run in the field.
Root Cause Analysis: when a regression needs a cause.
When to use Alloy