Compare two releases

Ask the agent to compare runs on two software versions. It lines up the runs, compares the signals you care about, and writes up what got better, what got worse, and the runs behind each claim.

When to use it

• You need to know if a new build is ready.
• A metric moved after a release and you want to know why.
• You compare builds by hand today, one log at a time.

How to do it

1. Make sure your missions carry the software version, for example in a metadata.json file with each run.
2. Ask the agent to compare the two versions, and name what matters: "docking accuracy", "localisation dropouts", "battery use per hour".
3. Narrow it if you need to: by robot, by site, by route or by test case.
4. Ask for a report to share with the team, and refine it in the chat.

Example

Fictional example.

Ask: "Compare 7.2.4 and 7.3.0 on this week's runs. Did anything get worse?"

Agent: 7.3.0 pauses at busy crossings: 214 costmap update timeout messages on 9 of 11 robots, none on 7.2.4. Docking accuracy and battery use are unchanged. [Runs for each version] [Chart]

Tips

• Compare like with like: the same route, site or test case on both versions.
• Ask for counts with their denominator: "6 of 52 runs", not "often".
• Turn the comparison into a recurring report to check each new build.

Related

• Validation & Testing
• Scenarios & Regression Testing
• Get a report on every new mission