Model testing

Alloy chooses, tests and routes the AI models behind the agent, so your team gets the model that tests best for each job without doing the eval work.

Part of what you get with Alloy is our model selection and testing. We track new models, test them on robotics tasks, and route each job to the model that wins on that job. Your team does not have to shop for models or build evals.

You do not pick a model in Alloy. We pick, and we keep picking as new models ship.

Why model choice matters for robot data

Robot data is not the text that general models are mostly trained on. It is logs, high-rate time series, binary sensor formats and camera frames, often all from one run. A model that scores well on public benchmarks can still fail on it in specific, repeatable ways. These examples are illustrative:

The unit slip. A model reads a nanosecond timestamp column as seconds and reports that a 40-minute run lasted decades. The answer looks confident and is wrong.
The agreeable answer. An engineer asks, "Why did the arm overheat?" The joint temperature stayed flat all run. A weak model invents a cause anyway; it should push back.
The false correlation. A model compares a 1 kHz IMU stream with a 10 Hz GPS topic without aligning time, and "finds" a link between vibration and position drift that is not there.

Public leaderboards do not test for these. So we test for them ourselves.

How we evaluate

Robotics test cases with known answers. Each case has a known answer, so we score a model on what it finds, not on how sure it sounds.
Graded against evidence. An answer scores only when it finds the fault and backs it with the data. Points come off for the confident wrong claim we expect a model to make.
Precision over recall. Some cases are questions built on a false premise: they state a fault that is not there. The agent must say so, not agree. We would rather the agent say "I'm not sure" than guess. One confident wrong answer costs more trust than ten right ones earn.
New models get tested, not adopted. When a new model ships, we test it on our cases. A strong benchmark score is not enough.

Example task

Fictional example.

Recording: a 20-minute quadruped patrol. A slow battery voltage sag starts at minute 12. Question: "QP-21 stopped early. Was it the leg servos?" A good answer: says the servo currents were normal, finds the voltage sag, gives the time it began and links to the data. Penalty: blaming the servos.

What this means for you

You get the best model we have tested for each job. No model-shopping for your team.
Findings link to evidence. You can check what the agent saw before you act on it.
Changes are tested and staged. We test a model change before it reaches your organisation. Changes roll out in stages, organisation by organisation, because no test suite catches everything.
You keep up with new models without the work. When a new model appears, we test it, and you get it only where it does better.

What you control

You choose how the agent works, not which model it uses. Alloy offers a few modes that trade speed for depth. The app shows the current ones. The model behind a mode can change when a better one tests better.

What we don't publish, and why

We don't publish model names, vendors, test cases or scores, because those change often and they are the result of work we treat as our edge.