# Welcome to Alloy
**Alloy is the data platform for robotics teams.** Stream mission logs, video, and sensor traces off your fleet — drones, marine systems, AMRs, agricultural robots, anything recording MCAP — and turn raw recordings into a searchable, queryable record of every test, mission, and field run your robots have ever done. Support for ROS bags and other formats is available upon request.
Alloy comes in two products. Use one or both.
## Mesh Storage [#mesh-storage]
Managed storage for robotics recordings. Ingest MCAPs from the browser, your devices, or cloud storage; replay and inspect them in the dashboard; query the whole catalog from SQL or any Iceberg-compatible tool — and connect AI tools through MCP.
Or read the [Mesh Storage overview](/docs/mesh-storage/overview) — what it is, how files move through it, and what each part of the page does.
## Agent Platform [#agent-platform]
AI-powered analysis on top of your mission data. Auto-generated summaries for every mission, semantic search across instances, recurring scenario scanners, and an Alloy AI agent in the dashboard and in Slack.
Or read the [Agent Platform overview](/docs/agent-platform/overview) — core concepts, the dashboard tour, workflows, and the integrations that connect it to your stack.
# Getting Started
Alloy is data-first — everything starts with getting your mission data into the platform. Once it's in, Alloy handles parsing, metadata extraction, AI analysis, and report generation automatically.
## Quick start: Browser upload [#quick-start-browser-upload]
Go to
**Missions**
in the sidebar
Click
**+ Add**
in the top right
Select your files, drag them in, or drop a folder
Add an optional
**description**
— Alloy's AI uses this as context to generate a better report
Click
**Upload**
— Alloy processes your data and generates a report
## Supported file types [#supported-file-types]
Alloy supports **MCAP** (`.mcap`) uploads by default. Support for ROS bags, CSV, PX4, ArduPilot, and other formats is available upon request.
**Folder upload**: When you upload a folder, Alloy uses the **folder name as the mission name**. Handy when your data is already organized into directories.
**Multiple files**: Select multiple files in a single upload. If they belong to the same session, Alloy groups them into one mission — useful for multi-robot operations where you have a fleet of robots that ran together.
## Other ways to get data in [#other-ways-to-get-data-in]
For teams that store mission data in cloud storage, Alloy can automatically watch a bucket and ingest new files as they appear — no manual uploads needed.
* Alloy detects new files automatically and creates missions
* Each folder becomes a mission, with the folder name as the mission name
* Include a `metadata.json` file to attach structured metadata (operator, vehicle, location, etc.)
* Data flows in continuously as new files land in the bucket
See the setup guides for [AWS S3](/docs/agent-platform/integrations/aws-s3) and [Google Cloud Storage](/docs/agent-platform/integrations/gcs).
If you're running Alloy's edge agent on your hardware, data uploads automatically.
Go to
**Devices**
→
**Setup Edge**
for the guided setup
Generate a
**provisioning key**
for the device
Deploy the pre-configured Docker Compose file to your device
The edge agent captures data and syncs it to Alloy automatically
Once provisioned, manage capture settings, view diagnostics, and monitor health from the Devices page.
## Folder structure [#folder-structure]
Alloy maps your folder structure to missions — this applies to browser uploads, S3, GCS, and edge device uploads:
* **Each folder** at the top level becomes a separate mission
* **The folder name** becomes the mission name
* **All files inside** a folder are grouped into that mission
* **Nested folders** are flattened — all files within are included regardless of depth
```
my-data/
├── mission-2024-03-15/ → Mission: "mission-2024-03-15"
│ ├── field-notes.txt ← operator notes (used as AI context)
│ ├── robot1/
│ │ ├── recording.mcap
│ │ └── metadata.json
│ ├── robot2/
│ │ ├── recording.mcap
│ │ └── metadata.json
│ └── metadata.json ← mission-level metadata
```
### Metadata JSON [#metadata-json]
You can include a `metadata.json` file in any mission folder to attach structured metadata. This metadata becomes filterable in the Alloy mission library.
```json
{
"operator": "Jane Smith",
"vehicle": "AUV-03",
"location": "Harbor East",
"firmware_version": "2.4.1",
"notes": "Post-maintenance checkout dive, calm conditions"
}
```
All fields are optional and flexible — use whatever key-value pairs make sense for your workflow. These values appear as filter options in the mission library sidebar.
## What happens during processing [#what-happens-during-processing]
After your data arrives (by any method), Alloy runs it through the ingestion pipeline:
**Parsing**
— reads log files, extracts topics, channels, and raw data
**Metadata extraction**
— identifies duration, timestamps, sensor types, device info
**AI analysis**
— detects anomalies, calculates metrics, generates visualizations
**Summary generation**
— compiles everything into the final mission view
Processing time depends on file size, but most missions complete in a few minutes. You'll see a processing indicator on the Missions page.
## Viewing your results [#viewing-your-results]
Once processing finishes, your mission appears with a **View report** button. The report includes a narrative summary, key events, metrics, timeseries plots, maps, trajectory visualizations, and anomaly highlights. See [Missions](/docs/agent-platform/navigating-alloy/missions) for a full breakdown of report components.
## What's next? [#whats-next]
# Overview
The **Agent Platform** is the analysis and reporting side of Alloy. Where [Mesh Storage](/docs/mesh-storage) is the data lake — files, replay, SQL, MCP — the Agent Platform turns those recordings into mission summaries, searchable instances, recurring scenarios, and natural-language workflows powered by Alloy AI.
If your org has the Agent Platform enabled, you'll see Missions, Reports, Scenarios, and the Alloy AI chat panel in your dashboard. The pages below explain how each piece works.
The Agent Platform is enabled per-org. If you don't see it in your dashboard and you'd like access, talk to your Alloy account manager.
## Start here [#start-here]
## Use the platform [#use-the-platform]
# Troubleshooting
## My device shows as Pending but I haven't approved it yet [#my-device-shows-as-pending-but-i-havent-approved-it-yet]
This is expected. The device appears as **Pending** as soon as the client contacts Alloy — it's waiting for an admin to approve it. Open the `devices/` folder in Mesh Storage, find the device, and click **Approve**.
## I approved the device but it still shows as Pending [#i-approved-the-device-but-it-still-shows-as-pending]
The client picks up the approval on its next sync (every `poll_secs` seconds, default 15). Wait a moment and refresh — the status should update automatically. If it doesn't:
* Check that the client process is still running (`systemctl status alloy-edge` or `docker ps`)
* Check the client logs for connection errors
* Verify the client can reach your Alloy endpoint on port 443
## My device isn't uploading files [#my-device-isnt-uploading-files]
First confirm the device is **Approved** and **Last seen** is recent. If so:
* Check that files exist in the watched directory (`input_dir` in the sync config, default `/recordings`)
* Check that files match the pattern (`file_pattern`, default `*.mcap,*.json,*.jsonl`)
* If `mcap_require_footer: true`, the client won't upload a `.mcap` file until it has a valid MCAP footer — in-progress recordings won't upload until the recording stops. The default is `false` (the age-based `upload_delay` fallback applies)
* Check `upload_delay` — the client waits this long after a file is last modified before uploading (default `30s`). See the [configuration reference](/docs/mesh-storage/reference/config) for the full list of sync settings
## My device was approved but now shows as disconnected [#my-device-was-approved-but-now-shows-as-disconnected]
**Last seen** stops updating when the client can't reach Alloy. Common causes:
* The client process crashed or was stopped
* Network connectivity to Alloy was interrupted
* The device's API key was rolled and the client hasn't picked up the new key yet (it should do so automatically on the next successful sync)
Check the client logs and ensure it can reach port 443.
## I uploaded a file from the web but I can't query it yet [#i-uploaded-a-file-from-the-web-but-i-cant-query-it-yet]
After upload, files go through several states before they're queryable:
1. **Queued** — the file has landed in object storage and is waiting to be processed
2. **Processing** — Alloy is parsing the MCAP and building queryable tables
3. **Ready** — the file is queryable in SQL Workbench and external Iceberg clients
4. **Failed** — something went wrong; check the row for the error
**Replay** and **Inspect** work as soon as the file is uploaded — you don't have to wait for processing. Only **SQL** requires the file to be **Ready**.
If a file gets stuck in **Processing** for an unusually long time, or shows **Failed**, click into the row for the error message.
## What ports does the client need? [#what-ports-does-the-client-need]
Outbound **port 443 (HTTPS)** only. No inbound ports are required. The client initiates all connections to Alloy — Alloy never connects back to the device.
## How do I rotate a device's API key? [#how-do-i-rotate-a-devices-api-key]
Open the `devices/` folder in Mesh Storage, open the actions menu for the device, and click **Roll Key**. Alloy revokes the old key and issues a new one. The client picks it up automatically on its next sync — no manual update needed.
## Diagnostics shows no data [#diagnostics-shows-no-data]
Diagnostics reads standard diagnostic topics from your MCAP files. If no data appears:
* Confirm your robot records system health data to a standard ROS diagnostic topic during missions
* Check that the MCAP file actually contains diagnostic messages (you can inspect it with `mcap info `)
* If the topic is present but under a non-standard name, it may not be recognized automatically
# Getting Started
Mesh Storage is the data lake at the heart of Alloy. Drop in MCAP recordings — from your laptop, a robot, or a fleet of devices — then browse, replay, and query everything from one place.
This page walks the canonical workflow in three steps. Pick the path through each step that matches your setup.
## Step 1 — Get data in [#step-1--get-data-in]
Mesh Storage accepts files through three primary paths:
Web uploads land in the **`uploads/`** folder. SDK uploads land in
**`uploads/sdk-uploads/`**. Alloy Edge uploads land in
**`devices//`**. All are queryable from the same place once processed.
A Docker deployment with a bundled ROS 2 recorder is also available from
**Mesh Storage → Add device**. Track Folder is the primary documented edge path.
Every file goes through the same lifecycle once it lands: **Queued → Processing → Ready** (or **Failed**). Replay and Inspect work as soon as a file lands — you don't have to wait for processing to finish.
## Step 2 — View the data [#step-2--view-the-data]
Once a file is in Mesh Storage you can browse, replay, and inspect it without leaving the browser.
**Open the file browser**
— the
[Mesh Storage page](/docs/mesh-storage/explore/browse)
lists every file in your data lake with a status badge, size, and last-modified time.
**Inspect**
any MCAP to see topics, schemas, message counts, and time range — useful for "what's actually in this recording?"
**Replay**
opens the 3D replay viewer with sensor data, robot poses, and other visualisable topics rendered directly in the browser.
For device folders, you also get
**ROS2 diagnostics**
and the
**ROS graph**
view to debug your robot's runtime configuration.
## Step 3 — Use the data [#step-3--use-the-data]
Once a file is **Ready**, pick the surface that matches where you want to work:
### From the browser — SQL Workbench [#from-the-browser--sql-workbench]
The in-app SQL editor runs DuckDB in your browser against the Iceberg tables. No setup, no credentials to manage. Best for ad-hoc analysis.
```sql
SELECT topic, count(*) AS messages
FROM "uploads"."my_recording__diagnostics"
GROUP BY topic
ORDER BY messages DESC
LIMIT 20;
```
### From your AI tool — MCP [#from-your-ai-tool--mcp]
Alloy ships an [MCP server](https://modelcontextprotocol.io/) so Claude, Cursor, Codex, Windsurf, or any MCP-aware tool can query missions, browse files, run SQL, and pull mission context into your workflow.
```bash
claude mcp add alloy --transport http https://aus.usealloy.ai/mcp
```
### From notebooks / BI / external compute — Iceberg REST [#from-notebooks--bi--external-compute--iceberg-rest]
Mesh Storage exposes the data lake as an Iceberg REST Catalog. Generate an API key from the Connect modal and point DuckDB, Spark, Trino, or PyIceberg at the endpoint — no copy step, no exports.
### From Python — Alloy SDK [#from-python--alloy-sdk]
Use `alloy-sdk` when you want Python code to upload files, list or download Mesh files, or query Ready data through hosted SQL.
```python
from alloy import storage, sql
with storage.connect() as store:
store.upload_folder("local/run-001", path="flights/run-001")
with sql.connect() as db:
rows = db.fetch("SELECT * FROM alloy.mesh.file_meta LIMIT 20")
```
## What's next? [#whats-next]
# Overview
Mesh Storage is the home for your data inside Alloy. Drop in MCAP recordings from the web, let your devices upload automatically, then browse, replay, or query everything from the same page — no separate tooling, no manual conversion step.
## What is Mesh Storage? [#what-is-mesh-storage]
Mesh Storage is a managed data lake for your organization. Files land in object storage, get indexed into queryable tables, and become available everywhere in Alloy — Replay, SQL Workbench, and any Iceberg-compatible client like DuckDB, Spark, or Trino.
You'll see two top-level folders when you first open it:
* **`uploads/`** — files you or a teammate upload directly. Web uploads land here, and Python SDK uploads land under **`uploads/sdk-uploads/`**.
* **`devices/`** — files uploaded automatically by your registered devices, organized by device
Both end up in the same data lake, queryable side by side.
## A tour of the page [#a-tour-of-the-page]
The Mesh Storage page has two parts: an **action bar** at the top and a **file browser** below.
The action bar exposes everything you can do at the org level:
* **Upload** — drop MCAP files in from your browser
* **Device Setup** — get the credentials and config snippets you need to connect a new device
* **SQL Workbench** — open the in-app query editor against your data lake
* **Connect** — generate an API key for external tools (DuckDB, Spark, Trino, Jupyter)
The file browser shows what's in the data lake — name, size, last modified, processing status, and a row menu for actions like Replay, Inspect, or Delete.
## How a file moves through Mesh Storage [#how-a-file-moves-through-mesh-storage]
Whether you upload from the web or a device pushes it up, every file goes through the same lifecycle:
1. **Queued** — the file has landed in object storage and is waiting to be processed
2. **Processing** — Alloy is parsing the MCAP and building queryable tables
3. **Ready** — the file is queryable in SQL Workbench and external Iceberg clients
4. **Failed** — something went wrong; check the row for the error
Replay and Inspect work as soon as the file lands — you don't have to wait for processing to finish to scrub through a recording.
## Next steps [#next-steps]
* [Getting data in](/docs/mesh-storage/ingest) — web uploads, Track Folder, or the Python SDK
* [Python SDK](/docs/mesh-storage/sdk) — upload files, query Ready data, and read Mesh files from Python
* [Managing your fleet](/docs/mesh-storage/manage) — key rotation, dashboard signals, and redaction
* [Exploring your data](/docs/mesh-storage/explore) — the file browser, SQL Workbench, external clients
* [Troubleshooting & FAQ](/docs/mesh-storage/faq) — common questions and fixes
# Core Concepts
Alloy structures your data around a few key concepts. Understanding these will help you get the most out of the platform.
***
# Missions [#missions]
A **mission** is a single data collection session — one flight, one survey run, one field test. When you upload log files to Alloy, they become a mission.
## What goes in [#what-goes-in]
Every mission starts with your raw inputs:
* **Data files** — sensor logs, telemetry recordings, camera feeds, and other data from your autonomous system. Alloy supports MCAP uploads by default; support for ROS bags, CSV, and other formats is available upon request.
* **Metadata** — attributes that describe the session and device: software version, firmware ID, serial number, test configuration, operator, and any custom fields specific to your setup
* **Description / field notes** — your context about what happened: goals, conditions, observations. Alloy's AI uses this to generate a more relevant report.
## What comes out [#what-comes-out]
Alloy processes these inputs and packages them into a single mission record containing:
* **AI-generated report** — a narrative summary with key events, anomaly detection, and findings
* **Key metrics** — distance, duration, and domain-specific measurements extracted from your data
* **Visualizations** — timeseries plots, maps, 3D trajectory views, and image galleries
* **Searchable instances** — every data point within the mission is indexed and searchable across your entire library
* **Structured metadata** — auto-extracted and user-provided attributes that power filtering and comparison
## How it connects [#how-it-connects]
Missions are the top-level unit of organization. Everything in Alloy connects back to missions:
* **[Search](/docs/agent-platform/navigating-alloy/search)** lets you find missions, reports, scenarios, and chats from anywhere in the platform — and Alloy AI can search across instances within missions
* **Reports** analyze one or many missions
* **Scenarios** scan across your mission library for patterns
Think of the [Missions page](/docs/agent-platform/navigating-alloy/missions) as the home base for your data — a searchable, filterable record of every test, scan, or session your systems have run.
***
# Instances [#instances]
An **instance** is a single moment in time of a robot's operation — think of it as a one- to few-second slice of what the robot was seeing, doing, and logging. It captures everything happening at that moment: the camera frames, the sensor readings, the log messages, the position, the state of the system.
A single mission might contain thousands of instances. Where a mission tells you *what happened overall*, instances let you zoom in on *what was happening right then*. They're the atomic unit of your data — the smallest piece you can search for, inspect, and reason about.
This is what makes Alloy's search powerful. When you ask "find all moments where the robot detected an obstacle" or "show me instances with GPS signal loss," you're searching across every instance from your entire history of missions.
The fastest way to find instances is through [Search](/docs/agent-platform/navigating-alloy/search) — open it with **Ctrl+K**, type what you're looking for, and press **Enter** to ask Alloy AI. Describe what you want in natural language, log patterns, or image descriptions, and AI searches across your entire mission library.
***
# Reports [#reports]
Alloy generates two kinds of reports, and they serve very different purposes.
## Mission summaries [#mission-summaries]
Every mission you upload automatically gets a **mission summary** — a report tied to that single session. It's generated during ingestion with no input from you, and it answers the question: *what happened during this mission?*
Mission summaries include a narrative overview, key events, metrics, visualizations, and anomaly highlights. They're scoped to one mission and one moment in time. Think of them as the automatic debrief.
## Custom reports [#custom-reports]
**Custom reports** are what you create through Alloy AI. They're fundamentally different from mission summaries because they can span any scope you need:
* **Longitudinal analysis** — compare performance across 50 missions over the last month
* **Cross-version comparison** — "compare software v2 with v1 across my last 2 sprints"
* **Cross-device comparison** — how does robot A perform vs. robot B on the same route?
* **Deep dives** — focus on a single metric or event type across your entire history
* **Trend analysis** — track how battery degradation progresses over weeks of operation
* **Targeted investigation** — "show me every mission where the IMU readings were anomalous"
You create them through conversation — describe what you want, and Alloy AI searches your data, runs analysis, and assembles the results. You can iterate until the report says exactly what you need.
Custom reports can contain narrative analysis, interactive charts, maps, metrics, timelines, and downloadable data files. They live on the [Reports page](/docs/agent-platform/navigating-alloy/reports) and are shareable by URL with your team.
## Recurring reports [#recurring-reports]
Set up a report prompt on a schedule — daily, weekly, or fortnightly — and Alloy generates it automatically. Useful for fleet performance summaries, anomaly digests, or any analysis you want refreshed regularly.
***
# Scenarios [#scenarios]
A **scenario** is a pattern scanner. You describe a pattern in plain language — through the chat — and Alloy continuously scans your mission library to find matches.
The prompt you provide is enriched by Alloy AI during the conversation. You might start with something simple like "find GPS issues" and the chat helps you refine it into a precise detection definition before the scanner starts running.
Examples:
* "Find all instances where battery voltage dropped below 11V during flight"
* "Flag any mission where the vehicle exceeded a 30-degree roll angle"
Scenarios run in the background — they scan existing missions and automatically check new ones as they're uploaded. Each scenario tracks its status, matches, and progress.
***
# Alloy AI [#alloy-ai]
Alloy AI is the conversational interface available throughout the platform. Use it to:
* Ask questions about your mission data
* Search for specific events or patterns
* Create and refine custom reports
* Set up scenarios
* Run analysis with SQL and Python
* Generate charts, maps, metrics, and timelines
Alloy AI has full context on your uploaded data, so you can ask natural-language questions and get answers grounded in your actual mission logs.
# AWS S3 Data Integration
Connect your AWS S3 bucket to Alloy for automatic data ingestion. Once set up, Alloy watches your bucket and ingests new files as they appear — no manual uploads needed.
**Setup time:** \~5 minutes
**Why S3 integration?**
* **Easy** — fast setup and simple to maintain
* **Secure** — uses permission-based IAM access and cloud security
* **Fast** — leverages cloud infrastructure for fast data transfer
* **Controlled** — you control the access point and what data is made available
## Prerequisites [#prerequisites]
* An AWS account with IAM permissions to create policies and roles
* An S3 bucket containing your mission data
## Step 1: Create an IAM Policy [#step-1-create-an-iam-policy]
In the AWS IAM console, go to
**Policies**
→
**Create policy**
Switch to the
**JSON**
editor and paste the following policy
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::",
"arn:aws:s3:::/*"
]
}
]
}
```
Replace `` with the name of your S3 bucket.
## Step 2: Create a Role [#step-2-create-a-role]
After the policy is created, head to the IAM
**Roles**
tab and create a new role
Attach the policy you created above
Click
**Create role**
Select trusted entity
**Web identity**
, with identity provider
**Google**
(
`accounts.google.com`
)
## Step 3: Update Trust Policy [#step-3-update-trust-policy]
In the IAM Roles console, select the role you created and click on the
**Trust Relationships**
tab
Click
**Edit Trust Policy**
, update with the following
The `accounts.google.com:sub` value below is for Alloy's standard `aus.usealloy.ai` environment. If your Alloy workspace uses a different domain or deployment, contact the Alloy team for the correct value before creating the trust policy.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "accounts.google.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"accounts.google.com:sub": "115295745068890142597"
}
}
}
]
}
```
Sometimes AWS will require the aud claim `accounts.google.com:aud` when first creating the trust policy. Make sure to remove this later.
## Step 4: Send us your details [#step-4-send-us-your-details]
Contact the Alloy team and provide:
1. **S3 bucket name** — the bucket you configured in the policy
2. **AWS IAM role ARN** — found on the role's summary page (e.g., `arn:aws:iam::123456789:role/alloy-access-role`)
We'll configure the connection on our end and your data will start flowing in automatically.
## Pull frequency [#pull-frequency]
Once your integration is set up, choose how often Alloy polls your S3 bucket from **Org settings → Integrations → Pull frequency**. Options: 1 min, 5 min, 15 min, 30 min, 1 hour. Default is 5 min.
On S3, polling cost is paid by **you** (the bucket owner), via S3 LIST API charges (\~$0.005 per 1,000 requests). Each poll is a full bucket walk paginated at 1,000 keys per LIST. Rough monthly cost at 5-min polling:
* **100K objects** → \~$0.43/month
* **1M objects** → \~$4.32/month
* **10M objects** → \~$43/month
Cost scales linearly with object count and inversely with poll interval — 1-min polling is 5× the cost of 5-min, 1-hour is 12× cheaper. Pick the slowest cadence that meets your latency needs.
## Folder structure and supported files [#folder-structure-and-supported-files]
See [Uploading your first mission](/docs/agent-platform/getting-started#folder-structure) for details on folder structure expectations, `metadata.json` format, and supported file types. These apply to all data ingestion methods including S3.
# Google Cloud Storage Integration
Connect your Google Cloud Storage bucket to Alloy for automatic data ingestion. Alloy uses a GCS Transfer Job to pull data from your bucket into our ingestion pipeline.
## How it works [#how-it-works]
Alloy sets up a [Storage Transfer Service](https://cloud.google.com/storage-transfer-service) job that periodically syncs data from your GCS bucket into Alloy's ingestion bucket. Once files land, they're automatically processed into missions.
## Setup [#setup]
**Share bucket access**
— grant read access on your GCS bucket to the Alloy service account (we'll provide the service account email)
**Contact the Alloy team**
— provide your GCS bucket name
**We configure the transfer job**
— Alloy creates a Storage Transfer Job that pulls new files from your bucket
The transfer job only copies new or changed files on each run — it won't re-process data that's already been ingested.
## What to provide [#what-to-provide]
Contact the Alloy team with:
1. **GCS bucket name** — e.g. `gs://my-org-mission-data`
2. **Bucket region** — for optimal transfer performance
We'll send you the service account email to grant access to, and handle the rest.
## Pull frequency [#pull-frequency]
Once your integration is set up, choose how often Alloy polls your bucket from **Org settings → Integrations → Pull frequency**. Options: 1 min, 5 min, 15 min, 30 min, 1 hour. Default is 5 min.
GCS LIST requests are very cheap (\~$0.04 per million requests), so pull frequency on GCS has negligible cost impact even on large buckets. Pick whatever latency you need.
## Folder structure and supported files [#folder-structure-and-supported-files]
See [Uploading your first mission](/docs/agent-platform/getting-started#folder-structure) for details on folder structure expectations, `metadata.json` format, and supported file types. These apply to all data ingestion methods including GCS.
# Integrations
Alloy integrates with the tools and infrastructure you already use — ingest data from cloud storage, bring mission insights into Slack, or connect AI coding tools directly to your data via MCP.
## Data ingestion [#data-ingestion]
## Tools [#tools]
# Slack
Alloy's Slack bot brings the full power of Alloy AI into your workspace 
. It has the same capabilities as the in-app chat — search missions, run analysis, create reports, and more — all without leaving Slack.
## Setting up [#setting-up]
In Alloy, open the **Integrations** panel for the organization you want to connect.
Click
**Connect Slack**
. Alloy will start the setup for the correct region and organization.
Authorize the app in Slack
Once connected, mention
**@Alloy**
in any channel or send it a direct message
## How to use it [#how-to-use-it]
Mention **@Alloy** in any channel with your question:
* "@Alloy what happened in our latest mission?"
* "@Alloy find all GPS signal loss events"
* "@Alloy create a report on this week's flights"
The bot responds in a thread, keeping the channel clean. Continue the conversation in the same thread — Alloy remembers the context.
Send a direct message to the Alloy bot — no `@` mention needed:
* "Show me my most recent missions"
* "Compare battery performance across firmware versions"
* "Set up a scenario for motor fault events"
## What it can do [#what-it-can-do]
The Slack bot has **all the same capabilities** as the in-app chat:
* **Search** your mission data with natural language, log patterns, or image descriptions
* **Analyze** data with SQL queries and Python code
* **Create reports** across one or many missions
* **Set up scenarios** to track patterns
* **View images and charts** — mission images and generated charts are shared directly in the thread, no need to open the app
* **Fetch metrics and summaries** for any mission
## Thread conversations [#thread-conversations]
The bot is thread-aware. When you reply in a thread, Alloy sees the full conversation history and responds in context:
1. "@Alloy show me missions from last week"
2. "@Alloy which one had the longest duration?"
3. "@Alloy create a report comparing that one to the previous week's longest"
Each reply builds on the previous context — no need to repeat yourself.
If you add a follow-up while Alloy is still working, you do not need another
**@Alloy** mention. After Alloy replies, mention **@Alloy** again to start a new
request in a normal channel. Messages sent while Alloy is already working are
best-effort updates to that request; they do not start a later request, even if
they mention **@Alloy**.
## Help-desk routes [#help-desk-routes]
Admins can configure selected Slack channels as help-desk routes. Alloy records visible messages in those channels as service cases, then uses the route's reply mode to decide when to post back:
* **Replies off** records cases without waking the bot.
* **Mentioned only** records all visible messages and replies when Alloy is mentioned.
* **Every message** records all visible messages and replies to each new message.
Mentioned-only routes can also have wake triggers for specific operational messages, such as an assignment or escalation. Triggered replies use the same full Alloy Slack agent as regular mentions.
## Progress indicators [#progress-indicators]
While the bot is working, you'll see emoji reactions on your message:
| Reaction | Meaning |
| -------- | ------------------------ |
| 👀 | Processing your request |
| ⚙️ | Running tools or queries |
| ✅ | Done |
If a request fails, Alloy removes its working state without posting an error in
the thread. The failure alerts the Alloy operations team.
## Troubleshooting [#troubleshooting]
### "Start Slack setup from Alloy" [#start-slack-setup-from-alloy]
Slack setup must be started from Alloy so the authorization request is securely bound to your organization. If you see this message, you followed a direct Slack authorization link. Return to Alloy, open the **Integrations** panel, and click **Connect Slack**.
### "This Slack workspace is already linked to a different organization" [#this-slack-workspace-is-already-linked-to-a-different-organization]
Disconnect the workspace from the other Alloy organization first, then start setup again from the **Integrations** panel. Reconnecting the same workspace to the organization it is already linked to is safe — it just refreshes the connection.
# Home
The Home page is where you land when you open Alloy. It's built around Alloy AI — your mission data analyst that lives inside the platform.
## Alloy AI [#alloy-ai]
Alloy AI is an AI agent that works within the Alloy platform. You can think of it as the data analyst or engineering assistant that lives inside your project. It's here to help you find answers and get things done fast.
Unlike general-purpose AI, Alloy AI is deeply connected to your mission data. It can reference your entire history of operational data, existing data analysis, and perform actions across the platform. Describe what you want in natural language and it can:
* **Query and analyze data** — search across missions, run SQL queries, execute Python scripts with pandas/numpy/scikit-learn, and compute statistics
* **Build visualizations** — create interactive timeseries plots, bar charts, scatter plots, trajectories, and maps
* **Reason across data types** — correlate logs, images, timeseries, and metadata to answer complex questions that span multiple sensors and missions
* **Create scenarios and reports** — define patterns to watch for, generate AI-powered reports, and schedule recurring analyses
* **Summarize and explain** — narrate what happened in a mission, surface anomalies, and break down performance metrics
You can be as broad or as specific as you like. Ask a high-level question like "How did last week's missions compare to the week before?" and Alloy AI will figure out what to query. Or get precise — "Show me all instances where motor temperature exceeded 80°C while heading was between 90° and 180°" — and it will build the exact analysis. For deep research tasks, Alloy AI can spin up multiple sub-agents that work in parallel — searching across missions, analyzing different data types, and synthesizing findings into a single answer.
The chat is available from the Home page and from the right edge of every screen — it's context-aware and knows what page you're on and what data you're viewing.
## What you'll see [#what-youll-see]
**Chat box** — front and center. Start typing a question or pick one of the suggested actions like "What's new in my missions?", "Search across my mission data", or "Create a custom report."
**Capability cards** — a visual showcase of what Alloy AI can do:
**Recent sessions** — your last few chat conversations, so you can pick up where you left off. Click any session to reopen it with the full history intact. Sessions also appear as nested items under Home in the sidebar.
## Memory [#memory]
Alloy AI remembers things across chat sessions. This is useful for:
* **Format preferences** — "Always include a chart when you talk about trends"
* **Testing set-up** — "Missions from Sep 20–30 used a temporary LiDAR mount on the front leg"
* **Frequently referenced context** — "Our standard survey route covers waypoints A through F"
Tell Alloy to "remember that..." and it saves the context. Memories can be scoped to just you or shared across your organization.
# Navigating Alloy
Alloy's sidebar gives you quick access to every part of the platform. Here's a brief overview — click through to each page for details.
* **Devices** — Register and manage your hardware fleet. Configure edge devices, generate provisioning keys, and view diagnostics.
*
The metadata filter buttons at the top of the page reflect the actual attributes in your data — so you'll see filters relevant to your setup. Click a filter to expand it, then select one or more values.
Combine multiple filters to narrow down precisely. Active filters are preserved in the URL, so you can bookmark or share filtered views with your team.
Sort by date (default), duration, or any mission-level metric. The sort dropdown updates dynamically based on the metrics available in your data.
## The Mission Report [#the-mission-report]
Click **View report** on any mission card to open its full AI-generated analysis. Here's what you'll find inside:
1. **Summary** — a narrative overview of what happened, with key metadata values (device IDs, locations, firmware versions)
2. **Key events** — a chronological timeline narrating the important moments: anomalies, state changes, and notable events, each with a timestamp, severity level, expandable detail, and associated images
3. **Metrics** — a grid of key performance indicators extracted from the data (distance, speed, depth, battery, or any domain-specific measurements)
4. **Timeseries plots** — interactive charts showing how values changed over the mission, with zoom, pan, hover tooltips, and key event markers overlaid
5. **Map** — a geographic visualization of the mission route with color-coded paths, clickable to set reference time across other visualizations
6. **3D trajectory viewer** — for missions with 3D position data, an interactive viewer you can rotate, zoom, and pan
7. **Image gallery** — a timeline-based view of camera feeds with multi-camera support and full-resolution expansion
8. **Available data** — a list of data topics from your original files used in this mission
All time-based components (plots, map, 3D trajectory, image gallery) are synchronized — clicking around in one updates them all.
### Mission name and description [#mission-name-and-description]
Both the mission **name** and **description** are editable. Click the mission name in the report header to rename it inline. For the description, hover over the description area and click **Edit** to modify it, or click **Add a description...** if one doesn't exist yet. Descriptions are searchable and visible to your team.
### More actions menu [#more-actions-menu]
The **⋯** (more actions) button in the report header — next to **Ask AI** — opens a menu with everything you can do to a mission beyond editing its name and description:
* **Download All Visualizations** — pulls every image, map, and chart from the mission page as a single bundle
* **Download Original Data** — re-downloads the source files you uploaded (MCAPs, logs, attachments)
* **Share link** — copies a link to this mission; anyone in your org with access can open it
* **Edit time format** — switch how timestamps are displayed across the report (e.g. local vs. UTC, absolute vs. mission-relative)
* **Sync key events, maps, and plots** — when on, each key event shows up as a small white bubble on the chart axes at its timestamp, and clicking around in one time-based component (key event, plot point, map location) updates the others. On by default
* **Delete Mission** — soft-deletes the mission so it stops appearing in your library. The summary, key events, plots, instances, and derived artifacts are hidden but recoverable; contact support if you need a deleted mission restored. The original uploaded files in Mesh Storage are unaffected — remove those separately from **Browse** if you also want the source data gone
### Comments & attachments [#comments--attachments]
Each mission has a discussion section where you and your team can attach files, leave notes, observations, and follow-up items. Alloy AI reads these too — so comments you leave become part of the context when you ask questions about a mission.
You can also ask Alloy AI to write comments for you — useful for bulk-commenting across multiple missions or having it note down interesting observations it finds during analysis.
## Replay Viewer [#replay-viewer]
Click **Replay** in the mission header to play back raw sensor recordings from your mission.
The replay viewer lets you scrub through the timeline and inspect sensor readings and other recorded topics at any point in time. You can rearrange panels, choose which data topics to display, and customize the layout to focus on what matters to you.
### Starting your first layout [#starting-your-first-layout]
If your organization has no saved Replay layout yet, choose how to begin before Alloy opens the recording:
* **Chat Layout** — use a layout Alloy has already prepared in the mission chat.
* **Auto Layout** — ask Alloy to arrange useful panels for the recording. The first successful Auto Layout is saved as **Overview**, so Replay can reopen with a useful starting point next time.
* **Empty View** — start with a blank workspace and add only the panels you need.
Alloy does not load the recording until you make one of these choices. This keeps first-time Replay setup fast and avoids opening a large recording just to show an empty workspace.
### Saving and reusing layouts [#saving-and-reusing-layouts]
Use the **Layout** picker in the Replay toolbar to switch between your organization's saved layouts. You can save the arrangement you are editing under a descriptive name, then reuse it across recordings. Saved layouts are shared with your organization, so use clear names that communicate their purpose.
You can also delete a saved layout from the picker when it is no longer useful. Auto Layout remains available in the toolbar after you have chosen a starting layout; its changes are temporary until you save them.
## Uploading new missions [#uploading-new-missions]
Click **+ Add** to upload data. See [Getting Started](/docs/agent-platform/getting-started) for all upload methods including browser upload, cloud storage auto-ingestion, and edge device uploads.
# Notifications
Alloy can notify you by email when important events happen — like when a mission finishes processing. You control which notifications you receive, and you can manage your preferences at any time.
## Notification bell [#notification-bell]
The **bell icon** in the sidebar shows your most recent notifications. Click it to open a dropdown with up to 20 recent items.
Each notification shows:
* The mission name (or names, if multiple were batched together)
* A brief label (e.g. "Mission")
* A relative timestamp (e.g. "5m ago", "2h ago")
Click a notification to expand it and see more detail.
The notification bell shows notifications for your entire organization — all team members see the same history.
## Managing your preferences [#managing-your-preferences]
Notification preferences are per-user — each person in your organization controls their own settings.
### How to access [#how-to-access]
Click your
**organization name**
at the bottom of the sidebar
In the management panel, click the
**Notifications**
tab
Toggle notifications on or off for each event type
### Available notification types [#available-notification-types]
| Event | Description |
| ---------------------- | --------------------------------------------------------------------------------------- |
| **My browser uploads** | Get an email when a mission you upload to Alloy through the browser finishes processing |
Use the toggle next to each event type to enable or disable email notifications. Changes take effect immediately.
The browser upload email setting is enabled by default and only applies to missions you upload in Alloy through the browser. Device, cloud integration, API, and legacy missions do not send this email.
## Slack notifications [#slack-notifications]
If your organization has the [Slack integration](/docs/agent-platform/integrations/slack) connected, Alloy can post notifications directly into a Slack channel. Unlike email preferences, Slack notifications are **org-wide** — everyone in the channel sees the same alerts.
### Available Slack notification types [#available-slack-notification-types]
| Event | Description |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| **New mission uploaded** | A new mission is uploaded to the platform — whether via the app or from an S3/GCS bucket you've integrated with Alloy |
| **Early access ready** | Mission data is available for querying (before the full report) |
| **Mission ready** | A mission finishes processing and is ready to review |
| **Triggered report ready** | A mission-triggered prompt finishes generating a report; Alloy posts it in the mission's Slack thread |
| **Scenario match found** | One of your scenarios detects a pattern match on a mission |
| **Scheduled report ready** | One of your scheduled prompts finishes generating a report |
### How to configure [#how-to-configure]
Most Slack notifications are set up from within Slack. Go to the Slack channel where you want to receive the alerts and ping **@Alloy** with what you want. For example:
* "@Alloy notify this channel when missions are ready"
* "@Alloy DM me when a scenario match is found"
* "@Alloy post here when a new mission is uploaded"
Once configured, the notification will appear in the Notifications tab with a toggle so anyone in your org can pause or resume it. Unconfigured types show as *Not configured* until you ask @Alloy to set them up.
Triggered report notifications are shown under **Mission ready** in the Slack section. After Mission ready has a Slack channel, you can enable or disable **Triggered report ready** directly from the Notifications tab. Triggered reports reply in the relevant mission thread; if there is no Mission ready Slack message for that mission yet, the first triggered report starts the thread and later triggered reports for that mission reply there. When Slack report notifications are re-enabled, Alloy only posts reports generated after they were turned back on.
Can't find Alloy in your Slack workspace? Your org may not have connected Slack yet — see [Slack integration](/docs/agent-platform/integrations/slack) to get started.
# Reports
The Reports page is where you create and manage AI-generated reports. Use it to ask Alloy AI to analyze your mission data and produce shareable, structured documents.
Reports are created through conversation — describe what you want to know, and Alloy AI searches your data, runs analysis, and assembles the results into a report. You can iterate on the output through follow-up messages until it says what you need.
## What reports can contain [#what-reports-can-contain]
* Narrative text and analysis
* Interactive charts and visualizations
* Maps with geographic data
* Metrics displays with trends
* Event timelines
* Downloadable data files
## Creating a report [#creating-a-report]
Open Alloy AI from anywhere and ask for a report, or start from the Home page chat. See [Creating a Report](/docs/agent-platform/workflows/creating-a-report) for a full walkthrough.
## Recurring reports [#recurring-reports]
You can set up reports that run automatically on a schedule — daily, weekly, or fortnightly. Useful for fleet performance summaries, anomaly digests, or ongoing trend analysis.
Click
**+ Add**
in the Recurring reports section at the top of the page
Describe what the report should cover
Set the frequency: daily, weekly, or fortnightly
Choose a start date and time
Click **Manage** to view, edit, pause, or delete your scheduled reports.
## Managing reports [#managing-reports]
* **Filter** — narrow reports by type
* **Sort** — by newest or oldest
* **View** — click any report to see its full content
* **Delete** — remove reports via the menu on each card
Reports are shareable by URL — anyone in your organization can view them.
# Scenarios
Scenarios are pattern scanners that continuously watch your mission library. Define a pattern in plain language, and Alloy scans every mission — past and future — to find matches.
## When to use scenarios [#when-to-use-scenarios]
* **Track recurring issues** — "motor fault events at high altitude"
* **Monitor for specific conditions** — "battery voltage below 11V during flight"
* **Build a dataset** — collect all instances of a particular event type across your fleet
* **Ongoing awareness** — know how often something happens without manually searching
## Creating a scenario [#creating-a-scenario]
Go to
**Scenarios**
in the sidebar
Click
**+ Create**
Describe the pattern you want to track in natural language — e.g., "Find instances where GPS accuracy dropped below 2 meters"
Alloy defines the detection pattern and starts scanning
Ask the chat to create one:
* "Create a scenario to find all emergency stop events"
* "Set up a scanner for navigation errors near waypoints"
* "Watch for instances where roll angle exceeded 30 degrees"
Alloy AI defines the pattern and starts scanning immediately.
## The scenario list [#the-scenario-list]
The Scenarios page shows all your scenarios in a table:
| Column | What it shows |
| ----------- | ------------------------------------------------- |
| **Name** | What the scenario is tracking |
| **Status** | Running, Paused, or progress ("Scanned 19 of 39") |
| **Matches** | Total instances found so far |
| **Created** | When the scenario was set up |
## How scanning works [#how-scanning-works]
When you create a scenario, Alloy starts working through your mission library in the background.
1. The scanner processes each mission, looking for instances that match your pattern
2. Progress shows as "Scanned X of Y" — where Y is your total mission count
3. New missions uploaded after the scenario was created are automatically scanned too
4. You don't need to re-run or restart anything — scenarios stay active until you pause or archive them
Scanning is non-blocking — you can keep using Alloy normally while scenarios run in the background. Large libraries may take some time to fully scan.
## Viewing matches and auto-bucketing [#viewing-matches-and-auto-bucketing]
Click a scenario to see its matches. Alloy automatically organizes what it finds:
### How matches are grouped [#how-matches-are-grouped]
Each match represents a cluster of related instances — not just a single data point. The AI groups instances that belong to the same event or condition together, so you see coherent results rather than individual fragments.
Every match includes:
* **Match reason** — an AI-generated explanation of why these instances matched
* **Time window** — when the matched event occurred
* **Mission** — which mission the match came from
* **Instance count** — how many data points are in this group
* **Hero image** — the most relevant image from the match (if images are involved)
### Filtering and sorting matches [#filtering-and-sorting-matches]
Alloy automatically aggregates metadata across all matches, so you can filter by any attribute:
* Filter by device, location, date, or any custom metadata field
* Combine filters to narrow down to specific conditions
* Sort by event time, detection date, upload timestamp, mission name, or metadata values
This auto-bucketing means you don't need to manually organize results. Alloy surfaces the metadata structure from your matches so you can slice the data however you need.
For example, if your scenario finds 50 matches across 20 missions, you can instantly filter to see only matches from a specific device, or sort by time to see when the pattern occurs most frequently.
## Editing a scenario [#editing-a-scenario]
Both the scenario **name** and **definition** are editable.
* **Name** — click the pencil icon next to the scenario name to rename it inline
* **Definition** — click **Edit** on the pattern definition card to open the AI chat. Describe how you want to change the pattern and Alloy AI will rewrite the definition for you. Depending on the change, Alloy may re-scan your missions to find new matches
The scenario definition is authored by Alloy AI when you first create it — the AI runs a preview search to find example matches and then writes a clear, specific description of the pattern. Editing works the same way: you describe what to change and the AI updates the definition, keeping it precise and well-structured.
## Managing scenarios [#managing-scenarios]
* **Pause** — stop scanning without losing progress or results. Resume at any time.
* **Resume** — pick up scanning where you left off
* **Archive** — remove from the active list while preserving all historical results
* **Unarchive** — bring back an archived scenario and resume scanning
# Search
Alloy has a unified search that lets you find anything across the platform — or hand your question straight to Alloy AI — without leaving the page you're on.
## Opening search [#opening-search]
Open the search dialog from anywhere in Alloy:
* Click the **Search** button at the top of the sidebar
* Press **Ctrl+K** (or **Cmd+K** on Mac)
A search overlay appears on top of whatever page you're on. Press **Esc** to close it.
## How it works [#how-it-works]
Type what you're looking for and press **Enter** to immediately ask Alloy AI. This is the fastest way to search your data — whether you're looking for specific missions, hunting for log patterns, finding images that match a description, or exploring a trend across your fleet.
For example:
* "instances with red cars" — AI searches across all your mission data for matching images
* "battery issues during outdoor missions" — AI finds and summarizes relevant events
* "GPS accuracy trends over the last month" — AI runs the analysis and shows you results
As you type, the search dialog also shows **instant results** from across the platform, grouped by type:
* **Missions** — matches against mission names, descriptions, summaries, key events, and metadata
* **Reports** — matches against report titles and content
* **Scenarios** — matches against scenario names and definitions
* **Chats** — matches against your Alloy AI conversation history
If one of these results is exactly what you're looking for, click it to jump straight there. Otherwise, just press **Enter** (or select the **Ask AI** option at the top) to hand your query to Alloy AI and get a full answer.
## Filtering by type [#filtering-by-type]
If you want to narrow the instant results to a single category, start typing a category name (like "missions" or "reports") and press **Tab** to activate the filter. A badge appears in the search bar showing the active filter. Press **Backspace** on an empty search to remove it.
## Searching your mission data with AI [#searching-your-mission-data-with-ai]
Search is the quickest entry point for asking Alloy AI to find things in your raw mission data. Type a natural-language query and press **Enter** — Alloy AI will search across all [instances](/docs/agent-platform/core-concepts#instances) in your mission library, including log messages, images, sensor readings, and metadata.
This replaces the need for separate log-pattern or image-description searches. Just describe what you're looking for:
* "find all moments where the robot detected an obstacle"
* "show me instances with GPS signal loss"
* "images of rocky coastline from last week's surveys"
Alloy AI can reason across multiple data types at once — correlating logs, images, timeseries, and metadata to answer complex questions. See [Using the Chat](/docs/agent-platform/workflows/using-the-chat) for the full range of what Alloy AI can do.
Found a pattern you want to track continuously? Ask Alloy AI to create a [Scenario](/docs/agent-platform/navigating-alloy/scenarios) from your search. It will scan all existing missions and watch for the same pattern in future uploads.
# Creating a Report
Every mission gets an auto-generated report. But you can also create **custom reports** through Alloy AI — reports that pull from multiple missions, focus on specific topics, or answer questions unique to your use case.
## Starting a report [#starting-a-report]
Go to
**Home**
in the sidebar
Click
**"Create a custom report"**
or type your request in the chat
Describe what you want the report to cover
Open the chat from anywhere — click **Alloy AI** on the right edge of any page, or use the Slack bot — and ask for a report. The chat has full context on your data regardless of where you start.
## Example prompts [#example-prompts]
* "Create a report summarizing performance across my last 10 missions"
* "Compare battery drain rates between firmware v2.2 and v2.3"
* "Analyze all campus survey missions for anomalies"
* "Build a cross-device comparison for missions from this week"
## What reports can contain [#what-reports-can-contain]
Custom reports can include a mix of:
* **Narrative analysis** — AI-written text explaining findings
* **Charts** — interactive line, bar, or scatter plots
* **Maps** — geographic visualizations with mission paths
* **Metrics** — key stats with trends (up/down/neutral)
* **Timelines** — chronological event sequences
* **Images** — galleries from mission data
* **Downloadable files** — CSV exports or processed data
## Refining your report [#refining-your-report]
After Alloy AI generates a report, iterate through conversation:
* "Add a section comparing latency across devices"
* "Focus more on the GPS signal loss events"
* "Include a chart showing altitude over time"
* "Remove the battery section and expand on navigation"
Go back and forth as many times as you need. The report updates with each request.
## Recurring reports [#recurring-reports]
Go to
**Reports**
and click
**+ Add**
in the Recurring reports section
Define what the report should cover
Set frequency: daily, weekly, or fortnightly
Choose a start date
Alloy generates the report automatically on schedule and adds it to your Reports page.
**Tip**: Be specific in your request — "compare battery performance for missions uploaded in the last week" gets better results than "make a report about batteries". Start broad, then iterate to refine.
# Using the Chat
Alloy AI is a conversational interface available throughout the platform. Ask questions, run analysis, build visualizations, and generate reports — all in natural language.
## Where to find it [#where-to-find-it]
* **Home page** — the chat is front and center
* **Chat panel** — click the **Alloy AI** button on the right edge of any page
* **"Ask AI" buttons** — quick-launch buttons on mission summaries and other pages, plus the **Ask AI** option at the top of [Search](/docs/agent-platform/navigating-alloy/search) results
The chat is **context-aware** — it knows which page you're on and what you're looking at. Ask a question on the Missions page and it references your current filters. Ask on a report and it can modify that report.
## What you can do [#what-you-can-do]
Find data across your mission library:
* "Find moments where two robots were at the same waypoint but reported different terrain classifications"
* "Can you find any joint temperature spikes that correlate with the gait stumbles?"
* "Find all perimeter patrols that completed without intervention and summarize the stats"
* "Search for images where the robot encountered standing water"
Supports text search, log pattern matching, and **image search** — describe what you're looking for visually.
Run analysis on your data, including SQL queries and Python code:
* "Walking speed should drop from 1.2m/s to 0.3m/s when the robot enters a confined space — find cases where that transition didn't happen"
* "Describe path-planning failures and how they lead to missed waypoints on the inspection route"
* "Track CPU, GPU, RAM, and joint motor temperatures per robot per mission — identify spikes and how they trend week to week"
Behind the scenes, Alloy can query your data warehouse and run Python with pandas, numpy, matplotlib, geopandas, scikit-learn, and more — all in a secure sandbox.
Create interactive visualizations for reports:
* "Plot where the leg servo errors begin and how they accumulate over the patrol"
* "Show me a map of all inspection routes overlaid on the facility layout"
* "Build a weekly comparison dashboard of system health metrics per robot by serial number"
* "Make a timeline of key events during this inspection run"
Visualizations can be embedded in reports for sharing.
Generate and refine custom reports:
* "That stair-climb failure you found for QP-07 during the warehouse inspection — make a report about what happened"
* "Create a report on how joint calibration drift builds up across back-to-back patrols"
* "Remove the battery section and focus on navigation"
See [Creating a Report](/docs/agent-platform/workflows/creating-a-report) for more.
Set up pattern scanners:
* "Create a scenario to flag any patrol where battery drops below 20% before returning to dock"
* "Set up a scanner for leg motor fault events"
See [Scenarios](/docs/agent-platform/navigating-alloy/scenarios) for more.
Open the [Replay Viewer](/docs/agent-platform/navigating-alloy/missions#replay-viewer) at exactly the moment you're discussing — with the right panels already laid out. Be as vague or as specific as you want:
* "Show me the altitude drop around 14:32 on yesterday's patrol"
* "Open the replay where the obstacle avoidance kicked in"
* "Show me battery and motor-current time-series with the map on top and camera5 in the bottom left"
Alloy AI picks the panels — 3D scene, map, time-series, text log — arranges them, and seeks every stream (video, telemetry, map cursor) to the exact instant in question.
**Save the layouts you want to reuse.** Once a layout works for the question you're asking, save it — next time you ask something similar, the agent will reopen it instead of rebuilding from scratch.
## Tips [#tips]
* **Iterate** — start broad, then refine with follow-ups
* **Use the context** — on a report, ask it to modify sections; from [Search](/docs/agent-platform/navigating-alloy/search), select **Ask AI** to explore a topic with AI
* **Highlight text** — select text on the Home page, Chat, Missions, or Reports and click "Ask about this" to include it as context
# Browse
Mesh Storage shows everything your org has uploaded — web uploads, device uploads, and the queryable tables built from them — in one file browser.
{/* TODO: reshoot — Mesh Storage file browser landing view */}
## Folder layout [#folder-layout]
You'll see a **`devices/`** folder at the top of the browser, alongside any files you've dropped in via the [web uploader](/docs/mesh-storage/ingest/upload). Web uploads sit at the root; everything pushed up by a registered device lives inside `devices//`.
Click into a folder to see what's inside. Folders show their aggregate size and a count of contained files.
## Status badges [#status-badges]
Every file shows a status badge that tells you what state it's in:
| Badge | What it means |
| -------------- | ----------------------------------------------------------------------------------------------------------- |
| **Queued** | The file has landed in object storage and is waiting to be processed |
| **Processing** | Alloy is parsing the MCAP and building queryable tables |
| **Ready** | The file is queryable in [SQL Workbench](/docs/mesh-storage/explore/workbench) and external Iceberg clients |
| **Failed** | Something went wrong; check the row for the error |
[Replay](#replay) and [Inspect](#inspect) work as soon as a file lands — you don't have to wait for processing to finish.
## Row actions [#row-actions]
Click the actions menu on any file row:
* **Replay** — open the file in the 3D replay viewer
* **Inspect** — view topics, schemas, message counts, and time range
* **Download** — generate a signed download URL (valid for 7 days)
* **Delete** — remove the file from object storage and Iceberg tables
For device folders, you also get **Settings** — open the device-specific configuration.
## Inspect [#inspect]
Click **Inspect** on any MCAP file to see what's inside without downloading it: topics, message counts, schemas, and time range. Useful for quickly checking what data a recording contains.
{/* TODO: reshoot — Mesh Storage MCAP Inspect modal */}
## Replay [#replay]
Click **Replay** on any MCAP file to open it in the 3D replay viewer. The viewer renders sensor data, robot poses, and other visualisable topics directly in the browser — no local tooling required.
{/* TODO: reshoot — Mesh Storage replay modal */}
This is the same replay viewer used for mission recordings. Add 3D, map, text-log, and time-series panels in any arrangement, then **save the layout** to reuse it on later files. From a mission, you can also ask Alloy AI to open Replay seeked to a specific moment with the right panels already laid out — see [Using the Chat → Replay](/docs/agent-platform/workflows/using-the-chat#what-you-can-do).
## Browsing a single device's files [#browsing-a-single-devices-files]
Click into `devices//` to see only that device's uploads. The folder header shows device-specific signals — hostname, status (Pending / Approved / Rejected), last seen, total files and size.
{/* TODO: reshoot — Mesh Storage devices/ folder view */}
From here you can also approve a pending device, roll its API key, or open device-level diagnostics and graph views.
## ROS2 diagnostics [#ros2-diagnostics]
In a device folder, click **Diagnostics** to view health metrics extracted from MCAP files uploaded by that device.
{/* TODO: reshoot — diagnostics view from inside a Mesh Storage device folder */}
Diagnostic levels (OK, WARN, ERROR, STALE) are plotted as a timeline so you can correlate device health with mission events. The bottom panel shows individual diagnostic messages with details.
Diagnostics requires that your robot records system health data to a standard ROS2 diagnostic topic during the mission. If the topic is absent, the diagnostics view shows no data.
## ROS graph [#ros-graph]
Click **ROS Graph** on a recording to visualise the node and topic graph. This shows which ROS nodes were active, what topics they published and subscribed to, and how data flowed through the system.
{/* TODO: reshoot — ROS graph view from Mesh Storage */}
Use the **Filters** panel to toggle system topics and Alloy infrastructure nodes. The graph shows node counts, topic counts, and connection counts at the bottom.
Useful for verifying your robot's configuration matches expectations — especially when debugging missing topics or unexpected node behaviour.
# Connect external tools
Mesh Storage exposes your data lake as an [Iceberg REST Catalog](https://iceberg.apache.org/concepts/catalog/). Any client that speaks Iceberg REST can query it directly — no copy step, no exports, no separate database.
This is how you connect notebooks, BI dashboards, CI pipelines, and external compute to your Alloy data.
## Generate an API key [#generate-an-api-key]
From the Mesh Storage data page, click **API keys**.
{/* TODO: screenshot — API keys button in the Mesh Storage data action bar */}
In the modal:
1. Enter a **name** describing where the key will be used — e.g. `Jupyter notebook`, `CI pipeline`, `local dev`. This makes keys easier to audit and revoke later.
2. Click **Create**.
3. Copy the key from the green box.
**Copy this key now — it won't be shown again.** If you lose it, generate a new one.
{/* TODO: screenshot — API keys modal with the new key revealed */}
## Get the catalog endpoint [#get-the-catalog-endpoint]
The Connect modal also shows the Iceberg REST catalog gateway URL. Copy it — you'll point your client at this URL.
The endpoint serves the standard Iceberg REST API. It works with any compatible client.
## Compatible clients [#compatible-clients]
| Client | Connection notes |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **DuckDB** | Use the [Iceberg extension](https://duckdb.org/docs/extensions/iceberg) — `ATTACH '' AS alloy (TYPE iceberg, TOKEN '', ENDPOINT '')` |
| **Apache Spark** | Configure the [Iceberg REST catalog](https://iceberg.apache.org/docs/latest/spark-configuration/) — point `spark.sql.catalog..uri` at the endpoint |
| **Trino** | Use the [Iceberg connector](https://trino.io/docs/current/connector/iceberg.html) with `iceberg.catalog.type=rest` |
| **PyIceberg** | `pyiceberg.catalog.load_catalog(name, type="rest", uri=...)` |
| **Any Iceberg REST client** | Most clients support OAuth-style bearer tokens — pass your API key as the `Authorization: Bearer ` header |
## Quick test (DuckDB) [#quick-test-duckdb]
```bash
# install duckdb + iceberg extension
duckdb
```
```sql
INSTALL iceberg;
LOAD iceberg;
ATTACH '' AS alloy (
TYPE iceberg,
TOKEN '',
ENDPOINT 'https://'
);
SHOW ALL TABLES;
SELECT count(*) FROM alloy.fleet.;
```
## List available tables [#list-available-tables]
After connecting, run `SHOW ALL TABLES;` to list the available namespaces, tables, and columns in your Alloy catalog.
```sql
SHOW ALL TABLES;
```
## Connect via MCP (for AI agents) [#connect-via-mcp-for-ai-agents]
If you're connecting an AI agent or assistant to your data lake, use the Alloy MCP server instead of a raw API key. See [MCP →](/docs/mesh-storage/explore/mcp).
## Revoking a key [#revoking-a-key]
Open the **API keys** modal and find the key in the list. Click its delete action and confirm — the key stops working immediately. Any clients still using it will start receiving auth errors on their next request.
# Overview
Once data lands in Mesh Storage, **Explore** is where you work with it. Five surfaces, one data lake:
* [**Browse**](/docs/mesh-storage/explore/browse) — the file tree, status badges, [Replay](/docs/mesh-storage/explore/browse#replay), [Inspect](/docs/mesh-storage/explore/browse#inspect), ROS graph, and ROS2 diagnostics. Best for "what's in this recording?"
* [**SQL Workbench**](/docs/mesh-storage/explore/workbench) — the in-app query editor. No setup, no credentials. Best for ad-hoc analysis straight from the browser.
* [**Python SDK**](/docs/mesh-storage/sdk) — upload files, query hosted SQL, list Mesh files, and download originals from Python. Best for notebooks, scripts, jobs, and services.
* [**Connect external tools**](/docs/mesh-storage/explore/connect) — point DuckDB, Spark, Trino, PyIceberg, or any Iceberg REST client at your data. Best for notebooks, BI dashboards, and CI pipelines.
* [**MCP**](/docs/mesh-storage/explore/mcp) — connect Claude, Cursor, Codex, Windsurf, or any MCP-aware AI tool to your missions and data lake.
All five surfaces hit the same Mesh data — a query in SQL Workbench sees the same Ready data as a Python SDK query, and an MCP tool call sees what Browse shows.
# MCP Server (AI)
Alloy exposes an [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) server that lets you connect your AI coding and analysis tools directly to your mission data. Query missions, search instances, view reports, and more — right from your preferred AI tool.
Some tools may not be available depending on how your Alloy account is set up and which features are enabled for your org. Your AI tool will only see the tools that apply to you.
## Setting up [#setting-up]
The MCP server URL:
```
https://aus.usealloy.ai/mcp
```
Connect it to your AI tool of choice using Streamable HTTP transport:
Add the Alloy MCP server, then authenticate via the `/mcp` menu inside a session:
```bash
claude mcp add alloy --transport http https://aus.usealloy.ai/mcp
```
This opens a browser-based OAuth flow. Tokens are stored locally and refreshed automatically.
If you need to reauthenticate later, run `/mcp` again and select the Alloy server.
See the [Claude Code MCP docs](https://docs.anthropic.com/en/docs/claude-code/mcp) for more details.
Remote MCP servers are added via the **Connectors** UI, not the JSON config file.
1. Open **Settings** → **Connectors** (or click **+** in chat → **Connectors** → **Manage connectors**)
2. Click **Add custom connector**
3. Paste the URL: `https://aus.usealloy.ai/mcp`
4. Click **Add** and complete the authentication prompt
The Alloy connector's tools will then be available in your conversations. You can toggle it on/off per conversation from the **+** menu.
For Team and Enterprise plans, an org owner must first add the connector in **Organization settings** → **Connectors** before members can use it.
See the [Claude Desktop custom connectors guide](https://support.claude.com/en/articles/11175166-get-started-with-custom-connectors-using-remote-mcp) for more details.
Add the MCP server using the [Codex CLI](https://github.com/openai/codex):
```bash
codex mcp add alloy --url https://aus.usealloy.ai/mcp
```
Then log in to authenticate with Alloy:
```bash
codex mcp login alloy
```
This opens the Alloy consent screen in your browser. Once authenticated, restart Codex (Desktop or CLI) and it should work.
If you get logged out at any point, re-run `codex mcp login alloy` to reauthenticate.
Add to `.cursor/mcp.json` in your project (or `~/.cursor/mcp.json` for global access):
```json
{
"mcpServers": {
"alloy": {
"url": "https://aus.usealloy.ai/mcp"
}
}
}
```
Restart Cursor after saving. When you first use an Alloy tool, you'll be prompted to authenticate in your browser.
You can verify the connection in the **Output** panel → **MCP** dropdown.
See the [Cursor MCP docs](https://docs.cursor.com/context/model-context-protocol) for more details.
Add to `~/.codeium/windsurf/mcp_config.json` (or manage via **Windsurf Settings** → **Cascade** → **MCP Servers**):
```json
{
"mcpServers": {
"alloy": {
"serverUrl": "https://aus.usealloy.ai/mcp"
}
}
}
```
Restart Windsurf after saving. Authentication will be handled via a browser-based OAuth flow on first use.
See the [Windsurf MCP docs](https://docs.windsurf.com/windsurf/cascade/mcp) for more details.
Paste the following into your AI tool for help getting set up:
```text
I need to connect to an MCP server. Here are the details:
- Server URL: https://aus.usealloy.ai/mcp
- Transport: Streamable HTTP (the URL is a single HTTP endpoint that accepts JSON-RPC POST requests)
- Authentication: OAuth 2.0 (browser-based consent flow — no API key or token needed upfront)
- Server name: "alloy"
Please configure this MCP server in my client. The server follows
the MCP Streamable HTTP transport specification. On first use it
will redirect me to a browser to authenticate with my Alloy
credentials via OAuth. No manual token setup is required.
```
The MCP server uses the same authentication as the Alloy web app. When you first connect, you'll be prompted to authenticate in your browser; the connection then persists for your session.
## What you can do [#what-you-can-do]
### Read your data [#read-your-data]
* `list_missions` — search and list missions across your org
* `get_mission_summary` — full mission details including key events, metrics, and comments
* `get_available_filters` — discover what metadata filters exist for your missions
* `get_plot_data` — data points for a specific plot in a mission summary
* `get_map_data` — trajectory and geospatial data points
* `search_instances` — semantic image, log, or similarity search across your instance library
* `get_instance` — instance metadata, logs, images, timeseries, and detections
* `list_scenarios` / `get_scenario` — scenario list and match details
* `list_reports` / `get_report` / `list_scheduled_reports` — read reports and check schedules
* `query_data` — ask a natural language question about your data; the agent writes SQL and returns results
* `search_docs` — search Alloy product documentation
### Create & edit (focused, single-shot) [#create--edit-focused-single-shot]
Each of these runs a small focused agent — supply IDs you already know to keep them fast. All finish well inside any AI tool's request timeout. Edit tools are owner-only.
* `create_report(prompt, mission_id?, mission_ids?)` — compose a new shareable report from a brief
* `update_report(report_id, prompt)` — edit an existing report you own
* `create_scheduled_report(prompt)` — set up a recurring report (daily / weekly / fortnightly)
* `update_scheduled_report(schedule_id, prompt)` — edit a recurring schedule you own
**Treat these as zero-reasoning tools.** They are deterministic compose-and-write passes on a tight budget. They cannot investigate, decide, hunt anomalies, or pick thresholds — if you ask them to, they will burn the budget exploring and time out without writing anything. Do the analysis upstream with `query_data` (or `alloy_request_start` for genuinely open-ended work) and then call these with concrete inputs.
**Do this:**
* *"Build a report titled 'Mission X Summary' for `mission_id m_abc` with sections: Overview, Metrics, Anomalies. The metrics are cpu=42%, latency\_p95=87ms."* → `create_report` works great
* *"Look at all our missions this quarter and write a report on the worst performers."* → use `alloy_request_start` (or break it down: `query_data` first to find the worst, then `create_report` with the names and numbers)
* *"Add this paragraph to report rpt\_xyz: …"* → `update_report` works great
* *"Make this report better."* → don't. Either decide the change yourself, or ask `alloy_request_start` to do the analysis and call `update_report` internally.
### Long-running tasks [#long-running-tasks]
For anything that needs multi-step exploration, deeper reasoning, or doesn't fit a single-shot tool — most notably creating or editing scenarios, where the scanner's detection strategy benefits from sample-mission validation — use the async pair:
* `alloy_request_start(prompt, timeout_seconds?)` — kick off a multi-step task, returns a `task_id` immediately. Defaults to a 25-minute server budget (matches the in-app agent's subagent cap); the agent self-terminates with whatever it has when the cap is hit.
* `alloy_request_poll(task_id, wait_seconds?)` — long-poll for completion (server blocks up to 60s); the response always carries `progress` so you can see what the agent is doing
### Mesh Storage [#mesh-storage]
* `browse_mesh_storage` — list files and folders in your mesh bucket (MCAP files come back with pipeline status baked in)
* `get_mesh_file_download_url` — presigned download URL for a specific file
* `get_mesh_replay_url` — dashboard URL that opens replay for one or more MCAPs (multi-file timeline). Files should overlap in time or be chronological for the best UX.
* `get_mesh_inspect_url` — dashboard URL that opens the Inspect MCAP modal for a single file (topics, diagnostics, ROS graph, capture config)
* `query_mesh_storage` — read-only DuckDB SQL against your Iceberg catalog
* `list_mesh_tables` — list tables or describe a specific table's schema
* `get_mesh_connection_info` — gateway URL and instructions for external clients (DuckDB, Spark, Trino). Alloy never issues mesh API keys over MCP — you generate those from the dashboard.
### Device fleet & edge setup [#device-fleet--edge-setup]
* `list_devices` — every device with status, last-seen, last-upload, and file count
* `approve_device` / `reject_device` — lifecycle actions for pending devices
* `get_edge_config` / `update_edge_config` — read and write recorder config files
* `get_docker_setup` — full Docker bundle (AR token, login/pull commands, compose yaml). See [Secret handling](#secret-handling) below.
* `get_binary_setup` — signed binary download URL plus `edge-manager.yaml`
* `get_edge_manifest` — available distros, tags, architectures, and binary releases
## Secret handling [#secret-handling]
Device-setup tools return real short-lived AR access tokens (1-hour, read-only) so `docker login` commands are immediately usable. The **provisioning key** is never returned over MCP — compose and yaml templates come back with `` placeholders. Open Mesh Storage → Device Setup in the dashboard to download the same files with the real key embedded, or copy the provisioning key from that page into the template by hand.
Other actions stay UI-only for safety: deleting devices or files, rolling API keys, creating Mesh Storage API keys, and resetting edge configs.
## Example usage [#example-usage]
Once connected, you can ask your AI tool questions about your Alloy data:
* "What missions were uploaded this week?"
* "Summarize the latest mission report"
* "Find instances with navigation errors across all missions"
* "What scenarios are currently running?"
* "Compare controller error across the last 10 flights and write up the worst three" — uses `alloy_request_start` (analysis + selection upstream, then a report). The async pair will fetch the data and call `create_report` internally with concrete values.
* "Write a report titled 'Mission M Latency' for mission\_id=`m_abc` with these metrics: p95=87ms, p99=142ms, errors=3" — uses `create_report` directly (concrete inputs, no analysis needed)
* "Schedule a weekly fleet-health summary for Monday 9am" — uses `create_scheduled_report`
* "Set up a scenario that flags any flight with a Z-axis position error over 0.5m" — uses `alloy_request_start` (scenarios need sample-mission validation, so they go through the async pair)
The AI tool uses Alloy's MCP server to fetch the data and respond in context — no need to switch to the Alloy web app.
# SQL Workbench
SQL Workbench is the in-app query editor. Open it from the Mesh Storage action bar — no credentials to manage, no client to install.
## Open the workbench [#open-the-workbench]
From the Mesh Storage action bar, click **SQL Workbench**. The editor opens as a full-screen modal.
{/* TODO: screenshot — SQL Workbench button in mesh-storage action bar */}
The workbench is disabled until your data lake is provisioned and you have at least one **Ready** file. Upload something first if you don't see results.
## Layout [#layout]
Three panes:
* **Schema tree** (left, \~20%): databases, namespaces, tables, and columns. Click a table to insert it into the editor.
* **Query editor** (top right): a Monaco editor with SQL syntax highlighting and autocomplete.
* **Results** (bottom right): tabular results with sortable columns and pagination.
You can drag the dividers to resize either pane.
{/* TODO: screenshot — SQL Workbench three-pane layout */}
## Run a query [#run-a-query]
Write standard SQL referencing tables from the schema tree. Results appear in the bottom pane as you run them:
```sql
SELECT topic, count(*) AS messages
FROM "uploads"."my_recording__diagnostics"
GROUP BY topic
ORDER BY messages DESC
LIMIT 20;
```
The workbench uses [DuckDB](https://duckdb.org/) running in your browser — queries execute locally against the Iceberg tables, with reads streamed from object storage. Most aggregations and filters run quickly even on multi-GB files.
## Export results [#export-results]
Click **Export CSV** in the top-right of the results pane to download all rows. Export bypasses the in-page row limit, so you get the full result set.
## How credentials work [#how-credentials-work]
The workbench fetches scoped, read-only credentials for your data lake on first open. They live **in memory only** — refreshing the page or closing the tab clears them. There is no `localStorage` or `sessionStorage` persistence.
The credentials have a TTL of roughly 20 hours. If a session is open longer than that, the next query will refresh them automatically.
## Limitations [#limitations]
* **Read-only.** No `INSERT`, `UPDATE`, `DELETE`, or `CREATE TABLE` against the data lake.
* **No cross-org joins.** You can only query your own org's tables.
* **In-flight files don't show up.** Tables only appear once the file is **Ready**. If you don't see your data, check the [status badge](/docs/mesh-storage/explore/browse#status-badges).
# Docker
Open Mesh Storage → Device Setup → and select **Docker** to run the interactive flow with pre-filled tokens and compose files.
The client needs outbound access on **port 443 (HTTPS)** only. No inbound ports required.
## Step 1: Pull the container image [#step-1-pull-the-container-image]
In the Device Setup modal, select **Docker**, then pick your ROS distro (Jazzy/Humble), architecture (x86\_64/ARM64), and optional variant. Click **Generate Access Command** to get a short-lived token (1 hour) for the private registry.
Authenticate with the registry:
```bash
echo "" | docker login \
-u oauth2accesstoken \
--password-stdin \
australia-southeast1-docker.pkg.dev
```
Then pull the image:
```bash
docker pull australia-southeast1-docker.pkg.dev/alloy-version-0/alloy-edge/alloy-edge:
```
## Step 2: Configure and download the compose file [#step-2-configure-and-download-the-compose-file]
In the **Download Compose File** section of the Device Setup modal, pick your middleware and download:
1. Leave **Middleware** on the default, **FastDDS**.
2. If alloy-edge will share topics with other ROS 2 containers on the same host, enter their UID in **Run as UID** (e.g. `1000:1000`). Otherwise leave it blank.
3. Click **Download docker-compose.yml**.
Required for Isaac Sim. Highest throughput via shared memory.
**UID must match.** FastDDS uses `/dev/shm` internally — if alloy-edge runs as a different UID than your other ROS 2 containers, topics discover but **data silently never flows**. When in doubt, run `echo "$(id -u):$(id -g)"` as the user that owns those containers and paste the result into **Run as UID**.
1. Set **Middleware** to **CycloneDDS**.
2. Click **Download docker-compose.yml**.
No UID matching needed. Works without `ipc: host` — good for rootless Docker and Kubernetes.
1. Set **Middleware** to **Zenoh**.
2. Click **Download docker-compose.yml**.
The downloaded file includes a `zenoh-router` sidecar service automatically. Best for edge / WAN / multi-robot fleets.
| | FastDDS | CycloneDDS | Zenoh |
| -------------------------- | ----------------------------- | -------------------------------- | ------------------------------- |
| RMW package | `rmw_fastrtps_cpp` | `rmw_cyclonedds_cpp` | `rmw_zenoh_cpp` |
| Requires `ipc: host` | **Yes** | No | No |
| Same UID across containers | **Required** | Not needed | Not needed |
| Isaac Sim compatible | **Yes** (required) | No | No |
| Best for | Default ROS 2, max throughput | Rootless Docker, K8s, mixed UIDs | Edge / WAN / multi-robot fleets |
## Step 3: Run [#step-3-run]
```bash test:quickstart
docker compose up -d
```
Your device will appear in Mesh Storage under the `devices/` folder as **Pending**.
## Step 4: Approve the device [#step-4-approve-the-device]
1. Open the `devices/` folder in Mesh Storage
2. Find the new device and click **Approve**
3. Alloy issues the device a permanent API key
4. The client picks up the new key on its next sync — no manual key distribution needed
## What happens next [#what-happens-next]
After approval, the edge client picks up its configuration and starts recording. Within a minute or two you should see:
* **Last seen** updating as the client syncs
* **Files** appearing in the device's folder in Mesh Storage
You can then [replay, inspect, or query](/docs/mesh-storage/explore/browse) any uploaded MCAP file directly from Mesh Storage.
## ROS domain ID [#ros-domain-id]
`ROS_DOMAIN_ID` (default `0`) isolates ROS 2 traffic on the same network. Set it when running multiple robots on the same LAN, or co-locating dev and prod stacks on one host:
```bash
ROS_DOMAIN_ID=42 docker compose up -d
```
All containers that need to communicate must share the same domain ID **and** the same middleware.
## Managing the container [#managing-the-container]
```bash
docker compose logs -f # view logs
docker compose down # stop
docker compose restart # restart
```
Folder structure (mounted volumes):
```
data/
├── config/ # client config, provisioning key
└── data/ # recorded bags, uploaded files
```
# Overview
Mesh Storage accepts files through three primary paths:
* [**Web upload**](/docs/mesh-storage/ingest/upload) — drag files from your laptop into Mesh Storage.
* [**Track a folder**](/docs/mesh-storage/ingest/track-folder) — point Alloy Edge at the directory where your recorder writes MCAP files.
* [**Python SDK**](/docs/mesh-storage/sdk/uploads) — upload from a script, notebook, backend job, or CI run.
Web uploads land in **`uploads/`**. SDK uploads land in **`uploads/sdk-uploads/`**.
Files uploaded by Alloy Edge land under **`devices//`**. All three paths
become queryable in the same Mesh Storage workspace.
## How Track Folder works [#how-track-folder-works]
Your recorder continues to own recording. Alloy Edge watches its output folder, waits
for each file to close, uploads it over HTTPS, and resumes automatically after network
interruptions. No inbound ports or cloud-storage credentials are required on the device.
Need Alloy to provide the ROS 2 recorder and supporting processors too? A Docker
deployment is also available from **Mesh Storage → Add device**. Track Folder is the
recommended documentation path for teams that already have a recorder.
Continue with [Track Folder](/docs/mesh-storage/ingest/track-folder).
# Web upload
The fastest way to get a recording into Alloy: drag the `.mcap` file into Mesh Storage and walk away. No client to install, no device to provision.
Use this for ad-hoc files, recordings shared with you by a teammate, or a single mission you want to inspect without setting up an edge client.
## Step 1: Open the upload modal [#step-1-open-the-upload-modal]
From the Mesh Storage action bar, click **Upload**.
{/* TODO: screenshot — Upload button highlighted in mesh-storage action bar */}
## Step 2: Add files [#step-2-add-files]
Drag MCAP files into the drop zone, or click to open the file picker. You can add multiple files at once.
Only `.mcap` files are accepted. Other file types are rejected with a warning telling you which ones were skipped — drop those out and continue.
{/* TODO: screenshot — upload modal with drag-drop zone and file list */}
## Step 3: Upload [#step-3-upload]
Click **Upload** to start. Each file shows a progress bar and a status badge:
| Badge | Meaning |
| --------- | -------------------------------------------- |
| Waiting | Queued, hasn't started uploading yet |
| Uploading | Multipart upload in progress |
| Done | Successfully uploaded to object storage |
| Error | Upload failed — see message; retry or remove |
| Cancelled | Aborted before completion |
The overall progress bar at the bottom shows total progress across all files.
You can click **Cancel** to abort all in-flight uploads. Cancelled multipart uploads are explicitly aborted on the backend so no orphaned parts are left behind.
## Step 4: Wait for processing [#step-4-wait-for-processing]
Uploaded files land in the **`uploads/`** folder of Mesh Storage with a status badge:
1. **Queued** — the file has landed in object storage and is waiting to be processed
2. **Processing** — Alloy is parsing the MCAP and building queryable tables
3. **Ready** — the file is queryable in [SQL Workbench](/docs/mesh-storage/explore/workbench) and external Iceberg clients
4. **Failed** — something went wrong; check the row for the error
[Replay](/docs/mesh-storage/explore/browse#replay) and [Inspect](/docs/mesh-storage/explore/browse#inspect) work as soon as the file lands — you don't have to wait for processing to finish.
## What about large files? [#what-about-large-files]
Web uploads use multipart upload. Each file is split into parts and uploaded in parallel. There's no per-file size limit imposed by the uploader, but very large files (multi-GB) are usually faster to push from a device using [Track Folder](/docs/mesh-storage/ingest/track-folder) over a stable connection.
## What about folder structure? [#what-about-folder-structure]
Web-uploaded files all land flat under `uploads/`. If you want files organized by device, mission, or run, use [device-based ingestion](/docs/mesh-storage/ingest) — files there are organized by device automatically.
# Manage
Day-to-day fleet operations after a device is connected:
* **API key rotation** — issue, roll, and revoke device API keys without re-provisioning.
* **Dashboard signals** — see device status, last contact, and upload activity.
* **[Redact](/docs/mesh-storage/manage/redact)** — strip or hash sensitive fields before recordings leave the device.
## API key rotation [#api-key-rotation]
Every device gets its own API key on approval. You can rotate it from Mesh Storage
without re-provisioning or connecting to the device over SSH.
A rotation may be needed when a device is decommissioned, a key may have been exposed,
or your security policy requires periodic credential rotation.
### How key rotation works [#how-key-rotation-works]
Key rotation is designed to avoid upload downtime:
1. Request a roll from the device actions menu under `devices/`. The old key remains valid.
2. On its next sync, the device receives and persists a new key.
3. Alloy revokes the old key after the device confirms the replacement.
You can cancel a pending roll before the device syncs.
## Dashboard signals [#dashboard-signals]
The `devices/` folder shows fleet status at a glance:
| Column | What it tells you |
| --------------- | ---------------------------------------------- |
| **Device** | Edge ID and device ID |
| **Info** | Hostname or machine identifier |
| **Status** | Pending, Approved, or Rejected |
| **Last Seen** | When the device last contacted Alloy |
| **Last Upload** | When the device last uploaded a file |
| **Files/Size** | Total files uploaded and their cumulative size |
{/* TODO: reshoot from Mesh Storage devices/ folder view */}
# Redact
This feature is in **beta**. Schema and CLI may change between releases — pin your `alloy-edge` version when authoring rules for production.
Some recordings carry data you don't want leaving the robot — operator names in metadata, microphone audio on `/audio/**`, hostnames echoed in `std_msgs/String` topics. Alloy Edge can rewrite or drop those records on the device before `edge-sync` uploads, so the cloud only ever sees the sanitised version.
The redactor is a streaming MCAP rewriter. It reads each record once, applies any matching rules, and writes a new file alongside the original — no buffering, no decode for channels you didn't list, and a self-documenting audit trail per file.
## What you get [#what-you-get]
| Capability | What it does |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Channel filter** | Drop or allow whole topics by glob (`/audio/**`, `/user/*`). Runs before any decode — costs nothing. |
| **Per-topic transforms** | Rewrite individual messages with `put` (whitelist — spell out what you keep) or `patch` (denylist — override specific fields). Templates use Jinja2. |
| **Metadata redaction** | Same shape applied to MCAP metadata records (operator info, calibration, config blobs). |
| **Regex redactors** | Strip patterns out of string fields with `regex_strip` (one pattern + replacement) or `regex_redact` (apply a list of patterns, all matches → `[REDACTED]`). Bring your own patterns for emails, phone numbers, IPs, employee IDs, etc. |
| **Hash salt rotation** | `hash(...)` reads `${VAR}` from the environment at config-load time. Rotate by changing the env var — no rule edits. |
| **Dry-run mode** | Run the filter with `upload_type: none` and route both originals and redacted artefacts to `/.dry-run/` to inspect locally before flipping it on. |
| **Self-documenting audit** | JSONL sidecar + embedded MCAP metadata record per file. Each carries `rules_hash` so you can prove later which rules version produced a given file. |
| **Failure policy** | Fail-closed by default — a broken rule can skip the file instead of uploading unredacted data. `pass_original` is opt-in. |
## When to use it [#when-to-use-it]
| You want to… | Use redaction? |
| ---------------------------------------------------- | -------------------------------------------------- |
| Drop a whole topic the cloud should never see | Yes — channel filter (`channels.deny`) |
| Hash an operator's name in metadata for compliance | Yes — `metadata:` rule with `hash(...)` |
| Replace one field in a known message with a constant | Yes — inline `patch` rule |
| Convert a high-bandwidth topic into a tiny summary | Yes — `put` rule with a Jinja template |
| Just stop recording the topic in the first place | No — narrow the topic list in your recording stack |
The recorder's topic list is your first line of defence — if a topic shouldn't be recorded at all, drop it there. Use redaction when you do need to record a topic (for replay, scenarios, or local diagnostics) but want to scrub something out before upload.
## How it fits together [#how-it-fits-together]
Redaction is configured in two files:
* **`edge-sync.yaml`** — a redaction **pipeline step** (v0.8+), or the legacy `redaction:` block (v0.7, still accepted), plus `lifecycle:` policy. Points at the rules file, sets failure policy and audit behavior, and controls what happens to original/redacted artefacts after upload.
* **`redaction.yaml`** — the rules themselves. Channel allow/deny, per-topic transforms, metadata-record transforms, named functions. This file is **identical across v0.7 and v0.8**.
The rules file is a separate file because operators rotate it independently of `edge-sync.yaml` (different review cadence, sometimes different reviewers).
```text
recorder writes ──► /recordings/*.mcap ──► edge-sync picks up
│
▼
edge-transform
(channel filter →
per-topic transform)
│
▼
/recordings/.alloy-redacted/*.mcap ──► upload
+ audit JSONL line
+ audit MCAP record (embedded)
```
## Quick start [#quick-start]
### Step 1: Author a rules file [#step-1-author-a-rules-file]
Create `/etc/alloy/redaction.yaml`:
```yaml test:redact-rules
enabled: true
# Cheapest filter — drop topics before any decode.
channels:
allow: ["*"]
deny:
- "/audio/**"
- "/user/*"
# Per-topic mappings. First match wins.
transforms:
# Replace the data field with a hash.
- match: "/robot_status"
transform:
type: patch
schema: "std_msgs/msg/String"
overrides:
data: '{{ original | sha256_short }}'
# MCAP metadata records — separate from message channels.
metadata:
- match: "operator_*"
transform:
type: patch
overrides:
operator_name: '{{ original | hash(algo="md5") }}'
# Strip emails out of free-text notes — replacement is the second arg.
operator_notes: '{{ original | regex_strip("[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "[EMAIL]") }}'
```
For multi-pattern stripping (emails *and* phone numbers *and* IPs in the same field), use `regex_redact` with a pattern list — it replaces every match with `[REDACTED]`:
```yaml
overrides:
description: '{{ original | regex_redact([
"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}",
"\\b\\d{3}[-.]?\\d{3}[-.]?\\d{4}\\b",
"\\b\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b"
]) }}'
```
The full schema (named functions, `put` vs `patch`, match selectors, hash salts, includes) lives in the [redaction reference](/docs/mesh-storage/reference/redaction).
### Step 2: Wire it into `edge-sync.yaml` [#step-2-wire-it-into-edge-syncyaml]
In **v0.8+** redaction is a `pipeline:` step; the failure-policy and audit knobs move under `lifecycle.transform`. The **v0.7** flat `redaction:` block still works in v0.8 (the loader auto-migrates it; run `alloy-edge migrate` to rewrite on disk).
```yaml
version: 1
pipeline:
- transform: /etc/alloy/redaction.yaml # rules file from Step 1
upload: true # upload the redacted artefact
original_after: move # set the unredacted original aside after upload
lifecycle:
original:
move_to: /var/lib/alloy/edge-sync/originals
transform:
on_rule_error: skip_file # drop the file if a rule fails (alt: skip_record [default] / pass_original)
audit:
jsonl_path: /var/lib/alloy/edge-sync/redaction-audit.jsonl # set to enable the JSONL sidecar; omit to disable
embed_in_mcap: true # also write the audit summary inside the redacted file
```
```yaml test:redact-edge-sync
redaction:
enabled: true
rules_file: /etc/alloy/redaction.yaml
on_rule_error: skip_file # drop the file (alt: skip_record [default — drop bad records, keep filtering] / pass_original)
audit:
jsonl_path: /var/lib/alloy/edge-sync/redaction-audit.jsonl # set to enable the JSONL sidecar; omit to disable
embed_in_mcap: true # also write the audit summary inside the redacted file
lifecycle:
original:
after: move
move_to: /var/lib/alloy/edge-sync/originals
redacted:
after: keep
```
### Step 3: Dry-run before flipping it on [#step-3-dry-run-before-flipping-it-on]
Run the pipeline with **no network**: set `upload_type: none` in `edge-sync.yaml`, and route the originals and the redacted artefacts into a `.dry-run/` sandbox using `lifecycle`. Scrub the result, confirm it matches expectations, then revert to normal uploads.
```yaml title="edge-sync.yaml — dry-run overrides"
upload_type: none # filter runs, nothing leaves the device
version: 1
pipeline:
- transform: /etc/alloy/redaction.yaml
original_after: move
lifecycle:
original:
after: move
move_to: .dry-run/originals # v0.8 rejects a shared move_to — keep the two distinct
transform:
after: move
move_to: .dry-run/redacted
```
```yaml title="edge-sync.yaml — dry-run overrides"
upload_type: none # filter runs, nothing leaves the device
redaction:
enabled: true
rules_file: /etc/alloy/redaction.yaml
lifecycle:
original:
after: move
move_to: .dry-run # relative to input_dir
redacted:
after: move
move_to: .dry-run
```
Or, for a one-shot run without editing the file, pass `--dry-run` to the CLI — it applies the same overrides at load time (and is exempt from the v0.8 shared-`move_to` check, routing both into one timestamped `.dry-run/` subdir):
```bash test:redact-dry-run
alloy-edge sync --config /etc/alloy/edge-sync.yaml --dry-run
```
Both the unredacted originals and the redacted artefacts land under `/.dry-run/`. Inspect them the same way you inspect any MCAP — `mcap info`, `mcap cat`, or drag-drop into the [web uploader](/docs/mesh-storage/ingest/upload) to open in Foxglove / Alloy Replay.
### Step 4: Enable and watch [#step-4-enable-and-watch]
Once dry-run looks right, remove the dry-run overrides (restore `upload_type` to whatever you normally use, drop the `lifecycle` block or set the actions you actually want for steady-state), restart edge-sync, and tail the audit log:
```bash
tail -f /var/lib/alloy/edge-sync/redaction-audit.jsonl
```
Each line is one file's redaction summary — messages and bytes in/out, per-channel dropped + transformed counts, denied channels, per-helper match counts, plus the `rules_hash` (`sha256:…`) of the config that produced it.
## The two transform shapes — `put` vs `patch` [#the-two-transform-shapes--put-vs-patch]
Each per-topic rule is one of two shapes. Pick based on how stable the upstream schema is.
### `put` — whitelist (safe by default) [#put--whitelist-safe-by-default]
Spell out the entire output. Anything not mentioned in the template disappears. New upstream fields don't leak.
```yaml
- match: "/operator/command"
transform:
type: put
schema: "your_msgs/OperatorCommand"
template: |-
{
"operator_id": {{ operator_id | hash(algo="sha256") | tojson }},
"command": {{ command | tojson }},
"timestamp": {{ timestamp | tojson }}
}
```
`operator_id` is hashed; `command` and `timestamp` pass through. Any other field upstream — `notes`, `location`, `payload` — disappears. That's the put semantic: spell out what you keep, everything else drops.
Use `put` for compliance channels — anything where "this field appeared upstream and we didn't redact it" would be a problem.
### `patch` — denylist (concise) [#patch--denylist-concise]
Pass everything through, override only the listed fields. New upstream fields flow through unchanged.
```yaml
- match: "/robot_status"
transform:
type: patch
schema: "std_msgs/msg/String"
overrides:
data: '{{ original | hash(algo="sha256") }}'
```
Use `patch` when the schema is stable and you only need to neutralise one or two fields. If upstream adds a new field you didn't anticipate, it'll flow through — that's the trade-off for the shorter rule.
Examples here use `hash(algo="sha256")` (salted) for de-identification. There's also `sha256_short` for content fingerprinting — same value always hashes to the same 8-char digest, but **unsalted** — so it's reversible with a wordlist and not safe for de-id. See the [helper reference](/docs/mesh-storage/reference/redaction#template-helpers).
## When a rule fails [#when-a-rule-fails]
Templates can break — a Jinja syntax error, a schema change that removes a field reference. Two distinct failure points:
* **Config-load time**: bad Jinja syntax (typo, unclosed tag, unknown filter) rejects the rules file before edge-sync starts. You'll see this immediately — not at upload time.
* **Runtime**: a syntactically-valid template that fails on a particular message (e.g. `{{ status[0].hardware_id }}` when a message has empty `status`). Behaviour is governed by `on_rule_error`:
* **`on_rule_error: skip_record`** (default) — the offending record is dropped, the rest of the file is filtered and uploaded. Lets one bad message not spike the whole file.
* **`on_rule_error: skip_file`** — the file is not uploaded when a rule fails. Pair with `lifecycle.original.after: move` if you want to retain failed originals for operator review.
* **`on_rule_error: pass_original`** — fail-open: the unredacted original uploads. Requires explicit opt-in because it can leak. Use only when "drop nothing on the floor" beats "leak nothing", and document why.
## The audit trail [#the-audit-trail]
Every redacted file gets two parallel audit records:
| Carrier | Where | Why |
| ------------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------ |
| **JSONL sidecar** | `{state_dir}/redaction-audit.jsonl` (one line per file) | Cheap to grep, cheap to stream — the operational logbook |
| **Embedded MCAP record** | A metadata record named `alloy.redaction.audit` inside the redacted file | The redacted file documents itself — survives re-uploads, copies, and forwards |
The two carriers contain the same JSON shape. Both default on. Turn `embed_in_mcap` off only when the rule layout itself is sensitive (e.g. you don't want the redacted file to disclose which topics were touched).
Each audit entry includes a `rules_hash` (formatted `sha256:`) over the merged config so you can prove later which version of the rules produced a given file.
## Hash salts and rotation [#hash-salts-and-rotation]
`{{ original | hash(algo="md5") }}` and friends use a salt configured at the top of `redaction.yaml`:
```yaml
hash_salt: "${ALLOY_HASH_SALT}"
```
The `${VAR}` form is interpolated from the environment at config-load time. Rotate by changing the env var and restarting edge-sync — no rule edit needed. The audit log records `hash_salt_fingerprint: sha256(salt)[:8]` per file so a compliance reviewer can prove a rotation happened without ever seeing the salt.
If the env var isn't set when the config loads, the loader fails fast — you can't accidentally ship with a missing salt.
## Channel filter is the fastest [#channel-filter-is-the-fastest]
Rules under `transforms:` decode the message; rules under `channels.deny` don't. If a topic should never leave the robot, deny it at the channel filter — it costs nothing and never decodes.
Only reach for a `transform:` when you need to keep some part of a message and remove or rewrite the rest.
## Limits and things to know [#limits-and-things-to-know]
* **MCAP only.** Other recording formats aren't supported by the redactor.
* **First match wins.** Order rules from most-specific to most-general.
## Next steps [#next-steps]
* [Redaction reference](/docs/mesh-storage/reference/redaction) — full `redaction.yaml` schema: every field, the function library, and the match selector forms.
* [Configuration reference](/docs/mesh-storage/reference/config) — the redaction pipeline step (and legacy `redaction:` block) in `edge-sync.yaml` (rules file path, output/audit knobs, failure policy) plus `lifecycle` retention controls.
# Redaction
The canonical reference for `redaction.yaml`. For the why-and-when, see [Redact](/docs/mesh-storage/manage/redact).
```yaml
# Master switch.
enabled: true
# Optional — merge named functions from additional files.
includes:
- /etc/alloy/transforms-common.yaml
# Topic-level allow/deny. Cheapest filter — runs before any decode.
channels:
allow: ["*"]
deny: ["/user/*", "/audio/**"]
# Salt for hash(...) helpers. ${VAR} is read from the environment at load time.
hash_salt: "${ALLOY_HASH_SALT}"
# Per-channel mappings. First match wins.
transforms: [...]
# Per-metadata-record mappings. Same shape as transforms.
metadata: [...]
# The named-function library. Referenced by transforms / metadata via `function:`.
functions: { ... }
# Optional — rule-file-level audit defaults. Operational overrides (CLI flags,
# edge-sync.yaml's `audit:` block) take precedence. Useful for "this rule set
# always wants its audit embedded" without forcing every call site to pass a flag.
audit:
embed_in_mcap: true
```
## Top-level [#top-level]
| Field | Type | Default | Description |
| ------------ | ------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | bool | `false` | Master switch. `false` → the redactor is bypassed even though edge-sync's `redaction.enabled` may be true. |
| `includes` | list of paths | `[]` | Additional files merged into the same `functions:` namespace, in declaration order. Last file wins on name collision; the entry-point file's own `functions:` always wins overall. Resolved relative to the directory of the file that contains them. |
| `channels` | object | `{}` | Topic-level allow/deny. See below. |
| `hash_salt` | string | — | Salt for `hash(...)` and friends. `${VAR}` is interpolated from the environment at load time. Missing env var → load-time error. |
| `functions` | map | `{}` | Named transform bodies — the "library" referenced by `transforms:` and `metadata:`. |
| `transforms` | list | `[]` | Per-channel mappings. First match wins. |
| `metadata` | list | `[]` | Per-metadata-record mappings. Same shape as `transforms` but `match:` globs against record names. |
| `audit` | object | — | Rule-file-level audit defaults. Currently one field: `embed_in_mcap` (bool). Absent → defer to `edge-sync.yaml`'s `audit:` block / CLI flags. |
## `channels` [#channels]
Topic-level allow/deny — the cheapest filter. Runs **before** any decode, so denied channels never pay the CDR-decode cost.
| Field | Type | Default | Description |
| ------- | ------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `allow` | list of globs | — | **Fail-closed.** `null` or `[]` → no topic passes. `["*"]` → all topics pass (only `deny` filters). `["/odom", "/tf*"]` → only matching topics pass. |
| `deny` | list of globs | `[]` | Always wins on conflict — a `deny` match drops the topic regardless of `allow`. |
Globs are POSIX-style: `*` matches any segment, `**` matches across segments.
## `transforms[]` / `metadata[]` [#transforms--metadata]
Each entry has **exactly one** of `function:` or `transform:`. Both or neither → load-time error.
| Field | Type | Required | Description |
| ----------- | ---------------- | -------- | --------------------------------------------------------------------------- |
| `match` | string or object | yes | Channel/record selector. See [Match selector](#match-selector) below. |
| `function` | string | one of | Reference to a named function in `functions:` (or merged from `includes:`). |
| `transform` | object | one of | Inline transform body — same shape as a `functions:` entry. |
## Match selector [#match-selector]
Two equivalent forms — pick the shorthand for the common case.
```yaml
# Shorthand — channel only (or record-name only for `metadata:`)
match: "/diagnostics"
# Object form — full selector
match:
channel: "/diagnostics"
schema: "diagnostic_msgs/DiagnosticArray" # disambiguates same-topic-different-schema (rare but legal)
encoding: "cdr" # or "ros2msg", "protobuf", etc.
```
For the `metadata:` block, the shorthand maps to `record_name` instead of `channel`:
```yaml
match: "operator_*" # shorthand → record_name
match:
record_name: "operator_*" # object form
key: "operator_email" # optional inner-key glob
```
| Field | Used by | Description |
| ------------- | ------------ | ------------------------------------------------------------------------------------------------- |
| `channel` | `transforms` | Topic glob (e.g. `/diagnostics`, `/sensors/*`). |
| `schema` | `transforms` | Optional schema-name glob — disambiguates two channels with the same topic but different schemas. |
| `encoding` | `transforms` | Optional encoding glob — `cdr`, `ros2msg`, `protobuf`. |
| `record_name` | `metadata` | MCAP metadata-record name glob. |
| `key` | `metadata` | Optional inner-key glob within a metadata record. |
## Transform bodies — `put` and `patch` [#transform-bodies--put-and-patch]
Each transform body has a `type:` discriminator. v1 ships two: `put` and `patch`.
### `type: put` — whitelist [#type-put--whitelist]
Spell out the entire output. Fields not mentioned in the template disappear. New upstream fields don't leak.
| Field | Type | Default | Description |
| ------------------ | --------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `schema` | string | — | ROS2 schema name the template targets (e.g. `diagnostic_msgs/DiagnosticArray`). Optional for `metadata:` rules. |
| `available_fields` | list of strings | `[]` | Documentation-only — human-readable list of fields the template references. Not enforced. |
| `template` | string (Jinja2) | — | Jinja2 template rendering the full output JSON. **Omit `template:` for the identity put** — the runtime short-circuits to a zero-copy passthrough (same cost as having no rule), useful when an operator wants the config to document a deliberate "this channel is fine, leave it alone." |
### `type: patch` — denylist [#type-patch--denylist]
Pass everything through, override only the listed fields.
| Field | Type | Default | Description |
| ----------- | ------ | ------- | ------------------------------------------------------------------------------------- |
| `schema` | string | — | ROS2 schema name. Optional for `metadata:` rules. |
| `overrides` | map | `{}` | Field-path → Jinja2 expression. Path syntax is `field`, `field.sub`, `array[].field`. |
## `functions` [#functions]
Named transform bodies. Same shape as a `transform:` entry, minus the `match:` key (which lives on the call site).
```yaml
functions:
# Identity put — runtime short-circuits to zero-copy passthrough.
identity_pass:
type: put
# Whitelist redactor for an operator-command message.
redact_operator_cmd:
type: put
schema: "your_msgs/OperatorCommand"
template: |-
{
"operator_id": {{ operator_id | hash(algo="sha256") | tojson }},
"command": {{ command | tojson }},
"timestamp": {{ timestamp | tojson }}
}
# Denylist counterpart — same schema, only override two fields.
redact_operator_cmd_lite:
type: patch
schema: "your_msgs/OperatorCommand"
overrides:
operator_id: '{{ original | hash(algo="sha256") }}'
notes: '""'
```
Functions from `includes:` land in the same namespace; local definitions in the entry-point file always win on collision.
## Template helpers [#template-helpers]
Inside `template:` and `overrides:` expressions, the rendering engine is [MiniJinja](https://docs.rs/minijinja/latest/minijinja/) (Jinja2-compatible). On top of the standard filters, the redactor adds:
| Helper | Use | Example |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `hash(algo=...)` | Full hex digest of `sha256(salt + value)`. `algo`: `md5`, `sha256`. **Use this for de-identification.** Requires `hash_salt`. | `'{{ original \| hash(algo="sha256") }}'` |
| `sha256_short` | First 8 hex chars of `sha256(value)` — **unsalted**. Content fingerprint, *not* de-identification (reversible with a wordlist). | `'{{ s.name \| sha256_short }}'` |
| `regex_strip(pattern, replacement)` | Replace regex matches in the value with a literal string. | `'{{ original \| regex_strip("[A-Z]+", "[X]") }}'` |
| `regex_redact([patterns])` | Apply a list of regex patterns; replace every match with `[REDACTED]`. Convenience wrapper for chaining several `regex_strip` calls — patterns are user-supplied. | `'{{ original \| regex_redact([email_re, phone_re]) }}'` |
| `redact_if(predicate, replacement)` | Replace the value only when the predicate matches; otherwise pass through. | `'{{ original \| redact_if("hostname", "[HOST]") }}'` |
| `zero` | Type-aware zero — `""` for strings, `0` for numbers, etc. | `'{{ original \| zero }}'` |
| `tojson` | JSON-encode a value (custom registration; minijinja's built-in `tojson` requires a feature this crate doesn't pull in). | `'{{ header \| tojson }}'` |
**`original` is a context variable, not a filter.** It's bound only inside `patch.overrides` (the pre-override field value) and `metadata:` rule expressions (the pre-redaction string). `put.template` does **not** see `original` — it sees the full decoded message and addresses fields by name (`{{ header | tojson }}`, `{{ status[0].level }}`).
`hash_salt` must be set if any rule calls `hash(...)`. `sha256_short` does not consume the salt.
## Resolution rules [#resolution-rules]
1. **Channel filter runs first.** Denied topics drop before any decode.
2. **First match wins** per channel / record name. Anything not listed in `transforms:` is an implicit passthrough — no decode, no allocation, byte-for-byte copy.
3. **`includes:` merge in declaration order** (last include wins on function-name collision); local `functions:` in the entry-point file override every include.
4. **Bad templates fail at load time.** A Jinja syntax error in any template / override expression rejects the rules file before the agent starts.
## Reload semantics [#reload-semantics]
Edits to `redaction.yaml` (or any included file) take effect only after `edge-sync` restarts. The rules file is read once at startup; there is no file watcher today. A future release plans hot-reload using the `source_files` set the loader populates.
The merged config is hashed (`sha256:`, recorded as `rules_hash` on every audit entry) so you can prove later which version of the rules produced a given file.
# List and Download Files
Use `alloy.storage` read helpers when you need to inspect upload object names or download uploaded files from Python.
```python
from alloy import storage
with storage.connect() as store:
listing = store.list_files(path="flights/run-001")
```
## Prefixes and keys [#prefixes-and-keys]
Upload helpers accept friendly SDK `path` values:
```text
flights/run-001
```
For SDK uploads, list helpers accept the same `path` value. For any Mesh-visible
folder, pass a raw `prefix` instead:
```text
uploads/sdk-uploads/flights/run-001
uploads/sdk-uploads/flights/run-001/run.mcap
```
Downloads use exact Mesh `key` strings, usually from an upload or list result.
SDK read helpers accept `uploads/...` prefixes and keys. That includes web uploads and SDK uploads; device-path reads are outside the SDK read surface.
## List files [#list-files]
```python
from alloy import storage
with storage.connect() as store:
listing = store.list_files(
path="flights/run-001",
recursive=False,
max_keys=100,
)
for directory in listing.directories:
print("dir", directory)
for file in listing.files:
print(file.key, file.size, file.last_modified)
```
`recursive=False` uses folder-like delimiters. Set `recursive=True` to return files below nested folders.
## Pagination [#pagination]
If a list response has more objects, `next_token` is set.
```python
from alloy import storage
prefix = "uploads/sdk-uploads/flights/run-001"
token = None
with storage.connect() as store:
while True:
page = store.list_files(prefix=prefix, recursive=True, continuation_token=token)
for file in page:
print(file.key)
token = page.next_token
if token is None:
break
```
## Download one file [#download-one-file]
```python
from alloy import storage
with storage.connect() as store:
store.download_file(
"uploads/sdk-uploads/flights/run-001/run.mcap",
"local-copy.mcap",
)
```
`download_file` downloads the exact key to the destination path.
## One-shot helpers [#one-shot-helpers]
```python
from alloy import storage
listing = storage.list_files(path="flights/run-001", recursive=True)
storage.download_file("uploads/sdk-uploads/flights/run-001/run.mcap", "run.mcap")
```
Async one-shots:
```python
from alloy import storage
listing = await storage.async_list_files(
path="flights/run-001",
recursive=True,
)
await storage.async_download_file(
"uploads/sdk-uploads/flights/run-001/run.mcap",
"run.mcap",
)
```
The SDK does not expose delete, move, rename, or overwrite operations. Use Mesh Storage deletion workflows when you need to remove or replace files.
# Python SDK
The Alloy Python SDK gives you a normal Python path into Mesh Storage. Use it from notebooks, scripts, backend jobs, CI checks, and internal tools that need to upload recordings or work with data that is already in Mesh.
Mesh still does the same work behind the scenes: files land in object storage, Alloy processes MCAPs into queryable tables, and Ready data becomes available through SQL, Replay, Inspect, and external tools.
The SDK is for Python code. If you want to browse files, replay MCAPs, or run ad-hoc SQL without writing code, use the Mesh Storage page and SQL Workbench instead.
## What you can do [#what-you-can-do]
## The public modules [#the-public-modules]
The SDK exposes two customer-facing modules:
| Module | Use it for |
| --------------- | ------------------------------------------------------ |
| `alloy.storage` | Upload files, list Mesh files, and download Mesh files |
| `alloy.sql` | Query Ready Mesh data through hosted SQL |
Keep imports explicit:
```python
from alloy import storage, sql
```
Keep using the module names in application code. The package root keeps SQL convenience exports, but storage client names stay under `alloy.storage` so `Client` and `AsyncClient` do not collide.
## How uploads fit Mesh Storage [#how-uploads-fit-mesh-storage]
SDK uploads land under:
```text
uploads/sdk-uploads//
```
You choose the `` part:
```python
from alloy import storage
with storage.connect() as store:
result = store.upload_folder("runs/run-001", path="flights/run-001")
print(result.prefix)
# uploads/sdk-uploads/flights/run-001/
```
After upload, files show up in Mesh Storage like any other uploaded file. Replay and Inspect are available once the file lands. SQL is available after processing reaches **Ready**.
## Sync and async [#sync-and-async]
Both storage and hosted SQL have sync and async clients. Use sync for scripts and notebooks. Use async for services, workers, and tools that already run on an event loop.
```python
from alloy import storage
with storage.connect() as store:
store.upload_file("run.mcap", path="flights/run-001")
```
```python
from alloy import storage
async with storage.async_connect() as store:
await store.upload_file("run.mcap", path="flights/run-001")
```
## Next steps [#next-steps]
# Limits
The SDK keeps Mesh Storage operations explicit: upload files into SDK paths, read Mesh objects by prefix or key, and query Ready data through hosted SQL.
## Upload limits [#upload-limits]
High-level upload helpers support `.mcap`, `.json`, `.yml`, and `.yaml` files under:
```text
uploads/sdk-uploads/
```
The SDK does not expose delete, move, rename, overwrite, raw R2 writes, direct object-store credentials, upload finalize calls, or direct mission creation.
`overwrite=True` raises immediately. Delete or replace files through Mesh Storage workflows before uploading a replacement.
## Read limits [#read-limits]
Read helpers list and download Mesh-visible objects by SDK `path`, raw `prefix`, or exact `key`. They do not provide file-id mapping or cross-org access.
The SDK validates final object key length before upload. Mesh object keys must fit the storage provider's key size limits.
## SQL limits [#sql-limits]
Hosted SQL is read-only. The SDK does not expose write SQL, table creation, public Flight SQL/gRPC, Postgres wire protocol, ODBC/JDBC, cursors, sessions, or stream resume.
## Processing limits [#processing-limits]
Uploading a file is not the same as processing it. After upload:
* Replay and Inspect work once the MCAP lands.
* SQL requires the file to reach **Ready**.
* Failed files need investigation from the Mesh Storage file row.
The SDK does not poll processing status. Use Mesh Storage, SQL Workbench, or MCP tools to inspect readiness.
## Dependencies [#dependencies]
The base SDK includes the transfer clients required by Mesh Storage. Optional extras are still used for some local result conversions:
| Feature | Dependency path |
| -------------------- | --------------------------- |
| Mesh sync transfers | included |
| Mesh async transfers | included |
| pandas conversion | install `alloy-sdk[pandas]` |
| Polars conversion | install `alloy-sdk[polars]` |
| DuckDB conversion | install `alloy-sdk[duckdb]` |
If a dependency is missing in a broken environment, the SDK raises a dependency error rather than silently falling back.
# Query Mesh Data
Use `alloy.sql` to query Mesh data after files reach **Ready**. The hosted SQL endpoint returns Arrow IPC; the SDK adapts it into rows, RowSets, Arrow tables, DataFrames, DuckDB relations, or streams.
```python
from alloy import sql
with sql.connect() as db:
rows = db.fetch(
"""
SELECT key, count(*) AS entries
FROM alloy.mesh.file_meta
GROUP BY key
ORDER BY entries DESC
LIMIT 20
"""
)
```
## Connect [#connect]
```bash
export ALLOY_API_KEY="ak_..."
export ALLOY_DATA_URL="https://data.usealloy.ai"
```
```python
from alloy import sql
with sql.connect() as db:
count = db.fetchval("SELECT count(*) FROM alloy.mesh.file_meta")
```
Explicit values are supported:
```python
from alloy import sql
with sql.connect(
base_url="https://data.usealloy.ai",
api_key="ak_...",
) as db:
rows = db.fetch("SELECT * FROM alloy.mesh.file_meta LIMIT 10")
```
## Small row results [#small-row-results]
Use `fetch`, `fetchrow`, and `fetchval` for bounded results that fit naturally in Python.
```python
from alloy import sql
with sql.connect() as db:
topics = db.fetch(
"""
SELECT topic, count(*) AS messages
FROM alloy.fleet.diagnostics
GROUP BY topic
ORDER BY messages DESC
LIMIT 100
"""
)
latest = db.fetchrow(
"""
SELECT file_id, updated_at
FROM alloy.mesh.file_meta
ORDER BY updated_at DESC
LIMIT 1
"""
)
file_count = db.fetchval("SELECT count(DISTINCT file_id) FROM alloy.mesh.file_meta")
```
## RowSet results [#rowset-results]
Use `fetch_rows` when you need columns, row counts, CSV output, or a local materialization cap.
```python
from alloy import sql
with sql.connect() as db:
rowset = db.fetch_rows(
"""
SELECT file_id, key, value, value_num
FROM alloy.mesh.file_meta
ORDER BY updated_at DESC
LIMIT 1000
""",
max_rows=1000,
)
print(rowset.columns)
print(rowset.row_count)
print(rowset.to_csv())
```
`max_rows` is a local SDK cap. It prevents unbounded Python row materialization, but it does not rewrite your SQL or reduce hosted SQL work. Use SQL `LIMIT` for query performance.
## Typed params [#typed-params]
Use typed params instead of formatting values into SQL strings.
```python
from alloy import sql
p = sql.param
with sql.connect() as db:
rows = db.fetch(
"""
SELECT file_id, key, value
FROM alloy.mesh.file_meta
WHERE key = $key
AND value = $mission_id
LIMIT 100
""",
params={
"key": p.utf8("alloy.mission_id"),
"mission_id": p.utf8("2U8NUGFHGifdCt7tdcnwTd"),
},
)
```
Parameter names do not include `$` in the `params` mapping. Params are values only; they cannot stand in for table names, column names, SQL clauses, or fragments.
Supported param constructors:
| Constructor | Use for |
| ------------------------------- | ---------------------- |
| `sql.param.utf8(value)` | strings |
| `sql.param.bool(value)` | booleans |
| `sql.param.int64(value)` | signed 64-bit integers |
| `sql.param.float64(value)` | floating point values |
| `sql.param.date32(value)` | dates |
| `sql.param.timestamp_us(value)` | timestamps |
| `sql.param.binary(value)` | bytes |
## Arrow and DataFrames [#arrow-and-dataframes]
Use `query` when you want an Arrow-backed result object with multiple conversion options.
```python
from alloy import sql
with sql.connect() as db:
result = db.query("SELECT * FROM alloy.mesh.file_meta LIMIT 1000")
print(result.column_names)
print(result.column_types)
print(result.row_count)
arrow_table = result.to_arrow()
pandas_df = result.to_pandas()
polars_df = result.to_polars()
duckdb_conn = result.to_duckdb(table_name="mesh_file_meta")
```
Direct helpers are shortcuts:
```python
with sql.connect() as db:
arrow_table = db.query_arrow("SELECT * FROM alloy.mesh.file_meta LIMIT 100")
pandas_df = db.query_pandas("SELECT * FROM alloy.mesh.file_meta LIMIT 100")
polars_df = db.query_polars("SELECT * FROM alloy.mesh.file_meta LIMIT 100")
df = db.query_df("SELECT * FROM alloy.mesh.file_meta LIMIT 100")
```
Pandas, Polars, and DuckDB helpers require their corresponding SDK extras, such as `alloy-sdk[pandas]`, `alloy-sdk[polars]`, or `alloy-sdk[duckdb]`.
## Streaming batches [#streaming-batches]
Use streams for large query results that should be processed batch by batch.
```python
from alloy import sql
with sql.connect() as db:
with db.stream("SELECT * FROM alloy.mesh.file_meta") as stream:
with stream.reader() as reader:
for batch in reader:
print(batch.num_rows)
```
## Async usage [#async-usage]
```python
from alloy import sql
with sql.connect() as db:
row = db.fetchrow("SELECT count(*) AS n FROM alloy.mesh.file_meta")
```
```python
from alloy import sql
async with sql.async_connect() as db:
row = await db.fetchrow("SELECT count(*) AS n FROM alloy.mesh.file_meta")
```
Async clients mirror the sync method names. Expensive result materialization is async so service code does not accidentally block the event loop.
Hosted SQL is read-only. Use it to query Mesh data, not to insert, update, delete, or create tables.
# Quickstart
This quickstart uploads one local run into Mesh Storage, lists the uploaded files, and runs a hosted SQL query once processing is Ready.
## Install [#install]
```bash
pip install alloy-sdk
```
The SDK includes the transfer clients it needs for Mesh uploads and downloads.
## Configure credentials [#configure-credentials]
Set your Alloy API key and data API endpoint:
```bash
export ALLOY_API_KEY="ak_..."
export ALLOY_DATA_URL="https://data.usealloy.ai"
```
You can also pass `base_url=` and `api_key=` directly to `storage.connect()` and `sql.connect()`.
Treat your Alloy API key like a password. Do not commit it to source control or print it in notebook output.
## Upload and query [#upload-and-query]
Put your MCAP and any metadata sidecars in one local folder.
Upload the folder into a Mesh path you can recognize later.
Wait for the files to show
**Ready**
in Mesh Storage.
Run hosted SQL against the tables created from the MCAP.
```python
from alloy import storage, sql
with storage.connect() as store:
upload = store.upload_folder(
"local/run-001",
path="flights/run-001",
overwrite=False,
)
print(upload.prefix)
# uploads/sdk-uploads/flights/run-001/
with sql.connect() as db:
rows = db.fetch(
"""
SELECT key, count(*) AS entries
FROM alloy.mesh.file_meta
GROUP BY key
ORDER BY entries DESC
LIMIT 20
"""
)
for row in rows:
print(row)
```
The exact table names depend on the MCAP contents and your Mesh catalog. Use [SQL Workbench](/docs/mesh-storage/explore/workbench), `list_mesh_tables` over MCP, or hosted SQL discovery queries to inspect available tables.
## List and download files [#list-and-download-files]
```python
from alloy import storage
with storage.connect() as store:
listing = store.list_files(path="flights/run-001", recursive=True)
for file in listing:
print(file.key, file.size)
store.download_file(
"uploads/sdk-uploads/flights/run-001/run.mcap",
"downloaded-run.mcap",
)
```
Upload helpers use friendly `path` values such as `flights/run-001`. List helpers accept the same SDK path for SDK uploads. Downloads use the Mesh `key` returned by upload or list results.
## Async version [#async-version]
```python
from alloy import storage, sql
async with storage.async_connect() as store:
upload = await store.upload_folder(
"local/run-001",
path="flights/run-001",
overwrite=False,
)
async with sql.async_connect() as db:
count = await db.fetchval("SELECT count(*) FROM alloy.mesh.file_meta")
```
Async storage transfers use the same method names; async SQL uses native async HTTP and decodes Arrow off the event loop.
## Keep going [#keep-going]
# API Reference
This page is a compact reference for the Python SDK surface used with Mesh Storage.
## Install [#install]
```bash
pip install alloy-sdk
```
## Environment variables [#environment-variables]
| Variable | Used by | Description |
| ---------------- | ---------------------------- | ---------------------------------------------------------- |
| `ALLOY_API_KEY` | `alloy.storage`, `alloy.sql` | Alloy API key for your org |
| `ALLOY_DATA_URL` | `alloy.storage`, `alloy.sql` | Alloy data API URL, for example `https://data.usealloy.ai` |
Explicit `base_url=` and `api_key=` values override environment defaults.
## alloy.storage [#alloystorage]
Import:
```python
from alloy import storage
```
### Connect [#connect]
```python
storage.connect(
*,
base_url: str | None = None,
api_key: str | None = None,
headers=None,
timeout: float = 120.0,
)
```
Returns a sync `storage.Client`.
```python
storage.async_connect(
*,
base_url: str | None = None,
api_key: str | None = None,
headers=None,
timeout: float = 120.0,
)
```
Returns an async `storage.AsyncClient`.
Both clients are context managers and close SDK-owned HTTP sessions.
### Client methods [#client-methods]
```python
client.upload_file(
local_path,
*,
path: str | None = None,
overwrite: Literal[False] = False,
) -> storage.UploadResult
```
Uploads one `.mcap`, `.json`, `.yml`, or `.yaml` file into a Mesh Storage path.
```python
client.upload_folder(
local_folder,
*,
path: str | None = None,
overwrite: Literal[False] = False,
) -> storage.UploadResult
```
Uploads supported files below a folder, preserving relative paths. Sidecars upload before MCAPs.
```python
client.list_files(
prefix: str | None = None,
*,
path: str | None = None,
recursive: bool = False,
max_keys: int = 100,
continuation_token: str | None = None,
) -> storage.StorageListResult
```
Lists objects below either an SDK upload `path` or a raw Mesh `prefix`.
```python
client.download_file(
key: str,
destination,
) -> None
```
Downloads one Mesh object by exact key.
`storage.AsyncClient` exposes the same methods with `await`.
### One-shot helpers [#one-shot-helpers]
```python
storage.upload_file(...)
storage.upload_folder(...)
storage.list_files(...)
storage.download_file(...)
await storage.async_upload_file(...)
await storage.async_upload_folder(...)
await storage.async_list_files(...)
await storage.async_download_file(...)
```
One-shot helpers open a client, run one operation, and close the client.
### Storage result objects [#storage-result-objects]
| Object | Fields |
| ------------------- | ---------------------------------------------- |
| `StorageFile` | `key`, `size`, `last_modified`, `etag` |
| `StorageListResult` | `files`, `directories`, `next_token`, `prefix` |
| `PlannedUpload` | `local_path`, `relative_path`, `key`, `size` |
| `UploadResult` | `files`, `total_bytes`, `path`, `prefix` |
## alloy.sql [#alloysql]
Import:
```python
from alloy import sql
```
### Connect [#connect-1]
```python
sql.connect(
*,
base_url: str | None = None,
api_key: str | None = None,
headers=None,
timeout: float = 120.0,
)
```
```python
sql.async_connect(
*,
base_url: str | None = None,
api_key: str | None = None,
headers=None,
timeout: float = 120.0,
)
```
### Query methods [#query-methods]
| Method | Returns | Use for |
| --------------------------------------------- | ------------------------ | ----------------------------- |
| `fetch(sql, params=None)` | `list[dict[str, Any]]` | small row results |
| `fetchrow(sql, params=None)` | `dict[str, Any] \| None` | one row |
| `fetchval(sql, params=None)` | first value | scalar queries |
| `fetch_rows(sql, params=None, max_rows=None)` | `RowSet` | rows plus schema/cap metadata |
| `query(sql, params=None)` | `QueryResult` | Arrow-backed result object |
| `query_arrow(sql, params=None)` | `pyarrow.Table` | Arrow handoff |
| `query_pandas(sql, params=None)` | pandas DataFrame | notebook analysis |
| `query_polars(sql, params=None)` | Polars DataFrame | Polars analysis |
| `query_df(sql, params=None)` | pandas DataFrame | alias for pandas |
| `stream(sql, params=None)` | `QueryStream` | Arrow batch streaming |
| `query_batches(sql, params=None)` | `QueryStream` | alias for stream |
Async clients expose the same method names. Await eager methods; use `async with` for streams.
### SQL result objects [#sql-result-objects]
| Object | Useful members |
| ------------------ | ------------------------------------------------------------------------------------------------------ |
| `RowSet` | `columns`, `rows`, `row_count`, `truncated`, `named_rows()`, `to_csv()` |
| `AsyncRowSet` | `columns`, `rows`, `row_count`, `truncated`, `named_rows()`, `to_csv()` |
| `QueryResult` | `column_names`, `column_types`, `row_count`, `to_arrow()`, `to_pandas()`, `to_polars()`, `to_duckdb()` |
| `AsyncQueryResult` | async equivalents for expensive conversions |
| `QueryStream` | context-managed Arrow batch stream |
| `AsyncQueryStream` | async context-managed Arrow batch stream |
## Errors [#errors]
Storage errors:
| Error | Meaning |
| ----------------------------- | --------------------------------------------------- |
| `AlloyStorageError` | Base storage SDK error |
| `AlloyStorageHttpError` | Storage Gateway returned a non-2xx response |
| `AlloyStorageResponseError` | Gateway or storage response had an unexpected shape |
| `AlloyStorageTransferError` | Local planning or transfer failed |
| `AlloyStorageConflictError` | `overwrite=False` found an existing target key |
| `AlloyStorageDependencyError` | Required transfer dependency is unavailable |
SQL errors:
| Error | Meaning |
| ------------------------- | --------------------------------------------------- |
| `AlloySqlError` | Base hosted SQL SDK error |
| `AlloySqlHttpError` | Hosted SQL returned a non-2xx response |
| `AlloySqlResponseError` | Hosted SQL response had an unexpected shape |
| `AlloySqlDecodeError` | Arrow IPC decoding failed |
| `AlloySqlDependencyError` | Optional DataFrame/DuckDB dependency is unavailable |
## Version notes [#version-notes]
Storage helpers, async transfers, and hosted SQL are available in the current `alloy-sdk` package. Upgrade before using this reference:
```bash
pip install --upgrade alloy-sdk
```
# Upload Files
Use `alloy.storage` when you already have files on disk and want them to land directly in Mesh Storage.
```python
from alloy import storage
with storage.connect() as store:
store.upload_folder("local/run-001", path="flights/run-001")
```
## Folder uploads [#folder-uploads]
`upload_folder(local_folder, path=...)` walks the folder recursively and preserves relative paths below that folder.
```text
local/run-001/
├── metadata.json
├── drone-a/
│ └── flight.mcap
└── drone-b/
└── flight.mcap
```
```python
from alloy import storage
with storage.connect() as store:
result = store.upload_folder(
"local/run-001",
path="flights/run-001",
overwrite=False,
)
print(result.prefix)
# uploads/sdk-uploads/flights/run-001/
```
The resulting Mesh keys are:
```text
uploads/sdk-uploads/flights/run-001/metadata.json
uploads/sdk-uploads/flights/run-001/drone-a/flight.mcap
uploads/sdk-uploads/flights/run-001/drone-b/flight.mcap
```
## Single-file uploads [#single-file-uploads]
`upload_file(local_path, path=...)` treats `path` as a folder and uploads the local file into that folder using the local basename.
```python
from alloy import storage
with storage.connect() as store:
result = store.upload_file(
"local/run-001/flight.mcap",
path="flights/run-001",
overwrite=False,
)
print(result.files[0].key)
# uploads/sdk-uploads/flights/run-001/flight.mcap
```
## Path rules [#path-rules]
`path` is a Mesh folder below the SDK upload root:
```text
uploads/sdk-uploads//
```
Pass this:
```text
flights/run-001
```
Do not pass:
```text
uploads/sdk-uploads/flights/run-001
uploads/flights/run-001
s3://bucket/flights/run-001
flights/run-001/
```
If you omit `path`, Alloy generates a dated folder such as:
```text
2026-06-24/4c9033de-6e7d-4d9a-8c34-9b4ab10e6bb7
```
The upload result includes the concrete generated `path` and `prefix`.
## Supported files [#supported-files]
The high-level upload helpers support:
| File type | Extension |
| --------------- | --------------- |
| MCAP recordings | `.mcap` |
| JSON sidecars | `.json` |
| YAML sidecars | `.yml`, `.yaml` |
Metadata sidecars upload before MCAP files. This makes sidecar metadata available to the ingestion path before the recording is processed.
## Overwrite behavior [#overwrite-behavior]
`overwrite=False` is the supported high-level SDK upload mode.
```python
from alloy import storage
with storage.connect() as store:
store.upload_folder("local/run-001", path="flights/run-001", overwrite=False)
```
Before uploading, the SDK checks every target key. If any key already exists, it raises `AlloyStorageConflictError` before uploading the new batch.
```python
from alloy import storage
try:
storage.upload_file("flight.mcap", path="flights/run-001", overwrite=False)
except storage.AlloyStorageConflictError as exc:
print(exc)
```
`overwrite=True` raises immediately. Delete or replace files through the Mesh deletion workflow before uploading a replacement.
## What happens after upload [#what-happens-after-upload]
Uploaded files appear in Mesh Storage under `uploads/sdk-uploads/`. Each MCAP goes through the normal lifecycle:
1. **Queued** - the file has landed and is waiting for processing
2. **Processing** - Alloy is parsing the MCAP and building queryable tables
3. **Ready** - SQL and external query paths can read the processed data
4. **Failed** - processing failed; check the file row for details
Replay and Inspect work as soon as the MCAP lands. SQL requires the file to be **Ready**.
## One-shot helpers [#one-shot-helpers]
For short scripts, use module-level helpers. They open and close a client for one operation.
```python
from alloy import storage
storage.upload_folder("local/run-001", path="flights/run-001")
storage.upload_file("local/run-001/flight.mcap", path="flights/run-001")
```
Async one-shots are available too:
```python
from alloy import storage
await storage.async_upload_folder("local/run-001", path="flights/run-001")
await storage.async_upload_file("local/run-001/flight.mcap", path="flights/run-001")
```
Use a long-lived client when you are doing more than one operation.
# Track Folder — v0.7.x
This guide targets the latest v0.7 patch, **v0.7.6**. New installations should use
v0.9.0.
## 1. Install v0.7.6 [#1-install-v076]
Open **Mesh Storage → Add device**, select binary version **v0.7.6**, download the
matching build, and put `alloy-edge` on `PATH`.
## 2. Create the Track Folder configuration [#2-create-the-track-folder-configuration]
```bash
alloy-edge init track-folder \
--track-dir /data/recordings \
--api-key
```
This writes `edge-manager.yaml`, `edge-sync.yaml`, and runtime state under
`/data/recordings/.alloy/`.
## 3. Run [#3-run]
v0.7 does not install a service through `init`. Run the manager in the foreground
under your existing service supervisor:
```bash
alloy-edge manager \
--config /data/recordings/.alloy/edge-manager.yaml
```
## 4. Approve and verify [#4-approve-and-verify]
Open `devices/` in Mesh Storage and approve the pending device. After approval,
**Last Seen** updates and completed files appear under the device's folder.
For retention, upload, or redaction tuning, use the
[advanced v0.7 Configuration reference](/docs/mesh-storage/reference/config/0.7.x).
# Track Folder — v0.8.5
This is the legacy v0.8 flow. New installations should use v0.9.0.
## 1. Install v0.8.5 [#1-install-v085]
Open **Mesh Storage → Add device**, select binary version **v0.8.5**, download the
matching build, and put `alloy-edge` on `PATH`.
## 2. Create the Track Folder configuration [#2-create-the-track-folder-configuration]
```bash test:init-track-folder
alloy-edge init track-folder \
--track-dir /data/recordings \
--api-key
```
This writes `edge-manager.yaml`, `edge-sync.yaml`, and runtime state under
`/data/recordings/.alloy/`.
For an unattended Linux system service:
```bash
sudo alloy-edge init track-folder \
--track-dir /data/recordings \
--api-key \
--config-profile system \
--systemd
```
## 3. Run [#3-run]
If `init` installed a service, it is already running. Otherwise start the manager:
```bash test:run-manager
alloy-edge manager \
--config /data/recordings/.alloy/edge-manager.yaml
```
## 4. Approve and verify [#4-approve-and-verify]
Open `devices/` in Mesh Storage and approve the pending device. After approval,
**Last Seen** updates and completed files appear under the device's folder.
For retention, upload, or pipeline tuning, use the
[advanced v0.8.5 Configuration reference](/docs/mesh-storage/reference/config/0.8.5).
# Track Folder — v0.9.0
If your recorder already writes MCAP files to a folder, Alloy Edge can watch that
folder and upload each completed recording to Mesh Storage.
Need Alloy to provide the ROS 2 recorder and supporting processors too? A Docker
deployment is also available from **Mesh Storage → Add device**.
The device needs outbound HTTPS access on port 443. It does not need an inbound port.
## 1. Install Alloy Edge [#1-install-alloy-edge]
Open **Mesh Storage → Add device**, choose the binary for the device's operating
system and architecture, and put `alloy-edge` on `PATH`.
For a standalone Linux binary:
```bash
chmod +x alloy-edge
mkdir -p ~/.local/bin
mv alloy-edge ~/.local/bin/
```
## 2. Link the recordings folder [#2-link-the-recordings-folder]
Use the provisioning key shown by Add device. Reading it from stdin keeps it out of
shell history:
```bash
printenv ALLOY_PROVISIONING_KEY | alloy-edge link \
--provisioning-key-stdin \
--track-folder /data/recordings
```
The default is a user installation. `link` validates the credential, creates the
configuration, waits briefly for approval, and prints the matching `run` command.
It does not start a background process.
| Installation | Link option | Configuration and state |
| -------------- | ----------------------------------- | ------------------------------------------------------ |
| User (default) | no scope option | `~/.config/alloy-edge` and `~/.local/state/alloy-edge` |
| Folder-local | `--local` | `/data/recordings/.alloy` |
| System | `--system` (run `link` with `sudo`) | `/etc/alloy-edge` and `/var/lib/alloy-edge` |
## 3. Run Alloy Edge [#3-run-alloy-edge]
For the default user installation:
```bash
alloy-edge run --user
```
Keep `run` in the foreground under a supervisor you control. For the other scopes,
use `alloy-edge run --local` from the tracked folder or
`sudo alloy-edge run --system`.
To validate the selected installation without backend calls or child processes:
```bash
alloy-edge run --user --dry-run
```
On Linux, `alloy-edge config --user systemd print` previews a service unit and
`alloy-edge config --user systemd install` installs it. Replace `--user` with the
scope used during `link`.
## 4. Approve and verify [#4-approve-and-verify]
If the device is pending, open `devices/` in Mesh Storage and select **Approve**.
After approval, **Last Seen** updates and completed recordings appear under the
device's folder.
For local retention, upload, or pipeline tuning, use the
[advanced v0.9.0 Configuration reference](/docs/mesh-storage/reference/config/0.9.0).
# Track Folder
Choose the version installed on your device. New installations should use v0.9.0.
Alloy remembers this selection when you move between Track Folder and the advanced
Configuration reference.
# Configuration — v0.7.x
This reference targets the latest v0.7 patch, **v0.7.6**. Do not add
`version: 1` or v0.8 pipeline fields to these files.
The v0.7 Track Folder flow uses two YAML files:
| File | Loaded by | Purpose |
| ------------------- | -------------------- | ----------------------------------------------------------------------- |
| `edge-manager.yaml` | `alloy-edge manager` | Cloud connection, device identity, and local process supervision |
| `edge-sync.yaml` | `alloy-edge sync` | Folder watching, uploads, flat cleanup limits, redaction, and lifecycle |
Durations use Go-style strings such as `5s`, `1m`, and `72h`. Sizes and bandwidth
limits use values such as `500MB`, `10GB`, and `5MB`.
Print the complete commented templates shipped with the installed binary:
```bash
alloy-edge manager sample-config
alloy-edge sync sample-config
```
## `edge-manager.yaml` [#edge-manageryaml]
### Top-level fields [#top-level-fields]
| Field | Purpose |
| ------------- | ---------------------------------------------------------------------------------------------------------------- |
| `backend_url` | Alloy backend endpoint. Use the value generated by **Add device**; `ALLOY_BACKEND_URL` overrides it. |
| `state_dir` | Manager state, credentials, and child configuration. Defaults to `.alloy/state`; `ALLOY_STATE_DIR` overrides it. |
| `seed_state` | Bootstrap identity and transport used until the device is approved. |
| `local_state` | Locally owned tags and processes that cloud configuration cannot overwrite. |
### `seed_state` [#seed_state]
| Field | Default | Purpose |
| -------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api_key` | — | Provisioning key used until approval. `ALLOY_PROVISIONING_KEY` overrides it before approval; `ALLOY_API_KEY` overrides the permanent key at runtime. |
| `edge_id` | Hostname, then generated UUID | Stable device identifier reported to Alloy. |
| `tags` | `{}` | Free-form strings for grouping and filtering devices. |
| `transport.http.poll_secs` | `15` | Seconds between desired-state checks. |
| `processes` | `[]` | Bootstrap processes used before the first successful cloud sync. Prefer `local_state.processes` for locally owned processes. |
### `local_state.processes` [#local_stateprocesses]
| Field | Default | Purpose |
| --------------- | -------------- | --------------------------------------------------------------------------------- |
| `name` | — | Required process name used in logs and reported state. |
| `command` | — | Required command to execute. |
| `enabled` | `true` | Keep the process declared but stopped when `false`. |
| `restart` | `on_failure` | Restart policy: `always`, `on_failure`, or `never`. |
| `shell` | `false` | Run through `/bin/sh -c` when shell expansion is required. |
| `trigger` | Start on apply | Start at `boot`, on a cron schedule, or when connectivity changes. |
| `duration` | — | Stop the process after a bounded duration. |
| `requires_auth` | `false` | Hold the process until Alloy approves the device. Set this for `alloy-edge sync`. |
| `files` | `{}` | Inline configuration files attached to the process. |
A locally owned Track Folder sync process has this shape:
```yaml
local_state:
processes:
- name: edge-sync
command: alloy-edge sync -c /data/recordings/.alloy/edge-sync.yaml
enabled: true
restart: on_failure
trigger: boot
requires_auth: true
```
## `edge-sync.yaml` [#edge-syncyaml]
v0.7 uses the unstamped configuration schema. Only `input_dir` is required.
Disk cleanup is disabled until you set one of the top-level limits.
```yaml
input_dir: /data/recordings
file_pattern: "*.mcap,*.json,*.jsonl"
upload_delay: 30s
max_folder_size: 10GB
max_file_age: 72h
max_file_count: 1000
lifecycle:
original:
after: keep
```
### Scanning and upload [#scanning-and-upload]
| Field | Default | Purpose |
| ------------------------ | ----------------------- | -------------------------------------------------------------------- |
| `input_dir` | — | Required folder to scan. |
| `file_pattern` | `*.mcap,*.json,*.jsonl` | Comma-separated file globs. `.tmp` entries are reserved and removed. |
| `cycle_time` | `1s` | Folder scan interval. |
| `upload_delay` | `30s` | Minimum age before a footer-less recording is eligible. |
| `mcap_require_footer` | `false` | Require an MCAP footer instead of falling back to file age. |
| `upload_type` | `signed_url` | `signed_url`, `none`, or feature-gated `opendal`. |
| `bwlimit` | Unlimited | Outbound bandwidth limit. |
| `max_concurrent_uploads` | `1` | Number of files uploaded in parallel. |
| `metadata` | `{}` | Free-form metadata included in upload requests. |
| `scan_exclude` | `[]` | Subdirectory basenames to skip; dot-directories are always skipped. |
### Cleanup and upload state [#cleanup-and-upload-state]
| Field | Default | Purpose |
| ----------------- | ------------------------ | ----------------------------------------------------------------------- |
| `max_folder_size` | Unbounded | Delete oldest eligible files when the tracked folder exceeds this size. |
| `max_file_age` | — | Delete eligible files older than this duration. |
| `max_file_count` | — | Delete oldest eligible files when the count exceeds this limit. |
| `state_dir` | `.alloy/state` | Holds sync state and the shared device credentials. |
| `txlog_path` | Derived from `state_dir` | Optional path for append-only upload bookkeeping. |
Files currently uploading or open by another process are not removed by cleanup.
### Lifecycle [#lifecycle]
Lifecycle controls the original and redacted artifact independently:
```yaml
lifecycle:
original:
after: keep
cleanup:
include_in_storage_cleanup: true
redacted:
after: move
move_to: .alloy-redacted
cleanup:
include_in_storage_cleanup: true
```
| Field | Default | Purpose |
| -------------------------------------------- | ---------------------------- | ------------------------------------------------------------------------ |
| `.after` | `keep` | `keep`, `delete`, or `move` the stage output. |
| `.move_to` | Stage-specific dot-directory | Destination when `after: move`; relative paths resolve from `input_dir`. |
| `.cleanup.include_in_storage_cleanup` | `true` | Include moved files in the global size, age, and count limits. |
The cleanup default above is an important v0.7 behavior. It changes to `false` in
v0.8.
### Redaction [#redaction]
v0.7 configures transformation through one flat `redaction` block:
```yaml
redaction:
enabled: true
rules_file: .alloy/redaction.yaml
on_rule_error: skip_record
on_reader_error: skip_tail
output_compression: inherit
output_suffix: redacted
audit:
jsonl_path: .alloy/state/redaction-audit.jsonl
embed_in_mcap: true
```
| Field | Default | Purpose |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `enabled` | `false` | Enable transformation for eligible files. |
| `rules_file` | — | Required rules file when redaction is enabled. |
| `on_rule_error` | `skip_record` | `skip_record`, `skip_file`, or explicitly fail open with `pass_original`. |
| `on_reader_error` | `skip_tail` | `skip_tail`, `abort`, or `recover` malformed MCAP data. |
| `output_compression` | `inherit` | `inherit`, `none`, `zstd`, or `lz4`. |
| `output_suffix` | `redacted` | Infix inserted before the file extension. |
| `audit.jsonl_path` | — | Optional append-only audit sidecar. |
| `audit.embed_in_mcap` | `true` | Embed the audit record into the transformed MCAP. |
Unlike v0.8, v0.7 has no `version: 1`, structured `cleanup:` block,
`lifecycle.transform`, or ordered `pipeline:`. Use `lifecycle.redacted` and the
flat `redaction:` block described in the
[Redaction guide](/docs/mesh-storage/manage/redact).
For complete transform-rule syntax, use the
[Redaction reference](/docs/mesh-storage/reference/redaction).
# Configuration — v0.8.5
This reference is for v0.8.5. Do not copy these files into a v0.9 installation.
The v0.8 Track Folder flow uses two YAML files:
| File | Loaded by | Purpose |
| ------------------- | -------------------- | ------------------------------------------------------------------------- |
| `edge-manager.yaml` | `alloy-edge manager` | Cloud connection, device identity, and local process supervision |
| `edge-sync.yaml` | `alloy-edge sync` | Folder watching, uploads, cleanup, lifecycle, and the processing pipeline |
Durations use Go-style strings such as `5s`, `1m`, and `72h`. Sizes and bandwidth
limits use values such as `500MB`, `10GB`, and `5MB`.
Print the complete commented templates shipped with your installed binary:
```bash
alloy-edge manager sample-config
alloy-edge sync sample-config
```
## `edge-manager.yaml` [#edge-manageryaml]
### Top-level fields [#top-level-fields]
| Field | Purpose |
| ------------- | ---------------------------------------------------------------------------------------------------------------- |
| `backend_url` | Alloy backend endpoint. Use the value generated by **Add device**; `ALLOY_BACKEND_URL` overrides it. |
| `state_dir` | Manager state, credentials, and child configuration. Defaults to `.alloy/state`; `ALLOY_STATE_DIR` overrides it. |
| `seed_state` | Bootstrap identity and transport used until the device is approved. |
| `local_state` | Locally owned tags and processes that cloud configuration cannot overwrite. |
### `seed_state` [#seed_state]
| Field | Default | Purpose |
| -------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api_key` | — | Provisioning key used until approval. `ALLOY_PROVISIONING_KEY` overrides it before approval; `ALLOY_API_KEY` overrides the permanent key at runtime. |
| `edge_id` | Hostname, then generated UUID | Stable device identifier reported to Alloy. |
| `tags` | `{}` | Free-form strings for grouping and filtering devices. |
| `transport.http.poll_secs` | `15` | Seconds between desired-state checks. |
| `processes` | `[]` | Bootstrap processes used before the first successful cloud sync. Prefer `local_state.processes` for locally owned processes. |
### `local_state.processes` [#local_stateprocesses]
| Field | Default | Purpose |
| --------------- | -------------- | --------------------------------------------------------------------------------- |
| `name` | — | Required process name used in logs and reported state. |
| `command` | — | Required command to execute. |
| `enabled` | `true` | Keep the process declared but stopped when `false`. |
| `restart` | `on_failure` | Restart policy: `always`, `on_failure`, or `never`. |
| `shell` | `false` | Run through `/bin/sh -c` when shell expansion is required. |
| `trigger` | Start on apply | Start at `boot`, on a cron schedule, or when connectivity changes. |
| `duration` | — | Stop the process after a bounded duration. |
| `requires_auth` | `false` | Hold the process until Alloy approves the device. Set this for `alloy-edge sync`. |
| `files` | `{}` | Inline configuration files attached to the process. |
A locally owned Track Folder sync process has this shape:
```yaml
local_state:
processes:
- name: edge-sync
command: alloy-edge sync --config /data/recordings/.alloy/edge-sync.yaml
enabled: true
restart: on_failure
trigger: boot
requires_auth: true
```
## `edge-sync.yaml` [#edge-syncyaml]
v0.8 uses strict schema version 1. Only `input_dir` is required. Cleanup limits
default to no size or age limit, with a default maximum of 100,000 files.
```yaml
version: 1
input_dir: /data/recordings
file_pattern: "*.mcap,*.json,*.jsonl"
upload_delay: 30s
cleanup:
max_folder_size: 10GB
max_file_age: 72h
max_file_count: 1000
lifecycle:
original:
after: keep
```
### Scanning and upload [#scanning-and-upload]
| Field | Default | Purpose |
| ------------------------ | ----------------------- | --------------------------------------------------------------------------------- |
| `version` | — | Set to `1` for the v0.8 schema. An absent value enters legacy compatibility mode. |
| `input_dir` | — | Required folder to scan. |
| `file_pattern` | `*.mcap,*.json,*.jsonl` | Comma-separated file globs. `.tmp` entries are reserved and removed. |
| `cycle_time` | `1s` | Safety-net scan interval. |
| `upload_delay` | `30s` | Minimum age before a footer-less recording is eligible. |
| `upload_order` | `oldest_first` | Dispatch ready files using `oldest_first` or `newest_first`. |
| `mcap_require_footer` | `false` | Require an MCAP footer instead of falling back to file age. |
| `upload_type` | `signed_url` | `signed_url`, `none`, or feature-gated `opendal`. |
| `bwlimit` | Unlimited | Outbound bandwidth limit. |
| `max_concurrent_uploads` | `1` | Number of files uploaded in parallel. |
| `part_concurrency` | `4` | Multipart upload parts sent in parallel for one file. |
| `metadata` | `{}` | Free-form metadata included in upload requests. |
| `scan_exclude` | `[]` | Subdirectory basenames to skip; dot-directories are always skipped. |
| `fs_event_enabled` | `true` | Use filesystem events for low-latency scans while retaining periodic scans. |
### Cleanup and upload state [#cleanup-and-upload-state]
| Field | Default | Purpose |
| ------------------------- | ------------------------ | -------------------------------------------------------------------------------------- |
| `cleanup.max_folder_size` | Unbounded | Delete oldest eligible files when the tracked folder exceeds this size. |
| `cleanup.max_file_age` | — | Delete eligible files older than this duration. |
| `cleanup.max_file_count` | `100000` | Delete oldest eligible files when the count exceeds this limit. |
| `state_dir` | `.alloy/state` | Holds sync state and the shared device credentials. |
| `index_store.backend` | `sqlite` | Per-file upload state: `sqlite` or `jsonl`. Switching backends migrates automatically. |
| `index_store.path` | Derived from `state_dir` | Optional JSONL path and SQLite migration source. |
### Lifecycle [#lifecycle]
Lifecycle controls what happens to the original and to each transformed artifact:
```yaml
lifecycle:
original:
after: keep
upload: false
cleanup:
include_in_storage_cleanup: false
transform:
after: move
move_to: .alloy-redacted
cleanup:
include_in_storage_cleanup: false
```
| Field | Default | Purpose |
| -------------------------------------------- | ---------------------------- | ------------------------------------------------------------------------- |
| `original.after` | `keep` | `keep`/`keep_continue`, `keep_stop`, `delete`, or `move` the source. |
| `transform.after` | `keep` | `keep`, `delete`, or `move` the transformed artifact. |
| `.move_to` | Stage-specific dot-directory | Destination when `after: move`; relative paths resolve from `input_dir`. |
| `.cleanup.include_in_storage_cleanup` | `false` | Include moved files in the global size, age, and count limits. |
| `original.upload` | `false` | Upload the untransformed original if it reaches the end of the pipeline. |
| `transform.on_rule_error` | `skip_record` | `skip_record`, `skip_file`, or explicitly fail open with `pass_original`. |
| `transform.on_reader_error` | `skip_tail` | `skip_tail`, `abort`, or `recover` malformed MCAP data. |
| `transform.output_compression` | `inherit` | `inherit`, `none`, `zstd`, or `lz4`. |
| `transform.audit` | Embedded audit enabled | Configure the JSONL sidecar and embedded MCAP audit record. |
### Processing pipeline [#processing-pipeline]
The optional ordered pipeline can filter, transform, upload, and retain different
artifacts from one recording:
```yaml
pipeline_trigger: delay-after-close
pipeline:
- transform: .alloy/redaction.yaml
transform_suffix: redacted
upload: true
transform_after: delete
original_after: delete
file_pattern: "**/camera/*.mcap"
filter:
require_topics: ["/nav/**"]
min_duration: 60s
min_messages: 100
min_size: 10MB
```
| Field | Default | Purpose |
| ------------------ | --------------------------- | ------------------------------------------------------------------------------------------- |
| `pipeline_trigger` | `delay-after-close` | Start after the close delay, or use `close` to start on a close event. |
| `transform` | — | Rules file used to create a transformed artifact. |
| `transform_suffix` | Step index | Infix added to the transformed filename. |
| `upload` | `false` | Upload the transformed artifact. |
| `original_after` | `lifecycle.original.after` | `keep`/`keep_continue`, `keep_stop`, `delete`, or `move` the source after this step. |
| `transform_after` | `lifecycle.transform.after` | Keep, delete, or move the transformed artifact. |
| `file_pattern` | All files | Restrict the step to matching paths. |
| `filter` | — | Gate the step by footer, topics, duration, message count, file size, or content expression. |
An unstamped v0.7-style sync file is migrated in memory. To inspect or write the
v0.8 form explicitly:
```bash
alloy-edge migrate --config /data/recordings/.alloy/edge-sync.yaml --dry-run
alloy-edge migrate --config /data/recordings/.alloy/edge-sync.yaml
```
For transform-rule syntax, use the
[Redaction reference](/docs/mesh-storage/reference/redaction).
# Configuration — v0.9.0
This is an advanced reference. Start with
[Track Folder v0.9.0](/docs/mesh-storage/ingest/track-folder/0.9.0); `link` creates
the required files and `run` selects them from the installation scope.
## Files and locations [#files-and-locations]
| File | Purpose |
| ------------------- | --------------------------------------------------------------------------------- |
| `edge-manager.yaml` | Cloud connection, identity seed, process supervision, and restart policy |
| `edge-sync.yaml` | Tracked folder, upload behavior, disk cleanup, lifecycle, and processing pipeline |
| Transform YAML | Optional per-recording transformation or redaction rules |
| Scope | Configuration | State |
| ------------ | ------------------------------------------------------- | ----------------------------------------------------------- |
| User | `$XDG_CONFIG_HOME/alloy-edge` or `~/.config/alloy-edge` | `$XDG_STATE_HOME/alloy-edge` or `~/.local/state/alloy-edge` |
| Folder-local | `/.alloy` | `/.alloy/state` |
| System | `/etc/alloy-edge` | `/var/lib/alloy-edge` |
Use `--user`, `--system`, or `--local` to select an installation explicitly.
Without a scope, Alloy Edge discovers the nearest valid folder-local installation,
then the user installation, then the system installation. A malformed installation
is an error; it does not silently fall back.
## `edge-manager.yaml` [#edge-manageryaml]
### Top-level fields [#top-level-fields]
| Field | Default | Purpose |
| ------------- | ------------------------------ | --------------------------------------------------------------------------------------------------- |
| `backend_url` | Alloy production backend | Alloy backend endpoint. `ALLOY_BACKEND_URL` overrides it. |
| `state_dir` | Scope-specific state directory | Manager state, credentials, desired state, and child configuration. `ALLOY_STATE_DIR` overrides it. |
| `seed_state` | `{}` | Bootstrap identity and transport used until the device is approved. |
| `local_state` | — | Locally owned tags and processes that cloud configuration cannot overwrite. |
| `supervisor` | Built-in defaults | Advanced restart, backoff, stability, and shutdown tuning. |
Credentials and device identity live under the installation state directory.
Deleting state is not a harmless cache clear. Use `alloy-edge unlink` with the
installation's scope when intentionally removing a linked installation.
### `seed_state` [#seed_state]
`alloy-edge link` writes and manages credentials for new installations. The seed
fields remain available for legacy or manually authored installations:
| Field | Default | Purpose |
| -------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `api_key` | — | Enrollment-only provisioning key used until approval. Prefer `alloy-edge link`; runtime commands use the persisted device key. |
| `edge_id` | Hostname, then generated UUID | Stable device identifier reported to Alloy. |
| `tags` | `{}` | Free-form strings for grouping and filtering devices. |
| `transport.http.poll_secs` | `15` | Seconds between desired-state checks. |
| `processes` | `[]` | Bootstrap processes used before the first successful cloud sync. Prefer `local_state.processes` for locally owned processes. |
### `local_state.processes` [#local_stateprocesses]
| Field | Default | Purpose |
| --------------- | -------------- | --------------------------------------------------------------------------------- |
| `name` | — | Required process name used in logs and reported state. |
| `command` | — | Required command to execute. |
| `enabled` | `true` | Keep the process declared but stopped when `false`. |
| `restart` | `on_failure` | Restart policy: `always`, `on_failure`, or `never`. |
| `shell` | `false` | Run through `/bin/sh -c` when shell expansion is required. |
| `trigger` | Start on apply | Start at `boot`, on a five-field cron schedule, or when connectivity returns. |
| `duration` | — | Stop the process after a bounded duration. |
| `requires_auth` | `false` | Hold the process until Alloy approves the device. Set this for `alloy-edge sync`. |
| `files` | `{}` | Inline configuration files attached to the process. |
Set `local_state.report: true` to report a safe, read-only projection
of locally pinned state to Alloy. The default is `false`.
A locally owned Track Folder sync process has this shape:
```yaml
local_state:
report: true
processes:
- name: sync
command: alloy-edge sync -c /data/recordings/.alloy/edge-sync.yaml
enabled: true
restart: on_failure
trigger: boot
requires_auth: true
```
### `supervisor` [#supervisor]
| Field | Default | Purpose |
| ----------------------- | ------- | ---------------------------------------------------------------------------- |
| `max_restarts` | `10` | Restarts allowed before a child is marked crashed. |
| `max_backoff_secs` | `60` | Maximum exponential restart backoff. |
| `stable_secs` | `60` | Runtime required before the restart count resets. Must be greater than zero. |
| `graceful_timeout_secs` | `10` | Grace after `SIGTERM` before `SIGKILL`. |
## `edge-sync.yaml` [#edge-syncyaml]
v0.9 uses strict schema version 1. Only `input_dir` is required. Cleanup has no
size or age limit by default, with a default maximum of 100,000 files.
```yaml
version: 1
input_dir: /data/recordings
file_pattern: "*.mcap,*.json,*.jsonl"
upload_delay: 30s
cleanup:
max_folder_size: 10GB
max_file_age: 72h
max_file_count: 1000
lifecycle:
original:
after: keep
```
Durations use Go-style strings such as `5s`, `1m`, and `72h`. Sizes and
bandwidth limits use values such as `500MB`, `10GB`, and `5MB`.
### Scanning and upload [#scanning-and-upload]
| Field | Default | Purpose |
| ------------------------ | ----------------------- | ------------------------------------------------------------------------------------------- |
| `version` | — | Set to `1` for the current strict schema. An absent value enters legacy compatibility mode. |
| `input_dir` | — | Required folder to scan. |
| `file_pattern` | `*.mcap,*.json,*.jsonl` | Comma-separated file globs. `.tmp` entries are reserved and removed. |
| `cycle_time` | `1s` | Safety-net scan interval. |
| `upload_delay` | `30s` | Minimum age before a footer-less recording is eligible. |
| `upload_order` | `oldest_first` | Dispatch ready files using `oldest_first` or `newest_first`. |
| `mcap_require_footer` | `false` | Require an MCAP footer instead of falling back to file age. |
| `upload_type` | `signed_url` | `signed_url`, `none`, or feature-gated `opendal`. |
| `bwlimit` | Unlimited | Outbound bandwidth limit. |
| `max_concurrent_uploads` | `1` | Number of files uploaded in parallel. |
| `part_concurrency` | `4` | Multipart upload parts sent in parallel for one file. |
| `metadata` | `{}` | Free-form metadata included in upload requests when supported. |
| `scan_exclude` | `[]` | Subdirectory basenames to skip; dot-directories are always skipped. |
| `fs_event_enabled` | `true` | Use filesystem events for low-latency scans while retaining periodic scans. |
The `upload_settings` block exposes advanced protocol controls:
| Field | Default | Purpose |
| ---------------------------- | -------------------------- | ----------------------------------------------------------------------------------- |
| `multipart` | `true` | Use the unified `/upload/*` protocol. Set `false` for the legacy single-PUT broker. |
| `multipart_endpoint` | `/upload/init` | Override the unified-protocol initialization endpoint. |
| `multipart_include_metadata` | `false` | Include `metadata` in the multipart initialization request. |
| `signed_url_endpoint` | Top-level/default endpoint | Override the legacy single-PUT broker endpoint. |
| `part_concurrency` | Top-level/default value | Override concurrent part uploads within one multipart upload. |
### Cleanup and upload state [#cleanup-and-upload-state]
| Field | Default | Purpose |
| ------------------------- | ------------------------------ | -------------------------------------------------------------------------------------- |
| `cleanup.max_folder_size` | Unbounded | Delete oldest eligible files when the tracked folder exceeds this size. |
| `cleanup.max_file_age` | — | Delete eligible files older than this duration. |
| `cleanup.max_file_count` | `100000` | Delete oldest eligible files when the count exceeds this limit. |
| `state_dir` | Scope-specific state directory | Holds sync state and shared device credentials. |
| `index_store.backend` | `sqlite` | Per-file upload state: `sqlite` or `jsonl`. Switching backends migrates automatically. |
| `index_store.path` | Derived from `state_dir` | Optional JSONL path and SQLite migration source. |
Files currently uploading or open by another process are not removed by cleanup.
Deprecated top-level cleanup fields, `txlog_path`, `credentials_dir`, `keep_files`,
and the flat `redaction` block remain readable for compatibility but should not be
used in new v0.9 configuration.
### Lifecycle [#lifecycle]
Lifecycle controls what happens to the original and to each transformed artifact:
```yaml
lifecycle:
original:
after: keep
upload: false
cleanup:
include_in_storage_cleanup: false
transform:
after: move
move_to: .alloy-redacted
cleanup:
include_in_storage_cleanup: false
on_rule_error: skip_record
on_reader_error: skip_tail
output_compression: inherit
audit:
jsonl_path: .alloy/state/redaction-audit.jsonl
embed_in_mcap: true
max_bytes: 5MB
max_files: 2
```
| Field | Default | Purpose |
| -------------------------------------------- | ---------------------------- | ------------------------------------------------------------------------ |
| `original.after` | `keep` | `keep`/`keep_continue`, `keep_stop`, `delete`, or `move` the source. |
| `original.upload` | `false` | Upload the untransformed original if it reaches the end of the pipeline. |
| `transform.after` | `keep` | `keep`, `delete`, or `move` the transformed artifact. |
| `.move_to` | Stage-specific dot-directory | Destination when `after: move`; relative paths resolve from `input_dir`. |
| `.cleanup.include_in_storage_cleanup` | `false` | Include moved files in the global size, age, and count limits. |
| `transform.on_rule_error` | `skip_record` | `skip_record`, `skip_file`, or explicitly opt into `pass_original`. |
| `transform.on_reader_error` | `skip_tail` | `skip_tail`, `abort`, or `recover` malformed MCAP data. |
| `transform.output_compression` | `inherit` | `inherit`, `none`, `zstd`, or `lz4`. |
| `transform.audit.jsonl_path` | — | Write an optional rotating JSONL audit sidecar. |
| `transform.audit.embed_in_mcap` | `true` | Embed the audit record in the transformed MCAP. |
| `transform.audit.max_bytes` | `5MB` | Roll the sidecar before it exceeds this size; `0` disables rotation. |
| `transform.audit.max_files` | `2` | Rotated generations to retain; must be at least one. |
### Processing pipeline [#processing-pipeline]
The optional ordered pipeline can filter, transform, upload, and retain different
artifacts from one recording:
```yaml
pipeline_trigger: delay-after-close
pipeline:
- transform: .alloy/redaction.yaml
transform_suffix: redacted
upload: true
transform_after: delete
original_after: delete
file_pattern: "**/camera/*.mcap"
filter:
require_topics: ["/nav/**"]
min_duration: 60s
min_messages: 100
min_size: 10MB
if: "done == true"
if_at: any
```
| Field | Default | Purpose |
| ---------------------------- | --------------------------- | -------------------------------------------------------------------------------- |
| `pipeline_trigger` | `delay-after-close` | Start after the close delay, or use `close` to start as soon as the file closes. |
| `transform` | — | Rules file used to create a transformed artifact. |
| `transform_suffix` | Step index | Infix added to the transformed filename. |
| `upload` | `false` | Upload the transformed artifact. Requires `transform`. |
| `original_after` | `lifecycle.original.after` | Control source flow after this step. |
| `transform_after` | `lifecycle.transform.after` | Keep, delete, or move the transformed artifact. |
| `file_pattern` | All files | Restrict the step to matching paths. |
| `filter.mcap_require_footer` | — | Wait and retry when an MCAP footer is absent. |
| `filter.require_topics` | `[]` | Require matching topics; glob patterns are supported. |
| `filter.min_duration` | — | Require a minimum recording duration. |
| `filter.min_messages` | — | Require a minimum message count. |
| `filter.min_size` | — | Require a minimum file size. |
| `filter.if` | — | Evaluate a Jinja2 expression against JSON or MCAP record content. |
| `filter.if_at` | `any` | Select records for `if`: `any`, `first`, `last`, an index, or a range. |
| `filter.require_fields` | — | Equality-map shorthand for simple `if` conditions. |
All configured filter conditions must match for a step to run. A content miss
continues to the next step rather than failing the file.
## Inspect, validate, and edit configuration [#inspect-validate-and-edit-configuration]
The v0.9 CLI is the canonical source for the complete schema installed on a
device:
```bash
# Authored manager and every referenced sync/transform file
alloy-edge config show
# Deterministic JSON plus the complete Draft 2020-12 schemas
alloy-edge config show --json --schema
# Validate the selected installation and every referenced file
alloy-edge config check
```
Select `manager` or an exact configured process name to narrow `config show`.
Add `--no-comments` for plain YAML, or use `--json --comments` for source-ordered
comments alongside the semantic JSON projection.
Prefer the configuration CLI for small edits. It preserves unrelated comments
and styles, validates a staged candidate, and atomically replaces the source:
```bash
alloy-edge config set track-folder /data/recordings
alloy-edge config set sync.input_dir /data/recordings
alloy-edge config set manager.state_dir /var/lib/alloy-edge
```
Process names form the first segment of an address, such as `sync.input_dir`.
Use `alloy-edge config show` to see the configured process names before editing.
After changing configuration, restart `alloy-edge run` or its service. Validate a
change without backend calls or child processes with:
```bash
alloy-edge run --user --dry-run
```
For transformation rules, continue to the
[Redaction reference](/docs/mesh-storage/reference/redaction).
# Configuration
Choose the version installed on your device. This reference covers advanced
configuration; most v0.9.0 Track Folder installations only need `alloy-edge link`
and `alloy-edge run`.