AWS S3 Data Integration

Connect your AWS S3 bucket to Alloy for automatic data ingestion. Once set up, Alloy watches your bucket and ingests new files as they appear — no manual uploads needed.

Setup time: ~5 minutes

Why S3 integration?

Easy — fast setup and simple to maintain
Secure — uses permission-based IAM access and cloud security
Fast — leverages cloud infrastructure for fast data transfer
Controlled — you control the access point and what data is made available

Prerequisites

An AWS account with IAM permissions to create policies and roles
An S3 bucket containing your mission data

Step 1: Create an IAM Policy

In the AWS IAM console, go to Policies → Create policy

Switch to the JSON editor and paste the following policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": [
        "arn:aws:s3:::<S3-Bucket-Name>",
        "arn:aws:s3:::<S3-Bucket-Name>/*"
      ]
    }
  ]
}

Replace <S3-Bucket-Name> with the name of your S3 bucket.

Step 2: Create a Role

After the policy is created, head to the IAM Roles tab and create a new role

Attach the policy you created above

Click Create role

Select trusted entity Web identity, with identity provider Google (accounts.google.com)

Step 3: Update Trust Policy

In the IAM Roles console, select the role you created and click on the Trust Relationships tab

Click Edit Trust Policy, update with the following

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "accounts.google.com:sub": "115295746068891424597"
        }
      }
    }
  ]
}

Sometimes AWS will require the aud claim accounts.google.com:aud when first creating the trust policy. Make sure to remove this later.

Step 4: Send us your details

Contact the Alloy team and provide:

S3 bucket name — the bucket you configured in the policy
AWS IAM role ARN — found on the role's summary page (e.g., arn:aws:iam::123456789:role/alloy-access-role)

We'll configure the connection on our end and your data will start flowing in automatically.

Pull frequency

Once your integration is set up, choose how often Alloy polls your S3 bucket from Org settings → Integrations → Pull frequency. Options: 1 min, 5 min, 15 min, 30 min, 1 hour. Default is 5 min.

On S3, polling cost is paid by you (the bucket owner), via S3 LIST API charges (~$0.005 per 1,000 requests). Each poll is a full bucket walk paginated at 1,000 keys per LIST. Rough monthly cost at 5-min polling:

100K objects → ~$0.43/month
1M objects → ~$4.32/month
10M objects → ~$43/month

Cost scales linearly with object count and inversely with poll interval — 1-min polling is 5× the cost of 5-min, 1-hour is 12× cheaper. Pick the slowest cadence that meets your latency needs.

Folder structure and supported files

See Uploading your first mission for details on folder structure expectations, metadata.json format, and supported file types. These apply to all data ingestion methods including S3.