General

Toborverse VPC set up

1. A user sends an HTTP/HTTPS request from the internet to the Application Load Balancer (ALB) sitting in the public subnets.
2. The ALB checks its target group and routes the incoming request to your private EC2 instance over the target port.
3. The private EC2 instance receives and processes the request locally through the running application process.
4. If the app needs database access, it queries the Aurora RDS cluster securely inside the isolated data subnets.
5. If the app interacts with S3 or DynamoDB, traffic routes internally through free Gateway Endpoints without hitting the internet.
6. If the app requires external internet access (like downloading packages), traffic is routed out via the Regional NAT Gateway.
7. The application sends the generated response back from the private EC2 instance to the ALB.
8. The ALB transmits the final response back over the internet to the user's browser.
9. For code deployments, GitHub Actions authenticates via OIDC and executes updates on the private EC2 instance using AWS SSM instead of SSH.

Extraction

The robot asks the API for presigned S3 URLs, uploads the archive in parts, and then tells the API the upload is complete. The archive is a zstd-compressed tar (.tar.zst) at orgs/<org_id>/episodes/<episode_id>/…. The episode row already exists. The worker updates it.

1. Upload data to S3. The robot uploads the archive to the data bucket.
2. Sends message to SQS. S3 sends the upload notification to the SQS queue.
3. Wakes up worker. The EventBridge rule starts one Fargate task. The task long-polls SQS, downloads the archive, and decompresses it without expanding the whole file on disk.
4. If success. Fargate writes each .mp4 and .parquet back to S3 under …/processed/, and transcodes each .mp4 to HLS. It writes rows and episode metadata to RDS (Postgres), and the episode status becomes processed. The Toborverse platform reads that result. The SQS message is deleted only after this run succeeds.
5. If fails. The message stays on SQS. After three receives it moves to the DLQ.
6. Email alert. CloudWatch alarms the DLQ, SNS publishes to the alerts topic, and an email notification goes out.

Columns, failure cases, and deploy steps are in Extraction worker.

Usage metering

1. Metering records each organization's S3 storage (GB-hours) and GPU time (GPU-hours) so a billing period can be invoiced.
2. Billable objects live in the data bucket under orgs/{org_id}/models/.... The first folder is the organization.
3. S3 sends each create and each delete to EventBridge. The rule puts the event on the ingest queue and sends a copy to the raw archive.
4. metering-ingest writes one row per message into metering.object_events: created, with a size, or removed. It records what S3 said.
5. Every 5 minutes the fold turns new events into one lifetime per object and its size intervals. An overwrite starts a new size interval.
6. Every night at 03:30 UTC the rollup writes one row per organization per UTC day into usage_daily. The application reads public.usage_daily.
7. A period can be invoiced 48 hours after it ends, once every day in it is final. The database freezes the figures in a closure, prices them from the plan, and returns the invoice columns. The application inserts the invoice and then calls finalize.
8. A closed period can still change, from a late message or a repair. The nightly drift scan records the difference as an adjustment.
9. A small adjustment is added to the next invoice. A large one, or one for an organization's last period, needs a corrected invoice.
10. Reconciliation, the multipart sweep, the canary, and health checks are in Usage metering. How to run the pipeline is in Operations.