Usage metering¶
Usage metering measures how much each organization uses, so that it can be billed for it. It measures two things:
- Storage: the objects (files) an organization keeps in S3, in GB-hours. One GB-hour is 1 GiB stored for one hour. Every object counts for at least one hour.
- GPU compute: the time its training jobs run on GPUs, in GPU-hours.
AWS cannot give these numbers per organization. Its storage metrics are one total per bucket, once a day. So the system records every object create and delete itself. It adds the results up per organization and per day. At the end of a billing period, it turns them into the numbers on an invoice. Checks run all the time to compare these numbers with what AWS really holds. A lost message becomes an alarm, not a wrong bill.
The invoice shows storage in GB-months and GPU time in GPU-hours. Prices come from the organization's subscription plan.
How it works¶
S3 data bucket ──► EventBridge ──► SQS queue ──► metering-ingest ──► object_events
orgs/{org_id}/… │ (what S3 said)
└──► raw archive (Firehose → S3) │
│ fold, every 5 min
▼
object_lifetimes + object_size_intervals
│ rollup, nightly
multipart sweep, hourly ─────────────────────────────────────►│◄──── GPU jobs
▼
usage_daily (organization × UTC day)
│ 48 h after the period ends
▼
closure ──► invoice (drafted, inserted, finalized)
│ nightly drift scan
▼
adjustments (changes found after a closure)
- An organization's object is stored in the data bucket under
orgs/{org_id}/…. The folder name tells us which organization owns it. - S3 sends a message to EventBridge. EventBridge puts it on a queue, and a copy goes to the raw archive.
- The
metering-ingestfunction writes one row per message intometering.object_events: "created, with this size" or "removed". It only records. It does not decide anything. - Every 5 minutes the fold reads the new events. It builds one lifetime per object (from when to when) and its size intervals (an object that is overwritten has several sizes over its life).
- Every night at 03:30 UTC the rollup adds this up into one row per
organization per UTC day in
usage_daily. The application reads a copy of this table,public.usage_daily, to show usage. - A billing period can be invoiced 48 hours after it ends, once every day in
it is final. Then the application asks the database for a draft invoice. The
database creates a closure (it freezes the period's figures), prices
them from the plan, and returns every invoice column. The application
inserts the invoice and then calls finalize.
How it works §7.2 has a short note on what the
application calls, and what it leaves alone.
§7.3 is how to freeze a
period and how to read
usage_period_closures. - A closed period can still change later, for example because of a late message or a repair. The nightly drift scan records the difference as an adjustment. A small adjustment is added to the next invoice. A large one, or one for an organization's last period, needs a corrected invoice.
Around this main path, other jobs check that nothing is lost:
- Reconciliation, every night at 02:15 UTC. AWS's own daily list of every object (S3 Inventory) is compared with the ledger, object by object. Each difference is recorded. When repairs are switched on, it is also fixed.
- Multipart sweep, every hour. A large upload uses storage while it is in progress, and also after it is abandoned. The sweep finds these uploads. An upload that stays open for more than 3 hours is billed from hour 3.
- Canary, every hour. It writes and deletes a test object with a difficult name in a test organization's folder. Then it checks that the object arrived in the ledger exactly. If the pipeline stops, the canary fails within two hours.
- Health, every 5 minutes. It publishes the pipeline's key numbers to CloudWatch. Alarms send email to the alerts topic.
- Raw archive. Every S3 message is also kept in a separate bucket for 7 years. It is evidence if a customer disputes a bill. It also makes it possible to rebuild the ledger.
GPU. A training job reports when it starts, sends a heartbeat every minute, and reports when it ends. A job whose heartbeats stop is closed by the reaper.
Off by default¶
Three parts are built and tested but start switched off. An operator switches each one on when its condition is met. Operations has the steps.
| Part | Setting | Switch it on when |
|---|---|---|
| GPU metering | GPU_METERING_ENABLED=false |
The training launcher sends start, heartbeat and end for every GPU job |
| Inventory repairs | APPLY=false |
About a week of nightly reconciliation findings has been read and explained |
| Pruning of old rows | PRUNE=false |
The retention margin has been chosen. Pruning is the only step that cannot be undone |
Words used in these pages¶
| Word | Meaning |
|---|---|
| Event | One S3 message as received: an object created (with its size) or removed |
| Fold | The job that runs every 5 minutes and turns events into lifetimes and size intervals |
| Lifetime | One object's existence: when it was created and when it was deleted (empty while it exists) |
| Size interval | A part of a lifetime at one size. An overwrite starts a new size interval |
| Ledger | Lifetimes and size intervals together: what existed, when, and at what size |
| GB-hour | 1 GiB (1,073,741,824 bytes) stored for one hour |
| GPU-hour | One GPU used for one hour |
| Billing period | One subscription period of an organization, usually a month |
| Closure | One row of usage_period_closures: the fixed figures of a finished billing period. The application creates it through draft_usage_invoice or freeze_usage_period, and never writes the table itself. How it works §7.3 |
| Adjustment | A difference found after a period is closed. It is billed on a later invoice or a corrected invoice |
| Quarantine | Messages or objects that cannot be assigned to an organization or processed. A person must look at them. They are never deleted without a decision |
| Reconciliation | The nightly comparison between the ledger and the S3 Inventory report |
| Diff A, B, C, D | A: in S3, not in the ledger (a missed create). B: in the ledger, not in S3 (a missed delete). C: the sizes differ. D: a delete with no create, while the object is still in S3 |
| Repair | A synthetic event that reconciliation writes to correct a difference |
| Pipeline epoch | The time measuring started. Objects older than this were accepted as the starting set |
| Canary | The hourly test object |
| Retention horizon | The point in time before which pruning may delete old rows |
| Dry run | A run that reports what it would do and changes nothing |
Where to read next¶
| Page | Read it to |
|---|---|
| How it works | Understand the design and the rules that keep the numbers correct. Code comments cite its sections as §N, for example §3.4 for the fold |
| Operations | Deploy it, configure the functions, act on an alarm, read the nightly findings, and switch on repairs, pruning or GPU metering |
| Schema reference, §4.8 | Look up a metering table, column or function |
lambda/metering-*/README.md |
Change or redeploy one function |
Where the code is¶
| Path | Contents |
|---|---|
database/schemas/metering.sql |
The metering schema: its tables and roles |
database/queries/metering_*.sql |
All behavior: ingest, fold, billing, invoices, GPU, multipart uploads, scheduled jobs, reconciliation, health, access rules, and the functions the application calls |
database/apply_metering.sh |
Applies all the metering_*.sql files in one transaction |
backend/src/db/migrations/103_*.sql to 108_*.sql |
The migrations for a database that already has the application schema |
database/tests/ |
The database tests (run_pipeline.sh runs all of them on PostgreSQL 18) |
lambda/metering-*/ |
The functions: ingest, jobs, inventory loader, multipart sweep, canary, health, and the GPU cost check |
infra/terraform/modules/usage_metering/ |
Terraform for part of the AWS resources: the queues, the event rule, the archive stream, four of the functions, their schedules and some alarms. Operations lists every part |