How usage metering works¶

Comments in the metering SQL and Lambda code cite sections of this page as §N, for example §3.4.

This page describes the design of the usage-metering system: how it produces its numbers and the rules that keep them correct. To run the system, see Operations. For every table and column, see the schema reference.

The system measures three things for each organization: objects stored in S3 (GB-hours), multipart uploads left open (GB-hours) and GPU compute (GPU-hours). AWS cannot give these figures per organization. Its storage metrics are one total per bucket, once a day. A daily total cannot tell an object kept for 24 hours from one kept for 40 minutes. Storage is billed by the hour, so the system records the life of every object itself.

The words used on this page are defined in the overview.

1. Assumptions¶

Each assumption must stay true. If one breaks, the system produces wrong numbers, not an error. So each assumption has a check.

#	Assumption	If it breaks	How it is checked
A1	Versioning is off on every metered bucket.	A delete leaves an old version that AWS still charges for, but the ledger closes the lifetime.	`metering-health` calls `GetBucketVersioning` on every run and publishes `BucketVersioningOk` (§8.3).
A2	Every billable object is under `orgs/{org_id}/`.	Nothing says who owns the object.	The bucket policy refuses writes outside `orgs/`. Ingest quarantines any other key as `unparseable_key` (§3.1).
A3	`org_id` is a valid UUIDv7 that exists in `organizations`.	A malformed segment creates an organization that does not exist.	Ingest quarantines a bad UUID as `malformed_uuid`. The fold parks an unknown org as `unknown_org` (§3.4). The ledger tables have a foreign key to `organizations`.
A4	Organizations are never hard-deleted.	Ledger foreign keys break.	`delete_org()` refuses an org with rows in `invoices` or `usage_daily` and says to call `deactivate_org()`, which sets `organizations.deactivated_at`.
A5	Subscription periods start and end at 00:00 UTC.	A period is not a whole number of hours, and the rounding correction must read every object (§6.4).	Constraints `subscriptions_period_start_utc_midnight` and `subscriptions_period_end_utc_midnight`. The invoice functions also refuse such a period (§7.2).
A6	A key is never used again after its object is deleted.	`(bucket, key)` stops identifying one lifetime. The schema depends on that identity.	The fold parks a create dated after the lifetime's `deleted_at` as `resurrected_key` (§3.4).
A7	Every `ObjectCreated` event carries a sequencer.	Two creates on one key cannot be ordered, so sizes after the first overwrite can be wrong.	Ingest quarantines the event as `ingest_error`, and `object_events.sequencer` is `NOT NULL` (§3.5).

A6 holds because each new episode gets a new episode_id, a unique UUIDv7, so a path that is free again is never written again. Objects are overwritten in place, though: updating an episode rewrites the same key, and S3 reports a second ObjectCreated with no ObjectRemoved. So an object's size changes over time (§3.2). This does not break A6, because it happens while the object exists. A6 applies to completed objects only: several multipart uploads can be open on one key, and a second completion is an ordinary overwrite (§4).

2. Architecture¶

The system is built in five layers. Each layer answers a question that the previous one raises.

2.1 The main path¶

S3 bucket -> EventBridge -> SQS -> metering-ingest -> object_events
  -> fold -> object_lifetimes + object_size_intervals -> rollup -> usage_daily -> closure -> invoice

Every other part of the system is attached to this path.

2.2 Why there are two kinds of table¶

SQS delivers messages out of order, and sometimes twice. One batch can hold an object's create and delete in either order. So ingest does not interpret events: metering-ingest writes down what S3 said, in object_events. The fold decides what the events mean later, when it sees all events of a key together. It writes two tables, because an object has two separate facts: object_lifetimes says when it existed, and object_size_intervals says how big it was, and when. This split lets ingest be one bulk INSERT with no reads (§3.3).

2.3 Three meters, one daily table¶

Three meters meet in metering.usage_daily:

objects, from the event path (§3);
multipart uploads: S3 sends no event for them, so an hourly sweep lists them (§4);
GPU jobs: heartbeats and a reaper limit the error for a crashed job (§5).

2.4 The safety net¶

Mechanism	Why it exists
Two dead-letter queues (§9.3)	Without them, a message that always fails is deleted at the retention limit, and nobody knows.
Quarantine (§3.1, §3.4)	Unattributed storage is real money with no owner. It is kept for a person, never dropped or guessed.
Raw event archive (§9.4)	The ledger can be rebuilt even if the ingest code wrote events wrongly.
Reconciliation (§8.2)	A missed delete bills a customer for ever. Only a key-by-key comparison with S3 Inventory finds it.
Canary (§8.3)	A stopped pipeline can look normal in totals for days. The canary fails within two hours.

2.5 Billing and closure¶

The rollup stores exact byte-seconds per day. When a period is invoiced, the closure stores both the exact and the rounded figure (§6.4), and the invoice is priced from it. A change that arrives later becomes an adjustment. The closure itself never changes (§7).

2.6 The whole flow¶

                    S3 data bucket  (orgs/<org_id>/...)
                          |  Object Created / Object Deleted
                          v
                    EventBridge rule ---- second target ----> Firehose -> raw event archive
                          |  `-- delivery fails --> EventBridge DLQ
                          v
                    SQS queue ---- received 5 times ----> SQS DLQ
                          |  batches of up to 100
                          v
                    metering-ingest ---- cannot attribute ----> usage_quarantine
                          |  one bulk INSERT, no reads
                          v
  reconciliation ---> object_events  (what S3 said)
  (02:15 UTC,             |  fold, every 5 minutes ---- cannot derive ----> fold_quarantine
   S3 Inventory)          v
                    object_lifetimes + object_size_intervals   (the ledger)
                          |  rollup, nightly at 03:30 UTC
                          v
  GPU intervals ----> usage_daily  <---- multipart sweep, hourly (mpu_lifetimes)
  (heartbeats,            |  same statement --> public.usage_daily (what the app shows)
   reaper)                |  draft_usage_invoice, 48 hours after the period ends
                          v
                    usage_period_closures (the closure) --> the app inserts the invoice
                          |  drift scan, nightly
                          v
                    usage_adjustments --> the next invoice, or a corrected invoice

  canary, hourly: writes and deletes a probe object; the ledger must show both
  health, every 5 minutes: metering.health_metrics() and AWS checks -> CloudWatch

3. Object storage¶

3.1 Path and attribution¶

Billable objects live under orgs/{org_id}/episode/{episode_id}/... or orgs/{org_id}/models/.... For each event, ingest reads org_id from the decoded key with ^orgs/([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})/ and checks that it is a UUIDv7. It does not look the org up; the fold does (§3.4).

Rule: when attribution fails, quarantine the whole event. Never drop it and never guess. The Lambda writes it to metering.usage_quarantine with one of these reason values:

unparseable_key: the key does not start with orgs/, or the bucket or key is missing;
malformed_uuid: the org segment is not a valid UUIDv7;
ingest_error: anything else. Examples are no sequencer, a create with no size, a bad timestamp and a key over 1,024 bytes. Others are a body that is not a JSON object, and a record that failed alone in the retry of §3.3.

raw_event holds the complete EventBridge envelope, so a replay needs nothing else; a body that was not JSON is stored as {"raw_body": ...}. The key column holds the key decoded, exactly as ingest used it (§3.6).

Resolving a usage_quarantine row means replaying the event, because it never reached object_events. metering.replay_quarantined(p_id, p_org_id, p_note) sends the event through metering.ingest_events(), then sets resolved_at, in one transaction. Pass p_org_id when a person has found the owner. It refuses a {"raw_body": ...} row, a create with no size and an event with no sequencer. A row that was never billable, such as an admin file outside orgs/, is resolved with a plain UPDATE.

3.2 Schema¶

Table	What it says	Primary key	Written by
`object_events`	What S3 said, not interpreted	`(bucket, key, event_type, sequencer)`	`metering.ingest_events()`; reconciliation repairs
`object_lifetimes`	When an object existed: its first create and its delete	`(bucket, key)`	the fold only
`object_size_intervals`	How big it was, and when: one row per create	`(bucket, key, sequencer)`	the fold only

The event key includes the sequencer, because a key has many creates. A message that arrives twice collides with itself. An overwrite has its own sequencer and lands as a new row.
seq_norm is the sortable sequencer, a stored generated column, lpad(sequencer, 32, '0'), so every query orders creates correctly (§3.5).
object_events.org_id has no foreign key. The log must record what S3 said, even for an org not known yet. A foreign key would make one unknown org fail a batch of 100, and would make the fold's unknown_org reason impossible. The ledger tables have the foreign key.
Some events are refused at the door. object_events_created_has_size requires size_bytes on every created event. A key over 1,024 bytes is refused, as S3 forbids it.
object_events is not partitioned. PostgreSQL requires the partition column in every primary key. Partitioning by event_time would put it in the key, so a repair and the real event would stop matching. Pruning is a ranged DELETE instead (§9.5).
Size lives only on the interval, because size_bytes and etag change during a lifetime. An object never overwritten has one interval, from created_at to deleted_at.
An interval is identified by its sequencer, not by from_at. Event times are whole seconds, so two creates can share a from_at. The first of them becomes a zero-length interval and bills nothing.

Rule: to find "the latest interval", order by from_at, then by the normalized sequencer: ORDER BY i.from_at DESC, lpad(i.sequencer, 32, '0') DESC. Never order by from_at alone.

object_size_intervals has a foreign key to object_lifetimes with ON DELETE CASCADE. It is the one table whose size is not limited by the number of objects: every overwrite adds a row.

3.3 Ingest¶

metering-ingest reads batches from SQS. For each record it decodes the key (§3.6), parses org_id (§3.1) and checks the fields. It reads nothing from the database. It sends the batch to metering.ingest_events(p_bucket, p_key, p_event_type, p_org_id, p_size_bytes, p_etag, p_sequencer, p_event_time, p_source), one multi-row INSERT ... ON CONFLICT:

Case	Key collides?	Result
The same event arrives again	Yes	Nothing changes.
An overwrite: same key, new sequencer	No	A new row. This is how a size changes.
Same key, type and sequencer, different size, ETag or time (both `source = 'event'`)	Yes	S3 contradicts itself. `conflicts` and `last_conflict_at` record it. The stored row is kept.
`source = 'inventory_correction'` on an existing create	Yes	Only `size_bytes` is replaced and `folded_at` cleared. `corrections` and `last_corrected_at` record it.

event_time is never rewritten. It is the billing time (§3.7).

Rule: de-duplicate on all four key columns before the insert. SQS can deliver one record twice in a batch, and PostgreSQL then raises ON CONFLICT DO UPDATE command cannot affect row a second time. Do not de-duplicate on (bucket, key, event_type): two records with different sequencers are an overwrite and its create, and both must be stored. The Lambda de-duplicates, and ingest_events() does it again with DISTINCT ON.

One bad record must not fail the other 99. The Lambda checks every record first and quarantines each failure on its own. An unknown but well-formed org_id is still inserted, for the fold to park; a lookup would be a read on the busiest path. If the bulk statement still fails, the Lambda retries one record at a time and quarantines each record that fails alone as ingest_error. It reports a message as failed to SQS (ReportBatchItemFailures) only when it cannot even quarantine it. Each run prints one CloudWatch embedded-metric line with IngestFallbackBatches and IngestRecordsQuarantined, zeros included.

3.4 The fold¶

The fold is the only writer of the ledger. Every 5 minutes, metering.fold() runs in one transaction. It takes the advisory lock hashtext('metering.fold'), and returns at once if no unfolded event exists outside a parked key. It picks the keys of up to fold_batch unfolded events (10,000 by default), skipping parked keys. It reads all events of each key as a set. So arrival order, duplicates and batch boundaries cannot change the result, and folding again changes nothing. The fold builds the lifetime and the intervals, and parks keys that fail the checks. Then it clears is_final where values moved, writes the ledger and marks events folded.

Rule: the first create plus the delete is the lifetime. Every create is an interval, closed by the next create or by the delete. Creates are ordered by seq_norm, and each interval ends where the next one starts:

LEAD(c.event_time) OVER (PARTITION BY c.bucket, c.key ORDER BY c.seq_norm) AS next_create_at

So an overwrite ends the old size and starts the new one at the same instant: no gap, no overlap, no byte billed twice.

A real event wins over a synthetic one. The fold ignores an inventory_backfill create when the key has a create with source = 'event'. The earliest real delete closes the lifetime; an inferred delete is used only when no real delete exists.

A key that cannot be derived is parked. Without this, one key that breaks a CHECK would fail the transaction every 5 minutes and stop billing for every org. The key goes to metering.fold_quarantine with a reason:

`reason`	Condition
`unknown_org`	The org is not in `organizations` (A3, A4).
`multiple_deletes`	More than one real delete on the key. Inferred deletes do not count.
`resurrected_key`	A create after a closed lifetime's `deleted_at` (A6), or the key's events were pruned (§9.5). Not an overwrite, which lands on a live key.
`delete_before_create`	The delete is before the create, or any interval would end before it starts.
`fold_error`	The key failed alone in the safety net (below). `detail` holds the SQLSTATE and message.

The check runs on every proposed interval, because a later create dated after the delete gives to_at < from_at. It uses the merged row, COALESCE(new deleted_at, stored deleted_at), because the lifetime upsert merges the same way.

Rule: a parked key keeps its intervals. If the fold removed them, the lifetime would remain with no bytes and bill zero, which looks healthy everywhere.

A parked key re-arms itself. The fold skips parked keys before it marks anything, so their events stay unfolded. Setting resolved_at is all that is needed: the next fold picks the key up. Each night metering.resolve_fold_quarantine() resolves unknown_org rows whose org exists again. The other reasons are contradictions in the events, which no later data clears.

Rule: clear is_final only for the days whose value moved. Deleting a year-old object changes no day before the delete, and clearing the whole lifetime would send a year of days back to the rollup. The fold matches stored and new intervals by sequencer. An interval that is added or removed, or whose size changed, clears its whole span. An interval whose end alone moved (a delete, or an overwrite that ends it) clears only the days between the old and new end. An open interval counts as ending at the current time.

Rule: day bounds are UTC. timestamptz::date uses the session time zone, so every day bound is written (ts AT TIME ZONE 'UTC')::date. Every job connection also runs SET TIME ZONE 'UTC'.

The rest of the write. The fold marks exactly the event rows it read, by (bucket, key, event_type, sequencer), so an event that commits during the run waits for the next one. The lifetime upsert rewrites every column from the events, so a real create replaces a backfilled created_at and time_source. Only deleted_at is merged with COALESCE, so a closed lifetime is never reopened. Intervals are deleted and inserted again for each touched key, so no old interval is left behind.

A safety net catches what the checks miss. metering.fold() wraps metering.fold_core(p_batch, p_buckets, p_keys). If the batch fails, fold() folds each key alone in a subtransaction and parks only a key that fails alone, as fold_error. Errors about the code or the database are raised, never parked. These are the SQLSTATE classes 08, 0A, 25, 28, 3D, 3F, 40, 42, 53, 54, 55, 57, 58 and XX. A batch of more than one key in which every key fails alone is also raised. Other errors park the key, such as data (22), constraint (23) and check (P0) errors. metering.fold_to_completion() repeats fold() until a pass neither folds nor parks anything.

A delete with no create produces no lifetime, so this "unmatched delete" is out of billing by structure. metering.unmatched_deletes() lists those older than 24 hours. Diff D (§8.2) or the archive (§9.4) resolves them. If they stay, creates are being lost.

The ledger can be rebuilt from the events. Delete the lifetimes (their size intervals go with them) and the fold quarantine, set folded_at back to empty on every event, and fold to completion. Clearing the ledger alone rebuilds nothing, because the fold reads only unfolded events. The exact steps, and the role they must run as, are in Operations, 10.4.

A rebuild reaches back only as far as the events that still exist. For older windows, replay the archive into object_events first (§9.4).

3.5 Sequencer¶

The sequencer is S3's ordering token for the events of one key. It is the only thing that orders the creates of an overwritten object. A wrong order puts the wrong size on every byte-second after the first overwrite, and no constraint would notice.

Compare sequencers only within one key.
Compare by length first, then as text. A plain text compare puts 'FF' after '100', which is wrong. seq_norm pads to 32 characters, and the fold orders by it.
Never guess an order from event_time. It is whole seconds and has ties. A create without a sequencer is quarantined (A7).
Synthetic events use reserved sequencers. metering.synthetic_create_sequencer() is 32 zeros and sorts before every real sequencer. metering.synthetic_delete_sequencer() is 32 lowercase f characters (all bits one) and sorts after every real one. They keep a repair and a real event in separate rows, and make a repair easy to identify.

3.6 Key encoding¶

S3 Event Notifications and EventBridge encode keys differently. A key decoded only in part can still look valid, name the wrong org and bill the wrong customer with no error. The KEY_DECODE setting of metering-ingest is none (the key as delivered, correct for EventBridge), urldecode or urldecode_plus (also + to space).

Rule: find the encoding by testing live events, not from documentation. With LOG_RAW_KEYS=true the Lambda logs every raw key; compare them with the keys S3 lists.

The canary proves the decoding every hour. Every probe key ends with metering.canary_key_suffix(), ' a+b&c%20d.bin': a space, a plus, an ampersand and a literal %20. metering.canary_check() counts only a probe whose exact key reached the ledger, so a wrong decoding makes the canary alarm fire. A HeadObject call per event would cost a call each time and fail for deleted objects. The tests also cover # and Unicode keys.

3.7 Timestamps¶

event_time is the envelope's time field, stored as received. It is the billing time. ingest_time is when the event was written.
The fold copies times and never computes them. created_at, deleted_at, from_at and to_at are event times. folded_at has no billing meaning.
Rule: never use now() for created_at or deleted_at. A DLQ redrive three days later would add three days of storage that never existed.
object_lifetimes.time_source is the source of the first create: event or inventory_backfill. An inferred delete has source = 'inferred' on its event row.

4. Multipart uploads¶

A multipart upload uses storage before it becomes an object. An upload that is never completed or aborted keeps costing money, and neither S3 events nor S3 Inventory show it. Only an hourly sweep can see it.

An upload is billable only after it has been open for 3 hours. Billing an upload in progress would charge the customer for the act of uploading; an upload open longer is treated as abandoned. This is a charge for leaked storage, and most uploads never become billable. An upload that really takes longer than 3 hours is billed for the bytes it holds from hour 3. The grace period moves the start of the window; it does not apply backwards. Billing runs from initiated_at + 3 hours, so the charge does not jump from nothing at 2h59m to three hours at 3h01m. The 3 hours are defined once, in metering.mpu_grace(). The sweep writes mpu_lifetimes.billable_from = initiated_at + mpu_grace(). It cannot be a generated column, because timestamptz + interval is only STABLE, so metering.mpu_audit() checks it nightly.

Changing the grace. The sweep rewrites billable_from on every upload it still lists, so an open upload follows a new value at its next sweep. Accrual only moves forward from last_seen_at. So a shorter grace bills an upload that was not yet billing from its new start, and never bills again the hours an upload that was already billing has passed. A closed upload keeps the billable_from it was billed from.

Rule: gather from AWS first, then open the transaction. Every hour, metering-mpu-sweep takes each bucket in METERED_BUCKETS. It records when the listing starts, calls ListMultipartUploads and then ListParts per upload, and reads the bucket's AbortIncompleteMultipartUpload age. This can take minutes, and an open transaction would hold the lock and block vacuum. Only then does it call metering.sweep_mpu(), once per bucket, with the listing start as p_now. An upload that completes between the two calls (NoSuchUpload) is left out, so it is simply absent.

Inside sweep_mpu() the order is close, then accrue, then observe.

1. Close what the listing does not show.

What the sweep sees	Result
Absent for the first time; no object on the key created at or after `initiated_at`	Stays open, and its cursor does not move. The create may not be folded yet.
Absent for a second sweep; still no such object	Closed at `p_now`: `lifecycle_abort` if older than the bucket's abort age, else `aborted`.
The only absent upload on the key; such an object exists	`completed`, closed at exactly the object's `created_at`.
Several absent; exactly one has `parts_etag` equal to the ETag of the object's first interval	That one is `completed` at `created_at`. The others are `aborted` at their own `last_seen_at`.
Several absent; no unique ETag match	All are `aborted` at their own `last_seen_at`. `metering.mpu_ambiguous_keys()` reports the key.

Rule: close a completed upload at exactly the object's created_at. Upload time and object time then touch but do not overlap. A 20-hour upload bills as an upload for hours 12 to 20, and as an object from hour 20. The sweep computes parts_etag from ListParts while the upload is still listed, which is the only time it can. It strips the quotes from the event ETag before comparing. public.upload_parts cannot be the source: it has no upload id, bucket or key.

2. Accrue. bytes_pending grows as parts arrive, so each sweep bills the time since the previous sweep at the size that sweep saw:

cursor = GREATEST(last_seen_at, billable_from)
target = GREATEST(LEAST(p_now, COALESCE(closed_at, 'infinity')), billable_from, first_seen_at)
delta  = bytes_pending × (target − cursor)          -- byte-seconds, can be negative

The delta is signed. A close at an object's created_at can be behind the cursor. The negative delta gives back seconds already billed, so they are not billed as upload and object.
Both ends are limited by billable_from and first_seen_at. Nothing before either was billed, so nothing before them is given back.
The delta is split at 00:00 UTC, so each day, and at a period boundary each invoice, gets its own share.
The increment and the cursor move commit together, so a retried sweep adds nothing. This is the one value in the system that is added to, not recomputed. The sweep also sets is_final = false and computed_at on the days it adds to.

3. Observe. Last, the sweep upserts every listed upload with its new size. So a growing upload is never billed at its final size for the whole hour. An upload whose key has no valid or known org cannot be stored; the sweep counts it as unattributable and skips it.

Multipart bytes are not rounded per object: gb_hours_mpu = Σ mpu_byte_seconds / 3600 / 1073741824. The lifecycle abort age limits the leak: the longest billable window is the abort age minus the grace, so a 7-day rule allows 6 days and 21 hours. Set the two values together.

Nightly audit. metering.mpu_audit() checks billable_from on every open upload. Per org, bytes accrued on uploads (live rows plus mpu_audit_baseline) must equal the sum of usage_daily.mpu_byte_seconds. A difference means a sweep committed only part of its work. A rising billable_open (open uploads past the grace period) means clients abandon uploads. Orphaned parts are a report, not a charge: metering.mpu_orphan_parts() finds episodes with upload_parts rows older than 7 days whose key (episodes.mp4_s3_path) has no lifetime and no open upload. The app deletes those rows on complete and on abort, so they point to a client that does not finish its uploads.

5. GPU compute¶

gpu_instance_types maps an instance type to a GPU model and count. gpu_jobs has one row per training job (job_id, org_id, provider, gpu_model, gpu_count, instance_type). gpu_job_intervals has one row per attempt (started_at, ended_at, last_heartbeat_at, end_reason).

gpu_seconds = Σ over intervals ( COALESCE(ended_at, last_heartbeat_at) − started_at ) × gpu_count
gpu_hours   = gpu_seconds / 3600

gpu_count is set at launch and never changes. Without it, a p5 instance (8 GPUs) would bill as one GPU. It is never read back from gpu_instance_types, so a later correction of the map cannot change an invoice already sent.
No guessed GPU model. The model sets the rate. gpu_start_job() refuses a NULL model or a placeholder such as unknown. Given an instance type, it takes the model from gpu_instance_types, and refuses an unmapped type or a model that does not match.
A job is a set of intervals. A stopped and restarted job has several attempts; one start and end would bill the gaps.
Heartbeats limit the error for a crash. A crashed job never sets ended_at. Heartbeats arrive every 60 seconds, and billing uses COALESCE(ended_at, last_heartbeat_at). The reaper, metering.reap_gpu_intervals(), closes an interval whose last heartbeat is older than three intervals, with ended_at = last_heartbeat_at and end_reason = 'heartbeat_timeout'. The error is at most about 3 minutes. The reaper clears is_final on the days it touches.
The orchestrator's close never ends before the last heartbeat: gpu_end_interval() sets ended_at = GREATEST(reported end, last_heartbeat_at). A day stays non-final while an open interval with last_heartbeat_at at or before the day's end covers it (§6.2). So no final day can change, and the orchestrator needs no access to usage_daily. A heartbeat never moves last_heartbeat_at back.
A timeout is not proof that the job died. A network break longer than three minutes closes an interval while the job runs on. A heartbeat for a job with no open interval opens the next attempt and returns reopened, which raises an alarm. The gap is not billed: there was no evidence the job was alive. If the latest attempt ended completed, the heartbeat returns finished and opens nothing. A failed attempt can reopen, because a retry keeps the job id.
Intervals recorded late still reach the meter. The rollup finds final days covered by an interval whose ingest_time is later than the day's computed_at (§6.2).

The orchestrator connects as usage_heartbeat and calls four functions:

metering.gpu_start_job(p_job_id TEXT, p_org_id UUID, p_gpu_model TEXT, p_gpu_count INTEGER,
    p_instance_type TEXT DEFAULT NULL, p_provider TEXT DEFAULT 'aws',
    p_created_at TIMESTAMPTZ DEFAULT now()) RETURNS BOOLEAN        -- false: the job exists
metering.gpu_start_interval(p_job_id TEXT, p_started_at TIMESTAMPTZ DEFAULT now())
    RETURNS UUID                                                    -- NULL: attempt 1 exists
metering.gpu_heartbeat(p_job_id TEXT, p_at TIMESTAMPTZ DEFAULT now())
    RETURNS TEXT                                                    -- extended | reopened | finished
metering.gpu_end_interval(p_job_id TEXT, p_end_reason TEXT DEFAULT 'completed',
    p_ended_at TIMESTAMPTZ DEFAULT now()) RETURNS BOOLEAN           -- false: nothing open

gpu_start_job() uses ON CONFLICT DO NOTHING, so a retried launch cannot change the GPU count. gpu_start_interval() is idempotent, reads org_id from the job row, and fails when no job row exists. gpu_end_interval() accepts completed, failed or preempted; heartbeat_timeout belongs to the reaper.

Rules for a launcher. A launcher that bills GPU time must:

take job_id from the scheduler (a job ARN or pod UID), never from the training container, so a job cannot bill another org;
call gpu_start_job() before the job runs, and cancel the job if that fails;
call gpu_start_interval() with the scheduler's start time when the job is running;
send heartbeats from a process that lives longer than the launch request;
call gpu_end_interval() with the scheduler's stop time.

GPU metering is off until GPU_METERING_ENABLED=true on metering-jobs. Switch it on only with a launcher that follows these rules (see Operations).

6. Billing¶

6.1 Unit: GB-hours¶

Bytes are priced per size interval. The hour rounding is applied once per object:

secs(i)             = LEAST(COALESCE(to_at, deleted_at, 'infinity'), period_end)
                      − GREATEST(from_at, period_start)         -- each size interval i
residency           = the object's own live seconds in the period
padding             = CEIL(residency / 3600) × 3600 − residency
last_size           = size_bytes of the object's last interval in the period

gb_hours_exact      = Σ_i size_bytes × secs(i) / 3600 / 1073741824
gb_hours_per_object = ( Σ_i size_bytes × secs(i) + Σ_objects last_size × padding ) / 3600 / 1073741824

Rounding up gives a one-hour minimum per object. A 40-minute object bills one hour.
An overwrite does not add a second minimum. A file overwritten five times in an hour bills one hour, not six; otherwise the bill would follow how often a file is edited.
The padding is priced at the last size in the period, the size the object actually holds.
Without an overwrite, this is size × CEIL(hours).

Rule: add up byte-seconds and divide once, at the end. Each size_bytes × seconds is an exact integer in NUMERIC. Dividing per interval makes each term a repeating decimal that NUMERIC cuts off, and six ten-minute intervals then sum to 744.00000000000000000000000002, not 744.

Rule: write the GiB divisor as 1073741824, never as 2^30. In PostgreSQL ^ returns double precision, so size_bytes::numeric / 2^30 turns every later figure into floating point. The drift scan (§7.1) depends on exact zeros. In floating point a sum that should cancel gives 5.55e-17, and a false adjustment appears every night.

A definition and a fast path. metering.gb_hours_definition(p_org, p_start, p_end) computes both figures from the ledger, exactly as above, but reads too many rows for production. metering.period_figures() gets the same figures from usage_daily plus a rounding correction (§6.4). The tests check that the two agree exactly.

The invoice unit is the GB-month. invoices.storage_gb (NUMERIC(14,4)) is the average GB held across the period, gb_hours_billed / period_hours, priced at subscription_plans.storage_rate_per_gb per GB-month. The hour rounding happens per object before this division, so it survives. GPU is billed as gpu_hours = gpu_seconds / 3600.

6.2 Daily rollups¶

A query over every interval of a period reads too many rows. So the nightly metering.rollup(p_as_of, p_grace, p_lookback) sums each UTC day once into metering.usage_daily, one row per org per day. It stores byte-seconds, not GB-hours, because byte-seconds add across days and per-object rounding does not: an object kept five days would be rounded five times. Rounding happens once, at the closure (§6.4).

Three writers own different columns. Each names only its own columns in DO UPDATE SET, and every column has a default, because any writer can create the row:

Columns	Writer
`byte_seconds`, `gpu_seconds`, `object_count_eod`, `is_final`, `computed_at`	the nightly rollup, recomputed from the ledger
`is_final = false`	the fold (§3.4) and the GPU reaper (§5), when they change a day's value
`mpu_byte_seconds`, with `is_final = false` and `computed_at`	the hourly multipart sweep (§4), added to with a sign

Do not upsert the whole row: a rollup that did would erase every multipart accrual since midnight.

The rollup recomputes these days:

days inside the grace window (p_grace, 48 hours);
every day with is_final = false, however old;
any (org_id, day) the ledger covers within p_lookback (35 days) that has no row yet;
any final day covered by a GPU interval whose ingest_time is later than the day's computed_at.

byte_seconds sums size intervals clipped to the day. So an overwrite counts the old size up to the change and the new size after it. gpu_seconds is clipped to the day and multiplied by gpu_count. An open GPU interval counts up to last_heartbeat_at.

is_final has one definition, and it needs both conditions:

is_final(org, day) :=
      now() > day_end + grace                      -- late object events have arrived
  AND NOT EXISTS (SELECT 1 FROM metering.gpu_job_intervals g   -- no GPU interval can still grow
                  WHERE g.org_id = org AND g.started_at < day_end
                    AND g.ended_at IS NULL AND g.last_heartbeat_at <= day_end)

The grace window alone would make a day final while a job still runs through it. The GPU condition alone would make every storage-only day final before late events can arrive. is_final is not a lock: a late event clears it, and the next rollup recomputes the day. For a closed period this moves computed_at past closed_at, the drift scan's signal (§7.1).

The app sees a copy. In the same statement, the rollup copies each row it writes to public.usage_daily. Billing never reads the copy, and the app never reads the metering table. If they differ, the metering row is correct, and the next rollup repairs the copy.

6.3 Billing periods¶

Periods come from subscriptions: an org that subscribed on the 14th has periods from the 14th to the 14th, never calendar months. subscriptions holds only the current period, and the app moves it forward at renewal, when the ended period still needs an invoice. So the trigger subscriptions_record_billing_period runs public.record_billing_period() on every insert and on every change to the period, plan or status. It is the only writer of metering.billing_periods, and the app changes nothing to feed it.

Change to the subscription	What is recorded
Renewal: new start at or after the old end	The old period as it ended, with its plan and status. Its invoice is priced from this row.
New start inside the old period (billing date moved)	The old period, from its start to the new start. The full-length row is removed.
Same start, different end (cancelled or extended)	The old row is removed. The new bounds replace it.
Dates cleared	The old period is kept.

A closed period is never removed. New bounds that overlap it wait as waiting_for = 'overlap' for a person (§7.2). A plan change in mid-period overwrites the row, so the plan at the end of the period prices all of it. The trigger never fails a write to subscriptions.

Allowances apply once per period, not per day: billable = GREATEST(0, used − included), for storage GB and for GPU-hours. By A5 a period is a whole number of UTC days, so it sums directly from usage_daily. organizations.timezone is for display only.

6.4 Rounding mode¶

per_object:   Σ size × CEIL(hours)      -- CEIL inside the sum, per object
per_org:      CEIL( Σ size × hours )    -- CEIL outside the sum
exact:        Σ size × hours

The mode is metering.billing_config.billing_mode, one value for the platform, default per_object. Rounding belongs to the platform, not to a plan, so a plan change never changes it.

Rule: every closure stores both object figures, whatever the mode. per_object bills gb_hours_per_object, exact bills gb_hours_exact, and per_org bills CEIL(gb_hours_exact). So changing the mode needs no migration and no recompute, and "what would last month cost in the other mode?" is answered from stored rows. Each closure records its billing_mode. Multipart GB-hours (gb_hours_mpu) are added unrounded in every mode.

The fast path rounds only boundary objects. By A5 a period is whole hours, so CEIL changes nothing for an object that exists for the whole period:

gb_hours_per_object = ( Σ usage_daily.byte_seconds + padding ) / 3600 / 1073741824
padding             = Σ last_size × ( CEIL(residency / 3600) × 3600 − residency )
                      over lifetimes with created_at or deleted_at inside the period

metering.period_rounding_padding() reads this set from two partial indexes on object_lifetimes. It runs in every mode. An overwrite moves neither created_at nor deleted_at, so the cost of a closure follows object turnover, not edit volume.

An object that crosses a period boundary is rounded in each period. A 90-minute file across a boundary bills one hour in each, two in total. Each period must close on its own, so this is expected. The extra charge is at most one hour per object per boundary.

7. Period close and late events¶

Late events are the main risk to a correct invoice. For example, September is invoiced on October 1. On October 3 a DLQ redrive delivers a create with an event_time of September 15, and the ledger disagrees with an invoice already sent. Five rules handle this:

Every row has two times. event_time places usage in the period where it happened. ingest_time records when it arrived.
A grace window. A period closes only 48 hours after it ends, and only when every day in it is final (§6.2). Otherwise a close at 01:00 would run before the 03:30 rollup and close a last day computed a day early.
The closure. The figures go to usage_period_closures, and the invoice is priced from that row, never from a fresh query. The ledger can change, so only the closure shows what was billed. Only its invoice_id is ever updated. The canary org is never closed or invoiced.
Drift becomes an adjustment in usage_adjustments (§7.1).
Carry or restate. A small adjustment goes on the next invoice. A large one, or any on an org's last period, needs a corrected invoice (§7.2).

7.1 Adjustments¶

One producer: the nightly drift scan. metering.drift_scan(p_horizon) takes each closure from the last p_horizon (90 days) that has a day with usage_daily.computed_at later than closed_at. It computes the period's figures again and, for each of byte_seconds, gb_hours_exact, gb_hours_per_object, mpu_byte_seconds, gb_hours_mpu and gpu_seconds, writes:

delta = figure_now − closure_figure − Σ drift deltas already settled on an invoice

computed_at moves for every writer of usage_daily. So this one scan finds every late change: a late object event, a GPU interval closed by the reaper, a size repair, a multipart give-back.

Subtract settled drift rows only. A manual row is an operator's credit or charge, not the ledger moving. Subtracting it would turn a settled −0.5 GB-hour credit into +0.5 of drift.
Compare multipart in exact byte-seconds (mpu_byte_seconds on the closure). Multiplying the rounded gb_hours_mpu back up leaves a remainder that never reaches zero.
Keep one unsettled drift row per period, and replace it. The remainder is cumulative, so appending each night would bill it once per night. The partial unique index usage_adjustments_unsettled allows one unsettled closed_period_drift row per period, and a manual row can exist beside it. When the remainder reaches zero, the row is deleted.
created_at is when the remainder first appeared. Replacing the figures never moves it. The last recompute is in detail->>'computed_at'.
The scan has a horizon, or it would read every closure every night. Set p_horizon to at least the longest delay an event can have: DLQ retention plus one inventory interval.

An adjustment is valued in the mode its period closed under, never the current mode. metering.adjustment_gb_hours() reads the closure's billing_mode. A drift row is worth gb_hours_object_delta in per_object and gb_hours_exact_delta in exact, plus gb_hours_mpu_delta. In per_org it is worth CEIL(settled + gb_hours_exact_delta) − CEIL(settled) plus gb_hours_mpu_delta, where settled is the closure's exact figure plus drift already settled: 744.2 → 744.5 is worth 0. A manual row is taken as the operator wrote it. In per_object mode adjustments come in steps: a late delete that moves a lifetime from 3h01m to 2h59m changes the bill by a whole hour. Set the restatement threshold with this in mind.

The view metering.unsettled_adjustments adds these columns to each unsettled row:

Column	Meaning
`billing_mode`, `gb_hours_delta`	The closed period's mode, and the value just described
`gpu_hours_delta`	`gpu_seconds_delta / 3600`
`storage_gb_delta`	`gb_hours_delta` divided by the hours of the row's own period, so a 30-day correction on a 31-day invoice keeps its value
`is_material`	`abs(gb_hours_delta) >= restatement_threshold_gb_hours` or `abs(gpu_hours_delta) >= restatement_threshold_gpu_hours`
`is_terminal`	No subscription of the org, not cancelled, runs past the period, so no later invoice can carry the row

Carry or restate. The next period's invoice carries every unsettled row of a previous period that is manual, or is drift that is not material. It prices them at its own rates. Material drift is not carried: its ids come back in restatement_needed_ids, and it needs a corrected invoice of its own period. Drift on a terminal period also needs one, whatever its size.

Credits are not clamped. A negative quantity cannot go on an invoice, and clamping it to zero would lose the credit. If carried rows would make storage or GPU negative, only the non-negative rows are carried; the rest wait. The application settles adjustments only through invoicing: the two finalize functions set applied_to_invoice_id (§7.2). metering.settle_adjustments() does the same for an operator who runs as usage_fold; the application cannot call it. Each settled row records what it added to that invoice, in billed_storage_gb and billed_gpu_hours.

7.2 Invoices¶

The database computes every invoice column. The application inserts the invoices row, with its own id, status, dates and payment reference.

Who calls. The application connects as tobor_app and reaches billing only through the SECURITY DEFINER functions in public defined by queries/metering_app.sql. A function that acts on an org refuses unless app.current_org equals that org. The comparison is on text, so set it to the lowercase UUID. The corrected-invoice functions take the org from the original invoice.

Rule: one invoice is one transaction: draft invoice, insert, finalize. The draft invoice holds the drift scan's lock until the transaction ends, so no carried row can change before finalize.

For the application. During the period the application does nothing about the invoice. The jobs fill usage_daily. A billing job, still connected as tobor_app, runs after the period has ended. Take September, invoiced on 3 October:

usage_periods_due(). Invoice a row only when ready is true. Otherwise stop and try again later. waiting_for says why: grace (the 48 hours are not up), rollup (a day is not final), plan (no plan on the period), or overlap (a person must decide). A period with no usage is still due, because the base fee is owed.
In one transaction, draft_usage_invoice returns the priced columns. Insert invoices by copying them. The application sets the id, status, issued_at and due_at, and later paid_at and payment_reference. It does not compute a price. Then finalize_usage_invoice, passing carried_adjustment_ids and carried_fingerprint back unchanged, and commit.
A small correction found after the invoice does not change that invoice. It is already inside the next period's draft, so the next run of these same steps bills it.
If restatement_needed_ids is not empty, an earlier period was wrong by a lot. Insert a second invoice for that period, set the original is_valid = false, and point superseded_by_invoice_id at the new id. Never change the cents on an invoice already issued.

The steps below are the same work, with the signatures and the errors.

Step 1: list the periods that are due.

public.usage_periods_due()
RETURNS TABLE (org_id UUID, subscription_id UUID, plan_id UUID, period_start TIMESTAMPTZ,
               period_end TIMESTAMPTZ, subscription_status TEXT, ready BOOLEAN, waiting_for TEXT)

It returns each recorded period that has ended and has no invoice. It covers the org in app.current_org, or every org when the setting is unset (for a billing worker). It returns ids and dates, never usage. It raises an error if app.current_org is set but is not a UUID. When ready is false, waiting_for names the first reason:

overlap: the period overlaps a closed period with other bounds, and a person must decide;
grace: less than 48 hours have passed since the end;
rollup: a day in the period is not final;
plan: no plan is recorded for the period.

The canary org is never listed. A period with no usage is still due, because the base fee is owed.

Step 2: get the draft invoice.

public.draft_usage_invoice(p_org UUID, p_start TIMESTAMPTZ, p_end TIMESTAMPTZ)
RETURNS TABLE (
  org_id UUID, subscription_id UUID, plan_id UUID, currency TEXT,
  period_start TIMESTAMPTZ, period_end TIMESTAMPTZ, base_fee_cents INTEGER,
  storage_gb NUMERIC, storage_rate_per_gb NUMERIC, storage_cents INTEGER,
  cold_storage_gb NUMERIC, cold_storage_rate_per_gb NUMERIC, cold_storage_cents INTEGER,
  streaming_gb NUMERIC, streaming_rate_per_gb NUMERIC, streaming_cents INTEGER,
  gpu_hours NUMERIC, gpu_rate_per_hour NUMERIC, gpu_cents INTEGER, total_cents INTEGER,
  -- not invoice columns: what the figures came from, and what finalize needs
  closure_id UUID, billing_mode TEXT, period_hours NUMERIC,
  storage_gb_used NUMERIC, included_storage_gb NUMERIC,
  gpu_hours_used NUMERIC, included_gpu_hours NUMERIC,
  carried_adjustment_ids UUID[], carried_fingerprint TEXT, restatement_needed_ids UUID[])

It creates the closure, or returns the existing one. It raises an error when:

the period does not start and end at 00:00 UTC (A5), or less than 48 hours have passed since p_end;
the org is the canary org, or the period is not in billing_periods, or it has no plan;
the plan has no currency, base_fee_cents, storage_rate_per_gb or gpu_rate_per_hour (use 0 for a real zero);
the period overlaps a closed period with other bounds, or a day in it is not final;
the period already has an invoice.

Pricing is in one function, metering.price_usage(). For storage and for GPU: quantity = round(GREATEST(0, used − included), 4) (the invoice column has 4 decimals), then cents = round(quantity × rate × 100), half away from zero. storage_gb_used is gb_hours_billed / period_hours plus the carried storage_gb_delta values; gpu_hours_used is gpu_seconds / 3600 plus the carried gpu_hours_delta values. Cold storage and streaming are 0 at the plan's rate (§13), so total_cents = base_fee_cents + storage_cents + gpu_cents.

Step 3: insert the invoice from the returned invoice columns, exactly. The app computes no price itself.

Step 4: finalize.

public.finalize_usage_invoice(p_org UUID, p_invoice_id UUID, p_start TIMESTAMPTZ,
                              p_end TIMESTAMPTZ, p_carried_ids UUID[], p_fingerprint TEXT)
RETURNS BIGINT    -- the number of adjustments settled

Pass back carried_adjustment_ids and carried_fingerprint unchanged. The function links the closure to the invoice and settles exactly the carried rows. Calling it again for the same invoice is safe. It raises an error when:

the invoice is missing, belongs to another org, has other bounds or is void;
there is no closure, or the closure is linked to another invoice;
a carried row is already settled on another invoice;
the fingerprint differs. The drift scan replaces rows in place, so one id can hold other figures later. carried_fingerprint is an md5 of the carried rows' ids and figures. On a mismatch, roll back and start again.

BEGIN;
SET LOCAL app.current_org = '<org_id>';
SELECT * FROM public.draft_usage_invoice('<org_id>', '<period_start>', '<period_end>');
INSERT INTO invoices (...) VALUES (...);          -- the returned invoice columns, unchanged
SELECT public.finalize_usage_invoice('<org_id>', '<invoice_id>', '<period_start>',
         '<period_end>', '<carried_adjustment_ids>', '<carried_fingerprint>');
COMMIT;

Corrected invoices. One is needed when a draft invoice returns ids in restatement_needed_ids, or when public.unsettled_usage_adjustments() shows a closed_period_drift row with is_terminal = true. That function returns SETOF metering.unsettled_adjustments for the current org, and refuses when app.current_org is not set.

public.draft_corrected_usage_invoice(p_original_invoice_id UUID)
RETURNS TABLE (
  org_id UUID, subscription_id UUID, plan_id UUID, currency TEXT,
  period_start TIMESTAMPTZ, period_end TIMESTAMPTZ, base_fee_cents INTEGER,
  storage_gb NUMERIC, storage_rate_per_gb NUMERIC, storage_cents INTEGER,
  cold_storage_gb NUMERIC, cold_storage_rate_per_gb NUMERIC, cold_storage_cents INTEGER,
  streaming_gb NUMERIC, streaming_rate_per_gb NUMERIC, streaming_cents INTEGER,
  gpu_hours NUMERIC, gpu_rate_per_hour NUMERIC, gpu_cents INTEGER, total_cents INTEGER,
  original_invoice_id UUID, restated_adjustment_ids UUID[], restated_fingerprint TEXT,
  storage_gb_used NUMERIC, gpu_hours_used NUMERIC)

public.finalize_corrected_usage_invoice(p_original_invoice_id UUID, p_corrected_invoice_id UUID,
                                        p_restated_ids UUID[], p_fingerprint TEXT)
RETURNS BIGINT    -- the number of adjustments settled

The original is the valid invoice of the adjustment's period. A corrected invoice is rebuilt, not derived again. It is the closure, plus exactly what the original absorbed from previous corrections (billed_storage_gb, billed_gpu_hours), plus every unsettled row of the period. It uses the original's rates, base fee and currency, and its plan's allowance. The corrected draft invoice refuses when:

the original is missing, void, has no storage or GPU rate, or is not linked to the closure;
nothing is unsettled for the period, or credits exceed the period's usage;
the original absorbed a row with no billed_storage_gb or billed_gpu_hours. Reading that as zero would bill a credit again, so such an invoice is corrected by hand.

BEGIN;
SET LOCAL app.current_org = '<org_id>';
SELECT * FROM public.draft_corrected_usage_invoice('<original_invoice_id>');
INSERT INTO invoices (...) VALUES (...);          -- the replacement, from the returned columns
UPDATE invoices SET is_valid = false, voided_at = now(), void_reason = '<reason>',
                    superseded_by_invoice_id = '<corrected_invoice_id>'
 WHERE id = '<original_invoice_id>';
SELECT public.finalize_corrected_usage_invoice('<original_invoice_id>', '<corrected_invoice_id>',
         '<restated_adjustment_ids>', '<restated_fingerprint>');
COMMIT;

The finalize moves the closure, and every row the original absorbed, to the replacement. Then it settles the restated rows on the replacement. It refuses when:

the original is still valid, or does not point to the replacement;
the replacement is void, or covers another org or period;
a restated row belongs to another period or is settled elsewhere, or the fingerprint differs.

The closure alone, without prices, is freeze_usage_period (§7.3). An invoice never needs that call: draft_usage_invoice already writes the same row.

7.3 How to use `usage_period_closures`¶

metering.usage_period_closures is one row per organization per billed period. It is what the ledger said when the period was frozen, and it is what the invoice was priced from. The application never inserts into this table and never selects from it. tobor_app has no grant on the metering schema. A row is written only by metering.close_period(), which the application reaches through the two calls below.

Do not write the table yourself. A second insert for the same organization and the same bounds fails on UNIQUE (org_id, period_start, period_end). Changing a figure would make the invoice impossible to reproduce. After the insert, the only column that changes is invoice_id, and only the two finalize functions write it. Usage that arrives after the close goes to usage_adjustments, not back into this row. The canary organization never gets a row. Rows are never pruned.

To freeze a period and invoice it, use the transaction in §7.2. draft_usage_invoice creates the row, or returns the one already there. closure_id in that result is usage_period_closures.id. finalize_usage_invoice sets invoice_id on that row. Calling the draft again for the same bounds returns the same row and does not recompute it. A period with no usage still gets a row, because the base fee is owed. Its figures are then zero.

To freeze a period without an invoice, call freeze_usage_period. Use it when something other than the invoice needs the frozen figures, for example a usage page for a period that has already closed. Connect as tobor_app. Set app.current_org to the lowercase UUID of the organization.

BEGIN;
SET LOCAL app.current_org = '<org_id>';
SELECT * FROM public.freeze_usage_period('<org_id>', '<period_start>', '<period_end>');
COMMIT;

public.freeze_usage_period(p_org UUID, p_start TIMESTAMPTZ, p_end TIMESTAMPTZ,
                           OUT closure_id UUID, OUT storage_gb NUMERIC, OUT gpu_hours NUMERIC,
                           OUT gb_hours_billed NUMERIC, OUT gpu_seconds NUMERIC,
                           OUT billing_mode TEXT, OUT period_hours NUMERIC)
RETURNS RECORD

Returned column	Meaning
`closure_id`	`usage_period_closures.id`
`storage_gb`	`gb_hours_billed / period_hours`, the same unit as the invoice column `storage_gb`
`gpu_hours`	`gpu_seconds / 3600`
`gb_hours_billed`	Object GB-hours for the mode in force at close, plus multipart
`gpu_seconds`	GPU time in the period, exact
`billing_mode`	`per_object`, `per_org`, or `exact`
`period_hours`	Hours from `period_start` to `period_end`

It raises an error when:

period_end is not after period_start, or either bound is not 00:00 UTC;
fewer than 48 hours have passed since period_end;
app.current_org is not this organization;
the period is not in billing_periods, or it overlaps a closure with other bounds;
a day in the period is not final (usage_daily.is_final is false);
the organization is the canary organization.

Calling it again for the same bounds returns the same closure_id and the same figures.

What each stored column is for. The calls above do not return every column. These are the columns on the row. Read them as usage_fold. The application cannot.

Column	Use it for
`id`	Match `closure_id` from a draft or a freeze
`org_id`, `subscription_id`	The organization, and the subscription whose plan produced the period
`period_start`, `period_end`	The billed window. Both are 00:00 UTC
`byte_seconds`	Exact object byte-seconds. The audit base. Do not bill from this column
`gb_hours_exact`	Object GB-hours with no rounding
`gb_hours_per_object`	Object GB-hours with `CEIL` per object. Always at least `gb_hours_exact`
`gb_hours_mpu`, `mpu_byte_seconds`	Multipart usage. Compare drift against `mpu_byte_seconds`. Do not multiply `gb_hours_mpu` back up
`billing_mode`	Which object figure was billed. `per_object` uses `gb_hours_per_object`. `exact` uses `gb_hours_exact`. `per_org` uses `CEIL(gb_hours_exact)`. Multipart is added in every mode
`gb_hours_billed`	That object figure plus `gb_hours_mpu`. Invoice storage is this divided by the hours in the period
`gpu_seconds`	Exact GPU seconds. Invoice GPU-hours are this divided by 3600
`closed_at`	When the row was written. The drift scan compares `usage_daily.computed_at` with it
`invoice_id`	NULL until finalize. A corrected invoice moves it from the original to the replacement

Both object figures are stored on every row, whichever mode was in force. A later change of billing_config.billing_mode can still answer what this period would have cost the other way, without recomputing the ledger.

Read a closure connected as usage_fold:

SELECT id, org_id, subscription_id, period_start, period_end,
       byte_seconds, gb_hours_exact, gb_hours_per_object,
       gb_hours_mpu, mpu_byte_seconds, billing_mode, gb_hours_billed,
       gpu_seconds, closed_at, invoice_id
  FROM metering.usage_period_closures
 WHERE org_id = '<org_id>'
 ORDER BY period_start;

invoice_id IS NULL means the period is frozen and not yet linked to an invoice. Finalize is what sets it. Do not set it by hand. One organization for one period:

SELECT * FROM metering.usage_period_closures
 WHERE org_id = '<org_id>'
   AND period_start = '<period_start>'
   AND period_end = '<period_end>';

8. Reconciliation¶

8.1 Two sources¶

Source	Granularity	Role
CloudWatch `BucketSizeBytes`	One total per bucket, daily	A quick check only. It has no prefix dimension, so usage moved between two orgs still matches.
S3 Inventory	One row per key, daily	The real comparison. Only it can check attribution.

The inventory report must be daily CSV with the optional fields Size, LastModifiedDate and ETag. LastModifiedDate dates a synthetic create. Without ETag, a synthetic create leaves object_size_intervals.etag empty, and §4 cannot tell which of several uploads completed.

8.2 Nightly comparison¶

At 02:15 UTC metering-inventory-load finds the newest report, checks its checksums, loads it into metering.inventory_staging and reads T, the report's own creation time. Then it calls metering.reconcile(p_manifest_t, p_apply). The loader stops before the comparison if T is in the future or more than 7 days old. It also stops if the loaded row count differs from the report, or if a key names an org that does not exist.

Rule: compare as of T, not as of the current time. A snapshot at T does not show later changes. So liveness is metering.was_live_at(created_at, deleted_at, T): created_at <= T AND (deleted_at IS NULL OR deleted_at > T).

The comparison runs in both directions. One direction finds only missed creates, but a missed delete bills a customer for an object that is gone, for ever.

Diff	Finds	Repair
A	In the inventory, not live in the ledger at `T`: a missed create. Keys with an unmatched delete are left to Diff D.	A synthetic create: `source = 'inventory_backfill'`, `event_time = LastModifiedDate`, the report's ETag, the all-zeros sequencer.
B	Live in the ledger at `T`, created before `T`, absent from the inventory: a missed delete.	A synthetic delete: `source = 'inferred'`, `event_time = T`, the all-`f` sequencer. The true delete lies between the previous `T` and this one, so the error is at most one inventory interval.
C	A size that differs from the interval current at `T`.	The size of that interval's create is updated in place and `folded_at` cleared, only when the inventory ETag equals the interval's.
D	An unmatched delete whose key is in the inventory.	A synthetic create at `LastModifiedDate`, only when the delete is after `T`. A delete at or before `T` is a contradiction: recorded, never repaired.

Rule: repairs are written as events, never as lifetimes. Every repair goes into object_events, and the fold applies it like any event. So the fold stays the single writer of the ledger, and every fold rule applies to repairs. A repair and the real event converge in either order, because the fold prefers real events (§3.4).

Diff C needs matching ETags. A size difference means either the create stored the wrong size, or the inventory shows a newer version overwritten just after T. In the second case, the new size on the old create would bill the old version's whole span at the wrong size. With a different or missing ETag, Diff C reports and writes nothing. It only updates an existing create, named by the interval's sequencer, because an insert would look like an overwrite.

Diff D orders the delete against T. A key in the snapshot existed at T. If the delete was after T, a create at LastModifiedDate is safe. If it was at or before T, that create would be later than the delete, and the fold would park the key as delete_before_create.

Findings are stored per org and per diff type in metering.reconciliation_findings, never per key, which would copy the ledger. The keys stay in inventory_staging. byte_delta is ledger minus inventory. The loader publishes ReconciliationKeyCount, ReconciliationByteDelta and ReconciliationRepaired per diff type, zeros included. Rule: compare sets of keys, never total bytes alone, because two opposite errors can sum to zero.

Repairs are off by default. reconcile() has p_apply DEFAULT false, and the loader applies repairs only when its APPLY setting is true; an invoke payload cannot switch them on. Diff B is the reason. A wrong inferred delete closes the lifetime of an object that still exists. The next night Diff A writes a create after deleted_at, and the fold parks the key as resurrected_key. Repairs stay off until several nights of findings have been read, as described in Operations. The pipeline epoch exemption is in §9.5.

8.3 Standing checks¶

Check	When	What it does
Canary	hourly probe, checked every 5 minutes	`metering-canary` writes and deletes a probe under `orgs/<CANARY_ORG_ID>/canary/`. `metering.canary_check()` is healthy only if a probe with its exact name completed a create and a delete in the ledger in the last 2 hours.
Versioning guard (A1)	every 5 minutes	`GetBucketVersioning` on each metered bucket. A failed call counts as not healthy.
Bucket-total check	every 5 minutes	`metering.ledger_live_bytes()` against the newest `BucketSizeBytes`, as `LedgerVsBucketPct`.
Conflicts (A6, A7)	every 5 minutes	Events whose `last_conflict_at` is in the last hour (`EventConflicts`). An overwrite is not a conflict.
GPU against the AWS bill	once per period, by hand	`lambda/metering-gpu-cur-check/check.py` reads a Cost and Usage export and calls `metering.gpu_cur_check()`.
Monthly bill check	monthly, by hand	What AWS charged against what the ledger says. Request fees appear here (§13).

The probe must be inside a real org prefix. A probe outside orgs/{uuid}/ is quarantined (A2) and would raise that alarm every hour. The org is billing_config.canary_org_id, and it is never closed or invoiced. The canary reports unhealthy, never healthy, when it has no data: no canary_org_id, no probe yet, a create without a delete, or an old probe.

Rule for the GPU check: convert the bill up, never the meter down. It compares instance_hours × gpu_count(instance_type) with metered GPU-hours; dividing the meter by gpu_count would hide an 8 times under-bill on a p5. It matches on instance_type, not gpu_model, because p4d and p4de are both A100. It covers provider = 'aws' only. An instance type missing from gpu_instance_types is returned with mapped = false, not dropped.

9. Operations design¶

This section explains why the running system has its shape. For deployment and procedures, see Operations.

9.1 Ingest concurrency¶

One org uploading many files makes SQS start many Lambdas. They could use up the connections of the database that also serves the application. So:

ingest is one multi-row statement per batch, with no reads (§3.3);
metering-ingest has a hard reserved concurrency of 20;
SQS sends batches of up to 100 messages, with a batching window;
every Lambda connects directly with a 15-minute IAM token over TLS, so no password is stored. metering-ingest replaces its connection after 10 minutes.

The limit is intended: the queue absorbs a burst and drains it, and billing is not real time. Alarm on the age of the oldest message, not on queue depth. A deep queue is normal in a burst. A rising age means ingest cannot keep up.

9.2 Scheduled jobs¶

The database schedules nothing. Lambdas started by EventBridge Scheduler send SQL, which gives logs, metrics, alarms and schedules that can be paused without a migration.

Lambda	When	Calls
`metering-jobs` `{"job":"fold"}`	every 5 minutes	`metering.fold()`
`metering-jobs` `{"job":"nightly"}`	03:30 UTC	`metering.run_nightly()`
`metering-jobs` `{"job":"gpu_reaper"}`	every 5 minutes; does nothing unless `GPU_METERING_ENABLED=true`	`metering.reap_gpu_intervals()`
`metering-inventory-load`	02:15 UTC	loads the report, then `metering.reconcile()`
`metering-mpu-sweep`	hourly	`metering.sweep_mpu()`, once per bucket
`metering-canary`	hourly	writes and deletes the probe object
`metering-health`	every 5 minutes	`metering.health_metrics()` and the AWS checks, to CloudWatch

Rule: every job is single-writer and idempotent. The job Lambdas run with reserved concurrency 1, and each job function takes a transaction-scoped advisory lock. So an early start or a long run waits and never overlaps. Every connection runs SET TIME ZONE 'UTC' first.

The nightly chain is an ordered list. metering.run_nightly() holds one lock, hashtext('metering.run_nightly'), for the whole chain, and returns a JSON report per step:

Step	Function	Why here
0	`metering.reconcile()` (02:15, separate Lambda)	Writes repair events. The clock puts it first.
1	`metering.resolve_fold_quarantine()`	Frees parked keys, so step 3 folds them the same night.
2	`metering.reap_gpu_intervals()`	So the rollup reads GPU figures that cannot still grow.
3	`metering.fold_to_completion()`	Folds every waiting event, repairs included. The 5-minute fold does not promise to finish.
4	`metering.rollup()`	Recomputes non-final days, which moves `computed_at`.
5	`metering.drift_scan()`	Reads the recomputed days.
6	`metering.mpu_audit()`	Checks the meter after the rollup and before any pruning.
7	`prune_events()`, `prune_quarantine()`, `prune_mpu_lifetimes()`, `prune_superseded_backfill()`	Last, so nothing it deletes is still needed. A dry run unless pruning is on (§9.5).

Rule: the drift scan runs after the rollup. computed_at is the scan's only signal, and the rollup moves it. In the other order nothing fails, but every correction appears one night late.

9.3 Dead-letter queues¶

Without a dead-letter queue (DLQ), a message that always fails is deleted at the retention limit, and nobody sees the loss. So there are two:

DLQ	Catches	Setting
SQS redrive DLQ	A message the Lambda cannot process	`maxReceiveCount` 5. Alarm on depth above 0.
EventBridge rule-target DLQ	EventBridge failing to deliver to a target: the SQS queue or the archive stream. It retries for about 24 hours, then drops the event.	A DLQ on both rule targets. Alarm on depth above 0.

Both keep messages for 14 days, the SQS maximum. The event source mapping uses ReportBatchItemFailures, so one bad record does not send back its whole batch. Redriving a DLQ is safe, because event_time comes from the event, not from the time of delivery (§3.7).

9.4 Raw event archive¶

The EventBridge rule has a second target: Firehose writes every S3 event to a separate bucket, <ARCHIVE_BUCKET>, as gzip-compressed JSON under raw-events/dt=YYYY-MM-DD/. Objects move to a Glacier storage class after 90 days and are kept for 7 years. The archive is not on the billing path. It is needed:

when the ingest code wrote events wrongly. The §3.4 rebuild reads object_events, so only the archive holds what S3 said;
when a window older than the retention horizon must be rebuilt. Replay the archive into object_events first;
as evidence in a billing dispute, and as history for testing a pricing change.

EventBridge's own archive cannot replace it. Its replay goes to the live targets, so it cannot rebuild into a separate environment, and its retention is limited.

9.5 Pipeline epoch, backfill and pruning¶

Pipeline epoch and backfill. Objects that existed before measuring started have no create event. metering.billing_config.pipeline_epoch records when measuring started; before it the ledger is not the authority. Diff A gives each such object a synthetic create (event_time = LastModifiedDate, the report's ETag, the all-zeros sequencer, source = 'inventory_backfill') through the normal write path.

The epoch exemption applies to Diff A only. Diff A skips a key whose lifetime came from inventory_backfill and starts before pipeline_epoch. This stops Diff A and Diff B from undoing each other's repair every night. Diffs B, C and D still apply: a backfilled object that is later deleted must still be closed. pipeline_epoch NULL turns the exemption off.

Retention horizon. metering.retention_horizon(p_margin) takes the earliest current_period_start among subscriptions whose current period has no closure, minus p_margin (90 days). With no such subscription it is -infinity, and nothing is pruned. It is derived, not set by hand, because a date set by hand would fall behind and delete events inside an open period.

What pruning deletes. metering.prune_events() deletes in batches of 50,000, driven by the lifetime, not by the event's own time:

SELECT x.ctid
FROM metering.object_events x
JOIN metering.object_lifetimes l ON l.bucket = x.bucket AND l.key = x.key
WHERE x.folded_at IS NOT NULL AND l.deleted_at IS NOT NULL AND l.deleted_at < horizon
LIMIT p_batch;

Clause	What it protects
`x.folded_at IS NOT NULL`	An unfolded event, such as a waiting repair, is not in the ledger yet.
`l.deleted_at IS NOT NULL`	A live object keeps its create event. Otherwise a later inferred delete finds no create, the fold produces nothing, and the object bills for ever.
`l.deleted_at < horizon`	Nothing in a period that can still be billed or corrected.
the `JOIN`	An event with no lifetime, such as an unmatched delete or a parked key, still waits to be resolved.

Table	Retention
`object_events`	`prune_events()`, as above. `prune_superseded_backfill()` deletes a synthetic create once the real create exists and both are folded.
`object_lifetimes`, `object_size_intervals`, `usage_daily`, `usage_period_closures`, `usage_adjustments`, `billing_periods`, `public.usage_daily`	Never pruned. They are the ledger and the financial record.
`usage_quarantine`, `fold_quarantine`	`prune_quarantine()` deletes rows resolved more than 90 days ago. An unresolved row is never pruned: it is the only record of storage nobody is billed for.
`mpu_lifetimes`	`prune_mpu_lifetimes()` deletes uploads closed before the horizon and, in the same statement, adds their byte-seconds to `mpu_audit_baseline`, so `mpu_audit()` stays exact.
`gpu_jobs`, `gpu_job_intervals`, `reconciliation_findings`	No pruning job. Rows are kept.
`inventory_staging`	The loader keeps the newest `KEEP_SNAPSHOTS` reports (2).

A new event for a pruned key counts no conflict (§3.3), because nothing is left to collide with. The fold still parks it as resurrected_key, because it checks the lifetime row, which is kept.

Pruning is the only step that cannot be undone, so it is a dry run by default. Each prune function has p_dry_run DEFAULT true and reports what it would delete. The nightly chain deletes only when metering-jobs has PRUNE=true. Switch it on only when the raw event archive (§9.4) holds every event that pruning would delete. After pruning, the archive is the only way to rebuild those windows.

10. Security and tenancy¶

The application has no path into the metering schema. The metering tables are cross-tenant by nature: the fold reads every org's events in one pass. So tobor_app has no USAGE on the schema and no table or function grant in it, and metering_rls.sql revokes all three anyway. A policy mistake inside metering cannot leak one tenant's usage to another.

Role (`NOLOGIN`)	Used by	Can do
`metering_owner`	nobody at run time	Owns the schema, tables and `metering` functions. Writes only `billing_config` and `gpu_instance_types`. Outside the schema: only `REFERENCES` on `organizations`, `subscriptions`, `subscription_plans`, `invoices`.
`usage_ingest`	`metering-ingest`	`SELECT`, `INSERT`, `UPDATE` on `object_events`; `INSERT` on `usage_quarantine`; `ingest_events()`, `quarantine_event()`, `org_id_from_key()`. Nothing else.
`usage_fold`	`metering-jobs`, `-inventory-load`, `-mpu-sweep`, `-canary`, `-health`	DML on every other metering table, and `SELECT` only on `billing_config` and `gpu_instance_types`. Reads `organizations`, `subscriptions`, `invoices`, `upload_parts`, `episodes` and `subscription_plans` across orgs. Writes `public.usage_daily` (no `DELETE`). Owns the app entry points.
`usage_heartbeat`	the training orchestrator	`SELECT`, `INSERT`, `UPDATE` on `gpu_jobs` and `gpu_job_intervals`; `SELECT` on `gpu_instance_types`; the four functions of §5. Not the reaper, not `usage_daily`.
`tobor_app`	the application	The seven entry points of §7.2. Reads `public.usage_daily` under its `org_isolation` policy; `INSERT`, `UPDATE`, `DELETE` on it are revoked.

A deployment gives the Lambda roles login through IAM authentication (see Operations). usage_ingest needs SELECT and UPDATE, not only INSERT, because the §3.3 upsert reads and updates the stored row. With INSERT alone it fails at run time. usage_fold cannot change billing_config, because that row decides what customers pay. usage_heartbeat cannot write usage_daily, which would be a second, unreviewed path to a bill.

Row-level security is enabled and forced on every metering table, with per-role policies: fold_writer, fold_reader, ingest_writer, heartbeat_writer, heartbeat_reader and owner_writer. No role has BYPASSRLS. Policies accept TO role, and permissive policies are combined with OR, so no bypass is needed.

Rule: every table with RLS needs a policy for each role that uses it. With no matching policy the table reads as empty and an UPDATE reports UPDATE 0, with no error. That is why billing_config and gpu_instance_types have owner_writer, and why gpu_instance_types has heartbeat_reader. A dedicated owner keeps FORCE meaningful: a table's owner can change its policies, and a superuser ignores RLS even with FORCE. So metering_owner, a plain NOLOGIN role, owns the tables. Every metering function is SECURITY INVOKER, so every grant stays in force; the view metering.unsettled_adjustments is security_invoker too.

The jobs read some public tables across orgs. Tenant tables in public have the policy org_id = app_current_org(). A job sets no app.current_org, so it would see nothing and fail quietly; the fold would park every key as unknown_org. So organizations, subscriptions, invoices, upload_parts and episodes have a second permissive policy, fold_reader (FOR SELECT TO usage_fold), and public.usage_daily has fold_writer. subscription_plans has no RLS, so a grant is enough. The orphan report (§4) is the only job that reads tenant content, and it returns counts per episode only.

The application reaches billing only through SECURITY DEFINER functions in public, defined by queries/metering_app.sql, owned by usage_fold, with search_path set to public, metering, pg_temp. EXECUTE is revoked from PUBLIC and granted to tobor_app only: usage_periods_due(), draft_usage_invoice(), finalize_usage_invoice(), draft_corrected_usage_invoice(), finalize_corrected_usage_invoice(), freeze_usage_period() and unsettled_usage_adjustments(). Those that act on one org check app.current_org, most through public.usage_assert_org() (§7.2). The definer, usage_fold, is bound by the policies above. The trigger function public.record_billing_period() is also SECURITY DEFINER. usage_fold has CREATE on public only while metering_app.sql runs; the file revokes it at the end.

11. Alarms¶

Principle: every failure path ends in a metric and an alarm. A failure without an alarm becomes a wrong invoice. Every metric is published on every run, zeros included, or a healthy system and a stopped publisher would look the same. HealthPublisherRun and ReconciliationRun are sent last, only after a complete run. Metrics use the namespace Toborverse/Metering. Most come from one read-only query, metering.health_metrics(), published every 5 minutes by metering-health.

The alarms fall into these groups:

the pipeline is alive: the canary, the publisher runs, Lambda errors;
ingest: queue age, both DLQs, the per-record fallback, usage_quarantine depth;
the fold: lag (without parked keys), fold_quarantine depth, old unmatched deletes, conflicts, corrections;
billing: old non-final days, old unsettled adjustments, a drift scan that owed a row, the number of adjustments created;
reconciliation and AWS checks: each diff type, bucket totals, versioning;
multipart: uploads past the grace period, accrual differences, ambiguous completions, orphaned parts;
GPU: reopens and heartbeat timeouts.

Both quarantines alarm at a depth above 0. A row in either is billable storage with no ledger row, not a queue for a weekly review. For every alarm, its threshold and what to do, see Operations.

12. Tests¶

Database tests. database/tests/run_pipeline.sh applies the whole schema in dependency order and runs every file in database/tests/, stopping at the first failure. The metering tests are files 14 to 25. Use a fresh PostgreSQL 18 container every time: queries/rls.sql creates tobor_app with no existence check, and roles are server-wide.

docker run -d --rm --name tobor-pg18 -e POSTGRES_PASSWORD=secret -e POSTGRES_DB=tobor_test \
  -v "$PWD/database:/database:ro" postgres:18
until docker exec tobor-pg18 psql "postgres://postgres:secret@localhost:5432/tobor_test" \
  -c 'select 1' >/dev/null 2>&1; do sleep 2; done
docker exec tobor-pg18 bash /database/tests/run_pipeline.sh \
  "postgres://postgres:secret@localhost:5432/tobor_test"
docker stop tobor-pg18

Run it from the repository root. Wait with a real psql connection, not pg_isready: the server reports ready once during its first start-up and then restarts.

Files	Cover
`14`, `15`	The fold, and the convergence property (below)
`16`, `17`	Billing math and drift; grants and RLS, run under `SET ROLE`
`18`, `19`, `20`	Ingest; the nightly order and pruning; reconciliation
`21`, `22`, `23`	Multipart uploads; GPU; health signals, each checked to fire and to stay quiet
`24`, `25`	The app entry points and the full invoice path

Convergence property. Test 15 folds random create, overwrite and delete streams once in one batch, and once shuffled into random batches with duplicates. The two ledgers must be identical. It checks unparked keys only: a parked key keeps its old state by design (§3.4).

Worked examples in the billing tests:

Object	`gb_hours_exact`	`gb_hours_per_object`
1 GiB for 40 minutes, no overwrite	0.666…	exactly 1.0
100 MiB for 2 hours, then 40 MiB for 3 hours	0.3125	0.3125
1 GiB, overwritten to 0.5 GiB at 20 minutes, deleted at 40 minutes	0.5	0.666… (0.833… would mean padding at the first size)

Two races need two concurrent sessions, which psql cannot drive. One is an event that commits while the fold runs. The other is two heartbeats that reopen a job at once. The code handles both (§3.4, §5).

Lambda tests replace AWS and PostgreSQL with stubs and need neither. Each exits non-zero on a failure. From the ToborverseDataManagement folder:

python3 -I lambda/metering-ingest/test_handler.py
python3 -I lambda/metering-mpu-sweep/test_sweep.py
python3 -I lambda/metering-mpu-sweep/test_parts_etag.py
python3 -I lambda/metering-health/test_handler.py
python3 -I lambda/metering-inventory-load/test_handler.py
python3 -I lambda/metering-gpu-cur-check/test_check.py

13. Not measured¶

Item	On the invoice	Why
Storage classes	`cold_storage_gb` and `cold_storage_cents` are 0. Everything bills as standard storage.	Lifecycle transitions are not tracked. A class changes over time like size, so it would fit as a column on `object_size_intervals`.
Data transfer out	`streaming_gb` and `streaming_cents` are 0. `streaming_rate_per_gb` and `included_streaming_gb` are not used.	S3 sends no event for a read. It needs CloudFront logs, access logs or CloudTrail data events, with its own attribution.
Request fees	Not billed.	Out of scope. They appear as a steady, expected difference in the monthly bill check (§8.3).

The draft invoice still returns the plan's cold_storage_rate_per_gb and streaming_rate_per_gb, so the invoice shows the rate it would use.

14. Settings¶

A setting that changes what a customer pays lives in the one row of metering.billing_config, which only metering_owner can write. A setting that changes only when or how a job runs is a function parameter or a Lambda environment variable. To change them, see Operations.

Setting	Value	Where it lives
Billing (rounding) mode	`per_object`	`billing_config.billing_mode` (`per_object`, `per_org`, `exact`)
Restatement threshold, storage	1 GB-hour	`billing_config.restatement_threshold_gb_hours`
Restatement threshold, GPU	1 GPU-hour	`billing_config.restatement_threshold_gpu_hours`
Canary org	none (canary reports unhealthy)	`billing_config.canary_org_id`
Pipeline epoch	none (exemption off)	`billing_config.pipeline_epoch`
Invoice storage unit	GB-month: `gb_hours_billed / period_hours`	`metering.draft_invoice()`; rate in `subscription_plans.storage_rate_per_gb`
Pricing precision	quantity to 4 decimals; cents half away from zero	`metering.price_usage()`
Grace window	48 hours	`rollup(p_grace)`, `run_nightly(p_rollup_grace)`, `periods_due(p_grace)`, `draft_invoice(p_grace)`, `health_metrics(p_rollup_grace)`; `ROLLUP_GRACE` on `metering-jobs` and `metering-health`; built into `freeze_usage_period()`
Rollup lookback	35 days	`rollup(p_lookback)`, `run_nightly(p_rollup_lookback)`; `ROLLUP_LOOKBACK` on `metering-jobs`
Drift horizon	90 days	`drift_scan(p_horizon)`, `run_nightly(p_drift_horizon)`, `drift_producer_gaps(p_horizon)`; `DRIFT_HORIZON` on `metering-jobs`
Fold batch	10,000 events	`fold(fold_batch)`, `fold_to_completion(p_batch)`, `run_nightly(p_fold_batch)`
Fold passes per night, at most	1,000	`fold_to_completion(p_max_passes)`
Multipart grace	3 hours	`metering.mpu_grace()`
Multipart abort age	7 days	the bucket's `AbortIncompleteMultipartUpload` rule, passed as `sweep_mpu(p_lifecycle_abort_age)`
Multipart audit tolerance	0.000001 byte-seconds	`mpu_audit(p_tolerance)`
Orphaned parts age	7 days	`mpu_orphan_parts(p_age)`
Metered buckets	`<DATA_BUCKET>`	`METERED_BUCKETS` on `metering-mpu-sweep` and `metering-health`
Heartbeat interval	60 seconds	`reap_gpu_intervals(p_heartbeat_interval)`, and the orchestrator
Reaper multiple	3	`reap_gpu_intervals(p_multiple)`
GPU metering	off	`GPU_METERING_ENABLED` on `metering-jobs`
Retention margin	90 days	`retention_horizon(p_margin)`
Resolved quarantine kept	90 days	`prune_quarantine(p_keep_days)`
Prune batch	50,000 rows	`prune_events(p_batch)`, `prune_superseded_backfill(p_batch)`, `prune_mpu_lifetimes(p_limit)`
Pruning	dry run	`PRUNE` on `metering-jobs`; `run_nightly(p_prune)`; `p_dry_run` on each prune function
Reconciliation repairs	off	`APPLY` on `metering-inventory-load`; `reconcile(p_apply)`
Inventory reports kept	2	`KEEP_SNAPSHOTS` on `metering-inventory-load`
Oldest usable inventory report	7 days	built into `metering-inventory-load`
Key decoding	`none`	`KEY_DECODE` on `metering-ingest`
Log raw keys	`true`	`LOG_RAW_KEYS` on `metering-ingest`
Unmatched delete age	24 hours	`unmatched_deletes(p_age)`
Canary window	2 hours	`canary_check(p_since)`, `health_metrics(p_canary_within)`; `CANARY_WITHIN` on `metering-health`
Health window	1 hour	`health_metrics(p_window)`; `WINDOW` on `metering-health`
Bucket-total tolerance	5 percent	`SMOKE_TOLERANCE_PCT` on `metering-health`
Ingest reserved concurrency	20	the `metering-ingest` function
SQS batch size	100	the `metering-ingest` event source mapping
Deliveries before the SQS DLQ	5	the ingest queue's redrive policy (`maxReceiveCount`)
DLQ retention	14 days	both dead-letter queues
Metric namespace	`Toborverse/Metering`	`METRIC_NAMESPACE` on the Lambdas