Operations¶

This page explains how to run usage metering: the AWS parts, the database deploy, the function settings, the alarms, the daily checks, and three settings that start switched off.

For how the system works and why, read how-it-works.md.
For the terms on this page (fold, ledger, Diff A, closure), see the glossary in index.md.
For every table and column, see the schema reference.

1. Placeholders¶

This page uses placeholders instead of real names and ids. Replace each one with the value for your environment. Real values are not kept in the repository.

Placeholder	Meaning
`<AWS_ACCOUNT_ID>`, `<AWS_REGION>`	The AWS account and region
`<PRIVATE_SUBNET_A>`, `<PRIVATE_SUBNET_B>`	The two private app subnets. They hold the interface endpoints
`<RDS_SG>`, `<APP_EC2_SG>`	The database security group, and the security group of the app server
`<APP_INSTANCE_ID>`	The app EC2 instance. It has no public address. Reach it with Session Manager
`<AURORA_WRITER_ENDPOINT>`	The Aurora writer endpoint. It contains `.cluster-`, not `.cluster-ro-`
`<AURORA_CLUSTER_RESOURCE_ID>`	The cluster resource id (`cluster-` and letters). IAM `rds-db:connect` ARNs use it. It is not the cluster name
`<DB_CLUSTER_ID>`	The cluster identifier (its name)
`<DB_MASTER_USER>`, `<DB_SECRET_ARN>`	The master user, which owns the schema, and the secret that holds its password
`<DATA_BUCKET>`	The metered data bucket
`<INVENTORY_BUCKET>`	The bucket that receives the S3 Inventory reports
`<ARCHIVE_BUCKET>`, `<FIREHOSE_STREAM>`	The raw-event archive bucket, and the Firehose stream that writes it
`<ARTIFACTS_BUCKET>`	The private bucket for build artifacts. Never the data bucket
`<COST_EXPORT_BUCKET>`	The bucket that receives the AWS cost export
`<ALERTS_SNS_TOPIC_ARN>`	The SNS topic that every metering alarm notifies
`<CANARY_ORG_ID>`	The canary organization

Some tasks also need a value that you find during the task. Examples are <KEY> (one object key), <ORG_ID> (one organization) and <MANIFEST_T> (the time of one inventory snapshot).

2. The parts¶

Part	What it does	Trigger and time (UTC)	Database role
Data bucket `<DATA_BUCKET>`	Holds customer objects under `orgs/{org id}/`. EventBridge notifications on. Versioning off. A lifecycle rule aborts incomplete multipart uploads after 7 days	—	—
EventBridge rule `metering-s3-events`	On the default bus. Matches `Object Created` and `Object Deleted` from `<DATA_BUCKET>`. Sends each event to the queue and to the archive stream	Each S3 event	—
SQS queue `metering-ingest`	Holds events for the ingest function. Standard queue, retention 14 days, visibility timeout 360 s, maximum receives 5	Each event	—
SQS queue `metering-ingest-dlq`	Receives a message after the ingest function fails on it 5 times. Retention 14 days	Redrive policy of `metering-ingest`	—
SQS queue `metering-eventbridge-dlq`	Receives events that EventBridge could not deliver to either target. Retention 14 days	Dead-letter queue of both targets	—
Lambda `metering-ingest`	Writes events to `metering.object_events`, and messages it cannot use to `metering.usage_quarantine`	SQS, batch size 100, batch window 5–20 s	`usage_ingest`
Firehose `<FIREHOSE_STREAM>`	Writes every event to `<ARCHIVE_BUCKET>` under `raw-events/dt=YYYY-MM-DD/`, gzip, one event per line. Buffer 128 MB or 300 s. The bucket moves files to Glacier Instant Retrieval after 90 days and deletes them after 2,555 days (7 years)	Second target of the rule	—
S3 Inventory `metering-daily`	A daily CSV list of every object in `<DATA_BUCKET>` with `Size`, `LastModifiedDate` and `ETag`. Current versions only. Lands in `<INVENTORY_BUCKET>`	Daily, at a time S3 chooses. The first report can take 48 hours	—
Lambda `metering-inventory-load`	Loads the newest report into `metering.inventory_staging`, then runs `metering.reconcile()`	`metering-inventory-load-nightly`, `cron(15 2 * * ? *)`: 02:15 UTC	`usage_fold`
Lambda `metering-jobs`	Runs the fold, the nightly chain (`metering.run_nightly()`) and the GPU reaper	`metering-fold`, `rate(5 minutes)`, payload `{"job": "fold"}`. `metering-nightly`, `cron(30 3 * * ? *)`: 03:30 UTC, payload `{"job": "nightly"}`. `metering-gpu-reaper`, `rate(5 minutes)`, payload `{"job": "gpu_reaper"}`, only with GPU metering (section 9)	`usage_fold`
Lambda `metering-mpu-sweep`	Lists open multipart uploads. Calls `metering.sweep_mpu()` once per bucket	`metering-mpu-sweep`, `rate(1 hour)`	`usage_fold`
Lambda `metering-canary`	Writes and deletes one test object under `orgs/<CANARY_ORG_ID>/canary/`	`metering-canary`, `rate(1 hour)`	`usage_fold`
Lambda `metering-health`	Publishes `metering.health_metrics()` and two AWS checks to CloudWatch	`metering-health-every-5-min`, `rate(5 minutes)`	`usage_fold`
Script `lambda/metering-gpu-cur-check/check.py`	Compares metered GPU-hours with the AWS cost export	By hand, once per billing period	`usage_fold`
SNS topic `<ALERTS_SNS_TOPIC_ARN>`	Receives every metering alarm	Alarm state changes	—

Rules for the parts:

S3 sends events to EventBridge only when EventBridge notifications are on for the bucket. They are off by default. Do not switch them on for <INVENTORY_BUCKET> or <ARCHIVE_BUCKET>.
Every schedule uses EventBridge Scheduler, time zone UTC, flexible time window off. A flexible window lets AWS move a run. The loader schedule retries twice, with a maximum event age of 1 hour.
The 02:15 UTC load must run before the 03:30 UTC nightly chain, so that its repairs become lifetimes before the rollup. A late load breaks nothing, but its repairs reach the meter a night later.
The queue policies on metering-ingest and metering-eventbridge-dlq must allow EventBridge only with aws:SourceArn equal to the rule's ARN. Without that condition, any rule can write into billing.

3. Deploying the database part¶

The database part is the metering schema: tables, functions, grants and row-level security (RLS). PostgreSQL 18 is required. The schema uses uuidv7() and uuid_extract_version().

3.1 Connect and choose a role¶

Aurora is reachable only from inside the VPC. <RDS_SG> allows port 5432 only from <APP_EC2_SG> and from the function security groups. Connect from the app instance:

aws ssm start-session --target <APP_INSTANCE_ID> --region <AWS_REGION>

export PGHOST=<AURORA_WRITER_ENDPOINT> PGPORT=5432 PGDATABASE=postgres PGUSER=<DB_MASTER_USER> PGSSLMODE=require
export PGPASSWORD="$(aws secretsmanager get-secret-value --region <AWS_REGION> --secret-id '<DB_SECRET_ARN>' \
  --query SecretString --output text | python3 -c 'import json,sys; print(json.load(sys.stdin)["password"])')"
export DATABASE_URL="postgres://<DB_MASTER_USER>@<AURORA_WRITER_ENDPOINT>:5432/postgres?sslmode=require"
psql -c 'select version();'          # must print PostgreSQL 18.x

The examples on this page use the database name postgres. Every table in metering uses forced RLS. Only the job roles have policies, and metering_owner has one on billing_config and gpu_instance_types. A role without a policy sees no rows and gets no error; an UPDATE reports UPDATE 0. So choose the role before you run SQL:

Task	Role
Read or change ledger tables, call metering functions	`SET ROLE usage_fold;`
Change `metering.billing_config` or `metering.gpu_instance_types`	`SET ROLE metering_owner;`
Deploy	The master user. `RESET ROLE;` returns to it

3.2 An existing database¶

Use this route on a database that already has the application schema. Connect as <DB_MASTER_USER>, the role that applied the migrations, and apply in this order:

Migrations 103–108 from backend/src/db/migrations/: 103_create_metering_schema.sql, 104_billing_metering_support.sql, 105_create_public_usage_daily.sql, 106_phase7_mpu_gpu.sql, 107_metering_fixes.sql, 108_organization_deactivation.sql.
database/queries/admin_lifecycle.sql: delete_org(), which refuses an organization with billing history, and deactivate_org().
database/apply_metering.sh: every metering function, the grants, the RLS policies and the application entry points.

cd ToborverseDataManagement
for f in backend/src/db/migrations/10{3,4,5,6,7,8}_*.sql; do
  echo "== $f"; psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f "$f" || break
done
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f database/queries/admin_lifecycle.sql
database/apply_metering.sh "$DATABASE_URL"

Migrations create and change tables only; the functions live in database/queries/. Do not deploy the migrations alone: every Lambda then fails on its first call.
apply_metering.sh runs all its files in one transaction, so a failure changes nothing. Run it again after any change to a file it lists. Run it as the schema owner: it uses SET ROLE metering_owner and SET ROLE usage_fold, and it creates a trigger on public.subscriptions. Migration 107 and apply_metering.sh change nothing on a second run.

3.3 An empty database¶

Do not run the migrations. Apply the schema files in the order of database/tests/run_pipeline.sh:

cd ToborverseDataManagement/database
for f in schemas/schema.sql schemas/metering.sql queries/triggers.sql queries/rls.sql \
         queries/versioning_triggers.sql queries/admin_lifecycle.sql queries/set_revisions.sql \
         queries/group_revisions.sql queries/invites.sql; do
  echo "== $f"; psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f "$f" || break
done
./apply_metering.sh "$DATABASE_URL"

schema.sql must come before metering.sql: the ledger has foreign keys to organizations, subscriptions and invoices. rls.sql must come before apply_metering.sh: metering_rls.sql revokes rights from tobor_app, which rls.sql creates. Do not run both the migrations and schema.sql on one database. They create the same objects and collide.

3.4 Check the deploy, and test SQL changes first¶

SELECT count(*) FROM pg_tables                    -- expect 0
 WHERE schemaname = 'metering' AND tableowner <> 'metering_owner';
SELECT p.proname FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
 WHERE n.nspname = 'metering' AND p.proname IN ('ingest_events','fold','run_nightly','reconcile',
       'sweep_mpu','health_metrics','canary_key_suffix') ORDER BY 1;          -- expect 7 rows

Every change under database/ must pass database/tests/run_pipeline.sh on a fresh PostgreSQL 18 container before you apply it; see how-it-works.md, §12. The tests write test data, so never run them against a real database. Use a fresh container for each run, because queries/rls.sql creates tobor_app without a guard. Record the result in the usage-metering section of DATABASE.md.

3.5 Let the functions log in¶

The roles are created NOLOGIN and have no password. Each function signs an IAM token from its execution role. The token is valid for 15 minutes, and TLS is required. Run once, as the master user:

GRANT rds_iam TO usage_ingest;  ALTER ROLE usage_ingest LOGIN;
GRANT rds_iam TO usage_fold;    ALTER ROLE usage_fold   LOGIN;
SELECT rolname, rolcanlogin, rolsuper, rolbypassrls FROM pg_roles
 WHERE rolname LIKE 'usage_%' OR rolname = 'metering_owner';   -- rolsuper and rolbypassrls: f

Do not give these roles a password. usage_heartbeat gets the same statements only for GPU metering (section 9). metering_owner never logs in. usage_fold also needs TEMPORARY on the database, because the fold, metering.reconcile(), metering.prune_events() and metering.sweep_mpu() create temporary tables. PUBLIC has it by default. Each execution role needs one rds-db:connect statement for exactly one database user. Get the resource id with aws rds describe-db-clusters --db-cluster-identifier <DB_CLUSTER_ID> --query 'DBClusters[0].DbClusterResourceId'.

{ "Effect": "Allow", "Action": "rds-db:connect",
  "Resource": "arn:aws:rds-db:<AWS_REGION>:<AWS_ACCOUNT_ID>:dbuser:<AURORA_CLUSTER_RESOURCE_ID>/usage_fold" }

Do not put the cluster name in the ARN. The login then fails with PAM authentication failed, which looks like a network problem. Do not use * as the user: the role can then log in as anyone.

3.6 Set the canary organization¶

The canary writes its test object into a real organization that is never invoiced. Create it once per environment, as the master user:

BEGIN;
SELECT uuidv7() AS canary_id \gset
SET LOCAL app.current_org = :'canary_id';
INSERT INTO organizations (id, name, slug) VALUES (:'canary_id', 'Pipeline Canary', 'pipeline-canary');
SET LOCAL ROLE metering_owner;
UPDATE metering.billing_config SET canary_org_id = :'canary_id', updated_at = now()
 WHERE singleton;                       -- must report UPDATE 1
COMMIT;

Until this is set, metering.canary_check() reports unhealthy and metering-canary fires. Also set
the pipeline epoch (section 10.5).

4. Function settings¶

4.1 Settings that every function shares¶

Setting	Value
Runtime and handler	Python 3.12, `handler.lambda_handler`. The console default `lambda_function.lambda_handler` does not exist in the zip
Code	A zip from the folder's `build.sh`. It adds `pg8000` and the RDS certificate bundle `global-bundle.pem`. `pg8000` is not in the Lambda runtime, so the console editor cannot be used
Network	Subnets `<PRIVATE_SUBNET_A>` and `<PRIVATE_SUBNET_B>`. Its own security group with no inbound rules. `<RDS_SG>` allows port 5432 from that group (name the group, not an address range)
Execution role	One role per function: `AWSLambdaVPCAccessExecutionRole`, plus `rds-db:connect` for its database user and the AWS calls listed for the function
Reserved concurrency	1. Only `metering-ingest` uses 20
`DB_HOST`	Required. `<AURORA_WRITER_ENDPOINT>`. The reader endpoint rejects writes, and the error appears only at the first write
`DB_PORT`, `DB_NAME`	`5432`. `DB_NAME` is required, for example `postgres`
`DB_USER`	`usage_ingest` for `metering-ingest`, `usage_fold` for every other function
`LOG_LEVEL`	`INFO`

Every connection runs SET TIME ZONE 'UTC'. Keep this in every handler: usage_daily.day is a UTC day, and a session in another time zone puts usage on the wrong day. The functions reach S3 through the S3 gateway endpoint. metering-inventory-load and metering-health also call CloudWatch. They need the NAT gateway or a com.amazonaws.<AWS_REGION>.monitoring interface endpoint.

To deploy a function, run its tests (python3 -I test_*.py), run ./build.sh, and upload the zip. The tests need no AWS account or database. metering-jobs and metering-canary have no tests.

aws lambda update-function-code --region <AWS_REGION> --function-name metering-jobs --zip-file fileb://metering-jobs.zip
aws lambda invoke --region <AWS_REGION> --function-name metering-jobs \
  --payload '{"job": "fold"}' --cli-binary-format raw-in-base64-out /dev/stdout     # run it by hand

Change one variable. update-function-configuration --environment replaces every variable, not only the one you name. Read, change, write back:

F=metering-inventory-load
aws lambda get-function-configuration --region <AWS_REGION> --function-name $F \
  --query '{Variables: Environment.Variables}' > env.json
python3 -c 'import json,sys; d=json.load(open("env.json")); d["Variables"][sys.argv[1]]=sys.argv[2]; json.dump(d,open("env.json","w"))' APPLY true
aws lambda update-function-configuration --region <AWS_REGION> --function-name $F --environment file://env.json

4.2 `metering-ingest`¶

Memory 512 MB, timeout 30 s, reserved concurrency 20, which caps database connections. The SQS trigger uses batch size 100, a batch window of 5–20 s, and report batch item failures; without it, one bad record sends the whole batch back. The queue visibility timeout must be at least six times the function timeout. Permissions: sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on metering-ingest, and rds-db:connect as usage_ingest. It needs no S3 access, because the event carries the bucket, key, size, ETag and sequencer.

Variable	Default	Meaning
`KEY_DECODE`	`none`	How to decode the object key. `none`: as delivered. `urldecode`: percent-decoding. `urldecode_plus`: percent-decoding, and `+` becomes a space. Use `none` (section 10.6)
`LOG_RAW_KEYS`	`true`	Logs every raw key as `RAW KEY …`. Set it to `false` for normal use, and to `true` only while you check key encoding
`METRIC_NAMESPACE`	`Toborverse/Metering`	Namespace of the two metrics below

The input is an SQS event. batchItemFailures in the response lists only records that could not even be quarantined; SQS delivers those again. Each invocation writes one CloudWatch embedded-metric log line, zeros included, so it needs no cloudwatch:PutMetricData permission. The metrics are IngestFallbackBatches (1 when the batch fell back to one record at a time) and IngestRecordsQuarantined.

4.3 `metering-jobs`¶

Memory 512 MB, timeout 15 minutes, reserved concurrency 1, statement limit 840 s. Permissions: rds-db:connect as usage_fold only.

Payload	What it does
`{"job": "fold"}`	Runs `metering.fold()` once. Returns `events_folded`, `keys_touched`, `keys_parked`. An event without `job` also runs the fold
`{"job": "nightly"}`	Runs `metering.run_nightly()`. Returns its report (section 6.4)
`{"job": "gpu_reaper"}`	Runs `metering.reap_gpu_intervals()` when `GPU_METERING_ENABLED` is `true`. Otherwise returns `"skipped": true`

Variable	Default	Meaning
`ROLLUP_GRACE`	`48 hours`	How long after its UTC day a day stays open for late events
`ROLLUP_LOOKBACK`	`35 days`	How far back the rollup looks for days that have ledger data but no row yet
`DRIFT_HORIZON`	`90 days`	How far back the drift scan checks closed periods. Keep it at least as long as the dead-letter queue retention plus one inventory day
`PRUNE`	`false`	`false`: pruning is a dry run that only counts. `true`: pruning deletes (section 8)
`GPU_METERING_ENABLED`	`false`	Lets the `gpu_reaper` payload run (section 9). The nightly chain runs the reaper in both cases

4.4 `metering-inventory-load`¶

Memory 1024 MB, timeout 15 minutes, reserved concurrency 1. Permissions: s3:GetObject and s3:ListBucket on <INVENTORY_BUCKET> and its objects, cloudwatch:PutMetricData with the condition cloudwatch:namespace equal to METRIC_NAMESPACE, and rds-db:connect as usage_fold. For what one run does, see how-it-works.md, §8.2.

Payload	What it does
`{}`	Loads the newest report. If that snapshot is already staged, skips the load and runs the diff again
`{"manifest_key": "<path of manifest.json>"}`	Loads one specific report
`{"force": true}`	Deletes and reloads a snapshot that is already staged
`{"apply": false}`	Runs once without repairs while `APPLY` is `true`. A payload cannot switch repairs on: while `APPLY` is `false`, `{"apply": true}` is ignored

Variable	Default	Meaning
`INVENTORY_BUCKET`	required	`<INVENTORY_BUCKET>`
`SOURCE_BUCKET`	required	`<DATA_BUCKET>`, the bucket that the report lists
`INVENTORY_ID`	`metering-daily`	The inventory configuration name. Reports are under `<SOURCE_BUCKET>/<INVENTORY_ID>/`
`APPLY`	`false`	`false`: record findings only. `true`: also write repairs (section 7)
`IGNORE_KEYS`	`deploys/frontend-dev-build.tar.gz`	Comma-separated keys left out of the snapshot. Use it only for an object that is never billed and must stay in the bucket. Build artifacts belong in `<ARTIFACTS_BUCKET>`
`COPY_BATCH`	`50000`	Rows per `COPY`
`KEEP_SNAPSHOTS`	`2`	Snapshots kept in `metering.inventory_staging`
`METRIC_NAMESPACE`	`Toborverse/Metering`	Must match the IAM condition
`SAFETY_MARGIN_MS`	`30000`	The function stops this long before the Lambda timeout, with a log message, so Lambda does not stop it during a `COPY`

The inventory configuration must be: name metering-daily, daily, CSV, current versions only, no prefix filter, and all three fields Size, LastModifiedDate and ETag. <INVENTORY_BUCKET> must be in the same region. Its bucket policy allows s3:PutObject for s3.amazonaws.com, limited by aws:SourceAccount and aws:SourceArn. Its lifecycle rule deletes reports after 90 days.

Each successful run publishes, zeros included: ReconciliationKeyCount, ReconciliationByteDelta and ReconciliationRepaired (dimension DiffType), ReconciliationStagedRows, UnattributableObjects, ReconciliationLoadSeconds, and last ReconciliationRun. When a load takes more than 15 minutes, run the same handler.py on the app instance or as a container task. Give it the same variables and permissions, and run python3 -c 'import handler; print(handler.lambda_handler({}, None))'. A None context switches off the deadline check.

Error or symptom	Cause
`no manifest under s3://…`	No report yet (allow 48 hours), the bucket policy refuses delivery, or `SOURCE_BUCKET` or `INVENTORY_ID` is wrong
No report after 48 hours	The inventory bucket policy. Remove any `s3:x-amz-acl` condition: the bucket uses bucket-owner-enforced ownership, so that condition refuses every delivery
`manifest T is in the future` or `more than 7 days old`	Inventory stopped delivering, or the manifest time is wrong
`fileSchema is missing …`, `fileFormat is …`	The inventory configuration lacks a field or is not CSV. A change applies from the next report
`checksum mismatch`, `is corrupt or truncated`	A file did not download fully. Run again. If it repeats, the report is bad
`org id(s) in the report do not exist in organizations`	A key names an unknown org. The diff did not run. A person must handle these keys

4.5 `metering-mpu-sweep`¶

Memory 512 MB, timeout 15 minutes, reserved concurrency 1. The payload is ignored. It returns one entry per bucket with counts such as observed, closed_completed, closed_aborted and closed_ambiguous. Permissions: s3:ListBucketMultipartUploads and s3:GetLifecycleConfiguration on each bucket, s3:ListMultipartUploadParts on its objects, and rds-db:connect as usage_fold. One variable: METERED_BUCKETS, required, a comma-separated list. If it is empty, the function refuses to run. Each metered bucket must have an AbortIncompleteMultipartUpload lifecycle rule of 7 days. It removes abandoned uploads, and the sweep records such a removal as lifecycle_abort. Without the rule, abandoned uploads stay, and MpuPastGrace only rises.

4.6 `metering-canary`¶

Memory 512 MB, timeout 60 s, reserved concurrency 1. The payload is ignored. One variable: CANARY_BUCKET, required, set to <DATA_BUCKET>. Permissions: s3:PutObject and s3:DeleteObject on arn:aws:s3:::<DATA_BUCKET>/orgs/*/canary/*, and rds-db:connect as usage_fold.

Each run reads canary_org_id from metering.billing_config. It writes the one-byte object orgs/<CANARY_ORG_ID>/canary/<YYYYMMDDTHHMMSSZ> a+b&c%20d.bin and deletes it. The name suffix comes from metering.canary_key_suffix(). If canary_org_id is NULL, it writes nothing and returns "skipped": true. metering.canary_check() passes only when a probe's create and delete both reached the ledger within 2 hours, with the exact name. Keep the canary and the health publisher in separate functions, so a publisher failure cannot look like a pipeline failure.

4.7 `metering-health`¶

Memory 512 MB, timeout 60 s, reserved concurrency 1. The payload is ignored. It returns metrics, unhealthy and seconds. Permissions: cloudwatch:PutMetricData with the namespace condition, cloudwatch:GetMetricStatistics, s3:GetBucketVersioning on each metered bucket, and rds-db:connect as usage_fold.

Variable	Default	Meaning
`METERED_BUCKETS`	empty	Comma-separated buckets for the two AWS checks. Set it: without a bucket, `BucketVersioningOk` and `LedgerVsBucketPct` are not published, and their alarms fire
`METRIC_NAMESPACE`	`Toborverse/Metering`
`WINDOW`	`1 hour`	The window of the rate metrics: conflicts, corrections, adjustments created, GPU signals, size intervals
`ROLLUP_GRACE`	`48 hours`	Must equal the value on `metering-jobs`
`CANARY_WITHIN`	`2 hours`	The maximum age of the newest complete canary probe
`SMOKE_TOLERANCE_PCT`	`5`	Above this percentage between the ledger and `BucketSizeBytes`, the function logs an error. The alarm has its own threshold

Every run publishes every metric, zeros included. HealthPublisherRun is sent last and only after success. A versioning check that fails publishes 0, not 1.

4.8 The Terraform module¶

infra/terraform/modules/usage_metering/ describes the ingest path, the archive, metering-jobs, metering-mpu-sweep, metering-canary, their schedules and four ingest alarms, with every name prefixed by name_prefix. It has no inventory loader, health function, or reconciliation and health alarms. Its Firehose stream lacks the AppendDelimiterToRecord processor; add it, or archive files have no line breaks. Its ingest function does not set LOG_RAW_KEYS, so it is true.

5. Alarms¶

5.1 How to read the tables¶

Every alarm notifies <ALERTS_SNS_TOPIC_ARN> (confirm email subscriptions, or nothing arrives). "Fires when" uses a short form: Max > 900, 2 of 2 means the maximum in a period is above 900 in 2 of the last 2 periods. Unless a row says otherwise, the namespace is Toborverse/Metering, the statistic Maximum, and the period 5 minutes. A fresh alarm shows INSUFFICIENT_DATA until its first data point.

Missing data. Most alarms treat missing data as OK. This is safe because every publisher sends all its metrics on every run, zeros included, so missing data means a publisher stopped. Five alarms treat missing data as a failure and catch that case ("missing = alarm" below): metering-health-stalled, metering-reconcile-stalled, metering-canary, metering-bucket-versioning and metering-ledger-vs-bucket. The two bucket alarms need it because their metrics carry a Bucket dimension: if a bucket's figures stop, only missing data shows it.

5.2 Ingest¶

Alarm	Watches	Fires when	Means	Do first
`metering-ingest-lag`	`AWS/SQS` `ApproximateAgeOfOldestMessage`, queue `metering-ingest`	Max > 900 s, 2 of 2	Events wait more than 15 minutes. Ingest is not keeping up	Check `metering-ingest-errors`, the function log, and that the database is reachable
`metering-ingest-process-fail`	`AWS/SQS` `ApproximateNumberOfMessagesVisible`, queue `metering-ingest-dlq`	Max > 0, 1 of 1	A message failed 5 times: a billable event that is not in the ledger	Read the message and the log. Fix the cause, then move the messages back (section 10.2)
`metering-eventbridge-dlq`	Same metric, queue `metering-eventbridge-dlq`	Max > 0, 1 of 1	EventBridge could not deliver an event to the queue or the archive. After 24 hours of retries EventBridge drops an event, so this queue holds the only copy	Read the message attribute `TARGET_ARN` to see which target failed, then `ERROR_CODE` and `ERROR_MESSAGE`
`metering-ingest-errors`	`AWS/Lambda` `Errors`, function `metering-ingest`	Sum > 0, 2 of 2	The function fails	Read the log. A timeout points to the network or `<RDS_SG>`. `PAM authentication failed` points to section 3.5
`metering-ingest-fallback`	`IngestFallbackBatches`	Sum > 0, 3 of 3	Whole batches fail for 15 minutes and are retried one record at a time	Read the log for the batch error. Usually a constraint or a changed SQL function

The lag alarm watches age, not queue depth: a deep queue during a burst is normal, but a rising age means the queue does not drain. IngestRecordsQuarantined has no alarm, because every record it counts is in metering.usage_quarantine, which metering-usage-quarantine watches.

If no events arrive at all, check in this order:

EventBridge notifications are on for <DATA_BUCKET>. If the rule metric MatchedEvents stays at zero, no event left S3.
The rule pattern names the correct bucket.
The queue receives messages (NumberOfMessagesSent). If not, check the rule target, the queue policy and metering-eventbridge-dlq.
The function runs. Unable to import module 'lambda_function' means a wrong handler setting.
The database accepts the login (section 3.5).

5.3 Reconciliation¶

The period of these alarms is 1 day.

Alarm	Watches	Fires when	Means	Do first
`metering-reconcile-diff-a`	`ReconciliationKeyCount`, `DiffType=A`	Max > 0, 1 of 1	S3 has objects that the ledger lacks: missed creates, under-billing	Read the findings (section 6). On a bucket with objects older than the pipeline epoch, it stays on until the first apply
`metering-reconcile-diff-b`	`ReconciliationKeyCount`, `DiffType=B`	Max > 0, 1 of 1	The ledger has objects that S3 lacks: missed deletes, over-billing	Ask S3 about one key (section 6.2)
`metering-reconcile-diff-c`	`ReconciliationKeyCount`, `DiffType=C`	Max > 0, 1 of 1	Sizes differ	Check `same_version` (section 6.2)
`metering-reconcile-diff-d`	`ReconciliationKeyCount`, `DiffType=D`	Max > 0, 1 of 1	A delete with no create, for a key still in S3	Compare the delete time with T (section 6.2)
`metering-unattributable-objects`	`UnattributableObjects`	Max > 0, 1 of 1	Objects outside `orgs/{uuid}/`. Nobody is billed, and no diff reports them	List them (section 6.2). Move them out of the data bucket
`metering-reconcile-stalled`	`ReconciliationRun`	Sum < 1, 2 of 2, missing = alarm	The loader has not finished a run for two days	Check `metering-inventory-load-errors`, the loader log, and that reports still arrive
`metering-inventory-load-errors`	`AWS/Lambda` `Errors`, function `metering-inventory-load`	Sum > 0, 1 of 1	The loader failed	Look up the error in section 4.4

If a known group of objects keeps Diff A above zero after the first clean apply, raise that threshold to a measured baseline. Write the number, the date and the reason in the alarm description.

5.4 Pipeline health¶

Alarm	Watches	Fires when	Means	Do first
`metering-health-stalled`	`HealthPublisherRun`	Sum < 1, 3 of 3, missing = alarm	The health publisher has not finished a run for 15 minutes. Every other health alarm is blind	Check `metering-health-errors` and the function log
`metering-canary`	`CanaryHealthy`	Max < 1, 2 of 2, missing = alarm	Highest severity. The pipeline does not carry events, or it decodes keys wrongly. Events lost during an outage cannot be recovered	`SELECT * FROM metering.canary_check();` and read `detail`. Then follow "If no events arrive" (section 5.2). Send this alarm to a person who can be woken
`metering-usage-quarantine`	`UsageQuarantineDepth`	Max > 0, 1 of 1	Messages with no owner wait in `usage_quarantine`	Section 10.1
`metering-fold-quarantine-delete-before-create`, `-unknown-org`, `-resurrected-key`, `-multiple-deletes`, `-fold-error`	`FoldQuarantineDepth`, `Reason=` `delete_before_create`, `unknown_org`, `resurrected_key`, `multiple_deletes` or `fold_error`	Max > 0, 1 of 1	The fold parked a key: billable storage with no ledger row. One alarm per reason, because each reason has a different fix	Section 10.3
`metering-fold-lag`	`FoldLagSeconds`	Max > 1800, 2 of 2	Events wait more than 30 minutes to be folded. Parked keys are not counted	Check `metering-jobs-errors` and the `metering-jobs` log
`metering-unmatched-deletes`	`UnmatchedDeletesOver24h`	Max > 0, 1 of 1	A delete older than one day has no create. A create was lost	Read the Diff D findings and the dead-letter queues
`metering-event-conflicts`	`EventConflicts`	Max > 0, 1 of 1	S3 sent the same key, type and sequencer twice with different content. An overwrite is not a conflict	Look at `metering.object_events` rows with a recent `last_conflict_at`
`metering-event-corrections`	`EventCorrections`	Max > 0, 1 of 1	Reconciliation corrected a size. Expected after an apply that found Diff C	The repair hides the cause. Find why the create had the wrong size

5.5 Billing¶

Alarm	Watches	Fires when	Means	Do first
`metering-rollup-behind`	`NonFinalDaysPastGrace`	Max > 0, 2 of 2, period 1 hour	Days are still not final 48 hours after their UTC day	Read the nightly report and `metering-jobs-errors`
`metering-adjustments-unsettled`	`UnsettledAdjustmentsPastPeriod`	Max > 0, 1 of 1, period 1 hour	Adjustments do not reach invoices. Invoicing has stopped	`SELECT * FROM metering.unsettled_adjustments;`. Check the invoicing run in the application
`metering-drift-producer`	`DriftProducerGaps`	Max > 0, 1 of 1, period 1 hour	A closed period's figures moved, and no drift row records it. The drift scan has stopped	`SELECT * FROM metering.drift_producer_gaps();` and the `drift` part of the nightly report
`metering-adjustments-volume`	`UsageAdjustmentsCreated`	Max > 5, 1 of 1	Many adjustments within one hour: events are lost and found late	Check the ingest alarms and Diffs A and B. 5 is a starting value; set it from real data

5.6 Multipart uploads and GPU¶

Alarm	Watches	Fires when	Means	Do first
`metering-mpu-leak`	`MpuPastGrace`	Max > 10, 1 of 1	Uploads left open past the 3-hour grace. A client error, or a very long upload; it is also billed	Find the client. 10 is a starting value; set it after a week of data and record why
`metering-mpu-orphans`	`MpuOrphanParts`	Max > 0, 1 of 1	`upload_parts` rows older than 7 days with no object and no open upload	`SELECT * FROM metering.mpu_orphan_parts();`. A report: fix the client
`metering-mpu-drift`	`MpuAccrualDriftOrgs`	Max > 0, 1 of 1	For an org, accrued multipart storage does not match the meter. A sweep was saved only in part	`SELECT * FROM metering.mpu_audit();`
`metering-mpu-grace`	`MpuGraceViolations`	Max > 0, 1 of 1	An open upload's `billable_from` is not its start plus `metering.mpu_grace()`	`SELECT * FROM metering.mpu_audit();`. Right after `mpu_grace()` is changed this is expected until the next sweep: invoke `metering-mpu-sweep` once
`metering-mpu-ambiguous`	`MpuAmbiguousKeys`	Max > 0, 1 of 1	A completed upload could not be told from an aborted one. It was closed at its last sighting	`SELECT * FROM metering.mpu_ambiguous_keys();`
`metering-gpu-timeouts`	`GpuHeartbeatTimeouts`	Max > 0, 2 of 2	GPU jobs stopped sending heartbeats: jobs crash	Read the launcher log. Zero while GPU metering is off
`metering-gpu-reopens`	`GpuIntervalReopens`	Max > 0, 2 of 2	A heartbeat arrived for a closed job and opened another attempt	Check the launcher's network. The reaper limit may be too short

5.7 Bucket checks¶

Create one pair per metered bucket. The period is 1 hour.

Alarm	Watches	Fires when	Means	Do first
`metering-bucket-versioning`	`BucketVersioningOk`, `Bucket=<DATA_BUCKET>`	Minimum < 1, 1 of 1, missing = alarm	Versioning is on, or the check failed. The ledger counts only current versions, so older versions are stored and not billed	Switch versioning off. Check the `s3:GetBucketVersioning` permission
`metering-ledger-vs-bucket`	`LedgerVsBucketPct`, `Bucket=<DATA_BUCKET>`	Max > 5, 2 of 2, missing = alarm	The ledger total differs from `BucketSizeBytes` (standard storage) by more than 5%	Look for objects outside `orgs/`, or older objects not backfilled yet. Otherwise read Diffs A and B

BucketSizeBytes is one daily total with no prefix. If usage moves between two orgs, this check does not see it; reconciliation does.

5.8 The functions themselves¶

These alarms watch AWS/Lambda Errors for one function each: Sum > 0, 1 of 1.

Alarm	Function	Period	Note
`metering-jobs-errors`	`metering-jobs`	5 minutes	The fold or the nightly chain fails. No other alarm reports this directly
`metering-health-errors`	`metering-health`	5 minutes	`metering-health-stalled` follows within 15 minutes
`metering-mpu-sweep-errors`	`metering-mpu-sweep`	1 hour
`metering-canary-errors`	`metering-canary`	1 hour	`metering-canary` follows within 2 hours

metering-ingest-errors and metering-inventory-load-errors cover the other two functions.

5.9 Create an alarm from the CLI¶

The CLI creates an alarm even before its metric exists. This helper takes one row of the tables. Write > N as GreaterThanThreshold N, < 1 as LessThanThreshold 1, "missing = alarm" as breaching, and every other alarm as notBreaching:

alarm () {  # name namespace metric dimension statistic period periods operator threshold missing
  aws cloudwatch put-metric-alarm --region <AWS_REGION> --alarm-name "$1" --namespace "$2" \
    --metric-name "$3" ${4:+--dimensions "$4"} --statistic "$5" --period "$6" \
    --evaluation-periods "$7" --datapoints-to-alarm "$7" --comparison-operator "$8" \
    --threshold "$9" --treat-missing-data "${10}" \
    --alarm-actions <ALERTS_SNS_TOPIC_ARN> --ok-actions <ALERTS_SNS_TOPIC_ARN>
}
alarm metering-ingest-lag AWS/SQS ApproximateAgeOfOldestMessage \
  Name=QueueName,Value=metering-ingest Maximum 300 2 GreaterThanThreshold 900 notBreaching
alarm metering-reconcile-stalled Toborverse/Metering ReconciliationRun "" Sum 86400 2 LessThanThreshold 1 breaching

6. Daily checks¶

6.1 Read the reconciliation findings¶

Reconciliation is the nightly comparison between the ledger and the S3 Inventory report. It writes one row per org and diff type to metering.reconciliation_findings. A clean night writes no rows.

SET ROLE usage_fold;
SELECT run_at, manifest_t, diff_type, org_id, key_count, byte_delta, repaired
  FROM metering.reconciliation_findings WHERE run_at > now() - INTERVAL '7 days'
 ORDER BY run_at DESC, diff_type, key_count DESC;

Diff	Meaning	Effect on the bill
A	S3 has the object; the ledger does not have it live. A create was missed	Under-billing
B	The ledger has the object live; S3 does not have it. A delete was missed	Over-billing
C	Both have the object, with different sizes	Either direction
D	A delete with no create, while the object is still in S3	A person must decide

byte_delta is signed: ledger minus inventory. A positive value means the ledger holds bytes that S3 does not have. repaired counts the repairs written; it is 0 while APPLY is false.

6.2 Find the keys behind a finding¶

Findings are per org. The keys stay in metering.inventory_staging only for the newest KEEP_SNAPSHOTS snapshots (2 by default), so look on the same morning. Set the finding's values with \set t '<MANIFEST_T>' and \set org '<ORG_ID>' in psql, as usage_fold.

For Diff A, the keys are the org's staged rows with no lifetime that was live at T. If such a key has a deleted_at at or before T, the ledger thinks S3 deleted an object that S3 still has.

Diff B. Ask S3 about one key from this list. A 404 means a delete was really lost. If the object exists, the loader did not stage it, and the fault is in the loader.

SELECT l.key, l.created_at, l.time_source FROM metering.object_lifetimes l
 WHERE l.org_id = :'org' AND metering.was_live_at(l.created_at, l.deleted_at, :'t') AND l.created_at < :'t'
   AND l.bucket IN (SELECT DISTINCT bucket FROM metering.inventory_staging WHERE manifest_t = :'t')
   AND NOT EXISTS (SELECT 1 FROM metering.inventory_staging s
                    WHERE s.manifest_t = :'t' AND s.bucket = l.bucket AND s.key = l.key)
 ORDER BY l.created_at LIMIT 50;

aws s3api head-object --region <AWS_REGION> --bucket <DATA_BUCKET> --key '<KEY>'

Diff C. same_version = true: the same object with a wrong size in the ledger; APPLY=true repairs it. same_version = false: S3 saw another version, usually an overwrite near T. Those rows are never repaired. If the deltas repeat one value, or are all powers of two, look for a unit error where the create was recorded.

SELECT s.key, s.size_bytes AS inventory_size, cur.size_bytes AS ledger_size, cur.from_at,
       (btrim(s.etag, '"') = btrim(cur.etag, '"')) AS same_version
  FROM metering.inventory_staging s
  JOIN metering.object_lifetimes l ON l.bucket = s.bucket AND l.key = s.key
  JOIN LATERAL (SELECT i.size_bytes, i.from_at, i.etag FROM metering.object_size_intervals i
                 WHERE i.bucket = s.bucket AND i.key = s.key AND i.from_at <= :'t'
                 ORDER BY i.from_at DESC, lpad(i.sequencer, 32, '0') DESC LIMIT 1) cur ON true
 WHERE s.manifest_t = :'t' AND s.org_id = :'org' AND cur.size_bytes IS DISTINCT FROM s.size_bytes
   AND metering.was_live_at(l.created_at, l.deleted_at, :'t')
 ORDER BY abs(s.size_bytes - cur.size_bytes) DESC LIMIT 50;

For Diff D, the keys are the org's staged rows that have a removed event in metering.object_events and no created event. Compare the delete's event_time with T. A delete after T is repairable: the object lived at T. A delete at or before T is a contradiction, because the object is still in the snapshot. Either the delete is wrong or the report is old. A contradiction is never repaired.

Unattributable objects: SELECT key FROM metering.inventory_staging WHERE manifest_t = :'t' AND org_id IS NULL LIMIT 20;

6.3 Patterns and what they probably mean¶

What you see	Probable meaning	What to do
Large Diff A on the first night	Objects written before EventBridge notifications were on	Set the pipeline epoch (section 10.5). The first apply backfills them
Diff A, the same every night	Repairs are not applied, or something writes outside the event path	Compare the keys night to night. Same keys: repairs are off. Different keys: a gap in ingest
Diff A grows every night	Ingest is losing events	Check the dead-letter queues and the ingest alarms at once
Any Diff B	The ledger holds objects that S3 does not have	Ask S3 about one key (section 6.2)
Large Diff B	Most likely a loader fault: a partial load, or keys not decoded	Check the staged count (section 6.4) before you trust any row
Diff C, `same_version = true`	A size was recorded wrongly at create	The repair fixes the number. Still find the cause
Diff C, `same_version = false`	An overwrite during the snapshot	Usually gone the next night. If it stays, overwrite events are being lost
Diff C on every object	A column-mapping or unit error in the loader	Run `test_handler.py`, then check the manifest `fileSchema`
Diff D with `repaired = 0` while `APPLY=true`	Contradictions	Investigate. Repairs never touch these
Every object in both Diff A and Diff B	The keys on the two sides never match	Compare one key from each side: decoding, the exact bucket name, a leading `/`, then `length(key)`

6.4 The nightly report and the snapshot count¶

metering-jobs logs one line per nightly run, job {"job": "nightly", "report": {…}}. Read it with aws logs filter-log-events --log-group-name /aws/lambda/metering-jobs --filter-pattern nightly. Expect fold.drained = true, fold.keys_parked = 0, and mpu_audit.grace_violations and mpu_audit.org_mismatches = 0. drift.rows_written above 0 means a closed period moved. The prune part shows enabled, horizon, and *_matched and *_deleted counts per kind of row. While PRUNE=false, every deleted count is 0, and the matched counts show what a live run deletes.

Check the staged count against S3's own object count:

SET ROLE usage_fold;
SELECT manifest_t, count(*) AS staged, count(*) FILTER (WHERE org_id IS NULL) AS unattributable,
       max(last_modified) AS newest
  FROM metering.inventory_staging GROUP BY manifest_t ORDER BY manifest_t DESC;

aws cloudwatch get-metric-statistics --region <AWS_REGION> --namespace AWS/S3 --metric-name NumberOfObjects \
  --dimensions Name=BucketName,Value=<DATA_BUCKET> Name=StorageType,Value=AllStorageTypes \
  --start-time "$(date -u -d '3 days ago' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 86400 --statistics Average

newest must not be later than manifest_t; if it is, the snapshot time is wrong and every diff is wrong. A difference of a few percent between staged and NumberOfObjects is normal, plus the keys in IGNORE_KEYS. A large difference is a loader problem.

7. Switching on inventory repairs (`APPLY=true`)¶

7.1 Why repairs start off¶

APPLY on metering-inventory-load starts as false: reconciliation records findings and changes nothing. With APPLY=true it also writes repairs. A repair is a synthetic event that the fold treats like a real one:

Diff A: a create dated to the object's LastModifiedDate, with source = 'inventory_backfill'.
Diff B: a delete dated to the snapshot time T, with source = 'inferred'.
Diff C: the size on the current create is corrected, only when both sides have the same ETag.
Diff D: a create, only when the delete is later than T.

Read about a week of nightly findings before you switch repairs on:

A repair changes billing, and nobody checks it first. If the comparison is wrong, the repair is wrong. This happens when a report is read only in part, a key is decoded differently, or an object is overwritten during the snapshot. The worst case is a false Diff B. It ends the lifetime of an object that still exists, so the customer stops being billed for it. Undoing it needs a ledger rebuild.
Timing problems show only over several nights. A finding that repeats is a real missed event. A finding that is gone the next night is timing noise, such as an overwrite during the snapshot. A finding that grows means ingest is losing events. A week also covers weekday and weekend traffic.
One check needs three nights. The staged count must match S3's own count three nights in a row.
Waiting costs nothing until invoicing starts. After that it costs money: a missed delete that is repaired late is dated to the night of the repair, so the customer is over-billed for the wait. Switch repairs on before production billing starts.

7.2 Checklist¶

All six must be true:

Diff B is zero, or you have explained every key in it.
The staged count matched NumberOfObjects on three nights (section 6.4).
Unattributable objects are zero, or the keys are known and accepted.
The pipeline epoch is set (section 10.5), or the bucket was empty when EventBridge notifications were switched on.
No billing period is closed over the time that the repairs change. Such a repair is valid, but it becomes an adjustment. Keep the first apply free of them.
You took the backup in section 7.3, step 1.

7.3 Steps¶

Back up the event and ledger tables from the app instance. --role and --enable-row-security are required: under forced RLS, only usage_fold sees every row.
Set APPLY=true on metering-inventory-load (section 4.1).
Invoke the loader once with {} and watch the result. The 02:15 UTC snapshot is already staged, so it skips the load and runs the diff with repairs. repaired shows the repairs per diff.

Fold the repairs at once:

 SET ROLE usage_fold;
 SELECT * FROM metering.fold_to_completion();     -- expect drained = true, keys_parked = 0
 SELECT reason, count(*) FROM metering.fold_quarantine WHERE resolved_at IS NULL GROUP BY reason;
 SELECT metering.rollup() AS days_rewritten;

The quarantine query must return no rows. delete_before_create or resurrected_key there means a repair made an order that the fold refuses. Find out why before the next run; do not just clear it.

Expect metering-event-corrections to fire if Diff C repaired a size.
If anything is not clean, set APPLY=false before the next 02:15 UTC run. Keep APPLY=true only after a clean apply, a clean fold and an empty fold quarantine.

7.4 Undo a bad apply¶

Diff A, B and D repairs can be undone. They are added rows with reserved sequencers: 32 × 0 for a create and 32 × f for a delete. ingest_time limits them to one run:

SET ROLE usage_fold;
\set since '<APPLY_START_UTC>'
SELECT source, event_type, count(*) FROM metering.object_events            -- look first
 WHERE sequencer IN (repeat('0',32), repeat('f',32))
   AND source IN ('inventory_backfill','inferred') AND ingest_time >= :'since'
 GROUP BY source, event_type;
DELETE FROM metering.object_events
 WHERE sequencer IN (repeat('0',32), repeat('f',32))
   AND source IN ('inventory_backfill','inferred') AND ingest_time >= :'since';

The ledger has already used these events, so then rebuild it (section 10.4).

A Diff C repair cannot be undone from the database. It overwrites size_bytes on a real create, adds 1 to corrections and sets last_corrected_at. The old size exists only in the backup. To restore it, write the old size_bytes from the backup back to that row as usage_fold, and set folded_at = NULL on the same row. The next fold rewrites that key.

8. Switching on pruning (`PRUNE=true`)¶

8.1 What pruning deletes¶

Pruning is the last stage of the nightly chain. PRUNE on metering-jobs starts as false, so each night is a dry run that counts and deletes nothing. With PRUNE=true:

What	Function	Deleted when	Never deleted
Events in `object_events`	`metering.prune_events()`	The lifetime closed before the retention horizon, and the events are folded. Up to 50,000 rows a night	Events of an object that still exists. Events not yet folded. Events with no lifetime (unmatched deletes, parked keys)
Superseded backfill creates	`metering.prune_superseded_backfill()`	An `inventory_backfill` create on a key that also has a real create, both folded. The fold already ignores it. Up to 50,000 rows a night	—
Rows in `usage_quarantine` and `fold_quarantine`	`metering.prune_quarantine()`	Resolved more than 90 days ago	An unresolved row, at any age
Closed uploads in `mpu_lifetimes`	`metering.prune_mpu_lifetimes()`	Closed before the retention horizon. Their totals go to `metering.mpu_audit_baseline` first, so `metering.mpu_audit()` stays exact. Up to 50,000 rows a night	Open uploads

Pruning never deletes the ledger or the financial record: object_lifetimes, object_size_intervals, usage_daily, usage_period_closures, usage_adjustments and billing_periods. No job prunes reconciliation_findings or the GPU tables. For the reasons behind each rule, see how-it-works.md, §9.5.

The retention horizon is the time before which pruning may delete. metering.retention_horizon() calculates it each night: the earliest current_period_start among subscriptions whose current period is not closed yet, minus a 90-day margin. A period is not closed while it has no row in metering.usage_period_closures. With no such subscription, the horizon is -infinity, and nothing is deleted. The margin is the default of the p_margin parameter, which metering.run_nightly() uses. To change the margin, change database/queries/metering_jobs.sql.

Why it starts as a dry run. Pruning is the only step that cannot be undone. After events are deleted, the ledger cannot be rebuilt from them, and reconciliation cannot use them. The raw archive is then their only record.

8.2 Check first¶

Read a week of nightly reports (section 6.4). Check that prune.horizon and the *_matched counts are plausible.
Check the horizon with SELECT metering.retention_horizon(); as usage_fold. It must lie before every billing period that can still be invoiced or corrected.

Check that the raw archive has a dt= folder for the day of the oldest event that pruning deletes:

 SELECT min(e.event_time) FROM metering.object_events e
   JOIN metering.object_lifetimes l ON l.bucket = e.bucket AND l.key = e.key
  WHERE e.folded_at IS NOT NULL AND l.deleted_at < metering.retention_horizon();

Then list the oldest folders with aws s3 ls s3://<ARCHIVE_BUCKET>/raw-events/ | head -3.

Check that metering-mpu-drift and metering-mpu-grace are OK.
Take a cluster snapshot. It is the only way back: aws rds create-db-cluster-snapshot --db-cluster-identifier <DB_CLUSTER_ID> --db-cluster-snapshot-identifier metering-pre-prune.

8.3 Steps¶

Set PRUNE=true on metering-jobs (section 4.1).
After the next 03:30 UTC run, read the report. prune.enabled must be true. Each *_deleted count equals its *_matched count, or 50,000 when more rows match.
Over the next days, check that metering-mpu-drift stays OK and that the resurrected_key fold quarantine stays empty.

To stop pruning, set PRUNE=false. Deleted rows do not come back.

9. Switching on GPU metering¶

GPU_METERING_ENABLED on metering-jobs starts as false. The SQL, the reaper and the alarms exist. The nightly chain runs the reaper every night either way; with no open GPU job it does nothing. The switch adds the 5-minute reaper run, which limits how long a crashed job is billed.

9.1 What must be in place first¶

The launcher. The orchestrator control plane writes GPU usage, never the training container. It connects as usage_heartbeat. It registers each job with metering.gpu_start_job() before the job runs, with the GPU count and instance type, and cancels the job if this fails. It calls metering.gpu_start_interval() with the scheduler's start time when the job runs. It calls metering.gpu_heartbeat() every 60 seconds, from a process that lives longer than the launch request. It closes with metering.gpu_end_interval() and the scheduler's stop time.
The login. GRANT rds_iam TO usage_heartbeat; ALTER ROLE usage_heartbeat LOGIN;. The launcher's IAM role needs rds-db:connect for usage_heartbeat (section 3.5).
The instance map. metering.gpu_instance_types lists every instance type you launch. Check each GPU count against the AWS specification: a wrong count bills a multi-GPU instance as one GPU. Edit it as metering_owner.
The alarms metering-gpu-timeouts and metering-gpu-reopens.
The cost export. An AWS Data Export (CUR 2.0, CSV) into <COST_EXPORT_BUCKET> (section 10.8).

9.2 Steps¶

Set GPU_METERING_ENABLED=true on metering-jobs and on the launcher at the same time.
Create the schedule metering-gpu-reaper: rate(5 minutes), time zone UTC, flexible time window off, target metering-jobs, payload {"job": "gpu_reaper"}.
Invoke metering-jobs once with {"job": "gpu_reaper"}. Expect closed and days_reset, not "skipped": true.
Launch one short job. As usage_fold, check its rows in metering.gpu_jobs and metering.gpu_job_intervals. After the next night, check gpu_seconds in metering.usage_daily for that org.

The reaper closes an interval when no heartbeat arrives for 3 × 60 seconds. These are the defaults of metering.reap_gpu_intervals().

10. Common tasks¶

Run the SQL in this section as usage_fold unless a step says otherwise (section 3.1).

10.1 Replay quarantined messages¶

The ingest function never drops a message. It puts a message it cannot use into metering.usage_quarantine, with the full EventBridge envelope. An org id that is well formed but does not exist is not quarantined here; the fold parks it as unknown_org (section 10.3).

Reason	Cause
`unparseable_key`	The key does not start with `orgs/`, or the bucket or key is missing
`malformed_uuid`	The key is under `orgs/`, but the id is not a valid UUIDv7
`ingest_error`	The body is not a JSON object, the event type is unknown, a field has the wrong type, the sequencer is missing, the time is missing or has no time zone, a create has no size, or the key is over 1,024 bytes. Also a record that failed alone in the one-at-a-time retry

SELECT id, reason, bucket, key, event_time, ingest_time
  FROM metering.usage_quarantine WHERE resolved_at IS NULL ORDER BY ingest_time;
SELECT metering.replay_quarantined('<QUARANTINE_ID>', '<ORG_ID>', 'owner confirmed');
UPDATE metering.usage_quarantine SET resolved_at = now(), resolution = '<REASON_FOR_THE_RECORD>'
 WHERE id = '<QUARANTINE_ID>';                     -- resolve by hand

metering.replay_quarantined(p_id, p_org_id, p_note) sends the event through the normal ingest path and marks the row resolved. Leave out p_org_id when the key parses correctly on a retry. It refuses a row that is already resolved, an unreadable body (raw_body), no org, an unknown event type, no sequencer, or a create with no size. Resolve such a row, or a row that was never billable, by hand with the UPDATE. Never delete an unresolved row: it is the only record of storage that nobody is billed for.

10.2 Empty the dead-letter queues¶

metering-ingest-dlq. Fix the cause first. Then move the messages back with aws sqs start-message-move-task --source-arn arn:aws:sqs:<AWS_REGION>:<AWS_ACCOUNT_ID>:metering-ingest-dlq. Without a destination, SQS moves them to the queue they came from. This is safe: the billing time comes from the event, and a duplicate event matches an existing row and changes nothing.

metering-eventbridge-dlq. Each message is an event that EventBridge could not deliver. Read TARGET_ARN first. If the queue target failed, send each message body to metering-ingest. If only the archive target failed, the archive misses that event.

10.3 Resolve parked keys¶

The fold parks a key in metering.fold_quarantine when its events describe an impossible lifetime. The key's events stay unfolded.

SELECT bucket, key, reason, detail, quarantined_at
  FROM metering.fold_quarantine WHERE resolved_at IS NULL ORDER BY quarantined_at;
SELECT event_type, source, sequencer, size_bytes, event_time, ingest_time, folded_at
  FROM metering.object_events WHERE bucket = '<DATA_BUCKET>' AND key = '<KEY>'
 ORDER BY event_time, seq_norm;

Reason	Usual cause	What to do
`unknown_org`	The org is not in `organizations`, often one deleted while events were in transit	Restore the org, or decide that the storage is not billable. The nightly chain clears the row when the org exists
`delete_before_create`	The delete is before the create	Find the wrong event. If it is a repair, remove it as in section 7.4
`resurrected_key`	A create arrived after the key's lifetime closed. The key was used again, a wrong inferred delete closed it, or its events were pruned	Compare with S3 and the raw archive
`multiple_deletes`	The key has more than one real delete	Compare with the raw archive
`fold_error`	The fold failed on this key alone	`detail` holds the SQLSTATE and the message. Fix that cause

After you fix the cause, mark the row resolved. The next fold picks up the key. If the cause is still there, the fold parks the key again.

UPDATE metering.fold_quarantine SET resolved_at = now(), resolution = '<REASON_FOR_THE_RECORD>'
 WHERE bucket = '<DATA_BUCKET>' AND key = '<KEY>';

10.4 Rebuild the ledger¶

The fold derives the ledger from metering.object_events. A rebuild deletes the ledger and folds every event again. Use it after a fault in the fold is corrected, or after you delete wrong repairs. It reaches back only as far as the events that still exist (section 8).

SET ROLE usage_fold;
BEGIN;
SELECT pg_advisory_xact_lock(hashtext('metering.fold'));   -- keeps the scheduled fold out
DELETE FROM metering.object_lifetimes;   -- also deletes object_size_intervals (ON DELETE CASCADE)
DELETE FROM metering.fold_quarantine;    -- the fold finds every violation again
UPDATE metering.object_events SET folded_at = NULL;
UPDATE metering.usage_daily SET is_final = false;   -- every day is computed again
COMMIT;
SELECT * FROM metering.fold_to_completion();
SELECT metering.rollup(p_lookback := INTERVAL '90 days') AS days_rewritten;

The first UPDATE is required, because the fold reads only events with folded_at IS NULL. It must report the number of rows in metering.object_events; 0 means the session is not usage_fold. The second UPDATE makes the rollup compute every day again, at any age. Without it, a day whose only usage came from deleted events keeps its old total, because no rebuilt interval covers that day. p_lookback only adds days that have ledger data but no usage_daily row yet. If the rebuild changes days in a closed billing period, the nightly drift scan records adjustments.

10.5 Set or check the pipeline epoch¶

The pipeline epoch (metering.billing_config.pipeline_epoch) is the time when EventBridge notifications were switched on for the data bucket, in UTC. It is not the bucket's creation date, the deploy date or the time of the first event. The first run with APPLY=true backfills each older object. After that, Diff A ignores a backfilled key that starts before the epoch; Diffs B, C and D still check it. Until the first apply, Diff A reports every older object; that is expected. NULL turns the exemption off, which is correct for a bucket that started empty. For details, see how-it-works.md, §9.5.

Find it. CloudTrail keeps the change for 90 days as PutBucketNotification: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=PutBucketNotification. Take the first call on <DATA_BUCKET> that switched EventBridge on. If CloudTrail does not have it, the epoch is at or before the first recorded event: SELECT min(event_time) FROM metering.object_events WHERE source = 'event';. When you are not sure, choose the older time. A time too early costs a few repeated findings. A time too late hides real under-billing.

Set it as metering_owner, then record the value and its source in the usage-metering section of DATABASE.md:

SET ROLE metering_owner;
UPDATE metering.billing_config SET pipeline_epoch = TIMESTAMPTZ '<EPOCH_UTC>', updated_at = now()
 WHERE singleton;                                  -- must report UPDATE 1
SELECT pipeline_epoch, canary_org_id, updated_at FROM metering.billing_config;
RESET ROLE;

10.6 Check key encoding for another bucket or event source¶

The ingest function stores each key as KEY_DECODE produces it. A wrong setting gives a wrong but valid org id, and the wrong customer is billed. EventBridge delivers keys exactly as S3 stores them, so KEY_DECODE=none. The canary proves this every hour: its name ends in a space, +, & and %20, and metering.canary_check() counts only an exact match. When you add a bucket or an event source, check again:

Set LOG_RAW_KEYS=true on metering-ingest (section 4.1).
Upload a test file with a space and a + in its name under orgs/<CANARY_ORG_ID>/canary/. The canary org is never invoiced.
Compare the RAW KEY log line with aws s3 ls. Identical: none. Percent escapes such as %20: urldecode. Spaces shown as +: urldecode_plus.
Set KEY_DECODE. Set LOG_RAW_KEYS=false. Delete the test file.

Inventory reports always percent-encode keys. The loader decodes them with unquote, never unquote_plus, because a + in a key is a real +. There is nothing to set.

A second bucket also needs its name in the rule pattern, and in METERED_BUCKETS on metering-mpu-sweep and metering-health. It needs its own two bucket alarms, versioning off, and an AbortIncompleteMultipartUpload rule. The loader reads one bucket, SOURCE_BUCKET.

10.7 Read the raw archive¶

The archive holds every event that the rule matched. Use it as evidence in a billing dispute, or to see what S3 sent for a key. Files are under s3://<ARCHIVE_BUCKET>/raw-events/dt=YYYY-MM-DD/, by delivery date. Each file is gzip, with one EventBridge event per line as JSON. Files older than 90 days are in Glacier Instant Retrieval, which you can still read directly. Files that Firehose could not write are under raw-events-errors/.

aws s3 ls s3://<ARCHIVE_BUCKET>/raw-events/dt=<YYYY-MM-DD>/
aws s3 cp s3://<ARCHIVE_BUCKET>/raw-events/dt=<YYYY-MM-DD>/<FILE> - | gzip -dc | grep -F '<KEY>'

Each output line is one event: time, detail-type, and detail.object with key, size, etag and sequencer. An event can arrive on the day after it happened, so read the next day's folder too. To order the events of one key, pad each sequencer with leading zeros to 32 characters, then sort. The fold orders them the same way.

10.8 Monthly checks¶

Two checks run by hand once per billing period, because their other half comes from AWS monthly.

The AWS bill against the ledger. Compare the S3 storage charges for <DATA_BUCKET> in Cost Explorer with these figures. A gap that they do not explain points to multipart leaks or request costs. There is no set threshold.

SELECT * FROM metering.ledger_live_bytes();   -- live objects and bytes per bucket
SELECT * FROM metering.mpu_audit();           -- billable_open, billable_bytes
SELECT * FROM metering.mpu_orphan_parts();    -- uploads with nothing behind them

GPU against the cost export. Run lambda/metering-gpu-cur-check/check.py from a host that reaches the database, such as the app instance. Its IAM role needs rds-db:connect as usage_fold and read access to <COST_EXPORT_BUCKET>. DB_HOST is required; DB_PORT, DB_NAME and DB_USER default to 5432, postgres and usage_fold. --end is not included in the period. --tolerance defaults to 0.5 GPU-hours per instance type. It reads AWS Data Exports (CUR 2.0) or a legacy cost report, CSV, gzip or plain. Exit code 1 means a result row below needs action.

cd ToborverseDataManagement/lambda/metering-gpu-cur-check
python3 -I test_check.py
DB_HOST=<AURORA_WRITER_ENDPOINT> DB_NAME=postgres DB_USER=usage_fold \
python3 check.py --start <PERIOD_START> --end <PERIOD_END> \
  --cur-bucket <COST_EXPORT_BUCKET> --cur-prefix <EXPORT_PREFIX>/

Result	Meaning
`mapped=False`	The bill has an instance ****type that `metering.gpu_instance_types` does not list. Add it before you trust any row
Negative `delta`	The meter is below the bill: under-billing. Check `gpu_count` on the jobs first
Positive `delta`	The meter is above the bill. Usually an interval did not close. Check `GpuHeartbeatTimeouts` for the period

Both sides are GPU-hours. The check multiplies the billed instance-hours by the GPU count. It never divides the meter, because dividing hides a wrong GPU count. Only AWS is compared; other GPU providers have no cost export.

10.9 Read a period closure¶

metering.usage_period_closures is the frozen figures of one organization for one billed period. Connect as usage_fold. The application does not read this table. It freezes a period by calling public.freeze_usage_period or public.draft_usage_invoice, as described in How it works §7.3.

SELECT id, org_id, subscription_id, period_start, period_end,
       billing_mode, gb_hours_billed, gpu_seconds, closed_at, invoice_id
  FROM metering.usage_period_closures
 WHERE org_id = '<ORG_ID>'
 ORDER BY period_start;

invoice_id empty means the period is frozen and not yet linked to an invoice. Do not insert a row, and do not change a figure. Finalize is the only writer of invoice_id. A difference found after closed_at is an adjustment, not an edit of this row.