Cost controls, quotas and alerts — operator runbook

Companion to the security remediation. That document records what the controls are; this one records the agreed policy and what to do when one of them fires. Covers issues #534 and #535.

Agreed policy

The application allowances in usage-quota.ts bound how much work TailSpend performs. They are not a dollar cap: rejected traffic, logs, storage and retained data still cost money. The budget above is the money-side tripwire, and the two are independent — you can breach the budget without any quota being hit, and vice versa.

Where alerts go

TailSpend-{stage}-Monitoring creates an SNS topic tailspend-{stage}-alerts and points every alarm at it.

Set the recipient by exporting TAILSPEND_ALERT_EMAIL before a deploy. It is deliberately optional: with no address the stack still deploys and still raises alarms, and prints the topic ARN as the AlertsTopicArn output so you can subscribe by hand. An unsubscribed topic means nobody is being told, so this is worth doing before relying on any of it.

aws sns subscribe --topic-arn <AlertsTopicArn> \
  --protocol email --notification-endpoint ops@example.com

The budget and the anomaly monitor are account-wide, so only the prod stack creates them. Deploying dev or beta alone will not give you spend alerting.

Responding to an alert

tailspend-{stage}-quota-denials — sustained 429s

Either abuse, or a real user stuck behind a limit that is set too low.

1. Find out which. In CloudWatch Logs Insights over /aws/http-api/tailspend-{stage}:

   fields @timestamp, routeKey, jwtSub, ip
   | filter status = "429"
   | stats count(*) as denials by jwtSub, ip, routeKey
   | sort denials desc
   

2. Many distinct subjects on one route → the route's limit is too tight. One subject or IP dominating → abuse, and the controls are doing their job. Leave it.

3. To raise a legitimate limit, edit the reviewed values and deploy — either USAGE_LIMITS in usage-quota.ts, the per-route throttles in service-stack.ts, or the weights in with-handler.ts.

Do not clear a user's quota row by hand, and do not disable the limiter to unblock someone. The quota rows are the abuse control; a manual edit is indistinguishable from the abuse it exists to stop and leaves no reviewable record. A daily allowance resets at 00:00 UTC on its own. If a user genuinely cannot wait, grant them credits through the admin balance adjustment (AdminService.adjustBalance) — that path is audited on the credit ledger — and raise the limit properly in code if it keeps happening.

tailspend-{stage}-failed-scans / -abandoned-scans

Both mean users have been charged for work they did not get.

• Failed: check the worker logs for the underlying Textract/Bedrock error. scans.job_failed carries the serialised cause.
• Abandoned: the job's lease expired twice without completing, so reconciliation gave it a terminal state (issue #538). Each one is a user who paid a credit for nothing.

Reconciliation is event-driven: every scan arms a one-off EventBridge Scheduler check (group tailspend-{stage}-scan-recovery) for its own recovery deadline, and the reconciler re-arms it while the job is still live. An hourly sweep is only a backstop. If scans.recovery_schedule_failed appears in the create-scan-job logs, checks are not being armed and stuck scans wait up to an hour for the sweep — check the function's scheduler:CreateSchedule / iam:PassRole permissions first.

There is no automatic refund, and that is deliberate: automatic refunds on failed scans are a loop that funds free OCR, and abandonment can be provoked on demand. Refund case by case with adjustBalance, which records the reason on the ledger.

tailspend-{stage}-throttled-* — DynamoDB throttling

A table or index hit the read/write ceiling the security release set. The bill is protected but the app is degraded. If the load is legitimate, raise maxReadRequestUnits / maxWriteRequestUnits for that table in data-stack.ts and deploy. Check first that it is not a single caller hammering one partition.

tailspend-{stage}-scan-worker-throttles

Scans are queueing behind the worker's reserved concurrency of 10, which is itself a cost control. Raise it only with the budget in mind: concurrency times Textract page rates is the worst-case spend per minute.

Budget or anomaly alert

1. Cost Explorer, grouped by service, for the current month.
2. Compare against the quota-denial and scan volumes for the same window. Spend rising with denials flat means real usage; spend rising with denials also rising means something is being rejected expensively.
3. The controls bound application work, not AWS charges. If spend is being driven by rejected traffic, the lever is the API Gateway route throttles, not the application allowances.

Rotating the signing secrets

Two secrets, and they must be treated differently.

/tailspend/{stage}/signup-fingerprint-secret

Keys signup-eligibility fingerprints — a 24-month fraud record. It is created automatically on first deploy of TailSpend-{stage}-Auth by GeneratedSecretParameter, which will not overwrite or delete an existing value.

Replacing this value without a migration re-grants signup credits to every identity that has already spent them. To rotate:

1. Add the outgoing derivation to legacySignupGrantKeys in identity-fingerprint.ts and in the provisioning-side signup-grant.ts. Both copies must agree.
2. Bump KEY_VERSION in both.
3. Deploy, then put the new value in SSM.

Eligibility checks look under the current version and every retained legacy one, and a legacy-only hit is promoted to the current version on next use, so markers keep matching across the rotation. identity-fingerprint.test.ts is the regression test for exactly this.

/tailspend/{stage}/cursor-signing-secret

Keys pagination cursors and invite tokens. Rotating it invalidates in-flight cursors and every saved invite link — recoverable, and owners can regenerate links.

Since #535 it no longer keys eligibility fingerprints, so rotating it is safe from an abuse standpoint. One caveat during the transition: pre-#535 markers (SIGNUP#<hex>, no version prefix) are still derived from this secret. They are promoted to the dedicated key the next time that identity is seen, but an identity that never signs in again keeps a marker that a rotation would orphan. If you need a clean rotation, accept that, or run the eligibility check across the affected identities first.

What is deliberately not automated

• No self-service quota reset. See above.
• No automatic scan refunds. See above.
• No kill switch on paid work. The daily global allowances (scanStartsGlobal, scanPagesGlobal, uploadsGlobal) already cap paid work per stage per day, and a switch that halts scanning for everyone is a bigger outage than the spend it prevents at this budget. Revisit if the ceiling rises.
• No budget action that stops resources. A budget action powerful enough to stop spend is powerful enough to take the service down on a false positive.