Real-time usage alerts.

Getting that decision wrong has a real cost. A missed alert lets a customer consume well beyond what they are entitled to. A premature alert interrupts someone who should still have access. And if the signals aren't reliable, customers and the teams running the product quickly lose confidence in the system.

That makes usage alerts a very different engineering problem from conventional alerts or notifications. The tricky part is making sure the number being compared is the right one, at the right moment, for the right customer, as usage, subscriptions, entitlements, pricing, and balances are all changing underneath it. Usage can arrive late or out of order. A subscription can renew while events from the previous term are still arriving. A credit balance can fall with usage and rise again with a top-up. And for spend alerts, the same amount of usage can mean different things depending on the pricing model.

So the system isn't just comparing a number to a cutoff. It's working with live billing state: the same aggregates, entitlements, and pricing logic that the invoice will use. The alert has to factor these and get it right, and it has to fire the webhook within seconds of the event.

That left us with a choice: build usage alerting as a separate system with its own representation of the billing state, or evaluate alerts against the billing state we already had. We chose the latter.

The temptation is to build this as a separate system. It's elegant, it decouples billing from alerting, but we knew it wasn't the right call. The moment your alerting service keeps its own version of billing state, that state can drift from the one that produces the invoice. A different rounding rule. A different opinion about which billing term an event belongs to. A stale copy of an entitlement. Each difference might look small, but together they can produce an alert that disagrees with the invoice.

So we decided to evaluate alerts close to the existing usage-ingestion and billing data plane instead, using the same semantics the invoice will use.

This costs more. The alert path is coupled to real billing internals instead of sitting behind a tidy, self-contained abstraction. But it means the alert and the invoice are working from the same billing state, eliminating the premise for drift.

And "the same billing state" looks different for each type of alert. Usage, spend, and credit each have their own source of truth, even though they share the same alerting behavior.

Usage doesn't arrive in a clean, in-order stream. Customers push it from their own backend systems, from batch jobs and ETL pipelines syncing off a data warehouse on their own schedule, or from client SDKs that buffer events locally and flush them once connectivity comes back. So an event we receive right now can describe something that happened an hour ago, and we can receive yesterday's event after this morning's. Meanwhile subscriptions, entitlements, meters, and alert definitions are all changing on their own independent timelines.

For any single incoming event, the system has to work out which billing term it belongs to, which configuration version applies right now, and whether historical usage needs folding in because a rule just went active, all without rescanning the customer's entire usage history every time.

The move that makes it tractable is incremental aggregates: we maintain running meter aggregates instead of recalculating a full billing term on every event. The evaluator combines the existing aggregate with the current alert, entitlement, and subscription context, which turns the common path into a bounded update instead of one that grows with history.

There's a real cost to this: renewals, new meters, late events, and historical backfills all have to agree on the exact boundaries of a term. If a boundary is wrong, the aggregate can be wrong too, without any obvious failure.

Getting the number right solves accuracy. It doesn't solve the scale. An alert is only useful if it can be scoped. The same meter might cap a free-tier account the moment it hits its limit, warn a self-serve account as it approaches the same limit, and let an enterprise account run into overage and flag it after the fact. Each of those needs its own threshold and its own action, chosen by plan or segment.

Filtering by plan becomes expensive if we compare every incoming event against every alert and every subscription; that cost multiplies as event volume or alert count grows. So we narrow the candidate set in stages instead:

1. At ingestion, the streaming layer runs a lightweight eligibility check, given the fields on this event, which meters could even be affected, and groups that by site, forwarding the union of eligible meters rather than running one full evaluation per event.
2. The threshold service then only considers subscriptions touched in that processing window.
3. In our columnar analytical database, a single set-based query resolves each subscription's current plan items, term, applicable alerts, filters, and overrides. Only the surviving alert/subscription combinations get joined against the usage or spend calculation.

Cost now scales with the sites, meters, subscriptions, and alerts actually affected, not with total events multiplied by the whole alert catalog. It also keeps plan membership and alert inheritance where they belong, in the subscription and configuration layer, instead of being reconstructed from raw events every time.

Two decisions made the biggest difference to latency: keeping incremental aggregates, and using the eligibility check to reduce the work before the expensive evaluation step. More recently, we added bounded parallelism per site for multi-tenant batches. The bottleneck there turned out to be repeated switching between database and schema contexts, not data insertion itself.

Accuracy and scope solve two of the three problems. The third is reliability, and it's the one we worried about most: losing the connection between usage we'd already accepted and the alert that should have followed from it. Billing ingestion is the source of truth, and we can't reject usage just because a downstream notification component happens to be unavailable, so usage has to get stored regardless. But if usage is stored and its threshold is never evaluated, that's a silently missed alert, which is worse than a late one because nobody knows it happened.

Here's how we handle it:

• Usage is stored durably first, before anything alert-related runs.
• Site- and batch-level processing identity is carried through the alert path, so work stays traceable.
• Evaluation runs over bounded processing windows, and progress advances only after a successful evaluation, so a crash re-does work instead of skipping it.
• Replays are made safe by the state machine: once an alert is already in_alarm, replaying the same effective input does not manufacture a new crossing. The customer is not notified twice for one state.
• For credits, the evaluator rejects balance updates older than the latest processed occurrence, protecting current state from out-of-order ledger messages.

We test the system around the boundaries where things can go wrong: term renewals, entitlement and threshold changes, late and out-of-order events, repeated processing, plan filters, credit top-ups, and the transition from in_alarm back to within_limit. Every evaluation also records the billing and configuration context it used, so when an alert changes state, we can see exactly why.

The infrastructure was never the hard part. A streaming engine, a queue, an analytical database, and a webhook service get you a working threshold in an afternoon. What's hard is keeping that threshold correct as subscriptions evolve, at volume, without ever losing an event. The engineering challenge was making sure we didn't have to trade speed for accuracy, or vice versa.

There are other scaling costs to account for too. Recalculating raw usage for every event, or evaluating every alert filter in application code, works well at a small scale. But as event history and alert count grow, those approaches get increasingly expensive in production.

If you're looking for the application-side implementation, our usage alerts recipe walks through alert configuration, webhook handling, and common response patterns.