Skip to main content

Alert routing

Updated

Alert routing is how OnCall decides which escalation policy fires when an alert comes in. Each incoming alert arrives through an alert source (a webhook endpoint). That alert source has an escalation policy attached to it. When an alert matches an active alert source, OnCall creates an incident and immediately fires step 0 of the attached policy.

Team leads configure alert routing by creating services and alert sources. Technicians benefit from it when alerts land in the right place without manual triage.

How Routing Works

  1. An external monitoring tool (or RMM/Defend internally) sends an alert payload to an OnCall webhook URL.
  2. OnCall looks up the alert source by the integration key in the URL.
  3. If the source is active, OnCall normalizes the payload using the source's adapter (Grafana, Prometheus, Datadog, PagerDuty, or generic).
  4. OnCall computes a fingerprint for the alert. If an open incident already exists with the same fingerprint, the alert is deduplicated — no new incident is created.
  5. If it's a new alert, OnCall creates an incident linked to the service associated with the alert source.
  6. The escalation policy attached to the alert source fires immediately (step 0).

Alert Sources and Services

An alert source is the specific webhook endpoint for one monitoring tool or integration. A service is the broader entity being monitored (for example, "Client A — Server Infrastructure"). An alert source is associated with a service, which sets the criticality, alert grouping, and auto-resolve settings for incidents created through it.

One service can have multiple alert sources — for example, one for Grafana, one for Datadog, and one for the RMM. Each can have a different escalation policy if different tools should wake different teams.

Alert Grouping

When you configure a service, you choose how it groups alerts into incidents:

  • Intelligent — OnCall groups similar alerts automatically (default behavior).
  • Time-window — Alerts arriving within a configurable number of minutes are grouped into one incident.
  • Off — Every alert creates its own incident, no grouping.

This is set on the service, not the alert source.

Deduplication

Deduplication prevents alert storms from creating hundreds of incidents. Each normalized alert has a fingerprint (computed by the adapter from the alert's content). If OnCall finds an open incident with the same fingerprint, the new alert is merged into the existing incident and no additional pages are fired. The incident's source_ref is updated.

Once the incident is resolved, a new alert with the same fingerprint will create a new incident.

RMM and Defend Routing

RMM and Defend alerts arrive through an internal integration rather than the public webhook. They still follow the same routing logic — each creates an incident with a service, escalation policy, and fingerprint. The key difference is that RMM alerts are automatically enriched with asset context from the CMDB when available (hostname, site, recent incidents for the asset).

For RMM and Defend, the escalation policy used is the one configured on the receiving service. You do not need to create a separate alert source for these.

FAQ

What happens if an alert source is deactivated? OnCall returns a 410 response and the alert is dropped. No incident is created. Reactivate the source under Alert Sources to resume processing.

Can I use the same escalation policy for multiple alert sources? Yes. Assign the same policy to as many alert sources as you like. Changes to the policy apply to all sources using it.

What if the monitoring tool sends an unsupported payload format? Use the generic adapter. It expects a simple JSON body with title, description, severity, and fingerprint fields. Your monitoring tool or a custom script formats the payload and sends it to the generic endpoint.

Related