Alerts & notifications

An alert policy decides what counts as a failure and who hears about it. When a check breaks your policy's conditions, Canary9 fires a notification to your integrations and records it in the event log.

Create an alert policy

Open Alerts in the sidebar and select + New policy. A policy has a few parts:

• Name: what you'll recognise the policy by (e.g. "Prod API down").
• Check type: which kind of endpoint the policy watches: HTTP, Ping, DNS, or TCP.
• Endpoints: apply it to all endpoints of that type, or pick specific ones.
• Condition: what makes a check "fail", with an optional warning level below the critical one for response-time and HTTP status (see below).
• Failure & recovery counts: how many consecutive failures trigger the alert (default 3) and how many consecutive successes clear it (default 2), so a single blip doesn't page you.

Conditions by check type

The available conditions depend on the check type:

Warning and critical levels

Response-time and HTTP status conditions support two levels, so Canary9 can nudge you before something is truly broken:

• Critical: the hard limit. Crossing it fires a down event (severity critical).
• Warning: an optional, lower bar. Crossing it (but not critical) fires a degraded event (severity warning), a heads-up rather than an outage.

The warning level is optional and only applies where "how bad" is a gradient:

• Response time: a warning threshold below the critical one (e.g. warn at 100 ms, alert at 200 ms).
• HTTP status range: a warning range alongside the critical range (e.g. warn on 400–499, alert on 500–599).
• Exact status code: your expected (ok) code, an optional warning code, and anything else is critical (e.g. expect 200, warn on 301).

Binary conditions have no warning level: DNS resolves or it doesn't, and a TCP port is open or closed. Both are ok or critical.

Certificate-expiry policies are warning-first: Canary9 emits a degraded warning when the observed HTTPS certificate enters your configured window (14 days by default), and treats an already-expired certificate as critical. A plain HTTP endpoint or a check with no certificate observation is ignored by this condition, rather than producing a false alert.

Preview against history

When you set a response-time policy, Canary9 charts the targeted endpoints' recent latency with your warning and critical thresholds drawn on top, and shows what share of recent checks would have crossed each line, so you can choose thresholds that fit reality instead of guessing. The dashed lines move as you type, and points that breach a threshold are coloured by the level they hit. (Rare outliers are left off so a single spike doesn't flatten the chart.)

Default policies

You don't start from a blank slate. Every organization ships with a set of default policies that watch all endpoints of their type from day one:

• DNS does not resolve: any DNS endpoint that stops resolving.
• Web response time: HTTP endpoints, warning at 1 000 ms and critical at 2 000 ms.
• Web error status (4xx/5xx): HTTP endpoints returning a 400–599 status.
• HTTP check failed: HTTP endpoints whose check fails outright, for whatever reason: the connection is refused or reset, the name does not resolve, the TLS handshake fails, the request times out, the status does not match the endpoint's expected code, or a content match fails. The alert names the failure ("connect failed: connection refused"), and it clears once the check passes again.
• TLS certificate expires soon: HTTPS endpoints whose observed certificate expires within 14 days; an already-expired certificate is critical.
• Ping response time: Ping endpoints, warning at 500 ms and critical at 1 000 ms.
• TCP port closed: TCP endpoints whose port stops accepting connections.
• Missing monitoring data: any active endpoint that stops reporting check results at all. A silent gap in monitoring is treated as an incident, not quietly ignored. An endpoint counts as silent after roughly three missed check intervals (at least 5 minutes), so a single delayed check never pages you, and a just-created or just-resumed endpoint gets a fresh grace window. When results resume, or you pause the endpoint on purpose, the alert resolves itself.

The response-time defaults are deliberately generous, because they apply to every endpoint you have, checked from every location you have chosen. A probe on the other side of the country or an ocean away pays that distance on every check: a healthy service answering in 30 ms locally can measure 200 ms or more from a distant region, purely because of the round trip. The defaults are set where a person would actually notice the wait, so they page you about your service rather than about geography.

If you want a tighter bar, that's exactly what a policy of your own is for: copy the default, set the thresholds you care about, and point it at the endpoints you hold to them. Two things make that easy to get right:

• The Preview on the policy form plots recent checks from every location the endpoint runs in against the thresholds you're typing, so you can pick a number you know your endpoint actually meets from all of them, rather than only from the nearest one.
• Which locations have to agree is yours to choose. By default a policy fires when any location breaches, which is what you want for an outage. Switch it to only when all regions are down on a policy you own if you'd rather hear about a problem only when it's visible from everywhere.

Default policies are managed, so they behave a little differently from your own:

• You can't delete them or change their conditions: they're always there as a safety net.
• You can mute one if it's too noisy for your setup.
• You can customise its notification message (see below): handy for adding a runbook link or routing it to a specific channel.
• You can copy one into a brand-new policy of your own, then edit every field: a fast way to start from a sensible baseline (e.g. copy "Web response time" and lower the critical threshold to 400 ms for an API you hold to a tighter bar).

Customizing notification messages

Every policy (default or your own) has an optional notification message. Leave it empty and Canary9 sends a sensible default to all of your enabled integrations. Fill it in to control the wording and where each kind of event goes.

Whatever the wording, every notification carries a link to the affected endpoint in Canary9 (https://app.canary9.com/endpoints?ep=...), rendered natively for each channel: a button in email and Microsoft Teams, a link in Slack and Discord, a click-through link on the PagerDuty incident, a link in the Jira issue, and endpoint_id, endpoint_name, and endpoint_url fields in webhook payloads. You can acknowledge, pause, or investigate without hunting for the endpoint first.

A message is made of state blocks, one per situation. Write only the ones you care about:

• {{ #critical }} … {{ #end_critical }}: when the check goes down.
• {{ #warning }} … {{ #end_warning }}: when it's degraded (slow but not down).
• {{ #critical_recovery }} … {{ #end_critical_recovery }}: when it recovers from down.
• {{ #warning_recovery }} … {{ #end_warning_recovery }}: when it recovers from degraded.
• {{ #no_data }} … {{ #end_no_data }}: when no results are coming in at all.

Inside any block, @-mention an integration's handle to route that situation there. No mention sends the block to every enabled integration. So you can page on-call for hard outages while keeping chatter in Slack:

{{ #critical }} API is down. Runbook: example.com/runbook  @pagerduty-oncall {{ #end_critical }}
{{ #warning }} API degraded  @slack-web {{ #end_warning }}
{{ #critical_recovery }} Recovered  @slack-web {{ #end_critical_recovery }}

In the editor, press Tab to autocomplete a block (type {{ #cr → Tab) or a handle (type @slack- → Tab). Known handles are highlighted as you type, so a typo'd @mention is easy to spot before you save.

Manage policies

• Mute: temporarily stop a policy from firing without deleting it (the bell icon on the row).
• Test: send a one-off test notification through the policy so you can confirm it reaches your integrations.
• Edit / Delete: change or remove a policy you created (default policies can't be deleted; copy or mute them instead).

The event log

The Event log tab is the timeline of everything that's happened: down, up, and degraded events, each with a severity (critical, warning, or info), the policy and endpoint involved, and when it occurred. Filter by policy to focus on one service. Each event names the endpoint it is about; click it to jump straight to that endpoint's row on the Endpoints page.

Event text describes the condition that actually tripped the policy. A response-time policy reports the measured latency against its threshold ("response time 2431ms exceeded the 2000ms threshold"), a status policy reports the HTTP status, and a certificate policy reports the days until expiry, so the message tells you which policy fired and why, and a recovery states which condition cleared.

A check that fails outright is a different kind of evidence from a slow or erroring response, and Canary9 keeps the two apart. Response-time and certificate policies only grade checks that completed: a refused connection returns in a few milliseconds and a timeout takes the full budget, but neither number is a response time, so a failed check never fires a latency alert and never counts as a latency recovery. The failure itself is reported by the HTTP check failed default (or your own policy on that condition), whose message names what went wrong rather than quoting a number. Status conditions still grade any status the server actually returned, so a 503 trips Web error status (4xx/5xx) as well as HTTP check failed; if you would rather hear about it once, mute whichever of the two you care about less.

When a web, browser, TCP, or ping check goes down, Canary9 automatically runs a network diagnostic from the probe and attaches it to that event, so the incident carries a snapshot of the DNS, network path, port, and TLS/response state captured at the moment it failed. Expand a down event's Diagnostics captured at failure to see it, so you can tell why it broke without re-running anything after the fact. See Diagnostic chains for how to read the chain.