Dashboards and monitors

The difference #

A dashboard and a monitor answer different questions, and using the right one is most of the work.

• A dashboard answers a question you thought of. You open it when you have decided to look.
• A monitor answers a question you did not think of. It watches on a schedule and tells somebody when the answer changes.

That makes them complements. A dashboard is where you work out why a monitor fired, and a monitor is what makes a dashboard worth having rather than something you have to remember to open.

Building your first dashboard #

Start from the question the dashboard will answer, and add one widget per part of it. A dashboard that answers ‘is checkout healthy right now’ wants a query value for the error rate, a timeseries for the same rate over time, and a top list for whichever service is contributing most.

• Choose the widgets, one question each.
• Drag them onto the grid to arrange them.
• Add a template variable so the same dashboard serves every environment.
• Share it with the team.

Building your first monitor #

A monitor is worth creating when a condition matters and nobody is watching for it at the time it happens. Give it a destination before you save it: a monitor that alerts into an empty channel is the easiest kind to miss.

• Pick the signal to watch.
• Choose the aggregation that matches the question.
• Set a warning threshold below the critical one, so there is lead time.
• Point it at a destination.

Putting the two together #

A monitor summary widget draws the current state of a set of monitors on a dashboard, which is how a team sees what is firing and the surrounding numbers in the same view. An alert that lands in a channel can link straight to the dashboard widget that shows the metric behind it.

What to keep in mind #

• Monitors evaluate on a fixed schedule rather than continuously, so a condition shorter than the interval can pass between evaluations.
• A monitor's status is the worst status across every group it evaluates, so one failing group alerts the whole monitor.
• Muting is permanent until you unmute. A downtime is the tool for a known, scheduled window.
• Every alert a monitor sends is retained, so a monitor that fires constantly costs attention long after it has told you anything.

Onward #

Dashboards is the view itself, Monitors is the alert, and Notification destinations is where an alert goes once it fires.