Database monitoring
A catalogue of the queries your databases run, with the samples behind them.
What is collected #
The database agent reads the engine's own statistics and reports which queries ran, how often, and how long they took. Each distinct statement shape is grouped into one catalogue entry so a query called ten thousand times appears once.
Agent-reported statements and statements seen in application traces are combined, so a query is in the catalogue whether it arrived through the agent or through a span.
Statements reach the catalogue two ways, and the difference is worth knowing when a query is missing. An agent-reported statement comes from the database's own statistics and is joined to your traces by host name. A span-reported statement comes from a traced database call and arrives with the attributes that call carried.
One engine
Queries are collected from PostgreSQL. The agent reads the engine's statement statistics view, which is where the catalogue, the plans and the recommendations all come from.
The query catalogue #
- Signature — a stable identifier for the statement's shape.
- Statement — the SQL with its values replaced by placeholders.
- Calls — how often the shape ran in the window.
- Total, mean and 95th percentile duration — the cost of running it.
- Rows affected and rows returned — the shape of the work it does.
Values are grouped out
The catalogue groups on the statement's structure, so WHERE id = 1 and WHERE id = 2 are one entry. This is what keeps a high-volume query from filling the catalogue with thousands of near-identical rows.
Sample collection #
Samples are individual query executions, kept so you can read the exact text, the host it ran on, the database and user, and how long it was blocked for.
- Statement text, query id and signature.
- Host, database, user and application.
- Client address and port.
- State, wait event type and wait event.
- Duration, and the sessions blocking it.
Explain plans #
Plans are captured for statements the agent planned but did not execute, which is where a missing index shows up. Planning a statement is safe in a way that running it is not: the query is described to the engine and the plan comes back, with the statement's effect never happening.
Plans are budgeted, because a plan is large and a busy database generates a great many candidate statements. The agent ranks candidates by how much time they cost, and sends up to five plans per report. A single statement shape is planned at most once every ten minutes, so a shape that stays expensive stays visible without filling the report.
Reading a plan
A plan is the sequence the engine intends to follow. The lines to look for are the scans: a sequential scan over a large table is the shape that a missing index produces, and the cost figure beside it is the engine's own estimate of the work.
Instances and clusters #
Instances are the individual database servers reporting in, each with its engine, version and role. Clusters group the instances that belong to one database, so you can see primary and replica together.
Recommendations #
Recommendations are derived from the catalogue and the plans, and cover these areas:
- Missing index — a statement whose plan shows a scan it does not need.
- Unused index — an index no statement in the window used.
- Long-running query — a statement exceeding 30 seconds.
- High-impact blocker — a session blocking many others.
- High row count — a statement touching an unusually large number of rows.
- Query regression — a statement that has become materially slower than it was.
Each kind has a threshold that decides when it is raised:
- Missing index — a table of at least 10,000 live rows, scanned sequentially at least 100 times in the window, averaging 1,000 rows per scan. Raised as high severity when the statement accounts for a fifth of the instance's query time.
- Unused index — no statement used the index across the whole replication group, which is why it is reported once against the primary rather than once per replica.
- High row count — a statement averaging 10,000 or more rows per call.
- Query regression — at least 50 calls in each half of the window, with the second half averaging at least twice the first. Raised as high severity at four times.
A recommendation can be dismissed from its detail page. Dismissal is scoped to the instance it was raised against, and it sticks: the identity of a recommendation is derived from what it is about, so the same problem dismissed yesterday stays dismissed today.
The agent #
The agent reads the engine's statistics view and reports the difference between two readings, which is how it counts calls without restarting anything. It needs read access to its statistics view and to the database it reports on.
Engine PostgreSQL
Statistics pg_stat_statements
Plans explain, captured before execution
Samples active sessions, on a scheduleLimits #
- Sample queries return up to 2,000 rows; the samples behind one plan return up to 200.
- Grouping accepts up to 3 tags and returns up to 50 groups.
- The query window reaches back 400 days.
- The agent reports at most 500 statements, 1,000 sessions and 50 plans per report.
- Sessions are read at most 200 at a time; tables and indexes are read every fourth pass, capped at 500 and 1,000 rows.
- An instance counts as stale 90 seconds after its last report.
Retention #
Each kind of database telemetry is kept for its own period:
- Instances, health, statements and plans — 30 days.
- Samples and table and index metadata — 15 days.
Start with the recommendations list. Each one links to the queries behind it, so you go from 'something is slow' to the statement causing it in two clicks.
Onward #
The collector that produces all of this is set up in Set up database and stream monitoring, and the queues and caches your services depend on are in Data streams monitoring.