Set up database and stream monitoring

The two halves #

Database monitoring and data stream monitoring are fed by different things. A collector reads your database directly. Data streams are discovered from the traces your services already send.

• Database monitoring needs a collector connected to each instance. Everything below this section is about that setup.
• Data stream monitoring needs spans carrying a stream attribute. There is nothing to install.

Preparing the database #

Two things are set up once per database: a role for the collector to connect as, and a function that returns a query plan without running the query.

• The reporting role — a login the collector authenticates with, granted the engine's built-in monitoring privilege, and set to read-only for every session it opens.
• The plan function — a function the collector calls to obtain a plan. It plans a statement, which is how a missing index becomes visible while the statement itself stays unexecuted.

The role is granted nothing on your tables. The plan function runs with rights of its own, which is why the collector reads nothing of your data.

Server settings #

Four settings make the statistics complete enough to be useful. The statement statistics module must be loaded at startup, and query text must be recorded for every statement rather than only top-level ones.

shared_preload_libraries   loads the statement statistics module
pg_stat_statements.track   all
track_io_timing            on
wal_level                  replica

Configuring the collector #

The collector is configured through environment variables. The one that matters most lists the instances to read.

[
  {
    "host": "orders-db.internal",
    "url": "jdbc:postgresql://orders-db.internal:5432/orders",
    "user": "sentinel_dbm",
    "passwordEnv": "ORDERS_DB_PASSWORD",
    "env": "prod"
  }
]

• host — matches the server your traces attribute queries to. This is what joins the collector's counts to the spans that ran them.
• url — the address the collector connects to.
• user — the reporting role.
• passwordEnv — the name of the variable holding the password. The collector reads whichever variable you name here.
• env — the environment this instance is reported as.

The remaining settings control how it runs:

• Interval — how often each instance is read. 15 seconds by default.
• Endpoint — where reports are sent.
• Raw text — send statements with their literal values in place. Off by default, which groups queries on shape with the values replaced.
• Threads and queue — sized from the number of instances unless you set them.

The first read is a baseline #

The collector reads each instance once to establish a starting point and reports nothing. Counting begins from the second read, so allow one interval to pass before you look for data. This is also what makes the report a difference between two readings rather than a running total since the database started.

Data streams need nothing installed #

A stream appears as soon as a span carries a destination name and is marked as producing or consuming. Attaching those two attributes to the spans your services already send is the whole setup.

Check your work #

• Databases lists the instances that reported, each with its engine, version and health.
• Query metrics lists the statement shapes the collector saw.
• Data streams lists the streams discovered from your spans.
• Recommendations appears once there is enough history to compare against.

Onward #

Queries are arriving: Database monitoring is the catalogue and the recommendations derived from it. Streams are arriving: Data streams monitoring is the topology and how to read it.