Data streams monitoring
The topology between your services and the queues, caches and databases they use.
What a data stream is #
A data stream is a dependency that carries messages: a queue, a topic or a cache. Streams appear from your spans, which is how a stream comes to be known at all — from the services that talk to it.
The topology #
The topology is a graph of two kinds of node and the edges between them. Services send and receive; queues carry in between.
- Service nodes — the applications on either side, with their language, environment and team.
- Queue nodes — the destinations themselves, with their technology and environment.
- Edges — one into a queue from each producer, and one out of it to each consumer, with the volume observed on each.
Two edges per queue
A queue has two directions and they are separate edges, because they are separate observations. The incoming edge comes from spans marked as producing and the outgoing edge from spans marked as consuming. A queue whose producers report but whose consumers do not will show you one side only.
Pathways #
A pathway is a producer-to-consumer route through one queue. Pathways show where latency accumulates along the way and which hop is responsible.
- Via — the producer, the queue and the consumer, in order.
- Latency — the total across both hops.
A pathway is assembled from the two observed edges rather than from a single end-to-end trace, so its latency is an approximation and is marked as one. A queue with four producers and six consumers produces every combination, which is what makes a pathway useful for spotting a slow pairing rather than a slow queue.
Overview #
The overview summarises the whole topology rather than the visible page: messages per second, the change in volume across the window, and the worst latency observed.
Change detection #
Volume is compared across the two halves of the window you are viewing. A stream is flagged when it moves by a fifth or more, and when the window holds at least ten messages in total. The floor matters because a single message arriving or being missed is otherwise a large-looking change.
Consumer lag #
Lag appears for streams whose technology reports it. It is the difference between the newest position a consumer has read and the newest position written, turned into seconds using the consumer's own rate. Because it is derived from that rate rather than from broker offsets, read it as an estimate: it holds while consumption is steady and drifts while it is not.
Figures that depend on the client
Bytes per second, partitions and the worst-affected partition are read from attributes the client library reports. A client that leaves them off leaves those figures blank, and a blank means the value was never reported rather than that it was zero.
Filters #
Narrow the topology by:
- kind
- technology
- environment
- language
- team
What to report to get good data #
A stream comes from the spans your services already send. Three things have to be on a span for it to appear: the span kind must mark it as producing or consuming, the service must be named, and the destination must be named.
span.kind producer or consumer
service.name the service on either side
messaging.destination.name the queue or topicThe technology behind a stream comes from messaging.system, which is what fills the technology filter, and message size comes from messaging.message.body.size, which is what makes throughput available.
Group the topology by team when you are reviewing ownership, and by environment when you are comparing a change you just shipped against the one before it.
Limits #
- A topology returns up to 2,000 nodes and 4,000 edges.
- Change detection needs at least ten messages in the window before it reports a movement.
Onward #
The Service map draws the same dependencies from a different angle, and Searching traces is where the spans behind a stream are. The databases those services also depend on are in Database monitoring.