Continuous profiling

What it collects #

A profiling agent samples your running processes and reports where the time went. The result is a flame graph you can read for a function, a service or a host.

Turn on the profiler #

Sentinel profiles four runtimes itself. The setup for each, with your service and this deployment's addresses filled in, is on a service's Profiling tab and on the Profiles page while it is empty.

• Java on Kubernetes — the collector chart's JVM profiler attaches to pods labelled profiling.sentinel.dev/language: java. No image change and no JVM flag.
• Python — pip install sentinel-probes, then start your command through sentinel-probes-run with SENTINEL_PROFILING_ENABLED=true.
• .NET — the Sentinel.Probes package's startup hook (DOTNET_STARTUP_HOOKS) with SENTINEL_PROFILING_ENABLED=true. CPU and allocations.
• PHP — composer require sentinel/probes with the excimer and apcu extensions, loaded through auto_prepend_file, and SENTINEL_PROFILING_ENABLED=true in the php-fpm pool.

SENTINEL_PROFILING_ENABLED=true
SENTINEL_SERVICE=checkout
SENTINEL_ENV=production
SENTINEL_PROFILING_URL=$SENTINEL_HOST/otlp/v1/profiles/pprof
SENTINEL_API_KEY=<ingest key>

What it costs #

Each profiler measures its own cost and keeps it under a budget, 1% by default (SENTINEL_PROFILING_MAX_OVERHEAD). Over it, the profiler samples less often, and if it is still over, it stops and logs why, once. It never throws into your application.

• Python samples every thread's CPU clock on its own thread, every 10 ms. Measured at under 0.5% of a busy server's CPU.
• PHP profiles 10 requests a second per php-fpm pool, after the response is sent. Every other request pays one shared-memory increment, under half a microsecond.
• .NET uses the runtime's own sampler, which pauses threads briefly at each sample, so it records 1 second a minute and scales the profile. Its budget is 1% of the CPU the process may use; a single-core container profiles less often.
• Java is profiled by async-profiler from the node, at about 1% of the JVM's CPU at the default 10 ms interval.

Other languages #

Profiles are accepted from Go, Java, Node.js, PHP, Python, Ruby, .NET, Rust, C and C++. Any profiler that writes pprof can upload one profile a minute.

Flame graphs #

The flame graph is a stack trace, visualised: width is the share of samples spent in a frame, and nesting is the call stack.

• Wide bars are where time is going.
• A tall stack is a deep call chain.
• A wide frame with few children is doing its own work.

Click any frame to see its own view. Selecting a service, host or version scopes the whole graph to that slice.

Comparing profiles #

Put two profiles side by side to see what changed between them. Comparing the same service before and after a release is the most direct way to find a regression.

• Compare across versions to evaluate a release.
• Compare across hosts to explain a single slow machine.
• Compare across time windows to find what changes under load.

Uploading a profile #

curl -X POST "$SENTINEL_HOST/otlp/v1/profiles/pprof?service=checkout&env=production&language=go" \
  -H "x-sentinel-api-key: $SENTINEL_API_KEY" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @profile.pb.gz

service and language are required; env, version and host are optional and become filters on the Profiles page.

Limits #

• Uploads are limited to 600 per minute per key.
• A single upload is limited to 4 MiB.
• A merge combines up to 500 profiles, keeping the top 20,000 stacks.