# Observability

Observability is the ability to understand a system’s internal state and behavior from the telemetry and context it exposes.

- Type: principle
- Domain: Engineering & architecture
- Level: Intermediate
- Last reviewed: 2026-09-22
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/observability/>)

## What it means

Observability describes how effectively teams can investigate a system through outputs such as traces, metrics, logs, profiles, and events. In software operations, it is not simply the presence of monitoring tools. Useful observability connects telemetry across requests and components, preserves relevant context, and supports questions that were not fully anticipated before an incident.

## Why it matters

Distributed systems fail in combinations that fixed dashboards may not predict. Rich, correlated evidence helps teams narrow uncertainty, distinguish symptoms from causes, understand user impact, and verify recovery. It also supports capacity work, performance analysis, release evaluation, and more precise service-level indicators.

## When to use it

Design observability alongside services, especially across asynchronous workflows, external dependencies, and multi-tenant systems. Prioritize important user journeys and operational decisions. Add telemetry when it answers a concrete question; collecting every possible event creates cost and noise without guaranteeing insight.

## How it works

1. Identify critical user journeys, dependencies, failure modes, and operational questions.
2. Emit structured telemetry with consistent time, service, environment, and correlation attributes.
3. Propagate context so a request or job can be followed across boundaries.
4. Define user-centered indicators and actionable alerts, then provide paths for deeper investigation.
5. Test telemetry during releases and incidents, and manage volume, retention, privacy, and access.

## A practical example

A booking request crosses an API, queue, inventory service, and payment provider. Shared trace context links the path, metrics show a latency increase, and structured logs identify retries from one provider region. Responders can quantify affected users and verify that rerouting restores the service-level indicator.

## Where it breaks

Telemetry can be incomplete, delayed, sampled, or misleading. High-cardinality data and long retention can be expensive, while logs may expose personal or secret information. Vendor tools do not create good instrumentation automatically, and teams still need domain knowledge and disciplined incident practices.

## Common mistakes

- Equating observability with buying a dashboard or log platform.
- Emitting data without consistent identifiers, ownership, or operational questions.
- Ignoring telemetry cost, sensitive data, sampling effects, and access controls.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/observability/#visualization-observability>)

## Sources

1. [Observability Primer](<https://opentelemetry.io/docs/concepts/observability-primer/>) — OpenTelemetry · Open-source project documentation
2. [Observability](<https://glossary.cncf.io/observability/>) — Cloud Native Computing Foundation Glossary · Institutional glossary

## Related topics

- [Service-Level Indicator](<https://wiki.soroush.uk/topic/service-level-indicator/>)
- [Incident Management](<https://wiki.soroush.uk/topic/incident-management/>)
- [Chaos Engineering](<https://wiki.soroush.uk/topic/chaos-engineering/>)
