Overview
Operating microservices across disparate Kubernetes clusters often results in fragmented operational visibility: metrics in one tool, logs in another, and distributed tracing either siloed or absent.
This initiative established a Unified Observability Platform centered around the Grafana telemetry ecosystem, deployed across Kubernetes environments.
The Telemetry Architecture
The platform was designed to provide comprehensive visibility into cluster state, workload health, and application execution across four core signals, with external instrumentation alongside them:
1. Metrics: Prometheus
- Deployed and operated Prometheus across clusters to collect platform and workload metrics exposed in each environment.
- Built dashboards and alerting rules around available workload, capacity, and resource-utilization signals.
2. Logs: Grafana Loki
- Deployed Loki across clusters for log aggregation, capturing container stdout/stderr streams with metadata labels.
- Enabled efficient log querying within Grafana directly alongside workload metrics.
3. Traces: Grafana Tempo
- Operated Grafana Tempo for distributed trace collection, providing request-level visibility across microservice boundaries.
4. Continuous Profiling: Grafana Pyroscope
- Integrated continuous profiling with Pyroscope for instrumented workloads.
Instrumentation & Unified Visualization
A primary architectural focus was providing engineering teams with centralized visualization in Grafana:
- Correlated Views: Brought available infrastructure metrics, application logs, trace timelines, and profiles into Grafana dashboards.
- Cross-Platform Instrumentation: Integrated New Relic across cloud infrastructure and Kubernetes clusters alongside the self-hosted Grafana stack.
- Cluster Cost Visibility: Leveraged cluster metrics and resource usage tracking to support rightsizing and committed-use discount (CUD) recommendations across cloud environments.
Operating the Stack
The observability components were cluster services with their own deployment and maintenance lifecycle. Prometheus collected platform and workload metrics; Loki captured container logs; Tempo and Pyroscope added request and code-level signals for instrumented services. Grafana brought those signals together for investigations, while New Relic supplied additional infrastructure and Kubernetes instrumentation.
This made routine cluster checks broader than pod status. Operators could inspect workload health, capacity and resource use, controller behavior, and the path of a request across services. New routing and TLS for telemetry endpoints were managed with the rest of the platform configuration as the stack evolved.
Trade-offs & Operational Considerations
| Approach | Trade-off Accepted | Context & Rationale |
|---|---|---|
| Self-Hosted Grafana Stack | Operational responsibility for deploying, scaling, and maintaining telemetry components | Kept operation of the core telemetry stack within the platform team. |
| Multi-Signal Deployment | Managing multiple specialized collectors (Prometheus, Loki, Tempo, Pyroscope) across clusters | Provided specialized backends tailored for each signal type rather than forcing unindexed logs or traces into a single generic datastore. |
Outcomes & Verification
- Shared Investigation Surface: Grafana provided one place to move between available metrics, logs, traces, and profiling data.
- Workload Profiling: Profiles from instrumented workloads added code-level context to runtime investigation.
- Resource Optimization: Provided detailed infrastructure visibility supporting cost optimization and committed-use recommendations across Azure, GCP, and AWS.