Title

AWS re:Invent 2022 - Open-source observability is the way forward, and here’s why (PRT039)

Summary

Tomer Levy, CEO of Logs.io, discusses the challenges of observability in the industry and how his company aims to address them.
Observability is considered broken due to the overwhelming amount of telemetry data generated, especially with Kubernetes, where most data is not useful and incurs unnecessary costs.
Companies often lack the knowledge to extract value from observability tools, leading to an increase in mean time to resolution (MTTR) despite the proliferation of technologies.
Logs.io focuses on simplifying observability by providing as-a-service solutions for open-source tools like ELK, Prometheus, and Jaeger, and by unifying data to reduce MTTR.
The company introduced Kubernetes 360 to correlate data from various sources and simplify monitoring.
Logs.io emphasizes data optimization by identifying and reducing the indexing of unnecessary data, which helps manage costs.
The company promotes the use of open-source tools to avoid vendor lock-in and encourages focusing engineering efforts on core business functions rather than managing observability infrastructure.

The observability market is saturated with tools that often lead to complexity and increased costs without proportional value, highlighting the need for optimization and simplification.
The trend towards open-source observability tools reflects a broader industry preference for flexibility, cost-effectiveness, and community-driven innovation.
Kubernetes has become a significant driver of observability challenges due to the volume of data it generates, indicating a need for better data management strategies in containerized environments.
The concept of "Data Hub" introduced by Logs.io, which filters out less useful data, suggests a growing recognition of the importance of data triage in observability to manage costs and improve efficiency.
The mention of vendor lock-in as a concern for businesses underscores the importance of interoperability and the ability to switch between different tools or providers without significant friction or cost.
The increase in MTTR despite advancements in observability tools suggests that tool proliferation without strategic integration can be counterproductive, reinforcing the value proposition of unified observability platforms.