7. Observability Coordinated: Prometheus Exemplars (Metrics) - Grafana Tempo (Traces) - Loki (Logs)
Vložit
- čas přidán 22. 08. 2024
- In this video, we’re gonna do 2 things. First we extend our digram with Grafana Tempo. Second, we enable Exemplars on Prometheus, to correlate Metrics to Traces, and after that we set up tracesToLogs to bind Traces and Logs
Github: github.com/nli...
OpenTelemetry with Spring Boot 3 Series: • OpenTelemetry with Spr...
References:
Exemplars Intro: grafana.com/do...
Exemplars Intro in Blog: grafana.com/bl...
Really great job. It is the best tutorial on Spring observability currently available
thank you!
Great! You solved my problem! Thx to share your knowledge!
thank you, glad it could help!
Excellent tutorial!
thank you, glad you liked it!
In this case, you have used Tempo directly with the agent, as an exporter.
What is the difference to use it inside the collector, just like we did in video 4 from your older tutorial?
Is is a good practice to set everything (Prometheus, Loki and Tempo) inside the collector?
Or is it better to connect Prometheus and Tempo directly with the agent, and use the collector only for things that doesn't have an exporter, like Loki?
How? how do you know what I'm about to do in the next video! 😜. From my perspective, it's better if we can have Otel Collector in between, so the service only needs to send telemetry to 1 destination (reduce overhead), and Collector will take care of the rest. Also, there are lots of cool stuff in Collector like Processor and Connector predefined for us to manipulate telemetry data (for example, some observability backends have some limitations about the maximum MB of telemetry data for their APIs, Batch Processor is a must in this case)
@@linhvudev haha 😅. I'm building an observability for my work, and these tutorials came in a precious moment.
In my case, I'm using this stack to monitor a lot of services. So it would be better for me to put everything in the collector, and not the application directly.
By doing this each service will depend on only one thing, and the observability will take care of the data. (Exactly as you mentioned)
I tried to bring prometheus to the otlp collector, it worked, but the exemplars didn't show up. So I can't navigate to Tempo from the Prometheus. Maybe some configuration that I missed (eagerly waiting for the next video haha).
And how can I connect multiple services (spring applications) so i can monitor them in grafana?
@@samuelgadiel how can I connect multiple services -> it's about context propagation, but if you use Otel Java Agent, you'll get it for free github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks. The most importance thing is you have to TAG your service with proper attributes to distinguish them among tons of telemetry data