Video

Thank you

Hey, quite a lot of questions for s small comment block 😅 but let's try: 1) have a look at this grafana.com/blog/2020/10/14/learn-grafana-share-query-results-between-panels-to-reduce-load-time/, plus Loki does a lot of caching, also take a look at recording rules to speed up high volume Loki queries czcams.com/video/qGyoJPUIOz8/video.htmlsi=BtYmT94Bt5_U21O3 2

2) it depends, generally you want to set as few labels as possible and with a rather lower cardinality during ingest; also it's rather bad practice to set a label is something that already is in the log message. On the other hand, at query time, you want to use as many labels as possible to speed up the query.

3) Grafana Scenes enters the conversation. "Hi there, let me help you" grafana.com/developers/scenes/ Scenes can help you achieve the connection of multiple dashboards while keeping the context when jumping back and forth.

4) Yes, learn transformations, but you can * also do arithmetics on queries directly * depending on the panel type you will already have suggestions in the suggestions tab of the visualization section that show %

5) just click "add query" below the first query of panel and add as many queries to one panel as you want.

Hey, generally speaking it's working the same way. You just need to define the pattern or a regex first to extract the information your want to visualize. Which metrics do you want to extract from which type of logs?

Thank you 🙂

There's nothing notable done in promtaill. What's your challenge?

thanks, I'm happy you found the video useful!

there's several ways you can accomplish this. Either host your own Loki and use Azure blob storage as a storage layer or ingest the logs into Grafana Cloud Loki and configure an export job (grafana.com/blog/2023/06/08/retain-logs-longer-without-breaking-the-bank-introducing-grafana-cloud-logs-export/)

Hi, thanks for the comment. this comes with a trade-off and any kind of query language has a certain learning curve. I'm trying to reduce the one for Loki with this video. In the near future you will see Grafana implementing an explore UI that allows you to query and aggregate logs without any query language at all. But users can still make use of the power of LogQL if they want

thanks for your kind words! considering this ;)

You mean application as in Grafana for creating dashboards? And Grafana Loki as the solution to store and query logs? I'm confused because I put this in the title. If you search online you should be finding tons of resources for both.

@@condla I wanted the application code which was generating the logs and trace. Thanks a ton for the video !!

Ahhhh 😁 I've used a dummy observability application that can be deployed to test things like this: follow the link for more information microbs.io/

Thanks for the feedback... Anything else that's commonly used but needs clarification?

hi, is there a way to get the difference of the timestamp for us to get an api latency and do a trendchart@@condla

Thanks for the feedback.

It didn't bother me

Hi @albogp thanks for the feedback. I haven't heard about this yet, but you can ask the community.grafana.com or join the community slack and ask your question there: grafana.slack.com

Thanks 😊. Yes, you can use the regex/pattern parser to do any kind of ad hoc filtering. Examples are dependent on the expressions and patterns of course. What's your log pattern and what would you like to filter for?

Hi @@condla , Thanks for your quick reply! This is my loggql for a huge file with more or less unstructured log rows, which shows up the amount of all errors occurred in the selected period: sum by(logMsgMasked) (count_over_time({env=~"$env", job="core-files", filename=~"activities.log"} |~ `(WARNING|ERROR)` | regexp `^\[(?P<datestr>.+) (?P<timestr>.+)\]\[PID:(?P<pid>\d+)\] level\.(?P<loglevel>\S+): (?P<logMsg>.*)` | regexp `((a|A)ccount #?(?P<accountId>\d+))` | label_format logMsgMasked="{{regexReplaceAll \"(\\\\d+)\" .logMsg \"\\\\d+\"}}" | line_format "{{.logMsgMasked}}" [$__range])) Suppose there was a log message in the "logMsg" pattern match section: "Memory for account 4711 exhausted by 123456 bytes.", this will be converted to "Memory for account \d+ exhausted by \d+ bytes.". So the converted message should be in an ad hoc filter panel. Activating the adhoc filter on it should display all messages in a corresponding raw message panel below of it, regardless of the number of bytes or the account where the error occurred. I hope I have been able to describe my problem clearly enough.

That's correct, given your log line is in json format, you can parse them on-read (at query time) with Loki's json parser.

Thanks for the nice words to you and your team ❤

Yes of course, you would just use the json parser instead of the logfmt one

Thank you, sounds like this solved an issue for you 😊

Thanks for the feedback

Haha, gern geschehen 😊

Thanks for the feedback. I wrote a blog post that accompanies the video, released yesterday: grafana.com/blog/2023/05/18/6-easy-ways-to-improve-your-log-dashboards-with-grafana-and-grafana-loki/

@@condla Thank you so much I was able to complete a task thanks to this, my logs were in JSON format, logfmt was not parsing and I guess ad-hoc variables would not work in that case

You can also do that with data links by just specifying the same dashboard. You can see an example of data links linking to the same dashboard in this video section "data links" czcams.com/video/EPLvB1eVJJk/video.html

Have a look at data links: grafana.com/docs/grafana/latest/panels-visualizations/configure-data-links/

Check this page, where you get further info and a link to the Faro repo grafana.com/oss/faro/

Thanks Andreas!

Thanks Alain!

Gerne und freut mich. Ist schon etwas älter das Video 😊

Sure, drag another processor into the board that has a name like "Put....." e.g. PutSQL to write it to a system that talks SQL. You need to add the jdbc driver just like you add it when you query it. There's plenty of other Put processors that use the Write APIs of many other databases.

@@condla thanks a lot ♥️

Thanks so much for the feedback. Highly appreciate it. What would you be interested in specifically?

@@condla If you could share us some knowledge on building our own customised processors , It will be wonderful

You're welcome

Hi Hamid, I know what you mean. However, I guess you can get as confused with code as with graphical user interfaces if you're not following clean code practices. If you want to get into ways on how to "write" clean NiFi flows, you can watch this video: czcams.com/video/v1CoQk730qs/video.html&ab_channel=NiFiNotes

You can add multiple columns to check for both new and updated records based on the timestamp in those columns.

@@condla Can you please show how and which version of driver should be installed?

In the answer of the following thread is a link that should help you on this issue. Let me know how it goes. community.cloudera.com/t5/Support-Questions/QueryDatabaseTable-giving-error-while-pulling-data-from/td-p/220949

@@condla Thanks for info. i tried by passing ssl details in query params for 'database connection url' property (jdbc:postgresql://host:port/databaseName?ssl=true&sslrootcert=C:\Certificate\DEV.cert&sslmode=verify-full ). But getting following exception .. 1. Caused by: org.postgresql.util.PSQLException: SSL error: Received fatal alert: handshake_failure .