Odtnhj

I'm trying to define a new timestamp in Logstash, in order to use the timestamp of the logs in my Kibana reportings instead of the Filebeat timestamp.

I had the following, which was working but didn't let me use the log timestamp as a time axis in Kibana:

grok {
 match => "message" => "(?<timestamp>%TIMESTAMP_ISO8601) %LOGLEVEL:loglevel (?<class>[A-Za-z0-9$_.]+) %NOTSPACE:method(%JAVAFILE:class:%NONNEGINT:line) %GREEDYDATA:message$"

So I modified it like this, but it's not working:

grok {
 match => "message" => "(?<timestamp>%TIMESTAMP_ISO8601:tstamp) %LOGLEVEL:loglevel (?<class>[A-Za-z0-9$_.]+) %NOTSPACE:method(%JAVAFILE:class:%NONNEGINT:line) %GREEDYDATA:message$"


date 
 match => ["tstamp", "TIMESTAMP_ISO8601"]

I tried to add what I think is a naming of the value inside of the Grok pattern (the ":tstamp" part), which I found here, and the define that as a date. But when I use that, I don't see any data in Kibana anymore.

I erased everything in Kibana in order to have a fresh start, and I deleted the "registry" file in Filebeat in order to make it stream the log file again.

What am I missing?

Edit: I was able to restore the data flow by changing my filters like that:

grok 
 match => "message" => "%TIMESTAMP_ISO8601:tstamp %LOGLEVEL:loglevel (?<class>[A-Za-z0-9$_.]+) %NOTSPACE:method(%JAVAFILE:class:%NONNEGINT:line) %GREEDYDATA:message$" 


date 
 match => ["tstamp", "ISO8601"]

But I still only see the @timestamp as Time Filter. tstamp is still considered as a string. I don't get it...

There were actually 2 distinct problems in what I made.

As I indicated my edit of the question, the date filter was wrong. I had to use

match => ["tstamp", "ISO8601"]

with "ISO8601" instead of "TIMESTAMP_ISO8601".

At that point, things were actually working, even if it wasn't obvious, because by default, Logstash overwrites the value of @timestamp with the result of the date filter. To avoid that, I had to use the target option:

target => "logdate"

I now have a logdate field in Kibana, of type date and I can select it as time filter when creating a pattern.

So my final filters in Logstash look like that:

grok 
 match => "message" => "%TIMESTAMP_ISO8601:tstamp %LOGLEVEL:loglevel (?<class>[A-Za-z0-9$_.]+) %NOTSPACE:method(%JAVAFILE:class:%NONNEGINT:line) %GREEDYDATA:message$" 


date 
 match => ["tstamp", "ISO8601"]
 target => "logdate"