logParser

Introduction

LogParser is a java program ( 1.7+ only ) which use Grok Engine ( see Plugin filter in Logstash ) for parsing file and send extracted informations to InfluxDB.

How to deploy it

To deploy it, you have to follow this documentation : and use the LogParser module. Usage : java -jar logParser.jar -help

application Input application name
debug Active debug output message
fromStart Read the file from start
help Print this message
info Active info output message
logfile Input log path files
paramfile Input a param file
pattern Input pattern path file
regex Name of the regex to apply
version Print the version information and exit

Pattern file

As logParser use Grok, you have to indicate a pattern file which contains all Grok pattern. Example :

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME:UNWANTED}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM:UNWANTED})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
LOGLEVEL ([A|a]lert|ALERT|[T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
LOGLEVELERROR [E|e]rr?(?:or)?|ERR?(?:OR)?
COSMOS .* %{LOGLEVELERROR:type} %{GREEDYDATA:message}

The first part indicate the name of the pattern et the second the pattern ( mostly regex format ).
Each pattern could be rename with the syntax indicate in the last line of the example ( i.e : %{GREEDYDATA:message} ).
The name is use to sendinf extracted data to influxdb.
For extract date and send it to influxdb you have to use specif name : year - month - day - hour - minute - second - mscond ( extract in numeric format only ).
For field and tag, all names in lowercase are sended as tag and all ends with " _value " as field. ( it's not mandatory ).
Please use and abuse of this site : http://grokconstructor.appspot.com/do/match to verify and test your pattern.

Parameter file format

The principal way to use logParser is with a parameter file. Each line reference for one application a list of files to be parsed by a grok patern : Example :

#Application LogFilePath FileName GrokName
repind /app/qal/REPIND/MAIN/tomcat7/s*/logs/appli/   .log  REPIND

Each parameter is separate by a tabulation.
Application = name of the application send as a tag to influxdb.
LogFilePath = Folder path which contains files to parse ( regex is possible ).
FileName = Name of files to parse ( regex is possible ) ( new file wich match to the regex will be parse too ).
GrokName = Name of the grok patternto apply on the file.

Influxdb

All extracted metrics are sended to the "qualif" database on the "one_week" retention each second. ( For now it's not parametable )

3.2 KiB Raw Blame History