# logParser

## Introduction

LogParser is a java program ( 1.7+ only ) which use Grok Engine ( see [Plugin filter in Logstash](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) ) for parsing file and send extracted informations to InfluxDB.

## How to deploy it

To deploy it, you have to follow [this documentation](http://diqmqs.airfrance.fr/?p=2899) :  and use the LogParser module. 
Usage : java -jar logParser.jar -help
- application <application name> Input application name
- debug Active debug output message
- fromStart Read the file from start
- help Print this message
- info Active info output message
- logfile <path to the logs file> Input log path files
- paramfile <path to the param file> Input a param file
- pattern <path to the pattern file> Input pattern path file
- regex <arg> Name of the regex to apply
- version Print the version information and exit

## Pattern file

As logParser use Grok, you have to indicate a pattern file which contains all Grok pattern.
Example : 
```
USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME:UNWANTED}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM:UNWANTED})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
LOGLEVEL ([A|a]lert|ALERT|[T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
LOGLEVELERROR [E|e]rr?(?:or)?|ERR?(?:OR)?
COSMOS .* %{LOGLEVELERROR:type} %{GREEDYDATA:message}
BOB_MSG ^%{WORD:event},\d+,\S+,%{NUMBER:mqtime_value},%{NUMBER:integration_value},%{NUMBER:transformation_value},ID.*
```
The first part indicate the name of the pattern et the second the pattern ( *mostly regex format* ).  
Each pattern could be rename with the syntax indicate in the last line of the example ( *i.e : %{GREEDYDATA:message}* ).  
The name is use to sendinf extracted data to influxdb.  
For extract date and send it to influxdb you have to use specif name : year  - month - day - hour - minute - second - mscond ( **extract in numeric format only** ).  
For field and tag, all names in lowercase are sended as tag and all ends with " _value " as field. ( *it's not mandatory* ).  
**Please use and abuse of this site** : http://grokconstructor.appspot.com/do/match to verify and test your pattern.  

## Parameter file format

The principal way to use logParser is with a parameter file.
Each line reference for one application a list of files to be parsed by a grok patern :
Example : 
```
#Application LogFilePath FileName GrokName
repind /app/qal/REPIND/MAIN/tomcat7/s*/logs/appli/   .log  REPIND
bob_mess        /app/qal/BOBCORE/*/tomcat7/s1/logs/appli        bobmess.*.log   BOB_MSG
```
Each parameter is separate by a tabulation.  
Application = name of the application send as a tag to influxdb.  
LogFilePath = Folder path which contains files to parse ( **regex is possible** ).  
FileName = Name of files to parse ( regex is possible )   ( **new file wich match to the regex will be parse too** ).  
GrokName = Name of the grok patternto apply on the file.  

## Influxdb

All extracted metrics are sended to the "qualif" database on the "one_week" retention each second. ( For now it's not parametable )