Add Readme.md file

This commit is contained in:
Maxime Chassagneux
2017-03-15 14:30:03 +01:00
parent 4e06c81d36
commit 5f03566957

62
Readme.md Normal file
View File

@@ -0,0 +1,62 @@
# logParser
## Introduction
LogParser is a java program ( 1.7+ only ) which use Grok Engine ( see [Plugin filter in Logstash](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) ) for parsing file and send extracted informations to InfluxDB.
## How to deploy it
To deploy it, you have to follow [this documentation](http://diqmqs.airfrance.fr/?p=2899) : and use the LogParser module.
Usage : java -jar logParser.jar -help
- application <application name> Input application name
- debug Active debug output message
- fromStart Read the file from start
- help Print this message
- info Active info output message
- logfile <path to the logs file> Input log path files
- paramfile <path to the param file> Input a param file
- pattern <path to the pattern file> Input pattern path file
- regex <arg> Name of the regex to apply
- version Print the version information and exit
## Pattern file
As logParser use Grok, you have to indicate a pattern file which contains all Grok pattern.
Example :
```
USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME:UNWANTED}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM:UNWANTED})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
LOGLEVEL ([A|a]lert|ALERT|[T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
LOGLEVELERROR [E|e]rr?(?:or)?|ERR?(?:OR)?
COSMOS .* %{LOGLEVELERROR:type} %{GREEDYDATA:message}
```
The first part indicate the name of the pattern et the second the pattern ( *mostly regex format* ).
Each pattern could be rename with the syntax indicate in the last line of the example ( *i.e : %{GREEDYDATA:message}* ).
The name is use to sendinf extracted data to influxdb.
For extract date and send it to influxdb you have to use specif name : year - month - day - hour - minute - second - mscond ( **extract in numeric format only** ).
For field and tag, all names in lowercase are sended as tag and all ends with " _value " as field. ( *it's not mandatory* ).
**Please use and abuse of this site** : http://grokconstructor.appspot.com/do/match to verify and test your pattern.
## Parameter file format
The principal way to use logParser is with a parameter file.
Each line reference for one application a list of files to be parsed by a grok patern :
Example :
```
#Application LogFilePath FileName GrokName
repind /app/qal/REPIND/MAIN/tomcat7/s*/logs/appli/ .log REPIND
```
Each parameter is separate by a tabulation.
Application = name of the application send as a tag to influxdb.
LogFilePath = Folder path which contains files to parse ( **regex is possible** ).
FileName = Name of files to parse ( regex is possible ) ( **new file wich match to the regex will be parse too** ).
GrokName = Name of the grok patternto apply on the file.
## Influxdb
All extracted metrics are sended to the "qualif" database on the "one_week" retention each second. ( For now it's not parametable )