Add Readme.md file
This commit is contained in:
62
Readme.md
Normal file
62
Readme.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# logParser
|
||||
|
||||
## Introduction
|
||||
|
||||
LogParser is a java program ( 1.7+ only ) which use Grok Engine ( see [Plugin filter in Logstash](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) ) for parsing file and send extracted informations to InfluxDB.
|
||||
|
||||
## How to deploy it
|
||||
|
||||
To deploy it, you have to follow [this documentation](http://diqmqs.airfrance.fr/?p=2899) : and use the LogParser module.
|
||||
Usage : java -jar logParser.jar -help
|
||||
- application <application name> Input application name
|
||||
- debug Active debug output message
|
||||
- fromStart Read the file from start
|
||||
- help Print this message
|
||||
- info Active info output message
|
||||
- logfile <path to the logs file> Input log path files
|
||||
- paramfile <path to the param file> Input a param file
|
||||
- pattern <path to the pattern file> Input pattern path file
|
||||
- regex <arg> Name of the regex to apply
|
||||
- version Print the version information and exit
|
||||
|
||||
## Pattern file
|
||||
|
||||
As logParser use Grok, you have to indicate a pattern file which contains all Grok pattern.
|
||||
Example :
|
||||
```
|
||||
USERNAME [a-zA-Z0-9._-]+
|
||||
USER %{USERNAME:UNWANTED}
|
||||
INT (?:[+-]?(?:[0-9]+))
|
||||
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
|
||||
NUMBER (?:%{BASE10NUM:UNWANTED})
|
||||
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
|
||||
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
|
||||
LOGLEVEL ([A|a]lert|ALERT|[T|t]race|TRACE|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|[E|e]rr?(?:or)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
|
||||
LOGLEVELERROR [E|e]rr?(?:or)?|ERR?(?:OR)?
|
||||
COSMOS .* %{LOGLEVELERROR:type} %{GREEDYDATA:message}
|
||||
```
|
||||
The first part indicate the name of the pattern et the second the pattern ( *mostly regex format* ).
|
||||
Each pattern could be rename with the syntax indicate in the last line of the example ( *i.e : %{GREEDYDATA:message}* ).
|
||||
The name is use to sendinf extracted data to influxdb.
|
||||
For extract date and send it to influxdb you have to use specif name : year - month - day - hour - minute - second - mscond ( **extract in numeric format only** ).
|
||||
For field and tag, all names in lowercase are sended as tag and all ends with " _value " as field. ( *it's not mandatory* ).
|
||||
**Please use and abuse of this site** : http://grokconstructor.appspot.com/do/match to verify and test your pattern.
|
||||
|
||||
## Parameter file format
|
||||
|
||||
The principal way to use logParser is with a parameter file.
|
||||
Each line reference for one application a list of files to be parsed by a grok patern :
|
||||
Example :
|
||||
```
|
||||
#Application LogFilePath FileName GrokName
|
||||
repind /app/qal/REPIND/MAIN/tomcat7/s*/logs/appli/ .log REPIND
|
||||
```
|
||||
Each parameter is separate by a tabulation.
|
||||
Application = name of the application send as a tag to influxdb.
|
||||
LogFilePath = Folder path which contains files to parse ( **regex is possible** ).
|
||||
FileName = Name of files to parse ( regex is possible ) ( **new file wich match to the regex will be parse too** ).
|
||||
GrokName = Name of the grok patternto apply on the file.
|
||||
|
||||
## Influxdb
|
||||
|
||||
All extracted metrics are sended to the "qualif" database on the "one_week" retention each second. ( For now it's not parametable )
|
||||
Reference in New Issue
Block a user