Module Name: omelasticsearch
Author:Rainer Gerhards <rgerhards@adiscon.com>
Available since:6.4.0+
Description:
This module provides native support for logging to Elasticsearch.
Action Parameters:
server [ http:// | https:// | ] <hostname | ip> [ : <port> ] An array of Elasticsearch servers in the specified format. If no scheme is specified, it will be chosen according to usehttps. If no port is specified, serverport will be used. Defaults to “localhost”.
Requests to Elasticsearch will be load-balanced between all servers in round-robin fashion.
Examples:
server="localhost:9200"
server=["elasticsearch1", "elasticsearch2"]
healthchecktimeout Specifies the number of milliseconds to wait for a successful health check on a server. Before trying to submit events to Elasticsearch, rsyslog will execute an HTTP HEAD to /_cat/health and expect an HTTP OK within this timeframe. Defaults to 3500.
Note, the health check is verifying connectivity only, not the state of the Elasticsearch cluster.
$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"
Which will produce this sort of documents (pretty-printed here for readability):
{
"message": " this is a test message",
"fromhost": "test-host",
"facility": "user",
"priority": "info",
"timereported": "2013-03-12T18:05:01.344864+02:00",
"timegenerated": "2013-03-12T18:05:01.344864+02:00"
}
errorfile <filename> (optional)
If specified, records failed in bulk mode are written to this file, including their error cause. Rsyslog itself does not process the file any more, but the idea behind that mechanism is that the user can create a script to periodically inspect the error file and react appropriately. As the complete request is included, it is possible to simply resubmit messages from that script.
Please note: when rsyslog has problems connecting to elasticsearch, a general error is assumed and the submit is retried. However, if we receive negative responses during batch processing, we assume an error in the data itself (like a mandatory field is not filled in, a format error or something along those lines). Such errors cannot be solved by simpy resubmitting the record. As such, they are written to the error file so that the user (script) can examine them and act appropriately. Note that e.g. after search index reconfiguration (e.g. dropping the mandatory attribute) a resubmit may be succesful.
Samples:
The following sample does the following:
module(load="omelasticsearch")
*.* action(type="omelasticsearch")
The following sample does the following:
module(load="omelasticsearch")
template(name="testTemplate"
type="list"
option.json="on") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}")
}
action(type="omelasticsearch"
server="myserver.local"
serverport="9200"
template="testTemplate"
searchIndex="test-index"
searchType="test-type"
bulkmode="on"
maxbytes="100m"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
This documentation is part of the rsyslog project. Copyright © 2008-2016 by Rainer Gerhards and Adiscon. Released under the ASL 2.0.