rsyslog futures (2012, RH Mini Summit on Logging)

rsyslog v8 improvements
and plugins in ANY language

Rainer Gerhards

First things first:
Version check :-)

Current stable is 8.2.x

Don't use outdated cruft like 5.8.x (!)

Packages available at
http://www.rsyslog.com

What's in this talk?

v8 improvementsInfrastructure things

The v8 engine

Writing plugins in languages other than C

I will probably run out of time - but that's fine, the slides at the end are optional.

The rsyslog doc project

The doc

The rsyslog doc project

The doc just sucks...

Spawned a new project to create better one: https://github.com/rsyslog/rsyslog-doc

Initiated by James Boylan (a sysadmin)

Please helpComplain ;-)

open issues

Write some doc...

We are especially interested to learn what is hard for beginners!

Community involvement

Rsyslog was traditionally open to contributions and has a lively community

Yet we still try to improve things and have made a couple of moves to encourage further contributions

Rsyslog & subprojects on github

Project Admins are David Lang (not affiliated), Tomas Heinrich (Red Hat) and me

Subprojects with different commit levels

What is new in the v8 engine?

Output part totally revamped

Simplified script executionEven faster execution speed

Less complex code

Even higher scalability

Global variable support

Required changes to output module interface

Support for plugins in any language

The v7 rule engine

rsyslogcoreQueue workerQueue workerQueue workerAction instancequeue

Single-thread
compartment

Filter processing

Message formatting

Actual output action, like sending msg

Kept simple & single threaded

Works well with fast actions

Has problems with slow ones, e.g.
via HTTP (like Elasticsearch)

What was the problem with the v7 engine and slow outputs?

Traditionally, an action instance is non-reentrant

Multiple threads are used forFiltering

Template generation

Execution of independent actions

But the SAME action could not benefit from the thread pool

Not a problem for traditional fast outputs

Why is this a problem e.g. for Elasticsearch?

Rsyslog-processing is NOT time-dominant

Transfer via HTTP takes its timeNeed to wait on result

Both ES and rsyslog mostly idle in this process

ES request relatively slow, scaling via multiple connections

ES can process many requests concurrently

With the new engine, we can spawn multiple concurrent connections from the same action, and thus keep both ES and rsyslog busy and de-tangle it from the slow parts!

The v8 rule engine

rsyslogcoreQueue workerQueue workerQueue workerAction wrkr inst.queue

Now multiple instances per action!

Queue worker pool automatically
scales outbound connection count
by spawning more worker instances

Works well with Elasticsearch etc.

Inherently serial outputs (e.g. local files!)
must serialize themselves

Action wrkr inst.Action wrkr inst.

What if the destination cannot handle multiple workers?

A typical example is file output

Framework will still call into multiple workers

Worker must use pData mutex to protect itself this is not done automatically!

Basically like in v7, but plugin needs to take care

Actioninstance1Actioninstance2wrkr 1wrkr 2wrkr 1wrkr 2wrkr 3

real codereal code

Action-instance mutexes

Writing plugins

Traditionally, pluginsAre written in C

Macros hide interface plumbing

Fairly easy to write for the C-literate

Still perceived as complicated

V8 goalEnable everyone to write plugins (sysadmins!)

Support any language (Python, Perl, ...)

Ability to execute security-sensitive plugin out of rsyslog security context

Types of Plugins

Output (actions)Deliver message to some destination system, e.g. file, ElasticSearch, MongoDB, Solr, ...

Any language supported in v8.2.0+

Message Modification Plugins (Modules)Permit on-the-fly modification of message content (e.g. anonymization, credit card removal)

Any language supported in v8.3.0+

InputAccept input messages

Currently being worked on (target 8.3.[3-5]+)

Writing external output plugins for rsyslog

Rainer Gerhards

Interface Overview

rsyslog
core
engineInternal pluginexternal plugin
connectorperl pluginpython plugin

process border

External Interface Design Goals

Keep it stupid simpleMust support almost any language

Dumb easy to use even for novice programmer

Do not require explicit threading

Speed is NOT the most important goalDon't make it unnecessarily slow

Many real-world log destinations are slow in any case (like when you connect via http...)

Focus on enable to build solution

If necessary, conversion to internal module can be done later

Interface Details: communication

use pipes

stdinone message per line

format can be customized via rsyslog templates

multi-line messags via JSON

stdout/stderrMust NOT be written in initial version

Will later convey back state information via plain text (e.g. ERR, ERRMSG:xxx, ...)

Template specifies input format (with JSON recommended for more complex cases)

Interface Details: Threading

Do NOT care about threading

Write app according to single-thread paradigm

rsyslog will spawn multiple instances of your plugin if there is need to do soHappens based on config in busy cases

Works well in most cases (e.g. http connects)

Can be disabled if necessary

If your program can run in multiple ter-minal sessions concurrently, it can also be run as multiple rsyslog action instances.

Startup & Termination

rsyslog will startup the plugin automatically

Plugin needs to read stdin until EOF

Do NOT terminate before EOF is reached

On EOF, cleanup and terminate

If the plugin dies, rsyslog restarts a new instance

Some signals (like sigint) are blocked and should remain so

Skeletons

The rsyslog project provides sample plugin skeletons

Available in ./plugins/external/skeletons

These containthe necessary plumbing

often a kind of abstraction layer to make writing plugins even easier

often performance-enhancement features

Can simply be copied to create your own plugins, don't care about the (minimal) plumbing!

Call to Action

If you need to send logs to a destination that is not yet supported, you can quickly write an external plugin in any language you know!

Writing rsyslog plugins is easyIf there is already a skeleton for your language, copy it and add your app-specific code

If not ... no problem, the interface is dumb easy

If you can write a script that reads stdin and does something useful with it, you can also write a rsyslog plugin!

Writing external output plugins for RSysLog
IN 2 MINUTES

Rainer Gerhards

Write the plugin itself

Choose any language you like

Implement the pseudocode belowMessages arrive via stdin, one message per line

Read from stdin until EOF

Process each message read as you like

Terminate when EOF is reached

That's it!

While not EOF(stdin) do { Read msg from stdin Process msg}

Make RsysLog call plugin

Regular filtering applies (as with any action)

You can specify message format via a template

Use omprog for the call

module(load=omprog) # needed only once in config!

if $rawmsg contains sometrigger then action(type=omprog binary=/path/to/your/plugin)

Optional: debugging your plugin

If something doesn't work, it's best to debug outside of rsyslog

Do this as you usually debug your programs (e.g. use your favorite debugger!)

For example, do

$ echo testmessage | /path/to/your/plugin

Questions about the plugin interface or plugin integration? Visit http://kb.monitorware.com/external-plugins-f53.html

Questions?

rgerhards@adiscon.com

www.rsyslog.com

https://github.com/rsyslog