Logging with Log4j and log aggregation with Apache flume

By

Arivoli.K,MDS201903

Naveen Kumar Reddy,MDS201909

Saager Babu NG,MDS201917

Suman Polley,MDS201935

Avinash Kumar, MDS201907

Why Logging is necessary?

Here comes log4j

Overview

• Log4j is a reliable, fast, and flexible logging framework (APIs) written in Java, which is distributed under the Apache Software License.

• Log4j has been ported to the C, C++, C#, Perl, Python, Ruby, and Eiffel languages.

• It views the logging process in terms of levels of priorities

Components

Log4j has three main components:

• Loggers: Responsible for capturing logging information.

• Appenders: Responsible for publishing logging information to various preferred destinations.

• Layouts: Responsible for formatting logging information in different styles.

History

• Started in early 1996 as tracing API for the E.U. SEMPER (Secure Electronic Marketplace for Europe) project.

• After countless enhancements and several incarnations, the initial API has evolved to become log4j, a popular logging package for Java.

• The package is distributed under the Apache Software License, a full-fledged open source license certified by the open source initiative.

Features

• It is thread-safe.

• It is optimized for speed.

• It is based on a named logger hierarchy.

• It supports multiple output appenders per logger.

• It is fail-stop but log4j does not guarantee that each log statement will be delivered to its destination.

• And many more!!!

Pros of logging

• Quick debugging

• Easy maintenance

• Structured storage of an application's runtime information.

Cons of logging

• Slows down an application.

• If too verbose, it can cause scrolling blindness.

To alleviate these concerns, log4j is designed to be reliable, fast, and extensible

Logger object

• Logger Object is the top-level layer is the Logger which provides the Logger object.

• The Logger object is responsible for capturing logging information and they are stored in a namespace hierarchy

Appender object

• This is a lower-level layer which provides Appender objects.

• The Appender object is responsible for publishing logging information to various preferred destinations such as a database, file, console, UNIX Syslog, etc

Layout object

• The Layout layer provides objects which are used to format logging information in different styles.

• It provides support to appender objects before publishing logging information.

• Layout objects play an important role in publishing logging information in a way that is human-readable and reusable.

Framework of log4j

Brief overview of the support objects

1)Level Object :

The Level object defines the granularity and priority of any logging information.

There are seven levels of logging defined within the API: OFF, DEBUG, INFO, ERROR, WARN, FATAL, and ALL.

Logging levels

Support objects

2) Filter Object :

The Filter object is used to analyze logging information and make further decisions on whether that information should be logged or not.

3)ObjectRenderer :

The ObjectRenderer object is specialized in providing a String representation of different objects passed to the logging framework.

4) LogManager:

The LogManager object manages the logging framework.

Syntax

Types of appenders

• RollingFileAppender

• SMTPAppender

• SocketAppender

• SocketHubAppender

• AppenderSkeleton

• AsyncAppender

• ConsoleAppender

Layout

• We have used PatternLayout with our appender.

• All the possible options are:

DateLayout ,HTMLLayout ,PatternLayout ,SimpleLayout, XMLLayout

Logging methods

• Logger class provides a variety of methods to handle logging activities.

• The Logger class does not allow us to instantiate a new Logger instance but it provides two static methods for obtaining a Logger object:

public static Logger getRootLogger(), public static Logger getLogger(String name)

static Logger log = Logger.getLogger(log4jExample.class.getName())

Logging methods

• Once we obtain an instance of a named logger, we can use several methods of the logger to log messages.

• The Logger class has the following methods for printing the logging information as shown in the next 2 slides.

Logging methods

Logging methods

Log 4j appenders

• Flume provides two log4j appenders that can be plugged into your application:

1) One that can write data to exactly one flume agent.

2) Another that can choose one of many configured Flume agents in a round-robin or random order.

Flume appender

Load balancing log4j appender

• Log4j appenders can be configured to load balance between multiple flume agents, using a round-robin or random strategy.

These log4j appenders come bundled up with flume and doesn’t require us to write any code which is another reason why it is extremely popular.

Apache Flume and log aggregating● Introduction● Philosophy● Apache Flume in HDFS ecosystem● Pros and cons

-Suman Polley MDS201935

INTRODUCTION:

● Apache Flume is a distributed system for

efficiently collecting, aggregating and

moving large amounts of log data from

many different sources to a centralized

data store.

● The use of Apache Flume is not only

restricted to log data aggregation. Since

data sources are customizable, Flume

can be used to transport massive

quantities of event data including but

not limited to network traffic data,

social-media-generated data, email

messages and pretty much any data

source possible.

PHILOSOPHY:

● Distributed pipeline architecture.● Pushing data into HDFS using an intermediate system is a common case

use.Flume acts as a buffer between source and destination.Thus by balancing out any inconsistency in data Flume maintains a smooth flow of data.

● Low-cost of installation ,operation and maintenance.● Highly customizable and extendable.

Flumes position in Hadoop ecosystem:

ARCHITECTURE:

PROS:

● RELIABILITY & RECOVERABILITY :

The events are staged in a channel on each agent. The events are then delivered to the next agent or terminal repository (like HDFS) in the flow. The events are removed from a channel only after they are stored in the channel of next agent or in the terminal repository.This ensures reliable data transfer and ensures recoverability.

PROS:● DOWNSTREAMING:

There could be hundreds even thousands of source .HDFS requires that exactly one clint writes at a time to the database.This could be a problem (!!!!!...)as it will create severe stress to the destination server.

PROS: Solution:

By connecting multiple agents to each other Flume creates a data pipeline .

It is possible to scale down the no of servers that write to the HDFS by adding intermediate Flume agents. This structure has its own problems!!

If n-th tier has same volume as (n-1)th tier then n-th tier will easily overflow creating flow back-pressure.

Points to remember:

● Event volume is least in the outermost tier.

● Event volume increases as flow converges.

● Event volume is the greatest in the innermost tier.

PROS:

● HANDLING AGENT FALIURE :

If the Flume agent goes down, then the all the flows hosted on that agent are aborted. Once the agent is restarted, then flow will resume.

PROS:

The flow using file

channel or other stable

channel will resume

processing events where

it left off.

CONS: ● Channels in Flume act as buffers at various hops. These buffers have a fixed capacity, and once that capacity is full it will create back pressure on earlier points in the flow. If this pressure propagates to the source of the flow, Flume will become unavailable and may lose data.

Rule of Thumb:

Event volume must be equal to worst case data ingestion rate (max data ingestion rate )sustained over the worst case downstream outage interval.

A BETTER SOLUTION:WHat if the single node goes down?

Adding another Flume agent balance the load and it gets better at downstream faliure handling.

Summery:

● All of the above points makes Apache Flume a great real time log aggregator.

● Although it was created for log aggregating ,since then it has evolved to handle many type of streaming data.

● Weak ordering nad prone to duplicacy hinders Flumes application beyond logging (like IoT, Instant messaging service).

NAME NODE CLOGGING

What if all the web

servers collecting log

data tried to connect to

hdfs and write at the

same time?

Map ReduceSpark

Impala

NAME NODE

1

FLUME: EVENTAn Event is the fundamental unit of data transported by flume from its point of origination to its final destination.

A Flume event is defined as a unit of data flow having a byte payload and an optional set of string attributes.

● Payload is opaque to Flume● Headers are specified as an unordered collection of string key-value pairs● These headers help in contextual routing

2

WHAT IS AVRO ?Avro is a row-based data format slash a data serialization system released by Hadoop working group in 2009.

The data schema is stored as JSON in the header while the rest of the data is stored in binary format. One

shining point of Avro is its robust support for schema evolution.

Row-based data formats are overall better for storing write-intensive data because appending new records is

easier.

An Avro Object Container File consists of:

● A file header, followed by

● one or more file data blocks.

A file header consists of:

● Four bytes, ASCII 'O', 'b', 'j', followed by the Avro version number which is 1 (0x01) (Binary values

0x4F 0x62 0x6A 0x01).

● File metadata, including the schema definition.

● The 16-byte, randomly-generated sync marker for this file.

3Ref: https://en.wikipedia.org/wiki/Apache_Avro

FLUME: CLIENT

An entity that generates events and sends them to one or more Agents.

● In our use case of log aggregation, this is done by log4j appender as discussed previously.● Only properties need to be changed for this to be used (no code needed).● Configure log4j to use flume log4j class.● Aim is to decouple a flume agent from the machine on which the flume is running.

4

FLUME: AGENT

A container for hosting Sources, channels, Sinks and other components that enable the transportation of

events from one place to another place. It is self-contained JVM process.

Connecting multiple Flume agents to each establishes a flow. This flow moves data.

Each Flume agent has three components: the source, the channel, and the sink. The source is responsible for

getting events into the Flume agent, while the sink is responsible for removing the events from the agent and

forwarding them to the next agent in the topology, or to HDFS. The channel is a buffer that stores data that

the source has received, until a sink has successfully written the data out to the next hop or the eventual

destination. 5

SourceAn active component that receives events from a specialized location or mechanism and places it on one or

more Channels.

Sources are active components that receive data from some other application that is producing the data.

There are sources that produce data themselves, though such sources are mostly used for testing purposes.

Sources can listen to one or more network ports to receive data or can read data from the local file system.

Each source must be connected to at least one channel. A source can write to several channels, replicating

the events to all or some of the channels, based on some criteria.

Flume’s primary RPC source is the Avro Source. The Avro Source is designed to be a highly scalable RPC

server that accepts data into a Flume agent, from another Flume agent’s Avro Sink or from a client

application that uses Flume’s SDK to send data. The Avro Source together with the Avro Sink represents

Flume’s internal communication mechanism (between Flume agents). With the scalability of the Avro Source

combined with the channels that act as a buffer, Flume agents can handle significant load spikes.

6

Source configuration

A source named usingFlumeSource of type avro, running in an agent started with the name usingFlume,

would be configured with a file that looks like:

7

CHANNELSChannels are passive components that buffer data that has been received by the agent, but not yet written

out to another agent or to a storage system.

Channels behave like queues, with sources writing to them and sinks reading from them. Multiple sources

can write to the same channel safely, and multiple sinks can read from the same channel. Each sink, though,

can read from only exactly one channel.

Channels allow sources and sinks to operate at different rates. Having a channel operating as a buffer

between sources and sinks has several advantages. Having a buffer in between the sources and the sinks also

allows them to operate at different rates, since the writes happen at the tail of the buffer and reads happen

off the head. This also allows the Flume agents to handle “peak hour” loads from the sources, even if the

sinks are unable to drain the channels immediately.

Channels allow multiple sources and sinks to operate on them. Channels are transactional in nature. Each

write to a channel and each read from a channel happens within the context of a transaction. Only once a

write transaction is committed will the events from that transaction be readable by any sinks. Also, if a sink

has successfully taken an event, the event is not available for other sinks to take until and unless the sink

rolls back the transaction. 8

TRANSACTIONS IN CHANNEL

ALL OR NOTHING

9

CHANNEL CONFIGURATIONThe following configuration shows a Memory Channel configured to hold up to 100,000 events, with each

transaction being able to hold up to 1,000 events. The total memory occupied by all events in the channel

can be a maximum of approximately 5 GB of space. Of this 5 GB, the channel considers 10% to be reserved

for event headers (as defined by the byteCapacityBufferPercentage parameter), making 4.5 GB available for

event bodies:

10

SINKThe component that removes data from a Flume agent and writes it to another agent or a data store or some

other system is called a sink. To facilitate this process, Flume allows the user to configure the sink, which

could be one of the sinks that comes bundled with Flume.

Sinks are the components in a Flume agent that keep draining the channel, so that the sources can continue

receiving events and writing to the channel. Sinks continuously poll the channel for events and remove them

in batches. These batches of events are either written out to a storage or indexing system, or sent to another

Flume agent.

Sinks are fully transactional. Each sink starts a transaction with the channel before removing events in

batches from it. Once the batch of events is successfully written out to storage or to the next Flume agent,

the sink commits the transaction with the channel. Once the transaction is committed, the channel removes

the events from its own internal buffers.

11

FLOW EXAMPLE 1

Multi agent flow

Where the data goes through multiple agents or hops.

The sink of the previous agent and source of the current hop need to be avro type with the sink pointing to

the hostname (or IP address) and port of the source.

12

FLOW EXAMPLE 2Typical scenario of log aggregation.

Multiple web servers producing log

data.

Logs collected from hundreds of

web servers sent to a dozen of

agents that write to HDFS cluster.

Configuring a number of first tier

agents with an avro sink, all pointing

to an avro source of single agent .

This source on the second tier agent

consolidates the received events

into a single channel which is

consumed by a sink to its final

destination. 13

FLOW EXAMPLE 3

Multiplexing Flow

Flume supports multiplexing the event

flow to one or more destinations. This

is achieved by defining a flow

multiplexer that can replicate or

selectively route an event to one or

more channels.

The above example shows a source

from agent “foo” fanning out the flow

to three different channels. This fan

out can be replicating or multiplexing.

14

FLUME PROPERTIES

15

REFERENCES

1. USING FLUME BY HARI SHREEDHARAN

2. FLUME USER GUIDE https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html3. REAL TIME DATA INGEST INTO HADOOP USING FLUME

https://www.youtube.com/watch?v=SR__hkCINNc

16