gorda kickoff meeting inria – sardes project

GORDA Kickoff meeting

INRIA – Sardes project

Emmanuel Cecchet

Sara Bouchenak

http://c-jdbc.objectweb.org/ - [email protected] 2 - 07/10/2004

Outline

INRIA, ObjectWeb & SardesGORDA


Budget: 120 M€ (tax not incl.)

900 permanent staff400 researchers500 engineers, technical and administrative

450 researchers from other organizations700 Ph.D students200 external collaborators750 trainees, post-doctoral students, visiting researchers from abroad

(universities or industry))6 Research Units

A scientific force of 3,000

INRIA key figures

Jan. 2003A public scientific and technological research institutein computer science and control

under the dual authority of the Ministry of Research and the Ministry of Industry

INRIA Rhône-Alpes


• Itanium-2 processors• 104 nodes (Dual 64 bits 900 MHz processors, 3 GB memory, 72 GB

local disk) connected through a Myrinet network• 208 processors, 312 GB memory, 7.5 TB disk• Connected to the GRID• Linux OS (RedHat Advanced Server)• First Linpack experiments at INRIA

(Aug. 03) have reached 560 GFlop/s

• Applications : Grid computing,classical scientific computing, high performance Internet servers, …

iCluster 2


ObjecWeb key figures

Open source middleware development Based on open standard

J2EE, CORBA, OSGi…

International consortiumFounded by INRIA, Bull and FT R&D in 2001

Academic partnersEuropean universities and research centers

Industrial partnersRedHat, Suse, MySQL, …NEC, Bull, France Telecom, Dassault, Cap Gemini,

…


Common Software & Architecture for Component Based Development

OS

Gi

Hardware & OS

Applications

J2E

E

CC

M

...

Dedicated Middleware Platform

Networked Resources

JVM

Tra

nsa

ctio

ns

Per

sist

ence

Dep

loym

ent

...

Co

mp

osi

tio

n

...

Application Servers

Database

Connectivity

Inte

roper

abili

ty

To

ols

Fra

mew

ork

s

Co

mp

on

ents

...

JOn

AS

Op

enC

CM

OS

CA

R

Pro

Act

ive

JOR

M/M

ED

OR

JOT

M

OS

CA

R

Fra

cta

lThink

JORAM

DotNetJ

C-JDBC

RmiJdbc

Octopus

Enhydra

Zeus

Kil

im

CAROL

Jon

ath

an

Sp

eed

o

RUBiS

Bonita

JAWE

XMLC

JMOB


Sardes project

Distributed Systems groupMain research themes

Reflective component technologyAutonomous systems management

Applications areashigh-availability J2EE serversdynamic monitoring, configuration and resource

management in large scale distributed systems(embedded system networks, ubiquitous computing)

Result dissemination by ObjectWeb


Outline

INRIA, ObjectWeb & SardesGORDA


Sardes experiences

Component-based open source middlewareObjectWeb (http://www.objectweb.org)

J2EE application serversJOnAS clustering (http://jonas.objectweb.org)

Database replication middlewareC-JDBC (http://c-jdbc.objectweb.org)

BenchmarkingRUBiS (http://rubis.objectweb.org)TPC-W (http://jmob.objectweb.org)CLIF (http://clif.objectweb.org)

MonitoringLeWYS (http://lewys.objectweb.org)


Common scalability practice

Internet

Web frontend

App. server

Well-known database

vendor here

Database

Well-known hardware +database vendors here

Cons Cost Scalability limit


Replication with shared disks

Internet

Web frontend

App. server Database Disks

Another well-knowndatabase vendor

Consstill expensive hardwareavailability


Master/Slave replication

Internet

Web frontend

App. server

Consconsistencyfailover time on master failurescalability

Master


Internet

Database tier should be scalable highly available without modifying the client application database vendor independent on commodity hardware

Atomic broadcast-based replication

Atomicbroadcast


Internet

JDBC compliant (no client application modification)database vendor independent

JDBC driver required heterogeneity support

no 2PC, no group communication between databasesgroup communication for controller replication only

C-JDBC

JDBC

JDBCJDBC


RAIDb - Definition

Redundant Array of Inexpensive Databasesbetter performance and fault tolerance than a

single database, at a low cost, by combining multiple database instances into an array of databases

RAIDb levels offers various tradeoff of performance and fault tolerance


RAIDb

Redundant Array of Inexpensive Databasesbetter performance and fault tolerance than a single

database, at a low cost, by combining multiple database instances into an array of databases

RAIDb controllergives the view of a single database to the clientbalance the load on the database backends

RAIDb levelsRAIDb-0: full partitioningRAIDb-1: full mirroringRAIDb-2: partial replicationcomposition possible


C-JDBC – Key ideas

Middleware implementing RAIDbTwo components

generic JDBC 2.0 driver (C-JDBC driver)C-JDBC Controller

C-JDBC Controller providesperformance scalabilityhigh availabilityfailovercaching, logging, monitoring, …

Supports heterogeneous databases


C-JDBC – Overview

Database Database

C-JDBC Controller Scalability - Fault tolerance - Failover -

Monitoring - Caching - Logging - ...

Database JDBC driver

DatabaseDatabase Database

Database

JVM

Java client program


JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

Servlet container Tomcat, Jetty, ...


MySQL, PostgreSQL, Oracle, DB2, InstantDB, ...


JVM

Java client program

C-JDBC driver

JVM

EJB Container

JOnAS,

WebLogic, JBoss, WebSphere, ...

Servlet container

Tomcat,

Jetty, ...

C-JDBC driver

C-JDBC driver

JVM

MySQL, PostgreSQL, Oracle, DB2, InstantDB, ...

JVM

C-JDBC

No scalability No fault tolerance

No failover

EJB Container

JOnAS,

WebLogic, JBoss, WebSphere, ...

Servlet container

Tomcat,

Jetty, ...


Heterogeneity support

unload a singleOracle DB withseveral MySQL

RAIDb-2 forpartial replication

MySQL MySQL

C-JDBC Controller RAIDb-2

MySQL JDBC driver

MySQLOracle

Java client program

C-JDBC driver

JVM

C-JDBC driver

C-JDBC driver

JVM

JVM

Oracle JDBC driver



Servlet container Tomcat, Jetty, ...


Inside the C-JDBC Controller

XML configuration file

C-JDBC Controller

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

XML engine

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Configuration &

administrationAdministration console

RMIRMI

Recovery Log

Authentication Manager

MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Recovery Log


Oracle

Oracle JDBC driver

Oracle JDBC driver

Oracle

SocketsSockets

JMX


C-JDBC features

unified authentication managementtunable concurrency controlautomatic schema detectiontunable replication: full partitioning, partial replication,

full replicationcaching: metadata, parsing, result with various

invalidation granularitiesvarious load balancing strategieson-the-fly query rewriting for macros and heterogeneity

supportrecovery log for dynamic backend adding and failure

recoverydatabase backup/restore using OctopusJMX based monitoring and administrationgraphical administration console


C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver

Client application (Servlet, EJB, ...)


Functional overview

connect myDBconnect login, passwordexecute SELECT * FROM t


C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver



Functional overview

execute INSERT INTO t …


C-JDBC Controller

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

C-JDBC driver



Failures

No 2 phase-commitparallel

transactionsfailed nodes are

automatically disabled

execute INSERT INTO t …


Controller replication

Prevent the controller from being a single point of failure

Group communication for controller synchronization

C-JDBC driver supports multiple controllers with automatic failover

MySQL PostgreSQL

C-JDBC Controller

MySQL JDBC driver

PostgreSQL JDBC driver

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC Controller

JGroups

MySQLPostgreSQL

MySQL JDBC driver


jdbc:c-jdbc://node1:25322,node2:12345/myDB


Controller replication

C-JDBC Controller

Distributed Request Manager

C-JDBC Controller

Distributed Request Manager

MySQL

C-JDBC driver

MySQL

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

Virtual database 2

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer


Oracle JDBC driver

OracleMySQL

MySQL JDBC driver

C-JDBC driver


C-JDBC driver



Request Manager

Query result cache

Scheduler

Load balancer


jdbc:cjdbc://node1,node2/myDB

Uniform total order reliable multicast


Mixing horizontal & vertical scalability

DB 6DB 5

DB native JDBC driver

DB 7

C-JDBC driver

DB 1 DB 2


DB 3


DB 4

C-JDBC controller Full replication




C-JDBC driverJVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

C-JDBC driver


Lessons learned

SQL parsing cannot be genericmany discrepancies in JDBC implementationsminimize the use of group communications

IP multicast does not scale

notification infrastructure neededusers want

no single point of failurecontrol (monitoring, plug-able recovery policies, …)no database vendor locking no database modification

need for an exhaustive test suitebenchmarking accurately is very difficult

load injection requires resourcesmonitoring and exploiting results is tricky


Sardes role in GORDA

provide inputGORDA APIsgroup communication requirementsmonitoring and management requirements

middleware implementation based on C-JDBCdissemination effort

ObjectWebpossible participation to JCP for JDBC extensions

hardware resources for experimentseCommerce benchmarks


Other interests

LeWYS (http://lewys.objectweb.org)monitoring infrastructuregeneric hardware/kernel probes for Linux/Windowssoftware probes: JMX, SNMP, …monitoring repository

autonomic behaviorbuilding supervision loopsself-healing clustersself-sizing (expand or shrink)SLAs


Q without A

do we consider distributed query execution?XA support?cluster size targeted?do we target grids or cluster of clusters?

reconciliationconsistency/cachingnetwork architecture considered?

are relaxed or loose consistency options?what will really cover the GRI?

do we impose a specific way of doing replication?access to read-set/write-set difficult to implement with legacy

databaseswhich workloads are considered?which WP deals with backup/recovery?licensing issues?

Q&A_________

Thanks to all users and contributors ...

Bonus slides

INTERNALS


Virtual Database

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

gives the view of a single database

establishes the mapping between the database name used by the application and the backend specific settings

backends can be added and removed dynamically

configured using an XML configuration file



MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Matches real login/password used by the application with backend specific login/ password

Administrator login to manage the virtual database


Scheduler

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Manages concurrency controlSpecific implementations for

Single DB, RAIDb 0, 1 and 2Query-levelOptimistic and pessimistic

transaction leveluses the database schema that is

automatically fetched from backends


Request cache

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

caches results from SQL requestsimproved SQL statement analysis to

limit cache invalidationstable based invalidationscolumn based invalidationssingle-row SELECT optimization

request parsing possible in theC-JDBC driver

offload the controllerparsing caching in the driver


Load balancer 1/2

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

RAIDb-0query directed to the backend having the

needed tables

RAIDb-1read executed by current threadwrite executed in parallel by a dedicated

thread per backendresult returned if one, majority or all commitif one node fails but others succeed, failing

node is disabled

RAIDb-2same as RAIDb-1 except that writes are

sent only to nodes owning the written table


Load balancer 2/2

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Static load balancing policiesRound-Robin (RR)Weighted Round-Robin (WRR)

Least Pending Requests First (LPRF)

request sent to the node that has the shortest pending request queue

efficient if backends are homogeneous in terms of performance


Connection Manager

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Connection pooling for a backend

Simple : no poolingRandomWait : blocking poolFailFast : non-blocking poolVariablePool : dynamic pool

Connection pools defined on a per login basis

resource management per logindedicated connections for admin


Recovery Log

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Checkpoints are associated with database dumps

Record all updates and transaction markers since a checkpoint

Used to resynchronize a database from a checkpoint

JDBCRecoveryLogstore information in a databasecan be re-injected in a C-JDBC

cluster for fault tolerance

SCALABILITY


C-JDBC scalability

Horizontal scalabilityprevents the controller to be a Single Point Of Failure

(SPOF)distributes the load among several controllersuses group communications for synchronization

C-JDBC Drivermultiple controllers automatic failover

jdbc:c-jdbc://node1:25322,node2:12345/myDB

connection cachingURL parsing/controller lookup caching


C-JDBC scalability

Vertical scalabilityallows nested RAIDb levelsallows tree architecture for scalable write broadcastnecessary with large number of backendsC-JDBC driver re-injected in C-JDBC controller


C-JDBC vertical scalability

RAIDb-1-1with C-JDBC

no limit tocompositiondeepness

MySQL JDBC driver

C-JDBC driver

MySQL JDBC driver MySQL JDBC driver

C-JDBC controller RAIDb-1




C-JDBC driverJVM

Client program C-JDBC

driver

JVM

Client program

C-JDBC driver

JVM

Client program

MySQL MySQL MySQL MySQL MySQL MySQLMySQLMySQL


C-JDBC vertical scalability

table x & ytable x & y


table x & y

C-JDBC driver

table w table w


table z


table z





C-JDBC driverJVM

Client program C-JDBC

driver

JVM

Client program

C-JDBC driver

JVM

Client programRAIDb-0-1

with C-JDBC

CHECKPOINTING


Fault tolerant recovery log

C-JDBC Controller

MySQL MySQL

Application virtual database

Database Backend

Database Backend

Request Manager

Request Cache

Scheduler

Load balancer

Recovery Log


MySQL

Database Backend

Recovery virtual database

Request Manager

SchedulerLoad balancer


C-JDBC driver


Checkpointing service

Database Backend

Database Backend

C-JDBC driver

MySQL

UPDATE statement


Checkpointing

Octopus is an ETL tool Use Octopus to store a dump of

the initial database state Currently done by the user using

the database specific dump tool

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



disabled

dump for initial

checkpoint

Recovery LogOctopus

disabled


Checkpointing

Backend is enabled All database updates are logged

(SQL statement, user, transaction, …)

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



enabled

dump for initial

checkpoint

Recovery LogOctopus

JDBC Recovery Log

enabled


Checkpointing

Add new backends while system online

Restore dump corresponding to initial checkpoint with Octopus

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus

PostgreSQL PostgreSQLdisabled disabledenabled

enabled

JDBC Recovery Log


Checkpointing

Replay updates from the log

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus

PostgreSQL PostgreSQLenabled

enabled

JDBC Recovery Log

disableddisabled


Checkpointing

Enable backends when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled


Making new checkpoints

Disable one backend to have a coherent snapshot

Mark the new checkpoint entry in the log

Use Octopus to store the dump

...

dump for last

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for last

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enableddisabled

dump for initial

checkpoint



Replay missing updates from log

dump for

checkpoint

Octopus...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



Recovery Log


enabled

JDBC Recovery Log

disabled enabled



Re-enable backend when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint


Recovery

A node fails!Automatically

disabled but should be fixed or changed by administrator

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enableddisabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint


Recovery

Restore latest dump with Octopus

...

dump for last

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for last

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enableddisabled

dump for initial

checkpoint


Recovery

Replay missing updates from log

dump for

checkpoint

Octopus...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



Recovery Log


enabled

JDBC Recovery Log

disabled enabled


Recovery

Re-enable backend when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

HORIZONTAL SCALABILITY


Horizontal scalability

JGroups for controller synchronizationGroups messages for writes only

DB1 DB2 DB3 DB4

Controller1

DB1 DB2 DB3 DB4

Controller1 Controller2

Client 1 Client... Client n Client 1 Client... Client n



Centralized write approach issues

Issues with transactions assigned to connections

DB1 DB2 DB3 DB4


commit

DB1 DB2 DB3 DB4


commit

commit commit

DB1 DB2 DB3 DB4

Controller2

? ? ? ?

DB1 DB2


DB1 DB2


T1(Read)

T1(R)

DB1 DB2


T1(Write)

T1(R)T1(W) T2(W)



General case for a write query3 multicast + 2n unicast

DistributedVirtualDatabase@controller2

DistributedRequestManager





execWriteRequest(AbstractWriteRequest)

Scheduler

Write query handler

Load Balancer

Scheduler

Load Balancer

Distributed scheduling handler

Wait for schedulers replies

Send request to all controllers

Send execute request order to all controllers

Wait for load balancer replies

Distributed load balancer handler

Scheduler

Load Balancer



Notify controllers of completion success or not



Notify scheduler of completion

Return result

return result





Solution: No backend sharing1 multicast + n unicast [+ 1 multicast]

DB1 DB2 DB3 DB4 DB5 DB6

Controller1 Controller2 Controller3






execWriteRequest(AbstractWriteRequest)DistributedRequestManager

Scheduler

Write query handler

Load Balancer

Scheduler

Load Balancer


Send request to all controllers

Wait for load balancer replies


Scheduler

Load Balancer



Notify controller(s) that have failed if other have succeeded



Wait if all backends failed

Return result

return result





Issues with JGroupsresources needed by a channelinstability of throughput with UDPperformance scalability

TCP better than UDP butunable to disable reliability on top of TCPunable to disable garbage collectionordering implementation is sub-optimal

Need for a new group communication layer optimized for cluster



JGroups performance on UDP/FastEthernet

USE CASES


Budget High Availability

High availability infrastructure “on a budget”

Typical eCommercesetup

http://www.budget-ha.com


OpenUSS: University Support System

eLearningHigh availabilityPortability

Linux, HP-UX, Windows

InterBase, Firebird, PostgreSQL, HypersonicSQL

http://openuss.sourceforge.net

Firebird

C-JDBC Controller Full replication

Firebird JDBC driver

Firebird

C-JDBC driver

JVM

C-JDBC driver

JVM


Firebird JDBC driver

C-JDBC driver

JVM

JOnAS J2EE server

C-JDBC driver

JVM

Enhydra server

OpenUSS OpenUSS OpenUSS OpenUSS

Enhydra Director

Apache

JOnAS J2EE server

Enhydra server

JOnAS J2EE server

Enhydra server

JOnAS J2EE server

Enhydra server

Enhydra Director

Apache

Enhydra Director

Apache


Flood alert system

Disaster recovery Independent nodes synchronized with C-JDBC VPN for security issues

http://floodalert.org

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

MySQL


MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

> 1000 miles


J2EE benchmarking

Large scaleJ2EE clusters

http://jmob.objectweb.org

Internet

emulated users

PERFORMANCE


TPC-W

Browsing mix performance


TPC-W

Shopping mix performance


TPC-W

Ordering mix performance


Result cache

Cache contains a list of SQL->ResultSetPolicy defined by queryPattern->Policy3 policies

EagerCaching: variable granularities for invalidationsRelaxedCaching: invalidations based on timeoutNoCaching: never cached

RUBiS bidding mixwith 450 clients

No cache Coherentcache

Relaxedcache

Throughput (rq/min) 3892 4184 4215

Avg response time 801 ms 284 ms 134 ms

Database CPU load 100% 85% 20%

C-JDBC CPU load - 15% 7%


Outline

MotivationsRAIDbC-JDBCPerformanceLessons learnedConclusion


Open problems

Partition of clustersUsers want control on

failure policyReconciliation must

also be user controlled

PostgreSQL

C-JDBC Controller

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC Controller

JavaGroups

PostgreSQL


LeWYS overview

Monitoring Pump

CPUprobe

diskprobe

pumpthread

DREAM

Monitoring Pump

CPUprobe

memoryprobe

pumpthread

DREAM

DREAM

DREAM

DREAM

Monitoring Pump

pumpthread

CPUprobe

networkprobe

ObserverObserver

Observer ObserverMonitoringrepository


LeWYS components

Library of probeshardware resources: cpu, memory, disk, networkgeneric sensors: SNMP, JMX, JVMPI, …

Monitoring pumpdynamic deployment of sensorsmanages monitoring leases

Event channelspropagate monitored events to interested observersallows for filtering, aggregation, content-based processing, …

Optional monitoring repository


LeWYS design choices

Component-based frameworkprobes, monitoring pump, event channelsprovides (re)configurability capabilities

Minimize intrusiveness on monitored nodes

No global clocktimestamp generated locally by pump

Information processing in DREAM channels


Centralized monitoring using a monitoring repository (1)

Monitoring Pump

CPUprobe

memoryprobe

pumpthreadDREAM

Monitoring Pump

CPUprobe

memoryprobe

pumpthreadDREAM

Monitoring Repository

Monitoring DB

storagethread

querythreads

Monitoring Console

eventsubscribeservice


Centralized monitoring using a monitoring repository (2)

Monitoring repositorystores monitoring informationservice to retrieve monitoring information

ProsDB allows for storing large amount of datapowerful queries

correlate data from various probes at different locationsresynchronize clocks

browsing history to diagnose failuresuse history for system provisioning

Consrequires a DB (heavy weight solution)


Outline

J2EE Cluster

Group communications

MonitoringmotivationsLeWYSimplementation

Status & Perspectives


Monitoring pump implemention

Component

Pro

beF

acto

ry

MonitoringPump

Thread

Probe

MonitoringPumpManager

ChannelOut

Pro

beM

anag

erBindingController

Probe

Probe

Repository

ProbeManager

OutputManager

Component

PullPushMultiplexerM

onitoringMumpManager

OutputManager

TimeStamp

Probe

Cache

Cac

hed

Pro

be

RMI


Hardware Probes

Pure Java probesusing /proccost: 0.01ms/call (Linux)

Linux Solaris

/proc

Hardware resources

Windows

.DLLC C

Linux

C

JNI JNI JNI

cpu, mem, disk, net,

kernel, …probes

cpu, … probes… …


Software Probes

Application level monitoringJMXad-hoc

JVM

Linux Solaris

JVM

Hardware resources

WindowsLinux

JMX JVMPI

SNMP, ad-hoc,

… probes

JVM probesJMX based

probes

gorda kickoff meeting inria – sardes project

Documents

objectweb sardesgordahttp

orgbenchmarkingrubis

orgmonitoringlewys http

orgclif http

ministry of research

orgtpcw http

gb memory

classical scientific