jonas developer workshop

JOnAS developer workshop – http://www.objectweb.org - [email protected] 1 - 23/02/2004

status

Emmanuel [email protected]


Outline

OverviewAdvanced conceptsQuery cachingHorizontal scalabilityMonitoringPerspectives


Motivations

J2EE performance scalability bounded by database performance

Database tier must be–scalable–fault tolerant (high availability + failover)–without modifying the client application–using open source databases–on commodity hardware


Internet

How do we want to use ?

• end-to-end open source solution for large scale J2EE clustered application servers

• cost-effective solution for both small and large configurations

Apache

MySQL


Redundant Array of Inexepensive Databases

RAIDb controller– gives the view of a single database to the client– balance the load on the database backends

RAIDb levels– RAIDb-0: full partitioning– RAIDb-1: full mirroring– RAIDb-2: partial replication– composition possible


C-JDBC

Middleware implementing RAIDbTwo components

– generic JDBC 2.0 driver (C-JDBC driver)– C-JDBC Controller

C-JDBC Controller provides– performance scalability– high availability– failover– caching, logging, monitoring, …

Supports heterogeneous databases


Using JOnAS with C-JDBC

• Without C-JDBC

Database

JDBCDriver


Pooling

Cache

ControllerDatabaseVirtual

Driver

Using JOnAS with C-JDBC

Update the datasource to use the C-JDBC driver

Backend

JDBCDriver

JOnAS


What C-JDBC offers

No application modificationNo JOnAS modificationNo database modification

DatabaseVirtual

BackendBackend recovery

Pooling

Cache

ControllerDatabaseVirtual

JDBCDriver

Load balancing

Driver


Outline



Inside the Controller

XML configuration file

C-JDBC Controller

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

XML engine

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Configuration &

administrationAdministration console

RMIRMI

Recovery Log

Authentication Manager

MySQL

Database Backend

Connection Manager

MySQL JDBC driver

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Recovery Log


Oracle

Oracle JDBC driver

Oracle JDBC driver

Oracle

SocketsSockets

JMX


Fault tolerant recovery log

C-JDBC Controller

MySQL MySQL

Application virtual database

Database Backend

Database Backend

Request Manager

Request Cache

Scheduler

Load balancer

Recovery Log


MySQL

Database Backend

Recovery virtual database

Request Manager

SchedulerLoad balancer


C-JDBC driver

Client application (Servlet, EJB, ...)

Checkpointing service

Database Backend

Database Backend

C-JDBC driver

MySQL


Cascading controller

DB 6DB 5

DB native JDBC driver

DB 7

C-JDBC driver

DB 1 DB 2


DB 3


DB 4

C-JDBC controller Full replication




C-JDBC driverJVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

C-JDBC driver


Heterogeneity support

application alreadywritten for a specific[commercial] database

user defined rulesfor on-the-fly queryrewriting to executeon heterogeneousbackends

MySQL MySQL

C-JDBC Controller RAIDb-2

MySQL JDBC driver

MySQLOracle

Java client program

C-JDBC driver

JVM

C-JDBC driver

C-JDBC driver

JVM

JVM

Oracle JDBC driver

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

Servlet container Tomcat, Jetty, ...


Outline



Query caching status

Cache contains a list of SQL->ResultSetPolicy defined by queryPattern->Policy3 policies

– EagerCaching: variable granularities for invalidations– RelaxedCaching: invalidations based on timeout– NoCaching: never cached

RUBiS bidding mixwith 450 clients

No cache Coherentcache

Relaxedcache

Throughput (rq/min) 3892 4184 4215

Avg response time 801 ms 284 ms 134 ms

Database CPU load 100% 85% 20%

C-JDBC CPU load - 15% 7%


RUBiS - Tomcat withoutC-JDBC caching

Tomcat~50% cpu

1 Database100% cpu

Throughput: 3900 pages/min


RUBiS - Tomcat withC-JDBC relaxed caching

Tomcat~55% cpu

1 Database~20% cpu

Throughput: 4200 pages/min

C-JDBC <10% cpu


JOnAS and C-JDBC query caching

RUBiS

Throughput(req/min)

Response time (ms)

No cache No cache

Servlet 3890 4140 744 310JOnAS SFCMP 1.1 2947 3418 2929 2045JOnASCMP 2.0 2178 2450 6372 4630


Outline



Horizontal scalability

JGroups for controller synchronizationGroups messages for writes only

DB1 DB2 DB3 DB4

Controller1

DB1 DB2 DB3 DB4

Controller1 Controller2

Client 1 Client... Client n Client 1 Client... Client n



Centralized write approach issues

Issues with transactions assigned to connections

DB1 DB2 DB3 DB4


commit

DB1 DB2 DB3 DB4


commit

commit commit

DB1 DB2 DB3 DB4

Controller2

? ? ? ?

DB1 DB2


DB1 DB2


T1(Read)

T1(R)

DB1 DB2


T1(Write)

T1(R)T1(W) T2(W)



General case for a write query– 3 multicast + 2n unicast

DistributedVirtualDatabase@controller2

DistributedRequestManager





execWriteRequest(AbstractWriteRequest)

Scheduler

Write query handler

Load Balancer

Scheduler

Load Balancer

Distributed scheduling handler

Wait for schedulers replies

Send request to all controllers

Send execute request order to all controllers

Wait for load balancer replies

Distributed load balancer handler

Scheduler

Load Balancer



Notify controllers of completion success or not



Notify scheduler of completion

Return result

return result





Solution: No backend sharing– 1 multicast + n unicast [+ 1 multicast]

DB1 DB2 DB3 DB4 DB5 DB6

Controller1 Controller2 Controller3






execWriteRequest(AbstractWriteRequest)DistributedRequestManager

Scheduler

Write query handler

Load Balancer

Scheduler

Load Balancer


Send request to all controllers

Wait for load balancer replies


Scheduler

Load Balancer



Notify controller(s) that have failed if other have succeeded



Wait if all backends failed

Return result

return result





Issues with JGroups– resources needed by a channel– instability of throughput with UDP– performance scalability

TCP better than UDP but– unable to disable reliability on top of TCP– unable to disable garbage collection– ordering implementation is sub-optimal

Need for a new group communication layer optimized for cluster



JGroups performance on UDP/FastEthernet


Outline



Design choices

JMX for both administration and monitoring– JMX server is MX4J

HTTP console for basic features but not suited for monitoring

Swing console for monitoring– uses JMX/RMI connector

Currently working on a scalable monitoring infrastructure


Online Monitoring


Viewing logs remotely


Outline



Current status

C-JDBC 1.0 rc2 releaseWeb site: >100.000 hits per monthDownloads: >1200 per monthMailing lists: 134 subscribers

Ready for production environments


Ongoing work

Scalable group communication middleware for clusters– Needed for horizontal scalability

– Decouple application streams from network channels

Monitoring– Continuous monitoring stored in repositories

– Generic monitoring console for both online monitoring and monitoring history browsing

Administration– Fully featured administration console– Graphical configuration and deployment of centralized/distributed backeds

and controllers (offline/online)– Dynamic reconfiguration

Misc.– Automated Load testing– RPM packaging (JPackage)


Questions & Answers_________

Thanks to all users and contributors ...

http://c-jdbc.objectweb.org


Virtual Database

MySQL

C-JDBC driver

Java client program

(Servlet, EJB, ...)

MySQL

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

MySQL JDBC driver

MySQL JDBC driver

Recovery Log


MySQL

Database Backend

Connection Manager

MySQL JDBC driver

gives the view of a single database

establishes the mapping between the database name used by the application and the backend specific settings

backends can be added and removed dynamically

configured using an XML configuration file


TPC-W Performance(Amazon.com)

0

200

400

600

800

1000

1200

1400

1600

0 1 2 3 4 5 6

Number of nodes

Th

rou

gh

pu

t in

req

ues

ts p

er m

inu

te

Single DB

RAIDb-0

RAIDb-1

RAIDb-2


Outline

OverviewFault tolerancePerformanceNew featuresRoadmap


Building the initial state

Octopus is an ETL tool Use Octopus to store a dump of

the initial database state

PostgreSQL

C-JDBC Controller

PostgreSQL JDBC driver

C-JDBC driver

JVM



disabled

dump for initial

checkpoint

Recovery LogOctopus

disabled


Journaling

Backend is enabled All database updates are logged

(SQL statement, user, transaction, …)

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



enabled

dump for initial

checkpoint

Recovery LogOctopus

JDBC Recovery Log

enabled


Adding backend on the fly

Add new backends while system online

Restore dump corresponding to initial checkpoint with Octopus

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus

PostgreSQL PostgreSQLdisabled disabledenabled

enabled

JDBC Recovery Log


Synchronizing backends

Replay updates from the log

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus

PostgreSQL PostgreSQLenabled

enabled

JDBC Recovery Log

disableddisabled


Expanded Cluster

Enable backends when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for initial

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled


Handling a backend failure

A node fails!Automatically

disabled but should be fixed or changed by administrator

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enableddisabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint


Restoring a backend

Restore latest dump with Octopus

...

dump for last

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for last

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enableddisabled

dump for initial

checkpoint


Re-synchronization

Replay missing updates from log

dump for

checkpoint

Octopus...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



Recovery Log


enabled

JDBC Recovery Log

disabled enabled


Healed Cluster

Re-enable backend when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint


Prototype

C-JDBC Management Framework

Shared design


Making new checkpoints

Disable one backend to have a coherent snapshot

Mark the new checkpoint entry in the log

Use Octopus to store the dump

...

dump for last

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for last

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enableddisabled

dump for initial

checkpoint



Replay missing updates from log

dump for

checkpoint

Octopus...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



Recovery Log


enabled

JDBC Recovery Log

disabled enabled



Re-enable backend when done

PostgreSQL

C-JDBC Controller


C-JDBC driver

JVM



dump for

checkpoint

Recovery Log

Octopus


enabled

JDBC Recovery Log

enabledenabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

jonas developer workshop – - [email protected] 1 - 23/02/2004 status emmanuel cecchet...

Documents

jonas modification

cjdbc middleware

cascading controller

heterogeneous databases

heterogeneous backends

controller sockets jmx

commodity hardware slide