apache geode clubhouse - wan-based replication

© Copyright 2016 Pivotal. All rights reserved. © Copyright 2013 Pivotal. All rights reserved.

WAN GatewayMulti-site and Active-active Design Patterns

© Copyright 2016 Pivotal. All rights reserved.

• The multi-site capability connects geographically separated distributed systems.

• It is important to understand that while you are presented with a façade that APPEARS like it is all one system, each of the distributed systems actually behaves autonomously

• A multi-site installation based on the WAN Gateway consists of two or more distributed systems that are loosely coupled.

• Each site manages its own distributed system, and region data is distributed to remote sites using one or more connections.

Using WAN Gateways


Using WAN GatewaysThe connections consist of a gateway sender in the sending site and a corresponding gateway receiver in the receiving site.


Multi-site TopologiesDISASTER RECOVERY

Disaster Recovery or Business Continuity is still the most popular design pattern for multi-site replication using the WAN Gateway.

One important distinctionGeode WAN Gateway is designed to be bi-directional. That makes fail-back much easier.


Multi-site TopologiesActive/Active

If your use-case is read-only, or updates are limited to specific entries belonging to a given user, then it is extremely easy to load balance across multiple sites hosting Geode clusters in an Active/Active configuration.

Apache Geode

IIS Farm

DB

Clients

Apache Geode

IIS Farm

DB

Site 1

Site 2

WAN Gateway


Multi-site TopologiesActive/Passive

Alternatively, if you have heavy updates on random data entries, you might want to use an Active/Passive configuration.

Apache Geode

IIS Farm

DB

Clients

Apache Geode

IIS Farm

DB

Site 1

Site 2

WAN Gateway


Multi-site TopologiesBusiness Unit Active/Passive

Or if you have different groups of users who update different datasets, you can make each cluster active for one Business Unit, and backup for the other.

Apache Geode

IIS Farm

DB

Equity Users

Apache Geode

IIS Farm

DB

Site 1

Site 2

WAN Gateway

Debt Users


Multi-site TopologiesGeographically Separated

Many times you are using a multi-site topology to achieve locality of reference for performance purposes


Multi-site Active-active Design Patterns1. Exchange Pattern

NYSE

LSE LSE

TSE

NYSE, TSE Read--only

LSE, TSE Read--only

NYSE, LSE Read--only

Client connects to all exchanges it needs for writing, uses local copy for read only access.


Multi-site Active-active Design Patterns2. The "Realm Manager" Pattern:Use the “Command” pattern to request that an action be performed on your behalf.

Request gets forwarded to all distributed systems but only the one with the right permission actually takes the action.

Read Only For This Customer

Read Only For This Customer

Write Permission For This Customer


Multi-site Active-active Design Patterns

3. Follow the Sun Pattern:This is the "Global book" pattern common in Financial Services.

The token is here



4. Inventory Allocation Pattern:

This pattern is commonly used when there are multiple trading venues and selling short is not allowed.

Partial Inventory

Partial Inventory

Partial Inventory

Partial Inventory



5. Apology based computing:This is the pattern that Max Feingold refers to when he says:

“At global scale, getting the truth is really really expensive.”


Configuring WAN GatewaySimple ConfigurationThe minimum configuration you need to do is this…

<cache> <gateway-sender id="NY" parallel=”true" remote-distributed-system-id="1” /> ...</cache>


Enabling Persistence for QueuesOverflowing Gateway Queues to diskTo overflow the Gateway Queues to disk to conserve memory do this…

<cache> <gateway-sender id="NY" parallel="false" remote-distributed-system-id="1" enable-persistence="true" disk-store-name="gateway-disk-store" maximum-queue-memory="200” /> ...</cache>


Multiple dispatcher threadsMultiple dispatcher threads for Parallel WAN and Async Event Queues

• Geode now defaults to 5 dispatcher threads for a parallel

WAN gateway or async event queue.

• If you detect that your system is using too much CPU,

modify the dispatcher-threads=1 in the gateway-sender

attributes.

• Default is ordering by key


Multiple dispatcher threadsConfiguring Dispatcher Threads and Ordering Policy for a Serial GatewayTo increase the number of dispatcher threads and set the ordering policy for a serial gateway sender, use one of the following mechanisms.

<cache> <gateway-sender id="NY" parallel="false" remote-distributed-system-id="1" enable-persistence="true" disk-store-name="gateway-disk-store" maximum-queue-memory="200" dispatcher-threads=10 order-policy="key"/> ...</cache>


Additional BenefitsThere are additional benefits that the WAN Gateway gives you

• Site-wide rolling release--roll each site independently

• Geode major release upgrades

• Mix of Cloud and On-prem


Technical Details


Using WAN GatewaysConsistency for WAN Updates• With a distributed WAN configuration, one or more gateway senders

asynchronously queue and send region updates to another Geode cluster.

• It is possible for multiple sites to send updates for the same region entry at the same time.

• It is also possible that, due to a slow WAN connection, a cluster might receive region updates after a considerable delay, and after it has applied more recent updates to the region.


Using WAN GatewaysConsistency for WAN Updates• To ensure that WAN propagated regions eventually reach a

consistent state, Geode first ensures that each cluster performs consistency checking to regions before queuing updates to a gateway sender for WAN distribution.

• In other words, region conflicts are first detected and resolved in the local cluster, using local timestamp and conflict detection algorithms


Using WAN GatewaysPartitioned Region Consistency• For a partitioned region, Geode maintains consistency by routing all

updates on a given key to the Geode member that holds the primary copy of that key.

• That member holds a lock on the key while distributing updates to other members that host a copy of the key.

• Because all updates to a partitioned region are initially processed on the primary Geode member, all members apply the updates in the same order and consistency is maintained at all times.


Using WAN GatewaysReplicated Region Consistency• For a replicated region, any member that hosts the region can

update an entry and distribute that update to other members without locking the entry.

• It is possible that two members can update the same entry at the same time (a concurrent update).

• It is also possible that, due to network latency, an update in one member is received by other members at a later time, after those members have already applied more recent updates to the entry (an out-of-order update)


Using WAN GatewaysReplicated Region Consistency• If two members update the same entry at the same time, conflict

checking ensures that all members eventually arrive at the same value, which is the value of one of the two concurrent updates.

• If a member receives an out-of-order update (an update that is received after one or more recent updates were applied), conflict checking ensures that the out-of-order update is discarded and not applied to the cache.


Using WAN GatewaysConsistency for WAN Updates• In a default configuration, the cluster that receives the event

examines the timestamp to determine whether or not the event should be applied.

• If the timestamp of the update is earlier than the local timestamp, the cluster discards the event.

• If the timestamp is the same as the local timestamp, then the entry having the highest distributed system ID is applied (or kept).


Using WAN GatewaysDiscovery for Multi-Site Systems• Each Geode cluster in a WAN configuration uses locators to

discover remote Geode clusters• In the configuration for each locator in a WAN configuration you

must define a unique distributed-system-id property that identifies the local cluster

• A locator uses the remote-locators property to define the addresses of one or more locators in remote Geode clusters to use for WAN distribution.


Using WAN GatewaysDiscovery for Multi-Site Systems• When a locator starts up, it contacts each locator that is configured

in the remote-locators property to exchange information about the available locators and gateway receivers in the cluster.

• The locator also shares information about locators and gateway receivers in any other Geode clusters that have connected to the cluster.

• Connected clusters can then use the shared gateway receiver information to distribute region events according to their configured gateway senders.

• Each time a new locator starts up or an existing locator shuts down, the changed information is broadcast to other connected Geode clusters.


Using WAN GatewaysGateway Senders• A Geode cluster uses a gateway sender to distribute region events

to another, remote Geode cluster.• You can create multiple gateway sender configurations to distribute

region events to multiple remote clusters, • A gateway sender always communicates with a gateway receiver in

a remote cluster. • Gateway senders do not communicate directly with other cache

server instances • Geode provides two types of gateway sender configurations: serial

gateway senders and parallel gateway senders


Using WAN GatewaysSerial Gateway Senders• A serial gateway sender distributes region events from a single

Geode server in the local cluster to a remote Geode cluster. • Although multiple regions can use the same serial gateway for

distribution, a serial gateway uses a single logical event queue to dispatch events for all regions that use the gateway sender.

• Because a serial gateway sender distributes all of a region's events through a single Geode member, it provides the most control over ordering region events as they are propagated across the WAN.

• However, a serial gateway sender does not provides horizontal scale of throughput for propagating events.


Using WAN GatewaysParallel Gateway Senders• While parallel gateway senders provide the best throughput for

WAN propagation, they provide less control for ordering events. • With a parallel gateway sender, you cannot preserve event ordering

for the region as a whole because multiple Geode servers distribute the region events at the same time.

• However, the ordering of events for a given partition can be preserved

Note: Replicated regions can only be configured to use serial gateway senders.


Using WAN GatewaysGateway Sender Queues• The queue that a gateway sender uses to distribute events to a

remote site can be overflowed to disk as needed, in order to prevent the Geode member from running out of memory.

• You should configure the maximum amount of memory that each queue uses, as well as the batch size and frequency for processing batches

• You should also configure these queues to persist to disk, so that a gateway sender can pick up where it left off when its member shuts down and is later restarted.


Using WAN GatewaysMulti-threaded dispatcher• By default gateway sender queues (even in serial gateway senders)

use 5 threads to dispatch queued events.• If ordering is required on a serial gateway sender you should set the

number of dispatcher threads to 1.


Using WAN GatewaysHigh Availability for Gateway Senders• When a serial gateway sender configuration is deployed to multiple

Geode members, only one "primary” sender is active at a given time. All other serial gateway sender instances are inactive "secondaries" that are available as backups if the primary sender shuts down.

• Geode designates the first gateway sender to start up as the primary sender, and all other senders become secondaries.

• As gateway senders start up and shut down in the distributed system, Geode ensures that the oldest running gateway sender operates as the primary


Using WAN GatewaysHigh Availability for Gateway Senders• A parallel gateway sender is deployed to multiple Geode members

by default, and each Geode member that hosts primary buckets for a partitioned region actively distributes data to the remote Geode site.

• When you use parallel gateway senders, high availability for WAN distribution is provided if you configure the partitioned region for redundancy.

• With a redundant partitioned region, if a member that hosts primary buckets fails or is shut down, then a Geode member that hosts a redundant copy of those buckets takes over WAN distribution for those buckets.


Using WAN GatewaysGateway Receivers• A gateway receiver configures a physical connection for receiving region

events from gateway senders in one or more remote Geode clusters.• A gateway receiver applies each region event to the same region or

partition that is hosted in the local Geode member. (An exception is thrown if the receiver receives an event for a region that it does not define.)

• Gateway senders use any available gateway receiver in the target cluster to send region events.

• You can deploy gateway receiver configurations to multiple Geode members as needed for high availability and load balancing.

There are issues with balancing of senders and receivers


Multi-site TopologiesParallel Multi-site Topology• This is the most often recommended topology• This is one where all sites know about each other. • This is the most robust configuration, where any one of the sites can go

down without disrupting communication between the other sites. • A parallel topology also guarantees that no site receives multiple copies

of the same message.Parallel Multi-site Topology is the recommended topology for most use-cases. Think of it as a “mesh” configuration.


Thank you

apache geode clubhouse - wan-based replication

Technology