7 secrets of wells fargo soa platform's high availability
DESCRIPTION
7 Secrets of Wells Fargo SOA platform's high availabilityTRANSCRIPT
Ram Lakshmanan
Sr. Systems Architect
Java One – 9/23/2013
secrets of Wells Fargo SOA
platform's 99.99% availability
About Wells Fargo
Just two data points
– Serving 1 in 3 US households
– Originating 1 in 3 US mortgages
About Wells Fargo
Just two data points
– Serving 1 in 3 US households
– Originating 1 in 3 US mortgages
Global SIFI (Systemically Important Financial Institution)
About Wells Fargo
Just two data points
– Serving 1 in 3 US households
– Originating 1 in 3 US mortgages
Global SIFI (Systemically Important Financial Institution)
4th in assets & 1st in market value among US peers
No. 1 most valuable financial services brand in the world (Brand Finance, The Banker)
Agenda
High Level Architecture
Secrets
1. AAA…
2. One 9
3. Lord protect me from my friends
4. 240k
5. Cockpit
6. Wackamole
7. *
Meta-Secret
High Level Architecture
High Level Architecture
Online App
Mobile App
Stores App
Call Center App
Other LOB Apps (100+)
High Level Architecture
SOR-2
SOR-60
SOR-5
SOR-3
SOR-4
SOR-6
SOR-1
Online App
Mobile App
Stores App
Call Center App
Other LOB Apps (100+)
High Level Architecture
SOR-2
SOR-60
SOR-5
SOR-3
SOR-4
SOR-6
SOR-1
Online App
Mobile App
Stores App
Call Center App
Other LOB Apps (100+)
SOA <SOAP/>
HTTP
JDBC
JMS
HTTPS
LDAP
Agenda
High Level Architecture
Secrets
1. AAA…
2. One 9
3. Lord protect me from my friends
4. Cockpit
5. 240k
6. Wackamole
7. *
Meta-Secret
Secret 1: AAA…
datacenters: active-active-active…
Secret 2: One 9
Secret 2: One 9
With “intelligent” Software that is running on one 9 (i.e. 90%) hardware availability you can achieve Four 9s (i.e. 99.99%) availability
SOR SOA
SOR SOA
Data Center 1
Data Center 2
Secret 2: One 9
With “intelligent” Software that is running on one 9 (i.e. 90%) hardware availability you can achieve Four 9s (i.e. 99.99%) availability
SOR SOA
SOR SOA
Data Center 1
Data Center 2
Secret 2: One 9
With “intelligent” Software that is running on one 9 (i.e. 90%) hardware availability you can achieve Four 9s (i.e. 99.99%) availability
SOR SOA
SOR SOA
Data Center 1
Data Center 2
Secret 2: One 9
With “intelligent” Software that is running on one 9 (i.e. 90%) hardware availability you can achieve Four 9s (i.e. 99.99%) availability
SOA JMS Server
Secret 2: One 9
With “intelligent” Software that is running on one 9 (i.e. 90%) hardware availability you can achieve Four 9s (i.e. 99.99%) availability
SOA JMS Server
storage
Secret 3: Lord protect me from my friends
SOR 2
SOR 60
SOR 3
SOR 1
SOA
Secret 3: Lord protect me from my friends
SOR 2
SOR 60
SOR 3
SOR 1
SOA
Secret 3: Lord protect me from my friends
SOR 2
SOR 60
SOR 3
SOR 1
SOA
SOR 2 SOA
Secret 3: Lord protect me from my friends
SOR 2
SOR 60
SOR 3
SOR 1
SOA
SOR 2 SOA
Secret 3: Lord protect me from my friends
SOR 2
SOR 60
SOR 3
SOR 1
SOA
SOR 2 SOA
Secret 3: Lord protect me from my friends
SOR 2
SOR 60
SOR 3
SOR 1
SOA
SOR 2 SOA
Secret 3: Lord protect me from my friends
SOR 2
SOR 60
SOR 3
SOR 1
SOA
SOR 2 SOA
Secret 3: Lord protect me from my friends
SOR 2
SOR 60
SOR 3
SOR 1
SOA
SOR 2 SOA
SOR 2
SOA
Secret 3: Lord protect me from my friends
Fault Containment
SOR 2
SOR 60
SOR 3
SOR 1
SOA
SOR 2 SOA
SOR 2
SOA
Secret 3: Lord protect me from my friends
Fault Containment
Fail Fast (Exponential Backoff)
SOR 2
SOR 60
SOR 3
SOR 1
SOA
SOR 2 SOA
SOR 2
SOA
Secret 4: 240k
Secret 4: 240k
80K regression tests
Secret 4: 240k
80K regression tests
3x production load testing
Secret 4: 240k
80K regression tests
3x production load testing
EDS – Enterprise Data Simulator
Secret 5: Cockpit
Pilot*
* - if one thing you want to take away from this session
Secret 5: Cockpit
Pilot*
Switches (installing without outage)
* - if one thing you want to take away from this session
Secret 6: Wackamole
http://www.youtube.com/watch?v=6apVZWu_1LU
Secret 6: Wackamole
Out of control – CPU will spike, memory will leak, JVM will crash.
sometimes
^
http://www.youtube.com/watch?v=6apVZWu_1LU
Secret 6: Wackamole
Out of control – CPU will spike, memory will leak, JVM will crash.
Strong monitoring & instant reactions
sometimes
^
http://www.youtube.com/watch?v=6apVZWu_1LU
Secret 7: .*
Secret 7: .*
Dangerous species
Secret 7: .*
Dangerous species
Restart Button
Secret 7: .*
Dangerous species
Restart Button Factory Reset
Secret 7: .*
Dangerous species
Restart Button
SODA – Service Oriented Development & Assembly
Factory Reset
Secret 7: .*
Dangerous species
Restart Button
SODA – Service Oriented Development & Assembly
Not ‘So Cool’
– Never chosen technology for the sake of “Coolness”
– But full stack is running almost on Open Source technologies
Factory Reset
Agenda
High Level Architecture
Secrets
1. AAA…
2. One 9
3. Lord protect me from my friends
4. 240k
5. Cockpit
6. Wackamole
7. *
Meta-Secret
Meta-Secret: Our Team
