redundant datacenter configuration with · pdf fileqnet mpls primary primary 3g private apn...
TRANSCRIPT
![Page 1: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/1.jpg)
Real Time Data Processing System
Real Time Data Acquisi7on System
Informa7on systems
Data Sources
Event informa7on ShakeMap informa7on
Event processing ShakeMap processing
Internet
Carrier O&M
BB Station
qnet
MPLS
primary
primary
3GPrivate APN
PrimaryData Center
BackupData Center
Data import & processing
en0
Data export
en0
CE
PE
CE
PE
NTP
SMS
Data export
en0
Data import & processing
en0
CE CE CE
SMS
NTP
BB Station
qnet
BB Station
qnet
BB Station
qnet
BB StationLAN
BB StationLAN
BB StationLAN Station LAN
SM Station
SM Station
SM Station
SM Station
SM Station
LAN switch
3G 3G
PEPE
PE
PE Internet
LAN switch
CE
Redundant Datacenter Configurat ion With Antelope
Stefan Radman Open Systems & Services Kinemetrics, Inc www.kmioss.com
Antelope User Group Mee5ng Baku, Azerbaijan
May 2014
![Page 2: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/2.jpg)
Outline
2
• System Layout
• Network Failover
• Applica9on Failover
![Page 3: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/3.jpg)
System Layout
3
Internet
Carrier O&M
BB Station
qnet
MPLS
primary
primary
3GPrivate APN
PrimaryData Center
BackupData Center
Data import & processing
en0
Data export
en0
CE
PE
CE
PE
NTP
SMS
Data export
en0
Data import & processing
en0
CE CE CE
SMS
NTP
BB Station
qnet
BB Station
qnet
BB Station
qnet
BB StationLAN
BB StationLAN
BB StationLAN Station LAN
SM Station
SM Station
SM Station
SM Station
SM Station
LAN switch
3G 3G
PEPE
PE
PE Internet
LAN switch
CE
![Page 4: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/4.jpg)
Network failover
4
Internet
Carrier O&M
BB Station
qnet
MPLS
primary
primary
3GPrivate APN
PrimaryData Center
BackupData Center
Data import & processing
en0
Data export
en0
CE
PE
CE
PE
NTP
SMS
Data export
en0
Data import & processing
en0
CE CE CE
SMS
NTP
BB Station
qnet
BB Station
qnet
BB Station
qnet
BB StationLAN
BB StationLAN
BB StationLAN Station LAN
SM Station
SM Station
SM Station
SM Station
SM Station
LAN switch
3G 3G
PEPE
PE
PE Internet
LAN switch
CE
Dynamic Re-‐rou7ng
![Page 5: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/5.jpg)
Applica9on Failover: System en99es
Real Time Data Processing System
Real Time Data Acquisi7on System
Informa7on systems
Data Sources
Event informa7on ShakeMap informa7on
Event processing ShakeMap processing
5
![Page 6: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/6.jpg)
Applica9on Failover: DataFlow -‐ MS Main System
6
orbserver:bulletin
IRISGSN
ORB ORBrtserve.iris.washington.edu
:18000SeedLink
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
WebEmailSMS
Data and information flow at Primary Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
orbserver orbserver ringserver
Data acquisition
ORB
ORB SeedLink
orbserver:acq
Data and information flow at Primary Data Center
Data export
Cooperating networks & institutions
orbserver:exp ringserver
Data export
orb2orborb2orb slink2orb
orb2genc
orb2ringserver
Primary Data CenterPublic IP
![Page 7: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/7.jpg)
orbserver:bulletin
IRISGSN
ORB ORBrtserve.iris.washington.edu
:18000SeedLink
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
WebEmailSMS
Data and information flow at Primary Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
orbserver orbserver ringserver
Applica9on Failover: DataFlow -‐ MS Main System
7
![Page 8: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/8.jpg)
Applica9on Failover: DataFlow -‐ MS Main System
8
orbserver:bulletin
IRISGSN
ORB ORBrtserve.iris.washington.edu
:18000SeedLink
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
WebEmailSMS
Data and information flow at Primary Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
orbserver orbserver ringserver
![Page 9: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/9.jpg)
Applica9on Failover: DataFlow -‐ MS Main System
9
orbserver:bulletin
IRISGSN
ORB ORBrtserve.iris.washington.edu
:18000SeedLink
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
WebEmailSMS
Data and information flow at Primary Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
orbserver orbserver ringserver
![Page 10: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/10.jpg)
Applica9on Failover: DataFlow -‐ BUS Backup System (BUS)
10
WebEmailSMS
orbserver:bulletin
IRISGSN
rtserve.iris.washington.edu:18000
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
Data and information flow at Backup Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
fromHQ
Analysts
Backup Data CenterPublic IP
Data acquisition
orbserver:acq
Data and information flow at Backup Data Center
Data export
Cooperating networks & institutions
orbserver:exp ringserver
Data Export
orb2orborb2orb slink2orb
orb2genc
orb2ringserver
![Page 11: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/11.jpg)
Applica9on Failover Backup System (BUS)
11
WebEmailSMS
orbserver:bulletin
IRISGSN
rtserve.iris.washington.edu:18000
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
Data and information flow at Backup Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
fromHQ
Analysts
• Concurrent data acquisi7on
• Processing at both data centers
• Two produc7on databases
• Unique IDs
• las7d replica7on
• One ac7ve informa7on system
![Page 12: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/12.jpg)
Applica9on Failover Backup System (BUS)
12
WebEmailSMS
orbserver:bulletin
IRISGSN
rtserve.iris.washington.edu:18000
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
Data and information flow at Backup Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
fromHQ
Analysts
• ID Server (dbids)
• ID Replica7on
(dbids.lastid)
• Monitoring
(idserver_connect)
• Failover 7me
• Authority
![Page 13: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/13.jpg)
Applica9on Failover Backup System (BUS)
13
WebEmailSMS
orbserver:bulletin
IRISGSN
rtserve.iris.washington.edu:18000
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
Data and information flow at Backup Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
fromHQ
Analysts
• Ini7ated by BUS
• Synchronize local dbids.las7d
• Ac7vate dbids
• Change db descriptor to use
local dbids
• Restart database writers
• Ac7vate informa7on systems
• No7fy operators (Email, SMS)
![Page 14: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/14.jpg)
Applica9on Failover Backup System (BUS)
14
• BUS providing database IDs
• BUS has authorita7ve database
• BUS feeding informa7on to the
world
• BUS prevents ac7va7on of
dbids on MS (armed) orbserver:bulletin
IRISGSN
ORB ORBrtserve.iris.washington.edu
:18000SeedLink
USGScatalog
M≥8
http://earthquake.usgs.gov
M≥4.5 archiving
M≥8
qed
RB:9500
RB:9500
RB:9500
RB:9500
RB:9500
SOH
SOH
SOH
SOH orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb
orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb
orb2dbt
orbdetect
orbassoc
orbevproc
orb2wf
orb2dbt
shake_mon
orbserver:acq
q330
q330
q330
q330
q3302orb
logs
orb2logs
orbdlalert
orbeqalert
dn dnids
orbserver:import
dbids
Officials
AnalystsEmailSMS
ShakeMap
dbloc2
orbeqalert
WebEmailSMS
Data and information flow at Primary Data Center
Data import and processing
Cooperating networks & institutions
Broa
dban
d st
atio
nsSt
rong
mot
ion
stat
ions
orbserver orbserver ringserver
![Page 15: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/15.jpg)
Applica9on Failover
15
MS • Check db consistency • Start rtexec (if not running) • Verify data flow
• Check db consistency vs dbids.las7d • Start dbids
• Start informa7on systems
BUS
• Stop db writers (orb2wf & orb2dbt) • Stop dbids • Replicate dbids.las7d to MS • Change db descriptor • Stop monitoring MS
• Start db writers • Start Monitoring MS • Stop informa7on systems
![Page 16: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/16.jpg)
BUS • Extract data to external disk (e.g. USB)
trexcerpt –v –e db/db <external disk> <star/me> <end1me> MS • Stop database writers • Merge data from external disk
trexcerpt –v –a –e <external disk> <star/me> <end1me> • Start database writers
Applica9on Failover
16
![Page 17: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/17.jpg)
Important • Wri_en procedures for stressful situa7ons • Drills and dry-‐runs for applica7on and network • At least once a year • Revise procedures if needed
Applica9on Failover
17
![Page 18: Redundant Datacenter Configuration With · PDF fileqnet MPLS primary primary 3G Private APN ... Antelope’User’Group’Mee5ng ... • BUS%providing%database%IDs%](https://reader034.vdocument.in/reader034/viewer/2022051722/5aa398b47f8b9a80378e722d/html5/thumbnails/18.jpg)
Redundant Data Center Configura9on
18
THANK YOU