Data mirrors: Data mirrors: Techniques and IssuesTechniques and Issues

Data mirrors: Data mirrors: Techniques and IssuesTechniques and Issues

ByBy

Sarah PonrathnamSarah Ponrathnam

sarah@iucaa.ernet.insarah@iucaa.ernet.in

http://www.iucaa.ernet.inhttp://www.iucaa.ernet.in//

Overview Introduction…What is Data Mirroring?Techniques of Mirroring…Motives for Mirroring…Mirror site Issues…Popular Mirroring Tools in Unix…Mirroring Experience at IUCAA…

What is data mirroring?

Creating an exact duplicate copy in real-time.

In terms of web sites, sites are often mirrored to reduce the traffic on one server.

Techniques of MirroringReplication is one way to solve

availabilityproblem.

Distributed Servers Cluster Servers Web site Mirrors

Techniques of Mirroring Distributed Servers:Large web destinations such as google,Yahoo have enough capital to set upand to support distributed servers.Eg. www.google.com, www.google.co.in

Techniques of MirroringCluster servers:

Techniques of Mirroring Website Mirrors:Mirrors simply compare and pull the contentsfrom a single master web site at a regular intervals and make identical contentsavailable on another computer, ideally closerto the users making use of it.

Motives for Mirroring Load Balancing - yahoo.co.uk yahoo.co.inHigh Availability - ADS mirrorsMultilingual replication - www.debian.org

Database Sharing – www.desert.net www.tucksonweekly.com

Franchise / Local Versions - quicken.excite.com quicken.com Virtual Hosting - sports.catalogue.com

www.accesports.com

Mirroring IssuesMaintenance of mirror-sites in different

geographical locations. Integrity of the mirrored contents Providing Host-Independent URLS. Initial transfer of data

Optimization Economical constraintsProviding Efficient routing

Popular Mirroring tools on Unix

RSync http://samba.anu.edu.au/rsync/Wget http://www.gnu.org/Mirror http://www.wehlus.de/mirror/download.html

Mirroring ExperienceIUCAA (Inter University CentreFor Astronomy and Astrophysics)works closely with ERNET(Educational and ResearchNETwork) to make this network a content orientednetworkhttp://www.iucaa.ernet.in/

ERNET India ? In 1986, then Department of Electronics

had initiated a project "ERNET" with thefunding from UNDP. The objective was toestablish and operate a nationwideInternet for the Indian academic andresearch community. Now it has become a full pledged ISP known as ERNET India.

ERNET partners

ERNET Network

ERNET NOC at IUCAA

Mirrors available in IUCAA

VizieR provides access to the most complete library of published astronomical catalogues and data tables available on line, organized in a self-documented database. http://urania.iucaa.ernet.in

Mirrors available in IUCAA

The NASA Astrophysics Data System (ADS) maintains four bibliographic databases containing more than 4 million records. http://ads.iucaa.ernet.in

VOI Data Archives at IUCAA

SDSS Sloan Digital Sky Survey – 1 Tb2MASS 2 Micron All Sky Survey – 194 Gb2dfGRS 2 degree field Galaxy Redshift

Survey – 5.5 Gb2QZ 2 Degree field QSO Survey – 630

Mb  FIRST Survey Faint Images of Radio Sky

at Twenty centimeters – 226 Gb

VOI Hardware

Mirror-site under construction

Chandra Data Archive (CDA) 424 GB data is available through ftp

service. ftp://cdaftp.iucaa.ernet.in

Incremental update is being done regularly.

Web based CDA service will be provided, once the new release of CXCDS software is made available.

AcknowledgementDr. Francois Ochsenbein - VizieRDr. Guenther Eichhorn and Dr.Alberto

Accomazzi - ADSDr. Ramadurai Padmanabhan - CDA

Questions

or

Comments?