EMu 24/7@ Museum Victoria

‘The bird that never sleeps’

Donna FothergillDavid Zhang

View from Nicholson St, blades featuring Melbourne Museum venue branding.Photographer: Jennifer McNairDate: June 2007Source: Museum VictoriaMV Images

Overview

Why?The decision to make EMu available 24/7 at Museum Victoria

How?The Infrastructure making this possible (hardware/scripts/resources)

What’s Next? Other Tools

Eight-hour day banner, Melbourne, 1856Date: 1856Source: Museum VictoriaMV Images

The MV EMu Guarantee

In the event of:

Primary Server Crash

System/Database Maintenance

EMu upgrades

EMu available 24/7EMu available 24/7

Quote:

“we can rebuild him, we have the technology”

-Oscar Goldman

Importance

System Integration- MV Images- MVWISE

Live Feeds- e.g. The Learning Federation

Online collections- e.g. Natural Sciences

Core business systemCore business system

EMu FigurinePhotographer: David PaulDate: 11/07/2011Source: Museum VictoriaMV Images

+ =

EMu Mobile

Technology

MVWISE

MVWISE

‘EMu in the palm of your hand’‘EMu in the palm of your hand’

MV Images

Butterflies on Display: Bugs Alive! Exhibition : Melbourne MuseumPhotographer: Andrew CurtisDate: 03/09/2010Source: Museum VictoriaMV Images

EMu

Core database for all Collection items

MVWISE

Application for location and inventory control

MV Images

Digital asset management system for all digital images

and audio in the State Collection and other

resources

Full synchronisation –

desktop and handheld devices

Data sharing using iMu web services

Systems Integration

Live Data Feeds‘The Learning Federation’‘The Learning Federation’

Online Collections‘Search Natural Sciences Collections’‘Search Natural Sciences Collections’

Remote Access‘regardless of connection, location or device’‘regardless of connection, location or device’

Extending Access

Fieldwork Projects/Special Projects

Out of Hours Availability

Testing environments

‘Where ever Internet Access is available, so too is EMu! ’

‘Where ever Internet Access is available, so too is EMu! ’

The Moon, from a Photograph taken by the Great Melbourne Telescope, Date: 1874: Source: Museum VictoriaMV Images

Primary Server Failure

Synchronisation Synchronisation

Maintenance/Upgrades

Snapshot Snapshot

Backend processes

EMu is synchronised from live server to backup server every hour

Full EMu access is available using the EMu client (except during maintenance)

Snapshot taken of backup service prior to maintenance

During maintenance, the access becomes read-only.

Quote:

“(EMu) ... looks good at the front, but has a big backend”

Anonymous!

Infrastructure

Live EMu Server

‘EMu Synchronisation process’‘EMu Synchronisation process’

Changes to live EMu are transported to EMu on the

backup server

Changes to live EMu are transported to EMu on the

backup serverBackup EMu Server

Email notification sent to support staffEmail notification sent to support staff

Data processing is carried out on the backup server for ‘fast’

synchronisation

Data processing is carried out on the backup server for ‘fast’

synchronisation

Primary Server

Dedicated 10 GB link and RAM for synchronisation temp space

Dedicated 10 GB link and RAM for synchronisation temp space

Synchronisation Performance

Tue Aug 30 12:02:05 EST 2011...start emu sync to backup serverTue Aug 30 12:02:05 EST 2011...stop emu loadTue Aug 30 12:02:11 EST 2011...get all the deletions search for new/changed multimedia objectsTue Aug 30 12:02:18 EST 2011... start searching new imgTue Aug 30 12:02:19 EST 2011...start exporting modified/new recordsTue Aug 30 12:02:22 EST 2011...move data to backup serverTue Aug 30 12:02:25 EST 2011...moving loads directory to backup serverTue Aug 30 12:02:25 EST 2011...restart backgroud loadTue Aug 30 12:02:26 EST 2011...finish sync on bronte

Live EMu service = 75GBLive EMu service = 75GB

Synchronisation Summary

A dedicated private link and RAM for temp space

Data processing is deferred to the backup server

It takes less than 30 seconds for synchronisation to complete

‘Checks and whistles’ in place to avoid data corruption

Infrastructure

Live EMu Service

‘EMu Snapshot prior to maintenance ’‘EMu Snapshot prior to maintenance ’

Backup of live EMu Service

Live EMu is synched to the Backup serverBackup Server

PrimaryServer

Snapshot EMuis

read only

Snapshot EMuis

read only

Snapshot created from backup of live EMu Service

Service changes from emu to snapshot EMu

And back again when maintenance completes

Maintenance iscarried out on

Live EMu

Maintenance iscarried out on

Live EMu

Snapshot Summary

Live EMu is synched to backup server

The snapshot is taken from the backup live service and placed on the primary server.

It is a read only copy of the live service.

New connections and/or requests after 5pm are directed to the snapshot

When maintenance is complete, the service switches back to the live service.

Synchronisation process starts again

Camera - Parisio Cantini Empoli, Bellows Glass Plate, circa 1880Photographer: Taryn EllisDate: 03/11/2010Source: Museum VictoriaMV Images

What’s Next?

Live EMu 24/7 ?Live EMu 24/7 ?

Other ToolsManaging our large data files Managing our large data files

fupdateCompactSortUpdate load/data file compaction tool

fselectRecover Deleted Records

getDbRefDetails Delete batches of records in the backend

fupdateCompactSort

Example: update 80 fields in 284,297 catalogue records + compaction

Example: update 80 fields in 284,297 catalogue records + compaction

Data fileData file texloadtexload texcompacttexcompact Index Rebuild Index Rebuild

Data fileData fileUpdate +Compaction

Update +Compaction

1 hour 37 minutes

Index Rebuild Index Rebuild < 1 minute

getDbRefDetails

delete batches of recordsdelete batches of records

/usr/local/emu/bin/getDbRefDetails etaxonomy

Primary DB = "etaxonomy"; Linked Modules ecatalogue => TaxTaxonomyRef_tab 10 BirCuckooRef 1 IdeFiledAsRef 1  enarratives => TaxTaxaRef_tab 45 SpeLookalikesRef_tab 20  etaxonomy => HisAccNameRef_tab 20 TaxAvailabilityCorrectNameRef_tab 20 HomNameRef_tab 20 TaxAvailabilityCompetingHomRef_tab 20 ClaCurrentNameRef 1 PriTypeAboveSpeciesNameRef 1 ClaHybridParent2NameRef 1 SynNameRef_tab 100

fselect

Recover deleted recordsRecover deleted records

usage: fselect [options] dbname Options are: -d alternative data file -D print out ins forms -Sstatus record status -c set the status of selected

records to current -a analyse -kirn get the records with this irn -un get nth record with this irn -fFile save processed records into

File

Computer Walking and TalkingPhotographer: Frey MicklethwaitDate: June 2011Source: Museum VictoriaMV Images