An Online Synoptic Search and Metadata Visualization Tool (UVI-OST)

G. A. GermanyUniversity of Alabama in Huntsville

Huntsville, AL

C.-C. HungSouthern Polytechnic State University

Marietta, GA

http://csds.uah.edu/uvi-ost/

Abstract

“The primary goal of the proposed work is to provide an online search and visualization tool of auroral and geophysical metadata covering the ascending phase of the current solar cycle.”

Project goals have been achieved and important new research targets of opportunity have been pursued.

The Problem• ~8 x 106 images in UVI image collection (and growing)

• Before this project… • No method existed for searching instrument info• No method existed for searching image info• Published studies almost exclusively event studies

0

10

20

30

40

50

1996 1997 1998 1999

Events

Surveys

Others

Unclassif ied

Use r In te rfa c e

RequestHandler

MetadataDisplay

ImageData

AuxSC Data Indices

Metadata

Database Tables

Dat

a Le

vel

Met

a L

eve

lU

ser

Leve

l

Clie

ntH

ost

Dat

a

The Solution• Database of instrument and image info (metadata) is assembled with software ‘miners’

• Online interface allows users to search and visualize metadata

• Result of query is list of times meeting search criteria

• Link is provided for data download service

• Includes image views and diagnostics

Project Impact• Usage statistics show continued access throughout project

• Some correlation with meeting and proposal deadlines

• Tool used for UVI operational planning and to diagnose instrument anomalies

• 8 papers or presentations directly related to OST

• Multi-year statistical studies in progressThru 3/23/05:• 6738 total queries • 324 unique IP addresses • ~100 different institutions

Worldwide ImpactUnique IP Addresses

Australia 2

Austria 4

Belgium 1

Canada 20

Finland 1

France 7

Great Britain 21

Iceland 1

India 2

Ireland 1

Italy 1

Japan 31

Norway 12

Poland 2

Russia 24

South Korea 13

Sweden 6

United States 171

Zambia 1

Centre Universitaire de Velizy Royal Institute of Technology UTA Telekom Uninett North Norway2C ComputingAKnetAPI DIGITAL COMMUNICATIONSAfrican Network Information Center America OnlineAmeritechAustrian Academy of SciencesBellsouthBoston UniversityBritish Antarctic Survey (NERC)Carnegie Mellon UniversityCentre d'Etude Spatiale des RayonnementsChungbuk National UniversityComcastCompuserveComputational Physics Consiglio Nazionale delle RicercheDartmouth CollegeEarthLink NetworkExchange Point BlocksFinnish University and Research NetworkGeorge Mason UniversityInstitute for InformaticsInstitute of Nuclear PhysicsJapan Network Information CenterJapan Network Information Center John Hopkins University Applied Physics LaboratoryKola Science Centre of Russian Academy of ScienceKorea Advanced Institute of Science and TechnologyKorea Network Information CenteKorea TelecomKyoto UniversityKyungpook National University

Leicester UniversityLevel 3 CommunicationsMarshal Space Flight CenterMassachusetts Institute of Technology Mission Research Corporation NEC CorporationNational Space Science and Technology

CenterNaval Research Laboratory Nicolaus Copernicus UniversityNorway UniversityOARnetOJSC RTCommObservatory Meteorology AeronomyPetrozavodsk State UniversityRiNet Dynamic DialupRoadRunnerRogers Cable IncRoyal Institute of TechnologyRussian Space Research InstituteRutherford Appleton LaboratorySRI InternationalSaskTelSatyam SouthwestBellSprint-IPDialSt.Petersburg State UniversityStanford UniversityStarwood HotelsState of Georgia/Board of RegentsSwipenetTelia Network ServicesThe University of ElectroCommunicationsTohuku UniversityUS West TechnologiesUniversite Louis PasteurUniversite Pierre et Marie Curie

University of AlabamaUniversity of AlaskaUniversity of AlbertaUniversity of CalgaryUniversity of California

(3),LA,Berkley,San DiegoUniversity of ColoradoUniversity of IowaUniversity of LancasterUniversity of LowellUniversity of MichiganUniversity of MinnesotaUniversity of New Brunswick University of NewHampshireUniversity of NewcastleUniversity of OsloUniversity of SouthamptonUniversity of Texas (2),Austin,DallasUniversity of TromsoUniversity of WalesUniversity of WashingtonUniversity of WaterlooUniversity of YorkUppsala UniversityUtah State UniversityVidesh Sanchar Nigam LtdWaterford Institute of Technology

Metadata Definitions

Instrumental Geophysical

Data availability Activity indices & solar wind

Operational modes Auroral boundaries

Image FOV info * Auroral areas

Data Quality Auroral intensities

Auroral morphology

Auroral energy flux

* On earth? | % oval seen | Ground station in FOV?

NOTE: This is not metadata in the normal sense, i.e. data used to describe elements of a data format, for example. The metadata above are descriptions of the image data and geophysical activity associated with the image data. The key point is that this metadata can be data based and searched on.

MetadataImageData

SegmentedImage

ImageMask

RegionMask

Polar Cap

Auroral Oval

FOV edge

Software routines ‘mine’ the UVI data set, extracting boundaries, areas, and total intensities. The location of the oval relative to the edge of the image is stored to help determine how much of the auroral oval we are seeing. Where appropriate, these parameters are calculated as a function of magnetic local time. Additional parameters needed to understand the extracted information, such as the solar zenith angle (SZA), are also cataloged.

Image Metadata• Quality Flags

– Low global power or low contrast

• FOV flags – what is in field of view?– Oval: 0 (all), 1 (nightside), 2 (some)– Cap: 0, 2

• Scalars (desired)– Fraction of oval or cap seen

Image metadata

Segmentation

• A modified K-means algorithm based on the histogram distribution was developed for the UVI image segmentation.

• From our experiments, it is unlikely that this segmentation will be trapped into the local optimal solution.

• Image segmentation is an essential step towards the higher level information processing such as substorms detection in image analysis approach.

Segmentation (cont.)

K-means Clustering Algorithm

Step 1: Choose initial cluster centers

Step 2: Distribute image pixels among

clusters

Step 3: Compute new cluster centers

Step 4: Repeat steps 2 and 3 until it converges.

k=2,3,4,5

Segmentation (cont.)• An adaptive K-means algorithm

based on the histogram distribution was recently developed for the UVI image segmentation.

• This adaptive version can segment the UVI image more accurately (Fine Segmentation) for further higher level UVI image analysis.

(a) An original image, (b) segmented using the static histogram, and (c) segmented using the dynamic histogram, with the same parameters for K-means algorithm.

(a)

(c)(b)

Dayglow Models

Mining techniques alloweddevelopment of a statisticaldayglow model based onmulti-year observations.

Model is more accurate andmore stable than previoustechniques.

Original

Dayglow

Corrected

CorrectionBias

http://csds.uah.edu/uvi-ost/

Site Overview

1. Search

2. Visualize

4. Info3. Delivery

1. Search

User can search on time, instrument ops, FOV constraints, geophysical conditions, and image info.

2. Visualize

3. Delivery

4. Info

User manual gives site instructions and background information necessary to understand the results of the data query

Targets of Opportunity• Not in original proposal

– Easy/moderate implementation– Significant impact

Thumbnails Links to UVI online data

Ground Stations Solar Wind Data delivery service Synoptic science studies

Papers• Germany et al., A simple automated algorithm for substorm identification in space-based auroral

images, to be submitted to Geophys. Res. Lett. 2005.

• Emery et al., Image Comparisons with the Auroral Electron and Ion Hemispheric Power after Intersatellite Adjustments and Geophysical Variations, Space Weather Week, 2005.

• Germany, G.A., J.F. Spann, C. Deverapalli, and C. Hung, The utility of auroral image-based activity metrics, Eos. Trans. AGU, 85(47), Fall Meet. Suppl., Abstract SA51B-0247, 2004.

• Germany, G.A., C.-C. Hung, R.A. Doe, D. Lummerzheim, and G.K. Parks, Multi-year analysis of FUV auroral images, Eos Trans. AGU, 84(46), Fall Meet. Suppl., Abstract SM51B-0516, 2003.

• Hung, C. C., and G. Germany, K-means and Iterative Selection Algorithms in Image Segmentation, in IEEE Southeast Conference, Session 1: Software Development, Jamaica, West Indies, 2003.

• Germany et al., Multi-year analysis of FUV auroral images, SM51B-0516, Fall AGU 2003.

• Doe et al., Impact of Global Averaging on UVI-based Bz North Conductance Estimates, SM51B-0517, Fall AGU 2003.

• Germany, G. A., C. Hung, D. Chua, Y. Tung, J.F. Spann, and G.K. Parks, Extended Synoptic Analysis Using a Database of Auroral Images, Eos Trans. AGU, 83(47), Fall Meet. Suppl., Abstract SM12A-0486, 2002.

“Polar UVI images are used herein to investigate global conductance morphology for Bz north conditions.

These brightness maps indicate a By-dependent variance structure in agreement with modeled and observed morphology of polar cap arcs.”

Doe et al., Impact of Global Averaging on UVI-based Bz North Conductance Estimates, SM51B-0517, Fall AGU 2003.

Temporal MiningA typical substorm bulge

Geographic Projection Magnetic Projection

Substorms characterizedby poleward bulge on thenightside and increased intensity.

Part of a regular growthand decay cycle with atime constant ~90 minutes.

Goal: Identify substormmorphology in single images, then mine forwardand backward in time toidentify temporal and spatial characteristics ofsubstorm.

Goal: Add this informationto the searchable database.

Substorm development

Multi-yearAnalysis

First complete synoptic view of full mission.

Anomalies found near fall equinox.

Daily MeanDaily Median

Image Contamination

Rad’n Belts Rad’n

Belts

LightLeak

LightLeak

Image Contamination (cont’d)

(Automated) Quality Determination

Open Items• Automated quality flags

• Temporal evolution, e.g. substorms

• Dynamic image segmentation algorithms

• Identify and resolve data contamination issues

• FOV corrections

• Higher temporal resolution

• Updated OMNI and indices

• Port IDL routines to C

• Extend database to IMAGE data

Acknowledgements• Work funded by NASA AISRP Grant NAG5-10743 to the University

of Alabama in Huntsville

• Subcontract to Southern Polytechnic State University, Marietta GA

• Special thanks to our Tiger Team of programmers!

Chakravarthy Deverapalli Lee AdarrVenkataraman Gopalakrishnan Haining YaoSudhakaran Dharmaraj Kiran PaiRuchi Tyagi Amirali JaliaaNuo Nuo Xu Vishal AgrawalWes Swift