2008 nvo summer school11 scientific data mining in astronomy kirk d. borne george mason university...

2008 NVO Summer School 11

Scientific Data Mining in Astronomy

Kirk D. BorneGeorge Mason University

[email protected] http://classweb.gmu.edu/kborne/

THE US NATIONAL VIRTUAL OBSERVATORY


OUTLINE

• Scientific Databases• Some key astronomy problems• Astronomy Data Mining examples• Suggested Reading• Some Data Mining Software• Summary


OUTLINE



10 Unique Features of Scientific Data

• Each of these characteristics requires special handling beyond what you read in standard data mining textbooks:

1. Scientific data depend on experimental equipment and conditions.2. Scientific data have noise.3. Scientific data have been (or need to be) calibrated.4. Scientific units on data values are imperative.5. Scientific databases often contain associated columns: { value, error }.6. Scientific data values are often non-linear (log values, magnitudes, asinh).7. History of scientific data creation, processing, and versioning is critical =

Provenance.8. Metadata, Metadata, Metadata = tells us “who, what, when, where, how”.

NOTE: Semantic Metadata are becoming more important = “why”.9. Context is critical (e.g., brightness in an optical catalog is expressed in mags,

but expressed in counts/sec in an X-ray catalog, or milli-Jansky in a radio catalog).

10. Scientific data have different levels of abstraction: raw, calibrated, reduced data products, derived information, extracted knowledge, published results.

• All of this makes the “Data Preparation” phase of any scientific data mining experiment even more critical and essential.


OUTLINE



Some key astronomy problems

• Some key astronomy problems that can be addressed with data mining techniques:

• Cross-Match objects from different catalogues• The distance problem (e.g., Photometric Redshift estimators)• Star-Galaxy Separation• Cosmic-Ray Detection in images• Supernova Detection and Classification• Morphological Classification (galaxies, AGN, gravitational lenses, ...)• Class and Subclass Discovery (brown dwarfs, methane dwarfs, ...)• Dimension Reduction = Correlation Discovery• Learning Rules for improved classifiers • Classification of massive data streams• Real-time Classification of Astronomical Events • Clustering of massive data collections• Novelty, Anomaly, Outlier Detection in massive databases


OUTLINE



Classification Methods:Decision Trees, Neural Networks, SVM (Support Vector Machines)

There are 2 Classes!

How do you ...-Separate them?-Distinguish them?-Learn the rules?-Classify them?

ApplyKernel

(SVM)


Decision Tree Classification Example: SKICAT Star-Galaxy DiscriminationReference: ftp://iraf.noao.edu/iraf/conf/web/adass_proc/adass_95/yooj/yooj.html


Decision Tree Classification Example: Classification of candidates for new supernova in galaxiesReference: http://spiff.rit.edu/richmond/sdss/sn_survey/scan_manual/sn_scan.html


Clustering is used to discover the different unique groupings (classes) of attribute values.The case shown below is not obvious: one or two groups?


This case is easier: there are two groups.(in fact, this is the same set of data elements as shown on the previous slide, but plotted here using a different attribute.)


Clustering in multiple dimensions: colors combined from SDSS & 2MASS magnitudes

2008 NVO Summer School

Clustering: Class Discovery and Rule Learning

• Clusters and the separation of classes depend on which attributes (dimensions) are chosen to be projected, as in the following star-galaxy discrimination test:

14

• Reference: http://www.cs.princeton.edu/courses/archive/spr04/cos598B/bib/BrunnerDPS.pdf

Not good Good


Semisupervised Learning:Outlier Detection

• Reference: http://www.cs.princeton.edu/courses/archive/spr04/cos598B/bib/BrunnerDPS.pdf

15

A demonstration of a generic machine-assisted discovery problem — data mapping and a search for outliers.

This schematic illustration is of the clustering problem in a parameter space given by three object attributes: P1, P2, and P3.

In this example, most of the data points are assumed to be contained in three, dominant clusters (DC1, DC2, and DC3).

However, one may want to discover less populated clusters (e.g., small groups or even isolated points), some of which may be too sparsely populated, or lie too close to one of the major data clouds.

In some cases, negative clusters (holes), may exist in one of the major data clusters.


Outlier Detection: Serendipitous Discovery of Rare or New Objects & Events


Principal Components Analysis &Independent Components Analysis

Cepheid Variables:Cosmic Yardsticks-- One Correlation-- Two Classes!

... Class Discovery!


Example: SOM (Self-Organizing Map)

• The SOM (Self-Organizing Map) is one technique for organizing information in a database based upon links between concepts.

• It can be used to find hidden relationships and patterns in more complex data collections, usually based on links between keywords or metadata.


Mega-Flares on normalSun-like stars = a star like our Sun increased in brightness 300X one

night!… say what??

Exploringthe Time Domain

Astronomy Data Mining in Action


Example: The Thinking Telescope

Sample Data Mining Applications: (credit: http://www.thinkingtelescopes.lanl.gov/ )Automated Feature Extraction: Real-time identification of artifacts and transients in direct and difference images.Classifiers: Automated classification of celestial objects based on temporal and spectral properties.Anomaly Detection: Real-time recognition of important deviations from normal behavior for persistent sources.


From Sensors to Sense

From Data to Knowledge:from sensors to sense

From Data to Knowledge:from sensors to sense

Data Information Knowledge


VOEventNet

Event Synthesis Engine

Pairitel

Palomar 60”

Raptor

PQ next-daypipelines

catalog

Palomar-Quest

knownVariables

knownasteroids

SDSS2MASS

PQ Event Factory

remote archives

baselinesky

eStar

VOEventNet

VOEventNet: a Rapid-Response Telescope Grid GRBsatellites

VOEventdatabase

Reference: http://voeventnet.caltech.edu/


Learning From Archived Temporal Data (Time Series):Classify New Data (Bayes Analysis or Markov Modeling)


Photometric-Redshift Estimation

Photometric vs. Spectroscopic Redshift Estimates:• Left panel: standard technique• Right panel: Machine Learning (data mining) application• Reference: http://arxiv.org/abs/0710.4482


Star-Galaxy Separation in Clustered Feature Space

* = star• = galaxy

http://arxiv.org/abs/astro-ph/9508012


Bayesian Probabilistic Estimationfor Catalog Cross-Matching

• Reference: http://arxiv.org/abs/astro-ph/0605216


Fundamental Plane for 156,000 cross-matched Sloan+2MASS Elliptical Galaxies: plot shows variance captured by first 2 Principal Components as a function of local galaxy density.

• Slide Content• Slide content• Slide content• Slide content

low (Local Galaxy Density) high

% o

f va

rian

ce c

ap

ture

d b

y P

C1+

PC

2Reference: Borne, Dutta, Giannella, Kargupta, & Griffin 2008


OUTLINE



Suggested Reading: Data Mining in Astronomy• Djorgovski et al. 2000, Searches for Rare and New Types of Objects.

http://arxiv.org/abs/astro-ph/0012453 • Djorgovski et al. 2000, Exploration of Large Digital Sky Surveys.

http://arxiv.org/abs/astro-ph/0012489 • Djorgovski et al. 2001, Exploration of Parameter Spaces in a Virtual Observatory.

http://arxiv.org/abs/astro-ph/0108346 • Mining the Sky, 2001, published proceedings of ESO conference.• Suchkov et al. 2003, Automated Object Classification with ClassX.

astro-ph/0210407

• Suchkov, Hanisch, & Margon 2005, A Census of Object Types and Redshift Estimates in the SDSS Photometric Catalog from a Trained Decision Tree Classifier. http://adsabs.harvard.edu/abs/2005AJ....130.2439S

• Giannella et al. 2006, Distributed Data Mining for Astronomy Catalogs. http://www.cs.umbc.edu/~hillol/PUBS/Papers/Astro.pdf

• Rohde et al. 2006, Matching of Catalogues by Probabilistic Pattern Classification. http://adsabs.harvard.edu/abs/2006MNRAS.369....2R

• Budavari & Szalay 2008, Probabilistic Cross-Identification of Astronomical Sources. http://adsabs.harvard.edu/abs/2008ApJ...679..301B


Suggested Reading, continued: Data Mining in Astronomy• Odewahn et al. 1993, Star-Galaxy Separation with a Neural Network. 2: Multiple

Schmidt Plate Fields. http://adsabs.harvard.edu/abs/1993PASP..105.1354O • Borne 2000, Science User Scenarios for a Virtual Observatory Design Reference

Mission: Science Requirements for Data Mining. astro-ph/0008307• Brunner et al. 2001, Massive Datasets in Astronomy. astro-ph/0106481• Gray et al. 2002, Data Mining the SDSS SkyServer Database.

http://arxiv.org/abs/cs/0202014 • Odewahn et al. 2004, The Digitized Second Palomar Observatory Sky Survey

(DPOSS). III. Star-Galaxy Separation. http://adsabs.harvard.edu/abs/2004AJ....128.3092O

• Ball, Brunner, et al. 2006, Robust Machine Learning Applied to Astronomical Data Sets. I. Star-Galaxy Classification of the Sloan Digital Sky Survey DR3 Using Decision Trees. http://adsabs.harvard.edu/abs/2006ApJ...650..497B

• Ball, Brunner, et al. 2007, Robust Machine Learning Applied to Astronomical Data Sets. II. Quantifying Photometric Redshifts for Quasars Using Instance-based Learning. http://adsabs.harvard.edu/abs/2007ApJ...663..774B

• Ball, Brunner, et al. 2008, Robust Machine Learning Applied to Astronomical Data Sets. III. Probabilistic Photometric Redshifts for Galaxies and Quasars in the SDSS and GALEX. http://adsabs.harvard.edu/abs/2008ApJ...683...12B


Suggested Reading, continued: Data Mining in Astronomy• Rogers & Riess 1994, Detection and Classification of CCD Defects with an

Artificial Neural Network. http://adsabs.harvard.edu/abs/1994PASP..106..532R • Feeney et al. 2005, Automated Detection of Classical Novae with Neural

Networks. http://adsabs.harvard.edu/abs/2005AJ....130...84F • Wadadekar 2005, Estimating Photometric Redshifts Using Support Vector

Machines. http://adsabs.harvard.edu/abs/2005PASP..117...79W • Bazell & Miller 2005, Class Discovery in Galaxy Classification.

http://adsabs.harvard.edu/abs/2005ApJ...618..723B • Bazell, Miller, & SubbaRao 2006, Objective Subclass Determination of Sloan

Digital Sky Survey Spectroscopically Unclassified Objects. http://adsabs.harvard.edu/abs/2006ApJ...649..678B

• Ferreras et al. 2006, A Principal Component Analysis approach to the Star Formation History of Elliptical Galaxies in Compact Groups. http://adsabs.harvard.edu/abs/2006MNRAS.370..828F

• Way & Srivastava 2006, Novel Methods for Predicting Photometric Redshifts from Broadband Photometry Using Virtual Sensors. http://adsabs.harvard.edu/abs/2006ApJ...647..102W

• Carliles et al. 2007, Photometric Redshift Estimation on SDSS Data Using Random Forests. http://arxiv.org/abs/0711.2477


OUTLINE



Some Data Mining Software & Projects

• General data mining software packages:– Weka (Java): http://www.cs.waikato.ac.nz/ml/weka/ – Weka4WS (Grid-enabled): http://grid.deis.unical.it/weka4ws/ – RapidMiner: http://www.rapidminer.com/

• Astronomy-specific software and/or user clients:• VO-Neural: http://voneural.na.infn.it/• AstroWeka: http://astroweka.sourceforge.net/• OpenSkyQuery: http://www.openskyquery.net/ • ALADIN: http://aladin.u-strasbg.fr/ • MIRAGE: http://cm.bell-labs.com/who/tkh/mirage/ • AstroBox: http://services.china-vo.org/

• Astronomical and/or Scientific Data Mining Projects:• GRIST: http://grist.caltech.edu/• ClassX: http://heasarc.gsfc.nasa.gov/classx/ • LCDM: http://dposs.ncsa.uiuc.edu/ • F-MASS: http://www.itsc.uah.edu/f-mass/ • NCDM: http://www.ncdm.uic.edu/


Weka:http://www.cs.waikato.ac.nz/ml/weka/

• Weka is in your NVOSS software distribution.• Weka is a collection of open source machine learning algorithms for

data mining tasks. • Weka algorithms can either be applied directly to a dataset or called

from your own Java code. • Weka comes with its own GUI.• Weka contains tools for data pre-processing, classification,

regression, clustering, association rules, and visualization.


AstroWeka:http://astroweka.sourceforge.net/

http://www.iterating.com/products/Wekahttp://weka.sourceforge.net/wekadoc/index.php/en:Knowledge_Flow_

%283.4.10%29


ALADIN: http://aladin.u-strasbg.fr/

36


MIRAGE:http://cm.bell-labs.com/who/tkh/mirage/

Java Package for exploratory data analysis (EDA), correlation mining, and interactive pattern discovery.

37


OUTLINE



Science is Knowledge Work

• Knowledge Discovery is the central theme of science.

• Knowledge Discovery in Databases (KDD) is the killer app for large scientific databases.

• Therefore, KDD (i.e., Data Mining) is an essential tool, since “big-data” science is here to stay (at petabytes and beyond).

Data Information Knowledge


Scientific Knowledge Discovery


Heliophysics Space Weather Example


Sun-Earth Space Environment – Rich Source of Heliophysical Phenomena


Multi-point Observations and Models of Space Plasmas Deliver a Deluge of Physical Measurements


Heliophysics Space Weather Example

CME = Coronal Mass EjectionSEP = Solar Energetic Particle


Data Mining:It is more than just connecting the dots

Reference: http://homepage.interaccess.com/~purcellm/lcas/Cartoons/cartoons.htm


Sample Astronomy Data Mining ApplicationIdeas for your Projects

– Neural Network for Pixel Classification: Event Detection and Prediction (e.g., Supernova or Cosmic-ray hit?)

– Bayesian Network for Object Classification (star or galaxy?)

– PCA for finding Fundamental Planes of Galaxy Parameters

– PCA (weakest component) for Outlier Detection: anomalies, novel discoveries, new objects

– Link Analysis (Association Mining) for Causal Event Detection (e.g., linking optical transients with gamma-ray events)

– Clustering analysis: Spatial, Temporal, or any scientific database parameters

– Markov models: Temporal mining, classification, and prediction from time series data

2008 nvo summer school11 scientific data mining in astronomy kirk d. borne george mason university...

Documents

scientific data values

data points

data mining techniques

nvo summer school44

nvo summer school66

nvo summer school12

standard data mining

reduced data products