arizona-noao temporal analysis and response to events...

Arizona-NOAO Temporal Analysis and Response to Events System

(ANTARES) Tom Matheson

National Optical Astronomy Observatory

Users Committee, NOAO, May 2015 (D2) 1

ANTARES What It Is and Why We Need It

•  ANTARES is an astronomical time-domain event broker •  A broker is an intermediary who sits between the source

of the alerts and the consumers of the alerts while adding value

•  LSST will compare each image with a reference and issue an alert for any 5σ change (brightness or position)

•  LSST will generate 1-10 million alerts each night for the ten-year duration of the survey

•  We need a system to rapidly winnow that number down to something commensurate with follow-up resources

•  Rate and volume will be unprecedented


ANTARES Need for a Broker

•  One of the primary recommendations from the ‘Spectroscopy in the Era of LSST’ workshop


3.1 Software Infrastructure Although time domain science is the primary beneficiary of a broker to sort through the alerts generated by changes in brightness (or position), the cosmological use of SNe Ia, the study of AGNs , and the study of brown dwarf weather all will rely on the ability to recognize interesting events when they happen. For each case, the signal hidden in the noise of one million alerts per night will be different, but they will all require a system to winnow alerts done to the ones they will find of interest. Hidden among those million alerts will be rare and interesting objects, and these will be lost without a working broker. Matheson et al. 2013 (astro-ph 1311.2496)

ANTARES Need for a Broker

•  One of the primary recommendations from the NRC report “Optimizing the U.S. Ground-Based Optical and Infrared Astronomy System” (Elmegreen report)


Recommendation 4a: The National Science Foundation should help to support the development of event brokers, which should use standard formats and protocols, to maximize LSST transient survey follow-up work.

ANTARES Building on Other Brokers

•  Astronomical brokers –  Humans –  Skyalert (http://skyalert.org/) –  PTF (Bloom et al. 2012) –  Requires time and information to sort alerts

•  LSST problem is one of scale and rate –  Multidisciplinary with computer science –  Collaboration between NOAO and UA Computer

Science –  NSF INSPIRE proposal was successful (IIA-1344024)


ANTARES How it Fits in the Transient Ecosystem

•  Alert generators find transients and assess validity

•  ANTARES characterizes, annotates, and ranks

•  Downstream brokers/users classify and follow up


ANTARESAnnotation

CharacterizationRanking

Distribution

CRTS LSSTLIGO ZTF

AlternateBrokersAlternate

BrokersAlternateBrokers

Downstream BrokersDownstream

Brokers

Downstream Brokers

Downstream Brokers

ClassificationDistribution

Robotic Telescopes

Investigators

ANTARES Environment

Potential Aggregator of Alerts from Multiple Sources

ALERT GENERATORS: Difference Imaging, Real/Bogus & Moving Object Assessment

Manually Triggered Followup Observing

System Data Flow Info to/from Investigators Feedback

ANTARES Overall Architecture

•  Core flow for alerts •  Draws on local databases for

annotation •  Various stages of filtering •  No alerts are lost, just diverted to

other channels •  Prototype focuses on ‘rarest of

the rare’ •  Open source/open access


LSST

Initial Ingest

ExistingCatalogs

1,000 images / night

Thumbnails Updated

CalibrationDatadaily

Camera AlertsUp to 10,000 alerts / image

quarterly

Measurement Recalibration

AstroObject & Alert

Association

AstroObject-Annotated Alerts

Feature Evaluation

Extended Alerts

Alert Characterization

Characterized Alerts

Initial Filtering

Level I Alerts

Advanced Feature Evaluation, Characterization and Filtering

Level II Alerts

1,000 / image

100 / image

1.0 / image

Taxonomy of Characterizations

(curated)

filteredalerts

filteredalerts

FilteringTrees

(curated)

CharacterizationAlgorithms(curated)

Locus-AggregatedAnnotated Alerts

PeriodicQueries

DeriveFeatures

New AstroObjectIdentification

PositionEstimation

MovingObjects(curated)

MovingObject

Identification

Feature Algorithms(curated)

LocusAggregation

ANTARES

Theoretical Predictions

AlgorithmVetting

Professionals,citizen scientists

forking

forking

start at imageacquisition

Other External Alerts

External Alert Ingestion

Theorists

Touchstone(curated)

Alert Throttling

Thumbnail Separation

Combo Assembly

Novel Alerts0.1 / image

"Rarest of the Rare"

Annotated Coalesced

Alerts

filtered andnovel alerts

External Alert

Brokers

Researchersand public

Ultra Characterization & Filtering

30 Dec 2014

Alert Replica Coalescing

Thumbnail Reattachment

black = usgreen = not usblue = catalog by-productsred = alert flow

Annotated Coalesced Alert

Rarest Alerts

AggregatedAstroObject Catalogs

(curated)Aggregated

AstroObject Catalogs(curated)Aggregated

AstroObject Catalogs(curated)

forking

Combo ReplicasSelected Alert Replicas

Combo Feature EvaluationAdvanced Feature Evaluation

Combo Coalescing

ANTARES Annotation Architecture

•  Added value is critical •  Multiwavelength catalogs,

including LSST itself •  Past history of alerts •  Other alert sources (e.g.,

LIGO) •  Theoretical predictions


Databases of astronomical information must be local

ANTARES Filtering Architecture

•  Derive features from alert and annotation data

•  Use features to characterize (not classify) objects

•  Alerts can be forked if more than one possibility

•  Filter based on features/characterizations

•  As alert numbers drop, more complex processing can occur


ANTARES: Progress I

•  NSF grant funded Sep 2013 (INSPIRE IIA-1344024) •  Weekly meetings to manage development •  Astronomy postdoc hired Sep 2014 •  3 CS grad students started Sep 2014

–  Faster algorithm testing –  Data and algorithm provenance control –  Parallel processing experimentation –  API developed –  Documentation


ANTARES: Progress II

•  Databases for annotation developed •  Can predict Galactic stellar variability •  Demonstration prototype operational

–  Utilizes major stages of architecture –  Modular components can become more complex


ANTARES Making Decisions When You Know Nothing

•  Some alerts will have no contextual information for annotation

•  Kepler provides empirical variability model for Galactic stars

•  We can assess variability for 155000+ stars over a wide range of spectral types

•  Biased sample, but still the best source


12000 10000 8000 6000 4000

65

43

21

0

T eff

log

g

Kepler Sample

T eff

log

g


•  Distribution of variability for Kepler sample in Q13

•  Bright excursions mainly M star flares

•  Faint excursions mainly eclipsing binaries


−1 0 1 2 3

110

010

000

1000

000

1000

0000

0

Full Kepler Sample

∆mag

Freq

uenc

y

Min. −1.105 Max. 3.335

0.005% 0.05% 99.95% 99.995%−0.276 −0.108 0.152 0.429


•  As an example, for the LSST reference field (RA 0 DEC 0, l 96 b -60), Besançon model provides distribution of stellar types

•  Scale Kepler sample to match Besançon distribution

•  Use Δmag of Galactic source to estimate rarity

•  Will develop Solar System model for similar approach


−1 0 1 2 3

110

010

000

1000

000

1000

0000

0

l96b−60.dat

∆mag

Freq

uenc

y

Min. −1.103 Max. 2.106

0.05% 99.95%−0.080 0.0830.005% 99.995%−0.177 0.3370.0005% 99.9995%−0.548 0.7720.00005% 99.99995%−1.105 1.027

ANTARES: Next Steps

•  Run more complex test alerts through system this 2015 •  Fully populate annotation databases •  Functional prototype in 2016 •  Address scaling through 2016 •  Run on live alert streams in 2017 •  Next proposal will be to make ANTARES generic to

handle the full breadth of user interests •  During LSST operations, will likely include NCSA

collaboration •  Visualization of ANTARES processes


ANTARES: Demo


LSST

Initial Ingest

ExistingCatalogs

1,000 images / night

Thumbnails Updated

CalibrationDatadaily

Camera AlertsUp to 10,000 alerts / image

quarterly

Measurement Recalibration

AstroObject & Alert

Association

AstroObject-Annotated Alerts

Feature Evaluation

Extended Alerts

Alert Characterization

Characterized Alerts

Initial Filtering

Level I Alerts

Advanced Feature Evaluation, Characterization and Filtering

Level II Alerts

1,000 / image

100 / image

1.0 / image

Taxonomy of Characterizations

(curated)

filteredalerts

filteredalerts

FilteringTrees

(curated)

CharacterizationAlgorithms(curated)

Locus-AggregatedAnnotated Alerts

PeriodicQueries

DeriveFeatures

New AstroObjectIdentification

PositionEstimation

MovingObjects(curated)

MovingObject

Identification

Feature Algorithms(curated)

LocusAggregation

ANTARES

Theoretical Predictions

AlgorithmVetting

Professionals,citizen scientists

forking

forking

start at imageacquisition

Other External Alerts

External Alert Ingestion

Theorists

Touchstone(curated)

Alert Throttling

Thumbnail Separation

Combo Assembly

Novel Alerts0.1 / image

"Rarest of the Rare"

Annotated Coalesced

Alerts

filtered andnovel alerts

External Alert

Brokers

Researchersand public

Ultra Characterization & Filtering

30 Dec 2014

Alert Replica Coalescing

Thumbnail Reattachment

black = usgreen = not usblue = catalog by-productsred = alert flow

Annotated Coalesced Alert

Rarest Alerts

AggregatedAstroObject Catalogs

(curated)Aggregated

AstroObject Catalogs(curated)Aggregated

AstroObject Catalogs(curated)

forking

Combo ReplicasSelected Alert Replicas

Combo Feature EvaluationAdvanced Feature Evaluation

Combo CoalescingAssociater Load Balancer

Alert Worker 1

Controller

alerts

Antares Load Balancing Diagram

alert status start or send command

send worker

status

send com

mand

Locus Aggregated

DB

LSST Throttler

Stages Configuration

(YAML)

Cached Status Table

for Alerts

maintain

Conductor(person)

Alert Worker Node

Master Node

Alert Worker n

initialize worker

ready(ask Controller)

load new or existing alertrunning

a stage

flush alert data to DB

abort alert worker

complete stage (including divert or fork)

switch to alertresumealert

ready for resume or switch to a different alert

Alert Worker Status Transition Chartstart worker

start alert

camera alerts

provenancemaintenance

Algorithms

Puppet

Github

Deploy codes for worker nodes

Night (fast) Daytime (slow)

reference

machine configuration

stage completed

Camera Alert

Constructor

AstroObject DB

commit

Creating replica or combo and loading data

Background Slides


ANTARES: Personnel

•  Tom Matheson (NOAO) •  Abhijit Saha (NOAO) •  Richard Snodgrass (UA Computer Science; PI of NSF

proposal; database expert) •  John Kececioglu (UA Computer Science; algorithm expert) •  Gautham Narayan (NOAO/UA Postdoc) •  Three UA CS grad students •  Volunteers

–  Tim Axelrod (UA Astronomy) –  Rob Seaman (NOAO) –  Tim Jenness (LSST)


ANTARES: Extant Broker Examples


ANTARES: ER Diagram


ANTARES: API


arizona-noao temporal analysis and response to events...

Documents