lecture 3 with every passing hour our solar system comes forty-three thousand miles closer to...
DESCRIPTION
Apache Point Observatory, Sunspot, New Mexico Apache Point Observatory 2.5m main survey telescope 0.5m photometric telescope 3.5m telescope (not used by SDSS) not a telescopeTRANSCRIPT
![Page 1: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/1.jpg)
Lecture 3
With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still there are some misfits who continue to insist that there is no such thing as progress. - Ransom K. Ferm
![Page 2: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/2.jpg)
Agenda
Homework 1 Questions? SDSS Lecture Study Questions EOSDIS Demo
![Page 3: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/3.jpg)
Apache Point Observatory, Sunspot, New Mexico
Apache Point Observatory2.5m main survey telescope
0.5m photometric telescope
3.5m telescope (not used by SDSS)
not a telescope
![Page 4: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/4.jpg)
Coarse Data Flow
![Page 5: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/5.jpg)
Detailed Data Flow
Data Acquisition
Data Processing (Fermilab)
Data Distribution
![Page 6: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/6.jpg)
Data Acquisition
![Page 7: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/7.jpg)
Good focus area ~ 30 full moons
Camera
Spectographs
Data Acquisition
![Page 8: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/8.jpg)
Data Acquisition: 2D Images
30 charge-coupled devices (CCDs)
Each has 4 million pixels Each night:
200 gigabytes of data on a dozen tapes
![Page 9: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/9.jpg)
Data Acquisition
![Page 10: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/10.jpg)
Data Acquisition: Spectra
![Page 11: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/11.jpg)
Data Acquisition: Spectra
![Page 12: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/12.jpg)
Spectra
Source: National Optical Astronomy Observatory
Sun Spectra with absorption lines
![Page 13: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/13.jpg)
Data Processing
![Page 14: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/14.jpg)
Data Processing
scanline strip = 6 scanlines stripe = 2 strips, offset frame (per CCD)
2048 x 1489 pixels 10% overlap
field = frames in all 5 filters
![Page 15: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/15.jpg)
Data Processing: Images
![Page 16: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/16.jpg)
Data Processing: Spectra
2D 3Dredshift = distance
ClassificationGalaxy or Star?
WavelengthsWhat substances
are involved?
![Page 17: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/17.jpg)
Data Processing: Spectra
![Page 18: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/18.jpg)
Data Processing: Spectra
![Page 19: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/19.jpg)
Data Distribution
![Page 20: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/20.jpg)
Data Distribution: Science Database
SpecObj
Telescope Configuration
Admin
PhotoObj
![Page 21: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/21.jpg)
Data Distribution: Science Database 200 million objects (photos, spectra,
etc.) Numerical attributes in a 100+
dimensional space Challenge: how can a relational
database scale to large volume of data?
![Page 22: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/22.jpg)
Improving Scalability SDSS data too large for one disk or one server Base-data objects spatially partitioned across
servers High-traffic data replicated Parallel and distributed query system Scan machine – continuously scans dataset and
evaluate user defined predicates (partitioned across multiple nodes)
Hash machine – performs comparisons within data clusters
![Page 23: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/23.jpg)
Overview of SDSS Schema
SDSS schema browser: http://cas.sdss.org/dr4/en/help/browser/browser.asp
PhotoObjAll – record describing all attributes of each photometric object
100s of columns Millions of photos Need good indexing/materialized views
![Page 24: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/24.jpg)
SDSS Schema (continued) PhotoObjAll table has many views:
PhotoObj- all primary and secondary objectsPhotoPrimary- all primary photo objects (best)
• Star• Galaxy• Sky• Unknown
PhotoSecondaryPhotoFamily (neither primary nor secondary)
Each view is Horizontal Partition (subset of rows)
![Page 25: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/25.jpg)
Other views
PhotoTag – Vertical partition of the PhotoObjAll table (subset of the columns)
Contains only columns that are most often requested (60 columns, 10% of PhotoObjAll)
Since rows are smaller (fewer columns), more rows can be loaded into memory and performance improves
![Page 26: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/26.jpg)
Indexes Hierarchical Triangular Mesh (HTM)
Spatially decomposes region of sky covered by SDSS data Enables faster spatial searches
Database indexes Primary key index –primary key of the table Foreign key index -primary key of another table Covering index – index covering one or more columns of a
table• Speeds up searches if any of the fields included in WHERE clause
mode, cy, cx, cz, htmID, type, flags, status, ra, dec, u, g, r, i, z, rhohtmID, cx, cy, cz, type, mode, flags, status, ra, dec, u, g, r, i, z, rhorun, camcol, type, mode, cx, cy, cz
![Page 27: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/27.jpg)
SDSS Database Indexes
PhotoObj and PhotoTag both indexed2% subset of PhotoObj
50x faster than reading whole PhotoObj table
5x faster than reading whole PhotoTag table
![Page 28: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/28.jpg)
Database Size for DR1 (GB)Filegroups BESTDR1 TARGDR1
data 1 200PhotoOther 18.1PhotoObjAll 165.4PhotoTag 78.1 73.7
PhotoTagIndex 53.6PhotoObjIndex 66.3PhotoObjProfile 80PhotoObjMask 22 17.2
SpecObj 6Neighbors 24.2
Frame 30 30Log 4.2 2Total 495.3 322.9
![Page 29: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/29.jpg)
Data Distribution
CASJobsFor long running queries
Personal Sky Server1% of total datapackaged for one-click installeducation, testing, demonstrations
Web servicesfor specific functions
![Page 30: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/30.jpg)
Data Distribution: Releases
![Page 31: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/31.jpg)
Data Distribution: Releases
![Page 32: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/32.jpg)
Study Questions
![Page 33: Lecture 3 With every passing hour our solar system comes forty-three thousand miles closer to globular cluster 13 in the constellation Hercules, and still](https://reader033.vdocument.in/reader033/viewer/2022052608/5a4d1b877f8b9ab0599bd2b4/html5/thumbnails/33.jpg)