compsci516 database systems - duke university•from “big data” wiki: –healthcare:...

CompSci 516DatabaseSystems

Lecture1Introduction

andDataModels

Instructor:Sudeepa Roy

1DukeCS,Fall2017 CompSci516:DatabaseSystems

CourseWebsite

• http://www.cs.duke.edu/courses/fall17/compsci516/

• Pleasecheckfrequentlyforupdates!

• NewRoom:LSRCD106

DukeCS,Fall2017 CompSci516:DatabaseSystems 2

Instructor• Sudeepa Roy

– sudeepa@cs.duke.edu– https://users.cs.duke.edu/~sudeepa/– officehour:Mondays11:30am-12:30pm,LSRCD325

• Aboutmyself– AssistantProfessorinCS– PhD:UPenn,Postdoc:Univ.ofWashington– JoinedDukeCSinFall2015– Researchinterests:

• Databases(theoryandapplications)• DataAnalysis,causality,explaininganswers• Uncertaindata,dataprovenance,crowdsourcing

Two(half-)TAs

• YilinGao– yilin.gao@duke.edu– officehour:Wed,3-4pm,Location:TBD

• Keping Wang– keping.wang@duke.edu– officehour:Thurs,3-4pm,Location:TBD

• BothCompSci 516veterans!4DukeCS,Fall2017 CompSci516:DatabaseSystems

Logistics

• Homeworksubmission:Sakai– Allenrolledstudentsarealreadythere

• Discussionforum:Piazza– Allenrolledstudentsarealreadythere– SendmeanemailifyouhavenotreceivedawelcomeemailfromPiazza

• Lectureslideswillbeuploadedbeforetheclass– butwillbeupdatedaftertheclass

Grading

• ThreeHomework:30%• Project:15%• TwoMidterms:25+25=50%• Classparticipation:5%

GradingStrategy• Relativegrading

– Theactualgradedistributionattheendwilldependontheperformanceoftheentireclassonallthecomponents.

– TopperoftheclassgetsA+irrespectiveofthenumber,andonly“aboveexpectation”performancesgetA+.

– Nofixedlowestgradeorgradedistribution.– SEveryone cangetgoodgradebyworkinghard!

Homework• Duein2-3weeksaftertheyareposted/previoushw isdue

– ALWAYSstartearly!

• Nolatedays– contacttheinstructorifyouhavea*valid*reasontobelate– Anotherexam,project,hw isNOTavalidreason– wewillalwaysbe

fairtoall– Computercrash/suddeninterviewtrips/medicalissues(following

officialprocedures)maycountasvalidreasons– Noguaranteethatyourrequestwillbegranted– again,startearly!

• Tobedoneindividually

HomeworkOverview• Youwilllearnhowtousetraditionalandnewdatabase

systemsinthehomework– Havetolearnthemmostlyonyourownfollowingtutorialsavailable

onlineandwithsomehelpfromtheTA

• HW1:TraditionalDBMS– SQLandPostgres

• HW2:Distributeddataprocessing– SparkandAWS

• HW3:NOSQL– e.g.MongoDB

Exams• Midterm-1– Oct11(Wed)• Midterm-2– Nov29(Wed)

• Inclass• Closedbook,closednotes,noelectronicdevices• Totalweight:25+25%=50%• Examswilltestyourunderstandingofthematerial

• Bothexamsarecomprehensive– wouldincludeeverylectureuptothemidterm

Projects• 15%weight• Ingroupsof3-4

– YoucanlookforgroupmembersthroughPiazzabyannouncingyourgeneralareaofinterestorifyouhaveaprobleminmind

– Eachgroupmembershoulddoapprox.equalwork

• Showyourcreativityandresearcher-side!• Workdoneshouldbeatleastequivalentto

– ahw *no.ofgroupmembers

• Allgroupmemberswillgetthesamegrade

ProjectTopics• Anythingrelatedto“Data”

– Datamanagement/processing/cleaning– Datavisualization– Dataexplorationoranalysis– Applicationsofdata(toanyfield)– Theoreticalfindingswithdata– Newtoolfordataanalysis

• Chooseaprojectaccordingtoyourresearchinterest• Youcancheckoutmajordatabaseconferencesforideas,e.g.

– Demonstrations (buildaprototypesolvingaproblemorimprovingUI)• SIGMOD’17:http://sigmod2017.org/sigmod-program/#posters• SIGMOD’16:http://sigmod2016.org/sigmod_demo_list.shtml• VLDB’17:http://www.vldb.org/2017/accepted_papers_demo_track.php• VLDB’16:http://vldb2016.persistent.com/demonstrations.php

– Researchpapers(solveaproblem,doexperimentswithdata)• CheckoutpapersinSIGMODandVLDBfromrecentyears

– Youcancheckoutpreviousyearstoo,andconferencesfromyourownresearcharea

ProjectDeliverables1. Projectproposal(due:9/20(W),1-3pages)

– problemselectionispartoftheproject– 3weeksfromnow– butstartasap,lookforproblems,dorelatedworkstudy,findan

interestingquestion,letmeknowyourinitialthoughts,allbythedeadline

2. Midtermprogressreport(due:10/25(W),3-5pages)3. Finalprojectreport(due:11/30(Th),4-8pages)4. Afinal5-10minsprojectpresentationand/ordemonstration

(inthelast1-2classes)

ProjectEvaluationCriteriaScaleof100:1. Well-motivated?102. Novel?103. Comprehensiverelatedworksurvey?104. Qualityofwriting?10

– shouldreflectallotherfactorstooexceptclasspresentation

5. Classpresentation/demo?15– shouldreflectallotherfactorstooexceptwriting

6. Technicalcontributions?45– Problemformulation/Algorithms/Experiments/Theory/System/

Userinterface/Efficiency/Usability/Datasetexplorationetc.

ClassParticipation• 5%weight• Includes

– Participationinclass(Q/A)– Pop-upquiz(youwillgettokenbyemailtoenrollin“gradiance”)

• Participation+correctanswering(lowesttwoscoreswillbedropped)– Evaluatingothers’projectsduringtheprojectpresentation

Ingeneral,• Activelyparticipateintheclass!

– Askquestionsinclassandonpiazza– Stopmeasmanytimesasyouneedtounderstandthelectures– Answereachother’squestionsonpiazza

• Alsosend(anonymousornot)feedback,suggestions,orconcernsonPiazza– thereisa“feedback”folder

ReadingMaterial

• Willmostlyfollowthe”cowbook”byRamakrishnan-Gehrke– Thechapternumberswillbeposted

• Youdonothavetobuythebooks,butitwillbegoodtoconsultthemfromtimetotime

• Youshouldbepreparedtodoquiteabitofreadingfromvariousbooksandpapers

Whatisthiscourseabout?

• Thisisagraduate-leveldatabasecourseinCS

• Wewillcoverprinciples,internals,andapplicationsofdatabasesystemsindepth

• Wewillalsohaveanintroductiontoafewadvancedresearchtopicsindatabases(laterinthecourse)

AQuickSurvey• Haveyoutakenanundergraddatabasecourseearlier

– CS316/equivalent?

• Areyoufamiliarwith– SQL?– RA?(σ, Π, ´, ⨝, r, È, Ç, -)– Keys, foreign keys?– Indexindatabases?– Logic:∧,∨,∀,∃,¬,∈, =>

– Transactions?– Map-reduce/Spark?

• Haveyoueverworkedwithadataset?– relationaldatabase,text,csv,XML

• Haveyoueverusedadatabasesystem?– PostGres,MySQL,SQLServer,SQLAzure

Whatwillbecovered?• Databaseconcepts

– DataModels,SQL,Views,Constraints,RA,Normalization

• Principlesandinternalsofdatabasemanagementsystems(DBMS)– Indexing,QueryExecution-Algorithms-Optimization,Transactions,

ParallelandDistributedQueryProcessing,MapReduce

• Advancedandresearchtopicsindatabases– e.g.Datalog,NOSQL,Datamining,Datawarehouse– Morewillbeaddedinthe“TBD”lectures

• Wewillgofastforsomebasictopicsindatabasescoveredinundergraddbcourses– Datamodel,SQL,RA– Butaskmetoslowdownifyouarenotfamiliarwiththem

WhatthiscourseisNOTabout

• Spark,AWS,clustercomputing…– PartiallycoveredinaHWandalecture

• Machinelearningbasedanalytics• Statisticalmethodsfordataanalytics• Python,R,…• Programming

Background• Youshouldhavesomeunderstanding(attheCS

undergraduatelevel)– datastructure,discretemaths,algorithms– databases– orhavetolearntheseyourselfasnecessary

• Needtopickupnewcodingframeworkandprogramminglanguagesonyourown– andhowtoprocessdatausingthem– Homeworkassignmentswillmostlybeself-taught– …withhelpfromtheTA

• Willinvolvesomemathematicalandanalyticalreasoningtoo

Whyshouldwecareaboutdatabases?

• Weareinadata-drivenworld

• “BigData”issupposedtochangethemodeofoperationforalmosteverysinglefield– Science,Technology,Healthcare,Business,Manufacturing,Journalism,Government,Education,…

• Wemustknowhowtocollect,store,process,andanalyzesuchdata

• From“BigData”wiki:“TheLargeHadronColliderexperimentsrepresentabout150millionsensorsdeliveringdata40 milliontimespersecond.Therearenearly600 millioncollisionspersecond.IfallsensordatawererecordedinLHC,….thisisequivalentto500quintillion(5×1020)bytesperday,almost200timesmorethanalltheothersourcescombinedintheworld.”

Science

DukeCS,Fall2017 CompSci516:DatabaseSystems

• From“BigData”wiki:– eBay.com usestwodatawarehousesat7.5PB(x1012)and40PBaswellasa40PBHadoopclusterforsearch,consumerrecommendations,andmerchandising

– Facebookhandles50 billionphotosfromitsuserbase– AsofAugust2012,Googlewashandlingroughly100 billionsearchespermonth

Technology

• From“BigData”wiki:– Healthcare:digitizationofpatient’sdata,prescriptiveanalytics

– Media:Tailorarticlesandadvertisementsthatreachtargetedpeople,validateclaims

• “ComputationalJournalism”projectinDukeDBgroup

– Manufacturing:supplyplanning– Sports:improvetraining,understandingcompetitors

HealthcareMediaManufacturingSports…..

• Simplystoringsuchlargedatasetsinaflatfilestopsworkingatsomepoint– Needefficientmodel,storage,andprocessing

• ADBMStakescareofsuchissues– theuseronlyhastorunqueriestoprocesssuchdatasets– muchsimplerthanwritinglowlevelcode

• DBMS• DataModels

• [RG]1.1,1.3-1.5

WhatisaDatabase?

• Adatabaseisacollectionofdata– typicallyrelatedanddescribingactivitiesofanorganization

• Adatabasemaycontaininformationabout– Entities

• students,faculty,courses,classroom

– Relationshipsbetweenentities• students’enrollment,facultyteachingcourses,roomsforcourses

Andwhatdoesitcontain?

WhyuseaDBMS• i.e.whynotusefilesystemandaprogramminglanguage?

• Supposeacompanyhasalargecollectionofdataonemployees,departments,products,salesetc.

• Requirements:– Quicklyanswerquestionsondata

• Notethatallthedatamaynotfitinmainmemory– Concurrentaccess:applychangesconsistently– Restrictedaccess(e.g.salary)

WhyuseaDBMS?

• ADBMSisapieceofsoftware(i.e.abigprogramwrittenbysomeoneelse)thatmakesthesetaskseasier– Quickaccess– Robustaccess– Safeaccess– Simpleraccess

• Next:somenicepropertiesofaDBMS

WhyuseaDBMS?

1. DataIndependence– Applicationprogramsshouldnotbeexposedtothedata

representationandstorage– DBMSprovidesanabstractviewofthedata

2. EfficientDataAccess– ADBMSutilizesavarietyofsophisticatedtechniquesto

storeandretrievedata(fromdisk)efficiently

WhyuseaDBMS?

3. DataIntegrityandSecurity– DBMSenforces“integrityconstraints”– e.g.check

whethertotalsalaryislessthanthebudget– DBMSenforces“accesscontrols”– whethersalary

informationcanbeaccessesbyaparticularuser

4. DataAdministration– Centralizedprofessionaldataadministrationby

experienceduserscanmanagedataaccess,organizedatarepresentationtominimizeredundancy,andfinetunethestorage

WhyuseaDBMS?

5. ConcurrentAccessandCrashRecovery– DBMSschedulesconcurrentaccessestothedatasuch

thattheusersthinkthatthedataisbeingaccessedbyonlyoneuseratatime

– DBMSprotectsdatafromsystemfailures

6. ReducedApplicationDevelopmentTime– Supportsmanyfunctionsthatarecommontoanumber

ofapplicationsaccessingdata– Provideshigh-levelinterface– Facilitatesquickandrobustapplicationdevelopment

WhenNOTtouseaDBMS?• DBMSisoptimizedforcertainkindofworkloadsand

manipulations

• Theremaybeapplicationswithtightreal-timeconstraintsorafewwell-definedcriticaloperations

• AbstractviewofthedataprovidedbyDBMSmaynotsuffice

• Toruncomplex,statistical/MLanalyticsonlargedatasets

DataModel• Applicationsneedtomodelsomerealworldunits• Entities:

– Students,Departments,Courses,Faculty,Organization,Employee,…

• Relationships:– Courseenrollmentsbystudents,Productsalesbyanorganization

• Adatamodelisacollectionofhigh-leveldatadescriptionconstructsthathidemanylow-levelstoragedetails

DataModelCanSpecify:

1. Structureofthedata– likearraysorstructs inaprogramminglanguage– butatahigherlevel(conceptualmodel)

2. Operationsonthedata– unlikeaprogramminglanguage,notanyoperationcanbeperformed– allowlimitedsetsofqueriesandmodifications– astrength,notaweakness!

3. Constraintsonthedata– whatthedatacanbe– e.g.amoviehasexactlyonetitle

ImportantDataModels

• StructuredData• Semi-structuredData• UnstructuredData

Whatarethese?

ImportantDataModels• StructuredData

– Allelementshaveafixedformat– RelationalModel(table)

• Semi-structuredData– Somestructurebutnotfixed– Hierarchicallynestedtagged-elementsintreestructure– XML

• UnstructuredData– Nostructure– text,image,audio,video

RelationalDataModel

• ProposedbyEdward(Ted)Codd in1970– wonTuringawardforit!

• Motivation:– Simplicity– Betterlogicalandphysicaldataindependence

RelationalDataModel

• ThedatadescriptionconstructisaRelation– Representedasa“table”– Basicallya“set”ofrecords(setsemantic)– orderdoesnotmatter– andallrecordsaredistinct

• however,itistruefortherelationalmodel,notforstandardDBM– allowduplicaterows(bagsemantic)– unlessrestrictedbykeyconstraints.Why?

sid name login age gpa53666 Jones jones@cs 18 3.453688 Smith smith@ee 18 3.253650 Smith smith1@math 19 3.853831 Madayan madayan@music 11 1.853832 Guldu guldu@music 12 2.0

Students

Bag:{1,1,2,2,3,2,1,5,6,1}Set:{1,2,3,5,6}

Bagvs.Set

• Why“bagsemantic”andnot“setsemantic”instandardDBMSs?– Primarilyperformancereasons– Duplicateeliminationisexpensive(requiressorting)– Someoperationslike“projection”s aremuchmoreefficientonbags

thansets

Students

RelationalDataModel

Students Attribute/Column/Field

Tuple/Row/Record

Whatisapoorlychosenattributeinthisrelation?

• Relationaldatabase=asetofrelations• ARelation:madeupoftwoparts

1. Schema2. Instance

SchemaandInstance• Oneschemacanhavemultipleinstances

• Schema:– Atemplatefordescribinganentity/relationship(e.g.students)– specifiesnameofrelation+nameandtypeofeachcolumne.g.Students(sid:string,name:string,login:string,age:integer,gpa:real).

• Instance:– Whenwefillinactualdatavaluesinaschema– atable,hasrowsandcolumns– eachrow/tuplefollowstheschemaanddomainconstraints– #Rows=cardinality,#fields=degreeorarity– examplebelow

Cardinality = 3, degree = 5sid name login age gpa

53666 Jones jones@cs 18 3.4

53688 Smith smith@ee 18 3.2

53650 Smith smith1@math 19 3.8

LevelsofAbstractionsinaDBMS

• Physicalschema– Storageasfiles,rowvs.

columnstore,indexes– willdiscussthesein

laterlectures

PhysicalSchema

LogicalSchema

ExternalSchema External Schema ExternalSchema

• Logical/Conceptualschema– describesthestoreddatainthe

physicalschema

• Decidedbyconceptualschemadesign

– e.g.ERDiagram• notcoveredinthiscourse

– Normalization• willbecovered

Students(sid:string,name:string,login:string,age:integer,gpa:real)

PhysicalSchema

LogicalSchema

• Externalschema– different“views”ofthe

databasetodifferentusers

– willdiscussviewslater

• Onephysicalandlogicalschemabuttherecanbemultipleexternalschemas

PhysicalSchema

LogicalSchema

DataIndependence

• Applicationprogramsareinsulatedfromchangesinthewaythedataisstructuredandstored

• AveryimportantpropertyofaDBMS

• LogicalandPhysical

LogicalDataIndependence• Userscanbeshieldedfromchangesinthelogical

structureofdata• e.g.Students:

Students(sid:string,name:string,login:string,age:integer,gpa:real)• Divideintotworelations

Students_public(sid:string,name:string,login:string)Students_private(sid:string,age:integer,gpa:real)

• Stilla“view”Studentscanbeobtainedusingtheabovenewrelations– by“joining”themwithsid

• AuserwhoqueriesthisviewStudentswillgetthesameanswerasbefore

PhysicalDataIndependence

• Thelogical/conceptualschemainsulatesusersfromchangesinphysicalstoragedetails– howthedataisstoredondisk– thefilestructure– thechoiceofindexes

• Theapplicationremainsunaltered– Buttheperformancemaybeaffectedbysuchchanges

Veryimportant

UnderstandtheCourse-Policy

See“whatisallowed/notallowed”

willberemindedineveryhwassignmenttoo

compsci516 database systems - duke university•from “big data” wiki: –healthcare:...

Documents

h. pre-engineered and prescriptive foundation designs · h....

mindsphere for prescriptive maintenance · prescriptive...

prescriptive analytics in the supply chain...prescriptive...

smartstart buildings program prescriptive lighting...

compsci516 data intensive computing systems lecture 6a...

compsci516 database systems - duke university

prescriptive intervention plan

advocating for prescriptive authority for psychologists: a...

prescriptive advising

prescriptive analytics

compsci516 data intensive computing systems lecture 6a...

prescriptive approach - california lighting...

compsci516 data intensive computing systems lecture 6a...

prescriptive rights study

map-reduce and spark -...

digitization workflow management system for massive...

smartstart buildings program prescriptive lighting...

aws prescriptive guidance - patterns · aws prescriptive...

icf prescriptive method

prescriptive method -...