data preprocessing part 1 - health...

HAP780DataMininginHealthCare Copyright©JanuszWojtusiak,2016

DataPreprocessingPart1

JanuszWojtusiak,PhDGeorgeMasonUniversity

Fall2016

HAP780DataMininginHealthCare


“Theworldisfullofobviousthingswhichnobodybyanychanceeverobserves.”

-SherlockHolmes


WhyPreprocessing?

• Multiplesources• Multipleformats• Multiplerepresentations• Errors,noise• Missingvalues• Unnecessaryattributes• Not-representativedata• ….andmanymore!


TwoTypesofPreprocessing

• Beforeloadingtodatabase/software– Howtogetdatafrommultiplesourcesintodatabase,datawarehouse,orotherformatonwhichDMtoolscanbeused.

• Afterloadingtodatabase/software– Thisiswhatistypicallycoveredbydatapreprocessing:datacleaning,transformation,reduction,discretization,normalization…..


SourcesofDataforDataMining

• EHRsystems• Billing• Surveys• Reports• Web• Excelspreadsheets• Sensors

• Sometimesweminetogetherdatafrommultiplesources. Simplyspeaking,wewanttobeabletomineanydataandallavailabledata.


FormsofData

• One-dimensional– Signalsfromsensors(EKG,accelerometer,etc.)

• Two-dimensional– Images

• Multidimensional– Flatdatatables(attribute-valuepairs)– RelationalDatabases

• Multimedia


FormatsofData

• Structured– Tables– Relationaldatabases– Non-relational/No-SQLdatabases– TextFiles(comaseparated,specialformats)– XML– Excelfiles– SASdatafiles– ….


FormatsofData

• Unstructured– Textfiles– Websites– Textfieldsindatabases/structureddata– Speech– Multimedia– ….


DirtyData

• Noise

• Incompleteness

• Inconsistency


DirtyDataPTID DOB Age Sex ProvID Dx1 Dx2 Dx3 Dx4 Dx5

1 1/2/70 48 M 345 250.0

2 30 N 010.0 Patientissuffering formberculosis …

473.0

2 1/1/80 33 3456 487

34 487

5 9/8/60 F 327.0 327.2

Thefollowingrecords are imported afterJanuary

6 8/8/54 M 320 250.0 487 296.7 361.0 E858

7 Unknown

M JohnSmith

8 25 F 377 150 151 038.9

Howmanyproblemsareinthisdataset?


DealingwithDirtyData

• Loaddatatodatabase– DataTypes– Obviousproblemsindatafiles

• Datacleaning&transformation– Inconsistencies,missingvalues,sampling,attributeselection,discretization,….

http://www.prosoftsolutions.net/blog/bid/146041/Dirty-Data-What-is-it-how-does-it-cause-problems-and-what-is-the-solution


DataTypes

• Differentnamesforthesame– Field:usedindatabases– Attribute:usedindataminingandmachinelearning– Variable:usedinstatistics– Feature:usedinmachinelearning(usuallymeansbinaryattribute)

• DatabaseAttributeTypes

• AnalyticAttributeTypes


FundamentalConcepts• Symbol:aphysicalentity,itsstate,oritsbehaviorthatconveysachoicefrom

apredefinedsetofchoices.Thechoicesmayrefertoanyentities(physicalorabstractobjects),totheirproperties,ortheiractions.Thechoiceindicatedbyasymboliscalleditsmeaning

• Data:arecordedsetofsymbolscharacterizingasetofentities• Information:interpreteddata;datawhosesymbolshavebeenassignedmeaning

• Knowledge:informationthatisverifiedtobetrueortruetosomedegree,whichcanbeobtainedbydirectobservationorbyinference

• Belief:hypothetical knowledge; knowledge that has not been validated, but is characterized by some measure of it’s the relationship to the reality it describes.


SymbolsData

Information

Knowledge

Belief


FundamentalConcepts

• Concept:asetofentitiesconsideredasaunit,andtypicallygivenaname• Language:asystemofsymbolsandrulesforcreatingexpressionsfromthesesymbolsforthepurposeofcommunicatinginformation

• Description:anexpressioninsomelanguagethatconveysinformationaboutasetofentities.Thesetbeingdescribediscalledthereferenceset.Aconceptdescription describesallentitiesbelongingtotheconcept(conceptinstances)

• Generalization:aprocessofextendingthereferencesetofadescription,oritsresult

• Abstraction:aprocessofreducinginformationaboutareferenceset,oritsresult


DatabaseAttributeTypes

• SystemSpecific• Forexample,inSQLServer2012:


Numeric andDate


StringsandOther


AnalyticDataTypes

• Symbolic– Symbolsusedtorepresententities

• Numeric– Numbers,usuallyusedforcalculations


AnalyticAttributeTypes


Extract,Transform,Load

• ETLisalmostalwaysusedincontextofdatawarehouses,butalsoappliedtodatamining– Extract datafromexternalsources(oftenmany)– Transform intouniformrepresentation– Load intothetargetsystem(DW,DM)


ETLinContext

EMR

Rx

Billing

PACS

ExtractTransform

Load DataWarehouse

FlatFiles

Reporting

DataMining

Analysis


ETLinContext

EMR

Rx

Billing

PACS

ExtractTransform

LoadFlatfilereadyforDataMining

FlatFiles

DataMining


ToolstoHave

• FileViewer• Textfileeditor,Editpad Pro,Notepad++– Notwordprocessor!

• Processingverylargetextfiles– awk,sed,grep,….

• Fileconverters,builtinsoftware…ornot– lotsoffreeones...


HAP780

JanuszWojtusiak,PhDGeorgeMasonUniversity

[email protected]

data preprocessing part 1 - health...

Documents