steps in the data mining process
TRANSCRIPT
Steps in the Data Mining Process
João Gama
December, 2018
Results of Data Mining Include:
• Forecasting what may happen in the future• Classifying people or things into groups by
recognizing patterns• Clustering people or things into groups based
on their attributes• Associating what events are likely to occur
together• Sequencing what events are likely to lead to
later events
Data mining is not–Brute-force crunching of bulk data –“Blind” application of algorithms–Going to find relationships where none exist
–Torturing Data Until They Confess–Presenting data in different ways–A database intensive task–A difficult to understand technology requiring an advanced degree in computer science
Data Mining Is•A hot buzzword for a class of techniques that find patterns in data
•A user-centric, interactive process which leverages analysis technologies and computing power
•A group of techniques that find relationships that have not previously been discovered
Data Mining versus OLAP
•OLAP - On-line Analytical Processing– Provides you with a very good
view of what is happening, – reporting– but can not predict what will
happen in the future or why it is happening
Predictive and Prescriptive Analytics
Predictive and Prescriptive Analytics
Predictive and Prescriptive Analytics
Methodologies for data mining
• 1997: Fayyad, Simoudis
• 2000: SEMMA – SAS
– Sample, Explore, Model, Modify, Assess
• 2000 - CRISP-DM,
– CRoss Industry Standard Process for Data Mining
The Data Mining ProcessFayyad & Simoudis (1997)
Phases of SEMMA (SAS)• Sample.
– The process starts with data sampling, e.g., selecting the data set for modeling. The data set should be large enough to contain sufficient information to retrieve, yet small enough to be used efficiently.
• Explore.
– This phase covers the understanding of the data by discovering anticipated and unanticipated relationships between the variables, and also abnormalities, with the help of data visualization.
• Modify.
– The Modify phase contains methods to select, create and transform variables in preparation for data modeling.
• Model.
– In the Model phase the focus is on applying various modeling (data mining) techniques on the prepared variables in order to create models that possibly provide the desired outcome.
• Assess.
– The last phase is Assess. The evaluation of the modeling results shows the reliability and usefulness of the created models.
SEMMA• SEMMA mainly focuses on the modeling tasks of
data mining projects, leaving the business aspects out (unlike, i.e., CRISP-DM and its Business Understanding phase).
• SEMMA is designed to help the users of the SAS Enterprise Miner software. Therefore, applying it outside Enterprise Miner can be ambiguous.
How Can We Do Data Mining?
How Can We Do Data Mining?
By using the CRISP-DM Methodology
– a standard process
– existing data
– software technologies
– situational expertise
Process Standardization
CRISP-DM: • CRoss Industry Standard Process for Data Mining• Initiative launched Sept.1996• SPSS/ISL, NCR, Daimler-Benz, OHRA• Funding from European commission• Over 200 members of the CRISP-DM SIG worldwide
– DM Vendors - SPSS, NCR, IBM, SAS, SGI, Data Distilleries, Syllogic, Magnify, ..– System Suppliers / consultants - Cap Gemini, ICL Retail, Deloitte & Touche, …– End Users - BT, ABB, Lloyds Bank, AirTouch, Experian, ...
CRISP-DM
•Non-proprietary
•Application/Industry neutral
•Tool neutral
•Focus on business issues
– As well as technical analysis
•Framework for guidance
•Experience base
– Templates for Analysis
– Developed by:– NCR, SPSS, Daimler-Chrysler, OHRA
Why CRISP-DM?
•The data mining process must be reliable and repeatable by
people with little data mining skills
•CRISP-DM provides a uniform framework for
–guidelines
–experience documentation
•CRISP-DM is flexible to account for differences
–Different business/agency problems
–Different data
BusinessUnderstanding
DataUnderstanding
EvaluationData
PreparationModeling
Determine Business Objectives
BackgroundBusiness ObjectivesBusiness Success
Criteria
Situation AssessmentInventory of ResourcesRequirements,Assumptions, andConstraints
Risks and ContingenciesTerminologyCosts and Benefits
Determine Data Mining Goal
Data Mining GoalsData Mining Success
Criteria
Produce Project PlanProject PlanInitial Asessment of Tools and Techniques
Collect Initial DataInitial Data Collection
Report
Describe DataData Description Report
Explore DataData Exploration Report
Verify Data Quality Data Quality Report
Data SetData Set Description
Select Data Rationale for Inclusion /
Exclusion
Clean Data Data Cleaning Report
Construct DataDerived AttributesGenerated Records
Integrate DataMerged Data
Format DataReformatted Data
Select ModelingTechnique
Modeling TechniqueModeling Assumptions
Generate Test DesignTest Design
Build ModelParameter SettingsModelsModel Description
Assess ModelModel AssessmentRevised Parameter Settings
Evaluate ResultsAssessment of Data
Mining Results w.r.t. Business Success Criteria
Approved Models
Review ProcessReview of Process
Determine Next StepsList of Possible ActionsDecision
Plan DeploymentDeployment Plan
Plan Monitoring and Maintenance
Monitoring and Maintenance Plan
Produce Final ReportFinal ReportFinal Presentation
Review ProjectExperience
Documentation
Deployment
Phases and Tasks
Phases in the DM Process: CRISP-DM
Phases in the DM Process (1 & 2)
•Business Understanding:
– Statement of Business Objective
– Statement of Data Mining objective
– Statement of Success Criteria
•Data Understanding
– Explore the data and verify the quality
– Find outliers
Phases in the DM Process (3)
• Data preparation:– Takes usually over 90% of our time
• Collection• Assessment• Consolidation and Cleaning
– table links, aggregation level, missing values, etc
• Data selection– active role in ignoring non-contributory
data?
– outliers?– Use of samples– visualization tools
• Transformations - create new variables
Phases in the DM Process (4)
• Model building
– Selection of the modeling techniques is based upon the data mining objective
– Modeling is an iterative process -different for supervised and unsupervised learning
• May model for either description or prediction
Types of Models
•Prediction Models for Predicting and Classifying
– Regression algorithms (predict numeric outcome): neural networks, rule induction, CART (OLS regression, GLM)
– Classification algorithm predict symbolic outcome): CHAID, C5.0(discriminant analysis, logistic regression)
•Descriptive Models for Grouping and Finding Associations
– Clustering/Grouping algorithms: K-means, Kohonen
– Association algorithms: apriori, GRI
Phases in the DM Process (5)
• Model Evaluation– Evaluation of model: how well it
performed on test data
– Methods and criteria depend on model type:
• e.g., coincidence matrix with classification models, mean error rate with regression models
– Interpretation of model: important or not, easy or hard depends on algorithm
Phases in the DM Process (6)
•Deployment– Determine how the results need to be utilized– Who needs to use them?– How often do they need to be used
•Deploy Data Mining results by:– Scoring a database– Utilizing results as business rules– interactive scoring on-line
Comparison between Methods
Final Comments
• Data Mining can be utilized in any organization that needs to find patterns or relationships in their data.
• By using the CRISP-DM methodology, analysts can have a reasonable level of assurance that their Data Mining efforts will render useful, repeatable, and valid results.
• In 2015, IBM Corporation released a new methodology called Analytics Solutions Unified Method for Data Mining/Predictive Analytics (also known as ASUM-DM) which refines and extends CRISP-DM.
Exercise• An insurance company has in its database, information on commercial insurance contract by
each of its clients . – The most important types of insurance are "Home" , "Auto" , "Life" , "Health" and "Leisure" .
– The insurer also has information on the type of area where the insured (rural , urban ) , occupation and age (numeric) resides.
– The Company now plans to sell a new insurance " Leisure " area , who are already your customer and not have this type of insurance . It intends to organize a promotional event you will be invited only the most promising customers .
– The Marketing of the insurer decided to use the data mining for it. For it has 6 person-months ( PM ) for a period of 20 months. The IT department has experience in using the software SPSS Clementine and Oracle DB . It is this database that are the sales figures of the insurance company .
– The management of the company wants results in the next 18 months and find positive it has sold 500 new policies in the universe of their customers who are currently 12000 people.
• In terms of methodology CRISP- DM :– review the tasks of the first two phases ( business understanding and data understanding ) and define what you
can on these two phases .
– do something similar for the last phase ( deployment )
Bibliography
• CRISP-DM user guide
– Availble in pdf at www.crisp-dm.org
– Glossário
• Articles– Shearer C. The CRISP-DM model: the new blueprint for data mining. Journal
Data Warehousing
• Books– “Data Mining Tecnhiques”, Berry e Linoff, Wiley, 97
• Other– Wikipedia:
http://pt.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining