Upload File

bayesian data mining

40
Bayesian Data Mining University of Belgrade School of Electrical Engineering Department of Computer Engineering and Information Theory Marko Stupar 11/3370 [email protected] 1/40

Upload: marziya

Post on 11-Apr-2016

6 views

Category:

Documents


3 download

Report

TRANSCRIPT

Bayesian Data Mining

Bayesian Data MiningUniversity of BelgradeSchool of Electrical EngineeringDepartment of Computer Engineering and Information Theory

Marko Stupar 11/3370 [email protected]/40Data Mining problemToo many attributes in training set (colons of table)

Existing algorithms need too much time to find solution

We need to classify, estimate, predict in real time

Marko Stupar 11/3370 [email protected] 1Value 2 . . . Value1000002/402Problem importanceFind relation between: All Diseases,All Medications,All Symptoms,

Marko Stupar 11/3370 [email protected]/40Existing solutionsMarko Stupar 11/3370 [email protected], C4.5Too many iterationsContinuous arguments need binning

Rule inductionContinuous arguments need binning

Neural networksHigh computational time

K-nearest neighborOutput depends only on distance based close values4/40Classification, Estimation, PredictionUsed for large data setVery easy to constructNot using complicated iterative parameter estimationsOften does surprisingly wellMay not be the best possible classifierRobust, Fast, it can usually be relied on to

Marko Stupar 11/3370 [email protected] Bayes Algorithm5/40Marko Stupar 11/3370 [email protected] Bayes algorithm ReasoningNew information arrived

How to classify Target?TargetAttribute 1Attribute 2 Attribute n................Training SetTargetAttribute 1Attribute 2 Attribute na1a2anMissing6/40Target can be one of discrete values: t1, t2, , tn

Marko Stupar 11/3370 [email protected]

Nave Bayes algorithm ReasoningHow to calculate?

Did not helped very much?Nave Bayes assumes that A1An are independent, So:

7/40AgeIncomeStudentCreditTargetBuys Computer1YouthHighNoFairNo2YouthHighNoExcellentNo3MiddleHighNoFairYes4SeniorMediumNoFairYes5SeniorLowYesFairYes6SeniorLowYesExcellentNo7MiddleLowYesExcellentYes8YouthMediumNoFairNo9YouthLowYesFairYes10SeniorMediumYesFairYes11YouthMediumYesExcellentYes12MiddleMediumNoExcellentYes13MiddleHighYesFairYes14SeniorMediumNoExcellentNoAttributes = (Age=youth, Income=medium, Student=yes, Credit_rating=fair)

P(Attributes, Buys_Computer=Yes) =P(Age=youth|Buys_Computer=yes) * P(Income=medium|Buys_Computer=yes) *P(Student=yes|Buys_Computer=yes) * P(Credit_rating=fair|Buys_Computer=yes) * P(Buys_Computer=yes)=2/9 * 4/9 * 6/9 * 6/9 * 9/14 = 0.028Attributes = (Age=youth, Income=medium, Student=yes, Credit_rating=fair)

P(Attributes, Buys_Computer=No) = P(Age=youth|Buys_Computer=no) *P(Income=medium|Buys_Computer=no) * P(Student=yes|Buys_Computer=no) *P(Credit_rating=fair|Buys_Computer=no) * P(Buys_Computer=no)=3/5 * 2/5 * 1/5 * 2/5 * 5/14 = 0.007

Nave BayesDiscrete Target ExampleMarko Stupar 11/3370 [email protected]/408

Nave BayesDiscrete Target - ExampleAttributes = (Age=youth, Income=medium, Student=yes, Credit_rating=fair)Target = Buys_Computer = [Yes | No] ?

P(Attributes, Buys_Computer=Yes) =P(Age=youth|Buys_Computer=yes) * P(Income=medium|Buys_Computer=yes) *P(Student=yes|Buys_Computer=yes) * P(Credit_rating=fair|Buys_Computer=yes) * P(Buys_Computer=yes)=2/9 * 4/9 * 6/9 * 6/9 * 9/14 = 0.028

P(Attributes, Buys_Computer=No) = P(Age=youth|Buys_Computer=no) *P(Income=medium|Buys_Computer=no) * P(Student=yes|Buys_Computer=no) *P(Credit_rating=fair|Buys_Computer=no) * P(Buys_Computer=no)=3/5 * 2/5 * 1/5 * 2/5 * 5/14 = 0.007

P(Buys_Computer=Yes | Attributes) > P(Buys_Computer=No| Attributes)

Therefore, the nave Bayesian classifier predicts Buys_Computer = Yes for previously given Attributes

Marko Stupar 11/3370 [email protected]/409Nave BayesDiscrete Target Spam filterAttributes = Text Document = w1,w2,w3 Array of wordsTarget = Spam = [Yes | No] ?

- probability that the i-th word of a given document occurs in documents, in training set, that are classified as Spam - probability that all words of document occur in Spam documents in training set

Marko Stupar 11/3370 [email protected]/40Nave BayesDiscrete Target Spam filter - Bayes factor

Sample correction if there is a word in document that never occurred in training set the whole will be zero. Sample correction solution put some low value for that

Marko Stupar 11/3370 [email protected]/40Gaussian Nave BayesContinuous AttributesContinuous attributes do not need binning (like CART and C4.5)Choose adequate PDF for each Attribute in training setGaussian PDF is most likely to be used to estimate the attribute probability density function (PDF)Calculate PDF parameters by using Maximum Likelihood MethodNave Bayes assumption - each attribute is independent of other, so joint PDF of all attributes is result of multiplication of single attributes PDFs

Marko Stupar 11/3370 [email protected]/40Gaussian Nave BayesContinuous Attributes - ExampleTraining set

Validation setsexheight (feet)weight (lbs)foot size(inches)male618012male5.9219011male5.58 17012male5.92 16510female51006female5.5 1508female5.421307female5.751509

Target = maleTarget = femaleheight (feet)

5.8850.0271755.41750.07291875weight (lbs)

176.25126.5625132.5418.75foot size(inches)

11.250.68757.51.25

sexheight (feet)weight (lbs)foot size(inches)Target61308

Marko Stupar 11/3370 [email protected]/40Nave Bayes - ExtensionsEasy to extendGaussian Bayes sample of extensionEstimate Target If Target is real number, but in training set has only few acceptable discrete values t1tn, we can estimate Target by:

A large number of modifications have been introduced, by the statistical, data mining, machine learning, and pattern recognition communities, in an attempt to make it more flexible Modifications are necessarily complications, which detract from its basic simplicity

Marko Stupar 11/3370 [email protected]/40Nave Bayes - ExtensionsAre Attributes always really independent? A1 = Weight, A2 = Height, A3 = Shoe Size, Target = [male|female]?How can that influence our Nave Bayes data mining?

Assumption?Bayesian networkMarko Stupar 11/3370 [email protected]/40Marko Stupar 11/3370 [email protected] NetworkBayesian network is a directed acyclic graph (DAG) with a probability table for each node.Bayesian network contains: Nodes and Arcs between themNodes represent arguments from databaseArcs between nodes represent their probabilistic dependencies

TargetA2A1A3P(A2|Target)A6A4A5P(A6|A4A5)A7P(A3|Target,A4A6)16/40Bayesian NetworkWhat to doRead Network

Construct NetworkTraining Set

Continue with nave Bayes

Joint ProbabilityDistributionNave BayesMarko Stupar 11/3370 [email protected]/40Marko Stupar 11/3370 [email protected] NetworkRead NetworkChain rule of probability

Bayesian network - Uses Markov Assumption

A7A2A5Joint Probability DistributionA7 depends only on A2 and A518/40Bayesian NetworkRead Network - ExampleHow to get P(N|B), P(B|M,T)?Expert knowledgeFrom Data(relative frequency estimates)Or a combination of both

MedicationBlood ClothTraumaHeart AttackNothingStrokeP(M)P(!M)0.20.8P(T)P(!T)0.050.95MTP(B)P(!B)TT0.950.05TF0.30.7FT0.60.4FF0.90.1BP(H)P(!H)T0.40.6F0.150.85BP(N)P(!N)T0.250.75F0.750.25BP(S)P(!S)T0.350.65F0.10.9

Marko Stupar 11/3370 [email protected]/40ManuallyFrom Database AutomaticallyHeuristic algorithms1. heuristic search method to construct a model2.evaluates model using a scoring methodBayesian scoring methodentropy based methodminimum description length method3. go to 1 if score of new model is not significantly better

Algorithms that analyze dependency among nodesMeasure dependency by conditional independence (CI) test

Bayesian NetworkConstruct NetworkMarko Stupar 11/3370 [email protected]/40Bayesian NetworkConstruct NetworkHeuristic algorithmsAdvantagesless time complexity in worst caseDisadvantageMay not find the best solution due to heuristic nature

Algorithms that analyze dependency among nodesAdvantagesusually asymptotically correct DisadvantageCI tests with large condition-sets may be unreliable unless the volume of data is enormous.

Marko Stupar 11/3370 [email protected]/4021Bayesian NetworkConstruct Network - Example1. Choose an ordering of variables X1, ,Xn2. For i = 1 to nadd Xi to the networkselect parents from X1, ,Xi-1 such that P (Xi | Parents(Xi)) = P (Xi | X1, ... Xi-1)

Marko Stupar 11/3370 [email protected] CallsJohn CallsAlarmBurglaryEarthquakeP(J | M) = P(J)? NoP(A | J, M) = P(A | J)? P(A | J, M) = P(A)? NoP(B | A, J, M) = P(B | A)? YesP(B | A, J, M) = P(B)? NoP(E | B, A ,J, M) = P(E | A)? NoP(E | B, A, J, M) = P(E | A, B)? Yes

22/40Create Network from databased(Directional)-Separationd-Separation is graphical criterion for deciding, from a given causal graph(DAG), whether a disjoint sets of nodes X-set, Y-set are independent, when we know realization of third Z-set

Z-set - is instantiated(values of its nodes are known) before we try to determine d-Separation(independence) between X-set and Y-set

X-set and Y-set are d-Separated by given Z-set if all paths between them are blocked

Example of Path : N1 N3 -> N4 -> N5


Bayesian Data Analysis

Bayesian Data Analysis - Arizona State Universityjtaylor/teaching/Spring2017/BIO545/... · Bayesian Analysis Bayesian Data Analysis: Overview Bayesian data analysis is based on the

Bayesian Network Learning for Data-Driven Designreliadesign.net/wp-content/uploads/2018/04/Bayesian... · 2018-04-18 · Bayesian Network Learning for Data-Driven Design Bayesian

DATA MINING ON CORPORATE FILLING BASED ON BAYESIAN

Uncertainty & Bayesian Belief Networks. 2 Data-Mining with Bayesian Networks on the Internet Internet can be seen as a massive repository of Data Data

Bayesian Inference Technique for Data mining for Yield ...xs3d.kaist.ac.kr/ismi2015/Presentations/ismi2015... · System Analysis Yield Learning Curve of Semiconductor Manufacturing

DATA MINING FILE SHARING METADATA A comparsion between ...823863/FULLTEXT01.pdf · DATA MINING FILE SHARING METADATA A comparsion between Random Forests Classificiation and Bayesian

Data Mining Cardiovascular Bayesian Networks Charles Twardy †, Ann Nicholson †, Kevin Korb †, John McNeil ‡ (Danny Liew ‡, Sophie Rogers ‡, Lucas Hope

CSE5230 - Data Mining, 2003Lecture 9.1 Data Mining - CSE5230 Bayesian Classification and Bayesian Networks CSE5230/DMS/2003/9

Statistical Data Mining€¦ · 3 Data Mining Data (re-design and maintain existing database) Mining (Analysis) -- our focus Statistical Data Mining What is Data Mining? Data mining

Bayesian Algorithms for Causal Data Miningproceedings.mlr.press/v6/mani10a/mani10a.pdf · Bayesian Algorithms for Causal Data Mining Subramani Mani SUBRAMANI. ... We now define some

Practice of Bayesian Networks Data Mining Lab 4. Bayesian Networks. Two tasks Infer the structure of the network from the data (in practice, the structure

A Bayesian Nonparametric Approach for Functional Data …mguindan/papers/f_5658-CIN.pdf · Keywords: functional data analysis, Bayesian analysis, Bayesian nonparametrics, computed

Bayesian Approaches - Queen's Universityresearch.cs.queensu.ca/home/xiao/doc/dm/Bayes.pdf · Bayesian Approaches Data Mining Selected Technique ... Bayesian Networks ... A Bayes net

Bayesian Networks for Data Miningpzs.dstu.dp.ua/.../bayes/bibl/Tutorial-BayesianNetworks.pdfKeywords: Bayesian networks, Bayesian statistics, learning, missing data, classification,

Mining Educational Data to Predict Students' Future Performance using Naïve Bayesian Algorithm

“Data Mining Approach for environmental Conditions ...marta/Enginfo/Palestra4_Intro_datamining.pdf · Decision Tree in SGI/MineSet. Bayesian Classification • Probabilistic learning:

Bayesian and Non-Bayesian Analysis of Soccer Data using

LNAI 7524 - A Bayesian Scoring Technique for Mining ... · A Bayesian Scoring Technique for Mining Predictive and Non-Spurious Rules IyadBatal 1,GregoryCooper2,andMilosHauskrecht

Data Mining 2012 Bayesian Network ClassifiersData Mining 2012 Bayesian Network Classi ers Ad Feelders Universiteit Utrecht October 25, 2012 Ad Feelders ( Universiteit Utrecht ) Data

Collective Mining of Bayesian Networks from Distributed ...hillol/PUBS/ddm_Bayes.pdf · Collective Mining of Bayesian Networks from Distributed Heterogeneous Data 5 are distributed

EXTENDING BAYESIAN NETWORK MODELS FOR MINING AND

Individualizing Bayesian Knowledge Tracing. Are Skill ... · Individualizing Bayesian Knowledge Tracing. Are Skill ... KDD Cup data mining challenge. ... 2013), draws on the

Software Engineering Laboratory1 Introduction of Bayesian Network 4 / 20 / 2005 CSE634 Data Mining Prof. Anita Wasilewska 105269827 Hiroo Kusaba

Bayesian Knowledge Tracing and Other Predictive Models in Educational Data Mining

Bayesian Networks for Data Miningpzs.dstu.dp.ua/DataMining/bayes/bibl/lemcs0564.pdf · data mining—the process of extracting knowledge from data. There are numerous representations

Collective Mining of Bayesian Networks From Distributed Hetrogeneous Data

Data Mining with Naïve Bayesian Methods

Introduction to Data Mining Bayesian Classificationusers.math.yale.edu/users/gw289/CpSc-445_2016/Slides...Bayesian classification Conditional probability estimation Example Suppose

Data Mining BY JEMINI ISLAM. Data Mining Outline: What is data mining? Why use data mining? How does data mining work The process of data mining Tools

Classification rules comparison into data mining …author used a total of seven data mining algorithms which are Artificial Neural Network, K-Nearest Neighbor, Naïve Bayes, Bayesian

What is Data Mining? Data Mining Motivation Data Mining Applications Applications of Data Mining in CRM Data Mining Taxonomy Data Mining Techniques

S3-SEMINAR ON DATA MINING -BAYESIAN NETWORKS- B. INFERENCE Master Universitario en Inteligencia Artificial Concha Bielza, Pedro Larrañaga Computational

INTEGRATION OF TECHNIQUES WITH DATA MINING ......The data mining techniques is Decision Tree, Rule-Based techniques. Naïve Bayesian and Decision Tree-Rule-Based (DT-RB) found that

CSE5230 - Data Mining, 2002Lecture 9.1 Data Mining - CSE5230 Bayesian Classification and Bayesian Networks CSE5230/DMS/2002/9