![Page 1: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/1.jpg)
Machine Learning in PracticeLecture 3
Carolyn Penstein Rosé
Language Technologies Institute/ Human-Computer Interaction
Institute
![Page 2: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/2.jpg)
Plan for Today Announcements
Assignment 2Quiz 1
Weka helpful hints Topic of the day: Input and Output More on cross-validation ARFF format
![Page 3: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/3.jpg)
Weka Helpful Hints
![Page 4: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/4.jpg)
Increase Heap Size
![Page 5: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/5.jpg)
Weka Helpful Hint: Documentation!!
Click on Morebutton!
![Page 6: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/6.jpg)
Output Predictions Option
![Page 7: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/7.jpg)
Output Predictions Option
Important note: Because of the way Weka randomizes the data forcross-validation, the only circumstance under which you can matchthe instance numbers to positions in your data is if you have separate train and test sets so the order will be preserved!
![Page 8: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/8.jpg)
View Classifier Errors
![Page 9: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/9.jpg)
Input and Output
![Page 10: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/10.jpg)
Representations
Concept: the rule you want to learn
Instance: one data point from your training or testing data (row in table)
Attribute: one of the features that an instance is composed of (column in table)
![Page 11: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/11.jpg)
Numeric versus Nominal Attributes What kind of reasoning does your
representation enable? Numeric attributes allow instances to be
ordered Numeric attributes allow you to measure
distance between instances Sometimes numeric attributes make too fine
grained of a distinction
.2 .25 .28 .31 .35 .45 .47 .52 .6 .63
![Page 12: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/12.jpg)
Numeric versus Nominal Attributes
.2 .25 .28 .31 .35 .45 .47 .52 .6 .63
Numeric attributes can be discretized into nominal values Then you lose ordering and distance Another option is applying a function that maps a range
of values into a single numeric attribute
Nominal attributes can be mapped into numbers i.e., decide that blue=1 and green=2 But are inferences made based on this valid?
![Page 13: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/13.jpg)
Numeric versus Nominal Attributes
.2 .25 .28 .31 .35 .45 .47 .52 .6 .63
.2 .3 .5 .6
Numeric attributes can be discretized into nominal values Then you lose ordering and distance Another option is applying a function that maps a range
of values into a single numeric attribute
Nominal attributes can be mapped into numbers i.e., decide that blue=1 and green=2 But are inferences made based on this valid?
![Page 14: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/14.jpg)
Example!
Problem: Learn a rule that predicts how much time a person spends doing math problems each day
Attributes: You know gender, age, socio-economic status of parents, chosen field if any
How would you represent age, and why? What would you expect the target rule to look like?
![Page 15: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/15.jpg)
Styles of Learning Classification – learn rules from labeled
instances that allow you to assign new instances to a class
Association – look for relationships between features, not just rules that predict a class from an instance (more general)
Clustering – look for instances that are similar (involves comparisons of multiple features)
Numeric Prediction (regression models)
![Page 16: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/16.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
![Page 17: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/17.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
What else would be affected if wheatwere to disappear?
![Page 18: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/18.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
How would you represent this data?
![Page 19: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/19.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
What would the learned rule look like?
![Page 20: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/20.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
What would the learned rule look like?
![Page 21: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/21.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
![Page 22: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/22.jpg)
Food Web What if you wanted a more general rule: i.e., Affects(Entity1, Entity2)
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
![Page 23: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/23.jpg)
Food Web What if you wanted a more general rule: i.e., Affects(Entity1, Entity2)
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
![Page 24: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/24.jpg)
Food Web What if you wanted a more general rule: i.e., Affects(Entity1, Entity2)
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
122 rows altogether!Now let’s look at the learned rule….
![Page 25: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/25.jpg)
Food Web What if you wanted a more general rule: i.e., Affects(Entity1, Entity2)
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
122 rows altogether!Now let’s look at the learned rule….
![Page 26: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/26.jpg)
Food Web What if you wanted a more general rule: i.e., Affects(Entity1, Entity2)
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
122 rows altogether!Now let’s look at the learned rule…. Does it have to be this complicated?
![Page 27: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/27.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
What would your representation for Affects(Entity1, Entity2) look like?
![Page 28: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/28.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
What would your representation for Affects(Entity1, Entity2) look like?
![Page 29: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/29.jpg)
Food Web
http://www.cas.psu.edu/DOCS/WEBCOURSE/WETLAND/WET1/identify.html
What would your representation for Affects(Entity1, Entity2) look like?
![Page 30: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/30.jpg)
More on Cross-Validation
![Page 31: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/31.jpg)
Cross Validation Exercise
What is the same?What is different?
1 2
3 45
What surprises you?
![Page 32: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/32.jpg)
Compare Folds with Tree Trained on Whole Set1 2
3 45
![Page 33: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/33.jpg)
Train Versus TestPerformance on Training Data Performance on Testing Data
![Page 34: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/34.jpg)
Which Model Do You Think Will Perform Best on Test Set?1 2
3 45
![Page 35: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/35.jpg)
Fold 1
![Page 36: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/36.jpg)
Fold 2
![Page 37: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/37.jpg)
Fold 3
![Page 38: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/38.jpg)
Fold 4
![Page 39: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/39.jpg)
Fold 5
![Page 40: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/40.jpg)
Total Performance
What do you notice?
![Page 41: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/41.jpg)
Total Performance
Average Kappa = .5
![Page 42: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/42.jpg)
Starting to think about Error Analyses
Step 1: Look at the confusion matrix Where are most of the errors occurring? What are possible explanations for systematic
errors you see? Are the instances in the confusable classes too similar
to each other? If so, how can we distinguish them? Are we paying attention to the wrong features? Are we missing features that would allow us to see
commonalities within classes that we are missing?
![Page 43: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/43.jpg)
What went wrong on Fold 3?1 2
3 45
![Page 44: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/44.jpg)
What went wrong on Fold 3?
Training Set Performance Testing Set Performance
Hypotheses?
![Page 45: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/45.jpg)
What went wrong on Fold 3?
Training Set Performance Testing Set Performance
Hypotheses?
![Page 46: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/46.jpg)
What’s the difference?
![Page 47: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/47.jpg)
Hypothesis: Problem with first cut
![Page 48: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/48.jpg)
Some Examples
![Page 49: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/49.jpg)
What do you conclude?
![Page 50: Machine Learning in Practice Lecture 3 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute](https://reader030.vdocument.in/reader030/viewer/2022012918/56649d085503460f949d9b5e/html5/thumbnails/50.jpg)
What do you conclude?
Problem with Fold 3 was probably just a sampling fluke.Distribution of classes different between train and test.