fundamentals of accuracy_assessment_session2_czaplewski
TRANSCRIPT
![Page 1: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/1.jpg)
Session 2
Fundamentals of Accuracy Assessment
Raymond L Czaplewski
United States Forest Service
Rocky Mountain Research Station
Fort Collins, Colorado USA
1
![Page 2: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/2.jpg)
Session 2 Topics
• Different sample designs– Simple Random Sampling (Systematic Sampling)
– Stratified Random Sampling
• Different sample survey estimators
• Different sample sizes, n=30, 60, 150
• How close are estimates to true value?
• Example of a 30×30 = 900 pixel world
2
![Page 3: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/3.jpg)
N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ NN N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N NN N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N NN N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ NN N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N NN N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ NN N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^N N N N N N N ^ N N N N N N N N N N N N N N ^ ^ ^ ^ N ^ N NN N N N N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N ^ ^ ^ ^
True (reference) population
Hypothetical “real world”
3
Nat
ural
Urb
an
Crop
N ^
Reference class30×30 = 900 pixels
![Page 4: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/4.jpg)
Hypothetical remotely sensed thematic map model for this “real world”
Map #1N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ NN ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ NN N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N
N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N NN N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^N N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ NN ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ NN ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^^ N N N N N ^ ^ N N N N N ^ N N N N N N N ^ N N ^ N N ^N N N N ^ N ^ N ^ N N N N ^ N N N N ^ ^ ^ ^ ^ ^ N N ^ N ^ 4
Natural NUrbanCrop ^M
ap c
lass
30×30 = 900 pixels
![Page 5: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/5.jpg)
N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ NN ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ NN N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N
N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N NN N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^N N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ NN ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ NN ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^^ N N N N N ^ ^ N N N N N ^ N N N N N N N ^ N N ^ N N ^N N N N ^ N ^ N ^ N N N N ^ N N N N ^ ^ ^ ^ ^ ^ N N ^ N ^
N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ NN N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N NN N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N NN N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ NN N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N NN N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ NN N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^N N N N N N N ^ N N N N N N N N N N N N N N ^ ^ ^ ^ N ^ N NN N N N N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N ^ ^ ^ ^
Remotely sensed thematic Map #1
True (reference) population Map #1
Error matrix presented by Steve Stehman
Traditional Analysis: Error (Confusion) Matrix
Reference Land CoverMapped Natural Urban Crop TotalNatural 0.25 0.03 0.08 0.36Urban 0.02 0.12 0.04 0.18Crop 0.10 0.04 0.32 0.46Total 0.37 0.19 0.44
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Overall accuracy
Reference class
Map
cla
ss
Reference class
Map
cla
ss
5
30×30 = 900 pixels
True Error Matrix
![Page 6: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/6.jpg)
True error matrix parameters
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%Overall accuracy
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Reference class
Map
cla
ss
6
![Page 7: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/7.jpg)
True error matrix parameters, graphical presentation
0% 50% 100%
True Reference Land Cover area
True Map Land Cover areaNatural
Urban
Crop
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%Overall accuracy
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Reference class
Map
cla
ss
7
44%37% 19%
![Page 8: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/8.jpg)
True error matrix parameters, graphical presentation
0% 50% 100%
True Reference Land Cover area
True Map Land Cover areaNatural
Urban
Crop
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%Overall accuracy
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Reference class
Map
cla
ss
8
46%36% 18%
![Page 9: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/9.jpg)
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappa
True
Producer’s AccuracyUser’s Accuracy
True error matrix parameters, graphical presentation
Natural Urban Crop TotalNatural 69% 8% 23% 100%
Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%
User's AccuracyReference class
Map
cla
ss
Natural Urban CropNatural 68% 16% 19%
Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%
Producer's AccuracyReference class
Map
cla
ss
Overall accuracy 69% kappa 51%
9
![Page 10: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/10.jpg)
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappa
True
Producer’s AccuracyUser’s Accuracy
True error matrix parameters, graphical presentation
Natural Urban Crop TotalNatural 69% 8% 23% 100%
Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%
User's AccuracyReference class
Map
cla
ss
Natural Urban CropNatural 68% 16% 19%
Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%
Producer's AccuracyReference class
Map
cla
ss
Overall accuracy 69% kappa 51%
10
![Page 11: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/11.jpg)
True error matrix parameters, graphical presentation
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappa
True
Natural Urban Crop TotalNatural 69% 8% 23% 100%
Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%
User's AccuracyReference class
Map
cla
ss
Natural Urban CropNatural 68% 16% 19%
Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%
Producer's AccuracyReference class
Map
cla
ss
Overall accuracy 69% kappa 51%
Producer’s AccuracyUser’s Accuracy
11
![Page 12: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/12.jpg)
True (reference) population
In the real world, we do not know the true classification for all 900 pixels
True error matrix
12
? ? 900
Natural NUrbanCrop ^M
ap c
lass
Nat
ural
Urb
an
Crop
N ^
Reference class
![Page 13: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/13.jpg)
True (reference) population
In the real world, we do know the true classification for 30 sampled pixels
Sample of true (reference) population True error matrix
13
? 900
Natural NUrbanCrop ^M
ap c
lass
Nat
ural
Urb
an
Crop
N ^
Reference class
![Page 14: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/14.jpg)
In the real world, we do know the true classification for 30 sampled pixels
Sample of true (reference) population
Natural Urban Crop TotalNatural 8 0 2 10
Urban 0 4 2 6Crop 2 2 10 14Total 10 6 14 30
73% kappa 58%
Natural Urban Crop TotalNatural 27% 0% 7% 33%
Urban 0% 13% 7% 20%Crop 7% 7% 33% 47%Total 33% 20% 47% 100%
Map
cla
ss
Overall accuracyReference class
Map
cla
ss
Reference class
Error matrix estimate from sample
True error matrix
14
? 900
![Page 15: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/15.jpg)
In the real world, we do not know the true classification for all 900 pixels
• Let us leave the real world for the next 30 minutes to compare– Known estimate of an error matrix with a sample
of 30 pixels
– Unknown true error matrix for all 900 pixels
15
![Page 16: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/16.jpg)
Comparison of true (unknown) error matrix with (known) sample estimate
16
Natural Urban Crop TotalNatural 8 0 2 10
Urban 0 4 2 6Crop 2 2 10 14Total 10 6 14 30
73% kappa 58%Overall accuracy
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%
Map
cla
ss
Overall accuracy
Reference classError matrix estimate from sampleTrue (unknown) error matrix
![Page 17: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/17.jpg)
Examples of random sampling error, simple random sample #1, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
17
![Page 18: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/18.jpg)
Examples of random sampling error, simple random sample #2, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
18
![Page 19: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/19.jpg)
Examples of random sampling error, simple random sample #3, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
19
![Page 20: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/20.jpg)
Examples of random sampling error, simple random sample #4, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
20
![Page 21: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/21.jpg)
Examples of random sampling error, simple random sample #5, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
21
![Page 22: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/22.jpg)
But how good is the sample estimate? Example, Producer’s Accuracy Urban
22
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
![Page 23: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/23.jpg)
Example: Producers accuracy for urban
23
Estimated Producer's Accuracy = 80%
`
0% 50% 100%
Natural
Urban
Crop
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
![Page 24: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/24.jpg)
Sample #1, n=60
24
Estimated Producer's Accuracy = 80%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 25: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/25.jpg)
Sample #2, n=60
25
Estimated Producer's Accuracy = 71%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 26: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/26.jpg)
Sample #3, n=60
26
Estimated Producer's Accuracy = 60%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 27: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/27.jpg)
Sample #4, n=60
27
Estimated Producer's Accuracy = 53%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 28: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/28.jpg)
Sample #5, n=60
28
Estimated Producer's Accuracy = 70%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 29: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/29.jpg)
Sample #6, n=60
29
Estimated Producer's Accuracy = 73%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 30: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/30.jpg)
Sample #7, n=60
30
Estimated Producer's Accuracy = 88%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 31: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/31.jpg)
Sample #8, n=60
31
Estimated Producer's Accuracy = 60%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 32: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/32.jpg)
Sample #9, n=60
32
Estimated Producer's Accuracy = 40%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 33: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/33.jpg)
Sample #10, n=60
33
Estimated Producer's Accuracy = 45%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 34: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/34.jpg)
Sample #11, n=60
34
Estimated Producer's Accuracy = 67%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 35: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/35.jpg)
Sample #12, n=60
35
Estimated Producer's Accuracy = 86%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 36: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/36.jpg)
Sample #13, n=60
36
Estimated Producer's Accuracy = 75%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 37: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/37.jpg)
Sample #14, n=60
37
Estimated Producer's Accuracy = 73%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 38: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/38.jpg)
Sample #15, n=60
38
Estimated Producer's Accuracy = 47%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 39: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/39.jpg)
Sample #16, n=60
39
Estimated Producer's Accuracy = 57%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 40: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/40.jpg)
Sample #17, n=60
40
Estimated Producer's Accuracy = 64%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 41: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/41.jpg)
Sample #18, n=60
41
Estimated Producer's Accuracy = 70%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 42: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/42.jpg)
Sample #19, n=60
42
Estimated Producer's Accuracy = 67%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 43: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/43.jpg)
Sample #20, n=60
43
Estimated Producer's Accuracy = 77%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 44: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/44.jpg)
Sample #21, n=60
44
Estimated Producer's Accuracy = 67%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 45: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/45.jpg)
Sample #22, n=60
45
Estimated Producer's Accuracy = 76%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 46: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/46.jpg)
Sample #23, n=60
46
Estimated Producer's Accuracy = 100%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 47: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/47.jpg)
Sample #24, n=60
47
Estimated Producer's Accuracy = 90%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 48: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/48.jpg)
Sample #25, n=60
48
Estimated Producer's Accuracy = 50%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 49: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/49.jpg)
Sample #26, n=60
49
Estimated Producer's Accuracy = 77%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 50: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/50.jpg)
Sample #27, n=60
50
Estimated Producer's Accuracy = 91%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 51: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/51.jpg)
Sample #28, n=60
51
Estimated Producer's Accuracy = 40%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 52: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/52.jpg)
Sample #29, n=60
52
Estimated Producer's Accuracy = 27%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 53: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/53.jpg)
Sample #30, n=60
53
Estimated Producer's Accuracy = 55%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 54: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/54.jpg)
Sample #31, n=60
54
Estimated Producer's Accuracy = 56%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 55: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/55.jpg)
Sample #32, n=60
55
Estimated Producer's Accuracy = 78%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 56: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/56.jpg)
Sample #33, n=60
56
Estimated Producer's Accuracy = 40%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 57: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/57.jpg)
Sample #34, n=60
57
Estimated Producer's Accuracy = 71%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 58: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/58.jpg)
Sample #35, n=60
58
Estimated Producer's Accuracy = 83%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 59: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/59.jpg)
Sample #36, n=60
59
Estimated Producer's Accuracy = 62%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 60: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/60.jpg)
Sample #37, n=60
60
Estimated Producer's Accuracy = 64%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 61: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/61.jpg)
Sample #38, n=60
61
Estimated Producer's Accuracy = 88%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 62: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/62.jpg)
Sample #39, n=60
62
Estimated Producer's Accuracy = 57%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 63: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/63.jpg)
Sample #40, n=60
63
Estimated Producer's Accuracy = 56%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 64: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/64.jpg)
Sample #41, n=60
64
Estimated Producer's Accuracy = 57%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 65: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/65.jpg)
Sample #42, n=60
65
Estimated Producer's Accuracy = 73%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 66: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/66.jpg)
Sample #43, n=60
66
Estimated Producer's Accuracy = 40%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 67: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/67.jpg)
Sample #44, n=60
67
Estimated Producer's Accuracy = 55%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 68: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/68.jpg)
Sample #45, n=60
68
Estimated Producer's Accuracy = 63%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 69: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/69.jpg)
Sample #46, n=60
69
Estimated Producer's Accuracy = 71%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 70: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/70.jpg)
Sample #47, n=60
70
Estimated Producer's Accuracy = 55%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 71: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/71.jpg)
Sample #48, n=60
71
Estimated Producer's Accuracy = 67%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 72: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/72.jpg)
Sample #49, n=60
72
Estimated Producer's Accuracy = 77%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 73: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/73.jpg)
Sample #50, n=60
73
Estimated Producer's Accuracy = 89%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
![Page 74: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/74.jpg)
0
100
200
300
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Estimated Producer's Accuracy = 78%
`
0% 50% 100%
Natural
Urban
Crop
74
True accuracy = 63%
Truth = 63%
![Page 75: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/75.jpg)
0
100
200
300
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Estimated Producer's Accuracy = 78%
`
0% 50% 100%
Natural
Urban
Crop
In the real world, we do not know the true value
75
True accuracy = 63%
![Page 76: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/76.jpg)
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
In the real world, we do not know the true value, and we have only 1 sample
76
Estimated Producer's Accuracy = 78%
`
0% 50% 100%
Natural
Urban
Crop
![Page 77: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/77.jpg)
Examples of random sampling error, simple random sample
• Any single sample estimate can differ from true error matrix from random sampling error
• Given our only sample with n=60, the estimated urban producers accuracy = 78% even though the true value is 63%
• However, the sample estimate is expected to equal the true value over all possible samples
77
![Page 78: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/78.jpg)
Examples of random sampling error, simple random sample
• How can we improve reliability of estimate?
• What if sample size increased from n=60 to n=150?
78
![Page 79: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/79.jpg)
Examples of random sampling error, simple random sample #51, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
79
![Page 80: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/80.jpg)
Examples of random sampling error, simple random sample #52, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
80
![Page 81: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/81.jpg)
Examples of random sampling error, simple random sample #53, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
81
![Page 82: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/82.jpg)
Examples of random sampling error, simple random sample #54, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
82
![Page 83: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/83.jpg)
Examples of random sampling error, simple random sample #55, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
83
![Page 84: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/84.jpg)
As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value
84
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
63%
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
63%
n=60 n=150
• When we do not know the true value and we have only 1 sample,• Chances that our 1 estimate is closer to the true value increases as
the sample size increases
![Page 85: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/85.jpg)
Examples of random sampling error, simple random sample, sample size n=150
• Effects of random sampling error become smaller as the sample size becomes larger
• Sampling error affected by sample size, not proportion of population sampled
• Sample size n=1 000 very accurate estimate of true error matrix
• 1 000 pixels = 0.01% of 1 000 000 pixel area
• 10% of a 1 000 000 pixel area = 100 000 pixels
85
![Page 86: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/86.jpg)
As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value
86
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
63%
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
63%
n=60 n=150
• When we do not know the true value and we have only 1 sample,
![Page 87: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/87.jpg)
As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value
87
63% 63%
n=60 n=150
• When we do not know the true value and we have only 1 sample,• Variance estimate gives an idea of spread among potential sample
estiamtes
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
![Page 88: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/88.jpg)
As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value
88
n=60 n=150
• When we do not know the true value and we have only 1 sample,• Variance estimate gives an idea of spread among potential sample
estiamtes
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
![Page 89: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/89.jpg)
Alternative Estimators
• Sample selection and sample estimator are separate, but related, components
• Following example– Simple Random Sample, n=30
– Unbiased Simple Random Sample Estimator
– Unbiased Stratification Estimator
89
![Page 90: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/90.jpg)
Estimator = Simple Random Sample n=30 Sample #51
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
90
![Page 91: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/91.jpg)
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 13% 10% 26% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 36% 13% 50% 99% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
18%2% = 3%27%
91
![Page 92: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/92.jpg)
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 10% 21% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 36% 9% 50% 95% Total 100%
Overall accuracy 66% kappa 49%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
18%9% = 13%27%
92
![Page 93: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/93.jpg)
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 36% 9% 47% 91% Total 100%
Overall accuracy 66% kappa 50%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
18%7% = 10%27%
93
![Page 94: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/94.jpg)
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 36% 9% 47% 91% Total 100%
Overall accuracy 66% kappa 50%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
94
![Page 95: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/95.jpg)
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 30% 38% Crop 46%Total 37% 9% 47% 93% Total 100%
Overall accuracy 66% kappa 49%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
46%8% = 7%37%
95
![Page 96: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/96.jpg)
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
96
![Page 97: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/97.jpg)
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
46%37% = 30%37%
97
![Page 98: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/98.jpg)
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
98
![Page 99: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/99.jpg)
Natural Urban Crop Total TotalNatural 26% 0% 10% 36% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
36%26% = 27%37%
99
![Page 100: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/100.jpg)
Natural Urban Crop Total TotalNatural 26% 0% 10% 36% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
100
![Page 101: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/101.jpg)
Natural Urban Crop Total TotalNatural 26% 0% 10% 36% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
36%10% = 10%37%
101
![Page 102: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/102.jpg)
Natural Urban Crop Total TotalNatural 26% 0% 10% 36% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
Stratification Estimator
Simple Random Sampling Estimator
102
![Page 103: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/103.jpg)
Estimator = Simple Random Sample n=30 Sample #51
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
103
![Page 104: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/104.jpg)
Estimator = Stratification n=30 Sample #51
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
104
![Page 105: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/105.jpg)
Alternative Estimators
• Stratification and Simple Random Sampling Estimators are both unbiased applied to the same Simple Random Sample
• Stratification Estimator is expected to be more precise than Simple Random Sampling Estimator when applied to the same Simple Random Sample
105
![Page 106: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/106.jpg)
Alternative Sampling Designs
• Simple Random Sampling– Equal probability sampling regardless of map– Sample selection at any time– n = 150
• Pre-Stratified Sampling– More intense sampling of pixels in more rare map
categories– Sample selection after map categories known– nNatural = nUrban = nCrop = 50– n = nNatural + nUrban + nCrop = 150
106
![Page 107: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/107.jpg)
True Sample 0 Total TotalNatural 38 4 8 50 Natural 15%
Urban 5 34 11 50 Urban 31%Crop 12 6 32 50 Crop 12%Total 55 44 51 150
Map
cla
ss
Sampling IntensityReference class
Map
cla
ss
Pre-Stratified Sample
True Sample 0 Total TotalNatural 45 6 11 62 Natural 19%
Urban 3 20 4 27 Urban 17%Crop 14 7 40 61 Crop 15%Total 62 33 55 150
Map
cla
ss
Sampling IntensityReference class
Map
cla
ss
Simple Random Sample
Sample Design = Simple Random Sample Sample #52 n=150 Sample Design = Pre-Stratified Sample #53 n=150
All map pixelsTrue Sample 0 Total
Natural 226 27 74 327Urban 18 108 36 162
Crop 89 36 286 411Total 333 171 396 900
Map
cla
ss
True value from all pixelsReference class
Pre-Stratified Sample #13
107
Simple Random Sample #12
![Page 108: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/108.jpg)
Sample Design = Simple Random Sample Sample #52 n=150
Estimator = Simple Random Sampling
108
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
![Page 109: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/109.jpg)
Sample Design = Pre-Stratified Sample #53 n=150Estimator = Simple Random Sampling
109
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
![Page 110: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/110.jpg)
Sample Design = Pre-Stratified Sample #53 n=150
110
Natural Urban Crop TotalNatural 28% 3% 6% 36%
Urban 2% 12% 4% 18%Crop 11% 5% 29% 46%Total 40% 21% 39% 100%
Overall accuracy 69% kappa 52%
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Overall accuracy 69% kappa 51%
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 5% 33%
Urban 3% 23% 7% 33%Crop 8% 4% 21% 33%Total 37% 29% 34% 100%
Overall accuracy 69% kappa 54%
Reference class
Map
cla
ss
Simple Random Sampling Estimator
True error matrix Stratification Estimator
![Page 111: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/111.jpg)
Sample Design = Pre-Stratified Sample #53 n=150Estimator = Pre-Stratified Design
111
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
![Page 112: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/112.jpg)
Sample Design = Pre-Stratified Sample #53 n=150Estimator = Simple Random Sampling
112
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
![Page 113: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/113.jpg)
Summary
• Estimator must be compatible with sampling design to be unbiased
• Simple Random Sampling– Simple Random Sample Estimator Unbiased
– Stratification Estimator Unbiased
• Pre-Stratified Sampling – Simple Random Sample Estimator Biased
– Stratification Estimator Unbiased
113
![Page 114: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/114.jpg)
Comments
• Stratification estimator improves estimates for Simple Random Samples and Pre-Stratified Samples
• Variance formulae differ among estimators– The variance estimator for Simple Random
Sampling will be biased for other estimators
– Congalton and Green(2009)
114
![Page 115: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/115.jpg)
Lessons Learned
• Unbiased sample estimates requires compatibility between sampling design and sample survey estimator
• Chances of a sample estimate being close to true (unknown) value improves with larger sample sizes
115
![Page 116: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/116.jpg)
116
![Page 117: Fundamentals of accuracy_assessment_session2_czaplewski](https://reader031.vdocument.in/reader031/viewer/2022032615/55a2ffd91a28ab1b4d8b45d3/html5/thumbnails/117.jpg)
Natural Urban Crop Total Natural Urban Crop TotalNatural 226 27 74 327 Natural 206 35 60 300
Urban 18 108 36 162 Urban 27 226 47 300Crop 89 36 286 411 Crop 68 39 193 300Total 333 171 396 900 Total 300 300 300 900
Natural Urban Crop Total Natural Urban Crop TotalNatural 25% 3% 8% 36% Natural 23% 4% 7% 33%
Urban 2% 12% 4% 18% Urban 3% 25% 5% 33%Crop 10% 4% 32% 46% Crop 8% 4% 21% 33%Total 37% 19% 44% 100% Total 33% 33% 33% 100%
Overall accuracy 0.69 kappa 51% Overall accuracy 0.69 kappa 54%
True value from all pixels
Reference class Reference class
Normalized value from all pixelsReference class Reference class
Map
cla
ssM
ap c
lass
Map
cla
ssM
ap c
lass
Normalization (MARGFIT) modifies error matrix so that margins are equal
117
Which error matrix is more accurate?