Mr. Nathan LowryHQ AFSPC/CEXIAFSPC GIO

Approaching Data Quality

USAF 2004 GeoBase Compass Conference 18 Aug 2004

2

Approaching Data Quality

Overview

• DoD Standards and Flexibility

• Extending Standards to meet needs

• Values and Objectives

• Spatial Data Accuracy

• Interpreting the SDSFIE

• Metadata Development

• Quality Assurance Procedures

• Consumption and Cost Effectiveness

3

DoD Installation Visualization Tool (IVT)

• Supports Base Realignment and Closure (BRAC)

• Explicit spatial, attribute, and metadata specifications

• Quality assured and auditedfor accuracy

DRAFT DELIBERATIVE DOCUMENT - FOR DISCUSSION PURPOSES ONLYDO NOT RELEASE UNDER FOIA

4

5

Standards and Flexibility

Advantages• Provides a common

interpretation of metadata and attribute values• Consistent• Comparable

• Allows implementation within a short time period

• Captures information of greatest interest to a specific audience

• “Measurable” quality

Disadvantages• Interpretations may not have

considered all alternatives

• Language may not be clear, concise, or easily understood

• May lack some information needed to communicate to a generic audience– Standard interpretation

may evolve over time

6

Extending Standards to Meet Needs

• Establish local, MAJCOM, or Service-level guidance to address immediate needs• Installation Boundaries• Generic uses and

distribution• Lower sensitivity risks

associated with BRAC• Strategic visualization:

• Address customer needs• Garrison mapping:

• Address generic needs

• Address data in many dimensions:• Completeness• Currency• Accuracy

• Positional Accuracy• Attribute Accuracy

• Geometric representation• Logical consistency • Metadata• Geospatial Data

Registration• File Format

7

Understanding Values and Objectives

• Values of the customer• Efficiency• Effectiveness• Quality of Service• Sensitivity of information• Perceptions on Cost

• Objectives of the customer• Intended uses of the data• Policy requirements• Timeline constraints• Desired end-state• Term of success

Will drive…• Level of detail needed• Level of accuracy needed• Amount and length of time

8

Measuring Spatial Data Accuracy

• Deductive estimate• Statements based on knowledge of errors occurring

during process steps, with assumptions and references to calibration tests

• Internal evidence• Tests based on repeated measure and redundancy

• Comparison to source• Graphic inspections, citing geometric tolerances and

registration methods used• Independent Source of Higher Accuracy

• “The preferred test for positional accuracy…”

From “Basic Implementation of SDSFIE, CADD-GIS Technology Center, ERDC, USACE, July 2004)

9

Measuring Accuracy Using the NSSDA

• Select well-defined points from

• Dataset being tested for accuracy

• Independent source of higher accuracy (ie. NSRS)

• Measure distances (x,y and/or z)

• Calculate statistics:

• Standard Error * Root Mean Square Error

10

So…

• Accuracy expressed using NSSDA =• Standard Error * Root Mean Square Error, or

• 1.7308 *

…in the horizontal plane, and

• 1.9600 *

…in the vertical.

(∆1)2+ (∆2)2

+ … + (∆n)n

n

(∆1)2+ (∆2)2

+ … + (∆n)n

n

11

Stating Accuracy Using the NSSDA

• “Tested to meet 0.181 meters horizontal accuracy at 95% confidence level."

• “Tested to meet 3.3 feet vertical accuracy at 95% confidence level

• “Compiled to meet 10m vertical and horizontal accuracy at 95% confidence level.”

12

Establishing Thresholds

• The critical value in the NSSDA• i.e.“…shall be tested meet or exceed 1m

horizontal accuracy at 95% confidence level.”• Required minimum value for accuracy

Value of Accuracy

Cost of Accuracy

1.010 0.1

•Project or agency defined standards for spatial accuracy•Feature-specific

•Utilities survey vs. biological sample site

13

Interpreting the SDSFIE

• Understand definitions• Read them…

• Understand structure• Entity Set, Class, Type,

etc.• Vendor-specific

• Understand relationships• Refer to data models

• Understand limitations of an evolving standard• Approaching

Comprehesive• Look for the “denotative

meaning” and intent• Recognize that there may

be occasional errors

• Browse by Alias

• Closest approximation to common names of items

• Alias to be shown in applications required by the GeoBase Architecture

• CIP written in CONOPS using alias names

14

Quality check the migration and add metadata

Populate the Access Database

(Migrate)

Final products (queries,

maps, etc.)

Generate a SDS-compliant

Access Database (Generator)

Build a filter containing the

Features (Filter Maker)

Create a correlation matrix between your data and SDSFIE

Browse the SDS to identify the

Features needed (Browser)

Identify the data to be

collected & stored in the

database

1 2

5

4

76

3 4

8

Step #

Step #

Implementing the SDSFIE

From “Basic Implementation of SDSFIE, CADD-GIS Technology Center, ERDC, USACE, July 2004)

15

The Correlation MatrixThe AFSPC CIP (Draft)

16

Metadata Development Strategies“Introduction to GeoBase Technologies” Metadata Development Approach (NGA)

Inventory• All source materials, prioritizing spatial data sets and related files for

documentation

Identify• Whose good will and cooperation you’ll need in order to gather and verify all

the information for your metadata

Select• Software that will meet your needs

Standardize• Your business rules and re-use components via “boilerplates”

Implement and Evaluate• Metadata maintenance and development progress

From “Introduction to GeoBase Technologies”, National Geospatial-Intelligence Agency (NGA) (DMS Course #4m/41-716)

17

Metadata Development Strategies

Margo Berendsen’s “Metadata for the Rest of Us” Approach

The Good News• There is really only one section of the Standard that is going to take a

significant amount of time:

Data Quality.• The other sections can usually

be recycled with minor changes.

The Bad News• If you do take the time to really look into your data's quality,

you'll discover you wish you hadn't• But look at it this way: better now than later

From “Metadata for the Rest of Us”, Margo Berendsen, Spatial Data and Visualization Center, University of Wyoming

2) Data Quality•Accuracy•Lineage (sources)•Process steps (procedure)

18

Metadata Development Strategies

Nathan’s “Pragmatist’s Guide to FGDC Metadata”

1) Gather and investigate all the information that you can on the data(documents, subject-matter experts, etc.)

2) At some point, recognize that YOU are the data “Expert”.

3) Write down (ie. in WordPad) what you feel is the most essential andrelevant information for “the next guy” to know in a coherent narrative.

THEN cut-and-paste from your narrative into the metadata editor.

19

Organize your informationOrganize your informationOrganize your information

Write your metadata fileWrite your metadata fileWrite your metadata file

Review your fileReview your fileReview your file

Have someone else reviewHave someone else reviewHave someone else review

ReviseReviseRevise

PublishPublishPublish

Six Stepsfor

WritingQuality

Metadata.

Metadata Development Strategies

From “Introduction to FGDC Metadata” Workshop,FGDC/NOAA-CSC, Denver Federal Center, Denver, CO, June 2004

20

Sense of Humor!

Chocolate

FGDCWorkbook

Metadata entry tool

Coffee

Metadata Development Strategies

21

• Get organized• Document your data as you go• Write so others can understand• Always review your document

• Keep your readers in mind• Write simply but completely• Document for a general audience• Be consistent in style and terminology• Define all acronyms• Avoid using jargon• Clearly state data limitations

Metadata Development Strategies

From “Introduction to FGDC Metadata” Workshop,FGDC/NOAA-CSC, Denver Federal Center, Denver, CO, June 2004

22

Quantitative and Qualitative Metadata

• Quantitative Measures– National Standard for

Spatial Data Accuracy (NSSDA)

– Tested with thresholds– “Tested to meet 0.5 meter

horizontal accuracy at 95% confidence level”

• Qualitative Value• Structure vs. Content

– Content driven

• What is good metadata?– Good writing…

23

Quality Assurance Procedures

• Product Assurance• SDSFIE Compliance• NSSDA Testing• Metadata Requirements

• Develop plans that assure standards for quality are met• Consistency• Repeatable process• Teamed Effort

• Checklist approach• Measurable• Meets USAF culture

• Independent Verification• Risk-reduction

• An evolving process…

• Process Assurance• Written procedures

• Process steps• Re-evaluate processes

24

Consumption and Cost Effectiveness

• Increase Consumption = • Increased Accuracy• Reduced Cost

• Benefits of Garrison Mapping• Increased Wing Capability• Increased quality for

Strategic requirements

Accuracy

Cos

t

10 20 30 40 50 60 70 80 90 100%

$

25

References

• Spatial Data Transfer Standard, USGS, 1988 • Geospatial Positioning Accuracy Standards (FGDC-STD-007-1998)• Content Standard for Digital Geospatial Metadata (CSDGM) Version 2.0 (FGDC-STD-001-1998)• FGDC Metadata Workbook Version 2.0 (FGDC-STD-001-1998) • USAF GeoBase Enterprise Architecture, Version 1.0, Headquarters Air Force Geo Integration Office, Jan 2003• USAF Garrison Mapping CONOPS, Version 2.0, Headquarters Air Force Geo Integration Office, Jun 2003• DoD Installation Visualization Tool, Quality Assurance Plan, Version 1.1, IVT Program Office, Headquarters Air Force Geo Integration Office, 31 Dec

2003• DoD Installation Visualization Tool Checklist, Version 1.1, Headquarters Air Force Geo Integration Office, Mar 2004• “Introduction to GeoBase Technologies”, National Geospatial Intelligence School (NGS), NIMA (DMS Course #4m/41-716)• “Error, Accuracy, and Precision”, Kenneth E. Foote and Donald J. Huebner, Department of Geography, University of Texas at Austin, 1995• “Managing Error”, Kenneth E. Foote and Donald J. Huebner, Department of Geography, University of Texas at Austin, 1996• “Basic Implementation of SDSFIE, CADD-GIS Technology Center, ERDC, USACE, July 2004• “Introduction to FGDC Metadata” Workshop,FGDC/NOAA-CSC, Denver Federal Center, Denver, CO, June 2004• “Train the Metadata Trainer” Workshop, FGDC/NOAA-CSC, Denver Federal Center, Denver, CO, June 2004• “Metadata Education Project: Data Quality / Error and Uncertainty”, University of Wyoming, (date and author not cited)• “Metadata for the Rest of Us”, Margo Berendsen, Spatial Data and Visualization Center,

University of Wyoming• “A Classification System for the National Standards for Spatial Data Accuracy (NSSDA)”, Gregory Tilley, VARGIS, LCC,(date not cited, available upon

request).• “Development of the National Spatial Reference System”, David R. Doyle, National Geodetic Survey (NGS), National Oceanic and Atmospheric

Administration (date not cited).• “How Good Are Your Maps?” , Bruce Joffe, GIS Consultants, GeoWorld, February 2002• “Map Error and Root Mean Square” , Joseph M. McCollum, USDA Forest Service, (date not cited)• “Metadata in Plain Language”, Peter N. Schweitzer, U.S. Geological Survey, Reston, VA• “Frequently-asked Questions on FGDC Metadata”, USGS http://geology.usgs.gov/tools/metadata/tools/doc/faq.html • Conversations with Mr. John Conroy, USGS/NMP Standards Team, August 2, 2004• Conversation with Mr.Joe Chumbley , CH2MHill, Colorado PLS, June, 2004• Conversations with Ed Riegelmann, Scott Beattie, and Joe Goutch, CH2MHill, 2003-2004• Conversation with Rick Johnston, AFMC Strategic GeoBase Coordinator, 2004