gis data quality evaluator version 4.0 datalogic, inc. datalogic corporation 72 dartmouth avenue...
TRANSCRIPT
GIS Data Quality EvaluatorVersion 4.0
DataLOGIC, Inc.
DataLOGIC Corporation72 Dartmouth AvenueAvondale Estates, GA 30002404-289-4050www.datalogic-systems.com
DataQE – Data Quality Evaluator • 2
Overview
• Concepts• Types of QA Checks Performed• Evaluation Process• Future Development
DataQE – Data Quality Evaluator • 3
What is DataQE?
• GIS Data Quality Evaluation Tool• Evaluations based on user-specified rules• Focuses on evaluations of attribute fields
and values• Includes:
– Layer Checks
– Data Structure Checks
– Field Checks
– Query Checks
DataQE – Data Quality Evaluator • 4
What Can You Evaluate with DataQE?
Compliance• Is data compliant with established standards, such as a
Data Dictionary?
Completeness• Has required data been appropriately populated?
Suitability• Is a data source appropriate for a given purpose?• Does data make sense from a real-world perspective?
DataQE – Data Quality Evaluator • 5
Who Can Use DataQE?
• Anyone who needs to evaluate the quality or suitability of a GIS data layer or tabular data– Data Entry Staff
– Data Migration Specialists
– GIS Managers
– Quality Assurance Staff
– Analysts
DataQE – Data Quality Evaluator • 6
Requirements
• ESRI ArcMap 9.3 or higher• Data Layers or Tables in the following formats
are currently supported:– Geodatabases (SDE, File, Personal)
– Shapefiles
– Coverages
– Database Tables (SQL Server, Oracle, MS Access, MS Excel, DBF, CSV)
DataQE – Data Quality Evaluator • 7
The Data Evaluation Process
• Create Rule Sets containing the rules your data should follow
• Assign a Rule Set to the data source being evaluated
• Launch an Evaluation of the data source using the selected Rule Set
• View Evaluation Results and identify data errors
• Correct any errors as necessary using ArcMap or other editing tools
• Re-evaluate as necessary
DataQE – Data Quality Evaluator • 8
Layer Checks
• Data Source/File Name Check– Verifies that the name of the evaluated data source name matches
a list of valid names
• Layer Type Check– Verifies that the layer type of the evaluated data source matches
specified requirements (GDB vs. Shapefile vs. Coverage, etc.)
• Projection Check– Verifies that a layer’s current projection matches established
projection requirements
Verify commonly used properties within a spatial data source.
DataQE – Data Quality Evaluator • 9
Data Structure Checks
• Field Exists Check– Verifies that required fields exist in the data source being
evaluated
• Field Type Check– Verifies that each field is the correct type (Text, Date, etc.)
• Field Width Check– Verifies that each field is the correct width
Verify that a data source’s field structure matches appropriate requirements.
DataQE – Data Quality Evaluator • 10
Field Checks
• Required Value Check– Verifies that each record in a field contains a value
• Unique Value Check– Verifies that each record in a field contains a unique value
• List of Values (LOV) Check– Verifies that each record in a field contains a value that matches a
specified list of values (domain)
• Valid Range Check– Verifies that each record in a numeric field contains a value that
falls within a specified range
Verify that attribute values within a field match established standards.
DataQE – Data Quality Evaluator • 11
Query Checks
• Used to perform in-depth evaluations of a data source• Query Rules can be created using the DataQE Query
Builder, or copied from existing ArcMap or database queries
• Allows queries to be stored permanently in Rule Sets• Can be used to expand basic compliance evaluations to
determine whether data meets established Business Rules• Can be used by analysts to evaluate whether data is
appropriate for a particular purpose• Can be used to validate whether data “makes sense”, as
opposed to simply meeting data dictionary standards
User-defined rules based on customizable queries.
DataQE – Data Quality Evaluator • 12
Data Quality Evaluator Window
• View Rule Set Assignments and properties
• Review Evaluation Results
• Used to perform evaluations of selected data sources
Data Quality Evaluator
ResultsWindowResultsWindow
Data SourcesWindow
Data SourcesWindow
Rule Set ExplorerRule Set Explorer
DataQE – Data Quality Evaluator • 14
Rule Set Manager
• Create Rule Groups and Rule Sets
• Add/Edit rules within a Rule Set
• Rule Sets can be shared among users
Used for managing and editing Rule Sets
Rule Set Manager – Properties Window
Rule Set ExplorerRule Set Explorer
PropertiesWindow
PropertiesWindow
Rule Set Manager – Field Rule Properties
Rule Set ExplorerRule Set Explorer
Field RulePropertiesField RuleProperties
Rule Set Manager – Field Rules Grid
Rule Set ExplorerRule Set Explorer
Field RulesGrid
Field RulesGrid
DataQE – Data Quality Evaluator • 19
Query Checks
• Allow users to build custom Query Rules.
• Query Builder window helps users build a query statement that can be used to evaluate a field or combination of fields.
DataQE – Data Quality Evaluator • 21
LOV Manager
• Create/Import New LOV’s
• Add/Edit Values within an LOV
• Assign LOV’s to Rule Sets
Used for managing and editing Lists of Values (LOV’s) for Domain Checks
LOV Manager – Create LOV’s from Existing Data
Value ListsValue ListsCreate LOV’sFrom Existing
Data
Create LOV’sFrom Existing
Data
Evaluation Manager
• Allows creation of Evaluations with pre-assigned properties
• Easy selection of Data Sources and Rule Sets
• Evaluation Definitions can be stored & shared
• Allows definitions to be saved and performed at any time, individually or in batches
DataQE – Data Quality Evaluator • 25
Used to manage and edit predefined Evaluation Definitions to support batch processing
EvaluationDefinitions
Window
EvaluationDefinitions
Window
PropertiesWindow
PropertiesWindow
Evaluation Manager
DataQE – Data Quality Evaluator • 27
Future Development
Coming in future versions of DataQE:
• Support for Citrix implementations
• Password Protected Public Rule Sets
• Associated Tables/Orphan Records Check
• Metadata Check
• User Definable Field Rules– Business Logic Check
– Spatial Overlay Check