GIS Mapping of the New York GIS Mapping of the New York City Sewer SystemCity Sewer System

Diagram Geodatabase Design Topic OutlineTopic Outline

Entering spatial/non-spatial data   -  Brief Overview of Baker's process Entering spatial/non-spatial data   -  Brief Overview of Baker's process (scanning, digitizing and scrubbing)(scanning, digitizing and scrubbing)

  Linking spatial and non-spatial data  -  Indexing Application, Sewer Database Linking spatial and non-spatial data  -  Indexing Application, Sewer Database design (database structure, Feature Class, table attribute, relationship)design (database structure, Feature Class, table attribute, relationship)

Data validation -  QA process (Automated and detailed feature inspection) . Data validation -  QA process (Automated and detailed feature inspection) . 

  

Entering spatial data 

Types of DEP source Document • As - built • Contract• Interceptor Plans• Index Maps• Emergency Contracts Drawings• Infiltration and Inflow Maps (I/I maps)• Connection Book• Field Survey

Types of Non - DEP source document • Department of Transportation • MTA• Bridge and Port Authority • NYC Orthophotos • DCPLION• NYC LANDBASE• GEOSUPPORT• Sanbon - Property Maps

Methods of Data InputsMethods of Data Inputs

Scanning Scanning The process of translating hardcopy media into digital form that can be The process of translating hardcopy media into digital form that can be

recognized by a computer. Create permanent records. recognized by a computer. Create permanent records. DigitizingDigitizing

The method of data capture that involves the conversion of data in hardcopy The method of data capture that involves the conversion of data in hardcopy or raster forms into Digital vector formats. or raster forms into Digital vector formats. Digitizing produces features (pipes and manholes) used in a GIS. Digitizing produces features (pipes and manholes) used in a GIS.

* Features are digitized relative to the NYCMAP with 1 foot * Features are digitized relative to the NYCMAP with 1 foot horizontal horizontal accuracy accuracy

* Precision : Offset value (distance btw feature and the curb line). * Precision : Offset value (distance btw feature and the curb line). Pipe Length Pipe Length

Scanning differs from digitizing in that the entire pages of data or map sheet Scanning differs from digitizing in that the entire pages of data or map sheet are captured all at onceare captured all at once

Geo-referencingGeo-referencingThe process of defining the position of a geographic object relative to a The process of defining the position of a geographic object relative to a standard reference. Images with spatial data (I/I maps) were geo-standard reference. Images with spatial data (I/I maps) were geo-referenced. referenced.

Field SurveyField Survey

Entering non-spatial dataEntering non-spatial data

SScrubbingcrubbing The act of cleaning a surface by rubbing it with a brush and soap and water The act of cleaning a surface by rubbing it with a brush and soap and water

A meticulous process of reviewing and highlighting relevant attribute A meticulous process of reviewing and highlighting relevant attribute information for inclusion in the database. information for inclusion in the database.

* Cleaned for inconsistency * Cleaned for inconsistency

* Information retrieved are used to populate and link to their * Information retrieved are used to populate and link to their

respective feature.respective feature.

* Standardization of domain and ranges (spatial and attribute)* Standardization of domain and ranges (spatial and attribute)

Linking spatial and non-spatial data Linking spatial and non-spatial data 

Indexing Indexing

Creating a database of information for easy search and retrievalCreating a database of information for easy search and retrieval Indexing aid in filtering through a huge amount of information.Indexing aid in filtering through a huge amount of information.

* * Document IndexingDocument Indexing

A procedure that associates a source document with its respective street centerlineA procedure that associates a source document with its respective street centerline * * Feature IndexingFeature Indexing

A procedure that associates a source document with its respective feature or A procedure that associates a source document with its respective feature or

attributeattribute

Where multiply source have been linked the most recent document acts as the defaultWhere multiply source have been linked the most recent document acts as the default

Source.Source.

Document are stored using a directories or folders and subfolders by their borough Document are stored using a directories or folders and subfolders by their borough

location. location.

Relationships are established within the database. Relationship require ID.Relationships are established within the database. Relationship require ID.

Quality Assurance ProceduresQuality Assurance Procedures

QA WorkflowQA Workflow

The QA process consists of several phases that are defined in the The QA process consists of several phases that are defined in the procedures below:procedures below:

Check of digital data delivery and hard copy supporting documentsCheck of digital data delivery and hard copy supporting documents

Automated database checks using QA softwareAutomated database checks using QA software

Context check of plots and Engineering Review within the delivery area.Context check of plots and Engineering Review within the delivery area.

Feature inspection based on a randomly selected sample set using ANSI Feature inspection based on a randomly selected sample set using ANSI standardsstandards

Workflow Workflow

7. Conversion Issue Resolution and Communication

5. Accept delivery6. Reject delivery

Accept?YES

DeliveryComplete?

Accept?

Accept?

NO

4. Conduct Sample Feature

3. Conduct Context Check &Engineering Review

2. Conduct Automated Checks

1. Check Data Delivery Package

Figure 1: QA Flow Chart

YES

YES

YES

NO

NO

NO

Quality Assurance Data Validation Techniques

The QA process ensures that database design specifications are adhered to during conversion. The automated routine runs checks for data completeness, spatial inaccuracies, context and attribution errors, scheme and data structure and connectivity error. The process is divided into three broad groupings – Automated Checks, Context Checks/ Engineering Review and Feature Inspections Checks. The compliance level for automated checks is 100% while the compliance level for feature inspection is 95%.

Automated Checks

Automated checks consists of three types of data evaluations—object design, geometry, and connectivity checks. Table 1 summarizes the Automated Validation Checks. The Automated application runs in batch mode with pre-configured tables controlling the parameters. Most feature characteristics and attributes can be checked through automated routines.

Object Design Review all features to ensure that all attribute values match domain tables Review all features to ensure that all mandatory values are populated and only

permissible null values are populated in the database All unique values such as Facility ID are in fact unique both in a single delivery,

as well as in the entire database Compare the attributes with the various area polygons such as Grid_no, etc., to

determine if the attributes are correctly populated Check facilities with assigned linked annotation to assure they have at least one

annotation feature linked to them.

Geometry test Review all junction features with elevation to ensure that the To and From Review all junction features with elevation to ensure that the To and From

attribution are correctly populated per pipelineattribution are correctly populated per pipeline Review all field-surveyed points in the Geodatabase located in the x,y Review all field-surveyed points in the Geodatabase located in the x,y

coordinates captured in the fieldcoordinates captured in the field Review all features to identify any features that occupy the same x, y Review all features to identify any features that occupy the same x, y

coordinate (i.e. duplication of geometry).coordinate (i.e. duplication of geometry).

Connectivity Test. The connectivity tests will: Validate that all features meet the topology business rules (i.e., identify all

floating junction features and/or edge features that are not terminated with a coded junction feature).

Conduct an audit and generate a report of all junction features and list features that have 0, 1, 2, and >2 edges connected to them.

Per each junction type, generate a list of features that have 0, 1, 2, or > 2 edges connected providing the ability to individually review each feature.

Data Structure and schemaData Structure and schema Type of ValidationType of Validation Description Description Purpose/Notes Purpose/Notes

Spatial Attribute Check Spatial Attribute Check Grid No, Maintenance Yard,Grid No, Maintenance Yard,

Drainage District, Drainage District,

Treatment PlantTreatment Plant

Features are validated Features are validated based on there spatial based on there spatial location within thelocation within the

Quality Assurance Quality Assurance LayersLayers

Document Link and Feature Document Link and Feature Linked Annotation Check Linked Annotation Check

Verifies that the each Verifies that the each feature is linked to a source feature is linked to a source document and its related document and its related piece of annotation. piece of annotation.

Domain Validation Check:Domain Validation Check: *Variation between *Variation between deliveriesdeliveries

*Attribute Domain*Attribute Domain

Test for changes in defined Test for changes in defined

domain set. Detect attribute domain set. Detect attribute

values that fall outside of values that fall outside of

the legal values. the legal values.

Ensure consistency btw Ensure consistency btw deliveriesdeliveries

Attribute Name CheckAttribute Name Check Attribute Name CheckAttribute Name Check Verifies attributes/field Verifies attributes/field consistency between consistency between deliveries. deliveries.

An error with a coded An error with a coded value attribute error value attribute error could point to an could point to an uncontrolled editing uncontrolled editing environment. Example, environment. Example, digitized features digitized features should have a value of should have a value of “Active” for the “Active” for the Operational_State field. Operational_State field.

Null and Uniqueness Value Null and Uniqueness Value ChecksChecks

Attribute/fieldsAttribute/fields The nulls test checks for The nulls test checks for unpopulated values in an unpopulated values in an attribute. The uniqueness attribute. The uniqueness test seeks out duplicate test seeks out duplicate values for an attribute field values for an attribute field that is designed to be that is designed to be unique, such as Facility-IDs. unique, such as Facility-IDs.

Null values for an Null values for an attribute may signal an attribute may signal an uncontrolled data uncontrolled data processing procedure. processing procedure. Unique values for an Unique values for an attribute are critical for attribute are critical for referential integrity, referential integrity, especially for primary especially for primary key attributes.key attributes.

These tests are critical for sewer dataset because invalid values can cause symbology and analysis errors..These tests are critical for sewer dataset because invalid values can cause symbology and analysis errors..

Context and Attribute Context and Attribute Error DescriptionsError Descriptions

Type of ValidationType of Validation Description Description Purpose/Notes Purpose/Notes

To and From Pipeline To and From Pipeline Invert Elevation CheckInvert Elevation Check

FlowFlow Identify instances where Identify instances where the From[Inv_Elevation the From[Inv_Elevation of a pipe is lower than of a pipe is lower than the To_Inv_Elevationthe To_Inv_Elevation

Review all junction Review all junction features with elevation features with elevation to ensure that the To to ensure that the To and From attribution are and From attribution are correctly populated per correctly populated per pipelinepipeline

Manhole has two out Manhole has two out sewer linesewer line

Multiple ExitMultiple Exit Identifies instance were Identifies instance were manhole have more than manhole have more than one exit sewer on it one exit sewer on it downstream section. downstream section.

Check for only usually Check for only usually configurationconfiguration

Isolated Junction Check Isolated Junction Check (Network Junction, (Network Junction, Simple Junction)Simple Junction)

ConnectivityConnectivity Identifies isolated Identifies isolated features e.g. manhole features e.g. manhole and sewer structure and sewer structure nodes that have no pipes nodes that have no pipes connected to them.connected to them.

Ensure that all features Ensure that all features are within the networkare within the network

Elevation Check at Elevation Check at ManholeManhole

FlowFlow Compares incoming pipe Compares incoming pipe invert elevations to invert elevations to outgoing elevations at outgoing elevations at each manhole. each manhole.

Ensure that incoming Ensure that incoming pipes have higher or pipes have higher or equal invert elevations equal invert elevations than outgoing pipesthan outgoing pipes

No Outlet Sewer -No Outlet Sewer - ConnectivityConnectivity

(Dead End Check)(Dead End Check)The Check identifies The Check identifies sewer sections that end sewer sections that end abruptly therefore abruptly therefore obstructing theobstructing the

general effluent flow. general effluent flow.

Check Plots

Reviews sewer configuration by comparing the geo-referenced I/I maps and digitized sewers. This check is to make certain that no significant segments of the sewer system are missing. Connectivity and flow tracing will support this step. Other step include sewer length verification.

Engineering Plots

Baker Delivery Plots (Engineering Review) - performed by a team of DEP Engineers/Staff. The check results in identification of configuration, attribute, and annotation errors. Valid errors are entered in the QA application along with other recorded errors

Context ChecksContext Checks

Infiltration and Inflow Maps (I/I Maps): Infiltration and Inflow maps (I/I) were originally produced from as-built drawings.  They cover 14 drainage areas within New York City .  The I/I maps show the following sewer information:  sewer type, size, shape, invert elevations, flow direction, and year of as-built.  Because they have not been maintained, they are considered incomplete, and are generally inaccurate.  They should be used for reference only as a secondary source to be used for an understanding of the sewer system within an area.

Sample Feature InspectionSample Feature Inspection

This process involves a detailed inspection on a selected set ofThis process involves a detailed inspection on a selected set of

feature. The selection is chosen randomly and based on statisticalfeature. The selection is chosen randomly and based on statistical

principals that establish an overall confidence on the quality of the principals that establish an overall confidence on the quality of the

data delivered.data delivered.

The ANSI test consists of four components: The ANSI test consists of four components: Calculate batch size, Calculate batch size, Select general inspection level, Select general inspection level, Calculate sample size and make a random selection from the Calculate sample size and make a random selection from the

batch size batch size Summing the errors for acceptance/rejection of the batch.Summing the errors for acceptance/rejection of the batch.

Table 2 shows how the ANSI standard (American National StandardTable 2 shows how the ANSI standard (American National Standard

Institute) is used in the Sewer Project.Institute) is used in the Sewer Project.

Table 2: ANSI Sample Size Code Letters14

General inspection levelsS-1 S-2 S-3 S-4 I II III

2 to 8 A A A A A A B9 to 15 A A A A A B C

16 to 25 A A B B B C D

26 to 50 A B B C C D E51 to 90 B B C C C E F91 to 150 B B C D D F G

151 to 280 B C D E E G H281 to 500 B C D E F H J501 to 1200 C C E F G J K

1201 to 3200 C D E G H K L3201 to 10000 C D F G J L M

10001 to 35000 C D F H K M N

35001 to 150000 D E G J L N P150001 to 500000 D E G J M P Q500001 and over D E H K N Q R

Table is a copy of the American National Standards Institute: Sampling Procedures and Tables for Inspection by Attributes

Prepared by the American Society for Quality Control Standards Committee for the American National Standards Institute Committee

Z-1 on Quality Assurance

ANSI/ASQC Z1.4-1993

Lot or batch size Special inspection levels

Table 3. Single Sampling Plans for Normal Inspection

TABLE I - Sample size code lettersTable A-2: Single sampling plans for normal inspection (Master table)

Sampling Acceptable Quality Levels (normal inspection)

size Sampling0.010 0.015 0.025 0.040 0.065 0.10 0.15 0.25 0.40 0.65 1.0 1.5 2.5 4.0 6.5 10 15 25 40

code size AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe AcRe

letter

A 2 0 1 1 2 2 3

B 3 0 1 1 2 2 3 3 4

C 5 0 1 1 2 2 3 3 4 5 6

D 8 0 1 1 2 2 3 3 4 5 6 7 8

E 13 0 1 1 2 2 3 3 4 5 6 7 8 10 11

F 20 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15

G 32 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

H 50 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

J 80 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

K 125 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

L 200 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

M 315 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

N 500 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

P 800 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

Q 1250 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

R 2000 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22

Table is a portion of a copy of the American National Standard: Sampling Procedures and Tables for Inspection by Attributes

Prepared by American Society for Quality Control Sandards Committee for American national Standards Committee

Z-1 on Quality Assurance

ANSI/ASQC Z1.4-1993

Table 4: ANSI Interpolated Number of Errors Allowed per Sample Size for 95% Acceptance and Rejection Criteria

Sample Size Code Letter

Sample Size

Interpolated Reject Threshold AC 95% RE 95%

A 2 1 0 1B 3 1 0 1C 5 1 0 1D 8 1.4 0 1E 13 2.4 1 2F 20 3.4 2 3G 32 4.8 4 5H 50 6.8 6 7J 80 9.2 8 9K 125 10.6 12 13L 200 17.8 17 18M 315 22 21 22N 500 22 NA NAP 800 22 NA NAQ 1250 22 NA NAR 2000 22 NA NAS 3150 22 NA NA

Total Features

Batch Number

Table I Code Letter

Sample Size

Table II-A Allowable

ErrorsSewers, Force Mains 3,460 G1 L 200 17Manholes 2785 M1 K 125 12Sewer Structures, Disch Pts, Pump Sta, Detention Systems 345 P1 H 50 6

375 35

Table 5: Feature Count and Sample Size

Batch (A)Total Feature Errors

FeatureChecked

TotalAttribute

Errors

Attributes

Checked

(B)Weighted Attribute Errors

(C)Allowable Errors

Pass/Fail

G1 Sewer/Force Main 7 200 222 4600 9.7 17 Pass

M1 Manholes 3 125 82 2000 5 12 Pass

P1 Other Point Features

3 50 8 163 2.5 6 Pass

Results of Random Sample Check

Gro

up Feature Count

Sample Letter

Sample Size

Not to Exceed Errors

Feature Errors

Attribute Errors

Weight Factor

Weighted Attribute Errors

Manholes M1 4848 L 200 17.8 0 39.96 0.06 2.50Sewers/FM S1 6570 L 200 17.8 11 33.98 0.05 1.62Point Ftrs P1 1125 J 80 9.2 0 0 0.15 0.00

Manholes M2 5137 L 200 17.8 0 42.92 0.06 2.68Sewers/FM S2 7390 L 200 17.8 8 47.95 0.05 2.28Point Ftrs P2 1248 K 125 12.6 0 4 0.13 0.51

Table 6: Computation of Attribute Weights

A B C D E F

Attributes

to be

checked

Number of

features in

batch

Total

attributes

Average

attributes

per feature

Attribute

weight

(1/average)

B * C D/C 1/ESewer/Force Main

ss_pressurizedmain 18 4 72ss_gravitymain 21 3456 72576

Group G 3460 72648 21.00 0.05

Manholes

ss_manhole 16 2785 44560Group M 2785 44560 16.00 0.06

Other Point Features

generic_node 2 292 584pump_station 10 3 30ss_detention_system 9 1 9ss_dischargepoint 17 12 204ss_sewer_structure 8 38 304

Group P 346 1131 3.27 0.31

Data validation -  QA processData validation -  QA process

Context CheckContext Check Baker Delivery Plots Baker Delivery Plots Feature InspectionFeature InspectionCustomized Tool for inspection Customized Tool for inspection Feature Dialog Feature Dialog Attribute Dialog Attribute Dialog Sample ToolsSample Tools