2017-09-25

1

Spatial Data Management I

GEOG 300, Lecture 6

Dr. Anthony Jjumba

1

Data Sources and Input Historic datasets and maps

Modern collection methods

2017-09-25

2

The Good Old Days

The Bad Old Days Digitizing centuries of hand-drawn maps…

Guess who got to do this job??

Prisoners

GIS Techs

Students!

Tedious and painstaking

2017-09-25

3

Digitizing

Digitizing is done in two ways:a. Tracing lines on maps initially using a tablet with map taped down, or

onscreen / ‘heads-up’ (= copying a map) – > 1995

Digitizing Procedure Lines = connected points

Manual point selection

Timed point selection

Interval point selection

http://forum.imagej.net/t/digitizing-contour-map/118

2017-09-25

4

Simplifying Lines Each vertex has a storage cost

How much is enough? Too many?

If too many, simplify in post process

Point remove: maintain essential shape

Bend simplify: maintain “important” bends

http://pro.arcgis.com/en/pro-app/tool-reference/cartography/how-simplify-line-works.htm

Digitizing Costs Labor $

Technicians (high)

Laypeople (medium)

Prisoners (presumably lower but not low)

Equipment $

$500-5,000 for a digitizer

Time

+/-1 boundary per minute

2017-09-25

5

Digitizing errors

9Source: Caitlin Dempsey, GIS Lounge

Sources of Error Precision:

If points +/- 25m on creation

similarly +/- 25m error introduced on digitization

Conceivably 50m total error

Accuracy:

Paper may have shrunk, stretched or torn

Symbols rearranged to prevent overlap

Map sheet boundaries

Human boredom, fatigue, humor or malice

2017-09-25

6

Input Error: Nature of the Beast Does not cause error messages in analysis

Outlier analysis sometimes catches mistakes

Easily goes unnoticed until publication

12

2017-09-25

7

13

Digitizing Tables in Modern Practice

Take a close look at Scott’s workbench…

2017-09-25

8

Scanners and Cameras High resolution raster

8,000X10,600 pixels (this model) https://www.youtube.com/watch?v=G7MjUrReUM4

http://www.library.unt.edu/digital-projects-unit/scanners-and-scanning-systems

Automatic Feature Recognition ArcScan toolbar in ArcMap

Automated, semi-automated, manual

Must be georeferenced

Toolbar:

2017-09-25

9

Modern Digitizing Error +/- 0.1mm precision from digitizing table

Difficult to confirm questionable points

Error potential depends on scale

0.1mm on 1:10,000 map less than 1:1,000,000

+/- 0.025mm precision from scanning

Less potential for user bias

Can be checked later if issues arise

Superior precision + accuracy

Lower operational cost

b. Onscreen from digital photography ('first hand data') – post 1985

2017-09-25

10

Air Photo Interpretation Stereoscopic Imagery

Contour lines

Slope angle

Elevation + Height

http://www.nrcan.gc.ca/earth-sciences/geomatics/satellite-imagery-air-

photos/satellite-imagery-products/educational-resources/9395

http://www.geog.ucsb.edu/~jeff/115

a/lectures/stereoscopy_and_height

_measurement.html

Modern Stereoscopy Done on desktop computers

Setups vary

Far less headache-inducing

2017-09-25

11

Air Photos for Stratified Sampling• Looking for distinct, uniform areas

• Crown size (age), harvest history

• Hardwoods (gray) and softwoods (green)

Stratified Sampling Generate sample points randomly

X points per area, e.g. 1 point every 3 hectares

Each point tied to polygon = unique stand

2017-09-25

12

Stratified Sampling: Population

A

BC

D

E

Digitising – editing is still needed: updates and errors

Coordinate locations are based on underlying georeferencing e.g. NAD 1983

Edits: e.g. adding new features, modifying existing features, creating a new layer

ArcEdit: https://www.youtube.com/watch?v=6dY3x-5qX6U

2017-09-25

13

Snapping Automatic connection to other features

Any features, selected features, feature class

Same feature class (roads)

Prevents slivers and disconnects

User-defined tolerance, radius… https://blogs.esri.com/esri/arcgis/2010/09/20/using-snapping-effectively-in-arcgis-10/

Aside: Toolbars

Snapping, Editing, GPS,

Georeferencing, Draw,

Scan, Advanced Editing, etc.

plus downloaded add-ins

Access via:

Customize>Toolbars

2017-09-25

14

Autotrace Snap an entire line to another line/feature

Follows established border precisely

Or follows at an offset

http://www.lutraconsulting.co.uk/blog/2014/02/11/finger-tired-from-digitising-try-autotrace/

https://blogs.esri.com/esri/arcgis/2010/11/04/getting-

the-most-out-of-editing-in-arcgis-10-tracing-features/

File Conversion

FME Universal Translator

GIS Lab has a license

ArcMap File formats

Read-only

Read + Write

Raster: http://desktop.arcgis.com/en/arcmap/10.3/manage-data/raster-and-images/supported-raster-dataset-file-formats.htm

Vector: http://desktop.arcgis.com/en/arcmap/10.3/manage-data/datatypes/about-geographic-data-formats.htm

QGIS: https://docs.qgis.org/2.2/en/docs/user_manual/working_with_vector/supported_data.html

2017-09-25

15

GPX File Conversion GPX is a standard GPS file format

QGIS reads it directly

ArcGIS has one intermediate step

Conversion tools>From GPS>GPX to Features

Input from GPS, output to shapefile

Rubber Sheeting Distortion to match-up boundaries

Precision + accuracy sacrificed

Best to re-check imagery

http://hds.essex.ac.uk/g2gp/gis/sect43.asp

2017-09-25

16

Keyboard Data Entry

Spreadsheet --> Attribute Table

• Spatial reference: GPS– GPS point # = row ID # (“Join” function)

• Row ID links spatial and tabular data– Field data usually entered in Excel or similar

– GPS data straight to Arc

Keyboard Data Entry 2• Coordinates + all data in spreadsheet

• Geographic data, no projection (unless…)

2017-09-25

17

Data Dictionaries (1) Trimble and other survey/map grade GPS

Establish database design first

Features may be culverts, bridges, signs, poles…

In this case, poles:

http://support.windenvironmental.com/knowledgebase.php?article=106

Data Dictionaries (2)

Populate attributes while collecting points

Takes ~90 seconds to average enough points

Data entry taken care of at no additional time cost

Attribute table ready to use

Requires proper prior planning

http://support.windenvironmental.com/knowledgebase.php?article=106

2017-09-25

18

One Bear’s Movement >

• Remote transfer

• Proximity transfer

• Direct transfer

Data Acquisition in 2016

Data BC

ESRI Open Data

GeoDiscover Alberta

BC MoFLNRO

GeoGratis

Prince George

Google

Spatially referenced 99% of the time

Most data publicly available (no $$)

2017-09-25

19

Canadian Data Sources

Canada: https://www.mcgill.ca/library/find/maps/geospatial-online

BC: https://catalogue.data.gov.bc.ca/dataset

Alberta: https://geodiscover.alberta.ca/geoportal/catalog/main/home.page

Saskatchewan: https://www.isc.ca/Pages/Content%20Gallery/GeoSask.aspx

Manitoba: http://www.manitoba.ca/iem/geo/gis/index.html

Ontario: https://www.ontario.ca/page/land-information-ontario

New Brunswick: http://www.snb.ca/geonb1/e/DC/catalogue-E.asp

Nova Scotia: https://geonova.novascotia.ca/

Quebec: https://www.mcgill.ca/library/find/maps (Gov’t data in a non-ESRI format)

Yukon: http://www.geomaticsyukon.ca/

Northwest Territories: http://www.geomatics.gov.nt.ca/dldsoptions.aspx (must register)

Nunavut: http://ntilands.tunngavik.com/maps/

U.S. and Other Data Sources

NASA: http://data.giss.nasa.gov/

Census Data

US Geological Society

Wikipedia’s List of GIS Data Sources

2017-09-25

20

Searching the Web: Keywords Google tips and tricks

Shapefile British Columbia Wildfire

Publicly Available KMLs Sometimes Google Earth is all you get

It’s enough…

▪ It comes in like this:

▪ Right click Polygons>File>Export data>Export as shapefile

▪ Result: usable shapefile/attribute table (turn off extra items)

2017-09-25

21

Shapefiles One “shapefile” = 3 or more files

.shp: shape format/geometry

.shx: shape index format (file navigation)

.dbf: attribute data (the ‘spreadsheet’)

.prj: projection data

.sbn, .sbx, .fbn, .fbx, .ain, .aih, .ixs, .mxs, .atx,

.shp.xml, .cpg, .qix: other formatting files

ALL HAVE TO MOVE TOGETHER

Geodatabases Layers in one place, ready to use + move

Pros: functionality, transferability within Arc

Cons: proprietary format = unreadable

Some newer software is getting around that

2017-09-25

22

Shapefile/GDB Layer InputAdd data from file

Polygon

Line

Point

Table

Raster

Geodatabases

Shapefiles (+text)

Shapefile/GDB Layer Input

Add data from ArcGIS Online

2017-09-25

23

Shapefile/GDB Layer Input

Catalog Tab

Inside ArcMap

Click + Drag to

Table of Contents

ArcCatalog

Click + Drag to

ArcMap Table of

Contents

Add using ArcCatalog or Catalog

Tab

WMS Layer Input

Add data->GIS Servers->add WMS Server

“Get Layers”

2017-09-25

24

Layer Files Layer file stores symbology

Where the files are

How to display them

Which attributes to display/hide

Does NOT include the data itself

Add to map just like a shapefile

Paper Towns: False Input Agloe, New York

Copyright “trap”

Agloe General Store

Beatosu (Beat OSU) and Goblu (Go Blue)

https://www.buzzfeed.com/krystieyandoli/welcome-to-the-agloe-general-store-come-back-soon?utm_term=.gy73AQXmy#.xdyE4mx8j

2017-09-25

25

Summary Digitizing tables are mostly out

Scanning, raster to vector is in

Keyboard data entry is still big… always?

File conversion remains critical

Digitized in 2002, file format…? Use FME

Countless data sources available

Input gets easier by the day

BUT is still the worst source of error