DMT 2008

The following was presented at DMT’08(May 18-21, 2008).

The contents are provisional and will besuperseded by a paper in the

DMT’08 Proceedings.

See also earlier Proceedings (1997-2007)http://ngmdb.usgs.gov/info/dmt/

Rescuing Legacy Digital Data: Rescuing Legacy Digital Data: Maps stored in Adobe Maps stored in Adobe

IllustratorIllustrator™™ formatformat

Andrew L. WunderlichAndrew L. WunderlichRobert D. Hatcher Jr.Robert D. Hatcher Jr.

Tectonics & Structural Geology Research GroupTectonics & Structural Geology Research GroupDepartment of Earth and Planetary SciencesDepartment of Earth and Planetary Sciences

and Science Alliance Center of Excellenceand Science Alliance Center of Excellence

University of Tennessee, KnoxvilleUniversity of Tennessee, Knoxville

IntroductionAs GIS databases become the standard for

storing spatial data, many organizations may be struggling to integrate “legacy” digital data into modern geospatial databases.

Map information that is stored in older digital data formats without spatial reference or attribution are in danger of being lost for future mapping and analysis purposes if the data are not converted into newer digital geospatial databases.

What are “legacy” data?• Digital file formats that do not natively accept

storage of geospatial information and feature attributes (Adobe Illustrator files)

• Other older, possibly obsolete digital file formats (programs have been discontinued, superseded by another format, etc.)

Also:• Attribute data in spreadsheet files, field

books, etc…• Analog map formats such as mylar or paper

Our Problem…

• Large collection of legacy geologic maps covering over 100 7.5-minute quadrangles.

• Maps are accompanied by a wealth of ancillary information that is not easily accessed. (station maps, field books containing detailed structural data and notes, photographs, etc.)

• Over 40 years of detailed geologic mapping (24K or larger scale) in the southern Appalachians is in danger of slipping into obsolescence...

Our Goal…

Transform and integrate these legacy data into geospatial databases to:– Prepare current and future analysis and

mapping projects (single quad maps, compilations, cross sections, etc.)

– Enhance the chances for long-term survival of the data

– Make the data available to the larger geologic community

Our Response…Develop a method to convert these legacy digital data

from Adobe Illustrator (and others) to a geospatial database that:– Makes the process modular, clearly defined, and

repeatable– Retains as many of the existing attributes in the

legacy files as possible– Saves time on repetitive tasks by automating

certain procedures with customized software tools

– Uses a template geodatabase for better interoperability with other geologic databases

Our Method…1) The original file is cleaned up and organized into

layers in Adobe Illustrator and exported to AutoCAD™ drawings. An image of the map is also exported to be used for attribution.

2) AutoCAD files are converted to shapefiles and spatially adjusted in ArcMap™ and appended into a geodatabase.

3) Geologic point data attributes that were not retained directly are either calculated from the feature (strike/rotation) or added to the features semi-automatically with the help of ArcMap utilities developed in-house (dip/plunge).

Our Method…4) The entire database is then checked for

topology errors, and if the map being processed is adjacent to existing data, the adjoining edges are reconciled.

5) A round of quality control measures are taken, including correcting mistaken attribution, after which any special considerations are addressed.

6) The finished database is then symbolized, labeled in ArcMap, and exported as a graphic for placement in a final map layout for editing and publication.

Method diagram

Step 1

• The original file is cleaned up and organized into layers according to feature type in Adobe Illustrator in order to maintain maximum attribution.

• Line and Polygon features are densified and straightened to remove Bezier curves

• Each topological type is individually exported to an AutoCAD™ drawing.

• A TIFF image of the map is also exported to be used for attribution of dip/plunge in Step 3.

Polygons layered by rock unitPolygons layered by rock unit

Illustrator file layering of polygons.

Faults and contacts layered by typeFaults and contacts layered by type

Illustrator file layering of lines.

Symbology layered by typeSymbology layered by type

Illustrator file layering of lines.

Add anchor points to Add anchor points to densifydensify featuresfeatures Simplify features to remove BeziersSimplify features to remove Beziers

Densify and simplify

Export selections to AutoCAD DWGExport selections to AutoCAD DWG Export entire map to TIFFExport entire map to TIFF

Export to TIFF and export to AutoCAD

Step 2

• AutoCAD files are checked in ArcCatalog for errors. (If found, back to Step 1)

• Selected feature classes within each AutoCAD file are converted to shapefiles.

• The shapefiles are spatially adjusted in an ArcMap edit session using the quadrangle boundary as a guide

• The adjusted shapefiles are saved and appended to their corresponding feature classes in a geodatabase

Check AutoCAD Check AutoCAD exports for content exports for content and attributesand attributes

Preview of AutoCAD classes in ArcCatalog. Check exports for content and attributes.

Export appropriate Export appropriate CAD feature class CAD feature class from each drawing from each drawing to shapefileto shapefile

AutoCAD class tree and export to shape dialog

Exported Exported shapefiles are shapefiles are loaded into loaded into ArcMap and ArcMap and spatially spatially adjusted by adjusted by snapping the snapping the corners of the corners of the footprint to the footprint to the corners of a corners of a projected projected quadranglequadrangleboundary boundary (HW_Footprint).(HW_Footprint).

Note the small Note the small residual error residual error (in meters)(in meters)

Spatial adjustment in ArcMap

Use Append tool to Use Append tool to load data from load data from adjusted line and adjusted line and polygon shapepolygon shape--files to the GDB.files to the GDB.

Use the Field Map Use the Field Map to match shape to match shape attributes to GDB attributes to GDB attributes.attributes.

Template GDB Template GDB with basic with basic classes to store classes to store geologic data geologic data from legacy mapsfrom legacy maps

ArcCatalog toolbox and Append to GeoDB showing tree

Step 3

• Symbology shapefile is processed in ArcMap and any unnecessary features are flagged as unneeded (operator intensive).

• Strike/rotation attribute is calculated for each feature using an ArcMap Field Calculator expression.

• Dip/plunge attribute (if applicable) is read from reference TIFF image and added to each feature semi-automatically by an operator with the help of an ArcMap utility developed in-house (AttributeFeatures tool)

Symbology shapefile needs to be processed to flag Symbology shapefile needs to be processed to flag unnecessary features so they are not appended to unnecessary features so they are not appended to databasedatabase

Use the Use the ‘‘LengthLength’’attribute to select attribute to select features that are features that are not needed.not needed.

Symbology cleanup in ArcMap 1

DonDon’’t delete anything! Use attributes to flag features. Mistakes cant delete anything! Use attributes to flag features. Mistakes can be fixed laterbe fixed later……

Use contrasting Use contrasting colors to help colors to help check for errors check for errors more quicklymore quickly

Symbology cleanup in ArcMap 2

With With ““Keep=1Keep=1”” features features selected (green), use the selected (green), use the Field Calculator on the Field Calculator on the ‘‘StrikeStrike’’ field to determine field to determine the azimuthal rotation the azimuthal rotation (see the azimuth example (see the azimuth example upperupper--right).right).

This number is used to This number is used to rotate the symbol later rotate the symbol later when it is pointwhen it is point--based.based.

P1

P2

P2

300°

N 60 W23 NE

23

0°

60°

Quadrantnotation:

30023

Azimuthalnotation:

Calculated Calculated azimuthazimuth

ArcMap Field Calculator for strike/rotation

Once symbology shapefile has been processed, it Once symbology shapefile has been processed, it can be converted to points and appended to the can be converted to points and appended to the ‘‘GeologicPointsGeologicPoints’’ class in the GDB.class in the GDB.

Some points will Some points will need to be need to be snapped to their snapped to their station locationsstation locations

Symbology convert to points in ArcMap

Use the custom tool Use the custom tool ‘‘AttributeFeaturesAttributeFeatures’’ to to quickly populate the quickly populate the ‘‘DipDip’’attribute for the selected attribute for the selected features from the image in features from the image in the background.the background.

User types in the number, User types in the number, hits Enter, and the tool hits Enter, and the tool automatically moves to automatically moves to the next feature.the next feature.

Attribute Features tool

After attribution, the points are symbolized and the rotation After attribution, the points are symbolized and the rotation checked. Note the inversion of the inclined bedding and checked. Note the inversion of the inclined bedding and overturned overturned antiformantiform symbols. These will have to be corrected.symbols. These will have to be corrected.

Symbolize and check for errors

Steps 4, 5, and 6

• The database is checked for topology errors and edge-matched to existing data if necessary.

• A round of quality control measures are taken and any special considerations are addressed.

• The database is ready for symbolization and labeling in ArcMap, and can be exported for placement in a final map layout for editing and publication.

Use topology to maintain relationshipsUse topology to maintain relationships EdgematchEdgematch adjoining quadranglesadjoining quadranglesTopology rules/Edgematching

QC and Special Items

• Quality control can consist of:– Checking tables for <null> or incorrect attributes

and fixing if found.– Comparing to reference map or checking field

books to confirm correct attributes.• Special cartographic or other items:

– Labels are converted to annotation so they can be positioned by hand.

– Cross-section lines, label points, fault names, fault teeth, etc. are created and stored.

– Any other items specific to a particular map.

QC/Cartographic features added

Map symbology and export

• The data are loaded into a template ArcMap layout and the layers are arranged appropriately.

• Layer files allow for implementation of standards and provide consistent replication of colors and symbols.

• DRGs, hillshades, or other data can be added to the map.

• Exports to PDF, Illustrator, or TIFF for placement in a layout for print or editing.

Map symbology labeling etc

Final map and labels in ArcMap

Finished poster

Final Thoughts

• Modular aspect makes it easier for multiple operators to work simultaneously.

• Method is less labor intensive than scratch digitizing, but requires greater care by operator not to lose or change data.

• Process can be tweaked to work with other input formats and different outputs (SDE, ArcIMS, etc.)

Future Developments

• Find a way to link CAD annotation to features to speed attribution.

• Could be automated further if AI files could be dissected programmatically.

• Other suggestions?

Thank you for your time Thank you for your time and attention!and attention!

Questions?Questions?