The Yale Map Collection

GIS Workshop #3

Finding GIS Data & Preparing It for Use

Census Data Download and Import into ArcGIS

Goals of this Tutorial: • To ensure that you can download both cartographic and demographic

data from the U.S. Bureau of the Census in a form that can be imported into ArcGIS.

• To ensure that you know how to join attribute data to cartographic data.

1) First, if you are using Internet Explorer, you must add the census sites to

your trusted sites list to allow downloads and pop-ups for this exercise.

2) In Internet Explorer, go to Tools>Internet Options and click on the Security Tab.

1

3) Highlight the green Trusted Sites icon, and click the Sites… button.

4) Uncheck the “Require server verification…” radio button.

5) Add census.gov and factfinder.census.gov to the Trusted Web Sites list.

6) Click OK twice to exit the Security setting dialog.

Downloading and Pre-processing Census Tract Data from the Internet

7) Create a directory called Census_Data_Download in your working directory (it is always a good idea to place your working directory on the C: Drive, and without spaces or special characters in the folder name).

8) Go to www.census.gov.

9) Click on “Geography”.

2

10) Click on “Census 2000 Geographic Products”.

11) Click on “ Census 2000 Map Series ” under the Census 2000 Maps heading.

12) Click on “BOUNDARY FILES.”

13) Click on the “Download Boundary Files” icon on the left hand side.

14) Click on “2000” next to the words “Census Tracts.” (Note that you are provided with MANY possible Census Boundary File choices on this page. However, the census does not provide access to the Block level boundary file from the website, although these can be obtained from the ESRI Data & Maps CD Set, or from the ESRI Download Website).

15) Scroll down to “Census 2000: Census Tracts in ArcView Shapefile (.shp) format.”

16) Click on “Connecticut - tr09_d00_shp.zip (220,667 bytes).” When prompted, save tfile to your \Data\Shapefile folder.

his

17) Next navigate to the \Data\Shapefile folder, right-click on the file you just downloaded and “Extract All.”

18) Make sure you have the three files with the extensions .dbf, .shx, .shp.

Census Tracts 2000

Subsetting the Census Boundary File to an Area of Interest (AOI)

The next step is to extract New Haven county census tracts from Connecticut state data using the FIPS (Federal Information Processing Standard) code attribute.

3

1) Open ArcMap with a new empty map and add the Connecticut census

tracts to your map.

2) Save your map in your Census_Data_Download directory as “Census_Data_Download.mxd.”

3) Go to the FIPS PUB 6-4 Lookup (http://www.census.gov/geo/www/fips/fips65/index.html) website to find out which FIPS code is associated with New Haven County (in this case, its 009).

4) In the Main Menu of ArcMap, Go to Selection>Select by Attributes.

5) Enter (or click on the appropriate Field name, operator, and value) "COUNTY" = '009' in the query dialog box. Click Apply and then Close.

6) The census tracts for New Haven County will be highlighted.

7) Right click on tr09_d00 layer in the Table of Contents on the left and Select Data>Export Data.

8) The Export: Selected Features option will be the default, since there is an active selection in the Map Document.

9) Browse to your working directory and Save the Export Shapefile as New_Haven_Tracts.shp.

4

C10) lick OK.

11) lick Yes when prompted to add the new shapefile to the map as a layer.

12) ight-Click on the tr09_d00 layer and choose Remove to remove it from

13) on the Full Extent

C Rthe current project.

Click button to zoom into New Haven County.

14) ight-Click on New_Haven_Tracts layer and Open the Attribute Table.

15) otice that you have 184 features. Notice that you have almost no

16) otice that you do have a column called Tract. This is going to be our Key

17) ave your map.

18) your data looks like the picture below… Congratulations!, you got the

R Nattribute data, other than numerical identifiers. NField that we use to join attribute data to our geography. S Ifcartographic data that you need.

5

Click Here

Click Here

1) Go to

Downloading and Pre-Processing Attribute Data

In this section we are going to download attribute data that pertains to the population that lives in the census tracts for which you downloaded boundaries in the previous section. The attribute data that we want is the racial split of the population. After we download the data we are going to edit it Excel in order to import it into the GIS, and we will do it in such a way that we can assign each census tract its appropriate population/race values.

http://www.census.gov/

2) Click on “American FactFinder” on the

left side column.

3) Click on “get data” under Decennial Census.

4) Make sure the radio button for “Census 2000 Summary File 1 (SF 1) 100-Percent Data” in the middle of the page.

5) In the “Select from the following” column in the right side of the page, select “Detailed Tables.”

6) In the new page under “Select a geographic type” choose “Census Tract” from the Dropdown menu.

7) Select “Connecticut” from the state dropdown menu.

6

8) Choose “New Haven County” from the county dropdown menu.

9) Under “Select one or more geographic areas…” menu select “All

Census Tracts” and then click the “Add” button.

10) Highlight the first “Census Tract 0” item and click Remove to remove it from the list. Click the “Next” button.

11) In the “Select one or more tables…” field choose table “P3. Race” and click on “Add”. Then click “Show Result”

You now have a table of attributes

where each column is a census tract and each row is a different racial attribute. The page shows only the first 10 census tracts, but you can scroll to the “next” page if you want to see more. We won’t be using all of the attributes (rows), but we will need to download the entire table and then cut out what we don’t need.

However, we are still missing one important element. We need a common identifier for each census tract that we can use to join this table with our table in the GIS.

12) Under “Options” on the top of the page choose “Show Geographic Identifiers”.

Now you see two tables of information

(if you scroll down). The first table shows all of the geographic identifiers for each tract, and the second table is the table of racial attributes that we saw before. We are going to download both of these tables so that we can edit them.

13) Under the “Print/Download” at the top of the page, choose “Download” to start downloading the table.

7

14) When the download table opens, scroll down to the section with the “Database compatible (data rows only)” heading:

a. Choose the Comma delimited (.txt).

b. Uncheck the “Include descriptive data element names” box..

c. Before you download the data set you have created, you should click on Technical Documentation (PDF) link, and save the file to your working folder. This PDF file continas information about the SF1 Data, including how the data is collected and a Data Dictionary.

d. Click “OK” to download the data file.

15) When the Save File dialog opens, Save the “output.zip” file to your \Data\Tables folder.

16) Browse to the folder you saved the output.zip file in and extract it to that folder. This should result in four files:

1. dt_dec_2000_sf1_u_data1.txt 2. dt_dec_2000_sf1_u_geo.txt 3. dt_readme.txt 4. readme_dec_2000_sf1.txt

8

Cleaning Up the Attribute File

17) Start Microsoft Excel and go to File>Open.

18) Browse to the folder that you extracted your attribute data to.

19) Change the “Files of Type” drop-down to “All Types” and Open the dt_dec_2000_sf1_u_data1.txt file.

20) In the Text Import Wizard, change the ‘Original Data Type” to Delimited, then click Next C

.

21) heck the “Comma” radio button and click Finish

22) lick on the A column e

23) croll to the right and ct

24) epeat the above step to

25) epeat the above step for

26) the Main Menu, select Save As.

27) rowse to the folder where you extracted the data (if it is not already

28) lick Save and Yes when warned about changing the Data Type.

Cidentifier at the top of thtable. S

Column Header

Shift-Click on X to seleall the columns from A to X. Right-Click on the X and select Delete. Rremove columns B through AO. R

(Right-Click here to delete)

columns E through W. In Bthere), and change the “Save as Type” drop-down to CSV (Comma Delimited). C

9

29) Close Excel and Click NO when prompted to save changes.

Openi g the Attribute Table in ArcMap

n

1) Return to ArcMap

2) Click the Add Data button and browse to the folder that you extracted e Census Attribute Data to.

3) data1.csv file to ArcMap.

ource Tab. This because you have added data that has no explicit geographic display.

reparing the Key Field in the Tract Boundary file for Joining to the Attribute File

ings the government does the census data is not perfect and ust be altered before it can be joined. The problem is that the TRACT records

in t

ght-Click on the dt_dec_2000_sf1_u_data1.csv table and OPEN it.

able.

3) o the right in both tables and find the TRACT field in each one.

re recorded with 6 significant places, while in the New_Haven_Tracts

The exdt_dec_2000_sf1_u_data1.txt table lies in the way that the Census Bureau

hen a Census lly

th Add the dt_dec_2000_sf1_u_

4) Notice that your Table of Contents View changes to the Sis

P

Like many th

mhe two separate tables are not formatted in the same way. We will need to

alter these records to provide ourselves with identically formatted and recorded records

1) Ri

2) Right-Click on the New_Haven_Tracts layer and Open the Attribute T Scroll t

4) Note that in the dt_dec_2000_sf1_u_data1.txt table the TRACT entriesatable the TRACT entries are recorded either with 4 or 6 significant places.

planation for the way the TRACT number is recorded in the

creates new Census Tracts in the face of increasing population. WTract becomes larger than the Census finds is appropriate, that Tract is typica‘split’ into two Tracts. When this happens, the Tract Number is appended with a suffix of 01, 02, 03, etc… (Depending on the number of times a Tract has been split).

10

What we need to do is create a Field in one of the tables that matches exactly a Field in the other table. These two identical fields can then be used to j

te Table, click on the Options button and select Add Field.

6) Y_TRACT, and give it the Type: LONG INTEGER.

oin the two datasets. In this case, there is a field in the Tracts Boundaryfile called NAME, in which the Census Tracts are recorded with the suffixes appearing after a decimal point. This means that we can simply multiply thisfield by 100 to arrive a value that corresponds to the Tract values in the dt_dec_2000_sf1_u_data1.txt table. 5) In the New_Haven_Tracts Attribu

Name the new field KE

11

7) Right-Click on the Field Header for the new KEY_TRACT field and Select Calculate Values.

8) In the Field Calculator, enter the following argument: 100 * (CDbl ( [NAME] )) In this case, the field NAME, in the New_Haven_Tracts table, is formatted as a STRING so that we must convert the STRING value in the field to a Double Value (Using the CDbl() function) before we can multiply it by 100. Click OK to calculate the new values for the KEY_TRACT field.

9) You can now close both Attribute Tables, if you have not already.

Joining the Attribute Data to the Boundary File

1) Right-Click on the New_Haven_Tracts layer in the Table of Contents and Select Joins and Relates>Join…

2) Assign the values to the Join Data dialog box as shown below:

a. Join attributes from a table b. …join will be based on: KEY_TRACT c. …the table to join…: dt_dec_2000_sf1_u_data1.csv d. …field in the table to base the join on: TRACT

3) Click OK to apply the Join.

4) Open the Attribute Table for the

12

New_Haven_Tracts layer and note the attribute data has now been joined to the boundary file.

If we would like to make the attribute data a permanent part of the boundary file, we can do so by exporting the Joined layer to a new shapefile.

5) Right-Click on the New_Haven_Tracts layer and go to Data>Export Data…

6) Make sure Export:All features is selected. Browse to your working folder and name the Export shapefile New_Haven_Tracts_Attribs. Click OK and Select Yes when prompted to add the layer to your map layout.

7) Open the Attribute Table of the new layer and note that the New_Haven_Tracts prefix has been removed from the field names in the Attribute Table. This is because the data is no longer “joined,” but is now part of the shapefile we have created.

Finally The procedure outlined here can be used to download any of the census boundary files and associate those boundaries with census attribute data, except in the case of census blocks. The census block boundary files can be downloaded from the ESRI Census Data Download website. ESRI provides the boundary and attribute data separately, just as the census does; however, the ESRI Census Data contains geographic identifiers that have already been altered to make them comparable for joining attributes to boundaries.

Additional Suggested Reading:

• Brewer, C., and T. Suchan. Mapping Census 2000: The Geography of U. S. Diversity. ESRI Press, 2001.

• Bureau, U. S. C. Summary File 1 Technical Documentation: 2000 (2003) • Bureau, U. S. C. Census of Population and Housing, Summary File 3:

Technical Documentation (2000) • Bureau, U. S. C. "Census 2000 Basics." Washington DC: US Government

Printing Office (2002)

13

Downloading Census & Other Data from the ESRI Census Data Website

1. Open your Web Browser and go to http://arcdata.esri.com/data/tiger2000/tiger_download.cfm

2. Select Connecticut from the drop-down menu, or the image map.

3. On the resulting page, under “Select by County,” Select New Haven.

4. Click on “Submit Selection.”

5. On the resulting page, select the checkboxes next to the items: Census Blocks Census Block Demographics (SF1)

6. Scroll to the bottom of the page and Click on the “Proceed to Download” button.

7. Your dataset will be assembled into a single *.zip file and you will be presented with a new page.

8. Click on “Download File” button. Note: You may need to add esri.com

14

to your trusted sites list, just as you did in the first part of this tutorial.

9. When prompted, Browse to your C:\Temp\initials folder, create a new folder called ESRI_Census and save the file there.

New Folder Button

10. Once the download has Completed, Browse to the folder where you saved the file.

11. UnZip the downloaded file and you should find the you have 2 new Zip files and a readme.html.

12. UnZip both of the resulting files into the same folder you have downloaded to.

13. You will now have four new file, three of which make up the Census Blocks Shapefile for New Haven County (tgr09009blk00.shp, etc…) and one which is the table containing the Census Block SF1 attribute data for the entire state of Connecticut (tgr09000sf1blk.dbf).

Joining the Attribute File to the Boundary File

1. Open ArcMap, or click on the New Document button to create a “New Empty Map.”

15

2. Use the Add Data Button to Open the Add Data dialog.

3. Browse to the folder where you saved and unzipped the files from the ESRI Census site.

4. Hold down the Ctrl-key and select both files, as shown on the right. Click Add to add them to your Map Document.

Note that you may be presented with a warning about the fact one of the layers you have added is missing spatial reference. This is because the Census Block shapefile you downloaded does not have a projection “explicitly defined.” This will not cause many problems in ArcGIS 9.1 and earlier because, when the coordinate values that record the positions of the points, lines and polygons of a shapefile fall within the normal range of Latitude (0 to 90 degrees) & Longitude (0 to 180 degrees) coordinates, these versions of ArcGIS assume that the shapefile is in Geographic Coordinate System (Lat/Lon) and will act as if the projection has been defined. This is no longer the case in ArcGIS 9.2, so you should get in the habit of defining the project for datasets, now, so that the lack of projection definition is not propagated through your collection of derivative shapefiles, as you subset and create new shapefiles from the initial file.

16

Defining the Spatial Coordinate System of Your Data

1. Right-click on the tgr09009blk00 layer and Open the Properties dialog.

2. Click on the Source Tab and note that the Coordinate System is GCS_Assumed_Geographic_1, and the Datum is NAD 1927. The “assumed” part of this item means that ArcGIS is assuming the projection information, based upon the range of the coordinates in the shapefile.

3. Close the Properties Dialog Box.

4. Open the ArcToolbox .

5. Click on the Search Tab at the bottom of the Arc Toolbox Panel.

6. Enter “define projection” as your search term and cthe Search button.

lick

7. Double click on the first item, which is the Define Projection Tool from the Data Management Toolbox.

8. Select the tgr09009blk00 layer as the Input Dataset.

9. Click on the Select Coordinate System

button.

10. In the Browse for Coordinate System dialog, browse to Geographic Coordinate Systems>North America.

11. Select the North American Datum 1983.prj and Click Add.

17

12. Click OK on the Spatial Reference Properties dialog to apply the selection.

13. Click OK on the Define Projection Tool to apply the definition.

In the Dialog box that shows the progress of the ArcToolbox tool application, you will likely see a warning that there is a “Datum Conflict between the Result and Map.” This is because the Map Document and the layers it contains can have different projections. ArcMap will usually successfully do an on-the-fly projection to overlay data properly. This is not always the case and, in fact, there is a particular NAD 1927 to NAD 1983 shift problem that causes many headaches. In the case of this Map Document, ArcMap Assumed the coordinate system was Lat/Lon with a Datum of NAD 1927. When the data layer was projected to NAD 1983, this Datum Shift problem triggered the warning message. You should redefine the Map Document Coordinate System, to avoid overlay problems.

14. Right-click on the Layers item at the top of

the Table of Contents, and open the Properties dialog box.

15. Click on the Coordinate System Tab.

16. In the “Select a coordinate system” panel, browse to Predefined>Geographic Coordinate Systems>North America.

17. Select the same North American Datum 1983.prj that you chose for the layer definition.

18

18. Click OK to apply the change.

Other Data Downloads and Preparation Skills

1. If you have not already, go to http://www.library.yale.edu/MapColl/gis_workshop_materials.html and download the Data file for the “Finding Data…” workshop to the C:\Temp folder you have been working in.

2. Unzip the downloaded file to the C:\Temp folder. Downloading from the Seamless Data Distribution Website The Seamless Data Distribution Website is maintained by the USGS and is the primary clearinghouse for raster data produced by the USGS, including ortho-imagery, Digital Elevation data, mosaics of Landsat imagery and more. Once familiar with the interface for the Seamless site, you should be able to successfully navigate and use many other ArcIMS-based data sites, as they are based upon the same architecture. In this exercise you will download elevation data for the New Haven, CT area.

1. Go to the USGS Seamless Data Distribution Website at http://seamless.usgs.gov/website/seamless/ and click on the “View and Download United States Data,” link at the right side of the page. Wait for the Map to load.

19

2. By default, the Zoom Tool will be active. Use it to zoom into New Haven, CT. This may take several Zooms.

3. On the right side of the Seamless Application, look for the “Download” tab and activate it.

Zoom Download Tab

Define Download

Area

Elevation

4. Scroll down the list (noting the available downloads), find the Elevation

Group and expand it (if not already) by clicking on the triangle next to the word Elevation.

5. Make sure the 1” NED item is checked, and that no other items are enabled. NED is the National Elevation Dataset, which is a raster dataset that describes the elevation at any given point, at several different resolutions (in this case ~30 meters). This type of data is commonly referred to as a Digital Elevation Model, or DEM.

20

6. Now click on the 1” NED layer name to open a description of the layer in a new window.

7. Enable the Define Download Area

Tool , under the Downloads Menu and Drag a box across the city of New Haven to define area of interest (AOI). Note that the box will remain green unless the area you have defined becomes larger that the Seamless Site allows (you can request up to 1.6GB, in 100MB files at once). Once you release the mouse button, a nbrowser window (you may have to enable pop-ups for the site). This window will provide some general info about the data you have requested (projection, bounding coordinates, cell size, etc…) as well as Download Link Buttons to begin downloading the data to your hard drive. C

ew page will be opened in a new new

8. lick on the Download Button to post the data request to the server. Yet

9. When the Save As… dialog box r

10. hen the download is complete,

11. he data will be extracted to a new

his

another window will be opened, which indicates the Current Status of your request on the server.

opens, browse to the \Data\Rastefolder to save the file. Wbrowse to find the ZIP file you justdownloaded, right-click on it and “Extract All.” Tsubfolder called something like “ned_70561049.” In the rest of ttutorial, this layer will be referred to as

ned_#######.

Applying an Appropriate Projection to You Data Data does not always come in a ‘ready to use’ format. One common necessity is to apply a “Projection” to the data, in order to transform the spatial reference from angular Lat/Long coordinates to planar/linear units, such as feet or meters. Here,

21

you will “project” your Digital Elevation Model to a projection that is appropriate for applying calculations that assume the data is recorded in linear units.

12. Open ArcMap.

13. Use the Add Data button to browse to the folder containing your elevation data. There should be two files there, one a polygon shapefile named “METADATA.shp,” (which contains the footprint of your elevation data, and its metadata), as well a raster layer with a name similar to the folder that contains it.

14. Select the ned_######## layer and add it to ArcMap.

15. Right-click on the ned_######## layer and Open its Properties Dialog box.

16. Select the Source Tab and scroll down to the Spatial Reference information.

17. Note that there is no Linear Unit assigned to this data. Scroll back up to the top and look at the Cellsize (X,Y) item.

18. This data has a Cellsize of about 30 meters, but here it is notes as 0.0002777777777999463. This is because the data has not been

22

projected and is currently spatially referenced in latitude & longitude coordinates, which are angular units of measurement.

19. Click OK to close the Properties Dialog.

20. Open the ArcToolbox and search for “Hillshade.” Open the Hillshade tool and use your ned_######## Layer as the Input Raster. Name the Output raster “hillshade01” and place it in the \Data\Raster folder. Change the Z factor option to 3 (this exaggerates the elevation for a better visual quality). Click OK to apply the tool.

When the tool is finished running, you should see a new layer in your Map View window. However, the effect is has produced is not very attractive. The Hillshade layer we have produced is very dark, and the topography it has created seems far more “extreme” than we might have expected. These poor results are related to what we observed earlier in the Spatial Reference and Cellsize of our Digital Elevation Model. Creating a Hillshade involves calculations that assume that the input parameters being used are in linear units, rather than the angular units that we currently have. This same problem would be true if we were to calculate slope, aspect and many other mathematical operations we might want to apply to this elevation data. What is necessary is that we “Project” our dataset from the current Latitude & Longitude Coordinates, which locate features on the face of the oblate spheroid that is the earth, to a projection

23

that records our data in linear measurements, as if the earth were flat. 21. Return to ArcToolbox and Search for “Project Raster.” Open the tool

and select your ned_####### layer as the Input raster, Browse to your \Data\Raster folder and save the Output raster as ned_proj. For the

Output Coordinate System, Click the Properties Icon to open the Spatial Reference Properties Dialog Box. Click on the Select… Button

and Browse to Projected Coordinate Systems>State Plane>Nad 83> NAD 1983 StatePlane Connecticut FIPS 0600.prj. Click Add. Click OK.

22. ArcToolbox adds the new layer to our Map Document. Right-Click on the new ned_proj layer and Open the Properties. Select the Source Tab and inspect the changes to the Cellsize and Linear Units items.

23. Use the Hillshade tool again, using the new projected elevation layer, to produce a new Hillshade Layer, called hillshade_02. Be sure to set the Z Factor to 3, like before.

24. You should find that you now have a much more pleasant looking result from the Hillshade Tool.

Converting from the Interchange (.e00) File Format to Shapefile. Interchange (.e00) format is a legacy data format from the days of Arc/INFO, when coverages and grids were the default data type for GIS modeling. Interchange files were a way of ‘packaging’ coverages and grids, whose essential data were

24

distributed across more than one folder. While most GIS data is now being produced in, or has been converted to, shapefile, you will still encounter Interchange format files. ArcCatalog retains the tools necessary to convert from Interchange format.

1. Open ArcCatalog and go to View>Toolbars to enable the “ArcView 8x Tools” toolbar. You should see a toolbar appear with a single dropdown

button labeled “Conversion Tools.”

2. Launch the “Import from Interchange File” tool from the Conversion Tools.

3. Browse to the \Data\Other folder and select the newhav.e00 file as your input file.

4. For the Output File, browse to the \Data\Shapefile folder and name the output file nh_wetlands. Click Save.

5. Before you apply the conversion, Click the Batch Button , and note that this tool provides the ability to convert multiple files at once. Click OK to apply the conversion.

6. In ArcCatalog, Browse to the \Data\Shapefile folder, and find the nh_wetlands coverage file. In the Catalog Tree window (on the left) expand the nh_wetlands coverage layer so that the four component layers are visible. We only want the polygon layer.

7. Right-click on the polygon component of the nh_wetlands coverage and select Export>To Shapefile (Single). Give the \Data\Shapefile folder as the Output Location, and name the Output Feature Class as New_Haven_NWI_Wetlands. Leave the remaining Options as their

25

default values. Click OK to apply the conversion.

8. Once the conversion is complete, you should see the New_Haven_NWI_Wetlands.shp appear in your \Data\Shapefiles folder.

26