DUSPviz Seminar Series Census and the American Community Survey, Session 2 October 27th, 2011

Exercise 1: Using data from Factfinder with ArcMap Factfinder provides US Population Census and American Community Survey (ACS) data that can be downloaded in a tabular format and joined with Census Tiger line files so the data can be displayed in ArcMap. This exercise will guide you through the process of downloading, joining and displaying Census 2010 data. Downloading Tiger files:

1. Go to http://www.census.gov/cgi-bin/geo/shapefiles2010/main to download 2010 TIGER/Line Shapefiles. Alternatively, you can go to the census home page ( http://www.census.gov) and click on TIGER and then click through a few links to access this page. These will provide a map to which you can join your census data.

2. Under select layer type, choose Counties (and equivalent) and press submit.

If you were interested in data for a smaller or larger area, you could have chosen other geographic divisions, such as census tracts or states.

3. Select Massachusetts for the state for 2010 and press download.

You also have the option of downloading all the counties for the entire United States. For certain divisions, such as census tracts, this is not an option due to the size of the file. Refer to this table to see which TIGER files are available for which geographic divisions: http://www.census.gov/geo/www/tiger/tgrshp2010/availability.html

4. Once the file downloads, right click to unzip it. Save the files in a local location, such as Documents or the Desktop.

Opening TIGER files:

1. Open ArcMap and choose a new blank document or press cancel to open a new map document.

2. Click the add data button ( ) in the top toolbar and navigate to the TIGER files you just unzipped. If you don’t

see your folder listed, click the “connect to folder” button to access the folder ( ). 3. Click “Add” to open the file. 4. Right click on the layer name and select “open attribute table.” You will notice that the table has columns of

geographic identifiers, but no census data. Next you will download tables of census data and join them to this table. Close the table.

Downloading Census Data:

1. Open the New American FactFinder using this link: http://factfinder2.census.gov/faces/nav/jsf/pages/index.xhtml

The left sidebar provides several ways to search for data, including by topic, geography, population group and industry code. You are interested in looking at housing data by county for the state of Massachusetts.

2. Click on “Geographies.” This will open a new “Select Geographies” window. 3. Under “Geographic Type,” click on “County.” 4. In the “Geography Results” box on the right, check the box next to “All counties in Massachusetts” and click the

“Add” button at the top of the box.

5. Close the “Select Geographies” window.

This will filter your results by the geography you selected. Your geographic selection has been added to the “Your Selections” box in the top, left. If you do not specify a geographic location before downloading a dataset, data for the default location usually downloads, which in most cases is Alabama, since it is the first state alphabetically. Now you are ready to choose your dataset and download the table.

6. Click on “Topics” at the left and notice that it expands to include several subcategories. 7. You are interested in housing data from Summary File 1 for the 2010 Census. Expand the “Dataset” subcategory

and click on 2010 SFI 100% data. (Notice another selection would be ACS 1-year estimates. You can download both from this site.)

8. Check the box next to General Housing Characteristics: 2010 (ID = QT-H1). If you want to learn more about any of the datasets, you can click on the “i” in the “About” column to see a sample table and list of variables included in the dataset.

There are buttons at the top of the “Search Results” box to both “View” and “Download” the data. You want to view the data first and then download it. This will give you more download options.

9. Click “View.” You will see a sample table for each county in Massachusetts. When you download the data all the counties will be included in one table.

10. Click Download. Select Comma Delimited format (.csv) and click ok. Once the file is ready, download it to your computer.

Preparing Census Data for Joining in ArcMap:

1. Unzip the .csv file that you just downloaded and then open it. You need to modify the table to make it compatible for joining in ArcMap.

a. Only one header row can be imported into ArcMap b. No special characters (. , - &) can be in the column name. c. The common variable that you are using to join the tables must be in the same format (i.e. both text). d. Save the table as an .xlsx file, which tends to be more compatible with ArcMap 10 than .xls (Excel 97-2003).

You need to delete all the header rows except for one. If you are working with tables of census data for your own work, you will probably want to spend some time re-naming the column headers so they are meaningful to you. You will also want to save a copy of the original excel file to refer to after some of the header rows are deleted. To save time in this workshop, you will keep one of the simplest header rows (so it is free of special characters) and then modify one column heading to use later in the exercise.

2. Copy and paste the headings “id,” “id2,” and “Geography” into the first row of the Excel table. See the graphic below for an example.

3. Rename the heading “S02” in column G to “occupiedhousingunits” so that you will be able to easily identify it later.

4. Delete all the header rows except for the first row (which contains the variable names S01, S02, etc.).

Important note for future use: If the id2 contains a leading zero, as is the case for many locations on the West coast, it will be missing from the table downloaded from FactFinder and will only appear as a 4-digit number. However, the leading zero will appear in table associated with the TIGER files, making it a 5-digit number. Because the numbers differ you will not be able to join them without adding that zero. You will have to create a new column in your Excel file and add the zero using the concatenate function. Your formula will look something like this: = CONCATENATE("0", B2). You can read more about this function in the Excel help menu.

5. The ID in the ArcMap attribute table to which you will be joining this table is in a text format. Id2 in this Excel file should also be text for the join to work properly. Convert the id2 column to text by highlighting the column, right clicking, and selecting “format cell.” Choose text for the format.

6. Save this table as an Excel workbook (not Excel 97-2003). Close the Excel file. Joining the Excel table to your TIGER ShapeFile in ArcMap:

1. Return to ArcMap and right click on the layer name (tl_2010_25_country10) and select “Joins & Relates” and then “Joins.”

2. Use the following parameters and click “Ok”: a. Choose the field in this layer that the join will be based on: GEOID10 b. Choose the table to join to this layer: Click on the folder icon and navigate to place where you saved your

Excel table. c. Choose the field in the table to base the join on: id2 d. Make sure “Keep all records” is selected e. Click “OK”

3. Look at the join you just completed by right clicking on the layer name and selecting “open attribute table.” Scroll to the right. Now all the census data has been added to your table. Close the table window.

4. The join is temporary until you export the table to a new file. To do this, right click on the file name and select data > export data. Name the file something that makes sense to you, rather than the default “export-output” and choose to save the file to a location on your computer. Change “Save as Type” to Shapefile. Click “ok”

5. When asked if you want to add the exported layer to your map, click “yes.” Visually Displaying Census Data:

1. Right click on the layer you just added to your map document and select “Properties.” 2. In the Symbology tab, select “Quantities” on the left and “Graduated colors.” 3. For the Value, choose “occupiedhousingunits” 4. Click on the “label” heading at the top of the table and select “Format labels…”

5. Because these numbers are percentages, choose “percentage” and then click “Numeric options…” In this menu you can choose the number of decimal places you wish to display.

6. Click “Ok” three times to exit out of all the windows.

Your map now visually displays the percentage of occupied housing units in each county of Massachusetts.

You could open the symbology tab again and change the number of classes (ranges of values) from 5 to another number, change the colors displayed, or choose another variable to display.

Exercise 2: Using Geolytics Neighborhood Change Database to compare Census data over time With Geolytics software, you can export Census data directly to a shapefile; you don’t have to join the table to the shapefile in a separate step. In the Neighborhood Change Database, which is one of several Geolytics software packages, the data for years 1970, 1980, 1990 and 2000 are recalculated and normalized to represent the data in 2000 boundaries. You can compare across years without having to control for changes in boundary definitions. This exercise will guide you through extracting data for 1970, 1980, 1990 and 2000. Then we’ll bring it into ArcMap to display it. Create a new request:

1. Close ArcMap if it’s open. 2. Create a folder on the desktop and name it with your Kerberos ID. (Delete this before you leave.) 3. Open the GIS folder on the desktop and launch the Neighborhood Change Database (NCDB).

In order to keep track of your files, and to prevent saving over needed data, start by creating/saving a new request. Unless you do this step, requests are labeled as NEWREQ and are saved over at the next request.

4. Go to File-> Save request as ->Navigate to your folder on the desktop-> and give your file a name “NCDBdata”. Geolytics software is pretty picky about names. It should be 8 characters or less, numbers and letters only.

Select time period:

5. Next, click on year and select “All years normalized to 2000.” This will extract data from each of the ’70, ’80, ’90, and ’00 Censuses, and display the data within 2000 boundaries.

Select the geography:

6. Within the Area menu, select Counties. A dialog box will come up. From the States column, select Massachusetts. All counties within Massachusetts will come up in the Counties window.

7. Select the top and bottom county numbers, then click Select between, on the lower right corner. This will include everything on the list in the selection. Then click Done. (For comparison’s sake, data will be at parcel level as county boundaries likely didn’t vary).

Note: County boundaries won’t change between Censuses, but Census tracts might. In order to extract data for all tracts within each county of MA, we’ll select County as the area, and NCDB will break the data down by tract. Other Geolytics

software packages are set up to allow you to select a subarea within an area (e.g. all counties within a state, several block groups within a tract). Choose tables:

8. Next we’ll choose data tables to extract. Select Counts from the top menu, and a dialog box will open. Scroll through the list to see available data and census year for each. You can browse through the list and select individual records, search by keyword, or if you’d like a single variable from all 4 censuses, find the variable for one census year and then do a keyword search for the code in the Name column. We’ll use the last option.

9. In this exercise we are going to select Total occupied housing units. I saved you the effort of finding the name of the variable and provided them below.

a. OCCHU – occupied housing units b. TOTHSUN – total housing units

10. Search for OCCHU. Click on Browse Last Search results to scroll through what was selected. To select these as the data you will extract, click on Add Selected Counts to Report.

Because the data is provided as a raw number rather than a percentage, we’ll also extract total housing units so we can calculate the percentage of occupied housing units per tract (We’ll do this in ArcMap).

11. Search for TOTHSUN. Click on Add Selected Counts to Report to add to the data you’ll extract.

12. The Report Counts window will show both of the counts you selected for all four Censuses. Click Done.

Run the query:

13. Next run the query. Select Map from the Run menu to generate the geographic information we’ll use in ArcMap.

Export map to shapefile: 14. From the File menu, select Export-> to Arcview’s Shape.

15. Open the folder that you created on the desktop. You’ll see the request you saved, plus the files you’ll need to draw the map in ArcMap. You’ll also see a Word document. The column names used by Geolytics are very abbreviated; this document is the key to what is in each column.

Bring the data into ArcMap:

16. Open ArcMap and choose a new blank document or press cancel to open a new map document.

17. Click the add data button ( ) in the top toolbar and navigate to your folder on the desktop. Add NCDBDATA.shp.

Data from Geolytics comes without a spatial reference. This means that while it looks accurate in terms of shape, where it is placed on the map in relation to the world is not defined. For example, your map of MA could end up on top of Mexico on a world map. Later we’ll define the projection to correct this.

18. We’ll take care of this in a few minutes. For now, click OK to dismiss the spatial reference warning.

Display data: 19. Right click on the layer you just added to your map document and select “Properties.” 20. In the Symbology tab, select “Quantities” on the left and “Graduated colors.”

a. In the Value field, select OCCHU0. b. In the Normalization field, select TOTHSUN0

This will calculate the number of occupied housing units as a portion of the total number of units. (i.e. percentage of occupied housing units).

21. Click on the “label” heading at the top of the table and select “Format labels…”

22. These numbers represent a percentage, so choose “percentage” and then click “Numeric options…” In this menu

you can choose the number of decimal places you wish to display. Click OK. 23. Because these numbers are calculated and therefore decimals, rather than already given as a percentage, select

“The number represents a fraction. Adjust it to show a percentage.”

24. Click OK twice to view the map. View different years:

25. To view different years, repeat the above step using OCCHU9 & TOTHSUN9, etc. To save them as separate layers, right click on NCDBdata and select Data->Export data. Name the different exports by the Census year.

Define projection

26. To be able to use this layer with other data layers, you’ll need to define the projection. Data from Geolytics uses NAD83 (North American Datum 1983).

27. Open the toolbox, by clicking on the icon.

28. Go to Data Management Tools-> Projections and Transformations ->Define Projection. 29. For Input Dataset or Feature Class, click the drop down box and select NCDBDATA. 30. Click on the box next to Coordinate system and a dialog box will come up. 31. Click on Select and another dialog box will open. Open the folder for Geographic Coordinate Systems -> North

America -> NAD1983.prj.

32. Click Add, then click OK through the remaining dialog boxes. You’ll see a note to indicate that tool is working.

Define Projection for the data frame

33. Next we need to define the projection for the background, also called the data frame. 34. From the top toolbar select View->Data Frame Properties

35. In the dialog box, navigate to Predefined-> Geographic Coordinate Systems-> North America->NAD 1983

It may not look like anything has changed, but if you add other data with the coordinate system defined, it will line up appropriately.