an introduction to mapping, gis and spatial modelling in r (presentation)

An Introduction to Mapping, GIS and Spatial Modelling in R Richard Harris, School of Geographical Sciences, University of Bristol

1 www.social-statistics.org

A Little about me…

•  My name is Richard Harris ([email protected]) •  I am a quanEtaEve human geographer with interest in ways of measuring and mapping processes of ethnic and social segregaEon, especially with respect to schools choices in the UK.

•  This is my first visit to China and I am very pleased to be here – thank you.

www.social-statistics.org 2

Example of my work (1)

•  Does the promoEon of school choice in England exacerbate ethnic or social segregaEon?


Mixed neighbourhood

Segregated schools

How could we measure this?

•  An ‘aspaEal’ way: •  (Measure of segregaEon for school) – (Measure of segregaEon for neighbourhood)

•  If result > 0 segregaEon appears to be increasing as a result of school allocaEons (which reflect choices)


How could we measure this?

•  A spaEal way •  Compare the ethnic or social composiEon of school A with the ethnic/social composiEon of other surrounding schools.

•  Σ wij | Xi – Xj | •  Where wij is part of a spaEal weights matrix



•  ContrasEng types of neighbourhoods at a small area scale

•  For example •  There is an area in Oldham (North England) where 61 of the 136 populaEon are White BriEsh and 31 Asian. Meanwhile, in an adjoining area, 345 of the 362 populaEon are Asian. None is White BriEsh.


Demo at http://glimmer.rstudio.com/drrichharris/segregation/


What do these have in common?

•  An interest in social geography •  The use of geographical data •  Geographical types of analysis •  And the use of R to do it all


Outline

•  What is R? •  Why use R? •  Why use R for mapping / GIS / spaEal analysis? •  How to do so – a demonstraEon


Note: •  I have created a set of tutorials on

a)  Gefng Started with R b)  A Geographical DemonstraEon of R c)  A Lihle More about the Workings of R d)  Using R as a Simple GIS e)  Defining Neighbours and CreaEng SpaEal Weights f)  SpaEal Regression Analysis


Tutorials and this presentation

•  Available at hhp://www.social-‐staEsEcs.org/

•  or hhp://www.researchgate.net/publicaEon/258151270_An_IntroducEon_to_Mapping_and_SpaEal_Modelling_in_R

•  I will add more to the tutorial in the future.


What is R?

•  “R is a free sonware environment for staEsEcal compuEng and graphics. It compiles and runs on a wide variety of UNIX plaoorms, Windows and MacOS” (hhp://www.r-‐project.org/)

•  Available from The Comprehensive R Archive Network: hhp://cran.r-‐project.org


The Mac version looks like this


RStudio also is available (http://www.rstudio.com/)


Sou

rce:

ht

tp://

ww

w.rs

tudi

o.co

m/id

e/

Why use R?

•  Because it is free and openly available across plaoorms

•  It is widely used with a large user-‐community •  The help documentaEon is generally excellent •  Moreover….


Why use R?

•  It does what you want it to do (because it is a programming language)

•  It allows for reproducibility of results and for collaboraEve working – essenEal for science and social science. For example: •  Can place your data on a website; record your stages of working as a script file; allow other people to run the script, check for errors, make suggesEons, etc.


Why use R for mapping / GIS / spatial analysis?

•  R has strong visualizaEon capabiliEes •  And strong staEsEcal capabiliEes •  Not many sonware bring the two together whilst also adding spaEal data handling and spaEal analyEcal capabiliEes

•  Even most GIS do not offer this; nor do most staEsEcal packages (which are not spaEal)



•  A large and growing number of libraries for handling spaEal data in R have been developed (and are freely available on CRAN)

•  See especially Bivand et al. Applied Spa*al Data Analysis with R and Gollini et al. athhp://arxiv.org/pdf/1306.0413.pdf



•  And, of course, the arguments for reproducibility, sharing and collaboraEon sEll apply.



‘ Core R’

Other libraries Other libraries


Other libraries are installed (once) and then called each time they are needed

Structure of R


‘ Core R’


e.g. GISTools Other libraries

e.g. spdep e.g. sp

e.g. maptools Structure of R


‘ Core R’


e.g. GISTools Other libraries

e.g. spdep e.g. sp

e.g. maptools Many spatial packages associated with the Spatial Task View

(see http://cran.r-project.org/web/ views/Spatial.html)

Packages that may be of particular interest

•  spdep •  To create spaEal weights, summarise those objects, and for permifng their use in spaEal data analysis

•  hhp://cran.r-‐project.org/web/packages/spdep/ •  spaEal

•  FuncEons for kriging and point pahern analysis. hhp://cran.r-‐project.org/web/packages/spaEal/



•  GISTools •  Some mapping and spaEal data manipulaEon tools •  hhp://cran.r-‐project.org/web/packages/GISTools/

•  raster •  For Reading, wriEng, manipulaEng, analyzing and modeling of gridded spaEal data

•  hhp://cran.r-‐project.org/web/packages/raster/



•  maptools •  Set of tools for manipulaEng and reading geographic data, in parEcular ESRI shapefiles

•  hhp://cran.r-‐project.org/web/packages/maptools/

•  sp •  A package that provides classes and methods for spaEal data

•  hhp://cran.r-‐project.org/web/packages/sp/ www.social-statistics.org 28


•  GWmodel •  FuncEons for compuEng geographically weighted models

•  hhp://cran.r-‐project.org/web/packages/GWmodel/

•  + many more!


To install the task view packages >  install.packages("ctv")>  library("ctv")>  install.views("Spatial")


And for other packages, e.g. >  install.packages("GWmodel")

Then to load a library, e.g. >  library("GWmodel")

A demonstration

•  Two data sets a)  A simple table of data about land parcel prices in

Beijing in .csv format (simulated, not real data) b)  Boundary file for districts in Beijing in .shp format


A demonstration

1)  Load the .csv data into R 2)  Convert the XY data into spaEal points 3)  Load the .shp data 4)  Map the points over the district boundaries 5)  Create a spaEal weights matrix based on

inverse distance weighEng 6)  Fit a regression model (a land price model) www.social-statistics.org 32

Demonstration

7)  Check the assumpEon of (spaEal) independence in the residuals of the model

8)  Try some spaEal approaches •  A spaEal error model •  A spaEally lagged y model •  Geographically Weighted Regression


You can repeat the analysis yourself

•  Just follow the code in these slides •  The data needed can be downloaded from: hhps://www.dropbox.com/sh/zzibpn2keilrhv3/rsuA7L_jlK •  You need the XY data, landprices.csv, and the boundary file, beijing_districts.shp

•  Note that landprices.csv is simulated (not real) data


Remember!

•  A full descripEon of how to do all this is included in the tutorials available athhp://www.social-‐staEsEcs.org/ or hhp://www.researchgate.net/publicaEon/258151270_An_IntroducEon_to_Mapping_and_SpaEal_Modelling_in_R


Loading and mapping the files >  landdata <- read.csv(file.choose())>  library(maptools)>  districts <- readShapePoly(file.choose())>  library(sp)>  attach(landdata)>  coordinates(landdata) <- c("x", "y")>  plot(districts)>  plot(landdata, pch=21, bg="yellow", cex=0.8, add=T)


Loading and mapping the files >  landdata <- read.csv(file.choose())>  library(maptools)>  districts <- readShapePoly(file.choose())>  library(sp)>  attach(landdata)>  coordinates(landdata) <- c("x", "y")>  plot(districts)>  plot(landdata, pch=21, bg="yellow", cex=0.8, add=T)


Read-in a table of data

Read-in a shapefile

Convert the XY data into a spatial object

Plot the map

Creating a choropleth map >  library(GISTools)>  attach(districts@data)>  shades <- auto.shading(POPDEN)>  choropleth(districts, POPDEN)>  plot(landdata, pch=21, bg="yellow", cex=0.8, add=T)>  choro.legend(462440,4407000,shades,fmt="%4.1f",title=

'Population density')>  north.arrow(461000, 4445000, "N", len=1000,

col="light gray")>  map.scale(425000,4400000,10000,"km",

5,subdiv=2,tcol='black',scol='black', sfcol='black')


Creating a choropleth map >  library(GISTools)>  attach(districts@data)>  shades <- auto.shading(POPDEN)>  choropleth(districts, POPDEN)>  plot(landdata, pch=21, bg="yellow", cex=0.8, add=T)>  choro.legend(462440,4407000,shades,fmt="%4.1f",title=

'Population density')>  north.arrow(461000, 4445000, "N", len=1000,

col="light gray")>  map.scale(425000,4400000,10000,"km",

5,subdiv=2,tcol='black',scol='black', sfcol='black')


Create a colour scheme Plot the map

Add information to the map

XY Maps with the point symbols shaded >  library(classInt)>  classIntervals(LNPRICE, 5, "fisher")>  break.points <- classIntervals(LNPRICE, 5, "fisher")

$brks>  groups <- cut(LNPRICE, break.points,

include.lowest=T, labels=F)>  library(RColorBrewer)>  palette <- brewer.pal(5, "Greens")>  plot(districts)>  plot(landdata, pch=21, bg=palette[groups], cex=0.9,

add=T)


XY Maps with the point symbols shaded >  library(classInt)>  classIntervals(LNPRICE, 5, "fisher")>  break.points <- classIntervals(LNPRICE, 5, "fisher")

$brks>  groups <- cut(LNPRICE, break.points,

include.lowest=T, labels=F)>  library(RColorBrewer)>  palette <- brewer.pal(5, "Greens")>  plot(districts)>  plot(landdata, pch=21, bg=palette[groups], cex=0.9,

add=T)


Put the observations into the map classes

Create a colour scheme

Plot the points, coloured by map class

>  legend("bottomright", legend=c("4.85 to <6.3", "6.3 to <7.105”, "7.105 to <7.865","7.865 to <8.82","8.82 to 11.06"), pch=21, pt.bg=palette, pt.cex = 0.9, title="Land value (log)")

>  north.arrow(461000, 4445000, "N", len=1000, col="light gray")

>  map.scale(425000,4400000,10000,"km",5,subdiv=2,tcol='black',scol='black', sfcol='black’)


>  legend("bottomright", legend=c("4.85 to <6.3", "6.3 to <7.105”, "7.105 to <7.865","7.865 to <8.82","8.82 to 11.06"), pch=21, pt.bg=palette, pt.cex = 0.9, title="Land value (log)")

>  north.arrow(461000, 4445000, "N", len=1000, col="light gray")

>  map.scale(425000,4400000,10000,"km",5,subdiv=2,tcol='black',scol='black', sfcol='black’)


Add information to the map


Land value (log)4.85 to <6.36.3 to <7.1057.105 to <7.8657.865 to <8.828.82 to 11.06

N

0 2 4 6 8 10km

Mapping as a raster grid >  library(raster)>  cell.length <- 1000>  xmin <- bbox(districts)[1,1]>  xmax <- bbox(districts)[1,2]>  ymin <- bbox(districts)[2,1]>  ymax <- bbox(districts)[2,2]>  ncol <- round((xmax - xmin) / cell.length, 0)>  nrow <- round((ymax - ymin) / cell.length, 0)>  blank.grid <- raster(ncols=ncol, nrows=nrow,

xmn=xmin, xmx=xmax, ymn=ymin, ymx=ymax)


Mapping as a raster grid >  library(raster)>  cell.length <- 1000>  xmin <- bbox(districts)[1,1]>  xmax <- bbox(districts)[1,2]>  ymin <- bbox(districts)[2,1]>  ymax <- bbox(districts)[2,2]>  ncol <- round((xmax - xmin) / cell.length, 0)>  nrow <- round((ymax - ymin) / cell.length, 0)>  blank.grid <- raster(ncols=ncol, nrows=nrow,

xmn=xmin, xmx=xmax, ymn=ymin, ymx=ymax)


The cell length

Set the grid to cover the geography (bounding box) of the Beijing districts

Define the blank raster

>  xs <- coordinates(landdata)[,1]>  ys <- coordinates(landdata)[,2]>  xy <- cbind(xs, ys)>  x <- LNPRICE>  land.grid <- rasterize(xy, blank.grid, x, mean)>  plot(land.grid)>  plot(districts, add=T)>  plot(landdata, pch=3, cex=0.3, add=T)


>  xs <- coordinates(landdata)[,1]>  ys <- coordinates(landdata)[,2]>  xy <- cbind(xs, ys)>  x <- LNPRICE>  land.grid <- rasterize(xy, blank.grid, x, mean)>  plot(land.grid)>  plot(districts, add=T)>  plot(landdata, pch=3, cex=0.3, add=T)


Aggregate the point data into the raster cells

Create the map

Summary so far…

•  Ways to create maps in R using, especially, the maptools and sp libraries to handle the spaEal data; classInt and RColorBrewer to create map classes and colours; sp and GISTools to map the data; and raster to create a raster grid.

•  But what about spaEal analysis and modelling?


Spatial Weights

•  Much spaEal analysis (e.g. localised staEsEcs, geographically weighted staEsEcs and spaEal regression) require a spaEal weights matrix to define neighbours / the strength of connecEon between spaEal objects.

•  For this, use the library spdep (spaEal dependency)


Creating spatial weights

•  Decide on the types of weight •  ConEguity? Distance-‐based? K Nearest neighbours?

•  Decide on the threshold (kernel length) at which the weight becomes zero

•  Decide on the decay funcEon (shape of the kernel)


Creating spatial weights (calibration) >  library(spdep)>  knear250 <- knearneigh(landdata, k=250, RANN=F)>  correlations <- apply(knear250$nn, 2, function(i)

cor(x, x[i]))>  plot(correlations, xlab="nth nearest neighbour",

ylab="Correlation")>  lines(lowess(correlations))>  abline(v=35, lty="dotted")


Creating spatial weights (calibration) >  library(spdep)>  knear250 <- knearneigh(landdata, k=250, RANN=F)>  correlations <- apply(knear250$nn, 2, function(i)

cor(x, x[i]))>  plot(correlations, xlab="nth nearest neighbour",

ylab="Correlation")>  lines(lowess(correlations))>  abline(v=35, lty="dotted")


This will produce a correlogram for the observed land price values Vs their kth nearest neighbour


The 35th nearest neighbour

Inverse distance weighting (Gaussian decay to the 35th nearest neighbour) >  d.matrix <- spDists(landdata, landdata)>  knear35 <- knearneigh(landdata, k=35, RANN=F)>  np <- knear35$np>  d.weights <- vector(mode="list", length=np)>  for (i in 1:np) {

neighbours <- knear35$nn[i,] distances <- d.matrix[i,neighbours] dmax <- distances[35] d.weights[[i]] <- exp(-0.5*distances^2/dmax^2)

}>  spknear35gaus <- nb2listw(knn2nb(knear35),

glist=d.weights, style="C")www.social-statistics.org 59

Inverse distance weighting (Gaussian decay to the 35th nearest neighbour) >  d.matrix <- spDists(landdata, landdata)>  knear35 <- knearneigh(landdata, k=35, RANN=F)>  np <- knear35$np>  d.weights <- vector(mode="list", length=np)>  for (i in 1:np) {

neighbours <- knear35$nn[i,] distances <- d.matrix[i,neighbours] dmax <- distances[35] d.weights[[i]] <- exp(-0.5*distances^2/dmax^2)

}>  spknear35gaus <- nb2listw(knn2nb(knear35),

glist=d.weights, style="C")www.social-statistics.org 60

Find the distances between points

Find the 35 nearest neighbours for each point

Calculate the Gaussian decay to the 35th neighbour

Create the spatial weights

•  Once we have a spaEal weights matrix we can use it. For example…


Fit a regression model, check for spatial errors

>  model1 <- lm(LNPRICE ~ DCBD + DELE + DRIVER + DPARK + Y0405 + Y0607 + Y0809, data=landdata)

>  moran.plot(residuals(model1), spknear35gaus)>  lm.morantest(model1, listw2U(spknear35gaus))> Global Moran's I for regression residuals

>  Moran I statistic standard deviate = 17.5633, p-value < 2.2e-16>  alternative hypothesis: greater>  sample estimates:>  Observed Moran's I Expectation Variance >  1.086196e-01 -3.256155e-03 4.057499e-05


Fit a regression model, check for spatial errors

>  model1 <- lm(LNPRICE ~ DCBD + DELE + DRIVER + DPARK + Y0405 + Y0607 + Y0809, data=landdata)

>  moran.plot(residuals(model1), spknear35gaus)>  lm.morantest(model1, listw2U(spknear35gaus))> Global Moran's I for regression residuals

>  Moran I statistic standard deviate = 17.5633, p-value < 2.2e-16>  alternative hypothesis: greater>  sample estimates:>  Observed Moran's I Expectation Variance >  1.086196e-01 -3.256155e-03 4.057499e-05


lm = linear model

Look for evidence of spatial autocorrelation in the residuals

Spatial Econometric models

•  For example, •  SpaEal error model •  SpaEally lagged y model


Spatial Econometric models >  # Standard OLS model>  model1 <- lm(LNPRICE ~ DCBD + DELE + DRIVER + DPARK +

Y0405 + Y0607 + Y0809, data=landdata)>  # Spatial error model>  model2 <- errorsarlm(LNPRICE ~ DCBD + DELE + DRIVER +

DPARK + Y0405 + Y0607 + Y0809, data=landdata, spknear35gaus)

>  # Spatially lagged y model>  model3 <- lagsarlm(LNPRICE ~ DCBD + DELE + DRIVER +

DPARK + Y0405 + Y0607 + Y0809, data=landdata, spknear35gaus)


>  logLik(model1)>  'log Lik.' -1438.354 (df=9)>  logLik(model2)>  'log Lik.' -1386.234 (df=10)>  logLik(model3)>  'log Lik.' -1426.541 (df=10)


Geographically Weighted Regression •  An alternaEve way of ahempEng to explain the spaEal variaEon in the land value prices is to allow the effect sizes of the predictor variables to themselves vary over space. Geographically Weighted Regression (GWR) offers this where the esEmate of βx at point locaEon i is not simply the global esEmate for all points in the study region but a local esEmate based on surrounding points weighted by the inverse of their distance away.


Geographically Weighted Regression >  library(GWmodel)>  d.matrix <- gw.dist(dp.locat=coordinates(landdata))>  bw <- bw.gwr(LNPRICE ~ DCBD + DELE + DRIVER + DPARK +

Y0405 + Y0607 + Y0809, data=landdata, adaptive=T, dMat=d.matrix)

>  gwr.model <- gwr.basic(LNPRICE ~ DCBD + DELE + DRIVER + DPARK + Y0405 + Y0607 + Y0809, data=landdata, adaptive=T, dMat=d.matrix, bw=bw)


Geographically Weighted Regression >  library(GWmodel)>  d.matrix <- gw.dist(dp.locat=coordinates(landdata))>  bw <- bw.gwr(LNPRICE ~ DCBD + DELE + DRIVER + DPARK +

Y0405 + Y0607 + Y0809, data=landdata, adaptive=T, dMat=d.matrix)

>  gwr.model <- gwr.basic(LNPRICE ~ DCBD + DELE + DRIVER + DPARK + Y0405 + Y0607 + Y0809, data=landdata, adaptive=T, dMat=d.matrix, bw=bw)


Calculate the distances between points

Calibrate the bandwidth; here using k nearest neighbours

Fit the model


This

var

iabl

e ap

pear

s to

var

y in

w

heth

er it

s ef

fect

is n

egat

ive

or

posi

tive

on la

nd p

arce

l pric

es

Mapping the local estimates


>  x <- gwr.model$SDF$DCBD>  break.points <- classIntervals(x, 5, "fisher")$brks>  groups <- cut(x, break.points, include.lowest=T,

labels=F)>  palette <- brewer.pal(5, "Spectral")>  subset=abs(gwr.model$SDF$DCBD_TV) > 1.96>  par(mfrow=c(1,2))>  plot(districts)>  plot(landdata, pch=21, bg=palette[groups], add=T)>  plot(districts)

Mapping the local estimates


>  x <- gwr.model$SDF$DCBD>  break.points <- classIntervals(x, 5, "fisher")$brks>  groups <- cut(x, break.points, include.lowest=T,

labels=F)>  palette <- brewer.pal(5, "Spectral")>  subset=abs(gwr.model$SDF$DCBD_TV) > 1.96>  par(mfrow=c(1,2))>  plot(districts)>  plot(landdata, pch=21, bg=palette[groups], add=T)>  plot(districts)

v Get the local estimates

Omit the ‘insignificant’ cases Plot two graphics together


>  plot(landdata[subset,], pch=21, bg=palette[groups[subset]], add=T)

>  n <- length(break.points)>  break.points <- round(break.points, 2)>  break.points[n] <- break.points[n] + 0.01>  txt <- vector("character",length=n-1)>  for (i in 1:(length(break.points) - 1)) {

txt[i] <- paste(break.points[i],"to <", break.points[i+1]) }

>  legend("bottomright",legend=txt, pch=21, pt.bg=palette, cex=0.7)


>  plot(landdata[subset,], pch=21, bg=palette[groups[subset]], add=T)

>  n <- length(break.points)>  break.points <- round(break.points, 2)>  break.points[n] <- break.points[n] + 0.01>  txt <- vector("character",length=n-1)>  for (i in 1:(length(break.points) - 1)) {

txt[i] <- paste(break.points[i],"to <", break.points[i+1]) }

>  legend("bottomright",legend=txt, pch=21, pt.bg=palette, cex=0.7)

v

Slightly automating the process of creating the legend


(omits locally insignificant estimate)

Much more we could do

•  MulElevel modelling: library(lme4)•  GeostaEsEcal approaches (e.g. kriging, semi-‐variogram): library(gstat)

•  SpaEal point pahern analysis: library(spatstat)•  Local Indicators of SpaEal AssociaEon (LISAs):

library(spdep)


In summary

•  R offers a flexible and customisable environment for geographical data handling and analysis

•  It takes a lihle while to learn but it is worth doing so

•  There is a large user community, many of whom make material available online


Getting Help

•  Perhaps the best all round introducEon to R is the An IntroducEon to R at CRAN •  hhp://cran.r-‐project.org/manuals.html


Getting Help

•  There is a mailing list for discussing the development and use of R funcEons and packages for handling and analysis of spaEal, and parEcularly geographical, data. It can be subscribed to at www.stat.math.ethz.ch/mailman/lisEnfo/r-‐sig-‐geo.


Getting Help

•  The spaEal cheat sheet by Barry Rowlingson at Lancaster University is really helpful:hhp://www.maths.lancs.ac.uk/~rowlings/Teaching/UseR2012/cheatsheet.html

•  There are some excellent R spaEal Eps and tutorials on Chris Brunsdon's Rpubs site, hhp://rpubs.com/chrisbrunsdon , and on James Cheshire's website, hhp://spaEal.ly/r/.


Getting Help

•  There is an excellent workbook on spaEal regression analysis in R by Luc Anselin. It is available at hhp://openloc.eu/cms/storage/openloc/workshops/UNITN/20110324-‐26/Basile/Anselin2007.pdf


Getting Help

•  GWmodel: an R Package for Exploring SpaEal Heterogeneity using Geographically Weighted Models: hhp://arxiv.org/abs/1306.0413

•  Analyzing spaEal point paherns in R: www.spatstat.org


Getting Help (textbooks using R)

•  Bivand, R.S., Pebesma, E.J. & Gómez-‐Rubio, V., 2008. Applied Spa*al Data Analysis with R. Berlin: Springer.

•  Ward, M.D. & Skrede Gleditsch, K., 2008. Spa*al Regression Models. London: Sage.

•  Chun, Y. & Griffith, D.A., 2013. Spa*al Sta*s*cs and Geosta*s*cs. London: Sage.


Thank you very much! www.social-‐staEsEcs.org


an introduction to mapping, gis and spatial modelling in r (presentation)

Technology