Causal Structure, Endogeneity, and the Missing Data Problem in Modeling the

Impact of Information and Communication Technology Use on Society

Tuesday, April 18, 2023

Hun Myoung Park

University Information Technology ServicesIndiana University

kucc625@indiana.edu

HICSS-41, January 7-10, 2008 2

Outline ICT Use and Society Competing Perspectives Review of Traditional Approaches Nature of Problems Alternative Approaches Data and Illustrations Findings Implications

HICSS-41, 2008 3

ICT Use and Society

Does ICT use influence society? Positive, negative, or negligible

effect? Technological determinism

Optimistic perspective Pessimistic perspective

Skeptical perspective

HICSS-41, 2008 4

Optimistic Perspective

ICT Use Society

Positive impact on societyTransformation TheoryRheingold (1993); Grossman (1995); Morris (1999) “Getting the general public engaged”

HICSS-41, 2008 5

Pessimistic Perspective

ICT Use Society

Negative impact on societyReinforcement theory David (1999, 2005); Norris (2001)Digital inequality (digital divide)“Engaging the engaged” rather than the disenfranchised

HICSS-41, 2008 6

Skeptical Perspective

ICT Use Society

ICT use shaped by societyReflection of the real worldNormalization theoryMargolis and Resnick (2000); Bimber

(2001, 2003); Uslaner (2004)”Politics as usual”

HICSS-41, 2008 7

Conflicting Evidence, How? Conflicting empirical results

depending on perspectives What is wrong? Failure to deal with the nature of

problems properly How do we assess the impact of

ICT use (treatment effect) more correctly?

HICSS-41, 2008 8

Review: T-test (ANOVA) Comparing means/proportions Scott (2006) Impact of ICT use: mean

difference Simplicity and easy interpretation Two groups are assumed to have

same characteristics except for the treatment

HICSS-41, 2008 9

Review: Linear Regression Least squares dummy variable

model (LSDV) Jennings and Zeitner (2003);

Uslaner (2004); Welch and Pandey (2007)

Impact: dummy coefficient δ What if the dummy d are related

to disturbance ε?

HICSS-41, 2008 10

Review: Binary Response Model

Binary logit and probit model for binary dependent variables

Bimber (2001, 2003) and Thomas and Streib (2003)

Impact: a discrete change of d, difference in predicted probabilities

Large N required

HICSS-41, 2008 11

Nature of Problems Measurement issues: categorical

and binary DVs Limited DVs (self-selected) Ambiguous causal structure Endogeneity: d and ε are related The “missing data problem” in

nonexperimental research

HICSS-41, 2008 12

Causal Structure

ICT Use Society

ICT Use Society

Unidirectional versus bidirectional Interactive and jointly determined? Iterative and virtuous circle: Norris

(2000)

HICSS-41, 2008 13

Endogeneity

ICT use may not be exogenous Disturbance ε is related to the

ICT use d violation of key OLS assumption

Jointly determined in a system Instrumental variable (IV)

approach?

HICSS-41, 2008 14

Missing Data Problem A subject is either ICT user

(participant) or nonuser, not both. NOT necessarily means many

missing values in data Users and nonusers may have

different characteristics, which are not controlled in research (survey): self-selection bias

HICSS-41, 2008 15

Nonexperimental Design

OBSpre Treatment OBSpost_treatment

OBSpre OBSpost_control

Treatment (?) OBSusers

OBSnonusers

Randomized control group pre-post test design

Non-randomized post test only design Is ICT use a real treatment?

HICSS-41, 2008 16

Propensity Score Matching 1 Rosenbaum and Rubin (1983, 1984) Binary Probit model to compute

predicted probabilities Match users and nonusers who have

similar likelihood (propensity score) Pair matching/subclassification; one-to-

one pair matching w/o replacement Controlling many covariates using one

dimensional propensity score

HICSS-41, 2008 17

Propensity Score Matching 2

Estimatingpropensity

scores

Splitting thesample into

blocks

Achievebalance?

Pair matchingor

Subclassifying

Estimatingtreatment

effects

Adjustingspecification

No

Yes

Rosenbaum and Rubin (1984); Dehejia and Wahba (1999)

Matching(paired) T-test

HICSS-41, 2008 18

Treatment Effect Model

Subjects decide whether or not to receive treatment: selection bias

Selection equation estimates predicted probabilities of ICT use

Impact is the dummy coefficient adjusted by correlation of ICT use and the dependent variable

When ρ=0, the impact is δ

HICSS-41, 2008 19

Recursive Bivariate Probit Model

Maddala (1983), Greene (1998) Two equations with an endogenous

IV variable, ICT use Correlation between disturbances If ρ≠0, both direct/indirect effects

are considered in RBPM If ρ=0, binary response model

(BRM) examines direct impact only

HICSS-41, 2008 20

Specification (RBPM)

ICT UseCivic

Engagement

InformationTechnology

Factors

e2 e1

Demographic Factors Political Factors

Correlation

HICSS-41, 2008 21

Secondary Data The PEW Internet and American Life

Project 2004 Post-Election Internet Tracking

Survey (Crosssectional) N=2,146

The American National Election Studies Longitudinal data of 1996, 1998, 2000,

2004 N=6,014

HICSS-41, 2008 22

Illustration 1: E-government Use

IV (d): whether citizens look for information from government websites

DV: whether citizens sent email about voting (deliberative civic engagement)

DV: Attendance at a rally during the election campaign (action-oriented)

HICSS-41, 2008 23

Illustration 1: E-government Use

Average effect: 9.8% vs. 2.2% Discrete change: 15.3% vs. 3.3%

Method Email RallyT-test 17.1%

(1,243)6.6%

(1,320)

PSM (Pair) 9.8%(509)

2.2%(558)

BRM (Probit) 14.1%(1,030)

3.3%(1,090)

RBPM 15.3%(931)

3.3%(974)

HICSS-41, 2008 24

Illustration 1: E-government Use

02

04

06

08

01

00

1 3 5 7 9 1 3 5 7 9

Sending Emails about the Campaign Attending a Campaign Rally

Users Nonusers

Pe

rcen

tage

of C

ivic

En

gage

men

t

Strata by Estimated Propensity Scores

HICSS-41, 2008 25

Illustration 2: Internet Use IV (d): whether citizens have

used the Internet for political information

DV: discussing politics (deliberative civic engagement)

DV: whether citizens gave money to a candidate (action-oriented engagement)

HICSS-41, 2008 26

Illustration 2: Internet Use

Average effect: 10.1% vs. 4.4% Discrete change: 8.3% vs. 5.2%

Method Discuss Give MoneyT-test 21.0%

(5,419)6.3%

(5,425)

PSM (Pair) 10.1%(1,091)

4.4%(1,090)

BRM (Probit) 9.9%(4,956)

5.4%(4,959)

RBPM 8.3%(4,956)

5.2%(4,959)

HICSS-41, 2008 27

Illustration 2: Internet Use0

20

40

60

80

100

0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1

Discussing Politics Giving Money to a Candidate

Users Nonusers

Pre

dict

ed

Pro

ba

bilit

y of

En

gage

me

nt

Political Interest

HICSS-41, 2008 28

Finding 1: T-test vs. PSM Robust estimation of PSM at

the expense of loss of N T-test overestimates the

impact on deliberative civic engagement due to missing data problem

No big difference in action-oriented engagement

HICSS-41, 2008 29

Finding 2: BRM vs. RBPM BRM overestimates the impact

on deliberative civic engagement: endogeneity matters

Both direct and indirect effects No big difference in action-

oriented engagement; the impact of ICT use is direct

HICSS-41, 2008 30

Finding 3: Deliberative Engagement

Both direct and indirect effects considered

Overall impact depends on signs and magnitude of effects

They may have opposite signs that cancel out each other

BRM may report misleading results

HICSS-41, 2008 31

Implication and Conclusion Types of civic engagement to be

differentiated; variety of civic engagement (Verba et al. 1995)

Characteristics of dependent variables carefully examined

Causal structure, endogeneity, missing data problem, and sample size considered

Specific use of ICT applications differentiated as well

HICSS-41, 2008 32

Questions?

Question or suggestion?