QnDReview:

Read 100 CHI Papers in 7 Hours

Ji Soo Yi

HIVE Lab

Industrial Engineering, Purdue University

717AB, 9am, 4/30/2014

Disclaimer

• Comprehensive reading a research paper requires lots of effort / time and sufficient background, so this talk is not about making you have magical, comprehensive understanding of 100 papers in 7 hours.

• I also have not created any fancy tools.

A Story in 2004

“Hey, Ji Soo, for the book chapter that we are working, can you survey what kinds of research methods were used in the papers of two journals, Ergonomics and Human Factors in the past 5 years?”

+ 900 articles

+ 900 articles“What the …”

+ 900 articles“What the …”

Super procrastination

+ 900 articles“What the …”

Super procrastination“OK, let’s do it tonight.”

Pleasantly Surprised!

I was able to • skim through papers quicker and quicker;• see the overall trends in the field;• generate interesting ideas; and• actually enjoy the process.

Leonard, V. K., Jacko, J. A., Yi, J. S., and Sainfort, F. Human factors and ergonomic methods. In Handbook of Human Factors and Ergonomics, G. Salvendy, Ed., 3rd ed. John Wiley & Sons, Feb. 2006, 292–321.

In 2009, I started to teach a graduate-level HCI methodology course.

What if

What if graduate students are forced

What if graduate students are forced asked

What if graduate students are forced asked to read 100 papers?

Quick and Dirty Review

QnDReview

Homework

1. Select 100 papers to read. Visit the ACM Digital Library to search for any papers that are seemingly interesting to you. I recommend you to start with CHI full papers because CHI is known to be the premier venue.

ver. 2013

Homework

2. Review them all very quickly. Use the template to make your own summary review. An Excel spread sheet for the quick and dirty review should contain the following information:

Column 1: Paper citation

Column 2: Interest rating: 1 (Worst) - 2 (Bad) - 3 (Neither good nor bad) - 4 (Good) - 5 (Best)

Column 3: A single sentence that describes the core idea of the paper.

Column 4: (Optional) Any new ideas that you have after skimming through this paper.

ver. 2013

Homework

3. Email your summary Email the complete review spreadsheet to the instructor by the due date with the following information in the body of the email.

• What is the overall experience? • What is the notable trends in the papers? • Which are the two most interesting papers you would like to read

more deeply? (Provide the full citation with DOI number, so that I can find the paper later)

• How much time did you spend on this homework? • What are the most difficult part of this homework? • How to make the quick and dirty review better?

ver. 2013

Participants

• N = 57 over 5 years• 27 F + 30 M• 2 U + 21 MS + 34 PhD• 11 non-IE1 + 46 IE

• IRBs• 1012010281 (retrospective one)• 1302013324

1 Engineering Education and Electrical and Computer Engineering.

Some Variations

2009: self-selected 100 HCI-related papers1

2010: preselected 50 CHI 2010 papers

2011: preselected 100 CHI 2011 papers

2012: preselected 120 papers• 30 from CHI 2012• 30 from IEEE TVCG• 30 from Decision Support Systems

2013: self-selected 100 CHI 2013 papers

1 CHI, HCI, IJHCI, and IJHCS.

Results

• Time Spent• Overall experiences

Total Time Spent

μ = 6.77 hσ = 3.11 h

Average Time Spent Per Paper

μ = 4.3 minσ = 1.8 min

Overall Impression

• Open coding• Avoid codify any general remarks (e.g., “It

was a good experience.”)

It’s so tiring

N = 26• At first I did not mind the reading at all, but I

definitely became less interested as time went on. (P2012-3)

• This was a pretty tiring experience. I found myself taking multiple breaks in order to regain focus and energy [...] (P2012-9)

• Interesting idea, but I have done this before for my lit. review for my masters; Too many papers. The exercise became tedious after a while; [...] (P2011-8)

But, it gets better.

N = 13• Well, I need to say that this hw is kind of tough

experience for me especially in the beginning. I spent much more time than I expected. But the more I did, the faster I read. [...] (P2009-4)

• I have never read so many papers. Its definitely a big challenge for me. At the beginning, its hard for me to skim each paper in 4 minutes. However, I think I gradually know how to get the core idea of the paper in 4 minutes. [...] (P2010-3)

It turned out good.

N = 44

big picture or trend (N = 20)

useful for future research (N = 15)

quite enjoyable (N = 15)

Big Picture (N = 20)

• At the beginning, I thought the task was impossible to complete in such a short time. I had to force myself to stop reading within 4 minutes. Gradually I became comfortable with this kind of quick review method. I began to enjoy reviewing so many interesting topics and was surprised by the diversity of the research. This activity helped me get familiar with the HCI area. Some research topics and design were pretty insightful. (P2010-2)

Useful for Future Research (N = 15)

• I think the overall experience of this project was good because it helped me learn how to quickly understand an area of research and to survey different types of research ideas within a particular field. I think I will definitely use this approach in my own research studies to especially before I write a literature review. I also really liked learning how to use Zotero and I think that is a great app to write papers with. (P2013-4)

Quite enjoyable (N = 15)

• I have to say it is unexpectedly pleasant process to do the quick and dirty review though I’ve exposed to positive comments from former students. […] (P2011-13)

Quite enjoyable (N = 15)

• The overall experience of the QnD review is surprisingly fantastic. […] I think it is because I skimmed the articles so quickly that only essence of each paper remained and hence the review process tended to yield understandings of breadth rather than depth. […] (P2010-10)

Some Observations

• Students were able to do it.• Students are different.

• For some, it is quite painful.• e.g., a student spent 20 hours.

• For others, it is enjoyable.

• Why?• Language?• Personality? (N vs. S in MBTI)

Some Observations

• Pre-selected vs. self-selected• CHI vs. other journals• 50 vs. 100 vs. 120?• QnDReview should not replace

comprehensive reading.• One highly rated paper in QnDReview turned

out to be quite poor.

Limitations

• It’s not a controlled study.• The sample size is small.• Students’ self reports could be biased.

• They don’t want to offend me.

Future Work

• How to understand overall trends?• How to generate new ideas from

literature?• How to read? vs. What to read?

Acknowledgements

• I appreciate all the students who have participated in this homework, and I also thank Aditya Srinath and Sung-Hee Kim for reviewing this paper and provided constructive feedback. In addition, I appreciate Dr. V. Kathlene Leonard for giving me an assignment of reviewing all the papers in Human Factors.

CHI 1982 75 CHI 1983 59 CHI 1985 35 CHI 1986 47 CHI 1987 46 CHI 1988 39 CHI 1989 54 CHI 1990 47 CHI 1991 56 CHI 1992 67 CHI 1993 62 CHI 1994 70 CHI 1995 66 CHI 1996 55 CHI 1997 55 CHI 1998 81 CHI 1999 78 CHI 2000 72 CHI 2001 69 CHI 2002 61 CHI 2003 75 CHI 2004 93 CHI 2005 93 CHI 2006 151 CHI 2007 182 CHI 2008 157 CHI 2009 277 CHI 2010 302 CHI 2011 410 CHI 2013 392

3,326 CHI papers× 1-2 hours / paper÷ 8 hours / day= 416 - 832 days