Managing a Research Project: Analytical Methods and Tools

Patricia EnsworthHarborlight Management Services LLC

Etic and emic approaches

• Etic– Researcher’s interpretation of participant’s experience

• Emic– Participant’s interpretation of own experience

Types of research

• Strategic– Making a yes/no decision– Choosing among several alternative solutions

• Exploratory– Entering a new market– Creating a new product, service, or system– Developing new procedures

• Forensic– Determining the cause of a problem– Improving performance

Recruitment methods

• Recommendations– Management– Experts– Salespeople– Customer service staff

• Volunteers• Screening service• Snowball sample• Data mining

Data collection methods

• Archive review• Literature review• Vendor review• Data analytics• Questionnaire• Unstructured interview• Structured interview

• Observation• Participant observation• Diary study• Survey• Brainstorming session• Focus group• Usability test

Data collection methods

Method Process GoalArchive review Search organization’s internal records for metrics, issue

logs, lessons learned, system diagrams.Explore history and evolution of issues, identify SMEs, determine current knowledge.

Literature review Search external printed and on-line books, articles, and documents for relevant material.

Same as for archive review, placing organization in context with comparable enterprises.

Vendor review Examine RFPs, contracts, and service level agreements. Review communications between vendor and client. Inspect service logs.

Identify patterns of issues and understand vendor’s perspective.

Data analytics Develop queries and algorithms to review large quantities of electronic data.

Identify patterns of issues and behaviors.

Questionnaire Create a standard list of questions to be asked by an interviewer.

Ensure that all interviewees are asked the same questions.

Unstructured interview Ask interviewee general open-ended questions and allow the conversation to develop spontaneously in directions related to the central topic.

Explore background issues, understand interviewee’s perspective, generate ideas.

Structured interview Ask interviewee specific closed-ended questions and ensure the conversation stays on topic.

Gather data for making decisions.

Data collection methods (cont’d)Method Process GoalObservation Watch a person or group of people engaging in the

behaviors under study.Identify actual behaviors in contrast to reported behaviors.

Participant observation Accompany a person or group of people engaging in the behaviors under study and learn to perform the behaviors as an apprentice or assistant.

Understand the lived experience of the person or group of people.

Diary study Provide participant with a form containing questions about the attitudes and behaviors under study and ask him/her to fill it out multiple times on a regular basis.

Explore the changes in the behaviors and attitudes over time.

Survey Send participant a form containing questions about the attitudes and behaviors under study and ask him/her to fill it out once.

Gather similar types of data from multiple participants for comparison, categorization, and statistical computation.

Brainstorming Assemble a group of people who have an interest in a topic, ask for their ideas, and record all their responses without giving them criticism or analysis.

Explore a broad range of perspectives.

Focus group Assemble a group of people who have an interest in a topic, lead them in a structured discussion about specific questions, and record all their responses without giving them criticism or analysis.

Obtain insights for making decisions.

Usability test Provide participant with an item under study, ask him/her to use it in specific ways, and record all their behaviors and responses without giving them criticism or analysis.

Verify the viability and functionality of a design.

Data collection activities

• Demo• Walkthrough• Task analysis• Card sorting• Affinity grouping• Model building

• Collaging• Mind mapping• Location mapping• Movement mapping• Social network analysis• Information network analysis

Data collection activitiesActivity Process GoalDemo Participant shows how to use an item. Explore participant’s understanding of

and relationship with the item.Walkthrough Participant show how to navigate a location or process. Explore participant’s understanding of

and relationship with the location or process.

Task analysis Participant explains how to perform a sequence of actions to accomplish a goal.

Explore behavior range of universals, specialties, alter-natives, and peculiarities. Establish dependencies between tasks, define roles and responsibilities, identify workarounds.

Card sorting Participant reviews a group of index cards that already have one idea written on each card, and sorts them into piles according to categories he/she creates. Cards can also be used for task analysis.

Explore alternative methods of organizing a group’s standard information and activities.

Affinity grouping Participant is presented with a topic and responds with ideas he/she associates with that topic. Participant records each idea on a separate card, then sorts them into piles according to categories he/she creates.

Explore alternative methods of organizing an individual’s personal information and activities.

Model building Participant creates a three-dimensional object that represents the topic under study.

Explore alternative visions and interpretations of the topic.

Collaging Participant creates a collage that represents the topic under study.

Explore alternative attitudes and experiences of the topic.

Data collection activities (cont’d)Activity Process GoalMind mapping Participant is presented with a topic and responds with

ideas he/she associates with that topic. Participant focuses on each first-level idea and responds with second-level ideas he/she associates with the topic. The process continues for several rounds. A map is drawn showing the initial topic at the center and the subsidiary ideas as radiating branches.

Explore alternative cognitive structures, imagined relation-ships, and specialized language usages.

Location mapping Participant draws a map of a place. Explore alternative views of important sites and relative distances.

Movement mapping Participant is given a map of a location and draws lines showing habitual travel patterns. Line weight varies according to the frequency, style according to the method, and color according to the reason.

Explore environmental constraints, space layout, and resource usage.

Social network analysis Participant draws a diagram showing people with whom he/she interacts. Line weight varies according to the frequency, style according to the method, and color according to the reason.

Explore community structure, collaboration methods, and technology trends.

Information network analysis

Participant draws a diagram showing the flow of information as inputs and outputs through the systems necessary to perform his/her work.

Explore current system usage in contrast to original design, identify workarounds, discover manual processes.

Research plan example1. Archive review2. Literature review3. Vendor review4. Data analysis5. Questionnaire6. Unstructured interview7. Observation8. Structured interview9. Diary study10. Survey11. Focus group12. Usability test

Observation goals

• Activities• Environments• Interactions• Objects • Users

• Tools employed• Sequence of actions• Types of interactions• Methods of organization• Workarounds

Source: Observing the User Experience

Behavior variation

Behavior Description ExamplesUniversals What everyone should do Wear shoesSpecialties What some roles do Wear high heels, cleats, ice skatesAlternatives What some people do as personal taste Wear pink gladiator sandalsPeculiarites What strange people do Go barefoot on a city street

Source: Ralph Linton, The Study of Man

Diary study

Survey – demographic info

Survey – single vs. multiple response

Survey – numerical answers

Survey - ranking

Survey - scales

Data analysis tools - quantitative

• Pie chart• Bar graph• Pareto chart• Histogram • Distribution curve

• Run chart• Comparison table• Cross-tabulation table• Scatter diagram• Spider/Radar chart

Data analysis tools - quantitativeTool Process GoalComparison table For multiple categories and numerical data elements that do

not need to be counted or calculated, create a table with the elements as columns and the categories as rows.

Determine the relative quantities for the elements.

Cross-tabulation table For multiple categories and data elements that need to be counted or calculated, create a pivot table with the elements as columns and the categories as rows.

Determine the relative quantities for the elements.

Histogram For one numerical data element, create a series of equal ranges, and sort the responses into the ranges. Graph the height of each range on the x-axis and the number of responses on the y-axis.

Determine the relative number of responses in each range.

Distribution curve For one numerical data element, create a histogram. Calculate the mean and superimpose a bell curve of normal distribution on the histogram.

Determine the unique pattern of the responses.

Run chart For one numerical data element that has been recorded on multiple occasions over time, graph the time sequence intervals on the x-axis and the measurements on the y-axis.

Determine the range of variation to establish acceptable values.

Bar graph For one textual data element, sort the responses into categories. Count the items in each category. Create a vertical or horizontal scale showing the range of totals as bars of different heights.

Determine the relative sizes of the categories.

Data analysis tools – quantitative (cont’d)

Tool Process GoalPie chart For one textual data element, sort the responses into

categories. Count the items in each category. Create a circle and divide it into segments showing the range of totals as segments of different sizes.

Determine the relative sizes of the categories.

Pareto chart For one textual data element, create a vertical bar graph with the categories arranged in order of size from left to right. Show the number of responses in a scale on the left and the percentages from zero to 100 as a scale on the right.

Determine the relative importance of the categories.

Scatter diagram For two numerical data elements, create a graph with a range for one on the x-axis and the other on the y-axis. Plot the two elements as a point where the values intersect.

Determine the strength of correlation between the two data elements.

Spider/Radar chart For multiple numerical data elements and multiple textual data elements, create a circular graph with zero at the center and the largest numerical quantity at the circumference. Sort the textual data elements into categories. Arrange the categories around the circle and draw a line connecting the zero midpoint to each category at the circumference. Draw a scale for the numerical data elements in the style of a target. Plot the numbers for each category on the line where it intersects the scale. Connect the data points.

Determine the similarities and differences in the numerical data between the categories.

Pie chart

Bar graph

Pareto chart

Histogram

Bar graph vs. Histogram

Data distribution curve

Run chart

Comparison table

Cross-tabulation table

Digital file unavailable

Printer unavailable Rush job

Personal business

Expense Reports

Large quantity

Document adjustments Other Total

Marketing 11 9 7 22 16 4 12 3 84Legal 15 2 8 10 4 12 5 1 57IT 1 5 2 14 3 4 3 6 38Communications 8 6 12 9 5 15 15 2 72Finance 3 28 6 39 85 4 10 0 175Total 38 50 35 94 113 39 45 12 426

Scatter diagram of correlation

Spider / Radar chart

Data analysis tools - qualitative

• Comparison table• T-chart• Mind map• Activity network diagram• Force field analysis

• Fishbone chart• Storyboard• Personas• Flowchart with swimlanes• Procedure analysis

Data analysis tools - qualitative

Tool Process GoalComparison chart Method:

AllActivity:

AllArtifact:

Chart with data elements as columns and categories as rows.

Understand the relative value of the data elements.

T-chart Method:All

Activity:All

Artifact:Table comparing two aspects of one data element.

Understand the relative value of the aspects.

Data analysis tools – qualitative (cont’d)Tool Process GoalMind map Method:

Structured interviewActivity:

Mind mappingArtifact:

Visual representation of associations.

Understand alternative cognitive structures, imagined relationships, and specialized language usages.

Activity network diagram Method:Interview, observation, participant observation

Activity:Demo, walkthrough, task analysis

Artifact:Flowchart representing the normal logical sequence of activities without consideration of resources available or the participation of specific individuals or groups.

Understand which tasks cannot begin until others have been completed, identify gaps and redundancies in processes, establish stakeholder dependencies.

Force field analysis Method:Interview, diary study, survey, brainstorming, focus group.

Activity:Discussion

Artifact:T-chart summarizing a group’s pro and con attitudes about a topic.

Understand political and operational issues without reference to specific individuals or groups.

Data analysis tools – qualitative (cont’d)Tool Process GoalFishbone chart Method:

Archive review, interview, observation, participant observation, brainstorming, focus group, usability test

Activity:Demo, walkthrough, task analysis

Artifact:Diagram showing the causes of a problem categorized by people, process, equipment, materials, environment, management etc.

Understand the range of factors to a problem without reference to specific individuals or groups.

Storyboard Method:Interview, observation, participant observation

Activity:Task analysis, location mapping, movement mapping, social network analysis, information network analysis

Artifact:Graphic novel-style narrative illustrating a process that involves multiple actors.

Understand the ways in which people interact with each other, their tools, and their environment.

Persona Method:Literature review, vendor review, data analytics, interview, survey, focus group

Activity:Affinity grouping, collaging, model building

Artifact:A fictional profile of an imaginary person who represents a category of customer, user, or stakeholder.

Understand the various needs and desires of different categories, develop acceptance criteria, form marketing plan.

Data analysis tools – qualitative (cont’d)Tool Process GoalFlowchart with swimlanes

Method:Data analytics, interview, observation, participant observation, diary study

Activity:Demo, walkthrough, task analysis, social network analysis, information network analysis

Artifact:Flowchart representing the normal logical sequence of activities in a process as performed by specific individuals or groups.

Understand the roles and responsibilities of stakeholders in a process.

Procedure analysis Method:Data analytics, interview, observation, participant observation, diary study, focus group

Activity:Demo, walkthrough, task analysis, social network analysis, information network analysis

Artifact:Textual narrative describing a process that involves multiple actors, comprising both normal scenarios and common exceptions, workarounds, and failure recovery methods.

Understand the behavior of stakeholders not only when conditions are ideal but also when problems arise.

Comparison table

T-chart

Mind map

Activity network diagram - simple

Activity network diagram - complex

Force field analysis

Fishbone diagram template

Fishbone diagram example

Storyboard

Persona

Flow chart with swim lanes

Procedure analysis