guide for engaging with research partners about data and

Activities to Support Research Partner Involvement in Data Analysis and Interpretation 1

Guide for Engaging with Research Partners about Data and Analysis:Activities to Support Research Partner Involvement

in Data Analysis and Interpretation

Guide for Engaging with Research Partners about Data and Analysis2

CONTENTSAcknowledgements ................................................................................................................................................... 4Recommended Citation ........................................................................................................................................... 4Disclaimer Statement ............................................................................................................................................... 4INTRODUCTION .......................................................................................................................................................... 4

Why Use This Guide? ....................................................................................................................................... 5When Should This Guide be Used? ................................................................................................................. 5

OVERVIEW ................................................................................................................................................................... 5Who Should Use This Guide? ........................................................................................................................... 6What Is Not Included in This Guide? ............................................................................................................... 7Helpful Tools and Resources .......................................................................................................................... 8

CHAPTER 1: Designing Your Research Study .......................................................................................................... 9Overview ............................................................................................................................................................ 9Important Concepts .......................................................................................................................................... 9Study Aims and Research Questions .............................................................................................................. 9Methods ........................................................................................................................................................... 10Suggested Tips for Navigating the Codesign of Research with Stakeholder Partners .......................... 12Further Reading ............................................................................................................................................. 13Case Study 1: Selecting the Study Design .................................................................................................... 14Chapter 1: Activities ........................................................................................................................................ 15

Activity 1a: Stakeholder Experience with Data Analysis and Interpretation ................................... 15Activity 1b: Personal Statement Worksheet ....................................................................................... 17Activity 1c: The PICOTS Framework: Designing Our Study ............................................................... 19Activity 1d: What Data Sources Does Your Study Use? ................................................................... 21

CHAPTER 2: Designing Your Data Analysis Plan ................................................................................................... 22Overview .......................................................................................................................................................... 22Important Concepts ........................................................................................................................................ 22Study Variables ................................................................................................................................................ 22Independent and Dependent Variables ....................................................................................................... 24Covariates ........................................................................................................................................................ 25Suggested Tips for Codesigning the Data Analysis Plan with Research Partners ................................... 27Further Reading ............................................................................................................................................. 28Case Study 2: Designing Your Data Analysis Plan ....................................................................................... 29Chapter 2: Activities ........................................................................................................................................ 30

Activity 2a: Understanding Research Design and Implications for Your Study .............................. 30Activity 2b: Identifying the Variables in Your Study ........................................................................... 32

CHAPTER 3: Analyzing Your Study Data ............................................................................................................... 34Overview .......................................................................................................................................................... 34Important Concepts ........................................................................................................................................ 34


Hypothesis Testing.......................................................................................................................................... 34Statistical Methods.......................................................................................................................................... 35Steps in Null Hypothesis Significance Testing ............................................................................................ 35Variations of Relative Risk .............................................................................................................................. 40Problems with Missing Data .......................................................................................................................... 41Suggested Tips for Facilitating Data Analysis with Research Partners ..................................................... 42Further Reading ............................................................................................................................................. 43Case Study 3: Analyzing STUDY DATA .......................................................................................................... 44Chapter 3: Activities ........................................................................................................................................ 45

Activity 3a: Using Descriptive Statistics in Your Study ....................................................................... 45Activity 3b: Using Inferential Statistics in Your Study ....................................................................... 47

CHAPTER 4: Interpreting Your Study Data ........................................................................................................... 48Overview .......................................................................................................................................................... 48Important Concepts ........................................................................................................................................ 48Internal Validity ............................................................................................................................................... 48When Research Doesn’t Go as Planned ...................................................................................................... 50External Validity (Generalizability) ................................................................................................................ 50Study Limitations ............................................................................................................................................ 52Suggested Tips for Interpreting Data with Research Partners .................................................................. 53Further Reading ............................................................................................................................................. 54Case Study 4: Interpreting Study Results ..................................................................................................... 55Chapter 4: Activities ........................................................................................................................................ 57

Activity 4a: Interpreting the Results—Overall Study Findings ......................................................... 57Activity 4b: Interpreting the Results—Sources of Bias and Study Limitations .............................. 58Activity 4c: Interpreting the Results—Sources of Bias and Study Limitations ............................... 59Activity 4d: Facilitating Stakeholder Interpretation of Results Using Data Visualization .............. 60

CHAPTER 5: Reporting and Presenting Study Results ........................................................................................ 62Overview .......................................................................................................................................................... 62Important Concepts ........................................................................................................................................ 62Tailoring Results Dissemination & Promotion to Community Audiences ............................................... 62Suggested Tips for Reporting Study Findings with Research Partners .................................................... 64Further Reading ............................................................................................................................................. 65Case Study 5: Reporting and Presenting Study Results ............................................................................. 67Chapter 5: Activities ........................................................................................................................................ 68

Activity 5a: Creating a Plain-Language Summary .............................................................................. 68Activity 5b: Write Key Messages Tailored for Each Target Audience ............................................... 71

GLOSSARY ................................................................................................................................................................. 75RESOURCE TABLE ..................................................................................................................................................... 85


ACKNOWLEDGEMENTS

The PCORI Public and Patient Engagement team thanks PCORI staff, community partners, and commissioned project team members who have contributed to the planning and development of this resource.

The PCORI Public and Patient Engagement team members gratefully acknowledge our community partners who provided critical input and user testing in the early stages of this project. They acknowledge Larry Kessler, Monique Does, James Harrison, Regina Greer Smith, and Michele Fritz.

A special thank you to Danielle Lavallee for bringing the concepts of research engagement during data analysis to life by leading the development and refinement of the statistical content, activities, and case studies for this resource. We would also like to thank other project team members from Westat including Russ Mardon, Jennifer Nooney, David Gilden, and Elena Tran.

Lastly, the PCORI Public and Patient Engagement team members gratefully acknowledge staff members, both current and former, who helped provide guidance and input throughout the development process. They acknowledge Julia Anderson, Ayodola Anise, Whitney Brower, Andrea Heckert, Esther Nolton, Regina Reid, Lisa Stewart, Robert Treadway, and Krista Woodward.

All authors and partners on this resource hope that it provides all research teams the help and support they need to meaningfully engage patients and stakeholders throughout the research process.

RECOMMENDED CITATION

To cite this document, please use: PCORI Guide for Engaging with Research Partners about Data and Analysis: Activities to Support Research Partner Involvement in Data Analysis and Interpretation. Patient-Centered Outcomes Research Institute. September 2021. www.pcori.org. Accessed [fill in date].

DISCLAIMER STATEMENT

Any resources or websites outside of PCORI linked to in this work were developed and are hosted by third parties. Opinions and information in these third-party sites do not necessarily represent the views of PCORI. Accordingly, PCORI cannot make any guarantees with respect to the accuracy or reliability of the information and data in this content. Please see the third party-website terms of use for additional information.

INTRODUCTION

http://www.pcori.org


Our mission at PCORI is to fund patient-centered, comparative clinical effectiveness research (CER)—otherwise known as patient-centered outcomes research (PCOR)—that can help patients and those who care for them make better-informed healthcare choices. Including the voices of patients, caregivers, advocacy groups, health system representatives, and others in study planning and design helps accomplish this.

When partnered with research teams, patient and stakeholder partners bring distinct lived experiences and expertise to a study that help ensure the results will be meaningful to the populations they represent, the healthcare community, and others. Research partners bring diverse perspectives and experiences to the team, helping to frame the ways evidence gaps affect a patient’s or clinician’s ability to make healthcare decisions. Research partners contribute to the interpretation of new data by framing results in the context of lived experience and can help research teams translate study findings for unique, diverse communities. Partner involvement also improves the potential for uptake of research results into clinical practice and policy.

Why Use This Guide? Many researchers have become comfortable engaging stakeholders as research partners in study design and implementation, however, the research community is just beginning to realize the need to engage diverse communities in data interpretation and translation. Historical underrepresentation of communities of color in data analysis, and research as a whole, suggests missing perspectives in study findings’ interpretations.

While comfort in engaging research partners is growing, many investigators are unsure how to expand engagement during quantitative data analysis, interpretation, and results reviews.

Many perceive statistics and modeling as complex topics, often requiring advanced study and experience. If researchers lack the skills or confidence to present complex information effectively, research partners may be left out of the data analysis process entirely. This collective uncertainty and lacking preparedness have led to an “engagement slump” that occurs during the data analysis phase in research projects.

To help close this gap, PCORI has developed this Guide for Engaging with Research Partners about Data and Analysis to prepare research staff to involve research partners in analyzing and interpreting quantitative data. This Guide will present concepts, terms, and examples that are specific to CER and PCOR methodologies. This guide provides a series of tips, resources, and activities to help research staff foster communication with partners and build their capacity to actively participate in data analysis and interpretation.

When Should This Guide be Used?This guide provides activities that support capacity building for data analysis and interpretation, from study beginning to completion. Discussions about data analysis and interpretation change over the course of the study, ranging from what research question the study should answer to how to present findings to diverse audiences. This Guide recommends preparing for data analysis and interpretation at the beginning of the study but provides some specific time points for teams to keep in mind (Figure 1).

OVERVIEW


Early in the proposal development and protocol refinement stage (Develop Study), teams work together to create the study design that will address gaps in evidence. This an opportunity for the entire research team, including stakeholder partners, to discuss how decisions about study design and outcomes will shape data analysis. Once the study is under way (Gather Data), the teams collaborate to prepare for data analysis and interpretation. This includes co-learning, when researchers learn from partners how they will use study findings and how the information will aid decision making. Research partners may need more in-depth knowledge about the process and technical aspects of conducting research. Through regular interactions and discussions, as recommended in this Guide, research teams can set a foundation for working together and identify important skills needed to analyze and interpret information (Analyze Data). Finally, teams discuss how to present data to diverse end users (Present Results). Research teams can adapt and use the content, resources, and tools in this guide for project meetings, patient and stakeholder advisory

board meetings, presentations, or other events to support partner engagement. They are meant to be flexible enough for use in any type of study and can be adapted for in-person or virtual discussion formats. These important conversations have a direct impact on the design and conduct of a study and, ultimately, data analysis and interpretation.

Who Should Use This Guide?The primary audience for this guide is the research community, including staff who are supporting stakeholder research partnerships. It is particularly aimed at supporting the work of study or project managers, engagement coordinators/managers, and other research staff that would be responsible for leading meetings and/or interactions with research partners. Suggested activities and exercises, when possible, are presented in plain language to support use by teams with research partners from diverse backgrounds.

Figure 1. Data Informs Every Step in the Research Process

DevelopStudy

GatherData

AnalyzeData

PresentResults

Co-create research to

answer important questions

Co-learn to prepare to

analyze and interpret data

Collaborate to analyze, understand, interpret and

present study findings


How is the Guide Organized?This guide is organized by phases of research development and conduct (Figure 1), recognizing that topics discussed will build on each other as team members work together and relationships deepen. We encourage research teams to use examples relevant to their study or topic of interest in discussions with research partners.

Each chapter in the guide includes these elements:

1. The Important Concepts section provides educational content that can be used to explain concepts to research partners. Guidebook users can extract content to create educational resources (handouts, slides, etc.) to distribute to research partners.

2. The Tips section highlights suggestions to support valuable conversations between research team members and partners when analyzing data and interpreting findings. Research staff can learn from others how to navigate complex conversations with research partners during the data analysis and interpretation phases.

3. The Case Study section presents example scenarios that could arise throughout the research process. The case studies and reflective discussion questions are intended to support teams early as they plan engagement with research partners.

4. The Activities section provide materials to help facilitate partner involvement throughout the research continuum. Guidebook users can use activities and cite resources during research partner meetings to reinforce new concepts.

5. The Further Reading section provides optional articles and resources to supplement the Data Guidebook important concepts. This section includes resources from PCORI and elsewhere to provide additional information for those interested in learning more.

What Is Not Included in This Guide?This Guide is not intended to be an introductory textbook or course for statistics. It covers statistical analytic concepts in enough depth to fulfill a research partner’s engagement needs without being overwhelming. We do not think that patients and other stakeholders need a comprehensive statistics background or education to meaningfully contribute to data analysis and interpretation in a research study.

While qualitative methods are important to patient-centered outcomes research (PCOR), they are beyond the scope of this guide. However, this guide includes several resources and tools on qualitative methods and data analysis that research terms are encouraged to review.


HELPFUL TOOLS AND RESOURCES

Several tools and resources are available for research teams to support stakeholder engagement. While specific references are provided within activities, a number of materials and resources exist. Some of the concepts and examples in this Guide are directly tied to PCORI’s Research Fundamentals learning package. The Guide aims to supplement content shared in Research Fundamentals to provide users with a multimedia experience, particularly when working with partners on data analysis and interpretation. If you plan to utilize the Research Fundamentals learning package as part of your partner research training and education, we encourage you to use this resource for additional and complementary exercises, tips, and resources.

Additionally, PCORI staff regularly update the PCORI Engagement in Health Literature Research Explorer and PCORI Engagement Tool and Resource Repository where resources developed by PCORI-funded teams are made available for public access. Also, PCORI has developed many tools and resources to support research capacity building:

• PCORI Research Fundamentals• PCORI Engagement in Health Research Literature Explorer• PCORI Engagement in Research Fact Sheet • PCORI Updated Engagement Plan Template• PCORI Research Methods 101• PCORI Methodology Standards• Building Effective Multi-Stakeholder Teams (need a link)

https://www.pcori.org/engagement/engagement-resources/Engagement-Tool-Resource-Repository

https://www.pcori.org/engagement/research-fundamentals

https://www.pcori.org/engagement/engagement-literature

https://www.pcori.org/sites/default/files/PCORI-Engagement-in-Research-Making-a-Difference-Webinar-Info-Sheet-091917.pdf

https://www.pcori.org/sites/default/files/PCORI-Updated-Engagement-Plan-Template.pdf

https://www.pcori.org/sites/default/files/PCORI-Methodology-101-Training-Booklet-and-Resource-Guide.pdf

https://www.pcori.org/research-results/about-our-research/research-methodology/pcori-methodology-standards


CHAPTER 1Designing Your Research Study

OVERVIEW

Research design provides the foundation for data analysis and interpretation. Decisions such as the data source, types of data collected, and duration of study follow-up inform how the data analysis is conducted. During this time, research teams have an opportunity to lay a foundation for working with stakeholder partners during data analysis and interpretation by 1) understanding stakeholder partners’ prior experiences with research and 2) providing early training on research conduct.

Involving stakeholder partners in study planning and design ensures that the project and results are useful and important to patients, caregivers, and others who will use or be affected by the results. During this phase of research, stakeholders inform question development, outcome selection, and data collection methods. Research partners also provide valuable perspective on ways to tailor interventions to the target community that will best meet the needs of study participation and improve recruitment and retention.

Expanding research teams to include diverse stakeholder perspectives creates a new opportunity to share decisions when designing studies. Traditionally, the selection of study design is made by research team members with extensive training in research methodology and statistics who make decisions about design based on the research question at hand, the specific aims of the study, the primary outcome(s) of interest, data source(s) available, and the time horizon needed to evaluate outcomes. Other aspects of study design, including the process for conducting the study and any effect on study participants, are also important to consider. Involving patients and stakeholders in the selection of study design may result in broader discussions not normally encountered and may require additional time and meetings to address issues stakeholders may raise.

Activities included in this chapter include the following: strategies for understanding partners’ experience and knowledge with data analysis and interpretation, PICOTS exercise, as well as data source questions.

IMPORTANT CONCEPTS

There are many ways to design a study to answer research questions. During the design of a research study, stakeholder partners will benefit from understanding the core elements that inform the study design and the implications of any decisions made. The study design will impact the findings and the ultimate use and application of them in real world settings. Consider these concepts as discussion topics for your research team and partners.

Study Aims and Research QuestionsThe term study aims refers to the main goals or overarching questions addressed by the research project. Study aims are general and high level and are almost always positioned at the very beginning of a Data Analysis Plan. Studies often have more than one study aim, typically termed primary, secondary, and tertiary aims.

Research questions provide more detail about the topics to be investigated in the research plan. Study aims are goal statements and related questions are written to investigate an aspect of validity in that statement. For example, a study aim could be, “To investigate which cardiac rehabilitation program is most effective at improving patient outcomes following a heart attack.” A related research question might then ask, “Are there group differences in blood pressure over time (or after one year) between individuals who receive the intervention compared to those that do not?” This research question can then be used to define specific variables that will be measured and used in statistical analyses. Many PCORI-funded studies have research questions that focus on the comparative effectiveness of different types of treatments. Find additional tips on writing study aims in the Resources section of this chapter.

Research partners should understand and inform the relationship between study aims and associated research questions whenever possible. The aims and research questions will inform and frame how the data are analyzed and how the results are interpreted at a later phase of the study.


HypothesesOften, the study aims section will include hypotheses along with research questions. Hypotheses and research questions are separate yet complementary. Hypotheses predict the answer to a research question based on theory and existing evidence. There are two types of hypotheses: null, the absence of an effect, and alternative, any hypothesis based on theory or evidence that indicates an effect/difference between groups.

• Null hypothesis (H0) = No effect• Alternative hypothesis (HA or H1, 2) =

Observed effect

Hypotheses are often stated as an expected relationship between two or more variables that are measured with data. For example, a hypothesis might state, “Individuals who attend weekly group classes as compared to individuals who receive weekly virtual one-on-one coaching for 12 weeks have better blood pressure control at 1 year.” Sometimes hypotheses are based on earlier exploratory research that led up to the current study, and sometimes they are based on studies in medical literature. If the Data Analysis Plan includes hypotheses, the study data will be analyzed using statistical tests to see if the results support the hypothesis/es.

MethodsStudy methods inform the data analysis plan and describe which outcomes will be measured and how the data will be collected and analyzed. Methods detail which treatments study participants receive, which groups will be compared, and which patients may or may not enroll in the study.

Each of these considerations of study methods influences the types of conclusions that can be drawn from the study and should be reviewed at its start and/or prior to data review. All of these concepts will ultimately lay the foundation for the study data analysis plan, which will be explored more deeply in chapter 2.

• Study population: This is the population in which the research team will test the study’s hypotheses and/or generalize the study observations. Depending on the research questions, it may be necessary to describe a study population based on demographics such as age, sex, race/ethnicity, marital status, education level,

place of residence, type of medical insurance, and socioeconomic status. It may also be important to describe the study population according to their health, such as past or present diseases or health conditions, medications received, ability to perform activities of daily living, etc.

The features describing the study population are usually presented in two lists: one naming things that would allow someone to enter a study (inclusion criteria) and the other listing things that will keep someone out of the study (exclusion criteria). These lists form the study’s eligibility criteria. Clearly defining the study population and using the eligibility criteria when participants are recruited ensures that the study’s conclusions are valid and apply to people outside the study who have shared characteristics with the study population.

Inclusion and exclusion criteria also help ensure a participant’s response to an intervention isn’t attributable to something other than the intervention being tested. Using the example above, patients on blood pressure medication could be excluded because changes in their blood pressure control might be due to the medication and not related to the coaching intervention.

The methods may also describe whether there will be any sampling from the study population, and if so, how the sample is selected. It may also point out any subgroups, such as men or women or those who have more severe illness versus less severe illness, that will be analyzed separately.

When designing a study, it is very important to think about the number of participants needed. If the population sample is too small, it will be hard to get a statistically significant result. If the sample is too large, it may yield statistically significant results that are not meaningful in real life. A too-large sample can also make the study more expensive to carry out. Research teams should discuss issues around sample size including feasibility, cost, and implications for data analysis early on.

• Comparators: The comparators, or comparison groups, are defined by


the treatments or interventions being compared. The comparisons may be between different treatments being tested or between a treatment and usual care. The Data Analysis Plan should clearly define how individuals are assigned to comparison groups and note whether group assignment is random or predetermined.

• Data Sources: The Data Analysis Plan should explain the sources of the data used in the study and describe where the data for each characteristic, or variable, will come from. It is important for research partners to review the data sources because their knowledge of the study population may help the research team rethink their data collection strategies and plans. See chapter 6 for more information and activities about data collection.

• Outcomes and other variables: Primary outcomes, usually defined at the time the study is designed, will be used to answer the main research questions. Secondary outcomes are additional measurements that may help answer other questions or generate hypotheses for future research. Outcomes may be measured using a variety of data sources.

• Statistical methods: The statistical methods section of a study will describe the types of statistics that will be used to analyze the data. The statistical tests used help us interpret the study results and indicate if comparisons or changes over time are important. These include descriptive statistics like the average or maximum that summarizes the characteristics and outcomes of the study population by size and frequency. Another type of statistics–inferential statistics–looks for the links between the comparators and the study outcomes

The statistical methods section may also describe how many people are needed in the study. This has to do with the size of outcome differences between comparator groups that will be considered significant (not due to chance) as well as clinically meaningful (large enough to be considered important in a care setting). This type of analysis, called a power analysis, helps researchers fit the size of the study sample to find an approximate number of participants needed so that the sample size is not too large or too small. Statisticians use these determinations to estimate the mandatory study population or sample size to detect significant and meaningful results.

• Timeline: Analytic plans usually include a timeline representing the dates when the main steps of the study will happen. For example, the timeline will indicate when recruiting starts, when data collection starts and ends, and when the results should be ready. The timeline may be shown in different ways such as a flow chart or a table. Throughout the study, the timeline helps the study team keep track of progress and shows what is coming up next. The timeline is also a helpful tool to share when and how research partners will be involved in study activities.


SUGGESTED TIPS FOR NAVIGATING THE CODESIGN OF RESEARCH WITH STAKEHOLDER PARTNERS

Expanding teams to include people with diverse experiences and perspectives broadens the view of how research may be leveraged to answer important questions. It also changes the traditional decision-making dynamic from one that is traditionally led by the principal investigator to one where decisions are made in partnership with others. Here is a checklist to consider as you prepare to engage partners at this stage of the research process.

Identify early with stakeholder partners any experience with research training and needs for partnering in research design and conduct.

Identify early with researchers any experience with and training needs for working with stakeholders throughout the design, conduct, and reporting of research.

Identify early within the research team any training needs for group facilitation, including consensus building, if necessary.

Identify a meeting facilitator to serve as discussion moderator and provide structure for the meeting.

● Facilitators should have knowledge of group processes and enough knowledge of engagement to provide adequate structure without imposing personal views or opinions.

● Ensure proper planning for meeting logistics, including how interactions will occur if remote participation is supported (e.g., use of video conferencing to support group discussion).

● Start discussions by acknowledging any ground rules, including the importance of respect for others’ opinions and experience.

Identify key personnel on the research team who have expertise explaining/teaching research design concepts and invite them to join/lead stakeholder meetings as needed.

Identify any reporting guidelines (e.g., CONSORT, SQUIRE, SPIRIT, etc.) relevant for managing future publications and presentations. Incorporate these checklists into discussions with stakeholders.

Provide stakeholder partners access to the study proposal and materials as reference throughout the study. Reference or call out areas that are pertinent to discussions as the research study is designed.

Solicit stakeholder partners’ help to pretest surveys, interview guides, or other data collection instruments.

Create a repository of resources and references relevant to the study to support involvement throughout study conduct and make it available to stakeholder partners. Reference specific materials, tools, and/or resources as needed prior to any group discussions to allow for preparation.


Any resources or articles included in the Further Reading list are optional and considered supplementary to the Data Guidebook. PCORI is not responsible for content accuracy or reliability developed by third parties. PCORI is not responsible for any fees associated with any of these resources.

Cashman, Susan B., Sarah Adeky, Alex J. Allen 3rd, Jason Corburn, Barbara A. Israel, Jaime Montaño, Alvin Rafelito, Scott D Rhodes, Samara Swanston, Nina Wallerstein, and Eugenia Eng. 2008. “The power and the promise: working with communities to analyze data, interpret findings, and get to outcomes.” Am J Public Health. Aug;98(8):1407-17. https://doi.org/10.2105/ajph.2007.113571.

“The Belmont Report. Ethical principles and guidelines for the protection of human subjects of research,” Department of Health, Education, and Welfare; National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. 1979, accessed September 3, 2018, https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/.

Domecq, Juan Pablo, Gabriela Prutsky, Tarig Elraiyah, Zhen Wang, Mohammed Nabhan, Nathan Shippee, Juan Pablo Brito, Kasey Boehmer, Rim Hasan, Belal Firwana, Patricia Erwin, David Eton, Jeff Sloan, Victor Montori, Noor Asi, Abd Moain Abu Dabrh, and Mohammad Hassan Murad. 2014. “Patient engagement in research: a systematic review.” BMC Health Serv Res. Feb 26;14:89. https://doi.org/10.1186/1472-6963-14-89.

Esmail Laura, Emily Moore, and Allison Rein. 2015. “Evaluating patient and stakeholder engagement in research: moving from theory to practice.” J Comp Eff Res. Mar;4(2):133-45. https://doi.org/10.2217/cer.14.79.

Forsythe Laura, Andrea Heckert, Mary Kay Margolis, Suzanne Schrandt, and Lori Frank. 2018. “Methods and impact of engagement in research, from theory to practice and back again: early findings from the Patient-Centered Outcomes Research Institute.” Qual Life Res. Jan;27(1):17-31. https://doi.org/10.1007/s11136-017-1581-x.

Hall, Budd L. 1992. “From margins to center? The development and purpose of participatory research.” Am Sociol, 23(1) (Winter): 15-28. https://doi.org/10.1007/BF02691928.

Iezzoni, Lisa I., and Lisa M. Long-Bellil. 2012. “Training physicians about caring for persons with disabilities: ‘Nothing about us without us!’” Disabil Health J. Jul;5(3):136-9. https://doi.org/10.1016/j.dhjo.2012.03.003

Maren Klawiter. 2004. “Breast cancer in two regimes: the impact of social movements on illness experience.” Sociol Health Illn. Sep;26(6):845-74. https://doi.org/10.1111/j.0141-9889.2004.00421.x.

Patient-Centered Outcomes Research Institute. 2014. “PCORI Engagement Rubric.” Last modified October 12, 2015, accessed June 3, 2021. https://www.pcori.org/sites/default/files/Engagement-Rubric.pdf.

Patient-Centered Outcomes Research Institute. 2015. “PCORI Methodology Standards.” Last modified February 26, 2019, accessed June 3, 2021. https://www.pcori.org/research-results/about-our-research/research-methodology/pcori-methodology-standards.

Shippee, Nathan D., Juan Pablo Domecq Garces, Gabriela J. Prutsky Lopez, Zhen Wang, Tarig A. Elraiyah, Mohammed Nabhan, Juan P. Brito, Kasey Boehmer, Rim Hasan, Belal Firwana, Patricia J. Erwin, Victor M. Montori, and M. Hassan Murad. 2015. “Patient and service user engagement in research: a systematic review and synthesized framework.” Health Expect. Oct;18(5):1151-66. https://doi.org/10.1111/hex.12090.

Swartz, Lee J., Karen A. Callahan, Arlene M. Butz, Cynthia S. Rand, Sukon Kanchanaraksa, Gregory B. Diette, Jerry A. Krishnan, Patrick N. Breysse, Timothy J. Buckley, Adrian M. Mosley, and Peyton A. Eggleston. 2004. “Methods and issues in conducting a community-based environmental randomized trial.” Environ Res. Jun;95(2):156-65. https://doi.org/10.1016/j.envres.2003.08.003.

FURTHER READING

https://doi.org/10.2105/ajph.2007.113571

https://doi.org/10.2105/ajph.2007.113571

https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/

https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/

https://doi.org/10.1186/1472-6963-14-89

https://doi.org/10.1186/1472-6963-14-89

https://doi.org/10.2217/cer.14.79.

https://doi.org/10.2217/cer.14.79.

https://doi.org/10.1007/s11136-017-1581-x

https://doi.org/10.1007/BF02691928

https://doi.org/10.1007/BF02691928

https://doi.org/10.1016/j.dhjo.2012.03.003

https://doi.org/10.1016/j.dhjo.2012.03.003

https://doi.org/10.1111/j.0141-9889.2004.00421.x

https://doi.org/10.1111/j.0141-9889.2004.00421.x

https://www.pcori.org/sites/default/files/Engagement-Rubric.pdf






https://doi.org/10.1111/hex.12090

https://doi.org/10.1111/hex.12090

https://doi.org/10.1016/j.envres.2003.08.003


CASE STUDY 1: SELECTING THE STUDY DESIGN

Background: Many PCOR questions address the comparative difference between one or more treatment options where the intensity and burden of treatment are significantly different. Randomized clinical trials produce the highest quality of evidence when a clear rationale for selecting one treatment over another does not exist. This is called clinical equipoise. It provides the ethical basis for randomly assigning people to receive different treatments. Yet even with equipoise, conducting a randomized clinical trial requires, in part, that patients be willing to participate.

Challenge statement: When clinical equipoise exists, randomization may be impeded by participants’ willingness to consent to take part in the study as designed.

Example scenario: In response to a systematic review identifying an important evidence gap in the management of a chronic condition, a research team decides to design a randomized controlled trial comparing patient-centered outcomes (e.g., function, survival, recurrent disease) between two accepted treatments. One treatment involves an extensive surgery, the other involves intensive medical management. Both treatments have potential for adverse events that may affect quality and quantity of life. Both treatments are used in practice, but a direct head-to-head comparison has not yet been made.

Conducting a randomized trial between the two interventions will help better guide treatment decisions patients and their care teams make. During early discussions with stakeholders, the research team received feedback that despite the need for the study, randomization would likely be impossible. Patient stakeholders (those with experience with both treatments) noted a lack of willingness to randomize into the study since there would be strong preferences for the type of treatment selected. Clinician stakeholders (and members of study sites where the trial would take place) indicated a lack of willingness to approach patients for participation. While clinicians recognized the need for evidence to guide decisions, the patient preferences that often drove

shared decisions around treatment make randomization difficult.

Resolution: Based on initial feedback from research partners, the research team conducted an anonymous survey among patients and care partners to understand willingness to randomize. Feedback from over 300 participants indicated that more than 80 percent of respondents would decline participation. Given the number needed for the study, the research team deemed it impossible to conduct the study with such a low expected willingness to participate. As a result, they proposed an observational study design and included the rationale for study selection in the study proposal. The study successfully received funding with reviewer comments recognizing that while a need exists for a randomized clinical trial to produce the highest quality of evidence, the feasibility and burden caused to participants would make the conduct very unlikely.

Discussion questions: Use the following questions to discuss your research study and research partnerships to support planning for study and protocol design:

• How will your study team decide on the final study design?

• Where can your team envision similar tension points while selecting your project’s study design? What design trade-offs can you offer given patient and stakeholder views?

• How will you negotiate opposing ideas related to study design to ensure patient-centeredness and authentic stakeholder engagement?

• How will stakeholder input inform decisions regarding the study design? What issues might be contentious?

• How can your team plan to support discussions with research partners?

• How will your team document results from discussions and decisions made about study design?


CHAPTER 1: ACTIVITIES

Activity 1a: Stakeholder Experience with Data Analysis and InterpretationBelow are several discussion questions for early research team meetings to help build a mutual understanding of the research process, with an emphasis on data analysis and interpretation. This can serve as a foundation for understanding the research team’s motivations for involvement. These questions can be sent to both researchers and research partners prior to the meeting. Note: This activity format is an agenda for a meeting with research partners. The content can be adapted as needed.

ObjectivesThis activity supports the following objectives:

1. Support team building to orient multi-stakeholder teams on motivations for partnering on a research study

2. Define roles for partnership including goals for data analysis and interpretation

Discussion Agenda

5 minsWelcome & Objectives

The facilitator shares his or her role and the objectives for the day.

10-15 mins

Round Robin Introductions

• Invite each person to introduce themselves, providing the following information:

o What makes you excited about being part of this research partnership?

o What contribution would you like to make to the study?

o What do you want to learn from participating on this study team?

10 mins

Present/Review: Roles of Research Partnership

• Provide a brief recap of the stages for research and goals for engagement across the research study.

• Share plans for study organization and structure, if developed.

25 mins

Group Discussion

• What experiences have you had collaborating on other research studies?

• What background and expertise do you want to contribute to the study?

• Have you been involved in data analysis or interpretation of research findings in other research studies?

5 mins

Wrap-Up

• Reflect on the discussion and craft a professional/personal statement using the Personal Statement Worksheet (Activity 1.b) as a follow-up to the meeting.


Session Preparation Materials for Stakeholder Partners

• PCORI Research Fundamentals Learning Package: Module 1 • PCORI Research Methods 101• PCORI Engagement Challenges, Strategies, and Resources • Optional: Study specific stakeholder engagement plan (if available)



https://www.pcori.org/sites/default/files/PCORI-Patient-Stakeholder-Engagement-Challenges-Strategies-Resources-Handout-120517.pdf


Activity 1b: Personal Statement WorksheetAll research team members should have short bios describing their role on the team, experience with the topic, and reason for being part of the study. Short bios are helpful to have at the beginning of the project so everyone knows who is involved. As a way to facilitate early discussions about data analysis and interpretation, we have included some specific prompts about experience and knowledge of data analysis. They are also good to have for presentation and publication materials. Discussion questions presented in Activity 1a are adapted below as a worksheet to help research partners formulate research bios. Users are welcome to add or tailor questions for their needs.

ObjectivesThis activity supports the following objectives:

1. Support team building to orient multi-stakeholder teams on motivations for partnering on a research study

2. Define roles for partnership, including goals for data analysis and interpretation

Activity OverviewDiscussion questions presented in Activity 1a are adapted below as a worksheet to help research partners formulate research bios. The worksheet can be completed during a group meeting, during a 1:1 discussion, or as an independent activity.

Personal Statement Worksheet

Name: Date:

Briefly Answer the Following Questions:

1. What excites you about being part of this research partnership?

2. What experiences have you had participating in or collaborating on other research studies?

3. What background and expertise do you want to contribute to the study?

4. Do you have experience analyzing or interpreting data for studies previously? If so, can you share more?


Create a Personal Statement: Reflect on the responses above and craft a personal statement highlighting your experience with the research topic and goals for your involvement. The information provided will help inform others learning about the research study about the breadth of interests and experiences on the research team.


Activity 1c: The PICOTS Framework: Designing Our StudyTo design a PCOR study comparing different treatments or interventions, a series of decisions are necessary at the beginning. This activity provides an opportunity for the research team to work with stakeholders to place the population, intervention, comparators, outcomes, timeframe, and setting into the context of the research question. This is the PICOTS framework. Through small group discussions, research team members can discuss each aspect of the framework to identify important features that should be accounted for in the study. Developing research questions and study design will impact the interpretation of study findings and should be accounted for at this stage of planning.

ObjectivesThe activities below support the following objectives:

1) Develop and refine research questions that support future data interpretation efforts with stakeholder partners

2) Advance stakeholder understanding of research concepts through applied learning

PICOTS Framework: Discussion Agenda



10 mins Present/Review: PICOTS

• Review the PICOTS framework• Discuss application to research study

20 mins

Breakout Group Discussion

• Use the PICOTS framework to discuss your study in small groups. If preferred, select a journal article to discuss the PICOTs framework following a journal club approach. If choosing an article, consider one from the literature that relates to your study. For example, select a study that informed the background and rationale for pursuing the study under development.

• Use the discussion questions in the table below to facilitate small group discussions. Space is allocated for documenting discussions.

20 mins

Report Out and Group Discussion

• A member of each breakout session reports the key points identified and discussed by the group.

• As a group, discuss the implications for the research questions and study design on how data can be analyzed and interpreted.

• Hold a group discussion on the potential for uptake of results, weighing pros and cons of study design selected.

5 minsWrap-Up

• Present to the group the next steps and how the information discussed will inform the study design.


PICOTS Breakout Discussion Questions

Discussion questions Group responses

Population • For which populations is this question important? Consider age, geography, co-existing conditions, ethnicity, culture.

• Would any populations require special monitoring if included in this study?

• Should any populations be excluded from the study due to high risk or ethical considerations?

Interventions & Comparators

• Which treatments or interventions are important to consider in the context of these questions?

• What treatments are you aware of for this condition? (Alt: What interventions or programs have been used/applied to address this issue?)

Outcomes • What are potential benefits for each treatment?

• What are potential harms for each treatment?

• What other aspects of health or health care might the treatment/intervention influence or affect?

Timeframe • How long should the treatment last?• When would one expect to see change?• How long do benefits of the treatment last?

Setting • In what setting is the treatment(s) given?• In what setting(s) does the intervention occur? • Is the study in one city or region? What factors

should be considered if more than one setting is included (e.g., geography, size, setting type [academic/community-based])?

Session Preparation Materials

• PCORI Research Fundamentals Learning Package: Module 1• PCORI Research Methods 101• The PICOTS Framework: How to Write a Research Question



https://www.pcori.org/engagement/engagement-resources/Engagement-Tool-Resource-Repository/picots-framework-how-write


Activity 1d: What Data Sources Does Your Study Use? In healthcare research, there are several common ways to obtain data. Each data source has strengths and weaknesses in terms of how complete and accurate the data are, time and financial costs, and how the data can be used to measure outcomes and other important variables. Gathering data and preparing it for analysis takes a lot of work. Research partners can help figure out the easiest and fastest ways to gather the needed data while considering study participants’ views. When research partners understand how a study collects its data, they may be better able to help interpret the study’s results.


1. Advance stakeholder understanding of research design through applied learning

2. Support future data interpretation efforts by understanding how the study design will inform data analysis and interpretation

Activity OverviewFacilitate a discussion among the research staff and research partners about how data are, were, or will be collected:

• If the data sources for your study are finalized, talk about what each data source is being used to measure and why it is the best way to obtain information for those measurements. How are you dealing with the weaknesses of each data source?

• If the data sources for your study are not finalized, discuss what the best ways to gather the information needed for the analysis, taking into account the strengths and weaknesses of the different sources of data.


• PCORI Research Methods 101• Research Fundamentals Learning Package: Module 2 Designing the Research Study



OVERVIEW

The Data Analysis Plan, also referred to as the study protocol, is an essential guide for planning the data-related aspects of the research project. Research staff can use this chapter as a topic roadmap for conversations with research partners about the main sections of the Data Analysis Plan at all phases of the study. This plan includes explanations of:

• The purpose of the study

• The variables to be measured

• The approach for collecting data

• What comparisons will be made

• Which statistical models and tests will be used

Research partners can help make many of these choices during the study design phase, bringing valuable perspectives and experiences to discussion about which outcomes should be measured, the best ways to collect data, and other factors that should be part of the analysis. Researchers and research partners can benefit from a discussion of the Data Analysis Plan as early in the project as possible, ideally when the plan is being made. Throughout the study, the Data Analysis Plan can serve as a guide for explaining why the data are being analyzed in a certain way. Using the plan as a guide, researchers and research partners can begin to interpret preliminary results with an eye toward what questions each analysis is helping to answer.

As you will read later in this resource, it is also important to keep a list of study protocol modifications and why they were made so that the team can consider them when analyzing and interpreting the data.

Activities in this chapter include a discussion guide on PCORI Methodology Standards and a variable identification worksheet.

IMPORTANT CONCEPTS

Patient-centered outcomes research uses statistical methods to understand the collected data, test the study hypotheses, and draw conclusions about the data. The methods utilized in a study are informed by decisions made early in the process when the study is designed and the Data Analysis Plan is finalized. This phase presents an opportunity to provide a basic understanding of core terminology of selected variables and lay the groundwork for how the collected data will inform discussions about the analysis. Here we present important concepts to discuss with research partners when discussing variables selected as part of the research study designed. For each explanation, consider selecting examples from your research topic or study. We encourage study teams to tailor the terminology below to their own study to facilitate meaningful engagement.

Study VariablesStudy variables are characteristics or qualities that can change from one time to another, one person to another, or one place to another. Examples of study variables include age, height, neighborhood, or year. Variables are used in research studies to better understand relationships and patterns and effects on study outcomes. They are the information captured in a study to answer the research questions and an important part of developing a Data Analysis Plan.

There are many different types of variables that are used to describe different types of data. The type of variable used informs the types of statistical tests that would be appropriate to use for analysis.

Numeric variables are measured in several ways and have values that answer such questions as “how much?” and “how many?” Examples can include heart rate, rainfall measured in inches, weight, and number of cardiac episodes.

CHAPTER 2Designing Your Data Analysis Plan


There are two types of numerical variables, discrete and continuous, that help us determine how to interpret the numeric values:

Discrete variables are always whole numbers and are considered countable elements, meaning, they can map to natural numbers. They can be finitely countable. Some general examples include age in years or number of students in a class.

Categorical data represent discrete groups or levels. They can classify or describe study targets such as people or organizations. They answer questions like “what type?” or “which group?” Many people characteristics, such as race, gender, and home location, are categorical variables. Within categorical data, there are additional variable types which allow for more precise organization and analysis.

Continuous variables, on the other hand, have infinitely uncountable elements and are typically represented in intervals or measurements, such as weight and height. Although the possible values are infinite, it is usually possible for us to count reasonable amounts of these variables. There are two types of continuous variables:

Ratio data are characterized by an equal and definitive ratio between each data point with absolute zero being treated as a point of origin. Meaning, the zero is meaningful because it represents the absence of something. It is also important to remember a negative numerical value cannot exist in ratio data.

Interval data are characterized by precisely equal spaces in between each number which can be measured, sharing similar properties to ratio data. Each number is equally distant from the next based on what that number represents. Examples are degrees in Celsius or measures of distance.

Categorical variables are still assigned numerical values for statistical analyses, but the type of variable it is determines how we interpret the numerical values. Here is more information about each of the three types of categorical variables, which are nominal, ordinal, and dichotomous:

Nominal variables have no order, which means the order is meaningless. The names assigned to each category (such as those for race/ethnicity, religion, sex, and political viewpoint) would be assigned numbers for statistical analyses, but those numbers have no significance relative to the rank-ordering or priority between groups.

Ordinal variables are categorical variables that have a meaningful ranking or order. Rankings include age group (baby, teenager, young adult…), disease stage, or attitude (e.g.: “strongly agree,” “agree,” “disagree,” and “strongly disagree”).

Dichotomous (or binary) variables are a special type of categorical variable. They can take only one of two values, such as yes/no variables that tell whether a certain condition is present. When assigned numerical values, 0 and 1 are usually used to represent the two groups within this variable.


Independent and Dependent VariablesVariables are used in different ways to analyze research data. Dependent variables are also called outcome variables. The outcomes represented by these variables are the changes in study participants’ health or experiences that most interest researchers and partners. An independent variable is a characteristic such as age that could potentially affect the dependent variable. Often in PCOR, it represents some aspect of medical treatment, healthcare delivery, or other characteristic that influences health. Independent variables are also called predictor variables. Researchers conduct studies to find out how changes in the independent variables affect the value of a dependent variable.

For example, the healthcare team might want to find out if virtual one-on-one coaching has an effect on blood pressure control at one year. In this study, the independent variable is the presence or absence of the intervention (one-on-one coaching sessions). The dependent variable could be some indication of health status, either clinical or functional, such as a numeric measure of a person’s blood pressure or a scaled measure of their ability to perform cardiovascular activities (e.g., rate of perceived exertion tests.

The diagram below shows the relationship between independent and dependent variables.

Health Status (clinical or functional measure)

Blood pressure classification (ordinal)1. Optimal2. Normal3. High normal4. Grade 1 hypertension5. Grade 2 hypertension6. Grade 3 hypertension

Rate of perceived exertion test (ordinal)1. Very light 2. Light3. Moderate 4. Vigorous5. Veryhard6. Maxeffort

Relationship between Independent and Dependent Variables

Independent Variable (Predictor)

Dependent Variable (Outcome)

One-on-One Coaching Program (dichotomous)

0. No coaching1. One-on- one coaching

Time (interval)1. Time 12. Time 23. Time 34. Time 4

REVISITING HOMEWORK TO HELP US UNDERSTAND INDEPENDENT AND DEPENDENT VARIABLESWe all have had homework assignments at one point in our life. We may have dreaded them during our elementary school years, but when thinking about the relationship between independent and dependent variables, homework assignments and final course grades can help make these concepts more relatable. Your independent work on each homework assignment influences your final course grade. Your final course grade is dependent on your independent behaviors or efforts on homework assignments throughout the semester. Therefore, your final course grade is dependent on the completion of your homework assignments.


CovariatesCovariates are other variables used in statistical analysis that help make the relationship between a study’s independent (predictor) and dependent (outcome) variables clearer. A highly useful covariate in the study above would be changes in diet associated with the coaching sessions. If the researchers were collaborating with research partners, they could work with them to develop ways to assess less obvious covariates, such as access to green space for exercise in study participants’ neighborhoods or the presence of informal caregivers. Covariates are important during data analysis to understand the interactions among variables, or effect modification.

Examining effect modification is one way researchers use covariates. Effect modification is also known as heterogeneity of treatment

effect, or interactions among variables. It means that the effect of the independent variables on the dependent variable is different for different people. For example, if the study is comparing different types of health class modes (independent variable) for people with high blood pressure, individuals may be more likely to show greater control in blood pressure or increased daily activities (dependent variables) if they have the support of informal caregivers. Caregiver support is an effect modifier since it creates the conditions for more successful rehab. If it is included in the analysis as a covariate, its importance can be measured as part of the study.

The diagram below shows the relationship between the different variable types.

Health status (clinical or functional measure)

1. Blood Pressure2. Rate of perceived exertion tests

Heterogeneity of Effect

Effect Modifier

Family and Caregiver Support

1. Low2. Medium3. High



One-on-One Coaching Program

0. No coaching1. One-on-one coaching


Another use of covariates is to reveal the effects of confounding. Confounding distorts the relationships between the predictor variables and study outcomes when unmeasured factors directly affect the predictors and outcomes. This type of distortion is one form of bias (see PCORI’s Research Fundamentals for further discussion of bias). For example, if we are studying whether level of smoking (predictor variable) increases the risk of high blood pressure (outcome variable), the result could be thrown off if we do not measure alcohol consumption. This is because smoking often accompanies alcohol consumption and both are causes of high blood pressure and cardiovascular disease. Failure to measure alcohol consumption and account for it in the analysis could increase the apparent impact of smoking on heart disease.

It is very important to think about potential confounding variables early in the project so that they can be included in the Data Analysis Plan and their values collected with the study data. In the example above, study teams can account for confounding variables by stating in the research protocol that research subjects must limit or eliminate smoking during the study to reduce the effect of this confounding variable. Here, too, research partners can draw on their knowledge of and experience with health and health care to suggest hidden variables such as marital or partnership status that might have a confounding effect in the study.

The diagram below shows the relationship between these types of variables.


Confounding Variable


One-on-one Coaching Program

0. No coaching1. One-on- one coaching

Alcohol Consumption

1. Number of drinks per week

Confounder

Health Status (clinical or functional measure)

1. Blood Pressure2. Rate of perceived exertion tests


SUGGESTED TIPS FOR CODESIGNING THE DATA ANALYSIS PLAN WITH RESEARCH PARTNERS

Research partners are vital contributors to the identification and selection of important variables for research. Discussing the Data Analysis Plan throughout the life of the project helps foster teamwork between research staff and research partners and supports co-learning, partnership, transparency, honesty, and trust. The table below suggests ways to support research partner involvement in designing the Data Analysis Plan.

The statistical methods that will be used to analyze data in the study will be shared in the Data Analysis Plan. The two basic types of statistical methods are descriptive and inferential. While detailed discussions about statistical methods occur during the data analysis and

interpretation phase, a basic understanding of how variables are described and interpreted should be presented and discussed early on and revisited as needed throughout the study. More detailed discussion and application of statistical concepts are provided in chapter 3.

Identify early with stakeholder partners any experience(s) with and training needs for co-developing Data Analysis Plan.

Prepare for a wide breadth of experience and comfort with data analysis topics. Some stakeholder partners may benefit from added discussions (or 1:1 time) to facilitate learning.

Identify key personnel on the research team who have expertise explaining/teaching analytic concepts used as part of the data analysis plan and invite them to join/lead stakeholder meetings as needed.

Provide resources (such as glossaries, reference materials) at least one week before meetings to provide research partners an opportunity to review materials.

Create a schedule of topics for discussion with stakeholder groups relevant to the goals for data analysis and interpretation as part of the research study. Revisit the schedule of topics as needed with stakeholders to ensure topics selected support learning needs.

Plan for topics that may be more challenging to cover and determine how best to discuss or break down content. For example, contentious topics regarding trade-offs for study design may be better suited to smaller group discussions among subsets stakeholder partners before being brought to a larger group discussion.

Document all decisions made and how stakeholder input shaped decisions.

Share with stakeholders how input and discussions were processed and incorporated. If dissenting opinions/views exist, ensure transparency and document the reasons/rationale.

Create the final study protocol and make it available to stakeholder partners. The final protocol will serve as the reference point for future discussion regarding study analysis and interpretation.



Centre for Evidence-Based Medicine. nd. “Study designs.” Accessed June 3, 2021. https://www.cebm.ox.ac.uk/resources/ebm-tools/study-designs.

Chan, An-Wen, Jennifer M. Tetzlaff, Peter C. Gøtzsche, Douglas G. Altman, Howard Mann, Jesse A. Berlin, Kay Dickersin, et. al. 2013. “SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials.” BMJ. 346:e7586. https://doi.org/10.1136/bmj.e758.

Gamble, Carrol, Ashma Krishan, Deborah Stocken, Steff Lewis, Edmund Juszczak, Caroline Doré, Paula R. Williamson, et al. 2017. “Guidelines for the Content of Statistical Analysis Plans in Clinical Trials.” JAMA. Dec 19;318(23):2337-2343. https://doi.org/10.1001/jama.2017.18556.

Grimes, David A., and Kenneth F. Schulz. 2002. “An overview of clinical research: the lay of the land.” Lancet. Jan 5;359(9300):57-61. https://doi.org/10.1016/S0140-6736(02)07283-5.

Röhrig, Bernd, Jean-Baptiste du Prel, Daniel Wachtlin, and Maria Blettner. 2009. “Types of study in medical research: part 3 of a series on evaluation of scientific publications.” DtschArztebl Int. Apr;106(15):262-8. https://doi.org/10.3238/arztebl.2009.0262.

Simpson, Scot H. 2015. “Creating a Data Analysis Plan: What to Consider When Choosing Statistics for a Study.” Can J Hosp Pharm. Jul-Aug;68(4):311-7. https://doi.org/10.4212/cjhp.v68i4.1471.

National Institute for Children’s Health Quality. nd. “QI Tips: A Formula for Developing a Great Aim Statement.” Accessed June 3, 2021. https://www.nichq.org/insight/qi-tips-formula-developing-great-aim-statement.

Nebeker, Camille, Gayle Simon, Michael Kalichman, Ana Talavera, Elizabeth Booen, and Araceli López-Arenas. 2015. “Module 2: Research Design.” In Building Research Integrity and Capacity (BRIC): An Interactive Guide for Promotores/Community Health Workers. San Diego: BRIC Academy. https://ori.hhs.gov/module-2-research-design.

FURTHER READING

https://www.cebm.ox.ac.uk/resources/ebm-tools/study-designs



https://doi.org/10.1136/bmj.e758

https://doi.org/10.1001/jama.2017.18556

https://doi.org/10.1016/S0140-6736(02)07283-5

https://doi.org/10.1016/S0140-6736(02)07283-5

https://doi.org/10.3238/arztebl.2009.0262

https://doi.org/10.3238/arztebl.2009.0262

https://doi.org/10.4212/cjhp.v68i4.1471

https://doi.org/10.4212/cjhp.v68i4.1471

https://www.nichq.org/insight/qi-tips-formula-developing-great-aim-statement

https://www.nichq.org/insight/qi-tips-formula-developing-great-aim-statement

https://ori.hhs.gov/module-2-research-design

https://ori.hhs.gov/module-2-research-design


CASE STUDY 2: DESIGNING YOUR DATA ANALYSIS PLAN

Background: Involving stakeholders when finalizing the study protocol and Data Analysis Plan helps ensure the team captures important variables for contextualizing results. In addition, it helps inform study procedures to support successful study conduct, such data collection timing or the burden on patients and staff for data collection.

Challenge statement: Comprehensive data collection supports researcher’s priorities for conducting robust data analyses, but it may place an unintended burden on research participants or partner sites. This may necessitate negotiation a balance between the quantity of data capture and the experience of research participants and partner sites.

Example scenario: A research team is finalizing the Data Analysis Plan for a study to compare two treatments for patients with a progressive condition resulting in significant impact on quality of life and health. Research participants often provide a substantial amount of data through completion of patient-reported outcome measures and study questionnaires. Several patient-reported outcome measures are identified as ideal for data capture and in some cases may serve as the best way to measure treatment-related changes. The battery of questions proposed for patients includes 180 items. A review of the full questionnaire intended to be administered at baseline and then quarterly through two-year follow-up reveals that it takes, on average, 35 minutes for participants to read through and complete.

Research team members feel it is important to measure all items to have robust data capture and fully understand the treatment outcomes and consider patient characteristics that could explain how different patient subgroups experience treatments. Patient partners express concern that study participants in poorer health will be challenged to finish questionnaires. Further, redundancy in some items creates unnecessary burden for study participants. Such a burden to participants may result in higher dropout rates over the course of time.

Resolution: The team works together to review the conceptual framework for the study, mapping out how all information collected from different data sources can inform the data analysis plan. They also review the patient-reported outcome measures, which patient partners prioritize. The team decides to remove questionnaires where significant overlap exists. In addition, while a comprehensive questionnaire is included at baseline, 12-month, and 24-month follow-up, only the prioritized measures are collected at 3, 6, 9 and 18 months to reduce the burden on study participants.

Discussion questions: With your team, use the following questions to discuss your research study and partnerships to support the Data Analysis Plan’s design.

• How will your study team involve research partners to finalize study variables?

• Where can your team envision similar tension points when finalizing your study protocol? What design trade-offs can you offer given patient and stakeholder views?

• How will you negotiate opposing ideas on the study protocol to ensure patient-centeredness and authentic stakeholder engagement (e.g., changes to data collection, enhanced patient materials, etc.)?

• How will stakeholder input inform decisions regarding the protocol?

• How will you document results from discussions about the final protocol?



Activity 2a: Understanding Research Design and Implications for Your StudyThere are many ways to design a study. Research partners can better engage in discussions about designing a research study when they understand the strengths and limitations of different designs. Study type is important to understand because it affects how the data are analyzed, the way in which the results are interpreted, and what conclusions can be drawn from the data.

In an effort to support high-quality research design, PCORI produced the PCORI Methodology Standards. These Standards include minimum requirements for PCOR development, conduct, and reporting. The Methodology Standards align with many existing reporting guidelines that support quality and transparency in research. The following activity leverages the PCORI Methodology Standards to discuss study design with research partners. In the discussion guide provided, crosscutting standards focused on content from chapter 2 (Standards for Data Integrity and Rigorous Analyses, Standards for Heterogeneity of Treatment Effect) guide the group’s conversation. The goal is to have discussions around data analysis and interpretation early. Research teams should select the Methodology Standards most relevant to and important for discussion with research partners and may adapt the guide accordingly. Multiple discussions or extended time may be warranted depending on the project’s needs.


1. Support future data interpretation efforts by understanding how the study design will inform data analysis and interpretation

2. Ensure research partners understand the role of PCORI Methodology Standards in supporting high quality research




10 mins

Review the PCORI Methodology Standards

• Review the goal of PCORI Methodology Standards• Review the Standards relevant to the research study in

development (or in process)• Review how Standards inform research results reporting

20 mins

Group Discussion: Heterogeneity of Treatment Effects

• Review Heterogeneity of Treatment Effects • Discuss, as a group, the following questions:

1. What experiences might modify patient outcomes or result in different outcomes for patients within the study? How will we measure this?

2. What subgroups of patients may experience a different outcome? How will we measure this?

• Review the responses to the PCORI Methodology Standards Checklist submitted as part of the proposal. Discuss any needed revisions.

20 mins

Group Discussion: Data Integrity and Rigorous Analysis

• Review the goals for Data Integrity and Rigorous Analysis • Discuss, as a group, the following questions:

1. What data sources are used in our study? How do data sources compare to alternatives?

1. What measures will we use in our study? What evidence exists that the measures are valid in the population studied?

1. What processes are in place to ensure rigor in data management?

• Review the responses to the PCORI Methodology Standards Checklist submitted as part of the proposal. Discuss any needed revisions.

5 minsWrap-Up

• Present to the group the next steps and how the information discussed will inform the study design and future reporting.

Discussion Guide

Session Preparation Materials (send at least two weeks in advance for review)

• Your research proposal or drafted specific aims• Your completed PCORI Methodology Standards Checklist• Research 101 Training: Intro and Research Design• PCORI Methodology Standards Checklist• PCORI Methodology Standards Academic Curriculum

https://www.pcori.org/sites/default/files/PPR-R101-Session-1-SMART.pdf

https://www.pcori.org/document/methodology-standards-checklist

https://www.pcori.org/research-results/about-our-research/research-methodology/methodology-standards-academic-curriculum


Activity 2b: Identifying the Variables in Your StudyAn important part of designing a study is deciding which variables are needed to answer the research questions. This important step informs how the research team will ultimately interpret study results. It is important to understand the data included in your study and how they are defined. There are several ways to measure and use variables in the analysis. Variables may be numeric or categorical and may be used as dependent (outcome) variables, independent (predictor) variables, or covariates (control variable).

ObjectivesThe activity below supports the following objectives:

• Understand the different data collected as part of the study and how it is measured• Foster discussion on relationships between variables and implications for study design• Support future data interpretation efforts by understanding how variables selected will inform

data analysis and interpretation

Activity Overview1. Share the table below with research partners.

2. Review the study protocol and list as a group the variables captured in the study.

a. Discuss the following questions concerning the variables and their uses:

i. Which outcome variables are primary, which are secondary, and why?

3. Review the study hypotheses and discuss the following questions:

a. Why do some variables function as predictors and others as covariates?

b. How does the study employ covariates? Some examples of covariate roles are to prevent confounding, alter the influence of other variables (effect modifiers), and separate the study population for subgroup analyses.

c. Would other variables be useful in the analysis if data on them is available?


Variable Listing Template

Variable Name Data Source Variable Type (numeric or categorical)

Notes on Variable Use

Outcome or Dependent Variables

Predictor or Independent Variables

Covariates


• PCORI Methodology 101 Training• Research 101 Training: Intro and Research Design• Research proposal or protocol, if applicable • Your draft study Data Analysis Plan• If research proposal is in development, consider selecting a published manuscript related to your

area of research, using a similar data source or study design.




CHAPTER 3Analyzing Your Study Data

OVERVIEW

This chapter will dive deeper into statistical methods and analytic tests that will ultimately lead to the study results. Carrying out the activities in this chapter using your study team’s project can help participants learn how to apply these statistical tests. These activities also serve as an effective way to familiarize the team with the study results and prepare for more in-depth conversations. Because data analysis can be an iterative process, the guide organizes activities first by descriptive analysis to familiarize research partners with basic data descriptions, followed by inferential statistics as data analysis becomes more complex.

We encourage patient and stakeholder partners who would like more in-depth statistical theoretical knowledge to seek out textbooks on biostatistics. The level of detail we share about statistical analysis does not match formal coursework.

Activities in this chapter include exercises and discussion questions on descriptive and inferential statistical tests.

IMPORTANT CONCEPTS

Patient-centered outcomes research often uses statistical methods to understand collected data and reach conclusions about the research questions. Discussing statistical concepts with stakeholder partners can be challenging if the concepts are new. A basic understanding of core terminology and statistical concepts is important during discussions about data analysis.

What are statistics? Statistical tests offer a way to organize and interpret data. Statistical methods sort through data points that represent the recorded observations, measurements, or counts that involve individual human beings. These tests also associate data to each other to describe or make inferences about the data set and relationships. Ultimately, researchers use statistical analyses to understand the intervention or treatment effects and generalize results to the study population.

Hypothesis TestingHypotheses, formed during study development, are usually stated as an expected relationship between two or more variables. A hypothesis could be that people who receive a certain treatment are more likely to have better outcomes than people who don’t get the treatment. Testing a hypothesis to determine if a relationship between variables is real is central to research. Statistical tests help scientists determine if the data support the hypothesis and the likelihood of a measurable effect occurring by chance.

Researchers often discuss the null hypothesis and alternative hypothesis at the beginning of a research study. Behind every relationship between variables, as identified through research questions, is a null hypothesis, which is the absence of a significant effect from intervention(s) or treatment(s) when exploring group differences. The null hypothesis states that there are no significant differences or effect. On the other hand, an alternative hypothesis predicts the intervention or treatment under question will have a significant effect when exploring group difference. Importantly, the alternative hypothesis may also state directionality of the predicted difference.

Data analysts use these hypotheses as the basis for their statistical tests. When a statistical test is run, the results tell us whether our hypothesis is true or false. Null Hypothesis Significance Testing (NHST) is the process of testing the assumption that there is no difference or effect in the relationship between variables. If this is true, that means that the null hypothesis is correct and there are no differences between groups. If this is false, there is a significant difference detected and the output from the testing tells us more about where and how much those differences occur, in what direction, and for which groups. With this information, we can then make inferences about what that means and how that information can be used to inform clinical decisions.

If a study is not able to reject the null hypothesis, that means the study team were not able to find a significant effect or relationship between the study’s independent and outcome variables. An


example of the most common type of null hypothesis would be, “Virtual one-on-one coaching does not have a significant effect on high blood pressure.” The statistical analysis will show whether the study team should accept or reject the null hypothesis and whether we can draw conclusions about the effects of virtual one-on-one coaching on blood pressure that are not due to chance.

Statistical MethodsThere are two basic types of statistical methods: descriptive and inferential.

Descriptive statistics summarize collected data, such as a range of values, their average, and the most common level or category. They give an overall sense of what is in the data. Researchers use inferential statistics to make comparisons, find patterns, test hypotheses, and draw conclusions from the data.

Descriptive Statistics

Descriptive statistics are helpful for getting a basic understanding of the data. An analyst’s first action with new data is usually to run a descriptive analysis to summarize the information in the data set. Different types of descriptive statistics can be used, depending on whether the data are numeric or categorical.

Numeric Data

As shared earlier in chapter 2, numeric data is any data that has a numeric value, such as heart rate, blood pressure, and weight. We can describe data that is measured in numbers using many methods, including measures of central tendency.

Categorical Data

For categorical data, the different categories can represent any type of subgroups, such as a geographic area, patient demographics, or aspects of patient health. Perhaps the most useful

STEPS IN NULL HYPOTHESIS SIGNIFICANCE TESTING The most appropriate statistical test for a study is dependent on the number of variables included, variable types, and the research questions that are being answered with those data. All are used in comparative effectiveness research in the same basic way and follow a common series of steps.

• Identify a null hypothesis, which finds no relationship between the study’s independent and outcome variables.

• Construct several alternative hypotheses about relationships between notable predictor and outcome variables.

• Theorize the independent and dependent variables and covariates that may affect these relationships.

• Decide what p-value is needed to reach a conclusion about the hypothesis. The p-value rates how often the relationship in the data would happen if the null hypothesis were true (and there was no real relationship between the predictor and outcome variables). Determining the a priori alpha level should usually happen right after the hypotheses are stated but before statistical tests are run.

• The standard p-values of .05 and .01 have been set by theory and are commonly used in practice. A p-value of .05 indicates there is a 5 percent chance that the observed connection between changes in predictor and outcome variable will occur by accident when the two variables are not linked at all. A p-value of .01 means that there is only a 1 percent chance of this happening. A lower p-value makes it more convincing that the study results show a real, nonrandom relationship between the variables.

• Choose the statistical test that is set up for the type of data that the study has produced. The tests can suggest that the observed relationships between variables are not due to chance and the null hypothesis is false. Statistical tests commonly assign a p-value to your results.

• Analyze the data. It is important to establish analytic procedures even before collection so that study teams are not tempted to tailor the analyses to what they think will yield their desired results.


type of descriptive information for categorical variables is the frequency distribution. This is a count or percentage indicating how many times each category or group appears in the data. A frequency distribution can be displayed as either a table or a chart. Take a look at Activity 3a to demonstrate a frequency distribution table in your own study.

Measures of Central Tendency

Measures of central tendency are values that point to the center of a collection of a numeric category’s range. The three most common measures are arithmetic mean (average), median (halfway point, with half of the study population having a value below the median and half above

it), and mode (the most common value for a study variable, most often appearing as the highest point in a chart). Mode is the largest category in the study population. The mean is the most used central measure, but the median is better when extreme values at one end of the range of results force the mean closer to that side. In this case the median is more representative of the study data.

The chart below shows the relation between mean, median, and mode in bell curve with varied skewness. If the mean, median, and mode are all the same then the distribution of your results will take the shape of a normal bell curve (see chart b below). Charts a and c below show when data is not distributed around the midpoint, which illustrates the concept of skewing. Skewing simply means the mean, median, and mode are all different.

a) Left Skewness b) Normal Bell Curve c) Right Skewness

Source: Hazra and Gogtay 2016, doi: 10.4103/0019-5154.173988 (Creative Commons License By-NC-SA 3.0)

Measures of Spread

Measures of spread show how spread out the data values are. For example, they can measure the spread between all the variable values and

the mean. The most common measures of spread are range, interquartile range (IQR), and standard deviation.

25% 25% 25% 25%

Interquartile range(from Q1 through Q3)

Q1 Q3Q2Median

One standard deviation One standard deviation


The simplest way to express spread is to state the difference between the highest (maximum) and lowest (minimum) values for a particular variable. This spread is called the range. The next simplest method is to report the IQR, which focuses attention on the middle 50 percent of the data range. If you were to split the curve into four quartiles, it is the distance between the 25 percent and 75 percent quartiles.

Standard deviation (SD) is another useful measure of spread for statistical analysis. The SD is an indicator of the average spread between each variable value in the data from the mean value for that variable. When the SD is

low, it indicates that the data points are tightly concentrated around the mean. If the SD is high, the data points are more spread out. The SD determines the shape of the data distribution bell curve, with a lower peak and wider curve if the SD is high (see the figure below).

The two main ways that curves can differ from the normal bell shaped curve are skewness and kurtosis. Skewness illustrates the side-to-side shift causing the data to shift to be heavier on one side and light at one of the ends. Alternatively, kurtosis describes the height of the peak in comparison to how wide the curve spans. There are data implications for why both happen.

- 3 - 2 - 1

1 2 3

- 3

- 2

- 1

φ μ,σ

2(

0.8

0.6

0.4

0.2

0.0

−5 −3 1 3 5x

1.0

−1 0 2 4−2−4

x)

0,μ=0,μ=0,μ=−2,μ=

2 0.2,σ =2 1.0,σ =2 5.0,σ =2 0.5,σ =

Bell Curves with Different Means and Standard Deviations

Normal/mesokurtic distribution

Normal/mesokurtic distribution

Platykurtic distribution

Leptokurtic distribution

Inferential Statistics

Inferential statistics include a range of statistical analyses that help the analyst understand differences across study individuals or groups on a study outcome. These tests can estimate the strength of each factor’s contribution to the outcome based on the patterns in the data. The goal of inferential statistical analysis is to make inferences about the study population and whether observed differences are due to intervention(s) or pure chance.

In a well-designed study, knowledge gained from inferential statistics allows investigators to generalize beyond the study sample to larger populations with similar characteristics. In the study example involving high blood pressure, you could make a table or line graph that shows how blood pressure increases as the rate certain behaviors such as smoking or alcohol consumption go up. You might make separate tables or graphs for men and women, people of different age groups, etc. These graphs would be descriptive statistics. Since inferential statistics allows you to combine all these variables to


predict high blood pressure in other populations with different makeups, it relies on complex mathematical formulas.

Some of the most important types of inferential statistics are measures of association between two or more variables. Often researchers are interested in the association between an intervention (such as a treatment) and an outcome (such as improved function). In this case, a measure of association can also be thought of as an estimation of the size of the effect. These are ways to measure the type, strength, and direction of the association between a predictor variable (such as a treatment) and an outcome variable. We present a few of the most common ways here.

Measuring the Association between Two Variables

The simplest measures of association look at two variables at a time. These are sometimes call bivariate statistics. Keep in mind that these measures in themselves do not say anything about causality. To reach conclusions about causality you must consider the study design, including randomization and control of other factors.

Coefficients of Correlation (r) and Determination (r2)

One commonly used measure of association between two continuous variables is the Pearson’s correlation coefficient (r). Imagine a plot of the data for two study variables where the horizontal axis represents the dose of a drug treatment and the vertical axis represents a patient outcome. Each dot shows one person’s dose and their outcome (see graphs on next page).

When the two variables have a linear association, all the dots can have a best fit line fitted to the model to calculate the correlation coefficient. The slope of that line represents the strength of the correlation between the two variables represented by a coefficient, r, which is bounded between -1 and +1. The correlation coefficient can fall anywhere between these two integers. A r = 1 would be a perfect positive correlation and r = -1 is a perfect negative correlation, both of which are extremely rare (see graphs below).

Correlation coefficients that fall between -1 and +1 indicate that the linear association is not perfect because some of the dots are not on the line. Numbers close to zero (either positive or negative) indicate weaker correlations. Coefficients closer to -1 or +1 indicate stronger correlations as they get closer to having a perfect relationship. When the correlation coefficient equals zero, no discernable relationship between the two variables exists (i.e., an increase in one variable is not necessarily related to an increase in the other).

Multiplying the correlation coefficient times itself produces r2, or the coefficient of determination (usually called r-squared). The coefficient of determination is always between 0 and 1 because it is a squared value and is a common and easy-to-grasp measure of association. Statisticians say that r2 represents the percentage of how much the outcome variable can be explained by the predictor. An r-squared of .8 means that when controlling for all other factors, 80 percent of the variability in the outcome variable can be explained by the predictor, which would be very high.

Essentially, r2 indicates the percent change in one variable that is associated with change in another. For example, imagine you are investigating the extent to which smoking history affects blood pressure. You plot estimated lifetime number of cigarettes versus blood pressure in your study population and find an r2 of 36 percent, or 0.36. You could then say that 36 percent of variability in blood pressure are explained by differences in smoking history, whereas person-to-person differences in biology and environment explain the other 64 percent. However, the term “explain” indicates an observed association. It does not mean that a change in one variable causes the other when used this way.

Usually for absolute values of r:

Size of correlation coefficient

Strength of correlation

0 No correlation

0 to 0.19 very weak

0.2 to 0.39 weak

0.4 to 0.59 moderate

0.6 to 0.79 strong

0.8 to 1.0 very strong


Correlation Coefficients (r) between Variables in Study Data

r = -1 -1 < r < 0

0 < r < +1 r = +1 r = 0

Source: https://en.wikipedia.org/wiki/Pearson_correlation_coefficient (Creative Commons Attribution-Share Alike 3.0 Unported License)

Multiplying the correlation coefficient times itself produces r2, or the coefficient of determination (usually called r-squared). The coefficient of determination is always between 0 and 1 because it is a squared value and is a common and easy-to-grasp measure of association. Statisticians say that r2 represents the percentage of how much the outcome variable can be explained by the predictor. An r-squared of .8 means that when controlling for all other factors, 80 percent of the variability in the outcome variable can be explained by the predictor, which would be very high.

Essentially, r2 indicates the percent change in one variable that is associated with change in another. For example, imagine you are investigating the extent to which smoking history affects blood pressure. You plot estimated lifetime number of cigarettes versus blood pressure in your study population and find an r2 of 36 percent, or 0.36. You could then say that 36 percent of variability in blood pressure are explained by differences in smoking history, whereas person-to-person differences in biology and environment explain the other 64 percent. However, the term “explain” indicates an observed association. It does not mean that

a change in one variable causes the other when used this way.

Measures of Differences in Risk

Another common type of bivariate statistic measures the difference in risk between two groups of people. Researchers often calculate the risk of the outcome for each group then compare these risks by calculating a risk difference or a relative risk. Relative risk aims to identify disparities in the incident of disease across groups based on exposure status. Imagine a scenario where researchers are studying hospital readmissions for chronic high blood pressure, which is manageable on an out-patient basis. They find that the low social support group has a 40 percent chance of readmission within 30 days after discharge and the high social support group has a 10 percent chance of readmission. The risk difference would be 40 percent minus 10 percent to equal 30 percent, while the relative risk would be 40 percent divided by 10 percent to equal 4 times the risk for the low-support group. (A relative risk of 1 would indicate no relationship between the variables.)


VARIATIONS OF RELATIVE RISKTwo variations of relative risk are odds ratios and hazard ratios. Both are special ways of showing disease risk in in one population compared to another. The calculation of odds ratio is slightly different than for relative risk, but the results are approximately the same except in groups with high disease risk. Odds ratios are used in certain analyses in which relative risk proves impractical. Analyses involving case-control studies are one example. Hazard ratios add up changes in relative risk over time. Chemotherapy, for example, is less effective in suppressing advanced, metastatic lung cancer as time goes on, and death rates of treated patients tend to catch up with those of untreated patients. Odds ratios and hazard ratios also are the products of some of the advanced statistical calculations outlined in the next section.

Lung Cancer

Present Absent

Smok

ing

Pres

ent

Abse

nt

a+b = totalsmoking

c+d = totalnon-smoking

a+c = totalcancer

b+d = totalno cancer

RR =a/(a + b)

c/(c + d)a b

c d

Lung Cancer

Present Absent

Smok

ing

Pres

ent

Abse

nt

a+b = totalsmoking

c+d = totalnon-smoking

a+c = totalcancer

b+d = totalno cancer

a b

c d

OR =a/b

c/d)=

ad

bc


Problems with Missing DataMissing data on study participants is a problem for many healthcare studies. Gaps in the data can happen for many reasons. Study participants may drop out of the study, miss appointments, not answer surveys, or skip questions. If the study is analyzing medical records or claims data, everyone may not receive all the services or measurements.

Gaps in the data cause problems in several ways. Study participants with missing data may be different from the study participants who don’t have missing data, but it would be hard to tell how they are different. In a drug study, for example, people who experience side effects may be more likely to drop out. When this happens, the data that can be used for analysis doesn’t accurately represent all the people in the study. This is a form of bias that can negatively affect the results of the analysis (see module 5 for more information on bias). Also, missing

data effectively shrinks the size of the study’s population sample. A smaller study population in turn reduces the study’s ability to reach conclusions based on the data.

Researchers work hard to keep people in the study and gather complete data on all participants. They often meet with research partners to figure out ways to keep people in the study. Research staff also try to anticipate how much data are likely to be lost and make up for it in several ways. One is to start with a large enough sample so that even after loss of data, enough complete data remain for the types of analysis planned. This can solve the problem of not having enough data, but it does not solve the problem of potential bias. A range of statistical methods can measure how much the study has been weakened by data gaps and adjust reduce this bias whenever possible. However, it is important to consult statisticians to make these assessments and judgements about missing data and its implications.


SUGGESTED TIPS FOR FACILITATING DATA ANALYSIS WITH RESEARCH PARTNERS

Successful involvement of research partners during data analysis occurs when partners understand its goals and have a basic understanding of key statistical concepts relevant to the study. Allot time to build stakeholder capacity for data analysis throughout study conduct. For prospective studies, this may happen as data are collected and decisions about study conduct are complete. Below are suggestions to boost efforts to involve research partners during data analysis.

Identify early with stakeholder partners any experience with and training needs for understanding statistics to focus capacity-building efforts.

Identify key personnel on the research team who have expertise explaining/teaching analytic concepts used as part of the data analysis plan and invite them to join/lead stakeholder meetings as needed.

Create a schedule of topics for discussion with stakeholder groups relevant to the goals for data analysis and interpretation as part of the research study. Revisit the schedule of topics as needed with stakeholders to ensure topics selected support learning needs.

When possible, leverage multiple modes of content delivery (such as written content, video/webcast materials, meeting discussions).

Tailor activities about data analysis and interpretation to your study. Examples include the following:

● Patient reported outcome measures: Review as a group a patient-reported outcomes measure used as part of the study. Discuss different aspects of measure validity, scoring, and interpretation.

● Discuss how to review a research study using a journal club/book club approach, discussing a published article that informed the development of the study under way.

Plan for topics that may be more challenging to cover and determine how best to discuss or break down content. For example, you may need to cover complex topics over multiple meetings or in person.



Australian Bureau of Statistics. nd. “Statistical Language.” Accessed June 3, 2021. http://www.abs.gov.au/websitedbs/a3121120.nsf/home/statistical+language.

Rice Virtual Lab in Statistics. nd. Last updated July 3, 2018. Accessed June 3, 2021. http://onlinestatbook.com/rvls.html.

Stat Trek. nd. Accessed June 3, 2021. https://stattrek.com/.

Gilmore, Stephen J. 2008. “Evaluating statistics in clinical trials: making the unintelligible intelligible.” Australas J Dermatol. Nov;49(4):177-86. https://doi.org/10.1111/j.1440-0960.2008.00465_1.x.

Hazra, Avijit, and Nithya Gogtay. 2016. “Biostatistics Series Module 1: Basics of Biostatistics.” Indian J Dermatol. Jan-Feb;61(1):10-20. https://doi.org/10.4103/0019-5154.173988. (Also see modules 2-10.)

Ioannidis, John, P. A. 2005. “Why Most Published Research Findings Are False.” PLoS Med. Aug;2(8):e124. https://dx.doi.org/10.1371%2Fjournal.pmed.0020124.

Moltusky, Harvey. 2015. Essential Biostatistics: A Nonmathematical Approach (1st Edition). New York: Oxford University Press.

Moltusky, Harvey. 2018. Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking (4th Edition). New York: Oxford University Press.

Szucs, Denes, and John P. A. Ioannidis. 2017. “When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment.” Front Hum Neurosci. Aug 3;11:390. https://doi.org/10.3389/fnhum.2017.00390.

Nuzzo, Regina. 2014. “Scientific methods: Statistical errors.” Nature. Feb 13;506(7487):150-2. https://doi.org/10.1038/506150a.

Schulz, Kenneth F., and David A. Grimes. 2005. “Multiplicity in randomised trials II: subgroup and interim analyses.” Lancet. May 7-13;365(9471):1657-61. https://doi.org/10.1016/s0140-6736(05)66516-6.

Schulz, Kenneth F., and David A. Grimes. 2005. “Multiplicity in randomised trials I: endpoints and treatments.” Lancet. Apr 30-May 6;365(9470):1591-5. https://doi.org/10.1016/s0140-6736(05)66461-6.

Grimes, David A., and Kenneth F. Schulz. 2005. “Compared to what? Finding controls for case-control studies.” Lancet. Apr 16-22;365(9468):1429-33. https://doi.org/10.1016/s0140-6736(05)66379-9.

Schulz, Kenneth F., and David A. Grimes. 2005. “Sample size calculations in randomised trials: mandatory and mystical.” Lancet. Apr 9-15;365(9467):1348-53. https://doi.org/10.1016/s0140-6736(05)61034-3.

Grimes, David A., and Kenneth F. Schulz. 2002. “Uses and abuses of screening tests.” Lancet. Mar 9;359(9309):881-4. https://doi.org/10.1016/s0140-6736(02)07948-5.

Schulz, Kenneth F., and David A. Grimes. 2002. “Sample size slippages in randomised trials: exclusions and the lost and wayward.” Lancet. Mar 2;359(9308):781-5. https://doi.org/10.1016/s0140-6736(02)07882-0.

Schulz, Kenneth F., and David A. Grimes. 2002. “Case-control studies: research in reverse.” Lancet. Feb 2;359(9304):431-4. https://doi.org/10.1016/s0140-6736(02)07605-5.

Grimes, David A., and Kenneth F. Schulz. 2002. “Bias and causal associations in observational research.” Lancet. Jan 19;359(9302):248-52. https://doi.org/10.1016/s0140-6736(02)07451-2.

Grimes, David A., and Kenneth F. Schulz KF. 2002. “Descriptive studies: what they can and cannot do.” Lancet. Jan 12;359(9301):145-9. https://doi.org/10.1016/S0140-6736(02)07373-7.

Grimes, David A., and Kenneth F. Schulz. 2002. “An overview of clinical research: the lay of the land.” Lancet. Jan 5;359(9300):57-61. https://doi.org/10.1016/s0140-6736(02)07283-5.

FURTHER READING

http://www.abs.gov.au/websitedbs/a3121120.nsf/home/statistical+language



http://onlinestatbook.com/rvls.html

http://onlinestatbook.com/rvls.html

https://stattrek.com/

https://stattrek.com/

https://doi.org/10.1111/j.1440-0960.2008.00465_1.x

https://doi.org/10.1111/j.1440-0960.2008.00465_1.x

https://doi.org/10.4103/0019-5154.173988

https://doi.org/10.4103/0019-5154.173988

https://dx.doi.org/10.1371%2Fjournal.pmed.0020124

https://dx.doi.org/10.1371%2Fjournal.pmed.0020124

https://economie.gouv.cg/sites/default/files/webform/pdf-essential-biostatistics-a-nonmathematical-approach-harvey-motulsky-pdf-download-free-book-c710834.pdf

https://economie.gouv.cg/sites/default/files/webform/pdf-essential-biostatistics-a-nonmathematical-approach-harvey-motulsky-pdf-download-free-book-c710834.pdf

https://books.google.com/books?hl=en&lr=&id=ZLDWAgAAQBAJ&oi=fnd&pg=PP1&dq=Intuitive+Biostatistics:+A+Nonmathematical+Guide+to+Statistical+Thinking+(4th+Edition).+&ots=J1d6ljZ9XY&sig=LcxgXrxTtmWK8UvgS5f-beHOyV8#v=onepage&q&f=false



https://doi.org/10.3389/fnhum.2017.00390

https://doi.org/10.3389/fnhum.2017.00390

https://doi.org/10.1038/506150a

https://doi.org/10.1016/s0140-6736(05)66516-6

https://doi.org/10.1016/s0140-6736(05)66516-6

https://doi.org/10.1016/s0140-6736(05)66461-6

https://doi.org/10.1016/s0140-6736(05)66461-6

https://doi.org/10.1016/s0140-6736(05)66379-9

https://doi.org/10.1016/s0140-6736(05)66379-9

https://doi.org/10.1016/s0140-6736(05)61034-3

https://doi.org/10.1016/s0140-6736(05)61034-3

https://doi.org/10.1016/s0140-6736(02)07948-5

https://doi.org/10.1016/s0140-6736(02)07948-5

https://doi.org/10.1016/s0140-6736(02)07882-0

https://doi.org/10.1016/s0140-6736(02)07882-0

https://doi.org/10.1016/s0140-6736(02)07605-5

https://doi.org/10.1016/s0140-6736(02)07605-5

https://doi.org/10.1016/s0140-6736(02)07451-2

https://doi.org/10.1016/S0140-6736(02)07373-7

https://doi.org/10.1016/S0140-6736(02)07373-7

https://doi.org/10.1016/s0140-6736(02)07283-5

https://doi.org/10.1016/s0140-6736(02)07283-5


CASE STUDY 3: ANALYZING STUDY DATA

Background: Data analysis is an exciting time in a research study as it brings the team one step closer to producing answers that inform healthcare decisions. The steps to clean and analyze data to support results interpretation requires time and teamwork to uphold data integrity. Throughout the data analysis, teams work together to review descriptive data and progress to more complex data analysis results.

While data analysis may be viewed as an objective process, it may evoke emotions for those involved with direct experience with the topic. Comparative effectiveness research compares the harms and benefits of alternative methods to prevent, diagnose, or treat illness or improve the delivery of care. This may cause research partners to reflect on their own personal experiences during data analysis.

Challenge statement: Comparative effectiveness research studies may challenge existing standards of care. As a result, research partners who have experience with interventions studied (e.g., as patients, caregivers, and clinicians) may have a more personal relationship to the data that research teams should recognize.

Scenario: A comparative effectiveness research study compares two common treatments for an acute condition (treatment A and treatment B). The research team comprises clinicians representing different clinical specialties. Among the clinician partners, some predominately use treatment A whereas others predominately use treatment B. In addition, patient partners on the team represent individuals with experiences using each of the treatments under study.

The research team plans to convene the full research team, including research partners, to view the initial study results as outlined in the Data Analysis Plan. The lead biostatistician produces the initial tables and data visualizations and recognizes that one treatment appears superior to the other. She is concerned that presenting the data to clinician and patient partners who have primary experience with a treatment that appears to perform worse may cause stress. Clinicians may reflect on the patients for whom they have

confidently prescribed the treatment; similarly, patient partners may reflect on their own treatment experience and results.

Resolution: The biostatistician consulted with the principal investigator and lead for stakeholder engagement. All recognized that data reflects treatment experiences in real-world practice. In preparation for the full meeting, the stakeholder engagement lead scheduled smaller focused discussions with subsets of the clinical and patient partners along with the biostatistician and principal investigator to review the main findings and answer questions or concerns. The discussions allowed an opportunity to hear initial reactions and frame questions for the larger group discussion about the findings and plans for analysis. During the full team meeting, the principal investigator began the discussion recognizing the implications for clinical practice and patient care and the importance of all partners helping to interpret findings in a way that would advance care.

Discussion questions: As a team, use the following questions to discuss your research study and research partnerships in preparation for data analysis.

• How will your study team approach data analysis with your research partners?

• Where can your team envision similar tension points during the data analysis?

• How might research findings cause research partners to reflect on their own personal experiences?

• How will stakeholder input inform decisions regarding the study design? What issues might be contentious and for whom?

• How will you negotiate opposing ideas related to data analysis to ensure patient-centeredness and authentic stakeholder engagement?

• How can you plan to support data analysis discussions with research partners?

• How will you document results from discussions and decisions made about data analyses?



Activity 3a: Using Descriptive Statistics in Your StudyDescriptive statistics are a way to make the collected data more understandable by summarizing the information in it. Different types of descriptive statistics are used depending on whether the data are numeric or categorical. An analyst’s first step with new data is usually to run descriptive statistics to see if any of the results are unexpected or extreme. This activity is designed to be conducted after researchers have collected some preliminary data before study completion.


• Discussing data with research partners• Identifying potential sources of bias within research findings

Activity OverviewUse this activity to establish familiarity with the study results. Research staff and partners should work together on the activity. Fill in the table below with descriptive statistics on key study variables or, if available, use a similar variable table from your study. Creating separate tables for the treatment or high-risk group and the comparison group will help highlight any differences in these groups.

Discuss with research partners the following questions about the descriptive statistics:

1. For each continuous variable

a. Is the mean higher than the median or the other way around? What does that say about the data?

b. Do the minimum and maximum values seem to be real observations, or might they be outliers (extreme values that might be random accidents or a signal of either technical problems or unknown influences on the study results)?

c. What do the standard deviations indicate about the amount of variability in each variable?

2. For each categorical variable

a. What do the counts in the different categories indicate about the people in the study?

b. Do the percentages for each category add up to 100 percent or more than 100 percent? Does this make sense for that variable?

3. Do the numbers make sense overall? Are there any surprises or items to check for accuracy?



Continuous VariablesName Mean Median Minimum Maximum Std. Deviation

Categorical VariablesName Category Count Percentage


• PCORI Methodology 101 Training• Research 101 Training: Data Analysis, Interpretation, and Presentation• Research Fundamentals Learning Package: Module 5—Understanding and

Sharing Research Findings


https://www.pcori.org/engagement/engagement-resources/Engagement-Tool-Resource-Repository/session-3-slides-data-analysis


Activity 3b: Using Inferential Statistics in Your StudyMost PCOR studies use multivariable models in the data analysis because these consider multiple factors simultaneously. There are many types of multivariable models and the output from them may look somewhat different. But most of them share common features. Multivariable model results will typically have:

• One dependent or outcome variable• One or more independent or predictor variables that are the focus of the analysis• One or more covariates, which are other factors that can influence the outcome, and which must

be accounted for to avoid confounding• Indicators of the effect size for each independent variable and covariate• Indications of statistical significance for each independent variable and covariate (confidence

intervals and p-values)


1. Begin early discussions around multivariable models 2. Discuss data interpretation concepts and implications for your study

Activity OverviewThis activity is a way to become familiar with the results produced by your study. Research staff and part-ners should work together on the activity once results are available to discuss. Use the table generated in activity 2b: Identifying the Variables in Your Study to help guide the discussion. Work with the lead statisti-cian to produce the multivariable model output from your study to discuss the following questions:

1. What research question is this analysis trying to answer?

2. What is the dependent variable and how do you know that?

3. What is the independent variable(s) of main interest in the study?

4. What are the covariates, and why are they included?

5. What levels of statistical significance does the model report for different predictor variables? What do significant differences mean for you? Is this an important (meaningful) difference?


• PCORI Methodology 101 Training• Research 101 Training: Intro and Research Design




CHAPTER 4Interpreting Your Study Data

OVERVIEW

How can you decide what your study results mean for patients, caregivers, clinicians, and other stakeholders? Research partners’ input will be central to deciding how to apply the findings to make a clinically meaningful difference in health outcomes. Partners can also identify trade-offs between a treatment’s potential benefit and harm. Research partners’ contribution shows how the study results can improve healthcare decision making and uptake into clinical practice. This module helps research teams look at the results of the analysis with their research partners and discuss what they mean for the various stakeholder communities.

Activities in this chapter include discussion questions for interpreting data while considering study biases, limitations and generalizability and a data visualization exercise.

IMPORTANT CONCEPTS

A study’s internal and external validity are yardsticks to judge how much confidence to have in study data supporting the effectiveness of an intervention or treatment. These are basic concepts in the consideration of how well a study was planned and conducted, if there are other explanations for the findings, and how the findings can be applied outside the study to real life.

Internal ValidityAt its most basic definition, internal validity is about whether the study can support that changes in the dependent (outcome) variables are caused by changes in the independent (predictor) variables. Internal validity is our ability to say that the study findings are trustworthy. In other words, if there is a real association, how likely is the study to find it? An example of the way statistical results can be misinterpreted is the correlation between ice cream sales and sunburn: ice cream sales go up, the sunburn rate increases. Does this mean that ice cream consumption leads people to spend extended time in the sun? No. Rather,

both ice cream sales and sunburn are linked to warmer temperatures. A study has high internal validity if other things that could be affecting the dependent variable have been ruled out.

Even before research teams and partners complete the data analysis, they can begin thinking about internal validity of the findings. Making and maintaining a list of things that could lower internal validity–such as failure to include an important variable in the study or a high drop-out rate for study participants–is a good way to track this. Discussions about internal validity can sometimes lead to changes in study conduct. For example, it may make sense to change the way data are collected to keep response rates high, like adding additional phone calls or changing where and when research staff meet with study participants. Before reaching any firm conclusions about the findings, research teams can brainstorm reasons why they are seeing significant effects for some variables and not for others.

Bias

An important part of checking internal validity is thinking about possible bias. Internal validity is based on both the design of the study and how it is carried out. Bias is defined as anything about the study design or how the study is carried out that pushes study results in a particular direction. For example, a study might be designed to look at breast cancer screening for all women. But if the study ended up recruiting mostly very educated women with a primary care doctor they see once a year, the results might not look the same as if the study had included a broader representation of women. This doesn’t mean the study results aren’t useful, but the team should share details about who was in the study so others can consider that information when looking at the results.

Several types of bias appear in research studies. Some of the most common are:


• Selection bias: Selection bias concerns who signs up for a study and how they are assigned to a treatment group or a comparison group. If the study participants do not represent the target population, there is selection bias. For example, if individuals are recruited for a study through their doctor’s office, then people who don’t go to the doctor very often will be underrepresented. Similarly, women who have family responsibilities that prevent study participation, elderly adults with mobility challenges, and working individuals who can’t make study visits during working hours may be underrepresented unless efforts are made to accommodate their needs. In some cases, selection bias can be addressed in the analysis. In this example study, the research team might decide to collect information on the how often participants visit the doctor so that study results can be weighted or adjusted to better represent the target population.

• Confounding bias: Confounding is another form of bias. Differences between the treatment and the comparison groups can cause confounding bias if they are either not recognized or not measured. It is not always possible to foresee or measure all sources of bias, but good planning and forethought may help the research team avoid major confounding bias. Randomly assigning people to study groups is one way to limit this type of bias. Any unrecognized personal differences will likely balance out across the groups and be discussed during the study’s design. Researchers and partners can discuss potential sources of confounding bias in the early stages of the study, making sure to measure any known predictors of the study outcome. The research team should also carefully review the characteristics of treatment and comparison groups to ensure that they are well balanced relative to important predictors.

• Information bias: This type of bias occurs when data are gathered differently for different people in the study. The participants who keep all appointments, answer surveys, and remain in the study until the end may be

healthier and have better social support than those who drop out or skip parts of the study. Returning to our blood pressure study example, imagine you are working on a study using coaching to educate and share resources with your target patient population. The coaching sessions are scheduled to meet weekly with patients for eight weeks during the study. Some patients received all eight coaching sessions, some received four, and some were coached only twice. Perhaps some patients were not available often enough to have ten visits in the timeframe of the study. Could these patients be different in ways that might affect the study outcome? If so, gaps in the data are not random and can cause bias.

In some studies, researchers must evaluate a person’s health or take other measurements about an individual. The researchers might subconsciously do this more thoroughly for people known to be in the control group or at higher risk than those in the intervention group. To prevent the resulting bias in the study data, it may be possible utilize a double-blind study. In such a study, the researchers are not aware of who is in which group. This prevents another form of bias known as researcher bias. Researcher bias is a process where the researcher influences the systematic investigation, intentionally or unintentionally, to arrive at desired outcome(s).

Here are some suggestions for maintaining Internal validity and protecting it from bias:

• Individuals are selected and assigned to intervention and control groups by chance (randomly).

• A high percentage of people selected for the study agree to participate and provide information on all important study concepts.

• The number of participants is large enough to support the analysis, determined by power analysis (see previous chapter).

• The study was implemented according to the study protocol (see the second section below for more on pragmatic study methods).


When Research Doesn’t Go as Planned Things don’t always happen as planned when conducting a research study, and sometimes the study cannot follow the original protocol. We can even cite global events such as the COVID-19 pandemic, which had significant effects on research project timelines, feasibility, and more. It is important to keep a list of modifications to the study protocol and why they were made so that the team can judge if the changes lower internal validity. Some possible reasons for differences between the conducted study and the planned study are listed in the sidebar. If the team thinks that any of these things might threaten internal validity, they can adjust parts of the study, adapting such features as the recruitment plan, sample size, intervention, measurement of variables, data collection process, or types of statistical analysis performed.

Research partners may have key insights to help identify needed adjustments to the protocol. Suppose that in the blood pressure study, the partners identify the need for coaching sessions at later hours than originally set, and this change to the protocol results in a larger number of coaching sessions per patient. Alternatively, suppose that the partners hear from study participants that some coaches are not making the necessary effort to schedule eight visits with each patient. The research team might consider tracking the timing and length of calls to ensure that coaches are following the protocol’s instructions on patient outreach.

When it is time to interpret the findings and reach conclusions, the research team must decide if the way the study was carried out adequately reflects the initial plan. The team can determine the study maintained fidelity to the protocol if all the changes were small and would not interfere with internal validity. It can be helpful for the team to define fidelity at the start of the study. Exactly how this is done depends on the study, but it might involve describing what deviations from the plan are acceptable in areas such as randomization, data collection, delivery of the intervention, and other key aspects of the research plan.

Thinking about internal validity is especially important if surprising or unexpected findings in the analysis arise. Such findings could indicate something genuinely new and exciting about real-world patient health and behavior. On the other hand, they could be misleading and the result of hidden confounding factors. Activity 5a suggests ways research staff and partners can examine bias and other issues that might lower internal

validity. If no significant problems with internal validity are identified, the team can have more confidence in the results. Research partners and staff can move on to proposing real-world explanations for any surprising findings.

External Validity (Generalizability)While internal validity is about understanding the relationships between the variables in the study, external validity is about applying those relationships to the outside world. If a study has high external validity, then its results can be applied to the real world across a range of healthcare settings, patient groups, and time periods. For example, if you want to modify current clinical practices for blood pressure management in patients who are 65 years or older, then the research team should limit the study population to that age bracket to generalize findings. If you include a significant number of patients who are under 65 years of age in the sample, the intervention or treatment might vary in acceptability or success and its external validity is compromised. External validity is higher if the study participants represent the target population in terms of age, sex, race, geographic location, healthcare setting, and other characteristics. Even a slightly different study sample will not allow findings to be applicable to the target population. Activity 5b presents ideas to spark conversations among research staff and partners concerning the results’ relevance to people outside the study.

WHY THE ACTUAL STUDY MAY BE DIFFERENT FROM THE PLANNED STUDY

• Not enough people participated in the study

• A lot of missing data

• Lack of the required technology, skilled staff, or permissions to do the intervention as planned

• Differences in how collecting data or recruiting participants at different sites

• Mistakes: unplanned problems having to do with staffing or training


CAUSATION

If a study has high internal validity, it points toward causation, an even stronger type of relationship between the predictors and outcomes than a statistically significant association. Several considerations will lead a research team to decide that an observed association is a sign of causation. The list below is condensed from Austin Bradford Hill’s classic lecture on the subject (Hill, 1972). Hill pioneered the use of randomized clinical trials as well as the relationship between smoking and lung cancer. If the research staff and partners think that most of the things on the list are true, they can consider drawing conclusions about causation.

Strength of associationThe size of the effect should be quite large in the treatment or at-risk group versus the comparison groups. (Smaller effects require greater confirmation.)

Consistency of associationThe more studies that report the same association, the more likely cause and effect are present.

Specificity of associationIf the changes in the predictor variable are related to changes in one outcome and not in others, that is a strong suggestion of cause and effect.

Biological gradient (dose-response relation)Greater changes in the predictor variable generally result in greater changes in the outcome variable when a simple, direct cause-and-effect relationship is at work.

Time sequenceIf changing a predictor variable causes change in an outcome variable, the predictor will always change before the outcome does. This is a basic law of nature.

Experimental evidenceRandomized controlled trials and other forms of rigorous testing that neutralize confounding factors can provide stronger evidence for causation.

External validity is the ability to say that the findings are applicable to the population, which cannot be true if internal validity has not been adequately assessed. If many limitations or alternative explanations are associated with the statistical findings, you can’t apply the study results to the real world. On the other hand, the steps researchers take to increase internal validity often lower external validity. For example, trials of a treatment for a particular disease tend to recruit patients that have that disease but no other major health conditions. This can make researchers more certain of the treatment results (higher internal validity), but it leaves open the question of whether a treatment with promising results will help people in the real world. These patients are likely to have other conditions that complicate treatment in ways not measured by

the trial (lower external validity). Similarly, study populations sometimes do not include many women, people of color, rural residents, or other disadvantaged groups with restricted healthcare access. This gap creates uncertainty about how the results apply to the more complicated health needs that these populations experience.

Fortunately, some studies focus on health disparities. They concentrate on recruiting overlooked populations and do not intend to apply their results to relatively privileged patient groups. Similarly, studies focused on aging may limit their enrollment to older adults. Ultimately, the purpose of a study dictates the target population for applying and interpreting results, which may be specific to a setting (hospitals, for example) or demographic group (such as patients

https://www.jstor.org/stable/41139488?seq=1


who are 65 to 90 years old or African American). It is becoming abundantly clear that studies addressing disparities must meaningfully and authentically engage affected communities from the start of a research project. It is imperative for researchers to understand all the layers and nuances of a community’s unique socioecological context to best prepare for study implementation and data interpretation.

External validity will be higher if the study takes place in a typical healthcare setting rather than a lab or artificial environment. This has led to an increased interest in pragmatic trials, which study whether an intervention works in real clinical practice. Pragmatic trials take place across different settings, have very large numbers of participants, and measure a variety of patient-centered outcomes. These studies provide rich information about how well interventions work when implemented by a variety of clinicians. They can help researchers understand what will happen if an intervention is broadly disseminated throughout the healthcare system.

Study LimitationsDisclosing the limitations of your research study is essential. If any factors lower internal or external validity, either in the design of the study or in the way the study was carried out, include these in the limitations section of any publications on the study results. Research teams should work with research partners to gain a solid understanding of the potential limitations others should be aware of when reading about this study and interpreting the results.

One final limitation common to all scientific research is that the collected study data does not

paint a complete picture of what people in the study experienced. Interpretation is necessary to give context. When people have these conversations, they are influenced by their own experience of the study as well as their cultural background. Therefore, it is important to invite people with a variety of perspectives to interpret the study results.

Imagine, for example, the different conclusions a patient with asthma living in a house with high amounts of allergens might reach compared to someone who works in a research lab studying causes of asthma. The researcher might think that the solution is simple: eliminate the allergens. But for the patient, removing the allergens is a complicated, expensive process and may not be the only source of their asthma diagnosis. The asthma diagnosis must be put into a greater context such as accounting for geographic area levels of air pollution (e.g., proximity to a major highway). Therefore, conducting an indoor environmental assessment in the home might be one of many solutions from the patient perspective. To paint the complete picture, there is no substitute for seeing the data from the points of view of those who everyday face the health problem that a study is investigating.

Awareness is growing in the research community about the need to engage diverse communities in data interpretation and analysis. Engaging communities of color in data analysis will help translate research findings to their families, neighbors, and other community members in a way that is understandable and relevant, which may lead to increased health equity, trust, and engagement with the healthcare system and the research that is so critical to it.


SUGGESTED TIPS FOR INTERPRETING DATA WITH RESEARCH PARTNERS

Research partners are an integral part of data interpretation. By approaching results from their respective experiences and perspectives, they add breadth to how the data might be interpreted. Research partners may add to the generation of new hypotheses that may emerge during this process. Notably, engaging communities of color in the data interpretation phase may introduce new programmatic and systemic innovations and ideas, particularly around health delivery, given their understanding of typical target populations in research.

Interpretating data with partners is often an iterative process that begins by looking at the primary research questions laid forth in the study and the data generated guided by the Data Analysis Plan. As the team analyzes and discusses data, considerations regarding validity, fidelity to protocol, and implications for real-world practice are considered. The below table provides a series of suggestions for planning and supporting discussions regarding data interpretation with research partners.

Allot adequate time for iterative discussions on data interpretation. Discussions with stakeholder partners will often require subsequent data analysis based on insight and feedback, warranting multiple meetings/discussions.

Recognize research partners, in particular patients and caregivers, may look to find or understand their own experiences in the data. For some, data interpretation may evoke memories from individual experiences. Research team members are advised to consider emotional needs of stakeholders prior to group discussions.

Utilize visual displays of data to facilitate group discussion.

Invite stakeholder partners to offer interpretations of data building from individual or group experiences.

Utilize data interpretation as an opportunity to generate new hypotheses or areas for future exploration.

Plan for topics that may be more challenging to cover and determine how best to discuss or break down content. For example:

● You may need to cover complex topics over multiple meetings or in person.

● Contentious topics may be better suited to smaller group discussions among subsets stakeholder partners before being brought to a larger group.



Basch, Ethan. 2013. “Toward patient-centered drug development in oncology.” N Engl J Med. Aug 1;369(5):397-400. https://doi.org/10.1056/nejmp1114649.

Gabriel, Sherine E., and Sharon-Lise T. Normand. 2012. “Getting the methods right--the foundation of patient-centered outcomes research.” N Engl J Med. Aug 30;367(9):787-90. https://doi.org/10.1056/nejmp1207437.

Grimes, David A., and Kenneth F. Schulz. 2002. “Bias and causal associations in observational research.” Lancet. Jan 19;359(9302):248-52. https://doi.org/10.1016/s0140-6736(02)07451-2.

Hill, Austin Bradford. 2015. “The environment and disease: association or causation? 1965.” J R Soc Med. Jan;108(1):32-7. https://doi.org/10.1177/0141076814562718.

Ioannidis, John P. A., Sander Greenland, Mark A. Hlatky, Muin J. Khoury, Malcolm R. Macleod, David Moher, Kenneth F. Schulz, and Robert Tibshirani. 2014. “Increasing value and reducing waste in research design, conduct, and analysis.” Lancet. Jan 11;383(9912):166-75. https://doi.org/10.1016/s0140-6736(13)62227-8.

FURTHER READING

https://doi.org/10.1056/nejmp1114649




https://doi.org/10.1016/s0140-6736(02)07451-2

https://doi.org/10.1177/0141076814562718

https://doi.org/10.1177/0141076814562718

https://doi.org/10.1016/s0140-6736(13)62227-8

https://doi.org/10.1016/s0140-6736(13)62227-8


CASE STUDY 4: INTERPRETING STUDY RESULTS

Background: Expanding research teams to include stakeholders with direct experiences with the research topic adds breadth and depth to data interpretation. By discussing the results of data analyses with stakeholders, it is possible to place results in context. Statistical tests guide how results are interpreted, but these tests must be placed into context when decisions are made.

Challenge statement: Researchers and research partners may draw different interpretations from the data when results are conflicting and lead to unanticipated conclusions.

Scenario: A research study comparing two healthcare delivery designs for caring for patients with a new-onset chronic condition recently completed data analyses. Patients newly diagnosed with the condition (symptoms persisting for three months or more) are randomized to one of two programs. In one treatment arm, patients participate in a three-month exercise program offering a weekly physical therapy session with virtual exercises and stretches to be completed at home, treatment X. In the other treatment arm, patients received usual care comprising patient education materials, medication management, and follow-up with primary care at three months, treatment Y. Patient-reported pain and fatigue are measured as primary study outcomes at three months, six months, nine months, and one year.

Upon review of the study results (see Figure),

the research team notices that over the course of the study, the patient-reported outcomes findings were different for pain versus fatigue. Patients who received treatment X reported being in more pain, on average, over time. At 12 months, patients receiving treatment X still reported more pain, on average, than did those receiving treatment Y. The differences were both clinically important (likely to be noticed by patients) and statistically significant (unlikely to have occurred by chance). However, for fatigue, patients receiving treatment Y reported being more fatigued over the course of the study compared to those on treatment X. Most of the important and significant differences in levels of fatigue were seen at three and six months. By 12 months, patients in both arms reported similar levels of fatigue; the difference in average scores was small (neither clinically meaningful nor statistically significant.

Researchers conclude that usual care (treatment Y) is the more effective strategy in this scenario as pain is significantly better. While fatigue is reported higher at the earlier timepoints, both groups report mild fatigue by the study’s end, which the researchers deem as the most important finding. Patient partners, however, disagree. While pain improvement is important, the early differences experienced with fatigue are concerning. Side effects from pain medications include fatigue, which impacts other aspects of daily living. Patient partners feel the ability to obtain mild pain levels (albeit not as low as with medication assistance) is ideal to remain independent of medications


and the associated fatigue side effects, even if at the end of one year fatigue is similar between the two treatment arms. This perspective represents both patients with direct experience with treatment approaches studies and leaders from patient advocacy organizations representing broader communities who have stated similar priorities relative to outcomes.

Resolution: This perspective modified how the research team interpreted the patient-reported outcomes in the study’s main findings to reflect that improved pain experienced with treatment Y is tempered by increased fatigue experienced early on with treatment Y as compared to treatment X. Based on study findings, recommendations supported starting with treatment X and evaluating at six months before initiating treatment Y. The discussion in the study’s main results journal publication, as well as talking points for presentations and FAQs accompanying study reports, shared how patient partners informed the interpretation when comparing the two treatments given their experiences with the condition.

Discussion questions: As a team, use the following questions to discuss your research study and research partnerships in planning for discussions on data interpretation. This discussion should be done during the planning stage of the research study, not discussed for the first time after data are collected.

• How will your study team discuss data interpretation with research partners?

• Where can your team envision similar tension points during data interpretation? What results could present conflicting interpretations between interventions studied?

• How will you negotiate opposing ideas related to data interpretation to ensure patient-centeredness and authentic stakeholder engagement?

• How can you plan to support discussions with research partners? How can data visualizations support discussions with research partners to interpret study results?

• How will you document results from discussions and decisions made about study design?

CASE STUDY 4: CONTINUED



Activity 4a: Interpreting the Results—Overall Study Findings When a study has finished collecting data, done some statistical analysis, and produced preliminary findings, the research team begins interpreting results. This is an iterative process that begins by looking at the primary research questions established in the study and the data generated guided by the Data Analysis Plan. Research partners can provide valuable insights into factors that may affect the interpretation of the findings. Some general questions for discussion by the research staff and research partners are given below. The research team may be able to tailor the questions or devise others specific to its study.


1. Discuss the findings of the data analysis plan relative to your study aims2. Discuss implications for subpopulations within the study

Activity OverviewThis activity is intended to support research team efforts to interpret results produced by your study with research partners. Research staff and partners should work together on the activity once results are available to discuss. Discussion questions should follow a brief presentation of results by a designated research team member.

Discussion questions:

1. Based on the analysis done in this study, what are the answers to the research questions in the Data Analysis Plan? Did the study make progress on all its aims?

2. Were the findings of the study the same for everyone or did the study look at different subgroups of people based on their health or personal characteristics and background? If so, how do the conclusions differ for different groups studied?

3. Was the study’s fidelity to the protocol high enough? What activities differed from the plan, and are these differences important in interpreting the finding?


Activity 4b: Interpreting the Results—Sources of Bias and Study Limitations The research team begins interpreting results after it has finished collecting study data, done some statistical analysis, and produced preliminary findings. During this iterative process the team considers issues of internal validity, possible biases, and fidelity to the protocol. Research partners can provide valuable insights into factors that may affect interpretation of the findings. Some general questions for discussion by the research staff and research partners are given below. The research team may be able to tailor the questions or devise others specific to its study.


1. Discuss potential sources of bias in the study that may affect interpretation 2. Discuss implications of bias for interpreting findings and limitations

Activity OverviewThis activity is intended to support research team efforts to interpret results produced by your study with research partners. Research staff and partners should work together on the activity once results are available to discuss. Discussion questions should follow a brief presentation of results by a designated research team member.


1. What are potential sources of bias in the study? Might there be:

a. Selection bias?

b. Information bias?

c. Confounding?

d. Some other type of bias?

2. If bias is present, what was or could be done to reduce its impact on the study findings?

3. Did the study have high enough fidelity to the protocol? What activities differed from the plan, and are these differences important in interpreting the finding?

4. Were there any surprising findings? If so, do you believe them? Do you think they occurred be-cause of an issue with how the study was designed or carried out, or do you think they indicate something real about the variables in the study?

5. What are the study’s limitations relative to internal validity?


Activity 4c: Interpreting the Results—Sources of Bias and Study Limitations One reason to conduct a research study is to learn something about real-world practice. When the data analysis is complete and any limitations are considered, it is time to discuss the key results and explain what it all means for healthcare practice. This means thinking about answers to the research questions that guided the study from the start. It also means thinking about external validity and generalizability to other populations and healthcare settings.

Most researchers want to reach the strongest and broadest conclusions possible, as long as the conclusions are backed up by the data. Research partners can help think of ways that the findings apply in the real world and how to frame the conclusions. Some general questions for discussion by the research staff and research partners are given below. You may be able to tailor the questions or devise others specific to your study.


1. Discuss the findings of the data analysis plan with respect to your study aims2. Discuss how the study findings apply to other settings of practice3. Identify ways to illustrate the findings in the context of research partner experience

Activity OverviewThis activity is intended to support research team efforts to interpret results produced by your study with research partners. Research staff and partners should work together on the activity once final results are established. Discussion questions should follow a brief presentation of results by a designated research team member.


1. This study was done with certain subgroups of people. Do you think the findings apply to other subgroups in the real world? If yes, then who, and how broadly do you think the findings apply?

2. This study was done in certain healthcare settings. Do you think the findings apply in other types of healthcare settings? If yes, then where, and how broadly do you think the findings apply?

3. Part of bringing a research study to life is telling stories that show the impact of the findings. Can you tell a story from your own experience that explains the real-world meaning of the findings?

4. Now that the study results are in, what is the main message people should draw from the study? Who needs to hear this message and what do you hope they do differently once they hear it?


Activity 4d: Facilitating Stakeholder Interpretation of Results Using Data VisualizationUsing effective visualization of analyzed study data can support discussions with research partners during the data interpretation phase. This activity presents a scenario highlighting how an analysis can reveal results that may lead to different interpretations by different end users.

This activity leverages design principles from a PCORI-funded study publication, Making a Picture Worth a Thousand Numbers: Recommendations for Graphically Displaying Patient Reported Outcomes Data. The study convened a multi-stakeholder panel to generate consensus recommendations for effective visual display of patient-reported outcome data. The research team is encouraged review the recommendations not only for this activity but for planned study analyses.


1. Discussing graphic representations of patient-reported outcomes data2. Interpreting patient-reported outcomes data with research partners3. Identifying potential sources of bias within research findings

Activity OverviewResearch partners participate in data interpretation by reviewing results and providing insight based on their experiences with the topic. Presenting data visualizations is one way to facilitate conversations with research partners.


1. As a group, review the Clinician’s Checklist to Evaluate Studies Using PROs, available at Clinician’s Checklist for Reading and Using an Article about Patient-Reported Outcomes, as an example of questions that research partners may consider when reviewing data. Note: While the checklist is targeted to clinician stakeholders, the questions are easily adapted for different team members.

2. Interpreting visualizations:

a. Allow time for research partners to review the visualizations presented below independently.

b. As a single group or in breakouts (depending on group size) discuss the following:

i. What observations could be made from the data as presented?

ii. What is unclear about the visualization (or what questions do the visualizations raise?)?

c. Review as a group important features of the data visualization, including:

i. How to interpret the x and y axis

ii. The elements of how data are displayed (e.g., confidence intervals, p-value, time)

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6363861/pdf/nihms-1509353.pdf



3. Discussing results:

a. Review with the group the potential conflict in results presented. In this example, the visu-alization of the data indicates there may not be an obvious advantage between treatments. Over the course of the study, the patient-reported outcomes findings were different for pain versus fatigue. Patients who received treatment X reported being in more pain, on average, over time. At 12 months, patients receiving treatment X still reported more pain, on average, than did those receiving treatment Y. The differences were both clinically important (likely to be noticed by patients) and statistically significant (unlikely to have occurred by chance).

Patients receiving treatment Y reported being more fatigued over the course of the study compared to those on treatment X. Most of the significant differences in fatigue levels were seen at the three- and six-month marks. By 12 months, patients in both arms reported sim-ilar levels of fatigue; the difference in average scores was small (neither clinically meaningful nor statistically significant).

Discuss how conflicting data such as these can be interpreted in the context of research findings. For example, pain and fatigue both affect overall quality of life. Patients may have different views on which domain is more important when interpreting findings. How might you handle this when interpreting results?

i. How do multiple data collection points help us understand treatment effect and pa-tient symptoms over time?

ii. What other observations could be made?

4. Review the Clinician’s Checklist to Evaluate Studies Using PROs with research partners. Discuss how to best adapt these questions for your study.


• Making a Picture Worth a Thousand Numbers: Recommendations for Graphically Displaying Patient Reported Outcomes Data

• Clinician’s Checklist for Reading and Using an Article about Patient-Reported Outcomes



https://www.mayoclinicproceedings.org/article/S0025-6196(14)00102-5/fulltext#sec1.1


CHAPTER 5Reporting and Presenting Study Results

OVERVIEW

PCORI defines dissemination as the intentional, active process of identifying target audiences and tailoring communication strategies to increase awareness and understanding of evidence and motivate its use in policy, practice, and individual choices. The purpose of dissemination is to spread and sustain knowledge and the associated evidence-based interventions. It is best to start thinking about dissemination early in the research process. The nature of the study goals, research questions, and analytic strategy all impact how the audience for the study results is comprised. Research partners know about key stakeholders and the problems they are trying to solve. Considering stakeholders’ issues when shaping study aims and goals makes the study more relevant to the community.

This chapter includes activities that will enable researchers and research partners to identify key messages important to highlight when communicating research findings as part of the dissemination plan. In activity 5a, research teams work together to create a plain-language summary of the study and its results. In activity 5b, research partners work together to consider important messages targeted to important stakeholder audiences.

IMPORTANT CONCEPTS

A dissemination plan is an organized strategy for spreading the word about a study’s findings so that targeted end users are aware of and appreciate the value of the findings. Research partners can help researchers better understand how end users, such as patients, obtain and use information to make decisions. Partners’ personal experience can help humanize and contextualize statistical findings by putting a personal face on the numbers. In addition, research partners often have networks that will help spread research findings. Discuss with stakeholder partners the important ways their involvement supports dissemination efforts, including the following:

• Identify target audiences. There may be several!

• Develop key messages tailored to target audiences.

• Create personal stories to humanize statistical results.

• Co-present study results.

• Coauthor journal articles and other publications.

The next step after identifying the target audiences is to think about the key messages that are most important for each audience. Make key messages clear and use language that the target audience will find relatable and understandable. Base the messages on the study results and prompt the target audience to act. One way to think of a key message is to ask yourself, “Why should this audience care about the study, and what do I want them to do differently because of the study results?”

Tailoring Results Dissemination & Promotion to Community AudiencesIf a target audience is members of the community, tailoring information to that audience includes adopting culturally appropriate terms and avoiding jargon, as described in chapter 4. If the research staff has engaged with community partners throughout the study, then the team will be prepared to tie the study results to local concerns and information needs. Having community partners on the research team helps give the study credibility, and those partners will be well positioned to share information about the study and its findings. Strong community support can in turn ensure broader uptake of the evidence-based, care-improvement strategies indicated by the study findings.


Based on your understanding of where each audience gets its information, choose a presentation format that fits the target audience (see box for example formats). No matter the format, presentations should carefully explain the questions that the study answers, how it arrived at those answers, and how it helps patients. Anticipate controversial points in the study and be open about its strengths and weaknesses.

Tables and graphics should reinforce the presentation. Be clear about the goals of the presentation and design tables or graphics that express the study’s key points. Create handouts and infographics that emphasize a few key, clear points (see chapter 4). Be sure that each graphic or table is clearly titled and that the predictor and outcome variables are properly labeled. Add text to further discuss the findings and what they mean for the audience.

Academic Journal Articles and Conference Presentations

Academic publications and conference presentations are important for researchers’ professional advancement. They are good ways to reach other researchers, clinicians, patients, stakeholders, and other health professional audiences.

Research partners can be active participants in these academic dissemination channels. Presenting study results at a conference is often a first step in refining the message and conclusions that will go into a journal article. Many researchers and research partners in patient-centered outcomes studies co-present at research conferences. These joint presentations highlight research partners’ important roles in conducting the study and bring home the value of the research results for improving peoples’ lives.

When the time comes to write the manuscript for a medical journal, research partners will be able to advise on discussing the study findings and conclusions, developing plain-language abstracts (see chapter 4), and writing and reviewing sections of the paper.

Writing journal articles is a long and difficult process and not all partners will want to participate. One of the earliest decisions is which journal to aim for since each journal has its own style, topic areas, formatting, and length requirements. A growing number of journals are interested in articles about engagement in research, and some journals want to know how

patients were involved as research partners and as part of the manuscript submission process.

The paper typically goes through many drafts while the research team tries to make it as clear as possible. Coauthors should understand that first drafts almost always get rewritten and that when this happens, it is not meant to question the value of a person’s experience or thoughts. Once the research team submits the paper to the journal, multiple rounds of critiques, revisions, or outright rejections can follow. Most journals send the paper for peer review by outside researchers. These reviewers often send back detailed lists of questions and comments that must be addressed if the paper is to be accepted for publication. Changing a paper to satisfy reviewers can be hard work, but it usually results in a better paper.

Other Types of Publications

Medical journals are an important type of publication for researchers, but they are not the only type of publication that matters. Getting the word out to some audiences may mean writing for trade publications aimed at people working in certain fields like hospital management or writing for magazines aimed at a variety of audiences. In addition, blogs, websites, video hosting sites, and social media are now common ways to quickly get information out to many local and national audiences. Traditional news media remain important for many people, especially those who don’t have easy access to computers. Newsletters, local and national newspapers, and

PRESENTATION FORMATS• Face-to-face communication at one-on-

one or group meetings is perhaps the most convincing way to get across the importance of study results.

• Videos can be attention-grabbing formats for storytelling and are getting easier to make.

• Health fairs and workshops, public lectures, town hall meetings, and patient-doctor interactions are all useful platforms for incorporation into dissemination plans.

• Community groups often have newsletters or bulletins that will carry your message to their readership.


SUGGESTED TIPS FOR REPORTING STUDY FINDINGS WITH RESEARCH PARTNERS

Research partners are vital to a research team’s ability to connect to end users of research. Research partners help teams understand important messages to communicate, effective forums for research-targeted audiences, and presentation strategies that make information accessible to inform decision making. The table below provides suggestions for working with research partners to craft and share research results.

popular magazines can be used to publicize study findings both through the Web and in print. This is also true of local TV news stories. In all these cases, the research team can work with news journalists to produce the report. Including the research partners in this effort will help make these stories clear to the audience.

It is important to discuss different products or publication formats with partners given their knowledge of preferred modes of information uptake. Research partners can leverage their

connections to the target community and share preferred outlets and/or methods for results promotion. They can also be regarded as trusted community members that can craft culturally sensitive and inclusive messaging around results to increase uptake.

For each type of publication, the basic steps are the same: pick an audience, decide on a message, figure out where the audience gets its information, and write in a style that works for that type of publication.

Identify early with stakeholder partners any experience participating in research publications and presentations.

Identify any professional experience among stakeholder partners regarding communication, writing, or marketing that may support results reporting.

Create a publication and presentation policy to guide plans for sharing results.

● Designate appropriate travel funds to support research partners attending meetings to co-present findings.

● Clarify with research partners all plans and opportunities to participate in co-presenting research findings.

● Identify venues and forums important to research partners where results should be presented.

Ensure transparency in how decisions are made for coauthorship for all research partners.

Designate a publication and presentation committee comprising study personnel and stakeholders to oversee planning and conduct of this work.

Invite research partners to recommend additional forums to present study results.

Review and update publication and presentation work with research partners regularly throughout the project. Frequency should correlate with phase of the study (e.g., more frequently as publications and presentations are generated for the public).

Review with research partners the standard process for submitting abstracts and publications for peer review set forth by the respective organization. Note: For many, the peer-review process is new and may seem harsh or onerous when undergoing revision and resubmission.

For public presentations, allot appropriate time to review slides and run through content with all presenters.



Haynes, Tiffany F., Ann M. Cheney, J. Greer Sullivan, Keneshia Bryant, Geoffrey M. Curran, Mary Olson, Naomi Cottoms, and Christina Reaves. 2017. “Addressing Mental Health Needs: Perspectives of African Americans Living in the Rural South.” Psychiatr Serv. Jun 1;68(6):573-578. https://doi.org/10.1176/appi.ps.201600208.

Ridpath, Jessica R., Sarah M. Greene, and Cheryl J. Wiese. 2007. PRISM Readability Toolkit. 3rd ed. Seattle: Group Health Research Institute. https://www.nhlbi.nih.gov/files/docs/ghchs_readability_toolkit.pdf.

Tai-Seale, Ming, Greer Sullivan, Ann Cheney, Kathleen Thomas, and Dominick Frosch. 2016. “The Language of Engagement: “Aha!” Moments from Engaging Patients and Community Partners in Two Pilot Projects of the Patient-Centered Outcomes Research Institute.” Perm J. Spring;20(2):89-92. https://doi.org/10.7812/tpp/15-123.

Smith, Katherine C., Michael D. Brundage, Elliott Tolbert, Emily A. Little, Elissa T. Bantug, Claire F. Snyder, and PRO Data Presentation Stakeholder Advisory Board. 2016. “Engaging stakeholders to improve presentation of patient-reported outcomes data in clinical practice.” Support Care Cancer. Oct;24(10):4149-57. https://doi.org/10.1007/s00520-016-3240-0.

Snyder, Claire F., Katherine C. Smith, Elissa T. Bantug, Elliott E. Tolbert, Amanda L. Blackford, Michael D. Brundage, and PRO Data Presentation Stakeholder Advisory Board. 2017. “What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability.” Cancer. May 15;123(10):1848-1859. https://doi.org/10.1002/cncr.30530.

Susan G. Komen. nd. “How to Read a Research Table.” Last modified December 16, 2020. Accessed June 3, 2021. https://ww5.komen.org/BreastCancer/HowtoReadaResearchTable.html.

George Mason University. 2013. “Understanding and Using Statistical Tables.” https://www.youtube.com/watch?v=tWhL3JYAJcY.

Rodrigues, Velany. 2013. “How to write an effective title and abstract and choose appropriate keywords.” Editage. https://doi.org/10.34193/EI-A-6376.

University of Michigan Center for Health Communications Research, Robert Wood Johnson Foundation. 2014. Visualizing Health: A Scientifically Vetted Style Guide for Communicating Health Data. http://www.vizhealth.org/.

Patient-Centered Outcomes Research Institute. 2015. “Dissemination and Implementation Framework and Toolkit.” https://www.pcori.org/research-results/putting-evidence-work/dissemination-and-implementation-framework-and-toolkit.

Ryan, Cynthia. n.d. “How to write an op-ed.” American Association for Cancer Research. Accessed June 3, 2021. https://www.aacr.org/ADVOCACYPOLICY/SURVIVORPATIENTADVOCACY/PAGES/HOW-TO-WRITE-AN-OP-ED.ASPX#.V_ZzJPkrLIU.

Bodison, Stefanie C., Ibrahima Sankaré, Henry Anaya, Juanita Booker-Vaughns, Aria Miller, Pluscedia Williams, Keith Norris, and Community Engagement Workgroup. 2015. “Engaging the Community in the Dissemination, Implementation, and Improvement of Health-Related Research.” Clin Transl Sci. Dec;8(6):814-9. https://doi.org/10.1111/cts.12342.

Bordeaux, Bryan C., Crystal Wiley, S. Darius Tandon, Carol R. Horowitz, Pamela Bohrer Brown, and Eric B. Bass. 2007. “Guidelines for writing manuscripts about community-based participatory research for peer-reviewed journals.” Prog Community Health Partnersh. Fall;1(3):281-8. https://doi.org/10.1353/cpr.2007.0018.

Flicker, Sarah, and Stephanie A. Nixon. 2018. “Writing peer-reviewed articles with diverse teams: considerations for novice scholars conducting community-engaged research.” Health Promot Int. Feb 1;33(1):152-161. https://doi.org/10.1093/heapro/daw059.

FURTHER READING

https://doi.org/10.1176/appi.ps.201600208

https://www.nhlbi.nih.gov/files/docs/ghchs_readability_toolkit.pdf



https://doi.org/10.7812/tpp/15-123

https://doi.org/10.7812/tpp/15-123

https://doi.org/10.1007/s00520-016-3240-0

https://doi.org/10.1007/s00520-016-3240-0

https://doi.org/10.1007/s00520-016-3240-0

https://doi.org/10.1007/s00520-016-3240-0

https://ww5.komen.org/BreastCancer/HowtoReadaResearchTable.html

https://ww5.komen.org/BreastCancer/HowtoReadaResearchTable.html

https://www.youtube.com/watch?v=tWhL3JYAJcY

https://www.youtube.com/watch?v=tWhL3JYAJcY

https://doi.org/10.34193/EI-A-6376

https://doi.org/10.34193/EI-A-6376

http://www.vizhealth.org/

https://www.pcori.org/research-results/putting-evidence-work/dissemination-and-implementation-framework-and-toolkit




https://www.aacr.org/ADVOCACYPOLICY/SURVIVORPATIENTADVOCACY/PAGES/HOW-TO-WRITE-AN-OP-ED.ASPX#.V_ZzJPkrLIU




https://doi.org/10.1111/cts.12342

https://doi.org/10.1353/cpr.2007.0018

https://doi.org/10.1353/cpr.2007.0018

https://doi.org/10.1093/heapro/daw059

https://doi.org/10.1093/heapro/daw059


Huang, Jennifer, Paula Darby Lipman, C. Daniel Mullins. 2017. “Bridging the divide: building infrastructure to support community-academic partnerships and improve capacity to conduct patient-centered outcomes research.” Transl Behav Med. Dec;7(4):773-782. https://doi.org/10.1007/s13142-017-0487-z.

Schillinger, Dean. 2010. An Introduction to Effectiveness, Dissemination and Implementation Research. P. Fleisher and E. Goldstein, eds. From the Series: UCSF Clinical and Translational Science Institute Resource Manuals and Guides to Community-Engaged Research, P. Fleisher, ed. Published by Clinical Translational Science Institute Community Engagement Program, University of California San Francisco. https://accelerate.ucsf.edu/files/CE/edi_introguide.pdf.

Robinson, Elizabeth T., Deborah Baron, Lori L. Heise, Jill Moffett, and Sarah V. Harlan. 2014. Communications Handbook for Clinical Trials: Strategies, Tips, and Tools to Manage Controversy, Convey Your Message, and Disseminate Results. New York: Microbicides Media and Communications and Research Triangle Park: Family Health International. https://www.fhi360.org/sites/default/files/media/documents/Commhandbk_full_052314.pdf.

https://doi.org/10.1007/s13142-017-0487-z

https://doi.org/10.1007/s13142-017-0487-z

https://accelerate.ucsf.edu/files/CE/edi_introguide.pdf

https://accelerate.ucsf.edu/files/CE/edi_introguide.pdf

https://www.fhi360.org/sites/default/files/media/documents/Commhandbk_full_052314.pdf




CASE STUDY 5: REPORTING AND PRESENTING STUDY RESULTS

Background: Expanding research teams to include stakeholders with direct experiences with the research topic adds breadth and depth to data interpretation. By discussing the results of data analyses with stakeholders, it is possible to place results in context. Statistical tests guide how results are interpreted, but these tests must be placed into context when decisions are made.

Challenge statement: Conflict within a team can arise when deciding if and when to share individual study results.

Scenario: A research study poised to determine the comparative effectiveness between two treatments at 30 days, 1 year, and 2 years recently completed patient recruitment. Early outcomes at 30 days assessed patient quality of life and adverse outcomes experienced between the two groups, which were randomized to each treatment. In addition to quality of life, adverse outcomes and exacerbations are assessed at both one and two years. Patient and stakeholder partners involved in the study want results analyzed and released once the 30-day data are available, with the understanding the long-term data will be released later. Publishing 30-day results will help patients actively making decisions about what treatment to choose. The research team wants to wait to release results until two-year data are accrued on all patients. This will allow people to understand the short-term outcomes in the context of long-term outcomes that will be important considerations for treatment decisions. Further, concern exists that 30-day outcomes could influence the study participants who have not completed long-term follow-up.

Resolution: To reach consensus on how to proceed, an iterative process occurred to decide when to release results for data analysis prior to ending patient enrollment. The principal investigator held multiple meetings with patient, clinician, and national

stakeholders to discuss the pros and cons of different strategies for the release of results. After multiple discussions, a compromise was reached to release results when all enrolled study participants reached one year. Patient partners supported this decision, understanding that while immediate data would not be available, long-term outcomes would better support high quality decision making by having the context of long-term outcomes available. The research team felt one year would allow the team to rigorously assess and disclose the most important aspects of study outcomes without affecting outcomes evaluation at two years.

Discussion questions: As a team, use the following questions to discuss your research study and research partnerships in planning how to release study results.

• How will your study team approach study data release?

• How will you shape plans with and by research partners?

• Where can your team envision similar tension points when planning the release of study data?

• How will you negotiate opposing ideas related to release of study results to ensure patient-centeredness and authentic stakeholder engagement?

• How can you plan to support discussions with research partners?

• How will you share plans for communicating study results with research partners and study site collaborators?

• When and how should research teams share results with study participants?



Activity 5a: Creating a Plain-Language Summary An important step in disseminating research findings is creating a plain-language summary of the study. The plain-language summary includes a list of key points that highlights what the study accomplished and how people can use the study results and conclusions.


1. Understand key messages to communicate to stakeholder audiences2. Identify potential avenues for disseminating study results



10 mins

Review Goals for Plain-Language Summary

• Use existing project summary (if available) to present content• Highlight goals for readability, clarity, and focus on main

findings

40 mins

Group Discussion

As a group, discuss each component of the plain-language summary by asking:

• What was the research about?• What did the research team do?• Who was in the study?• What were the results or findings?• What were the study’s limitations? (Discuss the study’s

uniqueness and generalizability. If the study findings are not particularly new or are not applicable to a large population outside the study, then a broad dissemination strategy may not be feasible.)

• How can people use the results? (Discuss the key findings to communicate to outside communities.)

5 mins

Wrap-Up

• Discuss next steps for preparing and circulating a draft of the lay summary based on the discussion. If preparation of the plain-language summary will be led by a research partner(s), allow this individual(s) to discuss plan and timeline.


Activity 5a Worksheet: Creating a Plain-Language SummaryUse the following worksheet to document responses to the following aspects of the research study in your own words.

1) What was the research about?

2) What did the research team do?

3) Who was in the study?

4) What were the results or findings?

5) What were the study’s limitations? (List the study’s uniqueness and generalizability.)

6) How can people use the results? (List the study’s key findings to communicate to outside communities.)



• Existing plain-language project summary (if available). For PCORI-funded initiatives, the plain-language project summary is available on PCORI’s website under the Portfolio of Funded Projects.

• PCORI Dissemination Framework and Toolkit, 2015. This toolkit contains a detailed review of study result dissemination and implementation strategies. It includes exercises enabling research teams to apply these strategies to their own study.

• The PCORI website includes a range of plain-language study summaries. You can search the Portfolio of Funded Projects to find summaries relevant to your study.

https://www.pcori.org/research-results?f%5B0%5D=field_project_type%3A298

https://www.pcori.org/sites/default/files/PCORI-DI-Toolkit-February-2015.pdf



Activity 5b: Write Key Messages Tailored for Each Target AudienceOnce the plain-language summary is established, it is time to write key messages tailored to each target audience and consider the best ways to reach them. When tailoring messages, use language that is familiar to members of the target audience and make points that will resonate with them. Be sensitive to the culture, language, knowledge, and experience of the target audience.


1. Construct key messages for target audiences2. Identify potential avenues to present study results



15 mins Identify Main Results

• As a group, create a comprehensive list of main study results (see activity 5a).

20 mins

Breakout Discussions

• Break into small groups with each group assuming the role of a targeted stakeholder audience (e.g., patients, policy makers, clinicians, payer).

• For each study result, discuss as group why the targeted audience should care about the study and what you want them to do differently because of the study results.

• Record ideas using the worksheet below (Note: The first two lines serve as examples).

15 mins

Group Discussion

• Reconvene as a group and discuss key messages and how to best disseminate them to the people you want to hear them.

• Discuss with research partners the following:• What venues do research partners feel are important for

presenting study results?• Who is best suited to present to the audience? • What considerations must be made regarding culture,

language, knowledge, and experience of the target audience?

5 minsWrap-Up

• Discuss next steps to prepare and communicate study results.


Worksheet: Gene

Key Study Finding Target Audience

Why Is This Finding Important to the

Audience?Key Message How They Get

Information

Example 1. RCT demonstrates treatment A is superior to treatment B

Professional societies

Current recommendations in practice are based on expert opinion (level C) and research- level evidence is needed to improve guidelines.

Guidelines should be updated to reflect latest evidence.

Peer-review publications, national meetings

Example 2. Virtual Cardiac Rehab program equivalent to six-week intensive program on clinical and patient-reported outcomes

Patients, care partners, patient advocates, and clinicians

Current practice standard focuses on in-person healthy heart classes and physical conditioning. Virtual adaptations offer convenience for some but do not have data on outcomes.

Both approaches to cardiac rehab support improved patient outcomes. Decisions should be tailored to patient preferences and support system available.

Patient advocacy organizations, media, cardiologists, hospital patient education/discharge materials


• The PCORI website includes a range of plain-language study summaries. You can search the Portfolio of Funded Projects to find summaries relevant to your study.



Optional Activity: Periodic Check-In to Review Engagement in Data Collection, Analysis, and Interpretation As the study enters each phase related to data collection or analysis, the research staff and research partners should discuss the engagement process, its progress, and how they would like it to continue in the next phase. Ideal times for these discussions are:

• At the start of each new phase of data collection

• At the start of data analysis

• Before developing any manuscripts or other dissemination materials

• At the end of the study to evaluate the engagement process

Below are some general questions for discussion about engagement in the study’s data-related phases. The questions can be modified as needed and included in surveys or used to guide discussions with research partners.

For Research Partners

These questions are meant for collaborative discussion between the research staff and research partners. They are suitable for guiding one-on-one conversations or group discussions, with answers made verbally or in writing.

1. How have you felt about the discussions on data collection, analysis, and interpretation? Have you felt included?

2. Have you felt that you were prepared to be part of these discussions? If not, what could have been done to help you feel more prepared?

3. Have you felt that the research staff listened to you and considered your advice?

a. Give an example of when your advice made a difference in the study.b. Give an example of when you thought your advice was not given full consideration,

if any.

4. How would you describe the amount of time you spent on the data aspects of this project?

a. Too littleb. Just rightc. Too much

5. What could the research team do differently to make it easier for you to take part in the data-related aspects of the study? These improvements might have to do with training, resources, meeting formats, communication, or other ways that impact how you work together.


For Researchers

These questions are meant for discussion among the research staff members who helped to engage research partners in the study’s data-related aspects. Their answers will suggest ways that the staff can refine or enhance its approach to partner engagement.

1. Have the research partners participated as much as you would like during data collection, analysis, and interpretation? If no, why not?

2. Were research partners prepared to participate in the data-related aspects of the study? If not, what could be done differently to improve their preparation?

3. What additional information and/or resources do research partners request that would improve their ability to contribute?

4. How have research partner contributions impacted the data-related aspects of the study? Give examples of valuable contributions made by research partners that significantly changed the study (e.g., defining outcomes, suggesting data collection approaches, developing presentation materials, interpreting study results).

5. How would you characterize the time needed for research partner engagement in data collection, analysis, and interpretation?

a. Less than expected

b. About as expected

c. More than expected

6. Are there things that the research staff could do differently to improve partner engagement in the data-related aspects of the study?


AAlternative Hypothesis

Predicts the intervention or treatment under question will have a significant effect when exploring group difference. Importantly, the alternative hypothesis may also state directionality of the predicted difference.

BBias (in Research)

In research, bias occurs when differences between groups might distort the study results. Bias makes study results less credible. For example, a study might compare patients at Clinic A taking Medicine A with patients at Clinic B using Medicine B. However, most patients at Clinic A are older than 65, and most patients at clinic B are between ages 30 and 45. If Medicine B appears to work better, it might be because of the medicine, or it might be because patients who got it were younger and healthier than those who got Medicine A. Bias can also occur when people in a study don’t reflect the larger patient population. For example, if patients in a study about diabetes are all the same race, results may not be accurate for people who are not the same race as those studied.

Bivariate Statistics

Measures of association used to look at two variables at a time.

CCategorical Data

Represent discrete groups or levels. They can classify or describe study targets such as people or organizations. They answer questions like “what type?” or “which group?”

Clinical Equipoise

The assumption that there is not one better intervention present during the design of a randomized controlled trial, for either the control or experimental group. A true state of equipoise exists when one has no good basis for a choice between two or more care options.

Clinical Trial

A type of research design in which researchers assign people to receive one or more treatments to evaluate the effects of the treatments on aspects of health.

Coefficient of Determination

Also called r-squared, the coefficient of determination is the percentage of the response variable variation that is explained by a linear model. It is always between 0 and 100 percent. R-squared is a statistical measure of how close the data are to the fitted regression line.

Comparator

A prevention, diagnosis, treatment, or healthcare delivery option being compared to another in a study.

Comparative Effectiveness Research

Research that compares two or more prevention, diagnostic, treatment, or healthcare delivery options to see which works better for certain patients under certain circumstances. For example, a researcher might compare two headache medicines to see which one works better for post-menopausal women who get severe headaches. Comparative effectiveness research is often referred to as CER.

GLOSSARY


Confounding

When the results of a study could be muddied by other unplanned factors that might provide an alternative explanation for the outcomes. Confounding can occur from any factor that was not planned to be part of the study. For example, a study might compare two ways to reduce obesity in children and test those two ways in different cities. If one city also started a youth sports program at the same time, and the study did not account for it, the study results in that city might be affected by both the obesity program and the sports program, making it difficult to know how well the obesity program worked.

Confounding Bias

A form of bias created when differences between the treatment and the comparison are either not recognized or not measured.

Continuous Variables

Variables that have infinitely uncountable elements and are typically represented in intervals or measurements, such as weight and height. They can take on almost any numeric value and can be meaningfully divided into smaller increments, including fractional and decimal values.

Covariates

Other variables used in statistical analysis to help make the relationship between a study’s independent (predictor) and dependent (outcome) variables clearer.

DData

Information that is collected and analyzed to answer a specific question in a research study. Data can be information from many sources, such as answers to questions asked in interviews with patients, or measurements of a patient’s symptoms from a patient’s medical record. Data can also be admissions records or insurance claims for healthcare services, such as visits to the emergency room or the use of rehabilitation services.

Data Analysis Plan

Also referred to as the study protocol, the Data Analysis Plan is an essential guide for planning the data-related aspects of the research project.

Data Sources

The sources of the data used in the study. This describe where the data for each characteristic, or variable, will come from. Examples include surveys, focus groups, interviews, electronic medical records, and other sources.

Dependent Variables

Response variables are also known as dependent variables and outcome variables. They help determine whether changes in the predictors are associated with changes in the response. The outcomes represented by these variables are the changes in study participants’ health or experiences that most interest researchers and partners.


Descriptive statistics are numbers that summarize data, such as the mean, standard deviation, percentages, rates, counts, and range. Descriptive statistics describe the data but do not try to generalize beyond the data.

Dichotomous Variables

Also called binary variables, they are a special type of categorical variable. They can take only one of two values, such as yes/no variables that tell whether a certain condition is present.

Discrete Variables

These are always whole numbers and are considered countable elements. They can map to natural numbers and can be finitely countable.

Double-Blind Study

A study including a control group and an intervention group in which researchers are not aware of who is in which group.


Dissemination

The intentional, active process of identifying target audiences and tailoring communication strategies to increase awareness and understanding of evidence, and to motivate its use in policy, practice, and individual choices.

E

Effect Modification

Also known as heterogeneity of treatment effect, or interactions among variables. Effect modification means that the effect of the independent variables on the dependent variable is different for different people.

Effect Size

The effect size is standardized mean difference between the two groups, meaning, a way of quantifying the size of the difference between two groups. It is particularly valuable for quantifying the effectiveness of a particular intervention, relative to some comparison.

External Validity

Applies the relationships between the variables in the study to the outside world. If a study has high external validity, its results can be applied to the real world across a range of healthcare settings, patient groups, and time periods.

FFidelity to the Protocol

Adherence to key aspects of the research plan such as randomization, data collection, and intervention delivery.

GGeneralizability

How well the outcomes of a study for those participating in the study reasonably apply to all other people with the same condition or circumstances.

HHazard Ratios

A variation of relative risk. It is a special way of showing disease risk in in one population compared to another. Hazard ratios add up changes in relative risk over time.

Heterogeneity of Treatment Effect

Interactions among variables, also known as effect modification. It means that the effect of the independent variables on the dependent variable is different for different people.

Hypothesis

Predicts the answer to a research question based on theory and existing evidence. There are two types of hypotheses: null, the absence of an effect, and alternative, any hypothesis based on theory or evidence that indicates an effect/difference between groups.

IIndependent variable

A characteristic such as age that could potentially affect the dependent variable. Also called predictor variables.

Inferential Statistics

A type of statistical tests that looks for the relationships between the comparators and the study outcomes. Inferential statistics use a random sample to draw conclusions about the population.

Information Bias

A type of bias that occurs when data are gathered differently for different people in the study.


Internal Validity

Allow researchers to say that study findings are trustworthy. Internal validity is about whether the study can support the assertion that changes in the dependent (outcome) variables are caused by changes in the independent (predictor) variables.

Interquartile Range

A measure of spread that shows how spread out the data values are. Interquartile range focuses attention on the middle 50 percent of the data range.

Interval Data

Characterized by precisely equal spaces in between each number which can be measured, sharing similar properties to ratio data. Each number is equally distant from the next based on what that number represents. Examples are degrees in Celsius or measures of distance.

Intervention

A healthcare prevention, diagnosis, treatment, or delivery activity being studied. When more than one activity is being studied, one is referred to as the intervention and the others are called comparators.

KKurtosis

A measure of spread that, on a normal bell shaped curve, describes the height of the peak in comparison to how wide the curve spans.

LLinear Association

When two variables have a linear association when all the data points, otherwise known as dots, can have a best fit line fitted to the model to calculate the correlation coefficient.

MMaximum

Highest. The simplest way to express spread is to state the difference between the highest, or maximum, and lowest, or minimum, values for a variable.

Mean

The mean describes an entire sample with a single number that represents the center of the data. It is the arithmetic average and is calculated by adding up all the observations and then dividing the total by the number of observations. The mean is the most used central measure.

Measure of Association

A measure of association quantifies the relationship between two or more variables, such as exposure and disease, among study groups.

Median

A measure of central tendency representing the halfway point, with half of the study population having a value below the median and half above it.

Measure

A specific outcome or result that the research team chooses as a way of answering the research question. Measures are based on data that the research team can collect in consistent ways. For example, a team might want to compare two blood pressure medicines. One measure might be patients’ blood pressure after three months of taking the medicine. Another measure might be how many patients had a heart attack or died after starting the medicine. Another measure might be a survey about how patients felt about their health after taking the medicine for a year.

Measures of Spread

Show how spread out the data values are. For example, they can measure the spread between all the variable values and the mean. The most common measures of spread are range, interquartile range (IQR), and standard deviation.


Method

A scientific process or plan for how a research team should answer a research question so that the findings are valid, reliable, and credible. A method lays out what kind of data to collect and how to collect it. Methods also help researchers analyze the data, or understand what the data means and how it answers the question.

Minimum

Lowest. The simplest way to express spread is to state the difference between the highest, or maximum, and lowest, or minimum, values for a variable.

Mode

The value that occurs most frequently in a set of observations and the most common value for a study variable, most often appearing as the highest point in a chart. You can find the mode by counting the number of times each value occurs in a data set.

NNominal Variables

Have no order, which means the order is meaningless. The names assigned to each category (such as those for race/ethnicity, religion, sex, and political viewpoint) would be assigned numbers for statistical analyses, but those numbers have no significance relative to the rank-ordering or priority between groups.

Null Hypothesis

The absence of a significant effect from intervention(s) or treatment(s) when exploring group differences. The null hypothesis states that there are no significant differences.

Numeric Variables

Have values that answer questions such as “how much?” and “how many?” Examples can include heart rate, rainfall measured in inches, weight, and number of cardiac episodes. The two types of numeric variables are discrete and continuous.

OObservational Study

A type of study in which researchers observe the results of a healthcare treatment. The researchers do not assign the treatment to people; rather they look at how the treatment is used in regular health care.

Odds Ratio

A variation of relative risk that is a special way of showing disease risk in one population compared to another. Odds ratios are used in certain analyses in which relative risk proves impractical.

Ordinal Variables

Categorical variables that have a meaningful ranking or order. Rankings include age group (baby, teenager, young adult…), disease stage, or attitude (e.g.: “strongly agree,” “agree,” “disagree,” and “strongly disagree”).

Outcomes

The measurable result of a healthcare activity, such as taking an x-ray, prescribing or giving someone medicine, or conducting surgery. Many results are measurable, for example: a diagnosis, like a broken bone on x-ray); a change (or no change) in health, like how long a patient lived or whether they felt better; or what a patient was able to do, such as climb stairs or go home from the hospital.

Outcomes Variables

Also called dependent variables, the outcomes represented by these variables are the changes in study participants’ health or experiences that most interest researchers and partners.


PPatient-Centered Outcomes Research

A type of comparative effectiveness research that compares outcomes that matter most to stakeholders, such as patients and those who care for them, healthcare providers, and healthcare advocates. Patient-centered outcomes research, or PCOR, requires the engagement of stakeholders as active partners in research. By sharing their lived experience and expertise, stakeholders influence research to be more patient-centered, relevant, and useful.

Pearson’s Correlation Coefficient

The Pearson correlation evaluates the linear relationship between two continuous variables. A relationship is linear when a change in one variable is associated with a proportional change in the other variable.

PICOTS

A brief overview of the essential characteristics of a study: Participants, Interventions and Comparators (the treatments), Outcomes (what is measured), Timeframe, and Setting.

Population

A group of people identified by a shared characteristic such as their type of health condition, age, race, gender, occupation, or geographic location. For example, in a research study, a population might be men who smoke or women of child-bearing age who live in Chicago.

Power Analysis

Using statistics to determine sample size, or how many people need to be in a study for the study to detect a difference between two treatments, if a difference exists.

Pragmatic Study

Study in which participants are assigned to treatments in the settings where they live, work, or receive health care. Other types of studies test how treatments work in carefully controlled settings, but pragmatic studies look at how treatments work in everyday care.

Predictor Variables

Also called independent variables, predictors variables are characteristics such as age that could potentially affect the dependent variable. A predictor variable explains changes in the response.

Principal Investigator

The person who leads and organizes the research study and team and is responsible for the study being completed.

Probability Value (p-value)

A statistical concept about the probability that the study would produce a result if there were no difference between the two treatments being studied. A p-value of .05 means that there is a 5 percent, or 1-in-20 chance, that a difference in results for Medicine A and Medicine B is just chance and there is no real difference between the two. The lower a p-value, the more confident researchers are that there is a real difference between two treatments or groups.

Prospective Observational Study

A type of study in which a research team collects data for people who happen to be getting a certain treatment. The study covers a specific period of time going forward. For example, it might track all emergency room visits for the next six months, or patients’ health for several years after they get a treatment.


Protocol

A plan for the procedures a research team will follow to conduct a research study. A protocol answers questions such as: (1) who will be in the study; (2) where, when, and how will data be collected; and (3) how will participants’ rights be protected. Protocols are reviewed by an Institutional Review Board.

QQualitative Research

Research methods that use people’s descriptions or perspectives to answer questions. Examples of qualitative research include interviews and focus groups to come up with ideas for how to improve mental health services in schools, or what matters most to people living with a chronic condition. Other qualitative research includes observing and analyzing how people interact or what they say or write.

Quantitative Research

Research methods that use numbers (statistics) to measure the relationships between treatments and their effects on people’s health. Examples of things that can be measured in numbers include blood pressure readings, responses on a ratings scale, or the number of days patients spend in the hospital. Researchers analyze the data using statistical tests to determine the relationship between a treatment and the result. For example, they might test whether patients spent less time in the hospital with Treatment A compared with Treatment B, and whether the difference was really because of the treatment, or just chance.

RR-Squared

The percentage of the response variable variation that is explained by a linear model. It is always between 0 and 100 percent. R-squared is a statistical measure of how close the data are to the fitted regression line.

Randomized Controlled Trial

A study design that randomly assigns participants into an experimental group or a control group. As the study is conducted, the only expected difference between the control and experimental groups in a randomized controlled trial is the outcome variable being studied. Randomized controlled trials and other forms of rigorous testing that neutralize confounding factors can provide stronger evidence for causation.

Randomization

A process used in experimental study designs in which a research team assigns patients to treatment groups by chance.

Range

One of the most common measures of spread, range is the difference between the highest (maximum) and lowest (minimum) values for a particular variable.

Ratio Data

Characterized by an equal and definitive ratio between each data point with absolute zero being treated as a point of origin. Meaning, the zero is meaningful because it represents the absence of something. A negative numerical value cannot exist in ratio data.

Recruitment

The process of inviting people to participate in a research study. Recruitment includes: (1) identifying potential study participants, (2) telling them about all aspects of the study including possible benefits and harms that may occur during or after the study, and (3) inviting them to consent (agree) to participate in the study.

Relative Risk

Aims to identify disparities in the incident of disease across groups based on exposure status. Two variations of relative risk are odds ratios and hazard ratios. Both are special ways of showing disease risk in in one population compared to another.


Reliability

The degree to which, if another research team did the same study using the same methods, they would get similar results.

Research Questions

Expand on the study aims to provide more detail about the topics to be investigated in the research plan.

Researcher Bias

A process where the researcher influences the systematic investigation, intentionally or unintentionally, to arrive at desired outcome(s).

Retention

Keeping participants involved to fulfill each step of a study.

Risk Difference

Aims to identify disparities in the incident of disease across groups based on exposure status.

SSample

A group of people participating in a study. When planning a study, researchers decide on the characteristics that are important to include so they can be sure the sample is a good match for the population the study is about. For example, they may want to be sure to recruit people of different ages, races, and ethnicities.

Sample Size

The number of participants in a study. The sample size should ideally be large enough for a study to detect differences between two or more treatments.

Screening

A test that detects signs of a condition in people who do not have symptoms. For example, a doctor might use a blood test as a screening method for early signs of diabetes. Based on the results of screening, a patient might get more tests to confirm that they have a condition.

Selection Bias

Concerns who signs up for a study and how they are assigned to a treatment group or a comparison group. If the study participants do not represent the target population, there is selection bias.

Setting

Where a study takes place. For example, a study about preventing falls at home might look different than a study about preventing falls in a nursing home.

Significance

Significance might refer to statistical significance or clinical significance. Statistical significance means how likely something happened by chance. Clinical significance means how important something is in daily life. It is important to clearly distinguish between the two.

Skewness

Skewed data are not equally distributed on both sides of the distribution—it is not a symmetrical distribution. Data are skewed right when most of the data are on the left side of the graph and the long skinny tail extends to the right. Data are skewed left when most of the data are on the right side of the graph and the long skinny tail extends to the left.

Standard Deviation

A measure of spread for statistical analysis. The standard deviation is an indicator of the average spread between each variable value in the data from the mean value for that variable.


Statistical Significance

The likelihood that a study result is caused by chance or by the treatment being studied.

Statistical Methods

Statistical methods aim to observe differences between individuals or groups based on a set of variables that are considered. Statistical methods sort through data points that represent the recorded observations, measurements, or counts that involve individual human beings. The statistical methods section of a study will describe the types of statistical analyses that will be appropriate to use to analyze the data.

Statistical Tests

Statistical tests help organize and interpret data. Research teams use statistical tests to determine if the data support the hypothesis and the likelihood of a measurable effect occurring by chance.

Statistics

The science of organizing and analyzing data that are in the form of numbers. By using different calculations, researchers can measure the relationship between treatments and results in a study. For example, a common calculation is whether patients’ results are likely caused by a treatment or might be caused by chance.

Statistician

The person on a research team who designs and leads the calculations needed to analyze the study results and how they relate to what treatments patients received.

Study Aims

Refers to the main goals or overarching objectives addressed by the research project. Study aims are general and high level and are almost always positioned at the very beginning of a Data Analysis Plan.

Study Population

The population in which the research team will test the study’s hypotheses and/or generalize the study observations. Depending on the research questions, it may be necessary to describe a study population based on demographics such as age, sex, race/ethnicity, marital status, education level, place of residence, type of medical insurance, and socioeconomic status. It may also be important to describe the study population according to their health, such as past or present diseases or health conditions, medications received, ability to perform activities of daily living, etc.

Subpopulations

Groups of study participants who have a characteristic in common. For example, a study might focus on how well a blood pressure treatment works for people older than age 65. The research team might also look at how well the treatment worked for specific groups within the study sample, such as women, African Americans, people with low incomes, or people who have diabetes.

Systematic Review

A type of study that examines all the evidence that can be used to answer a research question. For example, a research team might conduct a systematic review of all previous studies about how well a medicine works to treat a disease. The reviews are called systematic because they follow strict rules about the quality of evidence they include and how the results of different studies can be combined.

TTimeline

Analytic plans usually include a timeline representing the dates when the main steps of the study will happen. For example, the timeline will indicate when recruiting starts, when data collection starts and ends, and when the results should be ready. The timeline may be shown in different ways such as a flow chart or a table.


VValidity

How accurately a study measured what it intended to measure. Researchers refer to several types of

validity. Internal validity is being able to demonstrate that the study’s results are based on the treatment and not on other factors. External validity is being able to generalize the results of the study to other people or other settings. Content validity is making sure to measure all the aspects that make up the thing being studied. For example, a study about depression would ask not only about mood, but about energy level, self-worth, and disrupted sleep or eating.

Variables

Something that can be measured and have different values, and that might play a role in study results. For example, many studies collect information about patients’ age, race, ethnicity, and sex. Each is a variable, and the research team might compare results for men and women, or people younger than or older than 65.

Credit: Some definitions adapted from https://statisticsbyjim.com/glossary/. Copyright ©2021, Jim Frost; https://www.leeds.ac.uk/educol/documents/00002182.htm, Robert Coe School of Education, University of Durham; and Study Design 101 by Himmelfarb Health Sciences Library.

https://statisticsbyjim.com/glossary/

https://statisticsbyjim.com/glossary/

https://www.leeds.ac.uk/educol/documents/00002182.htm

https://www.leeds.ac.uk/educol/documents/00002182.htm


RESOURCE TABLETitle Description Chapter

ResourceActivity Resource

Goal Link

PCORI Methodology 101 Training

PCORI Merit Review Training Guide to support stakeholders participating in PCORI merit review

Full Guidebook

1a, 1b, 1c, 2b

Provides an overview of study design, includes PICO approach, basics on analysis.


PCORI Methodology Standards Checklist

PCORI developed standards to ensure research studies address important methodological standards in the conduct and reporting of research.

Full Guidebook

2a Provides a central checklist for awardees to document and track relevant methodlogy standards that must be addressed by the research team.


PCORI Methodology Standards Academic Curriculum

PCORI training materials regarding the methodology standards for researchers and stakeholders. Includes a teaching guide with self-assessment questions.

Full Guidebook

2a Provides recorded trainings focused on the 2013 version of the PCORI Methodology standards.


PCORI Engagement Challenges, Strategies, and Resources

PCORI handout highlighting engagement lessons learned and available resources.

Full Guidebook

Provides strategies and links to resources for PCORI community members.


PCORI Updated Engagement Plan Template

PCORI developed engagement template for awardees to document the engagement strategy planned for the study.

Full Guidebook

Provides a structured approach to documenting plans for stakeholder engagement throughout study conduct.







































Title Description Chapter Resource

Activity Resource

Goal Link

PCORI Dissemination Framework and Toolkit, 2015

PCORI generated a dissemination and implementation toolkit providing a detailed review of dissemination and implementation strategies.

Chapter 5 5a This document contains a detailed review of study result dissemination and implementation strategies. It includes exercises enabling research teams to apply these strategies to their own study.


IBEMC Stakeholder Network Governance

A document that outlines the roles and responsibilities of individuals involved in the network.

Chapter 1 Provides teams with an example for creating governance for the study to support clarity in decision making and roles.

https://www.pcori.org/sites/default/files/IBEMCSN-Governance-Document.pdf

University of California, Riverside, Facilitator Guide

A training guide for individuals facilitating group meetings.

Chapter 1 This is a training guide developed by the University of California, Riverside for small-group discussion facilitation. The guide includes information on facilitator role, tips for ensuring successful small-group conversation, and notetaking resources. The training guide could be used as an example or template for other teams interested in training facilitators.

https://www.pcori.org/sites/default/files/GoGM-Facilitator-Training-Guide-2395-UCR.pdf

IBEMC: How to be Effective in Meetings

A powerpoint used in planning for an upcoming meeting.

Chapter 1 Provides tips for planning for a successful meeting.

https://www.pcori.org/sites/default/files/Patient-Partner-Webinar-4_0.pdf
























Activity Resource

Goal Link

Comagine Health’s Building Successful Collaborations with Communities: A Guidebook for Researchers, Agency, and Organization Staff

A comprehensive Guidebook for researchers, agency, and organization staff to develop and sustain research partnerships.

Chapter 1 This comprehesive guide provides a compliation of recommendations from community leaders, community liaisons, and community-engaged researchers to support partnerships.

https://www.pcori.org/sites/default/files/Building-Successful-Collaborations-with-Communities-A-Guidebook-for-Researches-Agency-and-Organization-Staff.pdf

Comagine Health’s Building Successful Collaborations with Communities: A Guidebook for community organizations, leaders, and members

A comprehensive Guidebook for community organizations, leaders, and members to develop and sustain research partnerships.

Chapter 1 This comprehesive guide provides a compliation of recommendations from community leaders, community liaisons, and community-engaged researchers to support partnerships.

https://www.pcori.org/sites/default/files/Building-Successful-Collaborations-with-Communities-A-Guidebook-for-Community-Organizations-Leaders-and-Members.pdf

Designing an All-In meeting

A 10-step guide created by the Necrotizing Enterocolitis Society to host successful in-person meetings.

Chapter 1 and Chapter 5

The guide provides strategies for successful multi-stakeholder meetings.

https://necsociety.org/2019/07/12/nec-societys-10-step-guide-for-an-all-in-meeting/

The PICOTS Framework: How to Write a Research Question

An activity developed by NORD to help stakeholders understand PICOTS for writing research questions.

Chapter 1 1b An example for approaching the PICOTS framework using a self-guided worksheet.


RadComp Social Media Proposal

The RadComp leveraged social media to engage with the public and disseminate findings. The proposal outlines the strategy in detail.

Chapter 5 (if maintained)

An example for teams considering social media use. This plan is quite comprehensive and a good document for discussing/ co-producing plans with stakeholders.

https://www.pcori.org/sites/default/files/RadComp-Social-Media-Proposal.pdf












































Activity Resource

Goal Link

Survey Design for Community Members

The Univerisity of Southern Florida created this training for community members as part of the USF PathED Collaborative Stakeholder Research Capacity Building Workshop Series

Chapter 2 An overview of surveys, survey design, and administrations. Includes activities to conduct with research partners.

https://www.pcori.org/sites/default/files/PCORI-Survey-Training.pdf

Research 101 Training: Intro and Research Design

Research 101 training slides developed by the Global Healthy Living Foundation. The Research 101 training is divided into three sessions, and this session covers research design.


2a, 3b This slide deck provides an overview for research and research design to help research partners gain a basic understanding of research conduct. A series of weblinks provided at the end include additional resources on specific topics.


Research 101 Training: Research Instruments & Data Collection

Research 101 training slides developed by the Global Healthy Living Foundation. The Research 101 training is divided into three sessions, and this session covers research instruments and data collection with a focus on survey design.


This slide deck provides an overview for research instruments and data collection to help research partners gain a basic understanding of methods for collecting data from research participants. A series of weblinks provided at the end include additional resources on specific topics related to survey design and administrations.



















Activity Resource

Goal Link

Research 101 Training: Data Analysis, Interpretation, and Presentation

Research 101 training slides developed by the Global Healthy Living Foundation. The Research 101 training is divided into three sessions, and this session covers data analysis, interpretation, and presentation.


3a This slide deck provides an overview fordata analysis, interpretation, and presentation to help research partners gain a basic understanding of methods for collecting data from research participants. A series of weblinks provided at the end include additional resources on data analsysis and interpretation.


Making a Picture Worth a Thousand Numbers: Recommendations for Graphically Displaying Patient-Reported Outcomes Data

This publication presents recommendations for presenting graphic visualizations of patient-reported outcomes data.

Chapter 4 4d This manuscript provides recommendations generated by a PCORI-funded methods award for effectively presenting patient-reported outcomes data.


Clincian’s Checklist for Reading and Using an Article about Patient-Reported Outcomes

This publication presents a checklist of questions to consider when reviewing patient-reported outcomes data as part of generated evidence.

Chapter 4 4d The checklist is intended to help practicing clinicians understand clinical research articles that include patient-reported outcomes so that the information can be used for decision making.

https://www.mayoclinic proceedings.org/article/S0025-6196(14)00102-5/fulltext#sec1.1
























Activity Resource

Goal Link

Plain Language Summaries

The PCORI website includes a range of plain-language study summaries. You can search on the Explore Our Portfolio of Funded Projects page to find summaries relevant to your study.

Chapter 5 5a Plain-language summaries are required for all PCORI-funded studies to improve readability and access to broad audiences. Examples are available on the Explore our Portfolio page of the website.

https://www.pcori.org/research-results?f%5B0 %5D= field_project_type %3A298

Newsletter Examples

A link to newsletters generated by various studies and projects to keep people informed of study activities and results.

Chapter 5 Newsletters provide an important mechanism for keeping people informed of study progress. This includes research partners, study participants, and the broader community. The PCORI Engagement Tool and Resource Respository provides a number of examples for reference.

https://www.pcori.org/engagement/engagement-resources/Engagement-Tool-Resource- Repository ?keywords =newsletter








https://www.pcori.org/engagement/engagement-resources/Engagement-Tool-Resource-Repository?keywords=newsletter













Activity Resource

Goal Link

A Research Handbook for Patient and Public Involvement Researchers

A handbook covering different types of research design and conduct.

Full Guidebook

This handbook is written for patients and members of the public who want to understand more about the approaches, methods, and language used by health services researchers.

https://www.manchester openhive.com/ view/9781526136527/9781526136527.xml

Introduction to Study Design

An basic primer for study design.

Chapter 2 2a This document provides a brief overview of study design with a basic graphic to help people see differentiating characteristics in study design.

https://www.cebm.net/wp-content/uploads/ 2014/06/ CEBM-study-design- april-20131.pdf

Enhancing Community Health Center PCORI Engagement (EnCoRE) Research Training

A series of recorded webinars and slide presentations covering a range of research fundamentals.

Full Guidebook

Provides a series of recorded webinars covering research concepts to support stakeholder partnership in research

https://cdnencore.wordpress.com/live-session-library/

https://www.manchesteropenhive.com/view/9781526136527/9781526136527.xml







https://www.cebm.net/wp-content/uploads/2014/06/CEBM-study-design-april-20131.pdf















guide for engaging with research partners about data and

Documents