reviewing conference abstracts pat o’sullivan research and development in medical education
Post on 25-Dec-2015
215 Views
Preview:
TRANSCRIPT
REVIEWING CONFERENCE ABSTRACTSPat O’Sullivan
Research and Development in Medical Education
Acknowledgments
Paul Hemmer, MD, MPHUniformed Services University of the Health Sciences
G. Dodd Denton, MD, MPHUniformed Services University of the Health Sciences
Karen Szauter, MDUniversity of Texas Medical Branch School of Medicine
Sebastian Uijtdehaage, PhDUniversity of California Los Angeles
Brian Mavis, PhDMichigan State University
Creative Commons LicenseAttribution-NonCommercial-Share Alike 3.0 Unported
You are free:• to Share — to copy, distribute and transmit the work • to Remix — to adapt the work Under the following conditions:• Attribution. You must give the original authors credit (but not in
any way that suggests that they endorse you or your use of the work).
• Noncommercial. You may not use this work for commercial purposes.
• Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.
See http://creativecommons.org/licenses/by-nc-sa/3.0/ for full license.
Workshop Objectives
1. Define and describe major structural components of a conference abstract
2. Recognize what makes each component effective
3. Describe the elements required in a “Methods” and “Results”/ “Evaluation Plan” section
4. Avoid common pitfalls in a “Discussion” section
5. Describe the peer-review process for a conference abstract
6. Write reviews for abstracts being considered for a conference
Abstracts are different than manuscripts
• Often describe preliminary or pilot studies• Word restrictions (400 max) limit the scope • Exclude supporting documentation such as references,
tables, etc.
Anatomy of abstract
Title
1. IntroductionPurpose/Background
2. Methods
3. Results/Evaluation Plan
4. Discussion/Dissemination
5. Reflective Critique
THE INTRODUCTION
The Introduction provides background or rationale• Reference to what is already known• Reference to general knowledge or accepted
standards of practice (“conceptual framework”)• Addresses the “So what?” question
A clear Problem Statement/Purpose must be included
• Reviewer should clearly understand:• What question was being investigated, and• Why this question is important
Abstract is unacceptable without a clear purpose/research question
Problem Statement, Purpose, or Hypothesis
• Problem statement/Research Question • “We investigate the relationship between USMLE Step 1
scores and self-assessed study skills” • Statement of Purpose
• “Our purpose is to develop a curriculum for students to improve skills in communicating bad news”
• Hypothesis• “We hypothesized that our students would demonstrate
improvement in exam scores after participating in on-line remediation”
References to the literature are not required
• Word limits preclude full reference citations
• Authors might include abbreviated citation, e.g.: Smith, 2008.
• Allows author and reviewer to differentiate between “opinion” and building on another’s work
Example 1
INTRODUCTION
Increasing the enrollment of students from underrepresented groups in medical schools continues to be a challenge. Recent data (Smith, Jackson & Jones, 2011 ) showed that only 7% percnt of physicians are minority whereas this proportion is 25% in the US population overall. Pipeline programs are designed to improve diversity in medical schools. This study focused on the academic performance of underrepresented (UR) students on courses that are required for applying to medical school (“gateway courses”).
Example 2
INTRODUCTION
Increasing the enrollment of students from underrepresented population in medical schools continues to be a challenge. Minority students’ achievement in college courses that our required for application in medical school (“gateway courses”) is key to successful admission in medical schools. We sought to determine (a) whether minority students receive lower grades than non-minority students in pre-health gateway courses; (b) the extent to which lower grade performance might be explained by the differences in precollege academic achievement; and (c) whether minority students are less likely than non-minority students to persist in completing at least four gateway courses.
METHODS
General Considerations for Methods
• Operationalizes the research question or the innovation• Describes a quantitative, qualitative design, or mixed
designOR• Describes the framework for development of innovation• Explicit statement about IRB approval or review is optimal,
but not required
Methods describe:“Who, where, what, when, how?”
• Who is sampled?
• Patients, students, residents, fellows, faculty?
• What is the sample size?
• What is the setting?
• What is the source of data?
• Survey, papers, examinations, etc.
• What is the study design?
• When is this study done?
• How is the data analyzed?
What reviewers look for in Qualitative Studies• Was sampling appropriate to the question?• Was data collection method appropriate?
• Interviews, focus groups, survey responses, written comments
• Was data analysis done with rigor?• Who analyzed the data?• How were coding difference resolved? • Triangulation?
Innovation or Curriculum Development
• Guiding framework for development• Methods used for steps taken
• Ensure the quality of the innovation or curriculum developed• Data planned or collected
Example 1
METHOD
Using regression analysis, we analyzed academic performance of 5,000 college students in three California colleges who matriculated in the 2009–2010 or 2010–2011 academic year. We compared grade performance of students across ethnic groups in gateway courses and the degree in which they completed at least four gateway courses, using regression methods to control for students’ college admission test scores. Statistical significance was set at 0.05 .
Example 2
METHOD
We obtained retrospective, administrative data from 3 California colleges on 5,000 college students who matriculated in the 2009–2010 or 2010–2011 academic years and enrolled in at least one college gateway course. We compared students across ethnic groups in gateway course grade performance and persistence in completing at least four gateway courses, using regression methods to control for students’ college admission test scores.
Curricular Example 1• METHOD
We held weekly seminar taught by faculty from medicine and law schools for ten weeks. The curriculum focuses on areas where medicine and law intersect, including foster care, prisoner health, caring for refugees, and homeless health policy. Participants are exposed to academic and community leaders who express interest in developing mentoring opportunities with students. Eleven law and thirteen medical students enrolled in the seminar during the first year. Seminar sessions are conducted in lecture-based format for one hour followed by half hour small-group discussions. Students simultaneously interact with each other in the clinical setting when providing direct services with faculty supervision, and are encouraged to discuss these experiences.
Curricular Example 2• METHOD
We developed the joint medical and law seminar following a curriculum development framework where we analyzed needs, set objectives, chose teaching strategies, implemented and evaluated the seminar. Needs assessment indicated selecting topics where medicine and law intersected such as foster care, prisoner health, caring for refugees, and homeless health policy. Since we were emphasizing the intersection, we chose a format that allowed for sharing as well as didactics. Didactics were reviewed by the development team prior to sessions. Success was assessed by numbers enrolling from both schools, attendance, and learner and faculty satisfaction.
RESULTS
Results must present key findings• Actual data (numbers) must be presented• Congruent with Methods section• Descriptive data usually present• Inferential analyses, with p values, should be present
as appropriate.
Results: Essential components
• Response rate to a survey
• Participation rate in cohort studies
• Numbers in control and intervention groups
Preferably:• Numbers and/or reasons for exclusion
• Demographic information
Data and p values must be reported
RIGHT:
“Group A scored higher than group B (means: 10 ± 2 versus 4 ± 1, p<.05)”
WRONG:
“Group A scored significantly higher than group B (p < .05)”
What influences statistical significance?
• Sample size • As sample size increases, smaller differences needed for
significance
• Multiple comparisons• As number of statistical tests increases, likelihood of at least one
statistically significant finding by chance alone increases• the p value can and should be adjusted for multiple comparisons
Statistical vs Functional Significance
• Statistical analyses with significant p values ideally should be accompanied by a measure of functional significance
• Examples: Effect size, confidence intervals, proportion of variance explained (R2)
• Such measures help to show the importance of the statistical difference
“Negative” studies
• Negative findings can be important for learning what works (or doesn’t work) in medical education
• However, need to consider if negative or non-significant findings due to lack of statistical power
• Most common reason is insufficient sample size
• With negative findings, author should include a comment about power in the abstract• If absent, this is a feedback point in the written comments
Results: Qualitative Studies
Description of themes and sub-themes that emerged• Word limit may preclude example quotes
Differences among participants, groups may be discussed
Quantification of the results often not included or relevant
Results: Common Pitfalls
• No data (actual numbers) are reported• Results “will be presented at the meeting” is
unacceptable [*may depend on conference but helps if have preliminary or provide plan]
• No or low response rate (with surveys)• Not enough detail about measures• Statistical analyses inappropriate for type of data• Authors interpret data instead of just listing results:
“Analyses support our hypothesis that our intervention was effective and should be broadly implemented (60 vs. 35, p<.05)”
Example 1
RESULTS
Gateway courses included Biology, Organic Chemistry, Calculus and Physics . Calculating the GPA for these courses, we found that on average , the GPA was lower for UR students compared to non-UR students (p<.05). This gap could not be explained by differences in prior academic performance. However, no statistically significant differences were found in the rate in which UR students completed at least four gateway courses compared to white students. However , after accounting for the lower grades of UR students in their initial classes, UR students were more likely than non-URM students to complete four or more gateway courses (p<.01).
Example 2
RESULTS
On average, the GPA on gateway courses was 0.63 lower for students who are UR (N=1,800) compared to students who are not UR (N=3,200, R2=0.3, p<.05). This gap persisted after adjusting for measures of prior academic performance. However, logistic regression showed that students who are UR were nearly as likely students who are not UR to persist in completing at least four gateway courses. After accounting for the lower grades of students who are UR in their initial classes, UR students were more likely than non-UR students to complete four or more gateway courses (adjusted odds ratio = 1.25, p<.01).
EVALUATION PLAN OPTION
When evaluation is planned but not completed
• Most appropriate for curricular innovations where work has been done to develop the curriculum but it has not yet been implemented
• Indicate type of data to be collected• From whom• How analyzed• Timeline
• In a way, an extended methods section
DISCUSSION
Discussion describes what the results mean
• Interpretation of results
• General reaction to findings
• Statements of limitations
• Thoughts on study’s impact
• Proposals for next steps
Discussion pulls all information together
• Discussion must relate to problem statement/ hypothesis
• Interpretation of study’s findings• What do the results mean?• Do they answer the original hypotheses or problem statement?
Discussion: Common pitfalls• Assertions are made that are not supported by the data• Rehash of the results without interpretation:
“In this study we found that the intervention group performed significantly better than controls”
• Obvious limitations are ignored
Example 1
DISCUSSION
Scores in Gateway courses were statistically lower for UR students compared to white students. However, they completed four or more of these courses at the same rate. Our study clearly shows that academic support for UR students in college targeting gateway courses is crucial for increasing the diversity in medical schools.
Example 2
DISCUSSION
Despite academic challenges the minority students experience in college, many persist in their pre-health course requirements. Interventions at the college level to improve minority student performance in gateway courses are particularly important for increasing the diversity of medical schools
REVIEW CRITERIA
UCSF Education ShowcasePurpose: learly describe the project's goals and objectives. If you have a hypothesis, please state it here.Background: Describe how project was informed by knowledge of previous scholarly work or knowledge of educational principles.Methods: Describe the approaches used for this project. Who was studied or who will evaluate this project? What data were collected and how?Results/Evaluation Plan: Describe the findings of the study/evaluation including response rate, results, and statistical testing where appropriate.ORDescribe the outcomes that will tell you whether your curriculum is effective.
Discussion/Dissemination: State concisely the data-driven conclusion of your study/evaluation and how the results of the project are being (or will be) made available to colleagues.ORDescribe your plans for making the results of your project available to colleagues (talks, posters, publications, websites, etc.).Reflective Critique: Describe the steps taken to solicit feedback regarding your project and how it has been (or will be) modified based on the feedback.
WGEA: Oral Abstract Presentations & Posters
• :
Clarity of Purpose
Strength of Design
Reliability and Validity of Methods
Significance of Results/Outcomes
Soundness of Conclusions/Reflective Critique
Reviewers make a final recommendation
Definitely Include Highly rated in most/all categories; highly relevant
Probably Include Few concerning areas; can be improved before presenting. Relevant
Probably Exclude Several areas of concern, will require significant revision to remedy
Definitely Exclude Many areas of concern; fatal flaws present; needs substantial revision
REVIEWERS ETIQUETTE
Conforming to Submission Requirements
• As a reviewer, it is important to be familiar with the submission requirements
• Some submission systems reject an abstract that exceeds the word count• If the abstract seems long, check the word count and comment on
it in the review if excessive
General Rules• Always write comments
• Engage authors in feedback
• Your chance to convey:• Concerns/suggestions for improvement• Compliments• Reasons for ratings of abstract elements
• Consider writing comments FIRST, then use the rating form
Written Comments • Be respectful
• Be specific
• Be constructive
• Be focused
Example of Comments
• “The worst abstract I’ve ever read!”
• “Data presentation was pitiful.”
• “Wonderful study!”
• “The authors need to attend to several areas before resubmitting; these include:”
• “The authors need to present the actual data and statistical analyses.”
• “This was an excellent study for the following reasons:”
Poor Improved
Are you Available to Review? • Most reviews must happen within a discrete (and often
short) time period
• Look at your calendar before you agree to review
• If the timing is bad, politely DECLINE!
Comfortable with the Content?• Are you comfortable addressing the specific topics of
each submission?
• If you are not an appropriate reviewer, return it immediately so other reviewers can be found
REVIEWING EXERCISE
top related