proposal review board member training 2012 hgse student research conference
TRANSCRIPT
Proposal Review Board
Member Training
2012 HGSE Student Research Conference
WelcomeThank you for coming!!!
Grab some refreshments
Sign the attendance sheet that is going around
Look over the materials in your packet. You should have:
Proposal submission guidelines
2 scored sample proposals for training (1 empirical & 1 non-empirical)
2 proposals to score with blank rubrics (1 empirical, 1 non-empirical)
Sample comments for rubric
RoadmapFirst 20 minutes:
SRC History & Goals
PRB Responsibilities & Review Process
PRB Timeline
PRB Logistics
Next hour:
SRC Submission Guidelines & Scoring Rubric
Scoring Examples
SRC HistoryStarted 17 years ago by Susan Moore-Johnson and Kitty Bowles
Purpose to provide students with an opportunity to enter the research world in a non-threatening environment
Open to students from Harvard and beyond
Great experience for HGSE students in presenting and conference leadership
The Review Process
Each proposal will be blind-reviewed by two PRB members using the rubrics we will present today.
If the two reviews are discrepant (one recommends acceptance and the other recommends rejection), a third reviewer will be assigned to review the proposal to break the “tie.”
In late February, the PRB Chair will notify those who submitted proposals whether their proposal is accepted or rejected.
This year, we are not providing an opportunity for revision and re-submission
We may, however, make a recommendation regarding the presentation format (panel, roundtable, or poster)
PRB Member Responsibilities
Review proposals throughout the month of February. The exact number depends on the number of proposals received.
Review a few additional proposals in February if need arises (for instance to break a tie).
PRB Members & Submission
We encourage PRB members to also submit their own proposals to the SRC
We will not assign you to review your own proposal
Our system may be imperfect:If you get a proposal and you can tell who wrote it for whatever reason (it’s your own or you helped someone revise their proposal), please let us know by email ASAP
Timeline Now: Proposal submission open & ongoing
Jan. 27th: Deadline for applicants to submit proposals
Jan. 30th: Proposals go out to reviewers
Feb 13th: Reviews due back
Feb 16th – Feb 20th: Tiebreaker review
Feb 20th: Tiebreaker decisions due back
Feb. 24th: Decision emails sent to applicants
April 6th: 2012 HGSE Student Research Conference
PRB LogisticsSTEP 1: You will receive an email with a list of the proposals you are responsible for reviewing, which will be designated by a four-digit code.
STEP 2: You will then download your allocated proposals from the isite on a password protected tab called “Proposal Review Board Members.”
STEP 3: You will submit comments via an online poll similar to the one you filled out to RSVP to this training.
The Rubric
Complete one poll submission for each proposal you review
You don’t need to review them all in one sitting
You have a copy of the rubric in your packet
Submission Guidelines
Empirical v. non-empirical
Panel, roundtable & poster
Recommended outline
Empirical v. Non-empirical
Empirical studycollect & analyze
dataresearch questionqualitative or quantitative methods used to collect and analyze datafindings or hypothesized findings and conclusion based on analysis of data
Non-empirical paper
analytic papers, theoretical arguments, policy briefs and literature reviewsresearch probleman argument based more on logic and consideration of other perspectives and counter arguments conclusion based on thoughtful analysis
Presentation FormatsPanel
3-4 presenters with a common theme
15 minute presentation by each panelist followed by a question and feedback period
HGSE faculty member or doctoral moderator
Audience
Paper complete or in progress (at least preliminary results must be ready to present by April 6th), especially QPs, dissertation chapters, term projects, etc.
RoundtableSmall group of students with common theme
5-10 minute presentation by each participant followed by a question and feedback period
Especially suited to research projects in development
HGSE faculty member or doctoral moderator
Smaller audience (mostly participants)
PosterPresenters prepare a tri-fold poster for display
All posters displayed simultaneously in a conference space – presenter stands by poster
Poster session open to all conference attendees - succinctly describe work with students, faculty, and visitors who may or may not be familiar with the presentation topic
Most formal
Least formal
Recommended Outline1. RESEARCH PROBLEM:
A. Empirical study: State at least one research question
B. Non-empirical paper: State the research question, problem, or position.
2. CONCEPTUAL BACKGROUND: Explain the conceptual background for your research. Address why your research is important and include citations of published references.
3. METHODS/APPROACH:
A. Empirical study: Describe your research design and methods, including the plan for collecting and analyzing data to answer your research question
B. Non-empirical paper: Describe the argument, including how it will consider different perspectives and counter-arguments.
4. FINDINGS/CONCLUSIONS: State appropriate conclusions, tentative findings, or predictions.
5. IMPLICATIONS: Discuss the significance of your work for further educational research, theory, policy, or practice.
The RubricThis training is meant to familiarize you with the rubric for proposal evaluation, and to involve you in the process of increasing our reliability in using it.
We’ll go through each question on the rubric
We’ll go through an example of each element
We’ll then score proposals together to check our inter-rater reliability.
Overall Scoring Guidelines
We want to be as inclusive as possible
We want the conference to be well-attended
This is not a professional conference, but rather a student research conference
Rejection should be very rare – most likely you will not reject anyone (more on rejection later)
Goal is consistency across reviewersAlignment w/ PRB guidelines
Focus is substance rather than form
Quantitative Scoring
Criteria Evaluation Scale
3: Element meets criteria. It is fully articulated, convincing, and logical.
2: Element generally meets criteria. May have some small weaknesses.
1: Element may only be alluded to, or have serious weaknesses.
0: Element missing.
Majority 1 or lower – can be considered for rejection. Anything higher, recommend acceptance.
When in doubt…round up!
Qualitative Comments
Applicants will not see their quantitative scores (we use those internally), but they WILL receive these comments back, so please take care
Reviewer’s name will not be revealed to applicant
Goal = learning
If recommending acceptance: strengthening presentation
If recommending rejection: improving proposal for future submission
Suggested comments sheet in your packet – pull it out
Balance positive and constructive comments
Length of comments:
Length will vary based on quality of proposal
Up to you where to make your comments (for individual elements, overall, or both)
If you’re giving a 0 or 1 on a particular element, give a comment on that element
All comments should add up to at least a short paragraph
RejectionBefore you reject, think about:
Could comments help strengthen the presentation
Alternative formats:Roundtable (For Panel Requests)
If you do recommend rejection:Keep in mind the applicant will receive your comments
Please provide a clear rationale in your comments
Don’t be negative – please find something encouraging to say
Reach out to us if you would like help with phrasing etc.
Scoring: An Empirical
Proposal
Empirical Proposal Ex.
How does accountability impact school climate? A qualitative study of two urban middle schools
-Let’s go element by element - although the proposals themselves will not explicitly separate the elements (500 Word Max Proposal Format)
Empirical study
Conceptual backgroundDoes the proposal make connections to existing educational research, addressing why this research is important, and include citations?
3: According to the National Forum to Accelerate Middle-Grades Reform (2006), effective middle schools are academically rigorous, developmentally appropriate, and socially equitable. Comer's (2004) holistic view of child development supports the idea that schools need to balance academic rigor with a socially and emotionally nurturing environment. However, some education reformers claim that increased pressures to meet academic targets under No Child Left Behind, have caused many schools to focus on improving test scores at the expense of affective goals, negatively impacting students (Brigham et al, 2004; Lee, 2007).
2: According to the National Forum to Accelerate Middle-Grades Reform, effective middle schools are academically rigorous, developmentally appropriate, and socially equitable. Comer's holistic view of child development supports the idea that schools need to balance academic rigor with a socially and emotionally nurturing environment. However, some education reformers claim that increased pressures to meet the demands of educational accountability, such as those established by No Child Left Behind, have caused schools to focus too narrowly on teaching to the test.
1: According to the research literature that I have read, effective schools are academically rigorous, developmentally appropriate, and socially equitable. The holistic view of child development supports the idea that schools need to balance academic rigor with a socially and emotionally nurturing environment so that we can raise students who will be good citizens and there will be less bullying in schools. I do not think it is taking tests or having teachers teach to the test or pressuring young people to meet the demands of educational accountability which have been on the rise in America for many years.
Empirical study
Research questionDoes the proposal contain a clear research question?
3: In this mixed-methods pilot study I investigate how, if at all, the climate of two Boston urban middle schools is changing as a result of actions their school administrators are taking with the express purpose of achieving Annual Yearly Progress (AYP). I define “climate” in a broad sense, taking it to refer to characteristics associated with effective schools (Chenowith, 2009), including teacher and student perceptions of their school, the quality of interactions taking place within it, and the commitment of students to their school and learning.
2: This study examines the effect of current educational accountability reform, --i.e., No Child Left Behind—on the climate of urban middle schools. This study seeks to determine how the educational climate of two urban middle schools with similar demographics and size may be changing as a result of educational accountability. It also seeks to understand the decisions their school administrators may be making about diverse aspects of school administration that may have an effect on the climate of these schools, and how school administrators and teachers reflect on school climate, accountability, and how these two factors may be related.
1: This study looks at what is happening in American schools today. My study is seeking to determine how the educational climate of urban middle schools may be changing. It also seeks to understand school administrators’ decision making and school climate.
MethodsIs there a description of research methods that relate to the research question?
3: The sample consists of two Boston-area urban middle schools of similar size and student populations. One school meets AYP; the other has been labeled a “School In Need of Improvement.” I will evaluate changes in school climate by examining trends in the results of validated climate surveys (Shann, 1990; 1998) administered to approximately 90% of students at both schools over the past three consecutive years. I will conduct semi-structured interviews with teacher leaders (n=20), administrators (n=10), and school personnel (n=10) to assess their perceptions of how and why at the school climate has altered in recent years. I will also analyze school records related to attendance, behavior, budgets, professional development, and schedules to gain additional insights into the potential relationship between accountability and school climate at these schools and to triangulate my findings from the surveys and interviews (Wilkins, 2008).
2: The sample consists of two urban middle schools of similar demographics, but different accountability status. Several measures are used to document the climate of the schools. These measures include school climate surveys, interviews with some teachers, administrators at the two schools and some other school personnel, and analysis of school records. Surveys will be administered to measure student and teacher perceptions climate and factors they believe contribute to climate ir school, including the quality of relationships among school participants, administrative leadership, teacher autonomy, and student commitment to the school and learning. Interviews with teachers, administrators, and available school personnel will provide additional details of the climate and will also focus on how the climate may have changed. School records related to attendance, behavior, budgets, professional development, and schedules, as well as general observations of school activities, will also be analyzed to provide additional insight into accountability and school climate.
1: The sample consists of urban middle schools of similar demographics, but different accountability status. It is important that schools have similar demographics or our comparison will not be valid. Random sampling is also important. Several measures will be used to document the climate of the schools such as surveys, interviews, focus groups, and school records. We have already obtained permission slips from all participants. We will also be checking for threats to validity as well as internal consistency in our interviews.
Empirical study
Empirical study
Findings
3: I hypothesize that a greater emphasis on accountability is associated with some deterioration in school climate, as indicated by increases in student absenteeism among under-performing students and students generally commenting unfavorably about the school curriculum in their climate surveys. However, I believe that in some ways increased accountability may be associated with improvements in school climate, such as students appreciating their teachers’ efforts to help them pass their tests, particularly at the school that has met AYP.
2: I anticipate finding that a greater emphasis on accountability is associated with mainly negative changes in school climate. For example, students will say that they find school more boring and there will be an increase in absenteeism.
1: I anticipate finding, based on our study of two schools, that schools that have a greater emphasis on accountability always have a bad school climate.
Does the proposal indicate findings, predictions, or conclusions?
Empirical study
Implications:Does the proposal include implications for emerging educational research, theory, policy and/or practice?
3: Although my results will be tentative and based on a small sample size, my study could shed light on the ways in which accountability measures affect the climate for better or worse within middle schools. Furthermore, my results could indicate important areas for further research. For example, is climate affected differently in schools deemed to be failing rather than succeeding? If students and/or teachers express dissatisfaction with specific actions taken by school authorities to improve test scores, can alternative interventions be designed that are less detrimental to school climate?
2: The results of this study have implications for immediate actions that school leaders can take to improve school climate, while bolstering student academic outcomes. The results may also have implications for further research to be conducted by these and other researchers. Furthermore, I expect the findings to make an important contribution to theory about how schools work and how changes should be based on influences of the larger socio-cultural context. Administrators need to know what effect they are having on young people’s lives and whether students are coping with the stresses and strains of having to take high-stakes tests that will affect the Adequate Yearly Progress of their schools.
1: The results of this study have many important implications for research, theory, and practice which will be explored. We will also present specific ideas for immediate changes in school policy which should be advocated based on our expected findings.
Non-empirical Ex.
Language of Instruction Policy in Madagascar
Major Differences
Instead of a research question, non-empirical papers may have a research problem or a position statement.
Instead of methods, non-empirical papers may have an approach that includes a theoretical argument, with alternative perspectives or counter-arguments
Instead of findings or hypothesized findings non-empirical papers will have a conclusion.
Non-empirical study
Conceptual Background:Does the proposal make connections to existing educational research, addressing why this research is important, and include citations?
Research shows that children learn to read best when taught in the language they speak at home (International Reading Association, 2001; UNESCO, 2006). However, students in Malagasy schools who are taught in their mother tongues tend to score lower on the Progress in International Reading Literacy Study (PIRLS) exam than their counterparts in schools which use French and sometimes English, as languages of instruction (Boomer et al, 2007).
What rating would you give and why?3!
Existing researchConnected to RQ Citations!
Non-empirical study
Research Question:Does the proposal contain a clear research question, problem, or position statement?
Education in the mother tongue should not be of lesser quality than education in so-called prestige languages, such as English. What causes this to be the case? How can we improve the reading of Malagasy students? It is necessary to look more deeply at language of instruction policies. It is necessary both to provide culturally relevant and high quality educations in order to achieve international educational targets.
What rating would YOU give and why?
How would you phrase your comment(s)?
Compelling problem is presented, but would be stronger if “we” were defined:• Teachers?• Academics?• Policymakers?
The author IS presenting a question, but could more clearly state what she means by “this”
2!
Non-empirical study
Methods/Approach:Is there a description of the argument, including how it will consider different perspectives and counter-arguments?
One method is to look at language instruction policy from an international perspective. Here I will examine language policies in countries near to Madagascar.
What rating would you give and why? How might you phrase your
comment(s)?1 • No clear argument presented
• Counterarguments could be inferred, but they are not explicitly confronted
Which countries?How?
Non-empirical study
Findings/Conclusions:Does the proposal indicate findings, predictions, or conclusions?
I then propose a reform of the Malagasy language policy as it is my belief that it is not the mother tongue itself, but poor teaching practice, that causes Madagascar’s low test scores.
What rating would you give and why?
How might you phrase your comment(s)?
1Author is stating a belief rather than a finding
What kind of reform?
Non-empirical study
Implications:Does the proposal include implications for emerging educational research, theory, policy and/or practice?
The results of this study have many important implications for research, theory, and practice in international realms. Adoption of a language of instruction policy change in Madagascar will serve as an example for other countries.
What rating would you give and why?
Studies like this one are vitally important because a successful policy in one context may serve to inform decision-makers facing similar problems elsewhere. In fact, very few system-wide educational decisions carry the same importance or are as difficult as the choice of a language of instruction. Choosing one language over another to educate a society’s youth is a decision that carries substantial, political, social, and economic weight. This example could help to serve as a prototype for policies that protect the increasing number of economic migrants, who can be considered bereft of many of their rights since they fall between belonging in at least two nations, just like many of the marginalized ethnolinguistic groups of Madagascar.
1
2!
VagueWhat kind of countries?What kind of policy change?What is the importance of language instruction policy?
Putting it all together
Now let’s score an empirical study: “Fact or Fiction?”: Children’s Ontological Beliefs in
Historical Figures
Putting it all together
Scoring a non-empirical paper
EXAMPLE NON-EMPIRICAL PROPOSAL
Format Submitted: Roundtable
The purpose of this paper is to address the application of Hostetler’s framework of educational well-being to the globalization phenomenon occurring at a rapid pace in higher education. Globalization differs from internationalization in that it is marked by fluid borders, the concept of increased interconnections between nation states, and the idea that there is a global knowledge that is achievable through shared research and development. As the globalization trend has grown, a distinct lack of clarity of aims and intentions has presented itself. There is a lack of definition as to not only what makes an institution global, but also whether the institution is successful in its mission in the host country. While there is an imperative for globalization in higher education due to the framework of the global economy and mandate for human rights throughout the world, the outcomes of such efforts have proved themselves to be mixed for the hosting countries. Some African, South American, and developing Asian countries have been susceptible to heavy influence from Western interests (e.g., international institutions and corporations) that have the competitive advantage in setting the agenda for the institutions established. The paper discusses the effects of the transformation from the internationalization movement to one of globalization, and then goes on to address many outcomes of this development over the last two decades. Specific examples focused on are the UK’s movement into the Middle East, Australia’s institutes in the Pacific Rim, and American developments in India. While there have been attempts at regulation through standards provided by local governments (e.g., the UK’s QAA and the EU’s Lisbon Strategy) as well as ranking systems (The Times Higher Education Supplement Rankings of Universities and Shanghai Jiao Tong University), efforts at regulation by global organizations, such at The World Bank, UNESCO, and the Organization for Economic Co-operation and Development have proven themselves to be largely ineffective. The World Trade Organization’s framework for governance in the General Agreement on Trade in Services provides for economic equality but not for the well-being of the institution and its participants in the setting of the host country. Thus the authors of the paper propose to use Hostetler’s framework of educational well-being (2005) to examine the current state of globalization as well as propose a series of ethical guidelines to ensure that the interests of all nations and cultures involved are continuously considered as globalization transforms higher education.