integrating alternative assessment in a project … · web viewtitle integrating alternative...
TRANSCRIPT
Integrating Alternative Assessment in a Project-Based Learning Course
for Pre-service Science and Technology Teachers .
MOTI FRANK, Technion – Israel Institute of Technology
ABIGAIL BARZILAI, Technion – Israel Institute of Technology
Paper presented at the Learning Communities and Assessment Cultures Conference organised by the EARLI Special Interest Group on Assessment and Evaluation,
University of Northumbria, 28-30 August 2002
ABSTRACT This paper deals with a required methods course, based on the
national curriculum of science and technology for junior high schools. The course
participants are pre-service teachers who study towards a B.Sc. degree in Education
in Science and Technology parallel to their studies in one of the faculties of Sciences
or Engineering. Working in small teams the students carry out a project in a Project-
Based Learning (PBL) environment. The final outcomes of the project are group and
individual written reports, a portfolio, a multimedia presentation and a physical
model. The research question was: what implementation issues and processes do
higher education students encounter in a Project-Based Learning environment which
involves an alternative assessment approach? Qualitative and quantitative tools for
collecting data included ‘the participant as observer’ observations at the classroom,
semi-structured interviews with students, questionnaire, and analysis of students’
reports and products. The findings relate to the benefits of PBL from the students’
point of view, the challenges students experienced and perceived while conducting
their projects, the benefits students may gain from formative assessment, and the
difficulties located by the course teachers throughout the assessment meetings.
Introduction
A required methods course, based on the national curriculum of science and
technology in junior high schools, has been developed in the Department of Education
in Technology and Science. The course participants are pre-service teachers who
study towards a B.Sc. degree in Education in Science and Technology parallel to their
studies in one of the faculties of Sciences or Engineering. The learning environment
of the course is based on a constructivist approach, and designed to enable the
students to construct their knowledge through active learning, and through interaction
with staff and colleagues. The teachers act as facilitators and mentors. The assessment
in the course is also derived from the constructivist approach and is based on
formative assessment of students’ performance in microteaching, active learning in
the National Museum of Science and Technology, group assignments, and
interdisciplinary team projects.
This paper focuses on an approach that is implemented to assess the students
while carrying out the project in a Project-Based Learning (PBL) environment.
Students are required to design a system based on scientific, technological, social and
environmental principles. To emphasize technological, and not merely scientific
literacy, the unique quality of the project is that the starting point is the technological
requirements and needs rather than a research question. The students first define what
would be required of the technological system, including the tasks to be performed,
and determination of the system outcomes. They investigate alternatives for
implementation, collect and analyze data through a process of investigation and
collaboration, and conduct a feasibility study, after which they designed the system,
using a top-down approach.
During the course students learn how to use project-based learning in their
classrooms. The course covers the theoretical background of PBL and some field
research examples which describe the use of PBL as well as the strategy for
conducting a student-project: where to begin, how to coach students through a project,
and how to assess student products.
The final outcomes of the project are group and individual written reports, a
portfolio, a multimedia presentation in classroom in front of the course colleagues and
staff and a physical artifact, which can assist the secondary school teacher while
demonstrating a scientific and/or technological principle underlying the system.
We believe that alternative assessment should take into consideration intangible
parts of the project and not emphasize the final products or presentations only. The
real learning is often in the doing or in the process leading up to the product. Students
presenting a product that fails to work may still have learned the central principles,
especially if they can describe why their product does not work and what is needed to
correct it (Buck, 1999). Therefore, in our course, the project grade is based on the
assessment of tangible (about one third out of the project grade) and intangible
outcomes .
Following are examples of projects performed by students: a car driven by solar
energy, a water desalination system, a remote cardiologic testing system, an
automated watering system, a hot air balloon system, and an automated purification
system for aquarium water.
Project-Based Learning
What is the project-based learning approach? According to Krajcik, Czerniak and
Berger (1999), this approach engages learners in exploring important and meaningful
questions through a process of investigation and collaboration. Students ask questions,
make predictions, design investigations, collect and analyze data, use technology,
make products, and share ideas. According to Buck Institute of Education (1999),
PBL is an innovative model for teaching and learning. It focuses on the central
concepts and principles of a discipline, involves students in problem-solving
investigations and other meaningful tasks, allows students to work autonomously to
construct their own knowledge, and culminates in realistic products .
Thomas (2000) also tries to define this approach, and emphasizes that in the PBL
environment students are, in fact, investigating solutions to a problem. They build
their own knowledge by active learning, interacting with the environment as
suggested by the constructivist approach, working independently or collaborating in
teams, while the teacher directs and guides, and they make a real product.
We shall now discuss the advantages of the PBL approach for teaching and/or
learning, firstly from the students’ point of view, and then as concerns the instructor.
Green (1998) notes that learning by means of a project is likely to increase motivation
and give the students a sense of satisfaction. It is also helps to develop long-term
learning skills. Krajcik, Czerniak and Berger (1999) suggest four benefits for the
student. Firstly, learners develop deep, integrated understanding of content and
process. Secondly, students learn to work together to solve problems. Collaboration
involves sharing ideas to find answers to questions. In order to succeed in the real
world, students need to know how to work with people from different backgrounds.
Thirdly, this approach promotes responsibility and independent learning. As a final
benefit, this approach actively engages students in various types of tasks, thereby
meeting the learning needs of many different students.
Orevi and Danon (1999) also list the advantages of PBL from the students’ point
of view - it develops collecting and presenting data skills, develops thinking skills,
suits personal learning styles, enhances motivation, and develops independent
learners.
So far we have discussed the advantages for the student. Krajcik, Czerniak and
Berger (1999) also suggest three possible advantages for the teacher. Firstly, the
teacher may find the work enjoyable, interesting and motivating, since teaching will
vary every year as he/she will be exploring new projects with each new group of
students. Secondly, in project-based teaching, the teacher continually receives new
ideas, thus becoming a ‘lifelong learner’. Thirdly, classroom management is
simplified because when students are interested and involved, they are likely to cause
fewer disciplinary problems.
What has research discovered about the usefulness of the PBL approach? In a
research described by Shepherd (1998), it was found that students who applied the
PBL approach statistically received significantly higher grades on the Critical
Thinking Test than did those in the comparative group who had studied in the
traditional manner. The PBL students also demonstrated greater self-confidence and
improved learning abilities.
Interestingly, Rosenfeld and Rosenfeld (1999) found that students with low
academic records who studied in the conventional framework did better in courses
based on PBL, whereas those with higher grades in regular studies had lower
achievement when PBL methods were applied. Based on their findings, the
researchers suggest that teaching styles and learning environments be adapted to the
student’s learning mode. Low academic grades do not necessarily demonstrate lack of
ability, but rather the unsuitability of the pedagogic system. They advise that students
be exposed to PBL in order to give those who fail a chance of doing better, and to
encourage those with high academic achievements in subjects taught traditionally to
develop additional expertise.
The teaching role in PBL is changed. The teacher is no longer the expert lecturer,
facts provider, and director of instruction but rather a resource provider, learning
environment shaper, how-to-learn teacher, advisor, tutor and colleague (Buck, 1999).
Teaching by means of PBL presents several challenges. These include: teachers’
content knowledge, students’ lack of experience in this new approach and their
preference for traditional-structured approach; their preference for learning
environment which require less effort on their part; and problems arising from time
stress. Students struggling with ambiguity, complexity, and unpredictability and are
liable to sense frustration in an environment of uncertainty, where they have no notion
of how to begin or in which manner to proceed.
The system is rather time-consuming and requires that the teacher make a lot of
efforts over a long period of time. Certain teachers may feel that they don’t have
sufficient time to meet the curriculum guidelines. Class management, in which the
students have the freedom to talk together, is often more difficult.
Teachers often feel a need to direct lessons to insure that students get the “right”
information. Teachers frequently give students too much independence without
structuring the situation, or providing feedback. Sometimes, teachers have difficulty
incorporating technology into the classroom. Other problems are discussed by
Krajcik, Czerniak and Berger (1999, pp. 322-328) and by Buck Institute of Education
(1999, Potential Problems section).
To sum up, we will present the main advantages of the PBL method (PBL that
based on identifying needs and designing system to meet that needs rather than PBL
that based on answering research question). Students are engaged in active learning
and gain interdisciplinary knowledge while working in a real-world context. Students
develop deep and integrated understanding of content and process and focus in depth
on central ideas and salient issues. Since students deal with relevant issues, their
motivation is increased. PBL promotes responsibility and independent learning, offers
multiple ways for students to participate and to demonstrate their knowledge, and can
be matched to the various learning styles of students.
Through PBL the students are familiarized with the synthesis processes (not just
the analysis processes), experiment in teamwork, become familiar with the design
procedures and the underlying principles of systems engineering. The students learn
to apply the important engineering principle - how to design to an optimum (not
necessarily to the maximum).
The learning environment promotes the development of inquiry skills, problem
solving skills, and information skills. Students develop a capacity for systems
thinking, acquire lab experience, and gain a higher level of cognitive skills (such as
asking questions) and affective outcomes (such as curiosity and skepticism). Finally,
the awareness of students to Scientific-Technological-Environmental-Social aspects is
increased, academic achievement may be improved and students learn to simulate the
professional work of the scholar, researcher, engineer, reporter, planner, manager, and
other practitioners.
Alternative Assessment
The traditional assessment in most courses of the higher education is mainly based
on paper and pencil tests. The grade in a given subject is mainly based on a final test.
Usually these tests measure how well the student can repeat the declarative
knowledge transferred from the teacher to the passive student. In our course the
traditional assessment methods were found less appropriate for measuring the kind of
understanding the students acquired in PBL.
According to Krajcik, Czerniak and Berger (1999, p. 207), traditional questions
would fail to assess the multimode of ideas students learned. They would not measure
students’ ability to work as a team. They would not show how students could apply
their knowledge and skills to everyday life. They would not show that they could
design and plan investigations .
Therefore we decided to apply an alternative assessment in our course.
Assessment of students’ performance in the project was based on meetings and
discussions between the course staff and each team during the semester, observations
of the students’ work in classroom, group report and portfolio, personal reflective
reports, and an exhibition at the end of the course - posters, multimedia presentations,
and the artifacts. A rubric was also developed and validated for this assessment.
The following table presents the characteristics of alternative assessment (Buck,
1999; Krajcik, Czerniak and Berger, 1999) and the assessment strategies implemented
by us in the course:
TABLE 1. The characteristics of alternative assessment, and the strategy of the
assessment implemented in the course.
No. Alternative assessment characteristics Assessment strategies implemented in the course
1. The assessment is formative. It relates to the learning processes not just to the final result as a
continuous process embedded in instruction.
Throughout the course students were given feedbacks regarding their progress as an integral part of the course. The criteria for assessment
were given to the students in advance.2 It encourages teachers to become more
reflective practitioners. It helps them justify the grades, plan instruction, and identify the
difficulties and misconceptions.
The course design was dynamic. The instruction has been altered a number of times according to the progress in practice. The assessment criteria were changed following interaction with the
students.3 It is based on a number of diverse information
sources, not just on tests.The assessment was based on various forms, not just on a final test (see details in the Method
section).4 Assessing of knowledge, understanding, skills,
attitudes, values, motivation, higher-level cognitive skills and affective outcomes in a real
life context .
Assessing processes and products enabled to assess diverse cognitive and affective abilities.
5 It indicates that the student is responsible for the learning process. Engaging students in the assessment process helps them to be self-
reflective and self-regulated learners .
To help students monitor their progress they were required to report their progress and plans, keep journals and submit a reflective report at the end
of the course.
6The focus is on student improvement rather than on comparison with others. It may reduce the
level of stress and anxieties.
The assessment was based on the progress and outcomes of each team rather than on a final
standardized test .7 Teachers come to know their students better. Regular meetings with course staff were held in
The teacher can help each student according to individual needs.
order to identify difficulties and offer individual assistance.
8 The assessment is qualitative and verbal and not just quantitative.
An analytical rubric was used. Students were given verbal comments .
In light of the above characteristics, Krajcik, Czerniak and Berger (1999) claim
that alternative assessment is more valid and reliable, more closely matches today’s
educational goals, better accommodates cultural diversity, is consistent with cognitive
learning theory, and measures deep understanding.
Method
Research Design
This paper deals with a required methods course, based on the national curriculum
of science and technology for junior high schools. The research question was: what
implementation issues and processes do higher education students encounter in a PBL
environment which involves an alternative assessment approach? The purpose of the
current study was to learn about ideas (cognitive aspects), emotions (affective
aspects), difficulties, and behavior (behavioral aspect) of the students in the above
course.
The study is based, mainly, on qualitative data analysis. The merit of a qualitative
study lies in its internal validity, not in its generalization. Therefore, we do not
recommend making generalizations on the basis of the case study described here.
However, the findings and implementation tips presented in this paper may serve as a
basis for a follow up empirical/quantitative study based on a random sample and
inferential data analysis .
The strategies adopted in the study served to reinforce reliability, inner validity,
and external validity. This was carried out by cross-referencing sources and
triangulation. Denzin (1988) outlines four levels of triangulation. The first is data
triangulation that includes time, space and persons. The second level is investigator
triangulation - using several observers. The third is methodological triangulation -
using more than one research method, and the fourth level is theory triangulation -
creating a broad theoretical framework .
The data analysis strategy used was ‘content analysis’: definition of analysis units,
setting categories (outstanding repeated elements), and examination of frequencies.
The study was conducted by two researchers. Qualitative and quantitative tools for
collecting data included ‘the participant as observer’ observations (Adler & Adler,
1994) at the classroom, semi-structured interviews (Spradley, 1979; Mishler, 1986)
with students, questionnaire, and analysis of students’ reports and products. Data was
collected at different times and stages during the course.
The Subjects
The course participants were pre-service teachers who studied towards a B.Sc.
degree in the department of Education in Science and Technology parallel to their
studies in one of the faculties of Sciences or Engineering. The duration of the course
is a fourteen-week semester. Every weekly class included a one-hour lecture, two
hours microteaching and three hours team project.
Multiple Forms of Assessment – Tools for Assessing Processes and Products
The principle of triangulation was applied also to the assessment process. Multiple
forms of assessment were used - multiple formats, multiple units, and multiple
assessors (Gonella, 2001). Multiple formats involve group reports, portfolios,
observations, media product, and a physical model. Multiple units refer to self-
assessment, group (peer) assessment, and whole class assessment. Multiple assessors
were involved – two lecturers, the TA, and the students .
The project grade was 55% out of the final grade of the course: 10% - physical
model, 5% - Power Point presentation, 10% - meetings with the course staff, 20% -
group report, and 10% - personal (reflective) report. Several rubrics were developed
for assessing the above outcomes: analytical rubric for assessing the group report, and
holistic rubrics (Birenbaum, 1997; CPS, 2000) for assessing the personal report, the
PPt presentation, the physical model, and the meetings’ documentation. Using the
rubrics enabled the instructors and the students to monitor the students’ progress and
guided them through the project .
Analytical rubric for assessing the group report
As indicated the students were required to submit a group report describing and
explaining their work. They were directed to divide the report to seven chapters.
Chapter 1 refers to the needs and the system requirements, chapter 2 presents
literature review, chapter 3 explains the design process, chapter 4 explains the block
diagram of the optimal system and how the system works, chapter 5 describes the
physical model, chapter 6 includes the references list, and chapter 7 contains the
transcriptions of the meetings. An analytical rubric was developed to assess the group
report. The development was mainly based on rubrics for PBL that were found in the
literature. Some of the dimensions and criteria were developed specifically for this
study, since suitable items were not found in the literature. In the first phase of the
rubric designing the dimensions were determined according to the project stages. In
the second phase the criteria and the benchmark (three performance levels) were set
for each dimension (see appendix A).
The rubric was content validated by three experts in science and technology
education. First, analysis of the dimensions list was done. Then, the matching of each
criterion to its dimension was examined, and finally the level of clarity and the
detailing of each criteria and benchmark were reviewed .
Two lecturers assessed the group report – one is an expert in Biology teaching and
the other is an expert in Technology teaching. The Spearman Correlation between the
scores of the two experts (inter-judgment reliability) was highly positive – 0.854 .**
The questionnaire
The questionnaire was designed to study the students' attitudes towards PBL
environment and towards the alternative assessment strategies implemented in the
course. The items of the questionnaire were based on characteristics of PBL (Krajcik,
Czerniak and Berger, 1999; Buck, 1999).
The first person singular was used intentionally, to obtain each respondent’s
perception of his/her own subjective experiences in the classroom, rather than
perceptions of the experiences of the others (Taylor & Fraser, 1991). Placing students
in the center of the instruction process and the willingness to understand the student’s
personal point of view is a basic principle in the constructivist education .
The questionnaire scale of measurement was that of Likert (five extents of
agreement from strongly agree to strongly disagree) and it contains 27 items. The
questionnaire was content validated by three experts. The categories’ list and the
matching of each item to the appropriate category were examined. The students
completed the questionnaire at the end of the course. Appendix B presents the format
of the questionnaire.
Personal report
Each student was required to submit a personal report. The students were
instructed to refer to the following points: a general description of the processes
occurring in the group, what happened to the student while working with the team; the
construction and formation stages of the team; what was the unique contribution of
the student to the team; what did the others in the team contribute to the student.
Furthermore, the students were requested to answer two questions: in light of your
experience in the course, what are the advantages and disadvantages of learning
through projects? In light of your experience – as a junior high school teacher, will
you incorporate such kind of teaching and assessment?
Content analysis of the personal reports was performed by the two researchers.
Data found in the reports can support, broaden, explain and increase the validity of the
findings in the other research tools.
Major Findings and Discussion
Analysis of the raw data collected from group reports, personal reports,
questionnaire, interviews, observations, and students’ scores (given by the teachers
based on the rubrics) revealed several frequently repeated items. These were used as
categories according to which the collected data were classified. The items came
under four categories – those related to the project-based learning environment; and
those related to the alternative assessment techniques used in the course. The first
category relates to the benefits of PBL from the students’ point of view. The second
considers the challenges students experienced and perceived while conducting their
projects. The third category refers to the benefits students may gain from the
formative assessment. The fourth relates to the type of difficulties located by the
course teachers throughout the assessment meetings.
The Benefits of PBL
While working on the projects, the students were exposed to some advantages of
PBL. For instance, many students noted that in the course of the project they acquired
interdisciplinary knowledge, which they believe is one of PBL’s advantages:
We were required to cope with issues from the various disciplines –
Biology, Chemistry and Technology. Each of us studies a specific
subject, and the need to perform a joint project forced us to study
subjects from other disciplines. I understood that in this method the
student can acquire knowledge from other domains.
Other students marked the mutual inculcation between the members of the team as
a means of acquiring interdisciplinary knowledge:
Each of us contributed his share. We were exposed to various work
methods of our own … we were exposed to a variety of ideas … we
learnt from one another … I learnt from my colleagues unique
subjects .
Or:
There existed among us in the team a readiness to share information
and ideas … each of the team member arrived from a different domain,
and through our mutual work, a variety of ideas and scientific aspects
were raised … I majored in Chemistry, whereas the other members of
the team majored in Agricultural and Civil Engineering…. I helped
them understand concepts in chemistry, which were totally new to
them.
Another student emphasized the importance of work within a team, as a means of
coping with a wide variety of issues, and a large amount of information:
In teamwork, one is able to encompass a larger amount of contents, as
compared with learning by oneself. Each team member learnt a
portion of the sum required, and subsequently taught the others. The
integration among us was very good.
The students’ perception of the interdisciplinary knowledge acquisition as an
advantage of PBL was also manifested in their answers to the questionnaire. Based on
their experience in the course, 95% of the students maintained that PBL allowed them
to acquire knowledge and enhance their understanding in interdisciplinary subjects.
Interdisciplinary knowledge acquisition develops the capacity for systems thinking
and increases the awareness of social and environmental aspects .
Indeed, according to Krajcik, Czerniak and Berger (1999) and Buck (1999),
students in PBL are engaged in active learning and gain interdisciplinary knowledge
while working in a real-world context. Students develop deep and integrated
understanding of content and process and focus in depth on central ideas and salient
issues. The learning environment fosters the acquisition of inquiry skills, problem
solving skills, and information skills .
Another advantage identified by a large number of students, is that in the PBL
approach, the responsibility for the learning lies on the student. And thus, for
example, one of the students mentioned the social pressure within the team as a factor
that stimulated her to strive harder:
In the course of the teamwork, I realized the magnitude of my
responsibility. I undertook a task and my team friends expected me to
perform it in the best way that I could. I realized that I could not let
them down. If I should “screw up” the quality of the teamwork could
be adversely affected, and we would all stand to lose.
Another student expressed a similar idea, only in slightly different words, while
mentioning the relevance as a learning motivation rising factor:
In the work allocation among us I was in charge of a certain issue. I
promptly understood that I had to master that issue so that I could
later teach it to the other members in the team. The subject I had to
learn was selected by me, since it whetted my interest. It was
something I had encountered in a different course at the Technion. It
was really “fun” to dwell on it.
And indeed, in answering the questionnaire, about 85% of the students (16 out of
19) responded to the questions that related to the issue of raising their learning
motivation and responsibility in PBL with the answer: “to a large/very large extent”.
About 90% of the students (17 out of 19) agreed “to a large/very large extent” that
PBL allowed them to be engaged in everyday relevant issues.
By viewing the literature, it seems that many researchers believe that in PBL the
student’s responsibility for learning is higher compared with the traditional learning
methods, and that in certain conditions, the students’ motivation for learning is
increased. Thus, for example, according to Buck Institute of Education (1999), PBL
allows students to work autonomously to construct their own knowledge. Green
(1998) notes that learning in PBL is likely to increase motivation and give the
students a sense of satisfaction. It can also help to develop long-term learning skills.
Krajcik, Czerniak and Berger (1999) suggest that PBL promotes responsibility and
develops independent learners. Orevi and Danon (1999) indicate that since students
deal with relevant issues their motivation is increased .
The third advantage identified by the students refers to acquiring communication
skills and methods of decision making within a team. A number of students
emphasized an improvement in their discourse skills. For example:
In the course of our discussions we learnt to listen to one another, to
conduct a conversation and offer and receive criticism in a
constructive manner.
Other students emphasized the challenge in making team decisions:
… Usually, following a long discussion, we needed to make a decision
in order to pursue a matter. All our decisions were made by consensus,
and there was no need for voting over them.
Other students emphasized the social aspect and the improvement of the
interaction with the teachers of the course;
Throughout the work, the social ties among us improved. Furthermore,
the interaction and ties with the teachers of the course were better than
in other courses at the Technion.
From the students’ answers to the questionnaire, it seemed clear that the majority,
90%, believed that in the environment of PBL one can improve the communication
skills and the ways of obtaining decisions within a team.
Many researches have shown that interpersonal skills can be learnt and acquired
gradually and developmentally (see for instance: Hertz-Lazarowitz, 1990). One
example of the communication skills expected of the interlocutor is active listening. It
is not sufficient to tell a student that he should listen. In order for a student to
comprehend what is the behavior expected of him, it is advisable to break up the skills
into observable and clear behaviors. It is also advisable to discuss it with the students
and involve them in identifying the skills, i.e., the listening skill may be broken into
behavior skills (the body is turned in the direction of the speaker, looking into the
eyes of the speaker, showing interest, responding with head gestures) and into verbal
response skills (could you please repeat, have I heard correctly that …, did you mean
that …, I understand that you claim …). One of the training lessons in the course was
devoted to exercises of the above kind.
Another lesson in the course was devoted to methods of decision-making in a
team. Various techniques were viewed, such as making decisions by majority
(voting), making decisions by consensus, making decisions by brain storming, making
a decision by the chairman after consultation, the Nominal Group Technique method
and the Delphi method (Dyer, 1987; Parker, 1990; Jewell & Reitz, 1984). By
analyzing the students’ reports, it emerges that with all the teams, the decisions were
made by consensus. A possible explanation to this is that time pressure during the
course did not enable the students to apply more time-consuming methods. Perhaps
there were students who preferred not to insist on imposing their opinion, provided
they would meet their scheduled dates. This point will have to be taken into
consideration, while planning the next course.
Another advantage identified by some of the students relates to the enhancement
of one’s self-esteem in PBL environment. One of the students attests about himself:
At first I had many apprehensions, but the more we progressed in our
work and were able to meet more and more tasks, the more did my self
confidence increase, and I began to believe in our ability to complete
the project and meet all the course requirements.
Whereas another student attests about a team-mate:
In the beginning M was the weak link in the team. We demanded that
she be a more active participant, and the more we made progress, the
more active and creative she became. Suddenly she started to raise
many new ideas … her self-esteem rose…
Researchers of the PBL method relate to the issue of the self-esteem in an indirect
manner. For example, Krajcik, Czerniak and Berger (1999) indicate that PBL offers
multiple ways for students to participate and to demonstrate their knowledge, that can
be matched to the various learning styles of students.
Self esteem is a well-organized system of qualities and values a person attributes
to him/herself. It is affected, among others, by the discrepancy between the
expectations that the person consolidates about him/herself and his/her achievements.
The self-esteem evolves within reciprocal relations between man and his
surroundings. Many studies tested the correlation between the self esteem of the
student and variables such as academic achievements, the need for achievement, the
level of aspiration, the level of anxiety, curiosity, locus of control and teachers’
assessment. In some of these studies, a positive correlation was found between the
self esteem and success and receiving positive assessment (Battle, 1991). Therefore, it
is likely that PBL environment, which is based on formative assessment and
continuous support, as the case was with us, an increase of certain students’ self
esteem was found.
And finally, the fifth advantage identified by the students is that in the
environment of PBL learning is active and experiential. One student articulated it very
eloquently:
While deliberating on a certain issue and searching for information
sources on that issue, I was able to come to a conclusion of my own. It
was a great experience, and I am sure that I will never forget this
material.
Another student emphasized the intensive activity of searching and sorting
relevant interdisciplinary information:
After extensive reviewing of dozens of Internet sites, we put a lot of
effort into sorting the data and select the sites that deal with scientific,
technological and social aspects related to a car that operates by
means of solar energy.
And indeed, about 80% of our students (15 out of 19) very strongly believe that
learning in PBL environment is active and experiential learning (as opposed to other
courses in the faculties of Sciences and Engineering at the Technion, where they
solely have a passive role).
Active learning is a principal characteristic of PBL environment that is based on
the constructivism approach to teaching. Constructivism is a theory concerning
learning and knowledge, which suggests that the human being is an active learner who
constructs his/her knowledge on experience and on his/her efforts to give meaning to
that experience. In the course presented here, students were, in fact, required to
construct their knowledge by means of active experience and learning in the form of
‘trial and error’.
As constructivist researchers, we hope that through their experiencing, our
students learnt to understand the advantages and challenges of applying the PBL
approach to teaching. We hope that now, that they have experienced it by themselves,
they will be able to integrate this approach into their teaching in the future.
Challenges of Students in PBL Environment
According to Hertz-Lazarowitz (1990) the interpersonal skills required of a team
member can be divided into three groups. The first group is the communication skill –
required behaviors from the message transmitter and required behaviors from the
message receiver. The second group relates to the vote-of-confidence skills. The third
group refers to the negotiation skills and coping with conflicts within the team. These
skills can be improved through training and practice.
The list of the interpersonal skills required of a team member was discussed with
the students in one of the course lessons, and was even practiced to some extent. From
analysis of the personal reports of the students, we have learnt that some of them were
indeed exposed to situations of conflicts in the teamwork. Following are a number of
examples:
While working with the team, some social problems that affected the
work process arose.One weak link in the team can severely harm the whole teamwork. We
encountered such a weak link.
Not everybody knows how to work or is interested in working in a
team. There was one member within us who preferred to work on his
own.In the course of our work, struggles of forces were formed, yet the
more we progressed the better we learnt how to cope with them.I learnt to adapt my work habits to those chosen by the team.
We hope that experiencing conflict situations during the work with the team, in
conjunction with referring to resources where exercises of communication skills
improvement can be found, will allow our students to better prepare their future pupils
for working in a team.
The second challenge located by the students relates to the large amount of time
which the teacher invests while implementing the PBL method in his/her teaching, as
compared to the time and effort required for teaching the same contents over and over
again in a traditional teaching method. So, for example, though we met with the
students a great number of times (in relation to what we do in our regular courses),
there were students that claimed that it was not enough. These students concluded
that:
The PBL approach requires that the teacher continuously follow up the
activities of each one of the group members and their degree of
progress.
Another student expressed the same idea only she phrased it differently:
I don’t think I will be able to integrate such a teaching method in the
first years of my work as a teacher. It seems that one needs a great
deal of time and effort, and I will probably be busy with many more
basic issues.
And one student, concluding that in order to implement the PBL method the
teacher should have a broad interdisciplinary knowledge:
This method requires of the teacher a great deal of effort and a very
wide interdisciplinary knowledge.
Another challenge faced by the students is the need to cope with new contents in a
learning environment which is neither structured nor organized in advance:
In the beginning I was frustrated, as I had no idea where to begin… In the beginning we all wasted a lot of time, because we really didn’t
know what to do, and at the end we were encountered by the most
severe time stress.I prefer the traditional-organized method to this one.
According to Krajcik, Czerniak & Berger (1999), teaching by means of PBL
presents several challenges. One of these challenges is: “students’ lack of experience
in this new approach and their preference for traditional-structured approach.”
The students’ attitude towards alternative assessment
The formative assessment strategy that we applied in the course served, among
others, as a means of locating difficulties of students and choice of intervention means
in order to assist the students who face difficulties. Each group met with one of the
course teachers for one formal meeting once every three weeks (sometimes there were
additional informal meetings). The meetings had a constant format: at the beginning
the students reported about what they had performed since the previous meeting.
Subsequently, the hardships encountered by the students were discussed, as well as
the ways of coping with those hardships. Finally, the students presented a work plan
for the next three weeks. The interaction between the teacher and the team also
included a feedback and assessment regarding the degree of the progress made. The
students were required to arrive at the meetings with a written report, specifying what
they had performed, the difficulties they had encountered, and a work plan. Each team
kept the reports and the summary of the meetings with the teachers in a group
portfolio. We wanted to know to what extent these meetings helped the students. For
this purpose a number of students were interviewed, and a questionnaire was
submitted to all the students at the end of the course.
The summary of the students’ answers to the questionnaire demonstrates that
about 80% feel that giving the criteria for assessing the students’ achievements at the
beginning of the course, helped in the implementation of the project. The majority of
the students maintain that continuous assessment throughout the course, not just at its
end, advances the learning for the following reasons: it assisted the student to evaluate
the degree of progress (95%), cope with difficulties (80%), locate the points which
require correction or improvement (90%), test additional aspects related to the project
(80%) and cope with conflicts among the members of the team (80%). The students
indicated that the feedback they received in the meetings with the course teachers,
assisted their learning in any of the following project stages: defining the problem /
the technological need (95%), formulating the system objectives (85%), locating the
information sources (80%), presenting and testing alternative solutions (80%),
designing and constructing the system (80%) and presenting the work (95%).
Similar findings also emerged from analysis of the data collected during the
interviews. Part of the students related in the interviews to additional issues. For
instance, there were some who indicated the contribution of the tri-weekly meetings,
and the feedback / assessment they received one from the other to raise the level of
the performance:
In addition to the feedback we received from the teacher, we were also
requested to give feedback and evaluation to one another. I believe
that the criticism was fair and constructive, and helped to raise the
level of our performance.
Other students emphasized that the meetings allowed the course teachers to locate
the students who had difficulties, and assist them:
The meetings allowed the teacher to become more familiar with us and
identify the strong and weak points of each of us. Anyone who faced
difficulties received personal tutoring.
A number of students pointed at the potential difficulties in alternative assessment.
For example, two teams stated the fact that the course teachers had difficulties in
assessing the personal contribution of each student:
There was a problem for the TA to monitor each student individually,
since the emphasis in the meeting was on the group as a whole.
Another student indicated that:
The course teachers had a hard time assessing the contribution of each
member of the group.
Another difficulty identified by the students relates to the vast amount of time and
effort the teacher would have to invest in order to implement the techniques of the
formative assessment. All the students indicated that continuous follow-up of the
teacher was required to ascertain the progress of the whole team and each student. For
example:
Such work requires a continuous follow-up of the teacher to check the
activity of the team members and their progress …
A continuous follow-up of the teacher is required to check the
development of the project at each stage.
I suppose that in the junior high school I will have to invest a larger
number of meetings with the students, than the number of meetings you
held with us.
Difficulties that were located by the staff during the assessment meetings
As said, the formative assessment strategy, implemented by us, allowed us to find
out the difficulties the students encountered in the course of performing the project.
Following are a number of quotations from the students’ reports and transcripts of the
meetings:
I had difficulties in processing and concluding the material that was
gathered.
In constructing the theoretical background, we had difficulties in
selecting the relevant material for the project out of the sources we
gathered.
We had doubts as to the level of detailing would have to be in writing
the theoretical background.
In order to perform the project, basic engineering now-how is
required, including familiarization with design and planning processes
as well as scientific knowledge for the purpose of defining and
understanding the scientific principles underlying this system.
It was difficult for us to define in advance the system requirements, and
we retrospectively defined them only after choosing the system and the
analysis of its components.
Understanding the manner by which the heart functions did not suffice
in order to define the design requirements for an artificial heart.
We did not understand the meaning of detailed design.
We had difficulties in choosing the system: we were looking for a
system in which the needs it meets would be easy to define.
We have encountered a number of technical problems, such as:
generating the required level of voltage for the system operation.
From analyzing the above quotations and the assessments, based on the rubric, it
seems that the difficulties can be sorted into six groups: problems in choosing a topic
for the project (different areas of expertise and desires), problems in implementing the
project (not finding alternatives for a solution, etc.), social problems (conflicts,
incompetent team members, etc.), problems related to submitting the group report (it
was not clear which level of profoundness is required, incompatibility between the
team members with regards to what each of them prepared, searching and locating
suitable information resources, etc.), problems related to the artifact (technical
difficulties, etc.), and problems related to preparation of the presentation (what to
present in the limited time allocated, etc.).
Conclusions
Summary of Findings
The study question was: what implementation issues and processes do higher
education students encounter in a PBL environment which involves an alternative
assessment approach?
Benefits of PBL. While performing the projects the students were exposed to five
benefits of PBL: gaining interdisciplinary knowledge, acquiring that knowledge
through active and experiential learning, taking responsibility for the learning,
acquiring communication skills and methods of decision making within a team, and
enhancing of one’s self-esteem.
Challenges of students in PBL. At the same time the students experienced three
challenges: coping with conflict situations in the teamwork, investing a lot of time and
efforts (both on the teacher’s side and the student’s side), and coping with new
contents in a learning environment which is neither structured nor organized in
advance (experiencing frustration in an environment of uncertainty).
The students’ attitude towards the formative assessment. The majority of the
students maintained that continuous assessment throughout the course advanced the
learning for six reasons: it helped to understand the course goals and requirements, it
assisted to evaluate the degree of progress, it helped to cope with difficulties and to
locate the points which require correction or improvement, it emphasized the need to
test additional aspects related to the project, it assisted to cope with conflicts among
the members of the team, and it allowed the course teachers to locate the students who
had difficulties. The students pointed at two difficulties: sometimes the course staff
had difficulties in assessing the personal contribution of each student, and the teachers
had to invest vast amount of time and effort in order to implement the alternative
assessment techniques.
Difficulties that were located by the staff during the assessment meetings. The
students difficulties can be sorted into six groups: problems in choosing a topic for
the project, problems in implementing the project, social problems, problems relating
to submitting the group report, problems referring to the artifact, and problems
relating to preparation of the presentation.
Implications for using PBL in course for training pre-service teachers
Integrating PBL in teaching requires training the teaching staff both in the content
knowledge as well as the pedagogical knowledge. In addition, integration of PBL in
teaching science and technology requires a teaching staff that is familiar with the
design process as well as with a scientific knowledge. Usually, novice teachers lack
the training necessary to teach PBL effectively mainly because they have not been
trained in this area.
While being involved in active experiencing of the project implementation in the
PBL environment, our students were exposed to the advantages and challenges of the
method. We are hopeful that this experience will assist them to implement the method
in the future, when they themselves become teachers of Science and Technology in
the junior high school. By looking at the study findings, it seems that indeed the
students were exposed to the benefits and challenges inherent in Problem-Based
Learning and in the accompanying assessment approach. The students acquired both
content knowledge as well as pedagogical knowledge. In the course of their active
experiencing, the students acquired interdisciplinary scientific-engineering
knowledge, and familiarized themselves with the design process.
Data analysis revealed that students found the alternative assessment more
suitable for a course based on PBL than traditional methods, and they indicated that
they would use this type of method in their teaching. According to the students,
implementation of formative assessment throughout the course would help to reduce
anxiety, identify and solve problems, monitor progress and guide them through the
project.
The students perceived the alternative assessment as a positive experience. They
felt that they had been given an opportunity to share, reflect on, and discuss their
ideas, and that this type of assessment should also be applied in other courses. The
results of this study can provide educators from different disciplines who are
interested in PBL with a set of assessment instrument as well as with some
implementation tips as to how to apply PBL in the classroom .
More than ever, universities are now expected to help students develop skills and
competencies in real-life "authentic situation” and to enable them to demonstrate
these abilities by performance on alternative assignments rather than standardized
tests. Our study findings show that PBL environment enables to realize this goal.
Since our research was a case study we recommend that the findings presented in this
paper will serve as a basis for follow up studies .
REFERENCES
ADLER, P. A. & ADLER, P. (1994) Observational Techniques, in: DENZIN N. K. & LINCOLN Y. S. (Eds) Handbook of Qualitative Research, (Thousand Oaks, CA,
Sage) .
BATTLE, J. (1991) Self-Esteem Research: A Summary of Relevant Findings, (Edmonton Canada, James Battle & Associates).
BIRENBAUM, M. (1997) Alternatives in Assessment, (Tel Aviv University, Ramot, in Hebrew).
BUCK Institute for Education (1999) PBL Overview. Available at: http://www.bie.org/pbl/overview/index.html
CPS – Chicago Public Schools website (2000) Analytical Vs. Holistic Rubrics. Available at: http://intranet.cps.k12.il.us/Assessment
DENZIN N. K. & LINCOLN Y. S. (2000) Handbook of Qualitative Research, (Thousand Oaks, CA, Sage) .
DENZIN, N. K. (1988) Triangulation, in: KEEVES, J. P. (Ed.) Educational Research, Methodology and Measurement: An International Handbook, (Oxford, Pergamon
Press).
DYER, W. G. (1987) Team Building, (New York, McGraw-Hill) .
GONELLA, F. (2001) Assessing Student Work with Project-Based Learning. Available at: http://pblmm.k12.ca.us/PBLGuide/AssessPBL.html
TAYLOR, P. C. & FRASER, B. J. (1991) CLES: An Instrument for Assessing Constructivist Learning Environments, paper presented at the Annual Meeting of
the National Association for Research in Science Teaching (Fontane, Wisconsin).
GOODRICH, H. (1997) Understanding Rubrics, Educational Leadership, 54 (4), pp. 14-18.
GREEN, A. M. (1998) Project-Based-Learning: Moving Students Toward Meaningful Learning, ERIC Database, ED422466.
HERTZ-LAZAROWITZ, R. (1990) An Integrative Model of the Classroom: The Enhancement of Cooperation in Learning, paper presented at the Annual Meeting of the American Educational Research Association (Boston, MA, April 17-20).
JEWELL, L. N. & REITZ, H. J. (1984) Group Effectiveness in Organizations, (New York, Warner Books).
KRAJCIK, J., CZERNIAK, C. & BERGER, C. (1999) Teaching Children Science: A Project-Based Approach (New York, McGraw-Hill College).
MISHLER, E. (1986) Research Interviewing: Context and Narrative, (Boston, Harvard University Press).
MORSE, J. M. (1994) Designing Funded Qualitative Research, in: DENZIN, N. K. & LINCOLN, Y. S. (Eds) Handbook of Qualitative Research, (Thousand Oaks, CA,
Sage) .
OREVI, N. & DANON R. (1999) Learning Ecology with Educational Technologies, paper presented at the International Workshop on Science Teachers Education toward the New Millennium (Haifa, Technion - Department of Education in
Technology and Science).
PARKER, M. G. (1990) Team Players and Team Work, (New York, Prentice Hall).
ROSENFELD, M. & ROSENFELD, S. (1999) Understanding the surprise in PBL: an exploration into the learning styles of teachers and their students, paper presented at the 8th Conference of EARLI - European Association for Research in Learning
and Instruction (Goteburg, Sweden, EARLI)
SHEPHERD, H. G. (1998) The Probe Method: A Project-Based-Learning Model’s effect on Critical Thinking Skills, Dissertation Abstracts International, Section A, 59 (3A), p. 779.
SPRADLEY, J. P. (1979) The Ethnographic Interview, (New York, Holt, Rinehart & Winston).
THOMAS, J. W. (2001) A Review of research on Project-Based-Learning. Available at: http://www.autodesk.com/foundation/pbl/research
Appendix A: Criteria of the Project-Based Technology Rubric (PBTR)
Dimension Criteria
Defining the problem / formulating the need for the system design.
The problem was clearly and unambiguously stated.
Defining the system requirements. The system objectives are clearly defined, are compatible with the problem definition and are attainable.
Theoretical scientific-technological background needed for the project.
The team has conducted sufficient literature review to support the project. All information is clear, appropriate and correct. Covers topic completely and in depth. Information includes scientific, technological and social aspects relevant to the project.
Project planning. The project planning includes:Presenting a number of alternatives (solutions) that meet the problem.
Presenting a number of advantages and disadvantages of each alternative.
Choosing an optimal solution based on technological, economic and environmental considerations, as well as schedules and anticipated difficulties.
Stages of the system design . The system design is presented in the Top Down method in two stages:
Preliminary design, which includes drafting of a block diagram accompanied by annotations and details of the relations between the sub-systems.
Detailed design, which includes flowchart and/or detailed explanation of each “block” in the sub-system of the hyper-diagram.
System description: Specifying the system components and its mode of operation .
The system mode of operation is explained in a clear manner and integrates drawings and illustrations. The system components are explained in details, commencing with a general explanation of the exhibit, and subsequently accompanied by a detailed explanation.
Description of the lab model: details of the model goal and its mode of operation.
The goal of the model is clearly defined. The information attached to the model is sufficient to build the model and understand its operation principle.
The scientific/technological principle intended to be demonstrated is coherently explained and illustrated through the model.
The model’s mode of operation is explicitly explained and integrates drawings and illustrations.
References. Complete list of all sources used.A variety of significant and relevant sources (internet sites, books), presented according to the rules of the bibliographic writing.
Documentation of the project stages
(Portfolio.)The documentation includes organized listing of all the meetings’ protocols. Each protocol presents:
A report regarding the activity performed by the team according to the design and the goals set forth in the previous meeting. Planning the tasks to be performed towards the
following meeting.Problems related to the project performance.
Appendix B: Sample Questions from the “Students’ Attitudes towards PBL” Questionnaire
Scale Name #of Item
Description Sample Item
Assessment Method
11 The extent to which the assessment method helped the student
The feedback I received during the project helped me:
To monitor my progress in the project.
To locate the points which require correction or improvement.
The feedback I received assisted me in the following stages of the project:
To formulate the system requirements.
To examine and choose alternative solutions.
Teaching Method
16 The extent of benefits gained by applying PBL as realized during the instruction process.
In light of my experience in the course, PBL:
Allowed me to cope with interdisciplinary knowledge.
Encourage active learning .